Positioning method, device, electronic device and computer-readable storage medium

Abstract

The embodiments of the present disclosure disclose a positioning method, device, electronic device and computer-readable storage medium, which perform object detection on a captured current image frame to obtain at least one object to be identified in the current image frame, and compare a determined target image set with the at least one object to be identified to determine the target object in the current image frame, and determine the position information of the target object according to the area of ​​the target object in the image frame and the area of ​​at least one designated object in the current image frame, thereby improving the accuracy of identifying and positioning the target object.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, specifically, to the field of positioning technology, and more specifically, to a positioning method, device, electronic device, and computer-readable storage medium. Background Art

[0002] With the development of the information society, the application of target recognition and positioning technology is becoming more and more extensive. For example, the target object is recognized and located through image processing to analyze and record the behavior of the target object.

[0003] In related technologies, the position of a target object is usually determined based on face recognition technology. However, since the data acquisition device may not be able to simultaneously capture the recognizable face area of ​​the target object, or the target object's posture changes and affects the face recognition result, inaccurate recognition results may occur, leading to inaccurate positioning. Summary of the invention

[0004] In view of this, the embodiments of the present disclosure provide a positioning method, an apparatus, an electronic device, and a computer-readable storage medium to improve the accuracy of identifying and positioning a target object.

[0005] In a first aspect, an embodiment of the present disclosure provides a positioning method, the method comprising:

[0006] Acquiring a current image frame, wherein a coordinate sequence of at least one specified object is marked in the current image frame;

[0007] Performing object detection on the current image frame to obtain at least one object to be identified in the current image frame;

[0008] Comparing a determined target image set with the at least one object to be identified to determine a region of the target object in the current image frame, the target image set comprising full-body images of the target object at multiple different angles;

[0009] The position information of the target object is determined according to the area of ​​the target object in the current image frame and the area of ​​at least one of the designated objects in the current image frame, and the area of ​​the designated object in the current image frame is determined according to the coordinate sequence of the designated object.

[0010] Optionally, determining the position information of the target object according to the area of ​​the target object in the current image frame and the area of ​​at least one of the designated objects in the current image frame includes:

[0011] Determine a pixel overlap ratio of a region of the target object in the current image frame and a region of each of the designated objects in the current image frame;

[0012] In response to the pixel overlap rate being greater than or equal to a predetermined value, it is determined that the target object is within an area defined by a designated object corresponding to the pixel overlap rate.

[0013] Optionally, determining the target image set includes:

[0014] Acquire an image frame group within a pre-acquisition time, wherein a time difference between the pre-acquisition time and a current time is within a predetermined time difference range, and the image frame group includes at least one image frame;

[0015] Performing full-body image detection on the image frames in the image frame group to obtain a plurality of full-body images in the image frame group;

[0016] Performing face detection on the multiple full-body images respectively to obtain a first image set, wherein the full-body images in the first image set include recognizable face areas;

[0017] Comparing the target object with the recognizable face regions of the full-body images in the first image set respectively to determine a candidate image set;

[0018] The target image set is determined according to the candidate image set.

[0019] Optionally, determining the target image set according to the candidate image set includes:

[0020] Clustering is performed on the full-body images in the candidate image set to obtain the target image set.

[0021] Optionally, comparing the target object with recognizable face regions of full-body images in the first image set respectively to determine the candidate image set includes:

[0022] Acquire at least one first feature vector corresponding to the target object;

[0023] Extracting features of recognizable face regions of the full-body images in the first image set to obtain corresponding multiple second feature vectors;

[0024] Calculating the similarity between each of the first feature vectors and each of the second feature vectors;

[0025] For each of the second feature vectors, in response to a similarity between the second feature vector and at least one first feature vector being greater than or equal to a similarity threshold, a full-body image corresponding to the second feature vector is added to the candidate image set.

[0026] Optionally, performing object detection on the current image frame to obtain at least one to-be-identified object in the current image frame includes:

[0027] Inputting the current image frame into a target recognition network for feature extraction to obtain a feature map of the current image frame;

[0028] Perform object detection on the feature map to obtain at least one object to be identified in the current image frame.

[0029] Optionally, the target recognition network is trained and acquired according to an acquired training data set;

[0030] The training data set includes a plurality of unobstructed object images and semi-obstructed object images.

[0031] Optionally, the target recognition network includes a first direction mask branch group and a second direction mask branch group;

[0032] The first direction mask branch includes a first direction random return to zero module, and the second direction mask branch includes a second direction random return to zero module;

[0033] The first direction random zeroing module is used to zero a feature map of a predetermined width along the first direction in the acquired image feature map;

[0034] The second direction random zeroing module is used to zero the feature map of a predetermined width along the second direction in the acquired feature map;

[0035] Wherein, the position of the feature map of the predetermined width is random.

[0036] In a second aspect, an embodiment of the present disclosure provides a positioning device, the device comprising:

[0037] A collection unit, configured to collect a current image frame, wherein a coordinate sequence of at least one specified object is marked in the current image frame;

[0038] a to-be-recognized object determination unit, configured to perform object detection on the current image frame to obtain at least one to-be-recognized object in the current image frame;

[0039] a target object determination unit, configured to compare the determined target image set with the at least one object to be identified, and determine the target object in the at least one object to be identified, wherein the target image set includes full-body images of the target object at multiple different angles;

[0040] A positioning unit is configured to determine the position information of the target object based on the area of ​​the target object in the image frame and the area of ​​at least one of the designated objects in the current image frame, wherein the area of ​​the designated object in the current image frame is determined according to the coordinate sequence of the designated object.

[0041] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising a memory and a processor, wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method as described above.

[0042] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the method as described above.

[0043] The disclosed embodiment performs object detection on the acquired current image frame to obtain at least one object to be identified in the current image frame, and compares a determined target image set with the at least one object to be identified to determine the target object in the current image frame, and determines the position information of the target object according to the area of ​​the target object in the image frame and the area of ​​at least one designated object in the current image frame, thereby improving the accuracy of identifying and locating the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0045] Figure 1 is a flow chart of a positioning method according to an embodiment of the present disclosure;

[0046] Figure 2 is a flow chart of a method for determining a target image set according to an embodiment of the present disclosure;

[0047] Figure 3 is a flowchart of a method for determining a candidate image set according to an embodiment of the present disclosure;

[0048] Figure 4 is a schematic diagram of a target recognition network according to an embodiment of the present disclosure;

[0049] Figure 5 is a working diagram of a first direction random zeroing module according to an embodiment of the present disclosure;

[0050] Figure 6 is a working schematic diagram of a second direction random zeroing module according to an embodiment of the present disclosure;

[0051] Figure 7 is a scene schematic diagram of a positioning method according to an embodiment of the present disclosure;

[0052] Figure 8 is a schematic diagram of a positioning device according to an embodiment of the present disclosure;

[0053] Fig. 9 is a schematic diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0054] The present disclosure is described below based on embodiments, but the present disclosure is not limited to these embodiments. In the detailed description of the present disclosure below, some specific details are described in detail. It is possible for a person skilled in the art to fully understand the present disclosure without the description of these details. In order to avoid confusing the essence of the present disclosure, known methods, processes, flows, components and circuits are not described in detail.

[0055] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.

[0056] Unless the context clearly requires otherwise, words such as “include”, “including” and the like in the specification should be interpreted as including rather than being exclusive or exhaustive; that is, as meaning “including but not limited to”.

[0057] In the description of the present disclosure, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present disclosure, unless otherwise specified, the meaning of "plurality" is two or more.

[0058] In some application scenarios, the behavior of the target object can be judged by identity recognition and positioning technology. For example, the behavior of teachers and students in the classroom: if the teacher is in the blackboard area, he may be teaching, and if the teacher is in the student area, he may be patrolling, etc. Another example is to locate a designated dancer to determine the behavior trajectory of the designated dancer on the stage. At present, in order to achieve this demand, face recognition technology can be relied on, that is, the front face image of the target object is collected in advance, and then the face image information of the target object is detected from the pictures from multiple data acquisition devices, so as to determine the location of the target object. It is also possible to rely on pedestrian re-identification (Re-Identification, ReID) technology, that is, the full-body image of the target object is collected in advance, and then the full-body image information of the target object is detected from the pictures from multiple data acquisition devices, so as to determine the location of the target object. Among them, in the technology based on face recognition, due to angle reasons, the target object may not be detected because the front face cannot be captured or the angle of the collected image affects the face recognition result, so that the position of the target object cannot be determined. Based on the ReID technology, the whole body image of the human body is used as the comparison target, which solves some of the drawbacks brought by the use of face recognition technology. However, the ReID technology is sensitive to the clothing of the target object. Different clothing of the same target object will affect the recognition of the system, which may also lead to the inability to correctly identify and locate the target object. Therefore, this embodiment provides a positioning method, which enables the target object to be identified and located more accurately.

[0059] The smart classrooms that have emerged in recent years not only have intelligent audio-visual multimedia teaching hardware, but also require the ability to record and analyze the behavior of teachers and students in the classroom in real time to provide basic data for subsequent evaluation. In addition to standing in the blackboard area to teach in the classroom, the teacher will also patrol the student area in the classroom. In some classrooms with special layouts, the teacher will also discuss with students in the side auxiliary blackboard area. As one of the basic data of classroom behavior information, the most direct way to obtain the teacher's location information is to automatically filter out the teacher's location by analyzing the images from multiple data acquisition devices, and then judge the teacher's current behavior information. This embodiment is described using a smart classroom as an example. It should be understood that this embodiment is not limited to this. Any scene with a data acquisition device and a specified object can apply the positioning method of this embodiment, such as a conference room, a stage, and the like.

[0060] Figure 1 is a flow chart of a positioning method according to an embodiment of the present disclosure. The positioning method can be executed by a terminal, a server or other information processing device, wherein the terminal can be a terminal with a data acquisition device (for example, a smart camera, etc.), and the present disclosure does not limit this. Figure 1 As shown, the positioning method of this embodiment includes the following steps:

[0061] Step S100, capturing the current image frame. The current image frame has a coordinate sequence of at least one designated object calibrated therein. In this embodiment, the data acquisition device may be a fixed data acquisition device. For example, in a smart classroom, the data acquisition device may be a monitoring device in the smart classroom, and the designated objects may include fixed locations in the classroom such as a blackboard and a classroom corridor. Since the data acquisition device is a fixed device, in the image frame captured by it, the area of ​​the designated objects such as the blackboard in the image frame is fixed, and thus, the designated objects such as the blackboard and the corridor may be calibrated in advance in the corresponding data acquisition device.

[0062] When the positioning method is executed by a server or other information processing device, the current image frame sent by other terminals (e.g., smart cameras) can be collected. When the positioning method is executed by a terminal device, for example, when the terminal is a smart camera, the current image frame can be collected by the smart camera.

[0063] Step S200, performing object detection on the current image frame to obtain at least one object to be identified in the current image frame. Optionally, performing feature extraction on the current image frame to obtain a feature map of the current image frame, performing object detection on the feature map of the current image frame to obtain at least one object to be identified in the current image frame. For example, in a smart classroom, the current image frame includes multiple students and teachers, and object detection is performed on the feature map of the current image frame to determine the areas of multiple objects to be identified (including teachers and students) in the current image frame.

[0064] Step S300, compare the determined target image set with at least one object to be identified, and determine the area of ​​the target object in the current image frame. The target image set includes full-body images of the target object at multiple different angles. Since the angle of the target object may be different at different times, the image angle of the target object in the current image frame is uncertain. Therefore, in this embodiment, by using full-body images of target images at multiple different angles to perform similarity comparison with at least one object to be identified, the accuracy of object recognition can be improved.

[0065] In an optional implementation, the time difference between the acquisition time of the target image set and the current time is within a predetermined time difference range. Optionally, the acquisition time of the target image set can be the beginning stage of an activity (such as the start time of a class). Since the clothing of the target object in an activity basically does not change, the target object in the currently acquired image frame is determined based on the target image set at a later time of the activity, which can further improve the accuracy of target object recognition.

[0066] Step S400, determining the position information of the target object according to the area of ​​the target object in the current image frame and the area of ​​at least one designated object in the current image frame. The area of ​​the designated object in the current image frame is determined according to the coordinate sequence of the designated object. In this embodiment, the position information of the designated object is fixed, and the area of ​​the designated object in the image frame collected by the data acquisition device is also fixed. Therefore, this embodiment determines the position information of the target object according to the area of ​​the designated object in the current image frame and the area of ​​the target object in the current image frame, which can improve the positioning accuracy of the target object.

[0067] In an optional implementation, step S400 may include: determining a pixel overlap ratio between an area of ​​the target object in the current image frame and an area of ​​each designated object in the current image frame, and in response to a pixel overlap ratio among the pixel overlap ratios being greater than or equal to a predetermined value, determining that the target object is within an area defined by the designated object corresponding to the pixel overlap ratio. In this embodiment, a coordinate sequence of each designated object is pre-marked in the image frame collected by the data acquisition device, and therefore, the area R where each designated object is located isa After the target object is determined, the region R of the target object in the image collected by the corresponding data acquisition device is determined. o . Calculate the area R where the target object is located o The region R where a specified object A is located a The pixel overlap ratio T a =area(R a ∩R o ) / area(R a ). At the pixel overlap rate T a When it is greater than or equal to the predetermined value, it is determined that the target object is within the area defined by the designated object A. For example, assuming the predetermined value is 50%, the designated object includes the blackboard, and the area defined by the blackboard is the podium. If the overlap rate between the area of ​​the teacher in the image and the area of ​​the blackboard in the image is 55%, it can be determined that the teacher is on the podium, and the teacher may be giving a lecture at this time.

[0068] In an optional implementation, if there are multiple designated objects and the pixel overlap ratios corresponding to the target object are greater than or equal to a predetermined value, or all pixel overlap ratios are less than a preset value, then it is determined that the target object is located in the area defined by the designated object corresponding to the maximum pixel overlap ratio. The present disclosure does not limit the method of determining the position information of the target object based on the area of ​​the target object in the current image frame and the area of ​​at least one of the designated objects in the current image frame.

[0069] This embodiment performs object detection on the acquired current image frame to obtain at least one object to be identified in the current image frame, and compares the determined target image set with the at least one object to be identified to determine the target object in the current image frame, and determines the position information of the target object according to the area of ​​the target object in the image frame and the area of ​​at least one designated object in the current image frame, thereby improving the accuracy of identifying and locating the target object.

[0070] Figure 2 : is a flow chart of a method for determining a target image set according to an embodiment of the present disclosure. In an optional implementation, the positioning method of this embodiment further includes determining a target image set. For example, the target image set can be determined based on a group of image frames collected within a pre-collection time. The various states of the target object in the target image set determined based on the group of image frames collected within the pre-collection time are basically the same as those in the current image frame, and the accuracy of identifying the target object in combination with the target image set is high. Figure 2 As shown, the method for determining the target image set of this embodiment may include the following steps:

[0071] Step S210, collect image frame groups within the pre-collection time. Wherein, the time difference between the pre-collection time and the current time is within a predetermined time difference range, and the image frame group includes at least one image frame. Optionally, since the clothing, hairstyle, etc. of the target object will change over time, the pre-collection time can be the initial stage after the start of an activity, and the predetermined time difference range is related to the activity time of the activity. For example, in the field of smart classrooms, the pre-collection time can be the beginning stage of the first class of the teacher that day, and the preset time difference range can be the working hours of the teacher for a day. Assuming that the start time of the first class of teacher Q is 8 o'clock in the morning, the image frame group can be collected between 8 o'clock and 8:10 in the morning (pre-collection time), and the target image set determined based on the image frame group is used as the image comparison library of the teacher Q on that day. Thus, this embodiment realizes that the clothing and other states of the target object in the target image set are basically the same as those in the current image frame, thereby further improving the accuracy of target object recognition.

[0072] Step S220, performing full-body image detection on the image frames in the image frame group to obtain multiple full-body images in the image frame group. Optionally, performing full-body image detection on each image frame in the image frame group to obtain a full-body image of the object in each image frame. For example, in a smart classroom scenario, at the beginning of a class, an image frame group is collected, and a plurality of full-body images (which may include multiple full-body images of the teacher and multiple full-body images of the students) are obtained by performing full-body image detection on each image frame in the image frame group. It should be understood that in this embodiment, the full-body image does not specifically refer to an image including the full view of the object, but an image of all the features of the object that can be displayed in the image frame. For example, the full-body image of a teacher in a classroom does not specifically refer to all areas including the head, upper body, lower body, etc. If the lower body area is blocked by the table, the image including the head and upper body area can also be used as the full-body image of the teacher.

[0073] In an optional implementation, a target detection method may be used to perform full-body image detection on the image frames in the image frame group, such as YOLO, SSD (Single Shot MultiBox Detector) and other target detection methods. The present disclosure does not limit the method of full-body image detection.

[0074] Step S230, face detection is performed on the multiple full-body images obtained above to obtain a first image set. The full-body images in the first image set include recognizable face areas. In this embodiment, the target images in the target image set include recognizable face areas of the target objects. Therefore, face detection is performed on the multiple full-body images obtained by full-body image detection to screen full-body images with recognizable face areas. In an optional implementation method, the full-body images in the image frames can be captured, and face recognition is performed on the multiple full-body images by a face detection method (such as MTCNN, face key point recognition and other face detection methods), and the full-body images corresponding to the recognizable face areas are added to the first image set.

[0075] Step S240 , comparing the target object with the recognizable face regions of the full-body images in the first image set, to determine a candidate image set.

[0076] Figure 3 FIG. 1 is a flow chart of a method for determining a candidate image set according to an embodiment of the present disclosure. In an optional implementation, as shown in FIG. Figure 3 As shown, step S240 may include steps S241-S244:

[0077] Step S241, obtaining at least one first feature vector corresponding to the target object. In an optional implementation, the feature vector of at least one face area of ​​the target object is obtained from the corresponding database. That is to say, the feature vector of at least one face area image of the target object is obtained in advance, and these feature vectors are stored in the corresponding database, wherein the face angles in each face area image of the target object are different. In another optional implementation, the feature vector of at least one face area image of the target object is obtained in real time. Optionally, a face recognition network (such as DeepFace network, DeepID network, FaceNet network and other face recognition networks) can be used to extract features from the face area image of the target object to obtain the corresponding first feature vector.

[0078] Step S242, extract features from the recognizable face areas of the full-body images in the first image set to obtain corresponding multiple second feature vectors. Optionally, the recognizable face areas obtained in step S230 are input into a face recognition network for feature extraction to obtain multiple second feature vectors corresponding to the multiple recognizable face areas. It should be understood that there is no order of execution for step S242 and step S241, and step S242 can be executed after step S241, before step S241, or simultaneously with step S241, and this embodiment does not limit this.

[0079] Step S243, calculating the similarity between each first eigenvector and each second eigenvector, that is, calculating the similarity between each first eigenvector and each second eigenvector.

[0080] Step S244, for each second feature vector, in response to the similarity between the second feature vector and at least one first feature vector being greater than or equal to the similarity threshold, the full-body image corresponding to the second feature vector is added to the candidate image set. That is to say, when the similarity between the above-mentioned recognizable face region and the face region of one or more target objects is greater than or equal to the similarity threshold, the face in the recognizable face region can be determined as the target object. In the first image set, each recognizable face region has a corresponding full-body image. Therefore, when a recognizable face region is determined to be a target object, its corresponding full-body image is the full-body image of the target object.

[0081] In an optional implementation, the candidate image set includes a predetermined number of full-body images of the target object. The candidate image set determination method in this embodiment may also include step S245 ( Figure 3 (not shown): in response to the number of images in the candidate image set reaching a predetermined number, outputting the candidate image set.

[0082] This embodiment can improve the accuracy of face recognition by performing a similarity comparison between the recognizable face regions of each full-body image in the first image set and the face regions of target objects at multiple different angles.

[0083] Step S250, determining a target image set according to the candidate image set. In an optional implementation, clustering is performed on the whole body images in the candidate image set to obtain the target image set.

[0084] Since the recognition accuracy is higher if there are multiple full-body images of the target object in the target image set and the body orientations and postures in the full-body images are different, the present embodiment clusters the images in the candidate image set to obtain images with as large a difference as possible and add them to the target image set.

[0085] Wherein, step S250 includes the following steps:

[0086] Step S251, pre-set the number of images in the target image set to k, and assume that the number of candidate images in the candidate image set is m, where m is greater than or equal to k. The candidate image set is {x 1 ,x 2 ,…,x m}. Randomly initialize the set {μ1,μ2,…,μ k} are the k cluster centers of m candidate images, and set {c 1 ,c2 ,…,c m} respectively represent the category labels corresponding to the m candidate images.

[0087] Step S252, update the category label: for each c i , i = 1, 2, ..., m, calculate the distance between the corresponding candidate image and each cluster center, obtain the category with the smallest distance and update c i ,Right now:

[0088]

[0089] Wherein, j=1, 2, ..., k, Distance(·) represents an image similarity comparison method, which may use a traditional image hash comparison method or a neural network-based image similarity comparison method, and this embodiment does not limit this.

[0090] Step S553, update the cluster center:

[0091]

[0092] Where I(·) is an indicator function, which takes the value of 1 only when the internal equation holds, otherwise it takes the value of 0.

[0093] Repeat steps S252-S253 until the cluster centers of the k categories no longer change significantly, and obtain each cluster center to determine the target image set. Thus, the target image set can include full-body images of the target object at multiple angles, further improving the accuracy of object recognition.

[0094] This embodiment is based on the face images of the target object including multiple angles, and uses a face recognition network to screen out the full-body images of the target object from multiple full-body images to determine a candidate image set, and clusters the candidate image set to determine a target image set, which can make the target image set include full-body images of the target object at multiple angles. Therefore, when the images in the target image set are used as the target comparison library for the current target object recognition, the object recognition accuracy can be further improved, and then the object positioning accuracy can be further improved. As long as the target object image with a large difference can be obtained and used as the target object, the present disclosure does not limit the clustering method.

[0095] In an optional implementation, the present embodiment identifies the target object through a pre-trained target recognition network. That is, in the present embodiment, the acquired current image frame can be input into the target recognition network, and the target recognition network determines the target object in the current image frame according to the determined target image set. For example, the target recognition network can be used to extract features of the current image frame to obtain a feature map of the current image frame, and object detection can be performed on the feature map of the current image frame to obtain at least one object to be identified in the current image frame, and then each target image in the target image set is respectively compared with each object to be identified to determine the area of ​​the target object in the current image frame.

[0096] Optionally, the target recognition network of this embodiment is trained and acquired based on a pre-created training data set, and the training data set may include multiple training images. Among them, the training image is an unobstructed object image or a semi-obstructed object image. In an optional implementation method, taking the smart classroom scene as an example, by collecting the teaching video of the teacher in the classroom, and extracting the full-body image of the object (such as the teacher) in the image frame in the video without occlusion (including the head, upper body and lower body area) or semi-occluded (the lower body is blocked by the desk, etc.) as a training data set. Thus, by using the data training set obtained in the corresponding scene to train the target recognition network, the recognition accuracy of the target recognition network can be further improved.

[0097] Optionally, the training data set includes positive samples, negative samples and target samples, wherein the positive samples include full-body images of the same object as the target samples at multiple different angles, and the negative samples include full-body images of different objects from the target samples at multiple different angles, and the training makes the similarity between the positive samples and the target samples greater than a first preset value, and the similarity between the negative samples and the target samples less than a second preset value, so that the trained target recognition network can recognize the target object in the current image frame based on the full-body images of the target object at different angles. The first preset value is greater than or equal to the second preset value.

[0098] Figure 4 FIG. 1 is a schematic diagram of a target recognition network according to an embodiment of the present disclosure. Figure 4 As shown, the target recognition network 4 of this embodiment includes a global branch 41, a first direction mask branch group 42, a second direction mask branch group 43 and a feature extraction network 44. Among them, the feature extraction network 44 is used to extract features from the input image frame to obtain a corresponding feature map. Optionally, the feature extraction network 44 of this embodiment can be based on a network structure VGG, a network structure Mobilene or a network structure Resnet, etc., and this embodiment is not limited to this. Optionally, the first direction is a horizontal direction and the second direction is a vertical direction.

[0099] In this embodiment, the horizontal mask branch group 42 includes at least one horizontal mask branch, and the vertical mask branch group 43 includes at least one vertical mask branch. Figure 4 As shown, each branch (global branch, horizontal mask branch, and vertical mask branch) may include a global average pooling layer and a 1*1 convolution layer.

[0100] Each branch uses its own loss function. The loss function of the target recognition network 4 of this embodiment can be:

[0101]

[0102] Among them, L G is the loss function in the global branch, L Hh is the loss function of the h-th horizontal mask branch, L Vl is the loss function of the l-th vertical mask branch, 1≤h≤N, 1≤l≤M, α G , α H , α V are weight parameters of the global branch 41, the horizontal mask branch group 42 and the vertical mask branch group 43, respectively. G +α H +α V =1.

[0103] In an optional implementation, each loss function (L G , L H and L V ) is composed of a Triplet function and a Softmax function. The Triplet function can be:

[0104]

[0105] in, and denote target samples, positive samples, and negative samples respectively, the hyperparameter α denotes the isolation boundary, and D(·) denotes the measurement method, such as cosine similarity or Euclidean distance.

[0106] The Softmax function can be:

[0107]

[0108] Among them, f k Represents the input image.

[0109] In this embodiment, each horizontal mask branch further includes a horizontal random zeroing module ah (1≤h≤N), and each vertical mask branch further includes a vertical random zeroing module bl (1≤l≤M). The random zeroing module is equivalent to introducing noise during the training process, so that the target recognition network obtained by training can more accurately recognize the occluded image.

[0110] like Figure 4 As shown, the horizontal mask branch group 42 includes N horizontal mask branches, and the vertical mask branch group 43 includes M vertical mask branches. That is, in this embodiment, the feature map output by the feature extraction network 44 is input to the global branch 41, the horizontal mask branch group 42 and the vertical mask branch group 43 respectively.

[0111] Figure 5 1 is a schematic diagram of the operation of the first direction random zeroing module of an embodiment of the present disclosure. In this embodiment, the horizontal random zeroing module ah is used to zero the feature map of the acquired image feature map along a predetermined width in the horizontal direction. Figure 5 As shown, taking N=2 (i.e., the horizontal mask branch group includes 2 horizontal mask branches) as an example, in the horizontal mask branch group 42, the feature map is equally divided into 2 horizontal branches. N = 4 sub-feature maps 51-54, and then randomly add a height of ε in each sub-feature map h The eigenvalues ​​of the feature map are reset to zero. Figure 5 ε h = 2 as an example. As shown in the sub-feature graph 51, the feature values ​​of the 3rd and 4th rows are reset to zero.

[0112] Figure 6 FIG. 1 is a schematic diagram of the operation of the second direction random zeroing module of an embodiment of the present disclosure. In this embodiment, the vertical random zeroing module bl is used to zero the feature map of the acquired image feature map along the vertical direction with a predetermined width. The position of the feature map with the predetermined width is random. Figure 6 As shown, taking M=2 (i.e., the vertical mask branch group 43 includes 2 vertical mask branches) as an example, the feature map is equally divided into 2 along the vertical direction. M = 4 sub-feature maps 61-64, and then randomly add a width of ε in each sub-feature map w The eigenvalues ​​of the feature map are reset to zero. Figure 6 ε w = 1 as an example. For example, in the sub-feature graph 61, the first column in the sub-feature graph 61 is randomly selected, and the feature value of the first column in the sub-feature graph 61 is reset to zero.

[0113] This embodiment trains the target recognition network by extracting unobstructed object images and semi-occluded object images from the video of the corresponding scene, and sets a horizontal random zeroing module in each horizontal mask branch and a vertical random zeroing module in each vertical mask branch to introduce noise during the training process, so that various occluded images can be identified more accurately, further improving the accuracy of identifying and locating target objects.

[0114] Figure 7 is a scene diagram of a positioning method according to an embodiment of the present disclosure. This embodiment is described by taking a smart classroom as an example. Figure 7 As shown, in a classroom, there are data acquisition devices 71 in front and behind the classroom. Figure 7 The image frames shown are image frames acquired by the data acquisition device 71 at the back of the classroom. Among them, the coordinate sequence of the blackboard 72 is pre-calibrated in the image frames acquired by the data acquisition device 71. In this embodiment, at the beginning of the class, the target image set of the teacher is determined by whole-body image detection, face detection, face recognition and clustering processing. The image frames acquired by the data acquisition device 71 are input into the corresponding target recognition network to obtain the area of ​​each object to be identified (including the teacher 73, students 74-76) in the image frame, and the teacher 73 among the objects to be identified is determined based on the target image set of the teacher. Among them, the target image set is based on Figure 2 and Figure 3 Then, the overlapping area Rao is determined based on the area Ra where the teacher 73 is located and the area Ro where the blackboard is located, area(Rao)=area(R a ∩R o ) Calculate the region R where teacher 73 is located o The area R where the blackboard is located a The pixel overlap ratio T a =area(Rao) / area(R a )=60%, thus, the overlap rate between the area of ​​the teacher 73 in the image frame and the area of ​​the blackboard in the image frame is 60%, assuming that the predetermined value is 50%, and the area defined by the blackboard is the podium. It can be determined that the teacher is on the podium, and the teacher may be giving a lecture at this time.

[0115] Therefore, based on the positioning method of this embodiment, the teacher's position in the classroom and the length of time in the classroom, such as the time at the podium, the time in the classroom corridor, etc., can be more accurately determined according to the video or image frames collected by each data acquisition device in the classroom, thereby more accurately determining the teacher's behavior.

[0116] In an optional implementation, the pre-acquired target image set and the trained target recognition network parameters are transmitted to a data acquisition device, so that the data acquisition device determines the position of the teacher in real time based on the acquired image frames, the target image set, and the target recognition network. In another optional implementation, the pre-acquired target image set and the trained target recognition network parameters are transmitted to a preset server or other information processing device, and the data acquisition device acquires image frames in real time and transmits the image frames to the server or other information processing device, and the server or other information processing device determines the position of the classroom based on the acquired image frames, the target image set, and the target recognition network. It should be understood that this embodiment does not limit the execution subject of the above-mentioned positioning method, and any device that can execute the steps of the above-mentioned positioning method can be applied to this embodiment.

[0117] This embodiment determines the target image set by the image frames collected during the pre-collection time, and the target image set serves as the target comparison library of the target recognition network. Among them, the target object set includes full-body images of the target object at multiple angles or postures, which enables the target recognition network to recognize target objects at various angles or postures, thereby improving the accuracy of target recognition. Moreover, in this embodiment, the pre-collection time can be the beginning stage of an activity, so the target object in the target image set is basically consistent with the clothing, hairstyle and other states of the currently located target object, thereby further improving the accuracy of target recognition.

[0118] Figure 8 Schematic diagram of a positioning device according to an embodiment of the present disclosure. Figure 8 As shown, the positioning device 8 of this embodiment includes a collection unit 81 , a to-be-identified object determination unit 82 , a target object determination unit 83 and a positioning unit 84 .

[0119] The acquisition unit 81 is configured to acquire a current image frame in which a coordinate sequence of at least one designated object is marked.

[0120] The object to be identified determination unit 82 is configured to perform object detection on the current image frame to obtain at least one object to be identified in the current image frame. In an optional implementation, the object to be identified determination unit 82 may be further configured to input the current image frame into a target recognition network to perform feature extraction to obtain a feature map of the current image frame, perform object detection on the feature map to obtain at least one object to be identified in the current image frame.

[0121] The target object determination unit 83 is configured to compare the determined target image set with the at least one object to be identified to determine the area of ​​the target object in the current image frame, wherein the target image set includes full-body images of the target object at multiple different angles.

[0122] The positioning unit 84 is configured to determine the position information of the target object according to the area of ​​the target object in the image frame and the area of ​​at least one of the designated objects in the current image frame, and the area of ​​the designated object in the current image frame is determined according to the coordinate sequence of the designated object.

[0123] In an optional implementation, the positioning unit 84 includes a pixel overlap ratio determination subunit 841 and a positioning subunit 842. The pixel overlap ratio determination subunit 841 is configured to determine the pixel overlap ratio of the area of ​​the target object in the current image frame and the area of ​​each of the designated objects in the current image frame. The positioning subunit 842 is configured to determine that the target object is within the area defined by the designated object corresponding to the pixel overlap ratio in response to the pixel overlap ratio being greater than or equal to a predetermined value.

[0124] In an optional implementation, the positioning device 8 of this embodiment further includes a target image set determination unit 85. The target image set determination unit 85 is configured to determine a target image set. Optionally, the target image set determination unit 85 includes a collection subunit 851, a full body image detection subunit 852, a face detection subunit 853, a candidate image set determination subunit 854, and a target image set determination subunit 855.

[0125] The acquisition subunit 851 is configured to acquire an image frame group within a pre-acquisition time, wherein the time difference between the pre-acquisition time and the current time is within a predetermined time difference range, and the image frame group includes at least one image frame. The full-body image detection subunit 852 is configured to perform full-body image detection on the image frames in the image frame group to obtain a plurality of full-body images in the image frame group. The face detection subunit 853 is configured to perform face detection on the plurality of full-body images respectively to obtain a first image set, wherein the full-body images in the first image set include recognizable face regions. The candidate image set determination subunit 854 is configured to compare the target object with the recognizable face regions of the full-body images in the first image set respectively to determine a candidate image set. The target image set subunit 855 is configured to determine the target image set based on the candidate image set. Optionally, the target image set subunit 855 can be further configured to perform clustering processing on the full-body images in the candidate image set to obtain the target image set.

[0126] In an optional implementation, the candidate image set determination subunit 854 includes a first feature vector acquisition module 8541, a second feature vector acquisition module 8542, a similarity calculation module 8543 and a candidate image set determination module 8544. The first feature vector acquisition module 8541 is configured to acquire at least one first feature vector corresponding to the target object. The second feature vector acquisition module 8542 is configured to perform feature extraction on the recognizable face area of ​​the full-body image in the first image set to obtain a corresponding plurality of second feature vectors. The similarity calculation module 8543 is configured to calculate the similarity between each of the first feature vectors and each of the second feature vectors. The candidate image set determination module 8544 is configured to, for each of the second feature vectors, in response to the similarity between the second feature vector and at least one first feature vector being greater than or equal to a similarity threshold, add the full-body image corresponding to the second feature vector to the candidate image set.

[0127] In an optional implementation, the positioning device of this embodiment further includes a training unit 86, which is configured to train a target recognition network according to an acquired training data set. Wherein, the training data set includes a plurality of unobstructed object images and semi-obstructed object images. The target recognition network includes a first direction mask branch group and a second direction mask branch group; the first direction mask branch includes a first direction random zeroing module, and the second direction mask branch includes a second direction random zeroing module; the first direction random zeroing module is used to zero the feature map of a predetermined width in the acquired image feature map along the first direction; the second direction random zeroing module is used to zero the feature map of a predetermined width in the acquired feature map along the second direction; wherein the position of the feature map of the predetermined width is random.

[0128] The disclosed embodiment performs object detection on the acquired current image frame to obtain at least one object to be identified in the current image frame, and compares a determined target image set with the at least one object to be identified to determine the target object in the current image frame, and determines the position information of the target object according to the area of ​​the target object in the image frame and the area of ​​at least one designated object in the current image frame, thereby improving the accuracy of identifying and locating the target object.

[0129] Fig. 9 Schematic diagram of an electronic device according to an embodiment of the present disclosure. Fig. 9As shown, the electronic device 9 includes at least one processor 91, a memory 92 connected to the processor 91, and a communication component 93 connected to the scanning device, the communication component 93 receives and sends data under the control of the processor 91, wherein the memory 92 stores instructions executable by at least one processor 91, and the instructions are executed by at least one processor 91 to implement the positioning method of any of the above embodiments. The processor 91 is a CPU processor or an accelerated processor (such as a GPU processor, a multi-core processor, etc.).

[0130] Specifically, the electronic device 9 includes: one or more processors 91 and a memory 92, Fig. 9 In the example, a processor 91 is included, and the processor 91 is used to execute at least one step of the positioning method in this embodiment. The processor 91 and the memory 92 can be connected via a bus or other means. Fig. 9 In the example, the bus connection is used. The memory 92 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The processor 91 executes various functional applications and data processing of the device by running the non-volatile software programs, instructions and modules stored in the memory 92, that is, the positioning method of the embodiment of the present disclosure is implemented.

[0131] The memory 92 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store a list of options, etc. In addition, the memory 92 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 92 may optionally include a memory remotely arranged relative to the processor 91, and these remote memories may be connected to an external device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0132] The memory 92 stores one or more units, and when one or more units are executed by the processor 91, the positioning method in any of the above method implementations is executed.

[0133] This embodiment performs object detection on the acquired current image frame to obtain at least one object to be identified in the current image frame, and compares the determined target image set with the at least one object to be identified to determine the target object in the current image frame, and determines the position information of the target object according to the area of ​​the target object in the image frame and the area of ​​at least one designated object in the current image frame, thereby improving the accuracy of identifying and locating the target object.

[0134] Another embodiment of the present disclosure relates to a non-volatile storage medium for storing a computer-readable program, wherein the computer-readable program is used for a computer to execute part or all of the above method embodiments.

[0135] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0136] The above-mentioned product can execute the method provided in the embodiment of the present disclosure, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present disclosure.

[0137] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A positioning method, characterized in that: The method comprises: Acquiring a current image frame, wherein a coordinate sequence of at least one specified object is marked in the current image frame; Performing object detection on the current image frame to obtain at least one object to be identified in the current image frame; Comparing a determined target image set with the at least one object to be identified to determine a region of the target object in the current image frame, the target image set comprising full-body images of the target object at multiple different angles; Determine the position information of the target object according to the area of ​​the target object in the current image frame and the area of ​​at least one of the designated objects in the current image frame, wherein the area of ​​the designated object in the current image frame is determined according to the coordinate sequence of the designated object; The step of comparing the determined target image set with the at least one object to be identified is specifically to perform a similarity comparison between full-body images of target images at multiple different angles in the target image set and the at least one object to be identified; The target image set is obtained based on a set of full-body images matching the facial image of the target object in the image frame group acquired within the pre-acquisition time.

2. The method according to claim 1, characterized in that Determining the position information of the target object according to the area of ​​the target object in the current image frame and the area of ​​at least one of the specified objects in the current image frame includes: Determine a pixel overlap ratio of a region of the target object in the current image frame and a region of each of the designated objects in the current image frame; In response to the pixel overlap rate being greater than or equal to a predetermined value, it is determined that the target object is within an area defined by a designated object corresponding to the pixel overlap rate.

3. The method according to claim 1 or 2, characterized in that: Determining the target image set includes: Acquire an image frame group within a pre-acquisition time, wherein a time difference between the pre-acquisition time and a current time is within a predetermined time difference range, and the image frame group includes at least one image frame; Performing full-body image detection on the image frames in the image frame group to obtain a plurality of full-body images in the image frame group; Performing face detection on the multiple full-body images respectively to obtain a first image set, wherein the full-body images in the first image set include recognizable face areas; Comparing the target object with the recognizable face regions of the full-body images in the first image set respectively to determine a candidate image set; The target image set is determined according to the candidate image set.

4. The method according to claim 3, characterized in that Determining the target image set according to the candidate image set comprises: Clustering is performed on the full-body images in the candidate image set to obtain the target image set.

5. The method according to claim 3 or 4, characterized in that: Comparing the target object with the recognizable face regions of the full-body images in the first image set respectively, and determining the candidate image set comprises: Acquire at least one first feature vector corresponding to the target object; Extracting features of recognizable face regions of the full-body images in the first image set to obtain corresponding multiple second feature vectors; Calculating the similarity between each of the first feature vectors and each of the second feature vectors; For each of the second feature vectors, in response to a similarity between the second feature vector and at least one first feature vector being greater than or equal to a similarity threshold, a full-body image corresponding to the second feature vector is added to the candidate image set.

6. The method according to any one of claims 1 to 5, characterized in that Performing object detection on the current image frame to obtain at least one object to be identified in the current image frame includes: Inputting the current image frame into a target recognition network for feature extraction to obtain a feature map of the current image frame; Perform object detection on the feature map to obtain at least one object to be identified in the current image frame.

7. The method according to claim 6, characterized in that The target recognition network is trained and acquired according to the acquired training data set; The training data set includes a plurality of unobstructed object images and semi-obstructed object images.

8. The method according to claim 6 or 7, characterized in that: The target recognition network includes a first direction mask branch group and a second direction mask branch group; The first direction mask branch includes a first direction random return to zero module, and the second direction mask branch includes a second direction random return to zero module; The first direction random zeroing module is used to zero a feature map of a predetermined width along the first direction in the acquired image feature map; The second direction random zeroing module is used to zero the feature map of a predetermined width along the second direction in the acquired feature map; Wherein, the position of the feature map of the predetermined width is random.

9. A positioning device, characterized in that: The device comprises: A collection unit, configured to collect a current image frame, wherein a coordinate sequence of at least one specified object is marked in the current image frame; a to-be-recognized object determination unit, configured to perform object detection on the current image frame to obtain at least one to-be-recognized object in the current image frame; a target object determination unit, configured to compare a determined target image set with the at least one object to be identified, and determine a region of the target object in the current image frame, wherein the target image set includes full-body images of the target object at multiple different angles; a positioning unit, configured to determine the position information of the target object according to the area of ​​the target object in the image frame and the area of ​​at least one of the designated objects in the current image frame, wherein the area of ​​the designated object in the current image frame is determined according to the coordinate sequence of the designated object; The step of comparing the determined target image set with the at least one object to be identified is specifically to perform a similarity comparison between full-body images of target images at multiple different angles in the target image set and the at least one object to be identified; The target image set is obtained based on a set of full-body images matching the facial image of the target object in the image frame group acquired within the pre-acquisition time.

10. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the method according to any one of claims 1 to 8.

Images