Multi-dimensional identity recognition method and device, electronic equipment and program product

Abstract

The application provides a multi-dimensional identity recognition method and device, electronic equipment and program product, and relates to the technical field of artificial intelligence. The method comprises the following steps: collecting video stream data containing a to-be-recognized object, extracting multi-dimensional feature information of the to-be-recognized object based on a first target image in the video stream data, recognizing the identity of the to-be-recognized object according to face information, if the recognition fails, performing target tracking on the to-be-recognized object based on a second target image before the first target image in the video stream data to determine whether a third target image containing the identity recognition result of the to-be-recognized object exists in the second target image, and if not, recognizing the identity of the to-be-recognized object according to body key point information. Through face recognition, target tracking and identity recognition based on body key points, the high accuracy of face recognition is retained, and when face recognition fails, identity recognition can be accurately performed based on body key points, so that the balance between the high accuracy and the easy failure of face recognition is achieved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a multi-dimensional identity recognition method, device, electronic device, and program product. Background Technology

[0002] Facial recognition is a widely used identity verification technology. This method identifies individuals in images through face detection, feature extraction, and similarity comparison. However, facial recognition technology has limitations. In typical CCTV surveillance scenarios, people move freely, often with their backs to the camera or their faces not being captured. When faces are incomplete or unclear due to occlusion or shooting angle, facial recognition fails to identify the individuals in the image.

[0003] Because facial recognition-based identity verification technology has many limitations, methods have emerged to determine identity based on information other than facial features, such as multi-feature fusion methods. These methods compare similarity based on height, body shape, and clothing information to identify individuals. However, existing multi-feature fusion methods can only extract and fuse a limited number of features. While facial recognition-based identity verification methods are highly accurate, they are also prone to failure, exhibiting a high degree of vulnerability. Current technologies have failed to achieve a good balance between these two aspects. Summary of the Invention

[0004] This invention provides a multi-dimensional identity recognition method, device, electronic device, and program product to address the shortcomings of existing technologies in achieving a balance between high accuracy and vulnerability in face recognition-based identity recognition.

[0005] This invention provides a multi-dimensional identity recognition method, comprising:

[0006] Acquire video stream data containing the object to be identified; the video stream data includes multiple frames of images.

[0007] Based on the first target image in the video stream data, multidimensional feature information of the object to be identified is extracted; the multidimensional feature information includes facial information and body key point information;

[0008] The identity of the object to be identified is determined based on the facial information. If the identification fails, the object to be identified is tracked based on the second target image in the video stream data to determine whether there is a third target image in the second target image that contains the identification result of the object to be identified. The second target image is an image in the video stream data that is located before the first target image.

[0009] If not, the identity of the object to be identified is determined based on the key body information.

[0010] According to the multi-dimensional identity recognition method provided by the present invention, the step of identifying the identity of the object to be identified based on the body key point information includes:

[0011] Based on the body key point information, the body image of the object to be identified is extracted from the first target image;

[0012] Extract the body features of the object to be identified from the body image;

[0013] In the body image, the target pose of the object to be identified is determined based on the relative positions of the body key points located on both sides of the object's body in the body key point information.

[0014] The body features and target posture are matched with feature data recorded in a preset target body feature database, and the identity of the object to be identified is determined based on the matching results.

[0015] According to the multi-dimensional identity recognition method provided by the present invention, the step of extracting the body image of the object to be identified from the first target image based on the body key point information includes:

[0016] Based on the body key point information, determine the position coordinates and sharpness of each body key point in the first target image;

[0017] Obtain a preset clarity filtering threshold, and remove body key points with a clarity lower than the clarity filtering threshold to obtain target key points with a clarity greater than or equal to the clarity filtering threshold;

[0018] Based on the position coordinates of each target key point, calculate the first distance between each target key point and the first edge side of the first target image, and the second distance between each target key point and the second edge side of the first target image; the first edge side and the second edge side are two adjacent and perpendicular edge sides of the first target image;

[0019] Based on the first distance and the second distance, distinguish between the foreground image and the background image of the person in the first target image;

[0020] Extract the body image of the object to be identified from the foreground image of the person.

[0021] According to the multi-dimensional identity recognition method provided by the present invention, the step of tracking the object to be identified based on a second target image in the video stream data to determine whether there is a third target image in the second target image containing the identity recognition result of the object to be identified includes:

[0022] From the video stream data, obtain each of the second target images preceding the first target image;

[0023] The first target image is used as the current image, and the previous frame image of the current image is obtained from each of the second target images as the third target image;

[0024] A first bounding box is generated in the current image to identify the object, and a second bounding box is generated for each person object in the third target image.

[0025] Calculate the first overlap between the first location box and each of the second location boxes, and determine the first target box corresponding to the object to be identified in each of the second location boxes based on the first overlap, so as to perform target tracking on the object to be identified;

[0026] Based on the first target bounding box, determine whether the third target image contains the identity recognition result of the object to be identified;

[0027] If not, the third target image is used as the current image, and the process returns to and executes the step of obtaining the previous frame image of the current image from each of the second target images as the third target image, until the third target image contains the identity recognition result of the object to be identified, or the third target image is the last frame image in the second target images.

[0028] According to the multi-dimensional identity recognition method provided by the present invention, the first target image includes multiple objects to be identified, and before calculating the overlap between the first location box and each of the second location boxes, the method further includes:

[0029] Based on the first bounding box corresponding to each of the objects to be identified in the first target image, calculate the second overlap between each pair of the first bounding boxes;

[0030] Based on the second bounding boxes corresponding to each person object in the third target image, calculate the third overlap between each pair of the second bounding boxes;

[0031] Based on the second overlap, second target boxes in each of the first location boxes whose overlap is greater than a preset overlap threshold are removed;

[0032] Based on the third overlap, third target boxes in each of the second location boxes whose overlap is greater than the overlap threshold are removed.

[0033] According to the multi-dimensional identity recognition method provided by the present invention, after successfully recognizing the identity of the object to be identified based on the facial information, the method further includes:

[0034] If the recognition is successful, the first body feature data of the object to be identified is extracted from the first target image;

[0035] The first body feature data is labeled based on the identity recognition result of the object to be identified, and the labeled first body feature data is sent to a preset target feature database;

[0036] If the identification fails, and the second target image contains a third target image that contains the identification result of the object to be identified, the second body feature data of the object to be identified is extracted from the third target image;

[0037] Based on the identification result of the object to be identified in the second target image, the second body feature data is labeled, and the labeled second body feature data is sent to the target feature database;

[0038] Randomly select a person object from the target feature database as the target person object, and obtain the first target feature data corresponding to the target person object;

[0039] A sample dataset is constructed based on the first target feature data, and the sample dataset is used to iteratively train the body feature recognition model in the preset multidimensional identity recognition model to update the model parameters of the body feature recognition model; wherein, the multidimensional identity recognition model also includes a face recognition model and a target tracking model.

[0040] According to the multi-dimensional identity recognition method provided by the present invention, the step of constructing a sample dataset based on the first target feature data includes:

[0041] Based on the first target feature data, a sample group is constructed by selecting second target feature data and third target feature data from the target identity database;

[0042] Wherein, the person object corresponding to the second target feature data is the same as the target person object, the person object corresponding to the third target feature data is different from the target person object, and the posture difference between the person objects corresponding to the first target feature data, the second target feature data, and the third target feature data is less than a preset posture difference threshold.

[0043] The first target feature data is compared with the second target feature data, and the first target feature data is compared with the third target feature data to determine the target error corresponding to the sample group;

[0044] Return and execute the step of randomly selecting a person object from the target feature database as the target person object and obtaining the first target feature data corresponding to the target person object, until the number of the sample group reaches the preset sample number threshold and the target error of the sample group conforms to a normal distribution.

[0045] A sample dataset is constructed based on the sample group.

[0046] The present invention also provides a multi-dimensional identity recognition device, comprising:

[0047] The acquisition module is used to acquire video stream data containing the object to be identified; the video stream data includes multiple frames of images.

[0048] The extraction module is used to extract multidimensional feature information of the object to be identified based on the first target image in the video stream data; the multidimensional feature information includes facial information and body key point information;

[0049] The first recognition module is used to identify the identity of the object to be identified based on the facial information. If the identification fails, the module performs target tracking on the object to be identified based on the second target image in the video stream data to determine whether there is a third target image in the second target image that contains the identification result of the object to be identified. The second target image is an image in the video stream data that is located before the first target image.

[0050] The second identification module is used to identify the identity of the object to be identified based on the key body information if the object does not exist.

[0051] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-dimensional identity recognition method as described above.

[0052] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-dimensional identity recognition method as described above.

[0053] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the multi-dimensional identity recognition method as described above.

[0054] This invention provides a multi-dimensional identity recognition method, device, electronic device, and program product. It collects video stream data containing the object to be identified and extracts multi-dimensional feature information, such as facial information and body key point information, based on a first target image in the video stream data. The identity of the object to be identified is then determined based on the facial information. If identification fails, target tracking is performed on a second target image in the video stream data preceding the first target image to determine if a third target image containing the identity recognition result of the object exists in the second target image. If no third target image exists, the identity of the object to be identified is determined based on the body key point information. By combining facial recognition, target tracking, and body key point-based identity recognition, the high accuracy of facial recognition can be preserved to the greatest extent possible. Furthermore, when facial recognition fails, more body features can be extracted based on body key point information for identity recognition, ensuring the accuracy of the identity recognition result. This solves the problem of balancing high accuracy and susceptibility to failure in facial recognition. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0056] Figure 1 This is one of the flowcharts illustrating the multi-dimensional identity recognition method provided by the present invention;

[0057] Figure 2 This is a schematic diagram of the process for identity recognition based on key body information provided by the present invention;

[0058] Figure 3 This is a schematic diagram of the overlapping area of ​​the position frame provided by the present invention;

[0059] Figure 4 This is the second flowchart of the multi-dimensional identity recognition method provided by the present invention;

[0060] Figure 5 This is a schematic diagram of the structure of the multi-dimensional identity recognition device provided by the present invention;

[0061] Figure 6 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0063] To address the problems of existing technologies, this invention provides a multi-dimensional identity recognition method that can retain the high accuracy of facial recognition to the greatest extent possible, while also identifying individuals through body features when facial recognition fails. Specifically, multi-dimensional identity recognition refers to: when facial information is available, identity is recognized through facial recognition; when facial information is unavailable, target tracking is used to identify the same individual across different frames in a video stream, and the facial recognition result of that individual in a previous frame is used as the identity recognition result in the current frame; when target tracking is also unavailable, identity is recognized through the individual's body features.

[0064] Specifically, refer to Figure 1 , Figure 1 This is a flowchart illustrating the multi-dimensional identity recognition method provided in this embodiment of the invention, based on... Figure 1 The multi-dimensional identity recognition method provided in this embodiment of the invention includes:

[0065] Step 100: Acquire video stream data containing the object to be identified; the video stream data includes multiple frames of images;

[0066] First, video stream data containing the object to be identified is collected. This video stream data includes multiple frames of images. Using the same processing method, each frame of the video stream data is processed sequentially to identify the object to be identified in each frame of the video stream data.

[0067] Furthermore, in the same frame of the video stream data, there may be one or more objects to be identified. When there are multiple objects to be identified, the same method is used to identify different objects. The identification of multiple objects to be identified can be carried out simultaneously or sequentially. When carried out sequentially, the order can be determined without distinguishing the order, or according to certain rules, such as clarity, and / or the completeness of the facial information of the person in the image, and / or the completeness of the body outline of the person in the image. No specific limitation is made here.

[0068] Step 200: Based on the first target image in the video stream data, extract multidimensional feature information of the object to be identified; the multidimensional feature information includes facial information and body key point information;

[0069] Based on the first target image in the video stream data, multidimensional feature information of the object to be identified is extracted. This multidimensional feature information includes facial information and body key point information. The different dimensions of the multidimensional feature information can be extracted simultaneously or sequentially; no specific limitation is made here.

[0070] The first target image is the current image undergoing identity recognition processing. Body keypoint information includes some or all of the following: nose keypoint, neck keypoint, shoulder keypoint (including left and right shoulders), elbow keypoint (including left and right elbows), wrist keypoint (including left and right wrists), hip keypoint (including left, right, and mid-hip, with the mid-hip being the midpoint between the left and right hips), ankle keypoint (including left and right ankles), knee keypoint (including left and right knees), eye keypoint (including left and right eyes), and ear keypoint (including left and right ears), as well as the clarity of each body keypoint.

[0071] Step 300: Identify the identity of the object to be identified based on the facial information. If the identification fails, perform target tracking on the object to be identified based on the second target image in the video stream data to determine whether there is a third target image in the second target image that contains the identification result of the object to be identified. The second target image is an image in the video stream data that is located before the first target image.

[0072] The system identifies the target object based on the extracted facial information. If identification is successful, the object's identity information is obtained. If identification fails, the system tracks the target object based on a second target image in the video stream data to determine if a third target image containing the identification result of the target object exists in the second target image. The second target image refers to the frames in the video stream data preceding the first target image, and the identification result of the target object in the third target image is obtained based on facial recognition.

[0073] It is understandable that the motion trajectory of the same target can be detected based on consecutive frames in the video stream data, thereby achieving target tracking. In this embodiment, the object to be identified in the first target image is tracked based on the image preceding the first target image, realizing reverse tracing of the object's movement trajectory. If there is an image in the video stream data preceding the first target image that contains the identification result of the object to be identified, i.e., a third target image, then the identification result of the object to be identified in the third target image can be used as the identification result of the object to be identified in the first target image. By tracking the target, the identification result of the object to be identified is guaranteed to be based on face recognition to the greatest extent possible, thus preserving the high accuracy of face recognition.

[0074] Step 400: If not, identify the identity of the object to be identified based on the body key point information.

[0075] If the second target image does not contain a third target image that provides the identification result of the object to be identified, then the identity of the object to be identified is determined based on the extracted body key point information. Based on the extracted body key point information, not only can the body features of the object to be identified be obtained, but also its posture in the image can be identified. Therefore, the extractable body features are related not only to the object itself but also to its posture and the image's shooting angle. By extracting the body key points of the object to be identified, more feature information that can be used for identity recognition can be obtained, ensuring the accuracy of the identification of the object.

[0076] As is known, in typical camera surveillance scenarios, the movement of each person is relatively random and unpredictable. Therefore, when multiple objects to be identified exist in the first target image, the different objects will present different states in the same image, resulting in differences in the features that can be extracted. Therefore, for multiple objects to be identified in the same image, those that can be identified based on facial information will be identified using facial recognition; those that cannot be identified based on facial information will be identified using target tracking; and those that cannot be identified using either facial recognition or target tracking will be identified using extracted body key point information.

[0077] In one embodiment, the identification of the object to be identified is achieved based on a preset multidimensional identity recognition model, which includes a face recognition model, a target tracking model, and a body feature recognition model. The face recognition model is used for face detection and to identify the target's identity based on the detected face; the target tracking model is used for person tracking; and the body feature recognition model includes a body key point detection tool and a body feature detection module. The body key point detection tool is used to detect the body key points of a person in the image, and the body feature detection module is used to extract the person's body features based on the detected body key points and perform identity recognition.

[0078] Furthermore, the multidimensional identity recognition model also includes an object detection model and a human image segmentation model. The object detection model is used to detect the presence of human objects in the image and generate human body bounding boxes; the human image segmentation model is used to determine whether each pixel in the image belongs to the human image or the background image, thereby distinguishing between the human foreground image and the background image.

[0079] Based on this, for the first target image in the acquired video stream data, when identifying the object to be identified in the first target image, the target detection model is first called to detect the existence of the object to be identified; if the object to be identified exists, the face recognition model is called to extract the face information of the object to be identified and perform face recognition; if face recognition fails, the target tracking model is called to track the object to be identified; if the target tracking is successful, that is, if there is a third target image in the second target image containing the identity recognition result of the object to be identified, the identity recognition result of the object to be identified in the first target image is obtained based on the identity recognition result of the object to be identified in the third target image; if the target tracking fails, the body feature recognition model is called to extract the body key point information of the object to be identified, and the identity of the object to be identified is identified based on the extracted body key point information.

[0080] Furthermore, when invoking the face recognition model to identify the target object, the face information of the target object is matched with faces recorded in a preset target identity database. When invoking the body feature recognition model to identify the target object using body key point information, the body features of the target object are extracted based on the extracted body key point information, and the extracted body features of the target object are matched with body features recorded in a preset target feature database. In this embodiment, features of different dimensions in the multidimensional feature information of the target object are extracted only when the corresponding model is invoked, thereby enabling on-demand feature extraction and reducing the waste of computing resources.

[0081] In this embodiment, video stream data containing the object to be identified is collected, and multi-dimensional feature information such as facial information and body key point information of the object to be identified is extracted based on the first target image in the video stream data. The identity of the object to be identified is identified based on the facial information. If the identification fails, the object to be identified is tracked based on the second target image in the video stream data before the first target image to determine whether there is a third target image containing the identity recognition result of the object to be identified in the second target image. If not, the identity of the object to be identified is identified based on the body key point information. By combining facial recognition, target tracking, and identity recognition based on body key points, the high accuracy of facial recognition can be preserved to the greatest extent, and when facial recognition fails, more body features can be extracted based on body key point information for identity recognition, ensuring the accuracy of the identity recognition result. This solves the problem of balancing the high accuracy and susceptibility to failure of facial recognition.

[0082] Preferably, in step 400, identifying the identity of the object to be identified based on key body information specifically includes:

[0083] Step 410: Based on the body key point information, extract the body image of the object to be identified from the first target image;

[0084] Step 420: Extract the body features of the object to be identified from the body image;

[0085] Step 430: In the body image, based on the relative positions of the body key points located on both sides of the body of the object to be identified in the body key point information, determine the target pose of the object to be identified;

[0086] Step 440: Match the body features and the target posture with the feature data recorded in the preset target body feature database, and identify the identity of the object to be identified based on the matching result.

[0087] Reference Figure 2 The flowchart shown illustrates the process of identity recognition based on body keypoint information. First, based on the body keypoints in the body keypoint information, the body contour of the object to be identified can be determined from the first target image, thereby performing image segmentation on the first target image and extracting the body image of the object to be identified. Body features of the object to be identified are extracted from the extracted body image; these body features are vectors of a specific dimension, such as a 256-dimensional feature vector. In the extracted body image, the target pose of the object to be identified is determined based on the relative positions of the body keypoints located on both sides of the object's body in the body keypoint information. The extracted body features and the target pose of the object to be identified are matched with feature data recorded in a preset target feature database, and the identity of the object to be identified is determined based on the matching result.

[0088] Preferably, unlike simply calculating simple features such as height or body color distribution histograms, this embodiment employs neural network technology to extract the body features of the object to be identified. The structure of the deep neural network ensures that it can fully describe the complexity of body features. An image containing the object to be identified, detected by the object detection model, is input into the body feature recognition model. A body keypoint detection tool is used to obtain the keypoints of the object. A portrait segmentation model is then used to distinguish the foreground and background of the image, and body image is cropped based on the foreground. The cropped body image is input into the feature extraction network in the body feature detection module to obtain the body feature `bodyInfo` output by the body feature detection module. Then, pose analysis is performed using the body keypoints to calculate the target pose `posture` of the object to be identified. The target pose `Posture` and body features are matched with the feature data recorded in the target feature database for similarity. Based on the identity information corresponding to the feature data with the highest similarity, the identity of the object to be identified is determined.

[0089] Further, in step 410, based on body key point information, the body image of the object to be identified is extracted from the first target image, including:

[0090] Step 411: Based on the body key point information, determine the position coordinates and sharpness of each body key point in the first target image;

[0091] Step 412: Obtain a preset clarity threshold, and remove body key points with a clarity lower than the clarity threshold to obtain target key points with a clarity greater than or equal to the clarity threshold.

[0092] Step 413: Based on the position coordinates of each target key point, calculate the first distance between each target key point and the first edge side of the first target image, and the second distance between each target key point and the second edge side of the first target image; the first edge side and the second edge side are two adjacent and perpendicular edge sides of the first target image.

[0093] Step 414: Based on the first distance and the second distance, distinguish the foreground image and background image of the person in the first target image;

[0094] Step 415: Extract the body image of the object to be identified from the foreground image of the person.

[0095] First, based on the extracted body key point information, the position coordinates and sharpness of each body key point in the first target image are determined. A preset sharpness filtering threshold is initialized, and body key points with sharpness less than the sharpness filtering threshold are removed, resulting in target key points with sharpness greater than or equal to the sharpness filtering threshold. Based on the position coordinates of each target key point, a first distance between each target key point and a first edge side of the first target image, and a second distance between each target key point and a second edge side of the first target image are calculated. The first and second edge sides of the first target image are adjacent and perpendicular edge sides, such as the top and left edges, or the top and right edges, or the left and bottom edges, or the right and bottom edges of the first target image. Based on the calculated first and second distances, the foreground and background images of the person in the first target image are distinguished, and the body image of the object to be identified is extracted from the foreground image of the person.

[0096] The sharpness filtering threshold is set to 0.2. Taking the left and top edges of the first target image as examples, the distance *x* between each target keypoint and the left edge of the first target image, and the distance *y* between each keypoint and the top edge of the first target image are calculated. The minimum and maximum values ​​of the calculated distances *x* and *y* are selected. The minimum distance *x* is taken as the left edge of the foreground image of the person, and the maximum distance *x* is taken as the right edge. The minimum distance *y* is taken as the top edge of the foreground image of the person, and the maximum distance *y* is taken as the bottom edge. This distinguishes the foreground image of the person. Pixels in the background portion of the foreground image that are not human figures are set to specific pixel values, such as 0, to eliminate background interference, thereby obtaining the body image of the object to be identified.

[0097] The captured body image needs to be processed in two ways: the first input is to the feature extraction network in the body feature detection module, and the second is to perform pose analysis.

[0098] Specifically, for the first processing path, to improve the accuracy of the detection model, it is necessary to train the various parameters of the body feature extraction network. In this embodiment, an Inception-ResNet-v2-like convolutional neural network is used as the main neural network structure in the body feature detection module. This structure achieves a good balance between accuracy and computational efficiency. Furthermore, its residual block structure can solve the problems of gradient explosion and gradient vanishing in deep neural networks, thereby effectively improving model accuracy and training speed. Based on the original Inception-ResNet-v2, the size of the softmax layer in the output part is set to 256 (it can also be adjusted to 128 depending on the amount of training sample data, which is not limited here) to improve training efficiency. Finally, a feature extraction network with a 256-dimensional feature vector output is obtained through training. The extracted body features are denoted as bodyInfo, and this body feature vector contains rich body feature information.

[0099] In real-world scenarios, different body postures (including rotational postures, i.e., the angle of rotation) can also affect body characteristics. For example, the aspect ratio of a side view is significantly smaller than that of a front view. If body posture is combined with a body feature network to compare the similarity of body features, more accurate identity recognition results can be obtained.

[0100] Preferably, a pose analysis algorithm for calculating the target pose of a human body by means of the relative positions of key points on the left and right sides of the human body is as follows:

[0101] First, a sharpness threshold T is set, which characterizes the degree to which key body points are occluded. Each key body point is pre-numbered, for example: nose 1, neck 2, right shoulder 3, right elbow 4, right wrist 5, left shoulder 6, left elbow 7, left wrist 8, mid-hip 9, right hip 10, right knee 11, right ankle 12, left hip 13, left knee 14, left ankle 15, right eye 16, left eye 17, right ear 18, and left ear 19. The body rotation angle represents the angle of rotation of the human body relative to the camera. When performing posture analysis for the body rotation angle, if we define 0 degrees as the face facing the camera, 90 degrees as the right half of the body facing the camera, 180 degrees as completely facing away from the camera, and 270 degrees as the left half of the body facing the camera, then key points 2, 3, 6, 9, 10, and 13 are used for the following judgments:

[0102] Condition 1: If keypoints[3].accurate>T or keypoints[6].accurate>T, then take pointLeft=keypoints[6], pointRight=keypoints[3], pointMid=keypoints[2];

[0103] Condition 2: If keypoints

[10] .accurate > T or keypoints

[13] .accurate > T are satisfied, then set pointLeft = keypoints

[13] , pointRight = keypoints

[10] , and pointMid = keypoints[9].

[0104] If neither Condition 1 nor Condition 2 is satisfied, it means that most of the human body area is blocked and identity recognition cannot continue, and the process can be directly ended.

[0105] If pointLeft.accurate < T and pointRight.accurate > T are satisfied, it means that the right half of the object to be recognized is facing the camera directly, and the corresponding rotation posture value is 90;

[0106] If pointLeft.accurate > T and pointRight.accurate < T are satisfied, it means that the left half of the object to be recognized is facing the camera directly, and the corresponding rotation posture value is 270;

[0107] When pointLeft.accurate > T and pointRight.accurate > T, calculate dleft = pointMid.x – pointLeft.x and dRight = pointRight.x – pointMid.x;

[0108] posture = arcsin(dleft + dRight) / (maxLeft + maxRight).

[0109] Here, accurate is the clarity of the key point keypoints, and keypoints[3].accurate is the clarity of the right shoulder key point numbered 3, and so on.

[0110] Finally, the target posture posture obtained from posture analysis and the body feature bodyInfo obtained from feature extraction are jointly used as body features, and compared with the feature data recorded in the preset target feature database for similarity. The process of body feature similarity comparison is as follows:

[0111] The Euclidean distance between the body features of the object to be identified and the vectors of the body features in the target feature database is calculated and denoted as DBody. The similarity DPos of the pose is calculated as follows: DPos = person[a].pose – person[b].posture. The final similarity is: w1*DBody + w2*DPos. Here, w1 and w2 are weight values ​​obtained through model pre-training.

[0112] If the final similarity is less than the preset threshold Ts (e.g., 0.4), then the object to be identified is considered to have the same identity as the target corresponding to the feature data in the target feature database.

[0113] In this embodiment, the extracted body keypoint information includes three attributes for each keypoint: distance to the first edge of the image, distance to the second edge of the image, and sharpness. Sharpness characterizes the degree of occlusion of the body part corresponding to the keypoint. By extracting body features and performing pose analysis through body keypoints, and combining pose with body feature matching when identifying the object to be identified, the influence of pose on body features is taken into account, thereby improving the accuracy of identification.

[0114] Preferably, in step 300, target tracking is performed on the object to be identified based on the second target image in the video stream data to determine whether there is a third target image in the second target image that contains the identity recognition result of the object to be identified, including:

[0115] Step 310: Obtain each of the second target images preceding the first target image from the video stream data;

[0116] Step 320: Take the first target image as the current image, and obtain the previous frame image of the current image from each of the second target images as the third target image;

[0117] Step 330: Generate a first bounding box of the object to be identified in the current image, and generate a second bounding box of each person object in the third target image;

[0118] Step 340: Calculate the first overlap between the first location box and each of the second location boxes, and determine the first target box corresponding to the object to be identified in each of the second location boxes based on the first overlap, so as to perform target tracking on the object to be identified;

[0119] Step 350: Based on the first target bounding box, determine whether the third target image contains the identity recognition result of the object to be identified;

[0120] Step 360: If not, use the third target image as the current image, return and execute the step of obtaining the previous frame image of the current image from each of the second target images as the third target image, until the third target image contains the identity recognition result of the object to be identified, or the third target image is the last frame image in the second target images.

[0121] When performing target tracking, adjacent images are compared. If a person has previously appeared with clear facial information in the video stream data, and the person has been moving continuously within the camera area without any obstruction, then even if the person's facial information cannot be obtained in a later frame, their identity can still be confirmed through target tracking based on the historical facial recognition results in the video stream.

[0122] Target tracking uses the overlapping region of the bounding boxes of a person detected by the target detection model as the criterion for determining the continuous movement of the person target. The definition of the overlapping region of the bounding boxes is as follows: Figure 3 As shown, in Figure 3 In the diagram, 3-1 is the bounding box of the person in the current image, 3-2 is the bounding box of the person in the previous frame, and 3-3 is the overlapping area of ​​the bounding boxes. It can be seen that for consecutive frames in the video stream data captured by the same camera, the image size and resolution are the same; the difference lies in the position of the moving person in the image and the position and size of its bounding box.

[0123] During target tracking, firstly, the second target images preceding the first target image are obtained from the video stream data. The first target image is then used as the current image being processed. The frame preceding the current image is obtained from each of the second target images and used as the third target image. A first bounding box for the object to be identified is generated in the current image, and second bounding boxes for each person in the third target image are generated. The generation of the first and second bounding boxes can occur during target tracking or when detecting the presence of a person in the image; no specific limitation is made here. A first overlap between the first bounding boxes and each of the second bounding boxes is calculated. Based on this first overlap, the first target bounding box corresponding to the object to be identified in each of the second bounding boxes is determined, thereby determining the position of the object to be identified in the third target image and achieving the tracking of the object.

[0124] Based on the first target bounding box, determine whether the third target image contains the identification result of the object to be identified. If not, use the third target image as the current image, obtain its previous frame, and repeat the above steps until the third target image contains the identification result of the object to be identified, or the third target image is the last frame of each of the second target images, i.e., the first frame in the video stream data. If there is no third target image containing the identification result of the object to be identified among the second target images, target tracking of the object to be identified fails, and the body feature recognition model is invoked to identify the object to be identified based on the body key point information.

[0125] Furthermore, two overlap thresholds, denoted as iouT1 and iouT2, are used during target tracking. iouT1 is used to filter out human targets with high overlap within the same frame. If the overlap between the bounding boxes of two human targets exceeds iouT1, it is considered that human overlap exists in the image. Overlapping human targets will affect the accuracy of target tracking; therefore, if overlapping human targets are found, their bounding boxes need to be removed, ending the tracking of overlapping human targets.

[0126] IouT2 is used to detect the overlap between the bounding boxes of a person in two adjacent frames. Since the acquisition time interval between adjacent frames is very small, the overlap of the same person will be high. If the overlap of the bounding boxes of two people exceeds iouT2, the two people can be considered the same. Because the size of the overlapping area is directly affected by the size of the bounding box, and because people closer to the camera, even with a lower degree of overlap, have a larger overlapping area than people farther from the camera, this affects the setting of the overlap threshold. To address this issue, in this embodiment, the ratio of the overlapping area to the larger bounding box area is used to describe the degree of overlap.

[0127] The first target image may contain one or more objects to be identified, and the third target image may also contain one or more human figures. When the first target image contains multiple objects to be identified and the third target image contains multiple human figures, before calculating the overlap between the bounding boxes in the first and third target images, it is necessary to identify the overlapping human figures in the first and third target images and end the tracking of the overlapping human figures. Specifically, before step 340, the following may also be included:

[0128] Step 301: Based on the first location boxes corresponding to each of the objects to be identified in the first target image, calculate the second overlap between each pair of the first location boxes;

[0129] Step 302: Based on the second bounding boxes corresponding to each person object in the third target image, calculate the third overlap between each pair of the second bounding boxes;

[0130] Step 303: Based on the second overlap, remove second target boxes whose overlap is greater than a preset overlap threshold in each of the first position boxes;

[0131] Step 304: Based on the third overlap, remove the third target boxes in each of the second position boxes whose overlap is greater than the overlap threshold.

[0132] Based on the first bounding boxes of each object to be identified in the first target image, a second overlap between each pair of the first bounding boxes is calculated. Similarly, based on the second bounding boxes of each person in the third target image, a third overlap between each pair of the second bounding boxes is calculated. According to the calculated second overlap, second bounding boxes in each first bounding box with an overlap greater than a preset overlap threshold (iouT1) are removed, thus ending the tracking of the object to be identified corresponding to the second bounding box. Similarly, according to the calculated third overlap, third bounding boxes in each second bounding box with an overlap greater than the overlap threshold (iouT1) are removed, thus ending the tracking of the person corresponding to the third bounding box.

[0133] Preferably, in one embodiment, the variables are defined as follows:

[0134] Persons This represents an array consisting of all human figures in the i-th frame of the image;

[0135] Person [j] represents the j-th person in the i-th frame;

[0136] boundBox: Represents the bounding box of the human target generated by the object detection model. left, top, right, and bottom represent the left, top, right, and bottom borders of the bounding box, respectively.

[0137] The target tracking process provided in the embodiments of the present invention will be described in detail below, taking into account the meanings of the defined variables:

[0138] First, initialize the overlap thresholds iouT1 and iouT2;

[0139] S1: If the current frame image is the first frame image in the video stream data, end directly and the tracking fails; otherwise, record the current frame as the i-th frame image and the previous frame as the (i-1)-th frame, and execute S2.

[0140] S2: Load the list of people in the current frame image. And a list of Persons whose bounding boxes have been generated in the previous frame image. <i-1>Execute S3;

[0141] S3: Calculate Persons The overlap of the bounding boxes of all person targets is checked to determine if any overlap exists. Overlapping person targets will affect the accuracy of target tracking; therefore, if overlapping person targets are found, tracking of the overlapping person targets is terminated. For person targets without overlap, S4 is executed; where Persons The overlap of the bounding boxes of all characters and targets is calculated using the algorithm shown in the following pseudocode:

[0142] For m in range(0, len(Persons) ));

[0143] For n in range(m+1,len(Persons ));

[0144] boxM=Persons [m].boundBox;

[0145] boxN=Persons [n].boundBox;

[0146] areaM=(boxM.right–boxM.left)*(boxM.bottom–boxM.top);

[0147] areaN=(boxN.right–boxN.left)*(boxN.bottom–boxN.top);

[0148] width=max(min(boxM.right,boxN.right)-max(boxM.left,boxN.left),0);

[0149] height=max(min(boxM.bottom,boxN.bottom)-max(boxM.top,boxN.top),0);

[0150] areaIOU=height*width;

[0151] IOU[m][n]=areaIOU / max(areaM,areaN);

[0152] If(IOU[m][n]>iouT1);

[0153] Remove(Persons [m])# Delete overlapping targets;

[0154] Remove(Persons [n])# Delete overlapping targets.

[0155] S4: Calculate Persons according to the following pseudocode. Each location box in the Persons section <i-1>The overlapping area size of each location box is IOU[m][n], and then S5 is executed;

[0156] For m in range(0, len(Persons) ));

[0157] For n in range(0,len(Persons <i-1>));

[0158] boxM=Persons [m].boundBox;

[0159] boxN=Persons <i-1>[n].boundBox;

[0160] areaM=(boxM.right–boxM.left)*(boxM.bottom–boxM.top);

[0161] areaN=(boxN.right–boxN.left)*(boxN.bottom–boxN.top);

[0162] width=max(min(boxM.right,boxN.right)-max(boxM.left,boxN.left),0);

[0163] height=max(min(boxM.bottom,boxN.bottom)-max(boxM.top,boxN.top),0);

[0164] areaIOU = height * width;

[0165] IOU[m][n]=areaIOU / max(areaM, areaN);

[0166] If (IOU[m][n]>iouT2);

[0167] Save(IOU[m][n]) # Records IOU[m][n] as valid.

[0168] S5: Sort all valid IOU[m][n] values ​​in descending order and access them sequentially. For IOU[m][n], if Person... If [m] has not been visited, then the target Person will be... [m]'s identity marker is related to Person. <i-1>[n] are the same, thus successfully achieving the tracking and identification of the target m.

[0169] In this embodiment, by tracking and identifying the same person in the video stream data, the identity recognition result obtained by face recognition of the person in front is used as the identity recognition result of the person behind. Even when the face information is invalid, the high accuracy of identity recognition based on face recognition can still be maintained.

[0170] Preferably, the multi-dimensional identity recognition method provided in this embodiment of the invention includes an online detection module and an offline self-supervised training module. The online detection module is responsible for performing the identity recognition task and storing valid identity recognition information in a target feature database. The offline self-supervised training module uses the target feature database to perform self-supervised training on the parameters of the body feature extraction network, thereby improving the accuracy of the body feature extraction network.

[0171] Specifically, refer to Figure 4 Another flowchart illustrating the multi-dimensional identity recognition method is shown. The input data for the online detection module is a real-time or offline video stream. For each frame in the video stream, the online detection module performs the following... Figure 4 The recognition process is shown below. At the start of the detection process, an initialization process is executed first, loading the target feature database, starting various identity recognition-related models and tools, and then processing the input image begins.

[0172] First, the object detection module is called to detect whether there are human targets in the image. If they are found, the face recognition model is called for each human target. If face information is detected and face recognition is successful, the body feature recognition model is called to extract and record the body features (bodyInfo) of the human target and update the target feature database. If face recognition fails, target tracking is performed, and Persons is calculated. and Persons <i-1>Does a highly overlapping set of people exist (requirement)? A matching algorithm prioritizing overlap is used. If a match is successful, and Persons... <i-1>If a person has been successfully identified, the body feature recognition model is invoked to extract and record the person's body feature (bodyInfo), and the target feature database is updated. If tracking fails, the person's body feature is compared with the body feature of the person already recorded in the target feature database. If the distance between the body features is less than a preset value, the identification is successful; otherwise, the identification fails.

[0173] Based on this, after identifying the identity of the object to be identified based on facial information, step 300 may further include:

[0174] Step 3001: If the recognition is successful, extract the first body feature data of the object to be identified from the first target image;

[0175] Step 3002: Based on the identity recognition result of the object to be identified, the first body feature data is labeled, and the labeled first body feature data is sent to the preset target feature database;

[0176] Step 3003: If recognition fails, and the second target image contains a third target image that includes the identity recognition result of the object to be identified, extract the second body feature data of the object to be identified from the third target image;

[0177] Step 3004: Based on the identification result of the object to be identified in the second target image, the second body feature data is labeled, and the labeled second body feature data is sent to the target feature database;

[0178] Step 3005: Randomly select a person object from the target feature database as the target person object, and obtain the first target feature data corresponding to the target person object;

[0179] Step 3006: Construct a sample dataset based on the first target feature data, and use the sample dataset to iteratively train the body feature recognition model in the preset multidimensional identity recognition model to update the model parameters of the body feature recognition model; wherein, the multidimensional identity recognition model further includes a face recognition model and a target tracking model.

[0180] Specifically, if facial recognition is successful, the body feature recognition model is invoked to extract the first body feature data of the object to be identified from the first target image. Based on the identity recognition result of the object to be identified, the first identity feature data is labeled, and the identity recognition result of the object to be identified is used as the label information of the first body feature data. The labeled first body feature data is then sent to a preset target feature database to update the target feature database.

[0181] If facial recognition fails, and the second target image does not contain a third target image that includes the identification result of the object to be identified (i.e., the tracking and identification of the object to be identified has failed), the body feature recognition model is invoked to extract the second body feature data of the object to be identified from the first target image. Based on the identification result of the object to be identified, the second body feature data is labeled, and the identification result of the object to be identified is used as the label information of the second body feature data. The labeled second body feature data is sent to a preset target feature database to update the target feature database. The first body feature data and the second body feature data can be the same or different.

[0182] A person is randomly selected from the target feature database as the target person object. Then, the first target feature data corresponding to the target person object is obtained. A sample dataset is constructed based on the first target feature data. The constructed sample dataset is used to iteratively train the identity feature recognition model in the multidimensional identity recognition model to update the model parameters of the body feature recognition model and realize the dynamic calibration of the multidimensional identity recognition model.

[0183] Further, in step 3006, constructing a sample dataset based on the first target feature data specifically includes:

[0184] Step 3106: Based on the first target feature data, select the second target feature data and the third target feature data from the target identity database to construct a sample group;

[0185] Wherein, the person object corresponding to the second target feature data is the same as the target person object, the person object corresponding to the third target feature data is different from the target person object, and the posture difference between the person objects corresponding to the first target feature data, the second target feature data, and the third target feature data is less than a preset posture difference threshold.

[0186] Step 3206: Compare the first target feature data with the second target feature data, and compare the first target feature data with the third target feature data to determine the target error corresponding to the sample group;

[0187] Step 3306: Return to and execute the step of randomly selecting a person object from the target feature database as the target person object and obtaining the first target feature data corresponding to the target person object, until the number of the sample group reaches the preset sample number threshold and the target error of the sample group conforms to a normal distribution.

[0188] Step 3406: Construct a sample dataset based on the sample group.

[0189] Specifically, based on the first target feature data, second target feature data that is the same as the person corresponding to the first target feature data, and third target feature data that is different from the person corresponding to the first target feature data, are selected to construct a sample group. Furthermore, the pose differences of the person corresponding to the first, second, and third target feature data are less than a preset pose difference threshold.

[0190] The first target feature data and the second target feature data are compared, and the first target feature data and the third target feature data are compared to determine the target error of the constructed sample group. Then, a person object is randomly selected as the target person object, and the above steps are repeated until the number of constructed sample groups reaches a preset sample size threshold, and the target error of each sample group conforms to a normal distribution. A sample dataset is constructed based on the sample groups with the preset sample size threshold.

[0191] In this embodiment, to improve the accuracy of the model, it is necessary to train the various parameters of the body feature extraction network. In this embodiment, an Inception-ResNet-v2-like convolutional neural network is used as the main neural network structure. A crucial prerequisite for learning neural network parameters is having a large amount of high-quality labeled data. However, in practical applications, the cost of collecting sample data is high, and the data quality is inconsistent, ultimately leading to poor model performance. To address this issue, the multi-dimensional identity recognition method provided in this embodiment proposes a self-supervised training method, replacing manual annotation with reliable face recognition results, thereby significantly reducing the cost of collecting sample data.

[0192] In the online detection module, the target feature database stores a large amount of body feature data, which is registered in the target feature database when face recognition or target tracking is successful. Therefore, this data can be used as training data for the neural network. The process of one offline self-training iteration is as follows:

[0193] First, construct a sample dataset. From the target feature database, randomly select a reference record A, another record P with the same identity as record A, and a record N with a different identity from record A. The pose difference between A, P, and N is less than K. The pose difference is represented by the rotation angle, and the value of K is 20°.

[0194] For each group (A, P, N), calculate the body feature comparison result FP between A and P, and the body feature comparison result FN between A and N. Calculate the target error of the sample group (A, P, N): Loss = -FP + FN + margin (margin represents the minimum distance, used to highlight the difference between FN and FP, the default value can be 0.1).

[0195] Initialize the sample size threshold Batch_size (the default value can be 64). From the constructed sample group, further select Batch_size groups of samples, based on the principle that the target error Loss in this batch of data follows a normal distribution. The selected sample groups thus include both easily distinguishable and difficult-to-distinguish samples. The easily distinguishable samples can consolidate the existing training results, while the difficult-to-distinguish samples can further optimize the network parameters.

[0196] Finally, the final overall sample error LOSS = sum(Loss[i]) is calculated, where Loss[i] represents the target error of the i-th sample group in Batch_size. The final calculated overall sample error is backpropagated to correct the neural network parameters and the matching weights w1 and w2 of the body features. Through several iterations, until the final iteration termination condition is met, the training process ends, completing one dynamic calibration of the body feature recognition model.

[0197] In this embodiment, the identified human target features recorded in the target feature database are used to achieve dynamic calibration, which is used for comparison and subsequent offline self-supervised training to improve the accuracy of the body feature extraction network. Furthermore, it can automatically collect data from the scene to construct training samples, making the trained model adaptable to the monitoring scene, thus resulting in higher accuracy in identity recognition.

[0198] Furthermore, identity recognition relies solely on continuous image frames captured by the camera, requiring no additional input information, thus making it applicable to most scenarios. Moreover, due to its dynamic calibration characteristics, as long as a clear facial image appears in any frame of the video stream, subsequent identity recognition can overcome the influence of external environmental factors such as changes in brightness and distance, exhibiting higher adaptability and stability.

[0199] The multi-dimensional identity recognition device provided by the present invention is described below. The multi-dimensional identity recognition device described below can be referred to in correspondence with the multi-dimensional identity recognition method described above.

[0200] Reference Figure 5 The multi-dimensional identity recognition device provided in this embodiment of the invention includes:

[0201] Acquisition module 10 is used to acquire video stream data containing the object to be identified; the video stream data includes multiple frames of images;

[0202] Extraction module 20 is used to extract multidimensional feature information of the object to be identified based on the first target image in the video stream data; the multidimensional feature information includes facial information and body key point information;

[0203] The first identification module 30 is used to identify the identity of the object to be identified based on the facial information. If the identification fails, the first identification module 30 performs target tracking on the object to be identified based on the second target image in the video stream data to determine whether there is a third target image in the second target image that contains the identification result of the object to be identified. The second target image is an image in the video stream data that is located before the first target image.

[0204] The second identification module 40 is used to identify the identity of the object to be identified based on the body key point information if the object does not exist.

[0205] In one embodiment, the second identification module 40 is further configured to:

[0206] Based on the body key point information, the body image of the object to be identified is extracted from the first target image;

[0207] Extract the body features of the object to be identified from the body image;

[0208] In the body image, the target pose of the object to be identified is determined based on the relative positions of the body key points located on both sides of the object's body in the body key point information.

[0209] The body features and target posture are matched with feature data recorded in a preset target body feature database, and the identity of the object to be identified is determined based on the matching results.

[0210] In one embodiment, the second identification module 40 is further configured to:

[0211] Based on the body key point information, determine the position coordinates and sharpness of each body key point in the first target image;

[0212] Obtain a preset clarity filtering threshold, and remove body key points with a clarity lower than the clarity filtering threshold to obtain target key points with a clarity greater than or equal to the clarity filtering threshold;

[0213] Based on the position coordinates of each target key point, calculate the first distance between each target key point and the first edge side of the first target image, and the second distance between each target key point and the second edge side of the first target image; the first edge side and the second edge side are two adjacent and perpendicular edge sides of the first target image;

[0214] Based on the first distance and the second distance, distinguish between the foreground image and the background image of the person in the first target image;

[0215] Extract the body image of the object to be identified from the foreground image of the person.

[0216] In one embodiment, the first identification module 30 is further configured to:

[0217] From the video stream data, obtain each of the second target images preceding the first target image;

[0218] The first target image is used as the current image, and the previous frame image of the current image is obtained from each of the second target images as the third target image;

[0219] A first bounding box is generated in the current image to identify the object, and a second bounding box is generated for each person object in the third target image.

[0220] Calculate the first overlap between the first location box and each of the second location boxes, and determine the first target box corresponding to the object to be identified in each of the second location boxes based on the first overlap, so as to perform target tracking on the object to be identified;

[0221] Based on the first target bounding box, determine whether the third target image contains the identity recognition result of the object to be identified;

[0222] If not, the third target image is used as the current image, and the process returns to and executes the step of obtaining the previous frame image of the current image from each of the second target images as the third target image, until the third target image contains the identity recognition result of the object to be identified, or the third target image is the last frame image in the second target images.

[0223] In one embodiment, the first target image includes multiple objects to be identified, and the first identification module 30 is further configured to:

[0224] Based on the first bounding box corresponding to each of the objects to be identified in the first target image, calculate the second overlap between each pair of the first bounding boxes;

[0225] Based on the second bounding boxes corresponding to each person object in the third target image, calculate the third overlap between each pair of the second bounding boxes;

[0226] Based on the second overlap, second target boxes in each of the first location boxes whose overlap is greater than a preset overlap threshold are removed;

[0227] Based on the third overlap, third target boxes in each of the second location boxes whose overlap is greater than the overlap threshold are removed.

[0228] In one embodiment, the multi-dimensional identity recognition device further includes a self-supervised training module, used for:

[0229] If the recognition is successful, the first body feature data of the object to be identified is extracted from the first target image;

[0230] The first body feature data is labeled based on the identity recognition result of the object to be identified, and the labeled first body feature data is sent to a preset target feature database;

[0231] If the identification fails, and the second target image contains a third target image that contains the identification result of the object to be identified, the second body feature data of the object to be identified is extracted from the third target image;

[0232] Based on the identification result of the object to be identified in the second target image, the second body feature data is labeled, and the labeled second body feature data is sent to the target feature database;

[0233] Randomly select a person object from the target feature database as the target person object, and obtain the first target feature data corresponding to the target person object;

[0234] A sample dataset is constructed based on the first target feature data, and the sample dataset is used to iteratively train the body feature recognition model in the preset multidimensional identity recognition model to update the model parameters of the body feature recognition model; wherein, the multidimensional identity recognition model also includes a face recognition model and a target tracking model.

[0235] In one embodiment, the self-supervised training module is further configured to:

[0236] Based on the first target feature data, a sample group is constructed by selecting second target feature data and third target feature data from the target identity database;

[0237] Wherein, the person object corresponding to the second target feature data is the same as the target person object, the person object corresponding to the third target feature data is different from the target person object, and the posture difference between the person objects corresponding to the first target feature data, the second target feature data, and the third target feature data is less than a preset posture difference threshold.

[0238] The first target feature data is compared with the second target feature data, and the first target feature data is compared with the third target feature data to determine the target error corresponding to the sample group;

[0239] Return and execute the step of randomly selecting a person object from the target feature database as the target person object and obtaining the first target feature data corresponding to the target person object, until the number of the sample group reaches the preset sample number threshold and the target error of the sample group conforms to a normal distribution.

[0240] A sample dataset is constructed based on the sample group.

[0241] Figure 6 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 6 As shown, the electronic device may include: a processor 610, a communications interface 620, a memory 630, and a communication bus 640, wherein the processor 610, the communications interface 620, and the memory 630 communicate with each other via the communication bus 640. The processor 610 can call logical instructions in the memory 630 to execute a multi-dimensional identity recognition method, which includes:

[0242] Acquire video stream data containing the object to be identified; the video stream data includes multiple frames of images.

[0243] Based on the first target image in the video stream data, multidimensional feature information of the object to be identified is extracted; the multidimensional feature information includes facial information and body key point information;

[0244] The identity of the object to be identified is determined based on the facial information. If the identification fails, the object to be identified is tracked based on the second target image in the video stream data to determine whether there is a third target image in the second target image that contains the identification result of the object to be identified. The second target image is an image in the video stream data that is located before the first target image.

[0245] If not, the identity of the object to be identified is determined based on the key body information.

[0246] Furthermore, the logical instructions in the aforementioned memory 630 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0247] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the multi-dimensional identity recognition method provided by the above methods, the method comprising:

[0248] Acquire video stream data containing the object to be identified; the video stream data includes multiple frames of images.

[0249] Based on the first target image in the video stream data, multidimensional feature information of the object to be identified is extracted; the multidimensional feature information includes facial information and body key point information;

[0250] The identity of the object to be identified is determined based on the facial information. If the identification fails, the object to be identified is tracked based on the second target image in the video stream data to determine whether there is a third target image in the second target image that contains the identification result of the object to be identified. The second target image is an image in the video stream data that is located before the first target image.

[0251] If not, the identity of the object to be identified is determined based on the key body information.

[0252] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-dimensional identity recognition method provided by the methods described above, the method comprising:

[0253] Acquire video stream data containing the object to be identified; the video stream data includes multiple frames of images.

[0254] Based on the first target image in the video stream data, multidimensional feature information of the object to be identified is extracted; the multidimensional feature information includes facial information and body key point information;

[0255] The identity of the object to be identified is determined based on the facial information. If the identification fails, the object to be identified is tracked based on the second target image in the video stream data to determine whether there is a third target image in the second target image that contains the identification result of the object to be identified. The second target image is an image in the video stream data that is located before the first target image.

[0256] If not, the identity of the object to be identified is determined based on the key body information.

[0257] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0258] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0259] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-dimensional identity recognition method, characterized in that, include: Acquire video stream data containing the object to be identified; the video stream data includes multiple frames of images, and each frame of the video stream data contains at least one object to be identified. Based on the first target image in the video stream data, extract the multidimensional feature information of the object to be identified; The multidimensional feature information includes facial information and body key point information; The identity of the object to be identified is determined based on the facial information. If the identification fails, the object to be identified is tracked based on the second target image in the video stream data to determine whether there is a third target image in the second target image that contains the identification result of the object to be identified. The second target image is the image in the video stream data that precedes the first target image; If not, the identity of the object to be identified is determined based on the body key point information; The process of identifying the identity of the object to be identified based on the body key point information includes: Based on the body key point information, the body image of the object to be identified is extracted from the first target image; Extract the body features of the object to be identified from the body image; In the body image, the target pose of the object to be identified is determined based on the relative positions of the body key points located on both sides of the object's body in the body key point information. The body features and target posture are matched with feature data recorded in a preset target feature database, and the identity of the object to be identified is determined based on the matching results.

2. The multi-dimensional identity recognition method of claim 1, wherein, The step of extracting the body image of the object to be identified from the first target image based on the body key point information includes: Based on the body key point information, determine the position coordinates and sharpness of each body key point in the first target image; Obtain a preset clarity filtering threshold, and remove body key points with a clarity lower than the clarity filtering threshold to obtain target key points with a clarity greater than or equal to the clarity filtering threshold; Based on the position coordinates of each target key point, calculate the first distance between each target key point and the first edge side of the first target image, and the second distance between each target key point and the second edge side of the first target image; the first edge side and the second edge side are two adjacent and perpendicular edge sides of the first target image; Based on the first distance and the second distance, distinguish between the foreground image and the background image of the person in the first target image; Extract the body image of the object to be identified from the foreground image of the person. 3.The multi-dimensional identity recognition method of claim 1, wherein, The step of tracking the target object based on the second target image in the video stream data to determine whether there is a third target image in the second target image containing the identification result of the target object includes: From the video stream data, obtain each of the second target images preceding the first target image; The first target image is used as the current image, and the previous frame image of the current image is obtained from each of the second target images as the third target image; A first bounding box is generated in the current image to identify the object, and a second bounding box is generated for each person object in the third target image. Calculate the first overlap between the first location box and each of the second location boxes, and determine the first target box corresponding to the object to be identified in each of the second location boxes based on the first overlap, so as to perform target tracking on the object to be identified; Based on the first target bounding box, determine whether the third target image contains the identity recognition result of the object to be identified; If not, the third target image is used as the current image, and the process returns to and executes the step of obtaining the previous frame image of the current image from each of the second target images as the third target image, until the third target image contains the identity recognition result of the object to be identified, or the third target image is the last frame image in the second target images.

4. The multi-dimensional identity recognition method of claim 3, wherein, The first target image includes multiple objects to be identified. Before calculating the overlap between the first bounding box and each of the second bounding boxes, the method further includes: Based on the first bounding box corresponding to each of the objects to be identified in the first target image, calculate the second overlap between each pair of the first bounding boxes; Based on the second bounding boxes corresponding to each person object in the third target image, calculate the third overlap between each pair of the second bounding boxes; Based on the second overlap, second target boxes in each of the first location boxes whose overlap is greater than a preset overlap threshold are removed; Based on the third overlap, third target boxes in each of the second location boxes whose overlap is greater than the overlap threshold are removed.

5. The multi-dimensional identity recognition method of claim 1, wherein, After successfully identifying the identity of the object to be identified based on the facial information, the process further includes: If the recognition is successful, the first body feature data of the object to be identified is extracted from the first target image; The first body feature data is labeled based on the identity recognition result of the object to be identified, and the labeled first body feature data is sent to a preset target feature database; If the identification fails, and the second target image contains a third target image that contains the identification result of the object to be identified, the second body feature data of the object to be identified is extracted from the third target image; Based on the identification result of the object to be identified in the second target image, the second body feature data is labeled, and the labeled second body feature data is sent to the target feature database; Randomly select a person object from the target feature database as the target person object, and obtain the first target feature data corresponding to the target person object; A sample dataset is constructed based on the first target feature data, and the body feature recognition model in the preset multidimensional identity recognition model is iteratively trained using the sample dataset to update the model parameters of the body feature recognition model; wherein, the multidimensional identity recognition model also includes a face recognition model and a target tracking model.

6. The multi-dimensional identity recognition method of claim 5, wherein, The construction of the sample dataset based on the first target feature data includes: Based on the first target feature data, a sample group is constructed by selecting second target feature data and third target feature data from the target identity database; Wherein, the person object corresponding to the second target feature data is the same as the target person object, the person object corresponding to the third target feature data is different from the target person object, and the posture difference between the person objects corresponding to the first target feature data, the second target feature data, and the third target feature data is less than a preset posture difference threshold. The first target feature data is compared with the second target feature data, and the first target feature data is compared with the third target feature data to determine the target error corresponding to the sample group; Return and execute the step of randomly selecting a person object from the target feature database as the target person object and obtaining the first target feature data corresponding to the target person object, until the number of the sample group reaches the preset sample number threshold and the target error of the sample group conforms to a normal distribution. A sample dataset is constructed based on the sample group.

7. A multi-dimensional identity recognition apparatus, characterized by, include: The acquisition module is used to acquire video stream data containing the objects to be identified; The video stream data includes multiple frames of images, and each frame of the video stream data contains at least one object to be identified. The extraction module is used to extract multidimensional feature information of the object to be identified based on the first target image in the video stream data; The multidimensional feature information includes facial information and body key point information; The first identification module is used to identify the identity of the object to be identified based on the facial information. If the identification fails, the object to be identified is tracked based on the second target image in the video stream data to determine whether there is a third target image in the second target image that contains the identification result of the object to be identified. The second target image is the image in the video stream data that precedes the first target image; The second identification module is used to identify the identity of the object to be identified based on the body key point information if the object does not exist. The first identification module is specifically used for: Based on the body key point information, the body image of the object to be identified is extracted from the first target image; Extract the body features of the object to be identified from the body image; In the body image, the target pose of the object to be identified is determined based on the relative positions of the body key points located on both sides of the object's body in the body key point information. The body features and target posture are matched with feature data recorded in a preset target feature database, and the identity of the object to be identified is determined based on the matching results.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the multi-dimensional identity recognition method as described in any one of claims 1 to 6.

9. A computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the multi-dimensional identity recognition method as described in any one of claims 1 to 6.

Images