Face tracking method and device, electronic equipment and storage medium

Abstract

The present application provides a kind of face tracking method, device, electronic equipment and storage medium, the method comprises: obtaining current frame in video frame sequence, and obtaining the face key point information of current frame;Similarity calculation is carried out to the face key point information of current frame and the face key point information of last frame, and whether the similarity meets preset requirement is determined;If it is determined that the similarity meets preset requirement, the target that needs to be tracked is determined according to the face area depth information obtained, wherein the face area depth information indicates the actual distance from the feature point of the face area of current frame to camera equipment.Because of the introduction of similarity and face area depth information, the present application can realize correct tracking even in complex scene.

Description

Technical Field

[0001] This invention relates to the field of video analysis technology, and in particular to a face tracking method, apparatus, electronic device, and storage medium that integrates depth information. Background Technology

[0002] Face detection locates the position of a face in an image, while face calibration outputs the coordinates of key points such as the eyes, eyebrows, nose, and mouth. Currently, face tracking typically involves first detecting the face's position using face detection methods, then mapping the face using a bounding box to obtain the face bounding box. The camera is then rotated around the center of the face bounding box to achieve face tracking.

[0003] During face tracking, in complex scenarios such as those involving black masks, rapid facial changes (from left to right side of the face, from looking up to looking down), or passengers in the back changing positions, the bounding box may not accurately capture the face, leading to incorrect tracking. Therefore, achieving accurate tracking is crucial. Summary of the Invention

[0004] This invention provides a face tracking method, apparatus, electronic device, and storage medium to solve the problem in the prior art that the tracked target is incorrect in complex scenarios.

[0005] In a first aspect, the present invention provides a face tracking method, the method comprising:

[0006] Obtain the current frame in the video frame sequence, and obtain the facial key point information of the current frame;

[0007] The similarity between the facial key point information of the current frame and the facial key point information of the previous frame is calculated, and it is determined whether the similarity meets the preset requirements.

[0008] If the similarity is determined to meet the preset requirements, the target to be tracked is determined based on the obtained face region depth information. The video frame sequence is obtained from the camera device, and the face region depth information represents the actual distance from the feature points of the face region in the current frame to the camera device.

[0009] In one embodiment of the present invention, the step of obtaining the facial key point information of the current frame includes:

[0010] The current frame is input into the face detection model, and face bounding box information is output.

[0011] The face bounding box information is input into the face calibration model, and the face key point information of the current frame is output.

[0012] The facial key point information includes one or more combinations of key point coordinates, confidence level of key point coordinates, key point score, depth features, and key point visibility.

[0013] In one embodiment of the present invention, the step of determining whether the similarity meets the preset requirements includes:

[0014] Determine whether the number of publicly visible facial key points in the current frame and the previous frame meets the first preset requirement;

[0015] Determine whether the similarity score meets the second preset requirement;

[0016] Determine whether the confidence level of the key point coordinates in the facial key point information meets the third preset requirement;

[0017] If all of the above-mentioned first to third preset requirements are met, it is determined that the similarity meets the preset requirements.

[0018] In one embodiment of the present invention, the method further includes:

[0019] If any of the first to third preset requirements mentioned above are not met, the accumulated value is obtained by tracking the error accumulation through the confidence level of the facial key point coordinates in the current frame.

[0020] Determine whether the cumulative value is greater than a first preset threshold;

[0021] If the accumulated value is greater than the first preset threshold, then return to the step of obtaining the current frame in the video frame sequence;

[0022] If the cumulative value is less than or equal to the first preset threshold, then the similarity is determined to meet the preset requirements.

[0023] In one embodiment of the present invention, the step of calculating the similarity between the facial key point information of the current frame and the facial key point information of the previous frame includes:

[0024] Depth features are obtained based on the facial key point information of the current frame to obtain the feature vector of the current frame;

[0025] The feature vector of the current frame and the feature vector of the previous frame are multiplied by a dot product to obtain a similarity value, and the similarity between the faces in the current frame and the previous frame is measured based on the magnitude of the similarity value.

[0026] In one embodiment of the present invention, the step of obtaining the depth information of the face region includes:

[0027] Obtain the basic parameters of the camera device and the coordinates of at least one reference point. The basic parameters include the intrinsic parameters, extrinsic parameters, and distortion parameters of the camera device. The reference point is a point located near the camera device and whose position is determined relative to the camera device.

[0028] Based on the facial key point information of the current frame, the basic parameters, and the coordinates of the at least one reference point, the depth distance value corresponding to the facial region depth information is obtained.

[0029] In one embodiment of the present invention, the step of determining the target to be tracked based on the acquired face region depth information includes:

[0030] Determine whether the depth distance value corresponding to the depth information of the face region is greater than a second preset threshold.

[0031] If the depth distance value corresponding to the depth information of the face region is greater than the second preset threshold, then return to the step of obtaining the current frame in the video frame sequence;

[0032] If the depth distance value corresponding to the depth information of the face region is less than or equal to the second preset threshold, a cost matrix is ​​constructed based on the depth distance value corresponding to the depth information of the face region, and the cost matrix is ​​solved by the Hungarian matching algorithm. The identifier corresponding to the face box of the current frame is determined based on the solution result, so as to determine the target to be tracked.

[0033] In one embodiment of the present invention, the step of constructing a cost matrix based on the depth distance value corresponding to the depth information of the face region, solving the cost matrix using the Hungarian matching algorithm, and determining the identifier corresponding to the face bounding box of the current frame based on the solution result includes:

[0034] Get the face bounding box and its corresponding identifier from the previous frame, and get the face bounding box from the current frame;

[0035] The cost matrix is ​​constructed using the depth distance values ​​corresponding to the depth information of the face region, and the minimum value is obtained based on the relative depth distance values ​​of each face box in the previous frame relative to each face box in the current frame.

[0036] Assign the identifier corresponding to the face bounding box in the previous frame to the face bounding box in the current frame corresponding to the minimum value, so as to determine the identifier corresponding to the face bounding box in the current frame.

[0037] Secondly, the present invention also provides a face tracking device, the device comprising:

[0038] The detection module is used to obtain the current frame in the video frame sequence and obtain the facial key point information of the current frame;

[0039] The calculation module is used to calculate the similarity between the facial key point information of the current frame and the facial key point information of the previous frame, and to determine whether the similarity meets the preset requirements.

[0040] The tracking module is used to determine the target to be tracked based on the acquired face region depth information when the similarity meets the preset requirements. The video frame sequence is acquired from the camera device, and the face region depth information represents the actual distance from the feature points of the face region in the current frame to the camera device.

[0041] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the face tracking method as described in any of the first aspects above.

[0042] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the face tracking method as described in any of the first aspects above.

[0043] This invention provides a face tracking method, apparatus, electronic device, and storage medium. After detecting facial key point information based on a neural network model, it introduces similarity calculation and determines whether tracking operation is needed based on the similarity score. If tracking operation is required, it further incorporates facial region depth information to more accurately determine the target to be tracked. Because of the introduction of similarity and facial region depth information, this invention can achieve correct tracking even in complex scenes. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0045] Figure 1 This is a flowchart of the face tracking method provided by the present invention;

[0046] Figure 2 This is a flowchart of obtaining facial key point information provided in an embodiment of the present invention;

[0047] Figure 3 This is a flowchart of similarity calculation provided in an embodiment of the present invention;

[0048] Figure 4This is a flowchart illustrating whether a tracking operation needs to be performed, provided in an embodiment of the present invention.

[0049] Figure 5 This is a flowchart illustrating whether the current frame is the first frame, provided in an embodiment of the present invention.

[0050] Figure 6 This is a flowchart of obtaining depth information of a face region provided in an embodiment of the present invention;

[0051] Figure 7 This is a flowchart for determining the target to be tracked, provided in an embodiment of the present invention;

[0052] Figure 8 This is a schematic diagram illustrating the determination of the ID of the face bounding box in the current frame according to an embodiment of the present invention;

[0053] Figure 9 This is a schematic diagram of the face tracking device provided by the present invention;

[0054] Figure 10 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0056] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein.

[0057] The technical terms involved in this invention are described below:

[0058] Face tracking is used in special situations to track specific facial targets. To track faces in an image, the first step is to identify them. Face recognition uses computers to analyze still images or video frame sequences to find faces and output information such as the number, location, and size of the faces. The second step is face tracking, which involves capturing the face's position and size in subsequent frames after detecting it.

[0059] To address the problem of target tracking errors in complex scenarios in existing technologies, this invention provides a face tracking method, apparatus, electronic device, and storage medium. After detecting facial key point information based on a neural network model, a similarity calculation is introduced. The similarity score is used to determine whether tracking is necessary. If tracking is required, face region depth information is introduced to further accurately determine the target to be tracked. By incorporating similarity and face region depth information, this invention enables correct tracking even in complex scenarios.

[0060] The following is combined Figures 1-10 The present invention describes a face tracking method, apparatus, electronic device, and storage medium.

[0061] Please refer to Figure 1 , Figure 1 This is a flowchart of the face tracking method provided by the present invention. A face tracking method, the method comprising:

[0062] Step 110: Obtain the current frame in the video frame sequence and obtain the facial key point information of the current frame.

[0063] Facial landmarks refer to semantically meaningful location points on a person's face, also known as feature points. In this embodiment of the invention, multiple facial landmarks for multiple face regions can be predefined. Optionally, each face region can correspond to multiple facial landmarks, and the information of each facial landmark can be recorded using its location information in the image.

[0064] For example, the face can be divided into key points such as the jawline, left eyebrow, right eyebrow, left eye, nose, and mouth.

[0065] For example, the facial key point information includes one or more combinations of key point coordinates, key point confidence score, key point score value, depth features, and key point visibility score.

[0066] Step 120: Calculate the similarity between the facial key point information of the current frame and the facial key point information of the previous frame, and determine whether the similarity meets the preset requirements.

[0067] Since the relative changes in face position and features between adjacent frames are minimal, the coordinates of the face bounding box from the previous frame can be reused for tracking operations, reducing the need to call neural network models. Measuring the entire face as a region of interest involves significant computational overhead and a lot of redundant information. Facial landmarks, however, are prominent and representative features of the face region. Therefore, the similarity of facial landmarks between two frames can be used to determine whether the similarity meets preset requirements, thus deciding whether face tracking is necessary. This approach offers the advantage of moderate computational complexity and accuracy.

[0068] Step 130: If the similarity is determined to meet the preset requirements, then the target to be tracked is determined based on the acquired face region depth information. The face region depth information represents the actual distance from the feature points of the face region in the current frame to the camera device.

[0069] Building upon step 120, face region depth information is further integrated. For example, the face tracking method described in this invention can be applied to the field of Occupant Monitor Systems (OMS), enabling face tracking of occupants within a car cabin. By acquiring face region depth information (i.e., obtaining the actual distance from the passenger's face region feature points to the camera device), it can solve the problem of target tracking errors caused by ID switching when rear passengers exchange seats.

[0070] The following is a detailed description of steps 110 to 130 above.

[0071] Please refer to Figure 2 , Figure 2 This is a flowchart of obtaining facial key point information provided in an embodiment of the present invention. In step 110 above, the step of obtaining the facial key point information of the current frame includes:

[0072] Step 210: Input the current frame into the face detection model and output the face bounding box information.

[0073] Face detection models are used to locate the position of faces in an image. For example, they can be implemented using techniques such as MTCNN (Multi-Task Convolutional Neural Network), Retina Face, and Tina Face. The output of a face detection model is the face bounding box information. The specific process can be found in the relevant technologies, which will not be detailed here.

[0074] Step 220: Input the face bounding box information into the face calibration model and output the face key point information of the current frame.

[0075] A face calibration model is used to locate facial key points (such as the precise positions of facial feature points like eyes, eyebrows, nose, mouth, and the outer contour of the face). Specifically, during calibration, it is built upon a face detection model. A region of interest (ROI) is extracted from the face and input into a pre-trained face calibration model. The model outputs the coordinates of all key points, the confidence level of the key point coordinates, the key point score, depth features, and the visibility of the key points.

[0076] Furthermore, the present invention also normalizes the coordinates of key points output by the face calibration model. For example, if the coordinate values ​​of key points output by the face calibration model are between -0.5 and 0.5, they need to be normalized to between 0 and 1.

[0077] Please refer to Figure 3 , Figure 3 This is a flowchart of similarity calculation provided in an embodiment of the present invention. Step 120 above, the step of calculating the similarity between the facial key point information of the current frame and the facial key point information of the previous frame, includes:

[0078] Step 310: Obtain depth features based on the facial key point information of the current frame to obtain the feature vector of the current frame.

[0079] It should be noted that deep features refer to the features on the feature map corresponding to the coordinates of the key points of the facial key point information.

[0080] Step 320: Perform a dot product between the feature vector of the current frame and the feature vector of the previous frame to obtain a similarity value, and measure the similarity between the faces in the current frame and the previous frame based on the magnitude of the similarity value.

[0081] In some embodiments of the present invention, please refer to Figure 4 , Figure 4 This is a flowchart provided by an embodiment of the present invention regarding whether a tracking operation needs to be performed. To further ensure that the similarity is credible and reliable in order to determine whether the similarity is safe, step 120, which determines whether the similarity meets preset requirements, includes:

[0082] Step 410: Determine whether the number of publicly visible facial key points in the current frame and the previous frame meets the first preset requirement.

[0083] It should be noted that the visibility of the facial keypoint information refers to the number of visible facial keypoints in the current frame, while the common visibility is the number of commonly visible facial keypoints in the current frame and the previous frame. For example, if the visibility of the corresponding positions of the facial keypoints in the current frame and the previous frame is both true, it is called common visibility.

[0084] If the first preset requirement is not met, it means that the similarity is insecure.

[0085] Step 420: Determine whether the similarity score meets the second preset requirement.

[0086] The similarity can be calculated using steps 310 and 320. Specifically, based on the first preset requirement of step 410, the features (i.e., depth features) corresponding to the facial key points (excluding facial contour key points) on the feature map are obtained according to their coordinates. This yields the feature vector of the current frame. The feature vector of the current frame is then multiplied by the feature vector of the previous frame to obtain a similarity value. The similarity between the faces in the current frame and the previous frame is then measured based on the magnitude of this similarity value.

[0087] In some embodiments of the present invention, the similarity can be calculated by the above steps 310 and 320. However, in order to ensure that the similarity is reliable, the present invention further excludes unreliable information such as contour points (for example, considering the influence of side profile and hair, facial contour points and eyebrow key points are considered unreliable points) from the similarity calculation.

[0088] If the second preset requirement is not met, it means that the similarity is insecure.

[0089] Step 430: Determine whether the confidence level of the key point coordinates in the facial key point information meets the third preset requirement.

[0090] Based on meeting the second preset requirement mentioned above, in order to further ensure that the similarity measurement is reliable, the confidence of the facial key point coordinates is subject to relevant constraints (the confidence of the facial key point coordinates represents the degree of accuracy of the prediction of the facial key points; facial key points below a certain confidence level are also considered unreliable), and it is determined whether its value meets the third preset requirement.

[0091] If the third preset requirement is not met, it means that the similarity is insecure.

[0092] Step 440: If all the above-mentioned first to third preset requirements are met, it is determined that the similarity meets the preset requirements, indicating that the similarity is safe. Then, a tracking operation can be performed, including adjusting the face bounding box of the current frame to obtain a tracking box.

[0093] Therefore, the similarity is considered safe only when the similarity value calculated from the credible facial feature vector (e.g., the confidence of the public visible points, the coordinates of the key points, and the similarity value of the feature vector all meet the above preset conditions) and the confidence of the facial key points both meet the preset conditions. Then, the tracking box of the current frame is obtained, which is obtained by expanding the key points of the previous frame into a rectangle, and the tracking box of the current frame is updated.

[0094] In some embodiments of the present invention, the present invention further includes:

[0095] Step 450: If any of the first to third preset requirements are not met, the accumulated value is obtained by accumulating the confidence level of the facial key point coordinates in the current frame to track the error.

[0096] If any of the first to third preset requirements in steps 410 to 430 above is not met, it indicates that the similarity is insecure.

[0097] For example, the tracking error accumulation can be a counter, and in this embodiment of the invention, the error accumulation is tracked by the confidence level of the key point coordinates.

[0098] Step 460: Determine whether the cumulative value is greater than the first preset threshold.

[0099] Step 470: If the accumulated value is greater than the first preset threshold, then return to the step of obtaining the current frame in the video frame sequence.

[0100] In other words, if the accumulated value is greater than the first preset threshold, then return to execute step 110 above.

[0101] Step 480: If the cumulative value is less than or equal to the first preset threshold, then it is determined that the similarity meets the preset requirements.

[0102] Then, a tracking operation can be performed, including adjusting the face bounding box of the current frame to obtain a tracking box. Specifically, the tracking box or face bounding box of the previous frame is expanded outward by a certain proportion, and the face calibration model is called to obtain the face key point information of the current frame. The minimum expansion matrix is ​​calculated for the key points of the face key point information of the current frame to obtain the tracking box, and the obtained tracking box is updated.

[0103] It should be noted that the tracking bounding box is obtained by adjusting the face bounding box according to its left-right proportions. If tracking was already performed in the previous frame, the face bounding box in the previous frame is also referred to as the tracking bounding box.

[0104] In some embodiments of the present invention, please refer to Figure 5 , Figure 5This is a flowchart illustrating whether the current frame is the first frame, provided in an embodiment of the present invention. After inputting the face bounding box information into the face calibration model in step 220 above, the method further includes:

[0105] Step 510: If no face frame information is output, determine whether the current frame is the first frame.

[0106] Step 520: If it is the first frame, then the first frame is determined as a negative example to indicate that no face can be detected.

[0107] It should be noted that negative examples refer to images in which no face is detected by the face detection model, or in other words, images that do not contain a face.

[0108] Step 530: If it is not the first frame, then perform the step of outputting the face frame information.

[0109] In other words, if it is not the first frame, then continue to execute the output face bounding box information in step 220.

[0110] In summary, steps 110 and 120 above are based on similarity calculations, primarily utilizing facial key points and corresponding depth feature visual information to determine the safety of the results. The safety of the similarity score is then used to determine whether tracking operations are necessary. If safe, tracking is performed; otherwise, if unsafe, tracking errors are accumulated and a threshold is applied to determine whether further tracking is required. Therefore, introducing similarity can improve tracking accuracy.

[0111] Furthermore, during the tracking process, complex scenarios may occur, such as two passengers exchanging seats in a car cabin, which can easily lead to ID swapping. For example, assuming face A's ID is 1 and face B's ID is 2, after the seat swap, face A's ID might become 2, while face B's ID might become 1. Therefore, to solve the ID switching problem and further improve tracking accuracy, this invention introduces face region depth information (i.e., step 130) to address this issue.

[0112] Please refer to Figure 6 , Figure 6 This is a flowchart of obtaining face region depth information provided in an embodiment of the present invention. Step 130 above, the step of obtaining the face region depth information includes:

[0113] Step 610: Obtain the basic parameters of the camera device and the coordinates of at least one reference point.

[0114] The basic parameters include the intrinsic parameters, extrinsic parameters, and distortion parameters of the camera device, and the reference point is a point located near the camera device and determined relative to the position of the camera device.

[0115] Specifically, the camera equipment can be calibrated in advance and a 3D model of the monitored area (such as a car cabin) can be created to establish a world coordinate system and obtain the intrinsic, extrinsic, and distortion parameters of the camera equipment, as well as the coordinates of the reference point.

[0116] For example, the reference point can be a point near the camera device and determined relative to the camera device's position. For instance, the reference point can be a point displayed on the car's central control screen, and the number of selected reference points can be one or more. The system uses in-vehicle voice prompts to encourage the passenger to gaze at the reference point and remain stationary, and uses the camera device to capture video clips of the passenger gazing at the reference point.

[0117] Step 620: Based on the facial key point information of the current frame, the basic parameters, and the coordinates of the at least one reference point, obtain the depth distance value corresponding to the facial region depth information.

[0118] For example, facial feature localization is performed on the acquired passenger image (i.e., the current frame). Three-dimensional facial key point information of the passenger can be obtained through a face detection model and a face calibration model. Then, based on the feature points of the face region (e.g., the pupil center), the coordinates of the reference point, and the basic parameters of the camera device, the actual distance from the feature points of the passenger's face region to the camera device can be obtained. Combining this with the triangle similarity rule, the depth information of the distance from the feature points of the passenger's face region to the camera device can be obtained.

[0119] For example, the calculation steps for the depth information of the face region can be found in the patent document (CN111860292A, Human Eye Positioning Method, Apparatus and Device Based on Monocular Camera), and will not be repeated here.

[0120] It should be noted that the process of obtaining the relevant parameters of the camera device in step 610 above only needs to be executed once. Subsequently, the relevant parameters of the camera device and the facial key point information of the current frame can be directly called to calculate the facial region depth information.

[0121] Please refer to Figure 7 , Figure 7 This is a flowchart illustrating the process of determining a target to be tracked, provided in an embodiment of the present invention. Step 130 above, which involves determining the target to be tracked based on the acquired face region depth information, includes:

[0122] Step 710: Determine whether the depth distance value corresponding to the depth information of the face region is greater than the second preset threshold.

[0123] Step 720: If the depth distance value corresponding to the depth information of the face region is greater than the second preset threshold, then return to the step of obtaining the current frame in the video frame sequence.

[0124] In other words, if the depth distance value corresponding to the depth information of the face region is greater than the second preset threshold, then return to execute the above step 110.

[0125] Step 730: If the depth distance value corresponding to the depth information of the face region is less than or equal to the second preset threshold, a cost matrix is ​​constructed based on the depth distance value corresponding to the depth information of the face region, and the cost matrix is ​​solved by the Hungarian matching algorithm. Based on the solution result, the identifier corresponding to the face box of the current frame is determined to identify the target to be tracked.

[0126] The Hungarian Algorithm is commonly used to find the maximum number of matches and the minimum number of vertex covers in a bipartite graph. A bipartite graph is a graph consisting of two groups of data where consecutive groups can be connected directly, but members of the same group cannot be connected directly. For example, in face tracking, the bounding boxes of faces in consecutive frames of a video can be considered as two groups of data, and there is a matching relationship between these two groups; that is, for the same target, the bounding boxes of faces in consecutive frames are a pair.

[0127] The Hungarian algorithm requires a cost matrix (or benefit matrix) as input. In face tracking, existing techniques construct the cost matrix based on the distance between ReID (Re-identification) features extracted from consecutive frames, or using the IoU (Intersection over Union) distance. However, neither the ReID feature distance nor the IoU distance results in high matching accuracy. Therefore, to improve matching accuracy, this invention uses the distance of face region depth information to construct the cost matrix.

[0128] Specifically, in step 730 above, the step of constructing a cost matrix based on the depth distance value corresponding to the face region depth information, solving the cost matrix using the Hungarian matching algorithm, and determining the identifier corresponding to the face bounding box in the current frame based on the solution result includes:

[0129] Step 7301: Obtain the face bounding box and its corresponding identifier from the previous frame, and obtain the face bounding box from the current frame.

[0130] Step 7302: Construct the cost matrix using the depth distance values ​​corresponding to the depth information of the face region, and find the minimum value based on the relative depth distance values ​​of each face box in the previous frame relative to each face box in the current frame.

[0131] Step 7303: Assign the identifier corresponding to the face frame of the previous frame to the face frame of the current frame corresponding to the minimum value, so as to determine the identifier corresponding to the face frame of the current frame.

[0132] Examples are given below:

[0133] The previous frame contained two face frames, A and B, with corresponding face IDs of 1 and 2 respectively.

[0134] The current frame contains three face bounding boxes, D, F, and G, whose corresponding face IDs are yet to be assigned.

[0135] A 2*3 code matrix C is constructed using the depth distance values ​​corresponding to the depth information of the face region. That is, the two face bounding boxes A and B of the previous frame are used as the rows of the code matrix C, and the three face bounding boxes DF and G of the current frame are used as the columns of the code matrix C.

[0136] Assume the relative depth distances between A and D, F, G are 0.25, 0.62, and 0.50; and the relative depth distances between B and D, F, G are 0.76, 0.32, and 0.90.

[0137] Since the smaller the distance, the better the match, which minimizes the total cost, the Hungarian matching algorithm can be used to obtain: A->D, B->F.

[0138] Therefore, A's face ID and face bounding box are assigned to D, B's face ID and bounding box are assigned to F, and G is the newly added trajectory with a face ID of 3.

[0139] It should be noted that each face bounding box A, B, D, F, G has its own depth distance value, so their relative depth distance values ​​can be calculated relative to the same coordinate system and reference point. For example, as shown... Figure 8 As shown, points O, E, and D are the pupil centers of the face region in the current frame, the face region of passenger ID1 in the previous frame, and the face region of passenger ID2 in the previous frame, respectively.

[0140] The coordinates of the pupil center in the image coordinate system are obtained through steps 110 and 120 above. d1 and d2 are calculated using Euclidean distance. If the result of the key point similarity is safe, the face region depth information (i.e., the actual distance from the pupil center of the face region in the current frame to the camera device) obtained in steps 610 and 620 above is used to construct a cost matrix. The Hungarian matching algorithm is then used to determine whether the face bounding box in the current frame is ID1 or ID2, thereby determining the target to be tracked and avoiding the problem of ID1 and ID2 being interchanged, which would cause the target to be tracked to be incorrect.

[0141] In summary, the face tracking method provided by this invention, by providing a similarity calculation step based on facial key points and a face tracking step based on fused facial region depth information, can achieve correct tracking in complex scenarios, improves the face tracking effect and efficiency, and solves the ID switching problem to a certain extent.

[0142] The face tracking device provided by the present invention is described below. The face tracking device described below can be referred to in correspondence with the face tracking method described above.

[0143] Please refer to Figure 9 , Figure 9 This is a schematic diagram of the face tracking device provided by the present invention. A face tracking device 900 includes a detection module 910, a calculation module 920, and a tracking module 930.

[0144] For example, the detection module 910 is used to obtain the current frame in the video frame sequence and obtain the facial key point information of the current frame.

[0145] For example, the calculation module 920 is used to calculate the similarity between the facial key point information of the current frame and the facial key point information of the previous frame, and to determine whether the similarity meets the preset requirements.

[0146] For example, the tracking module 930 is used to determine the target to be tracked based on the acquired face region depth information when it is determined that the similarity meets the preset requirements. The video frame sequence is obtained from the camera device, and the face region depth information represents the actual distance from the feature points of the face region in the current frame to the camera device.

[0147] For example, the detection module 910 is also used for:

[0148] The current frame is input into the face detection model, and face bounding box information is output.

[0149] The face bounding box information is input into the face calibration model, and the face key point information of the current frame is output.

[0150] The preset neural network model includes a face detection model and a face calibration model, and the face key point information includes one or more combinations of key point coordinates, confidence of key point coordinates, key point score, depth features, and key point visibility.

[0151] For example, the computing module 920 is also used for:

[0152] Determine whether the number of publicly visible facial key points in the current frame and the previous frame meets the first preset requirement;

[0153] Determine whether the similarity score meets the second preset requirement;

[0154] Determine whether the confidence level of the key point coordinates in the facial key point information meets the third preset requirement;

[0155] If all of the above-mentioned first to third preset requirements are met, it is determined that the similarity meets the preset requirements.

[0156] For example, the computing module 920 is also used for:

[0157] If any of the first to third preset requirements mentioned above are not met, the accumulated value is obtained by tracking the error accumulation through the confidence level of the facial key point coordinates in the current frame.

[0158] Determine whether the cumulative value is greater than a first preset threshold;

[0159] If the accumulated value is greater than the first preset threshold, then return to the step of obtaining the current frame in the video frame sequence;

[0160] If the cumulative value is less than or equal to the first preset threshold, then the similarity is determined to meet the preset requirements.

[0161] For example, the computing module 920 is also used for:

[0162] Depth features are obtained based on the facial key point information of the current frame to obtain the feature vector of the current frame;

[0163] The feature vector of the current frame and the feature vector of the previous frame are multiplied by a dot product to obtain a similarity value, and the similarity between the faces in the current frame and the previous frame is measured based on the magnitude of the similarity value.

[0164] For example, the detection module 910 is also used for:

[0165] If no face frame information is output, then determine whether the current frame is the first frame;

[0166] If it is the first frame, then the first frame is judged as a negative example to indicate that no face can be detected;

[0167] If it is not the first frame, then the step of outputting the face frame information is performed.

[0168] For example, the tracking module 930 is also used for:

[0169] Obtain the basic parameters of the camera device and the coordinates of at least one reference point. The basic parameters include the intrinsic parameters, extrinsic parameters, and distortion parameters of the camera device. The reference point is a point located near the camera device and whose position is determined relative to the camera device.

[0170] Based on the facial key point information of the current frame, the basic parameters, and the coordinates of at least one reference point, the depth distance value corresponding to the facial region depth information is obtained.

[0171] For example, the tracking module 930 is also used for:

[0172] Determine whether the depth distance value corresponding to the depth information of the face region is greater than a second preset threshold.

[0173] If the depth distance value corresponding to the depth information of the face region is greater than the second preset threshold, then return to the step of obtaining the current frame in the video frame sequence;

[0174] If the depth distance value corresponding to the depth information of the face region is less than or equal to the second preset threshold, a cost matrix is ​​constructed based on the depth distance value corresponding to the depth information of the face region, and the cost matrix is ​​solved by the Hungarian matching algorithm. The identifier corresponding to the face box of the current frame is determined based on the solution result, so as to determine the target to be tracked.

[0175] For example, the tracking module 930 is also used for:

[0176] Get the face bounding box and its corresponding identifier from the previous frame, and get the face bounding box from the current frame.

[0177] The cost matrix is ​​constructed using the depth distance values ​​corresponding to the depth information of the face region, and the minimum value is obtained based on the relative depth distance values ​​of each face box in the previous frame relative to each face box in the current frame.

[0178] Assign the identifier corresponding to the face bounding box in the previous frame to the face bounding box in the current frame corresponding to the minimum value, so as to determine the identifier corresponding to the face bounding box in the current frame.

[0179] It should be noted that the face tracking device provided in this embodiment of the invention can implement all the method steps implemented in the above method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0180] Figure 10 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present invention, such as... Figure 10As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the face tracking method, the method including:

[0181] The current frame in the video frame sequence is obtained and input into a preset neural network model to obtain the facial key point information of the current frame;

[0182] The similarity between the facial key point information of the current frame and the facial key point information of the previous frame is calculated, and a determination is made based on the similarity to determine whether a tracking operation needs to be performed.

[0183] If it is determined that a tracking operation needs to be performed, the target to be tracked is determined based on the obtained face region depth information, wherein the face region depth information represents the actual distance from the feature points of the face region in the current frame to the camera device.

[0184] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0185] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to execute the face tracking methods provided by the above methods.

[0186] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the face tracking methods provided above.

[0187] The present invention provides an electronic device, a computer program product, and a processor-readable storage medium, wherein the computer program stored thereon enables the processor to implement all the method steps implemented in the above method embodiments and achieve the same technical effect. Here, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail.

[0188] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0189] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0190] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A face tracking method, characterized in that, The method includes: Obtain the current frame in the video frame sequence, and obtain the facial key point information of the current frame; The similarity between the facial key point information of the current frame and the facial key point information of the previous frame is calculated; however, facial contour points and eyebrow key points are not included in the similarity calculation. Determine whether the number of publicly visible facial key points in the current frame and the previous frame meets a first preset requirement; determine whether the similarity score meets a second preset requirement; determine whether the confidence level of the key point coordinates in the facial key point information meets a third preset requirement; If the first to third preset requirements are met, the similarity is determined to be safe; if safe, a tracking operation is performed, including determining the target to be tracked based on the acquired face region depth information, wherein the video frame sequence is acquired from the camera device, and the face region depth information represents the actual distance from the feature points of the face region in the current frame to the camera device; If any of the first to third preset requirements is not met, the similarity is determined to be unsafe. In the case of insecurity, the accumulated error is tracked by the confidence score of the facial key point coordinates in the current frame to obtain an accumulated value. It is determined whether the accumulated value is greater than a first preset threshold. If the accumulated value is greater than the first preset threshold, the step of obtaining the current frame in the video frame sequence is returned. If the accumulated value is less than or equal to the first preset threshold, the similarity is determined to be safe, and the tracking operation is performed.

2. The face tracking method according to claim 1, characterized in that, The step of obtaining the facial key point information of the current frame includes: The current frame is input into the face detection model, and face bounding box information is output. The face bounding box information is input into the face calibration model, and the face key point information of the current frame is output. The facial key point information includes one or more combinations of key point coordinates, confidence level of key point coordinates, key point score, depth features, and key point visibility.

3. The face tracking method according to claim 1 or 2, characterized in that, The step of calculating the similarity between the facial key point information of the current frame and the facial key point information of the previous frame includes: Depth features are obtained based on the facial key point information of the current frame to obtain the feature vector of the current frame; The feature vector of the current frame and the feature vector of the previous frame are multiplied by a dot product to obtain a similarity value, and the similarity between the faces in the current frame and the previous frame is measured based on the magnitude of the similarity value.

4. The face tracking method according to claim 1, characterized in that, The steps for obtaining the depth information of the face region include: Obtain the basic parameters of the camera device and the coordinates of at least one reference point. The basic parameters include the intrinsic parameters, extrinsic parameters, and distortion parameters of the camera device. The reference point is a point located near the camera device and whose position is determined relative to the camera device. Based on the facial key point information of the current frame, the basic parameters, and the coordinates of at least one reference point, the depth distance value corresponding to the facial region depth information is obtained.

5. The face tracking method according to claim 1, characterized in that, The step of determining the target to be tracked based on the acquired face region depth information includes: Determine whether the depth distance value corresponding to the depth information of the face region is greater than a second preset threshold. If the depth distance value corresponding to the depth information of the face region is greater than the second preset threshold, then return to the step of obtaining the current frame in the video frame sequence; If the depth distance value corresponding to the depth information of the face region is less than or equal to the second preset threshold, a cost matrix is ​​constructed based on the depth distance value corresponding to the depth information of the face region, and the cost matrix is ​​solved by the Hungarian matching algorithm. The identifier corresponding to the face box of the current frame is determined based on the solution result, so as to determine the target to be tracked.

6. The face tracking method according to claim 5, characterized in that, The step of constructing a cost matrix based on the depth distance value corresponding to the depth information of the face region, solving the cost matrix using the Hungarian matching algorithm, and determining the identifier corresponding to the face bounding box of the current frame based on the solution result includes: Get the face bounding box and its corresponding identifier from the previous frame, and get the face bounding box from the current frame; The cost matrix is ​​constructed using the depth distance values ​​corresponding to the depth information of the face region, and the minimum value is obtained based on the relative depth distance values ​​of each face box in the previous frame relative to each face box in the current frame. Assign the identifier corresponding to the face bounding box in the previous frame to the face bounding box in the current frame corresponding to the minimum value, so as to determine the identifier corresponding to the face bounding box in the current frame.

7. A face tracking device, characterized in that, The device includes: The detection module is used to obtain the current frame in the video frame sequence and obtain the facial key point information of the current frame; The calculation module is used to calculate the similarity between the facial key point information of the current frame and the facial key point information of the previous frame, wherein facial contour points and eyebrow key points are not included in the similarity calculation; determine whether the number of commonly visible facial key points in the current frame and the previous frame meets a first preset requirement; determine whether the similarity meets a second preset requirement; determine whether the confidence of the key point coordinates in the facial key point information meets a third preset requirement; if all of the first to third preset requirements are met, the similarity is determined to be safe; if any of the first to third preset requirements are not met, the similarity is determined to be unsafe, and the accumulated error is tracked by the confidence of the facial key point coordinates in the current frame to obtain an accumulated value; determine whether the accumulated value is greater than a first preset threshold; if the accumulated value is greater than the first preset threshold, return to the step of obtaining the current frame in the video frame sequence; if the accumulated value is less than or equal to the first preset threshold, the similarity is determined to be safe. The tracking module is used to perform tracking operations under safe conditions, including determining the target to be tracked based on the acquired face region depth information, wherein the video frame sequence is acquired from the camera device, and the face region depth information represents the actual distance from the feature points of the face region in the current frame to the camera device.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the program, it implements the steps of the face tracking method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the face tracking method as described in any one of claims 1 to 6.

Images