Face tracking methods, devices, electronic equipment and storage media

Abstract

This invention provides a face tracking method, apparatus, electronic device, and storage medium. The method includes: acquiring the current frame in a video frame sequence and determining the type of the current frame; if the current frame is a detection frame, performing face detection on the detection frame and performing different processing based on the face detection result, including: performing similarity calculation based on facial key point information when no face is detected, and selecting the optimal face based on depth information when a face is detected; if the current frame is a tracking frame, performing face calibration on the tracking frame and calculating the similarity between the facial key point information of the current frame and the facial key point information of the previous frame based on the face calibration result; when the similarity calculation result of the detection frame or the tracking frame meets a first preset requirement, acquiring a tracking box to perform a tracking operation. By introducing similarity and face region depth information, this invention enables correct tracking even in complex scenes.

Description

Technical Field

[0001] This invention relates to the field of video analysis technology, and in particular to a face tracking method, apparatus, electronic device, and storage medium that integrates depth information. Background Technology

[0002] Face detection locates the position of a face in an image, while face calibration outputs the coordinates of key points such as the eyes, eyebrows, nose, and mouth. Currently, face tracking typically involves first detecting the face's position using face detection methods, then mapping the face using a bounding box to obtain the face bounding box. The camera is then rotated around the center of the face bounding box to achieve face tracking.

[0003] During face tracking, in complex scenarios such as those involving black masks, rapid facial changes (from left to right side of the face, from looking up to looking down), or passengers in the back changing positions, the bounding box may not accurately capture the face, leading to incorrect tracking. Therefore, achieving accurate tracking is crucial. Summary of the Invention

[0004] This invention provides a face tracking method, apparatus, electronic device, and storage medium to solve the problem in the prior art that the tracked target is incorrect in complex scenarios.

[0005] In a first aspect, the present invention provides a face tracking method, the method comprising:

[0006] Obtain the current frame in the video frame sequence and determine the type of the current frame;

[0007] If the current frame is a detection frame, then face detection is performed on the detection frame, and different processing is performed according to the face detection result, including: when no face is detected, similarity calculation is performed based on the face key point information, and when a face is detected, the optimal face is selected based on the depth information.

[0008] If the current frame is a tracking frame, then face identification is performed on the tracking frame, and similarity calculation is performed on the facial key point information of the current frame and the facial key point information of the previous frame based on the face identification result.

[0009] When the similarity calculation result of the detection frame or the tracking frame meets the first preset requirement, a tracking box is obtained to perform a tracking operation.

[0010] In one embodiment of the present invention, the step of performing face detection on the detection frame if the current frame is a detection frame, and performing different processing based on the face detection result, includes: performing similarity calculation based on facial key point information when no face is detected, and selecting the optimal face based on depth information when a face is detected, including:

[0011] If the face detection result is that no face is detected, then continue to determine whether the current frame is a preset specified frame;

[0012] If so, it is determined that no face was detected in the current frame;

[0013] If not, the face bounding box information of the previous frame is obtained from the preset face detection model, and the preset face calibration model is called to calibrate the face bounding box information of the previous frame to obtain the face key point information of the current frame, and the similarity calculation of the face key point information of the current frame and the face key point information of the previous frame in the video frame sequence is performed.

[0014] If the similarity calculation result does not meet the first preset requirement, it is determined that no face was detected in the current frame.

[0015] In one embodiment of the present invention, the step of performing face detection on the detection frame if the current frame is a detection frame, and performing different processing based on the face detection result, includes: performing similarity calculation based on facial key point information when no face is detected, and selecting the optimal face based on depth information when a face is detected, further comprising:

[0016] If the result of the face detection is that a face is detected and at least one face bounding box is obtained, then it is further determined whether the value of the depth information corresponding to each face bounding box meets the second preset requirement.

[0017] If satisfied, the intersection-union ratio (IUU) of the tracking box or detection box of the previous frame in the video frame sequence with at least one face box in the current frame is calculated, and the face box with the largest IUU is selected as the optimal face to obtain the detection box corresponding to the optimal face.

[0018] If the conditions are not met, then the next frame in the video frame sequence will be processed.

[0019] In one embodiment of the present invention, the step of performing face identification on the tracking frame if the current frame is a tracking frame, and performing similarity calculation on the facial key point information of the current frame and the facial key point information of the previous frame based on the face identification result includes:

[0020] Obtain the facial key point information of the previous frame in the video frame sequence, and call the preset face calibration model to perform face calibration on the current frame to obtain the facial key point information of the current frame;

[0021] If the similarity calculation result between the facial key point information of the current frame and the facial key point information of the previous frame does not meet the first preset requirement, then the tracking error accumulation operation is continuously performed to obtain the error accumulation value.

[0022] In one embodiment of the present invention, the step of performing face calibration on the tracking frame if the current frame is a tracking frame, and performing similarity calculation on the facial key point information of the current frame and the facial key point information of the previous frame based on the face calibration result, further includes:

[0023] Compare the accumulated error value with a preset threshold.

[0024] If the cumulative error value is less than or equal to the preset threshold, a tracking box is obtained to perform a tracking operation;

[0025] If the cumulative error value is greater than the preset threshold, then a face tracking error is determined and a preset face detection model is called to perform face detection to obtain the face detection result.

[0026] In one embodiment of the present invention, the step of performing face calibration on the tracking frame if the current frame is a tracking frame, and performing similarity calculation on the facial key point information of the current frame and the facial key point information of the previous frame based on the face calibration result, further includes:

[0027] If the face detection result is that no face is detected, then it is determined that no face is detected in the current frame;

[0028] If the result of the face detection is that a face is detected and at least one face bounding box is obtained, then it is further determined whether the value of the depth information corresponding to each face bounding box meets the second preset requirement.

[0029] If satisfied, the intersection-union ratio (IUU) of the tracking box or detection box of the previous frame in the video frame sequence with at least one face box in the current frame is calculated, and the face box with the largest IUU is selected as the optimal face to obtain the detection box corresponding to the optimal face.

[0030] If the conditions are not met, then the next frame in the video frame sequence will be processed.

[0031] In one embodiment of the present invention, the step of performing face detection on the detection frame if the current frame is a detection frame further includes:

[0032] The detection frame is input into a preset face detection model for face detection. If the face detection result is that a face is detected, at least one face bounding box corresponding to the detection frame is output.

[0033] In one embodiment of the present invention, if the current frame is a tracking frame, the step of performing face identification on the tracking frame further includes:

[0034] The tracking frame is input into a preset face calibration model for face calibration, and the face key point information corresponding to the tracking frame is output. The face key point information includes one or more combinations of key point coordinates, confidence of key point coordinates, key point score, depth features and key point visibility.

[0035] Secondly, the present invention also provides a face tracking device, the device comprising:

[0036] The detection module is used to perform face detection on the current frame in the video frame sequence when the current frame is a detection frame, and to perform different processing based on the face detection results, including: performing similarity calculation based on facial key point information when no face is detected, and selecting the optimal face based on depth information when a face is detected.

[0037] The tracking module is used to perform face calibration on the tracking frame when the current frame in the video frame sequence is a tracking frame, and to perform similarity calculation on the face key point information of the current frame and the face key point information of the previous frame based on the face calibration result. When the similarity calculation result of the detection frame or the tracking frame meets the first preset requirement, the tracking box is obtained to perform the tracking operation.

[0038] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the face tracking method as described in any of the first aspects above.

[0039] Fourthly, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the face tracking method as described in any of the first aspects above.

[0040] This invention provides a face tracking method, apparatus, electronic device, and storage medium. It distinguishes between detection frames and tracking frames, calculates similarity, and determines whether tracking is necessary based on whether the similarity result meets a first preset requirement. Furthermore, it incorporates depth information based on the face detection result, selecting the optimal face to accurately acquire the detection bounding box. By introducing similarity and depth information, this invention enables correct tracking even in complex scenes. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0042] Figure 1 This is a flowchart of the face tracking method provided by the present invention;

[0043] Figure 2 This is a schematic diagram of the process of not detecting a face provided in an embodiment of the present invention;

[0044] Figure 3 This is a schematic diagram of the process for detecting a human face provided in an embodiment of the present invention;

[0045] Figure 4 This is a flowchart illustrating the tracking frame processing steps provided in an embodiment of the present invention;

[0046] Figure 5 This is a schematic diagram of the face tracking device provided by the present invention;

[0047] Figure 6 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0049] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein.

[0050] The technical terms involved in this invention are described below:

[0051] Face tracking is used in special situations to track specific facial targets. To track faces in an image, the first step is to identify them. Face recognition uses computers to analyze still images or video frame sequences to find faces and output information such as the number, location, and size of the faces. The second step is face tracking, which involves capturing the face's position and size in subsequent frames after detecting it.

[0052] To address the problem of target tracking errors in complex scenarios in existing technologies, this invention provides a face tracking method, apparatus, electronic device, and storage medium. By distinguishing between detection frames and tracking frames and introducing similarity calculation, the invention determines whether tracking operation is necessary based on whether the similarity calculation result meets a first preset requirement. Furthermore, depth information is introduced based on the face detection result, and the optimal face is selected based on this depth information to accurately obtain the detection bounding box. Due to the introduction of similarity and depth information, this invention can achieve correct tracking even in complex scenarios.

[0053] The face tracking method described in this invention can be applied to the field of passenger monitoring systems (OMS), enabling face tracking of occupants in a car cabin. It achieves face tracking by acquiring face region depth information (i.e., acquiring the actual distance from the passenger's face region feature points to the camera device). Specifically, it relates to a face tracking method that integrates depth information. This method mainly relies on a new strategy of combining facial key point similarity with face region depth information to achieve face tracking.

[0054] The following is combined with Figures 1-6 The present invention describes a face tracking method, apparatus, electronic device, and storage medium.

[0055] Please refer to Figure 1 , Figure 1 This is a flowchart of the face tracking method provided by the present invention. A face tracking method, the method comprising:

[0056] Step 110: Obtain the current frame in the video frame sequence and determine the type of the current frame.

[0057] The current frame refers to the current frame image in the current video stream. The current frame can be divided into detection frames and tracking frames. Detection frames and tracking frames can be distinguished based on the different models that need to be called. For example, frames that require calling a face detection model, or frames that require both a face detection model and a face calibration model, are called detection frames; frames that only require calling a face calibration model are called tracking frames. However, it is also possible to pre-define a frame from a predetermined number of frames every fixed interval in the video stream as a detection frame. For example, if the predetermined number of frames for a fixed interval is 25, then frames 1, 26, 46, 66, etc., can be designated as detection frames.

[0058] For example, the detection frame is input to a preset face detection model. If the face detection result is that a face is detected, then at least one face bounding box corresponding to the detection frame is output.

[0059] For example, the tracking frame is input to a preset face calibration model for face calibration, and the corresponding facial key point information is output. The facial key point information includes one or more combinations of key point coordinates, confidence scores of the key point coordinates, key point scores, depth features, and key point visibility.

[0060] It should be noted that the aforementioned preset face detection model is used to locate the position of a face in an image. For example, it can be achieved through techniques such as MTCNN (Multi-Task Convolutional Neural Network), Retina Face, and TinaFace. The output of the face detection model is the face bounding box information. The specific process can be referred to the relevant technologies for implementation, which will not be detailed here.

[0061] The aforementioned pre-set face calibration model is used to locate facial key points (such as the precise locations of facial feature points like eyes, eyebrows, nose, mouth, and the outer contour of the face). Specifically, during calibration, it is built upon a pre-set face detection model. A region of interest is extracted from the face and input into the pre-trained face calibration model to output information such as the coordinates of all key points, the confidence level of the key point coordinates, the key point score, depth features, and the visibility of the key points.

[0062] Step 120: If the current frame is a detection frame, then perform face detection on the detection frame and perform different processing based on the face detection results, including: performing similarity calculation based on facial key point information when no face is detected, and selecting the optimal face based on depth information when a face is detected.

[0063] The video frame sequence is obtained from the camera device, and the depth information refers to the actual distance from the feature points of the face region in the current frame to the camera device.

[0064] Specifically, the optimal face is selected based on depth information, i.e., by comparing whether the depth information meets a preset threshold. The preset threshold can be set based on experience, and this invention does not limit it. Generally speaking, the larger the preset threshold, the farther the optimal face is from the camera device, and the smaller the face; the smaller the preset threshold, the closer the optimal face is to the camera device, and the larger the face.

[0065] It should be noted that facial key points refer to semantically meaningful location points on a person's face, also known as feature points. In this embodiment of the invention, multiple facial key points for multiple face regions can be predefined. Optionally, each face region can correspond to multiple facial key points, and the information of each facial key point can be recorded using its position information in the image. For example, a face region can be divided into key points such as the jawline, left eyebrow, right eyebrow, left eye, nose, and mouth.

[0066] Step 130: If the current frame is a tracking frame, perform face calibration on the tracking frame, and calculate the similarity between the face key point information of the current frame and the face key point information of the previous frame based on the face calibration result.

[0067] Step 140: When the similarity calculation result of the detected frame or the tracked frame meets the first preset requirement, the tracking box is obtained to perform the tracking operation.

[0068] It should be noted that the similarity calculation can be based on the characteristic that the face positions and features of adjacent frames between the current frame and the previous frame do not change significantly. By reusing the coordinates of the face bounding box in the previous frame to perform the tracking operation, the number of calls to the face detection model can be reduced. If the entire face is used as the region of interest for measurement, the computational load is large and there is a lot of redundant information. However, facial landmarks are relatively significant and representative features of the face region. Therefore, this invention uses the similarity of facial landmarks between adjacent frames to determine whether to perform face tracking operations. Its advantage lies in moderate and accurate computation.

[0069] The following is a detailed description of steps 110 to 140 above.

[0070] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the process for handling undetected faces provided in an embodiment of the present invention. For example, in step 120 above, if the current frame is a detection frame, face detection is performed on that detection frame, and if the face detection result determines that no face was detected, similarity calculation is performed based on facial key point information; and if a face is detected, the optimal face is selected based on depth information.

[0071] Step 210: If the face detection result is that no face is detected, then continue to determine whether the current frame is a preset specified frame.

[0072] It should be noted that if no face is detected, it means that the current frame image itself does not contain a face, for example, it is a background image. Then, it is further determined whether the current frame is a preset specified frame. The preset specified frame is a frame that is pre-defined; that is, it can be a frame in a pre-defined video frame sequence, or it can be a frame that is forcibly called by the preset face detection model to recalculate a fixed number of frames as the preset specified frame, as described in step 416 below.

[0073] For example, assuming there are 25 frames per second in a video frame sequence, we can pre-set 25 frames as a fixed frame, meaning the preset number of frames for a fixed frame is 25. Then, we can set the first frame of every fixed frame as a preset designated frame, which can be designated as the detection frame. However, the system can also set the nth frame of every fixed frame as the preset designated frame, in which case the nth frame becomes the designated detection frame.

[0074] Step 220: If the current frame is a preset specified frame, then it is determined that no face was detected in the current frame.

[0075] In other words, assuming the first frame is a preset designated frame, if the current frame is not the first frame, it is directly determined that no face was detected in the current frame, and the current frame is judged as a negative example.

[0076] If a fixed frame has a preset frame count of 25, then the preset designated frames are frame 1, frame 26, frame 46, frame 66, etc., and other frames are not preset designated frames. The purpose of setting preset designated frames is to prevent inaccurate tracking boxes in extreme scenarios, such as when a face is turned to the side or when there are large angles such as vertical tilt, so that timely correction can be made.

[0077] Step 230: If the current frame is not a preset specified frame, obtain the face bounding box information of the previous frame from the preset face detection model, and call the preset face calibration model to calibrate the face bounding box information of the previous frame to obtain the face key point information as the current frame.

[0078] Step 240: Perform similarity calculation on the facial key point information of the current frame and the facial key point information of the previous frame in the video frame sequence.

[0079] It should be noted that the previous frame in a video frame sequence could be either a detection frame or a tracking frame. By calculating the similarity between the facial landmark information of the current frame and the facial landmark information of the previous frame in the video frame sequence, false positives in face detection can be prevented. This is because the relative changes in the face position and features between two adjacent frames are not significant, so the similarity of the facial landmarks between the two frames can be used to determine whether face tracking is necessary.

[0080] In some embodiments of the present invention, the method for calculating the similarity of detection frames or tracking frames may be:

[0081] Step 2401: Obtain depth features based on the facial key point information of the current frame to obtain the feature vector of the current frame.

[0082] Step 2402: Perform a dot product between the feature vector of the current frame and the feature vector of the previous frame to obtain a similarity value, and measure the similarity between the faces in the current frame and the previous frame based on the magnitude of the similarity value.

[0083] Step 250: If the similarity calculation result meets the first preset requirement, then obtain the tracking box to perform the tracking operation.

[0084] In some embodiments of the present invention, the method for determining whether the similarity calculation result meets the first preset requirement may be:

[0085] Step 2501: Determine whether the number of publicly visible facial key points in the current frame and the previous frame meets the preset requirement 1.

[0086] It should be noted that the visibility of the facial keypoint information refers to the number of visible facial keypoints in the current frame, while the common visibility is the number of commonly visible facial keypoints in the current frame and the previous frame. For example, if the visibility of the corresponding positions of the facial keypoints in the current frame and the previous frame is both true, it is called common visibility.

[0087] Step 2502: Determine whether the similarity score meets the second preset requirement.

[0088] The similarity can be calculated using the methods provided in steps 2401-2402 above. That is, based on satisfying the first preset requirement of step 2501, the features (i.e., depth features) corresponding to the facial key points (facial contour key points are not included in the calculation) are obtained on the feature map to obtain the feature vector of the current frame. The feature vector of the current frame and the feature vector of the previous frame are then multiplied to obtain a similarity value. The similarity between the faces in the current frame and the previous frame is measured based on the magnitude of this similarity value.

[0089] In some embodiments of the present invention, the similarity can be calculated by the method provided in steps 2401 to 2402 above. However, in order to ensure that the similarity is reliable, the present invention further excludes unreliable information such as contour points (for example, considering the influence of side profile and hair, facial contour points and eyebrow key points are considered unreliable points) from the similarity calculation.

[0090] Step 2503: Determine whether the confidence level of the key point coordinates in the facial key point information of the current frame meets the preset requirement three.

[0091] Based on satisfying the second preset requirement of step 2502 above, in order to further ensure that the similarity measurement is reliable, the confidence of the facial key point coordinates is subject to relevant constraints (the confidence of the facial key point coordinates represents the degree of accuracy of the prediction of the facial key points, and facial key points below a certain confidence level are also considered unreliable), and it is determined whether its value meets the third preset requirement.

[0092] Step 2504: If all the preset requirements one to three of steps 2501 to 2503 are met, it is determined that the similarity meets the first preset requirement.

[0093] Therefore, only when the similarity value calculated from the credible facial feature vector (e.g., the confidence of the public visible points, the coordinates of the key points, and the similarity value of the feature vector all meet the above preset conditions) and the confidence of the facial key points both meet the preset requirements, is it considered that the similarity meets the first preset requirements. Then, the tracking box of the current frame is obtained, which is obtained by expanding the key point information of the current frame into a rectangle.

[0094] Step 260: If the similarity calculation result does not meet the first preset requirement, it is determined that no face was detected in the current frame.

[0095] In other words, if the similarity calculation result does not meet the first preset requirement, it can be directly judged as a negative example, indicating that no face was detected in the current frame.

[0096] In summary, steps 210-260 above are the processing steps for cases where no face was detected, while the processing steps for cases where a face was detected are as follows:

[0097] Please refer to Figure 3 , Figure 3 This is a schematic diagram of the face detection process provided in an embodiment of the present invention. For example, in step 120 above, if the current frame is a detection frame, face detection is performed on the detection frame, and different processing is performed based on the face detection results, including: performing similarity calculation based on facial key point information when no face is detected, and selecting the optimal face based on depth information when a face is detected.

[0098] Step 310: If the face detection result is that a face is detected and at least one face bounding box is obtained, then continue to determine whether the value of the depth information corresponding to each face bounding box meets the second preset requirement.

[0099] In other words, face detection can detect one face or multiple faces. For example, the current frame might contain the faces of the driver and passengers. Since current technology typically selects the face with the largest area, if the face of a rear passenger or passerby is larger than the driver's, face switching can easily occur. That is, the system might be tracking a passenger instead of the driver. Therefore, introducing depth information can solve this problem of tracking errors caused by face switching.

[0100] For example, the depth information value corresponding to the face frame can be calculated by calling the relevant parameters of the camera device and inputting the key facial information of the current frame. For details, please refer to the patent document (CN111860292A, Human Eye Positioning Method, Device and Equipment Based on Monocular Camera), which will not be repeated here.

[0101] Step 320: If satisfied, calculate the intersection-union ratio (IUR) between the tracking box or detection box of the previous frame in the video frame sequence and at least one face box in the current frame, and select the face box with the largest IUR as the optimal face to obtain the detection box corresponding to the optimal face.

[0102] Intersection over Union (IoU) is a concept used in object detection. IoU calculates the overlap ratio between the predicted bounding boxes and the ground truth bounding boxes; that is, the ratio of their intersection to their union. Ideally, they overlap completely, meaning the ratio is 1.

[0103] For example, if only one face A is detected, it is determined whether the value of the depth information corresponding to the face bounding box A meets the second preset requirement. If it does, the face bounding box A is taken as the optimal face.

[0104] For example, if two faces A and B are detected, it is determined whether the depth information value corresponding to face bounding box A meets the second preset requirement, and whether the depth information value corresponding to face bounding box B also meets the second preset requirement. Assuming that the IoU value of A is 0.6 and the IoU value of B is 0.7, the face bounding box B with the largest IoU is selected as the optimal face.

[0105] Step 330: If the condition is not met, proceed to process the next frame in the video frame sequence.

[0106] In other words, if the depth information value corresponding to each face frame does not meet the second preset requirement, then the next frame in the video frame sequence will be processed.

[0107] In summary, the above Figure 2 and Figure 3 The processing steps for a current frame that is a detection frame, and the processing steps for a current frame that is a tracking frame, are as follows:

[0108] Please refer to Figure 4 , Figure 4 This is a flowchart illustrating the tracking frame processing steps provided in an embodiment of the present invention. For example, in step 130 above, if the current frame is a tracking frame, the step of performing face mapping on the tracking frame and calculating the similarity between the facial key point information of the current frame and the facial key point information of the previous frame based on the face mapping results includes:

[0109] Step 410: Obtain the facial key point information of the previous frame in the video frame sequence, and call the preset facial calibration model to perform facial calibration on the current frame to obtain the facial key point information of the current frame.

[0110] Step 411: Perform similarity calculation on the facial key point information of the current frame and the facial key point information of the previous frame.

[0111] Step 412: If the similarity calculation result meets the first preset requirement, then obtain the tracking box to perform the tracking operation.

[0112] It should be noted that the method for calculating the similarity of the detection frame or the tracking frame can be referred to in steps 240 to 250 above, and will not be repeated here.

[0113] Step 413: If the similarity calculation result does not meet the first preset requirement, then the tracking error accumulation operation is continuously performed to obtain the error accumulation value.

[0114] For example, the tracking error accumulation can be a counter, and in this embodiment of the invention, the error accumulation is tracked by the confidence level of the key point coordinates.

[0115] Step 414: Determine whether the cumulative error value is greater than a preset threshold.

[0116] Step 415: If the cumulative error value is less than or equal to the preset threshold, then obtain the tracking frame to perform the tracking operation.

[0117] Step 416: If the cumulative error value is greater than the preset threshold, then a face tracking error is determined and a preset face detection model is called to perform face detection to obtain the face detection result.

[0118] In other words, if the accumulated error value exceeds the preset threshold, face tracking is considered lost, and only a preset face detection model can be used for face detection. It should be noted that calling the preset face detection model here forces face detection to be called and restarts the frame count calculation.

[0119] Step 417: Determine whether the face detection result detects a face.

[0120] Step 418: If the face detection result is no face detected, it is determined that no face was detected in the current frame.

[0121] Step 419: If the face detection result is that a face is detected and at least one face bounding box is obtained, then continue to determine whether the value of the depth information corresponding to each face bounding box meets the second preset requirement.

[0122] Step 420: If satisfied, calculate the intersection-union ratio (IUR) between the tracking box or detection box of the previous frame in the video frame sequence and at least one face box in the current frame, and select the face box with the largest IUR as the optimal face to obtain the detection box corresponding to the optimal face.

[0123] Step 421: If the condition is not met, proceed to process the next frame in the video frame sequence.

[0124] It should be noted that the steps for determining depth information can be referred to in steps 310 to 330 above, and will not be repeated here.

[0125] In summary, the face tracking method provided by this invention, by providing a similarity calculation step based on facial key points and a face tracking step based on fused depth information, can achieve correct tracking in complex scenarios, improves the face tracking effect and efficiency, and solves the problem of target switching to a certain extent.

[0126] The face tracking device provided by the present invention is described below. The face tracking device described below can be referred to in correspondence with the face tracking method described above.

[0127] Please refer to Figure 5 , Figure 5 This is a schematic diagram of the face tracking device provided by the present invention. A face tracking device 500 includes a detection module 510 and a tracking module 520.

[0128] For example, the detection module 510 is used to:

[0129] When the current frame in the video frame sequence is identified as a detection frame, face detection is performed on the detection frame, and different processing is performed based on the face detection results, including: when no face is detected, similarity calculation is performed based on the face key point information, and when a face is detected, the optimal face is selected based on the depth information.

[0130] For example, the tracking module 520 is used for:

[0131] When the current frame in the video frame sequence is identified as a tracking frame, face identification is performed on the tracking frame. Based on the face identification result, similarity calculation is performed on the face key point information of the current frame and the face key point information of the previous frame. When the similarity calculation result of the detection frame or the tracking frame meets the first preset requirement, a tracking box is obtained to perform the tracking operation.

[0132] For example, the detection module 510 is also used for:

[0133] If the face detection result is that no face is detected, then continue to determine whether the current frame is a preset specified frame;

[0134] If so, it is determined that no face was detected in the current frame;

[0135] If not, the face bounding box information of the previous frame is obtained from the preset face detection model, and the preset face calibration model is called to calibrate the face bounding box information of the previous frame to obtain the face key point information of the current frame, and the similarity calculation of the face key point information of the current frame and the face key point information of the previous frame in the video frame sequence is performed.

[0136] If the similarity calculation result does not meet the first preset requirement, it is determined that no face was detected in the current frame.

[0137] For example, the detection module 510 is also used for:

[0138] If the result of the face detection is that a face is detected and at least one face bounding box is obtained, then it is further determined whether the value of the depth information corresponding to each face bounding box meets the second preset requirement.

[0139] If satisfied, the intersection-union ratio (IUU) of the tracking box or detection box of the previous frame in the video frame sequence with at least one face box in the current frame is calculated, and the face box with the largest IUU is selected as the optimal face to obtain the detection box corresponding to the optimal face.

[0140] If the conditions are not met, then the next frame in the video frame sequence will be processed.

[0141] For example, the tracking module 520 is also used for:

[0142] Obtain the facial key point information of the previous frame in the video frame sequence, and call the preset face calibration model to perform face calibration on the current frame to obtain the facial key point information of the current frame;

[0143] If the similarity calculation result between the facial key point information of the current frame and the facial key point information of the previous frame does not meet the first preset requirement, then the tracking error accumulation operation is continuously performed to obtain the error accumulation value.

[0144] For example, the tracking module 520 is also used for:

[0145] Compare the accumulated error value with a preset threshold.

[0146] If the cumulative error value is less than or equal to the preset threshold, a tracking box is obtained to perform a tracking operation;

[0147] If the cumulative error value is greater than the preset threshold, then a face tracking error is determined and a preset face detection model is called to perform face detection to obtain the face detection result.

[0148] For example, the tracking module 520 is also used for:

[0149] If the face detection result is that no face is detected, then it is determined that no face is detected in the current frame;

[0150] If the result of the face detection is that a face is detected and at least one face bounding box is obtained, then it is further determined whether the value of the depth information corresponding to each face bounding box meets the second preset requirement.

[0151] If satisfied, the intersection-union ratio (IUU) of the tracking box or detection box of the previous frame in the video frame sequence with at least one face box in the current frame is calculated, and the face box with the largest IUU is selected as the optimal face to obtain the detection box corresponding to the optimal face.

[0152] If the conditions are not met, then the next frame in the video frame sequence will be processed.

[0153] For example, the detection module 510 is also used for:

[0154] The detection frame is input into a preset face detection model for face detection. If the face detection result is that a face is detected, at least one face bounding box corresponding to the detection frame is output.

[0155] For example, the tracking module 520 is also used for:

[0156] The tracking frame is input into a preset face calibration model for face calibration, and the face key point information corresponding to the tracking frame is output. The face key point information includes one or more combinations of key point coordinates, confidence of key point coordinates, key point score, depth features and key point visibility.

[0157] It should be noted that the face tracking device provided in this embodiment of the invention can implement all the method steps implemented in the above method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0158] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiment of the present invention, such as... Figure 6 As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the face tracking method, the method including:

[0159] Obtain the current frame in the video frame sequence and determine the type of the current frame;

[0160] If the current frame is a detection frame, then face detection is performed on the detection frame, and different processing is performed according to the face detection result, including: when no face is detected, similarity calculation is performed based on the face key point information, and when a face is detected, the optimal face is selected based on the depth information.

[0161] If the current frame is a tracking frame, then face identification is performed on the tracking frame, and similarity calculation is performed on the facial key point information of the current frame and the facial key point information of the previous frame based on the face identification result.

[0162] When the similarity calculation result of the detection frame or the tracking frame meets the first preset requirement, a tracking box is obtained to perform a tracking operation.

[0163] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0164] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to execute the face tracking methods provided by the above methods.

[0165] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the face tracking methods provided above.

[0166] The present invention provides an electronic device, a computer program product, and a processor-readable storage medium, wherein the computer program stored thereon enables the processor to implement all the method steps implemented in the above method embodiments and achieve the same technical effect. Here, the parts that are the same as those in the method embodiments and the beneficial effects will not be described in detail.

[0167] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0168] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0169] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A face tracking method, characterized by, The method comprises: acquiring a current frame in a video frame sequence and judging a type of the current frame; if the current frame is a detection frame, performing face detection on the detection frame and performing different processing according to the result of face detection, including: when no face is detected, performing similarity calculation of face key point information according to face key point information of the current frame and face key point information of a previous frame in the video frame sequence, and when a face is detected, selecting an optimal face according to depth information; if the current frame is a tracking frame, performing face calibration on the tracking frame, and performing similarity calculation of face key point information of the current frame and face key point information of a previous frame according to the result of face calibration; when the result of similarity calculation of the detection frame or the tracking frame meets a first preset requirement, acquiring a tracking frame to perform a tracking operation.

2. The face tracking method of claim 1, wherein, The step of, if the current frame is a detection frame, performing face detection on the detection frame and performing different processing according to the result of face detection, includes: if the result of face detection is that no face is detected, continuing to judge whether the current frame is a preset designated frame; if yes, determining that the current frame does not detect a face; if no, acquiring face frame information of a previous frame from a preset face detection model, calling a preset face calibration model to calibrate the face frame information of the previous frame to obtain face key point information of the current frame, and performing similarity calculation of face key point information of the current frame and face key point information of a previous frame in the video frame sequence; if the result of similarity calculation does not meet the first preset requirement, determining that the current frame does not detect a face.

3. The face tracking method of claim 2, wherein, The step of, if the current frame is a detection frame, performing face detection on the detection frame and performing different processing according to the result of face detection, further includes: if the result of face detection is that a face is detected and at least one face frame is obtained, continuing to judge whether the value of depth information corresponding to each face frame meets a second preset requirement; if yes, calculating the intersection over union of a tracking frame or a detection frame of a previous frame in the video frame sequence and at least one face frame of the current frame, and selecting the face frame with the maximum intersection over union as an optimal face to obtain a detection frame corresponding to the optimal face; if no, performing processing on a next frame in the video frame sequence.

4. The face tracking method of claim 1, wherein, The step of, if the current frame is a tracking frame, performing face calibration on the tracking frame and performing similarity calculation of face key point information of the current frame and face key point information of a previous frame according to the result of face calibration, includes: acquiring face key point information of a previous frame in the video frame sequence, and calling a preset face calibration model to perform face calibration on the current frame to obtain face key point information of the current frame; If the similarity calculation result of the face key point information of the current frame and the face key point information of the previous frame does not satisfy the first preset requirement, a tracking error accumulation operation is continuously performed to obtain an error accumulation value.

5. The face tracking method of claim 4, wherein, If the current frame is a tracking frame, the tracking frame is subjected to face labeling, and the step of performing similarity calculation on the face key point information of the current frame and the face key point information of the previous frame according to the face labeling result further includes: comparing the error accumulation value with a preset threshold value; if the error accumulation value is less than or equal to the preset threshold value, a tracking frame is obtained to perform a tracking operation; if the error accumulation value is greater than the preset threshold value, it is determined that face tracking is incorrect, and a preset face detection model is called to perform face detection to obtain a face detection result.

6. The face tracking method of claim 5, wherein, If the current frame is a tracking frame, the tracking frame is subjected to face labeling, and the step of performing similarity calculation on the face key point information of the current frame and the face key point information of the previous frame according to the face labeling result further includes: if the face detection result is that no face is detected, it is determined that no face is detected in the current frame; if the face detection result is that a face is detected and at least one face frame is obtained, it is further determined whether the value of the depth information corresponding to each face frame satisfies a second preset requirement; if yes, the intersection over union of the tracking frame or the detection frame of the previous frame in the video frame sequence and at least one face frame of the current frame is calculated, and the face frame with the maximum intersection over union is selected as an optimal face to obtain a detection frame corresponding to the optimal face; if no, the next frame in the video frame sequence is processed.

7. The face tracking method of claim 1, wherein, If the current frame is a tracking frame, the tracking frame is subjected to face labeling, and the step of performing similarity calculation on the face key point information of the current frame and the face key point information of the previous frame according to the face labeling result further includes: the detection frame is input into a preset face detection model for face detection, and if the face detection result is that a face is detected, at least one face frame corresponding to the detection frame is output.

8. The face tracking method of claim 1, wherein, If the current frame is a tracking frame, the tracking frame is subjected to face labeling, and the step of performing similarity calculation on the face key point information of the current frame and the face key point information of the previous frame according to the face labeling result further includes: the tracking frame is input into a preset face labeling model for face labeling to output face key point information corresponding to the tracking frame, wherein the face key point information includes one or more combinations of key point coordinates, confidence of key point coordinates, key point score value, depth feature and visibility of key points.

9. A face tracking apparatus, characterized by comprising: The device includes: a detection module configured to, when a current frame in a video frame sequence is detected as a detection frame, perform face detection on the detection frame, and perform different processing according to the face detection result, including performing similarity calculation of face key point information according to face key point information of the current frame and face key point information of a previous frame in the video frame sequence when no face is detected, and selecting an optimal face according to depth information when a face is detected; A tracking module is configured to, when a type of a current frame in the video frame sequence is a tracking frame, perform face labeling on the tracking frame, perform similarity calculation on face key point information of the current frame and face key point information of a previous frame according to a result of the face labeling, and obtain a tracking box to perform a tracking operation when a result of the similarity calculation of the detection frame or the tracking frame meets a first preset requirement.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, The processor implements the steps of the face tracking method according to any one of claims 1 to 8 when executing the program. 11.A non-transitory computer-readable storage medium having stored thereon a computer program. The computer program, when executed by the processor, implements the steps of the face tracking method according to any one of claims 1 to 8.

Images