Face tracking method and terminal device

Abstract

The application discloses a face tracking method and a terminal device, and relates to the technical field of computer vision. The method comprises the following steps: acquiring a current image frame in a video stream and a tracked face list; in the case that the tracked face list is empty or the interval between the current image frame and a previous detection frame is greater than or equal to a preset detection interval, determining the current image frame as a detection frame, performing face detection on the current image frame, and updating the tracked face list according to the face detection result; in the case that the tracked face list is not empty and the interval between the current image frame and the previous detection frame is less than the preset detection interval, determining the current image frame as a tracking frame, performing face tracking on the current image frame according to the tracked face list, and updating the tracked face list according to the face tracking result.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a face tracking method and terminal device. Background Technology

[0002] Face tracking is a core technology in computer vision, widely used in video surveillance, human-computer interaction, virtual reality, and augmented reality. With the development of deep learning technology, face detection and keypoint prediction methods based on deep learning have achieved significant improvements in accuracy. One face tracking method performs a complete face detection for each frame. This method has high tracking accuracy and can detect newly appearing faces or recover lost tracking in a timely manner, but it has a huge computational cost. Taking the SCRFD (Sample and Computation Redistribution for Efficient Face Detection) model as an example, it takes tens to hundreds of milliseconds to process a 1920×1080 image frame on a mobile device, which is difficult to meet the real-time requirements of a 30fps video stream. Summary of the Invention

[0003] In view of the above problems, this application provides a face tracking method and terminal device that significantly reduces computational overhead while ensuring tracking accuracy. The specific solution is as follows: The first aspect of this application provides a face tracking method, including: Obtain the current image frame and the list of tracked faces in the video stream, wherein the list of tracked faces is initially empty; If the list of tracked faces is empty or the interval between the current image frame and the previous detection frame is greater than or equal to the preset detection interval, the current image frame is determined as the detection frame, face detection is performed on the current image frame, and the list of tracked faces is updated according to the face detection results. If the list of tracked faces is not empty and the interval between the current image frame and the previous detection frame is less than the preset detection interval, the current image frame is determined as the tracking frame. Face tracking is performed on the current image frame according to the list of tracked faces, and the list of tracked faces is updated according to the face tracking results.

[0004] In one possible implementation, performing face detection on the current image frame and updating the tracked face list based on the face detection results includes: Preprocess the current image frame to generate a preview image of the first preset size; Map the bounding boxes of face objects in the tracked face list to the preview image to obtain the tracked region in the preview image. The region in the preview image other than the tracked region is the untracked region. Perform face detection on the untracked region to obtain a candidate face list corresponding to the untracked region; The tracked face list is updated based on the candidate face list.

[0005] In one possible implementation, face detection is performed on the untracked region to obtain a list of candidate faces corresponding to the untracked region, including: Perform face detection on the untracked region to obtain a list of candidate face frames corresponding to the untracked region; Remove candidate face frames from the candidate face frame list whose minimum side length is smaller than the preset minimum face pixel size; Facial landmark detection and face confidence detection are performed on the region within the candidate's face frame to obtain the coordinates of the landmarks and the face confidence corresponding to the candidate's face frame; Create a face object corresponding to the candidate face bounding box. The face object includes: face object identifier, bounding box, key point coordinates, and face confidence score. If the number of face objects is less than or equal to N, the face object is added to the candidate face list, where N≥1; If the number of face objects is greater than N, the face objects are sorted in descending order of face confidence, and the top N face objects are added to the candidate face list.

[0006] In one possible implementation, updating the tracked face list based on the candidate face list includes: Initial tracking processing is performed on each face object in the candidate face list; The face object that has undergone initial tracking is added to the list of tracked faces.

[0007] In one possible implementation, performing face tracking on the current image frame based on the tracked face list and updating the tracked face list based on the face tracking results includes: Based on the affine transformation matrix corresponding to the face object in the tracked face list, extract the face bounding box in the current image frame; Facial key point detection is performed on the region within the face bounding box to obtain the key point coordinates of the face object in the current image frame coordinate system; Establish the target affine transformation matrix between the key point coordinates of the face object in the current image frame coordinate system and the target key point coordinates in the average shape space of the standard face; The list of tracked faces is updated based on the face bounding box corresponding to the face object in the current image frame, the key point coordinates in the current image frame coordinate system, and the target affine transformation matrix.

[0008] In one possible implementation, the step of extracting a face bounding box in the current image frame based on the affine transformation matrix corresponding to the face object in the tracked face list includes: Based on the affine transformation matrix corresponding to the face object in the tracked face list, the first face bounding box is extracted in the current image frame; Perform face confidence detection on the region within the first face bounding box to obtain the face confidence corresponding to the first face bounding box; When the face confidence scores corresponding to all the first face bounding boxes are greater than or equal to the face confidence threshold, the affine transformation matrix corresponding to the face object is mathematically amplified, and a second face bounding box of a second preset size is extracted from the current image frame.

[0009] In one possible implementation, the face tracking method further includes: If the face confidence score corresponding to at least one of the first face bounding boxes is less than the face confidence score threshold, the current image frame is determined as the detection frame, and the steps of performing face detection on the current image frame and updating the tracked face list according to the face detection results are performed.

[0010] In one possible implementation, establishing the target affine transformation matrix between the keypoint coordinates of the face object in the current image frame coordinate system and the target keypoint coordinates in the average shape space of a standard face includes: Calculate the similarity transformation matrix using the key point coordinates of the face object in the current image frame coordinate system as the source point set and the average shape point set of the standard face as the target point set. Using the similarity transformation matrix, the key points of the face object in the current image frame coordinate system are mapped to the average shape space of the standard face; Using a preset average shape constraint algorithm, the coordinates of key points mapped to the average shape space of a standard face are corrected to obtain the coordinates of the target key points; The target affine transformation matrix is ​​calculated based on the key point coordinates of the face object and the target key point coordinates in the current image frame coordinate system.

[0011] In one possible implementation, the face tracking method further includes: After updating the list of tracked faces based on the face detection results, or after updating the list of tracked faces based on the face tracking results, the key point coordinates of each face object in the list of tracked faces are normalized to obtain the key point normalized coordinates. Output the structure corresponding to the list of tracked faces. The structure includes: the number of face objects, the key point coordinate array of each face object, and the bounding box. The key point coordinate array includes the normalized coordinates of each key point of the face object.

[0012] A second aspect of this application provides a terminal device, including at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the terminal device can implement the face tracking method of the first aspect or any implementation thereof.

[0013] By using the above technical solution, the face tracking method provided in this application changes the complete face detection operation from being executed every frame to being executed periodically by setting a preset detection interval. During the preset detection interval, face tracking is performed on the image frame to maintain the face state, thereby significantly reducing the computational overhead while ensuring tracking accuracy. Attached Figure Description

[0014] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0015] Figure 1 A flowchart illustrating a face tracking method provided in an embodiment of this application; Figure 2 This is a partial flowchart illustrating a face tracking method provided in an embodiment of this application. Figure 3 This is a partial flowchart illustrating a face tracking method provided in an embodiment of this application. Figure 4 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application. Detailed Implementation

[0016] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0017] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0018] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0019] Real-time face tracking solutions can be mainly divided into two categories: Full-frame detection scheme: Performs complete face detection, key point prediction, and pose estimation for each frame of the image. This method is accurate in tracking and can detect new faces in a timely manner, but the deep learning model has a large computational load, and inference of full-resolution images is time-consuming. It is difficult to meet the real-time requirements of 30fps and above on mobile devices, and the power consumption and performance overhead are too high.

[0020] Pure tracking scheme: Detection is performed only in the initial frame, and subsequent frames rely on tracking algorithms such as optical flow, Kalman filtering, and correlation filtering to maintain the state. This method has low computational overhead, but it is prone to tracking loss and drift when the face moves rapidly, is partially occluded, experiences sudden changes in lighting, or undergoes significant scale changes. Furthermore, it cannot automatically detect newly entering faces in the frame, resulting in poor robustness.

[0021] Furthermore, existing technologies suffer from two common drawbacks: first, the detection frames still employ full-resolution image inference, resulting in persistently high detection time; second, in multi-face scenarios, no special processing is applied to the tracked regions, leading to repeated identification of tracked faces by the detection algorithm, resulting in redundant computation and duplicate detection results. This necessitates additional post-processing for filtering, further burdening the device. These issues prevent existing solutions from simultaneously achieving low computational overhead, high tracking accuracy, strong stability, and non-duplication detection on mobile devices, making them ill-suited for the facial landmark detection and tracking requirements of real-time video streams.

[0022] This application provides a face tracking method applicable to terminal devices such as smartphones, tablets, wearable devices, and embedded terminals. The terminal device can employ local offline inference, requiring no network connection, and can still stably achieve face tracking even in network-free environments, ensuring privacy and real-time performance. The face tracking method of this application embodiment will be described in detail below with reference to the accompanying drawings.

[0023] Reference Figure 1 , Figure 1 This is a flowchart illustrating a face tracking method provided in an embodiment of this application, as shown below. Figure 1As shown in the embodiment of this application, a face tracking method may include steps 101 to 103, which are described in detail below.

[0024] 101: Get the current image frame and the list of tracked faces in the video stream.

[0025] Video streams can be video streams in scenarios such as video surveillance, human-computer interaction, virtual reality, and augmented reality. A video stream consists of multiple consecutive image frames.

[0026] The current image frame is the image frame to be processed in the video stream. If the current image frame is the first image frame in the video stream, the tracked face list is in its initial state and is empty. If the current image frame is not the first image frame in the video stream, the tracked face list includes the face objects detected or tracked in the previous image frame.

[0027] To facilitate subsequent processing, a frame processing object (FrameProcess object) corresponding to the current image frame can be generated. The frame processing object includes: image pixel data ( Parameters such as image width, image height, image stride, and image format (RGBA or BGRA) can be used.

[0028] It should be noted that this embodiment requires the use of a face detection model, a facial landmark detection model, and a face confidence model. Before executing the face tracking method provided in this embodiment, the Configuration method needs to be called, passing in the paths of the above three models to load and initialize them. Specifically, the face detection model can be a SCRFD (Sample and Computation Redistribution for Efficient Face Detection) model, the facial landmark detection model can be a PFLD (Principal Facial Landmark Detector) model, and the face confidence model can be an RNet (Refined Network) model.

[0029] 102: If the list of tracked faces is empty or the interval between the current image frame and the previous detection frame is greater than or equal to the preset detection interval, the current image frame is determined as the detection frame, face detection is performed on the current image frame, and the list of tracked faces is updated according to the face detection results.

[0030] 103: If the list of tracked faces is not empty and the interval between the current image frame and the previous detection frame is less than the preset detection interval, the current image frame is determined as the tracking frame. Face tracking is performed on the current image frame according to the list of tracked faces, and the list of tracked faces is updated according to the face tracking results.

[0031] The preset detection interval indicates how many frames are between each complete face detection. For example, a preset detection interval of 30 means that a complete face detection is performed every 30 frames, and face tracking is performed on the remaining frames.

[0032] For example, in this embodiment, the data input includes: the frame processing object of the current image frame and the list of tracked faces. The judgment logic is to determine whether any of the following conditions are met: (1) The list of tracked faces is empty (trackingFace.empty()), which means that no face has been tracked at present; (2) The interval between the current image frame and the previous detection frame is greater than the preset detection interval (e.g., 30 frames). Output: A boolean value, detection_executed, indicating whether face detection is performed. If any of the above conditions are met, detection_executed = true, and step 102 is executed; otherwise, detection_executed = false, and step 103 is executed.

[0033] This embodiment provides a face tracking method that, by setting a preset detection interval, changes the complete face detection operation from being performed every frame to being performed periodically. During the preset detection interval, face tracking is performed on the image frames to maintain the face state, thereby significantly reducing computational overhead while ensuring tracking accuracy.

[0034] In one possible implementation, please refer to Figure 2 One implementation of step 102 in the above embodiments includes the following steps 1021-1024: 1021: Preprocess the current image frame to generate a preview image of the first preset size.

[0035] Preprocessing the current image frame includes: rotating it so that the face is facing upwards and scaling it to a first preset size.

[0036] In this method, the current image frame is rotated to face-up by setting the image rotation mode, in order to be compatible with the rotation of the terminal device's camera.

[0037] By calling the `SetPreviewSize` method, the first preset size of the preview image is set (e.g., 192×160). Then, the `ExecutePreviewImageProcessing(true)` method is called to scale the current image frame and generate the preview image (`image_detect`). The preview image is a low-resolution image of the original input image, i.e., the current image frame, after scaling, used to accelerate the face detection process. The preview image size is usually smaller than the original input image.

[0038] 1022: Map the bounding boxes of face objects in the tracked face list to the preview image to obtain the tracked region in the preview image. The region in the preview image other than the tracked region is the untracked region.

[0039] Specifically, perform the following operations on each face object in the tracked face list: Get the bounding box of the face object (face.GetRect()), and obtain the rectangular region m_mask_rect corresponding to the bounding box of the face; The affine transformation matrix corresponding to the face object maps the coordinates of the rectangular region m_mask_rect to the tracked region mask_rect in the preview image. The affine transformation matrix is ​​a geometric transformation matrix used to transform the face region from the original image coordinate system to a standard aligned coordinate system (such as 112×112 pixels), supporting rotation, scaling, and translation operations.

[0040] 1023: Perform face detection on the untracked regions to obtain a list of candidate faces corresponding to the untracked regions.

[0041] By calling the BlackingTrackingRegion method to fill the mask_rect region with black, the face detection model can be invoked to perform face detection only on the untracked region, avoiding repeated detection in the already tracked region, reducing computational waste, and ensuring that newly appearing faces in the current image frame can be correctly detected.

[0042] For example, one specific implementation of step 1023 includes the following steps A1-A6: A1: Perform face detection on the untracked regions to obtain a list of candidate face bounding boxes corresponding to the untracked regions.

[0043] Specifically, since the tracked area is filled with black, the DetectFace method can be called to perform face detection on the preview image. The face detection model is used to detect faces in the preview image, resulting in a list of candidate face boxes, each containing at least one candidate face box.

[0044] A2: Remove candidate face frames from the candidate face frame list whose minimum side length is less than the preset minimum face pixel size.

[0045] The preset minimum face pixel size can be set according to the actual application scenario. By eliminating candidate face boxes that are too small, subsequent tracking of overly small faces can be avoided, thus reducing computational waste.

[0046] A3: Perform facial landmark detection and facial confidence detection on the region within the candidate's face frame to obtain the number of landmarks and facial confidence corresponding to the candidate's face frame.

[0047] It is understandable that the candidate face frames here are the candidate face frames remaining after removing smaller candidate face frames.

[0048] The facial landmark detection model is used to detect the coordinates of the key points corresponding to the face within the candidate's face bounding box.

[0049] The face confidence model is used to detect the face confidence corresponding to the region within the candidate's face box. The face confidence represents the probability value that the region within the candidate's face box is a face, and is used to characterize the credibility of the presence of a face within the candidate's face box. The value usually ranges from 0 to 1 (or 0 to 100%).

[0050] A4: Create the face object corresponding to the candidate's face frame.

[0051] Create a face object (FaceObjectInternal) corresponding to the candidate face bounding box. The face object includes: face object identifier, bounding box, key point coordinates and face confidence score. The bounding box is the aforementioned candidate face bounding box.

[0052] A5: If the number of face objects is less than or equal to N, add the face objects to the candidate face list, where N≥1.

[0053] A6: If the number of face objects is greater than N, sort the face objects in descending order of face confidence and add the top N face objects to the candidate face list.

[0054] N is set according to the actual application scenario. By setting N, the computational load and memory consumption of the terminal device are controlled, avoiding the increase in inference time, decrease in frame rate and reduction in tracking stability caused by too many faces in the picture. At the same time, priority is given to retaining face targets with high confidence and strong effectiveness, and redundant and invalid targets are eliminated to ensure that the face tracking method can run continuously, stably and efficiently offline in resource-constrained environments.

[0055] 1024: Update the list of tracked faces based on the list of candidate faces.

[0056] For example, one specific implementation of step 1024 includes the following steps B1-B2: B1: Perform initial tracking processing on each face object in the candidate face list.

[0057] At this point, the face object is in the detection state (face.TrackingState() == ISF_DETECT). The initial affine transformation matrix corresponding to the face object is calculated. The initial affine transformation matrix is ​​used to extract the face region from the current image frame, that is, the bounding box corresponding to the face.

[0058] Based on the initial affine transformation matrix corresponding to the face object, a first face bounding box is extracted from the current image frame. Face confidence detection is performed on the region within the first face bounding box to obtain the face confidence corresponding to the first face bounding box. If the face confidence corresponding to all first face bounding boxes is greater than or equal to the face confidence threshold, the affine transformation matrix corresponding to the face object is mathematically amplified. A second face bounding box of a second preset size is extracted from the current image frame. Face keypoint detection is performed on the region within the second face bounding box to obtain the keypoint coordinates corresponding to the second face bounding box. The keypoint coordinates are then subjected to an inverse affine transformation to obtain the keypoint coordinates in the current image frame coordinate system. Please refer to steps 1031-1032 in the following embodiments for details.

[0059] B2: Add the face object that has undergone initial tracking to the list of tracked faces.

[0060] Furthermore, the following operations can be performed on each face object in the tracked face list: Get the coordinates of key points of a face object (faces[t].landmark_). For example, get the coordinates of 106 key points of the face object. Get the bounding box of the face object (faces[t].getBbox()); Normalize the coordinates of key points on the face object: x = landmark.GetX() / width, y = landmark.GetY() / height (normalized coordinates of key points).

[0061] By normalizing the key point coordinates of the face object, the key point coordinates are converted from fixed pixel coordinates to relative coordinates (0~1) that are independent of the image size, ensuring consistent output across different devices and sizes, thus achieving universality across resolutions, devices, and sizes.

[0062] Write the above processing results for each face object into an output structure (p_human_action). The structure includes: Number of face objects (face_count); An array of keypoint coordinates for each face object, which includes the normalized coordinates of each keypoint of the face object; The bounding box (left, top, right, bottom) of each face object.

[0063] Depending on the specific application scenario, the above structure can be used to perform different processing on facial images. For example, the above structure can be used to perform beautification processing on facial images.

[0064] In one possible implementation, please refer to Figure 3 One implementation of step 103 in the above embodiments includes the following steps 1031-1034: 1031: Extract face bounding boxes in the current image frame based on the affine transformation matrix corresponding to the face objects in the tracked face list.

[0065] For example, one specific implementation of step 1031 includes the following steps C1-C3: C1: Extract the first face bounding box in the current image frame based on the affine transformation matrix corresponding to the face object in the tracked face list.

[0066] By calling the ExecuteImageAffineProcessing method, the first face bounding box is extracted from the current image frame using the transformation matrix corresponding to the face object in the tracked face list of the previous image frame.

[0067] C2: Perform face confidence detection on the region within the first face bounding box to obtain the face confidence corresponding to the first face bounding box.

[0068] The face confidence detection model is used to detect the face confidence in the region within the first face bounding box, and the face confidence corresponding to the first face bounding box is obtained. The face confidence corresponding to the first face bounding box represents the probability value that the region within the first face bounding box is a face.

[0069] C3: When the face confidence scores corresponding to all first face bounding boxes are greater than or equal to the face confidence threshold, the affine transformation matrix corresponding to the face object is mathematically amplified, and a second face bounding box of the second preset size is extracted in the current image frame.

[0070] The fact that the face confidence score corresponding to all first face bounding boxes is greater than or equal to the face confidence score threshold indicates that the tracking of all face objects is successful.

[0071] For example, the affine transformation matrix corresponding to the face object is magnified by 1.2 times, and the second face bounding box cropped in the current image frame is a second preset size, such as 112×112. Specifically, magnifying the affine transformation matrix corresponding to the face object by 1.2 times means magnifying the center by 1.2 times while keeping the center point unchanged.

[0072] By mathematically amplifying the affine transformation matrix corresponding to the face object, the field of view of face cropping is expanded without changing the model input size (112×112). Sufficient safety margins are left around the face to prevent the face from being cropped out of the model input area in two consecutive image frames due to slight movement, rotation, or offset. This ensures stable key point prediction and no loss of tracking.

[0073] If the face confidence score corresponding to at least one first face bounding box is less than the face confidence score threshold, it indicates that the face object tracking has failed. The current image frame is determined as the detection frame, and the face detection is performed on the current image frame in step 102 of the above embodiment. The list of tracked faces is updated according to the face detection results.

[0074] 1032: Perform facial landmark detection on the region within the face bounding box to obtain the coordinates of the key points of the face object in the current image frame coordinate system.

[0075] For example, a face landmark detection model is used to detect landmarks in the region within the face bounding box, resulting in the coordinates of 106 landmarks. The predicted landmark coordinates are then mapped back to the current image frame coordinate system (lmk_back, i.e., the original image coordinate system) through an inverse affine transformation: lmk_back = ApplyTransformToPoints(lmk_predict, affine_scale.GetInverse()), to obtain the landmark coordinates of the face object in the current image frame coordinate system (landmark_back).

[0076] 1033: Establish the target affine transformation matrix between the key point coordinates of the face object in the current image frame coordinate system and the target key point coordinates in the average shape space of the standard face.

[0077] For example, one specific implementation of step 1033 includes the following steps D1-D4: D1: Calculate the similarity transformation matrix using the key point coordinates of the face object in the current image frame coordinate system as the source point set and the average shape point set of the standard face as the target point set.

[0078] The mean shape set of a standard face is a set of 106 standard facial landmarks used for face alignment and pose correction.

[0079] Specifically, the similarity transformation matrix is ​​calculated using similarity transformation estimation, a method that calculates the similarity transformation matrix based on the source point set and the target point set, and is used to estimate the alignment transformation of the face region.

[0080] D2: Using a similarity transformation matrix, the key points of the face object in the current image frame coordinate system are mapped to the average shape space of the standard face.

[0081] Using a similarity transformation matrix, the coordinates of key points of the face object in the current image frame coordinate system are transformed in the forward direction and mapped to the average shape space of the standard face to obtain the center-aligned key points.

[0082] D3: Using a preset average shape constraint algorithm, the coordinates of key points mapped to the average shape space of a standard face are corrected to obtain the coordinates of the target key points.

[0083] The preset average shape constraint algorithm, such as the FixPointsMeanshape method, filters, smooths and corrects the key point coordinates mapped to the average shape space of a standard face, suppresses noise and jitter, and makes it closer to the average shape (inside_points) of a standard face, outputting stable corrected key points.

[0084] D4: Calculate the affine transformation matrix of the target based on the key point coordinates of the face object and the target key point coordinates in the current image frame coordinate system.

[0085] Using the key point coordinates of the face object in the current image frame coordinate system as input and the target key point coordinates as the target, calculate the target affine transformation matrix (trans_m) which includes rotation, translation, and scaling. This matrix can correct rotated and deflected faces into normal faces.

[0086] This embodiment calculates a similarity transformation matrix using the keypoint coordinates of the face object in the current image frame coordinate system as the source point set and the average shape point set of a standard face as the target point set. The keypoints of the face object in the current image frame coordinate system are mapped to the average shape space of the standard face and corrected / optimized. Then, the affine transformation matrix is ​​reconstructed by combining the keypoints of the face object in the current image frame coordinate system with the corrected keypoints. This affine transformation matrix effectively corrects rotated and deflected face poses, converting non-frontal faces into frontal standard faces adapted to the PFLD model, overcoming the limitation of the PFLD model which can only recognize frontal faces. Simultaneously, the corrected affine matrix is ​​updated to the face object, providing accurate transformation basis for predicting face keypoints in the next frame. This significantly improves the accuracy of face keypoint detection in rotational and pose-changing scenarios, reduces keypoint prediction errors and tracking drift problems, greatly enhances the robustness and stability of the face tracking system under complex poses, and ensures the continuity and reliability of the entire face tracking process.

[0087] 1034: Update the list of tracked faces based on the face bounding box corresponding to the face object in the current image frame, the key point coordinates in the current image frame coordinate system, and the target affine transformation matrix.

[0088] In this context, the bounding box corresponding to the face object in the current image frame is the bounding box of the face object.

[0089] Furthermore, the following operations can be performed on each face object in the updated list of tracked faces: Get the coordinates of key points of a face object (faces[t].landmark_). For example, get the coordinates of 106 key points of the face object. Get the bounding box of the face object (faces[t].getBbox()); Normalize the coordinates of key points on the face object: x = landmark.GetX() / width, y = landmark.GetY() / height (normalized coordinates of key points).

[0090] Write the above processing results for each face object into an output structure (p_human_action). The structure includes: Number of face objects (face_count); An array of keypoint coordinates for each face object, which includes the normalized coordinates of each keypoint of the face object; The bounding box (left, top, right, bottom) of each face object.

[0091] This application also provides a terminal device in its embodiments. (See reference...) Figure 4 The diagram illustrates a structural schematic suitable for implementing the terminal device in the embodiments of this application. The terminal device in the embodiments of this application may include, but is not limited to, fixed terminals such as smartphones, tablets, wearable devices, and embedded terminals. Figure 4 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0092] like Figure 4 As shown, the terminal device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. When the terminal device is powered on, the RAM 403 also stores various programs and data required for the operation of the terminal device. The processing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0093] Typically, the following devices can be connected to I / O interface 405: input devices 406 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 407 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 408 including, for example, memory cards, hard drives, etc.; and communication devices 409. Communication device 409 allows the terminal device to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 4 Terminal devices with various means are shown, but it should be understood that it is not required to implement or have all of the means shown. More or fewer means may be implemented or have alternatively.

[0094] This application also provides a computer program product including computer-readable instructions, which, when executed on a terminal device, cause the terminal device to implement any of the face tracking methods provided in this application.

[0095] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by a terminal device, the terminal device can implement any of the face tracking methods provided in this application.

[0096] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0097] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0098] In the above embodiments, the implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, in the form of a computer program product.

[0099] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A face tracking method, characterized in that, include: Obtain the current image frame and the list of tracked faces in the video stream, wherein the list of tracked faces is initially empty; If the list of tracked faces is empty or the interval between the current image frame and the previous detection frame is greater than or equal to the preset detection interval, the current image frame is determined as the detection frame, face detection is performed on the current image frame, and the list of tracked faces is updated according to the face detection results. If the list of tracked faces is not empty and the interval between the current image frame and the previous detection frame is less than the preset detection interval, the current image frame is determined as the tracking frame. Face tracking is performed on the current image frame according to the list of tracked faces, and the list of tracked faces is updated according to the face tracking results.

2. The face tracking method according to claim 1, characterized in that, The step of performing face detection on the current image frame and updating the list of tracked faces based on the face detection results includes: Preprocess the current image frame to generate a preview image of the first preset size; Map the bounding boxes of face objects in the tracked face list to the preview image to obtain the tracked region in the preview image. The region in the preview image other than the tracked region is the untracked region. Perform face detection on the untracked region to obtain a candidate face list corresponding to the untracked region; The tracked face list is updated based on the candidate face list.

3. The face tracking method according to claim 2, characterized in that, Face detection is performed on the untracked region to obtain a candidate face list corresponding to the untracked region, including: Perform face detection on the untracked region to obtain a list of candidate face frames corresponding to the untracked region; Remove candidate face frames from the candidate face frame list whose minimum side length is smaller than the preset minimum face pixel size; Facial landmark detection and face confidence detection are performed on the region within the candidate's face frame to obtain the coordinates of the landmarks and the face confidence corresponding to the candidate's face frame; Create a face object corresponding to the candidate face bounding box. The face object includes: face object identifier, bounding box, key point coordinates, and face confidence score. If the number of face objects is less than or equal to N, the face object is added to the candidate face list, where N≥1; If the number of face objects is greater than N, the face objects are sorted in descending order of face confidence, and the top N face objects are added to the candidate face list.

4. The face tracking method according to claim 2, characterized in that, The step of updating the tracked face list based on the candidate face list includes: Initial tracking processing is performed on each face object in the candidate face list; The face object that has undergone initial tracking is added to the list of tracked faces.

5. The face tracking method according to claim 1, characterized in that, The step of performing face tracking on the current image frame according to the list of tracked faces, and updating the list of tracked faces according to the face tracking results, includes: Based on the affine transformation matrix corresponding to the face object in the tracked face list, extract the face bounding box in the current image frame; Facial key point detection is performed on the region within the face bounding box to obtain the key point coordinates of the face object in the current image frame coordinate system; Establish the target affine transformation matrix between the key point coordinates of the face object in the current image frame coordinate system and the target key point coordinates in the average shape space of the standard face; The list of tracked faces is updated based on the face bounding box corresponding to the face object in the current image frame, the key point coordinates in the current image frame coordinate system, and the target affine transformation matrix.

6. The face tracking method according to claim 5, characterized in that, The step of extracting a face bounding box in the current image frame based on the affine transformation matrix corresponding to the face object in the tracked face list includes: Based on the affine transformation matrix corresponding to the face object in the tracked face list, the first face bounding box is extracted in the current image frame; Perform face confidence detection on the region within the first face bounding box to obtain the face confidence corresponding to the first face bounding box; When the face confidence scores corresponding to all the first face bounding boxes are greater than or equal to the face confidence threshold, the affine transformation matrix corresponding to the face object is mathematically amplified, and a second face bounding box of a second preset size is extracted from the current image frame.

7. The face tracking method according to claim 6, characterized in that, The face tracking method also includes: If the face confidence score corresponding to at least one of the first face bounding boxes is less than the face confidence score threshold, the current image frame is determined as the detection frame, and the steps of performing face detection on the current image frame and updating the tracked face list according to the face detection results are performed.

8. The face tracking method according to claim 5, characterized in that, The step of establishing the target affine transformation matrix between the key point coordinates of the face object in the current image frame coordinate system and the target key point coordinates in the average shape space of a standard face includes: Calculate the similarity transformation matrix using the key point coordinates of the face object in the current image frame coordinate system as the source point set and the average shape point set of the standard face as the target point set. Using the similarity transformation matrix, the key points of the face object in the current image frame coordinate system are mapped to the average shape space of the standard face; Using a preset average shape constraint algorithm, the coordinates of key points mapped to the average shape space of a standard face are corrected to obtain the coordinates of the target key points; The target affine transformation matrix is ​​calculated based on the key point coordinates of the face object and the target key point coordinates in the current image frame coordinate system.

9. The face tracking method according to claim 1, characterized in that, The face tracking method also includes: After updating the list of tracked faces based on the face detection results, or after updating the list of tracked faces based on the face tracking results, the key point coordinates of each face object in the list of tracked faces are normalized to obtain the key point normalized coordinates. Output the structure corresponding to the list of tracked faces. The structure includes: the number of face objects, the key point coordinate array of each face object, and the bounding box. The key point coordinate array includes the normalized coordinates of each key point of the face object.

10. A terminal device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the terminal device can implement the face tracking method as described in any one of claims 1 to 9.

Images