Face tracking method and apparatus for mobile terminal, and storage medium and electronic device

Abstract

Disclosed in the present application are a face tracking method and apparatus for a mobile terminal, and a storage medium and an electronic device. The method comprises: acquiring a first transformation matrix determined on the basis of a previous original image frame; on the basis of the first transformation matrix, performing alignment on a current original image frame to obtain a current facial image frame; on the basis of a CNN regression model, performing detection on the current facial image frame to obtain positions of facial landmarks and the position of a face bounding box in the current facial image frame; on the basis of the positions of the facial landmarks and the position of the face bounding box in the current facial image frame and a transpose of the first transformation matrix, determining positions of facial landmarks in the current original image frame; and on the basis of the positions of the facial landmarks in the current original image frame and positions of reference points, determining a second transformation matrix, so as to detect positions of facial landmarks in a next original image frame on the basis of the second transformation matrix and the CNN regression model, wherein when the previous frame is an initial frame, the first transformation matrix is determined on the basis of a target detection model, an initial original image frame, the regression model and the positions of the reference points.

Description

Mobile face tracking methods, devices, storage media and electronic devices Technical Field

[0001] This application relates to the field of image processing technology, and more specifically, to a mobile face tracking method, apparatus, storage medium, and electronic device. Background Technology

[0002] Face tracking is an important task in computer vision, with the primary goal of continuously locating faces in a video stream. This typically involves first detecting a face in the initial frame of the video and then tracking the movement of that face in subsequent frames.

[0003] In related technologies, face tracking is achieved by using computationally intensive and complex object detection models. However, for mobile applications running on mobile devices, due to the limited computing power of mobile devices, using computationally intensive and complex object detection models to perform face detection in every frame would result in excessive computational consumption, making real-time tracking impossible. Summary of the Invention

[0004] This application provides a mobile face tracking method, apparatus, storage medium, and electronic device, which can solve the problem that using a computationally intensive and complex target detection model for frame-by-frame face detection on mobile devices with limited computing power leads to excessive computational consumption and makes real-time tracking impossible.

[0005] The specific technical solution is as follows:

[0006] In a first aspect, embodiments of this application provide a mobile face tracking method, the method comprising:

[0007] Obtain a first transformation matrix, wherein the first transformation matrix is ​​a transformation matrix that transforms the facial key points detected from the previous frame original image to reference points, and the reference points are facial key points on a standard face image;

[0008] Align the original image of the current frame with the first transformation matrix to obtain the face image of the current frame;

[0009] The face image in the current frame is detected based on the convolutional neural network (CNN) regression model to obtain the positions of facial key points and the positions of facial bounding boxes in the current frame face image.

[0010] The positions of facial key points in the original image of the current frame are determined based on the positions of facial key points in the current frame face image, the positions of facial rectangles in the current frame face image, and the transposed first transformation matrix.

[0011] Based on the positions of facial key points in the current frame original image and the position of the reference point, a second transformation matrix is ​​determined to transform the facial key points in the current frame original image to the reference point, so as to detect the positions of facial key points in the next frame original image based on the second transformation matrix, the next frame original image and the CNN regression model;

[0012] Wherein, when the previous frame original image is the initial frame original image, the first transformation matrix is ​​determined based on the initial frame face image detected by the target detection model from the initial frame original image, the CNN regression model, and the position of the reference point.

[0013] As can be seen from the above scheme, the embodiments of this application only require a highly complex and computationally intensive target detection model when detecting facial key points in the initial frame original image. The detection of facial key points in subsequent frames original images only requires simple detection and mathematical calculation based on a lightweight CNN regression model and transformation matrix. Therefore, for mobile devices with limited computing power, the method provided by this application can meet the real-time requirements of face tracking and ensure accuracy under limited computing resources.

[0014] In a first possible implementation of the first aspect, determining the first transformation matrix based on the initial frame face image detected from the initial frame original image by the object detection model, the CNN regression model, and the position of the reference point includes:

[0015] The CNN regression model is used to detect the face image in the initial frame to obtain the positions of facial key points and the positions of face bounding boxes in the initial frame face image.

[0016] The positions of facial key points in the original image of the initial frame are determined based on the positions of facial key points in the initial frame face image and the positions of facial rectangles in the initial frame face image.

[0017] The first transformation matrix is ​​determined based on the positions of facial key points in the original image of the initial frame and the positions of the reference points.

[0018] In a second possible implementation of the first aspect, the initial frame face image is detected from the original initial frame image by an object detection model, including:

[0019] Based on the target detection model, face detection is performed on the original image of the initial frame to obtain the face bounding box of the original image of the initial frame.

[0020] The face bounding box is extracted from the original image of the initial frame to obtain the face image of the initial frame.

[0021] In a third possible implementation of the first aspect, before extracting the face bounding box from the original image of the initial frame to obtain the face image of the initial frame, the method further includes:

[0022] The face rectangle of the original image of the initial frame is expanded outward to obtain an expanded face rectangle, wherein the expanded face rectangle is a sub-image of the original image of the initial frame;

[0023] The step of extracting the face bounding box from the original image of the initial frame to obtain the face image of the initial frame includes:

[0024] The expanded face rectangle is cropped to obtain the initial frame face image.

[0025] As can be seen from the above scheme, after detecting the face bounding box of the original image of the initial frame based on the target detection model, the face bounding box of the original image of the initial frame is first expanded outward, and then the expanded face bounding box is cropped as the face image of the initial frame. This makes the face region of the initial frame face image partially redundant, which can prevent the situation where the face region is missing when the subsequent frames are aligned according to the transformation matrix, thus ensuring the integrity of the face region and improving the accuracy of face key point detection in subsequent frames.

[0026] In a fourth possible implementation of the first aspect, a second transformation matrix is ​​determined based on the positions of facial key points in the current frame original image and the position of the reference point, including:

[0027] When the confidence level of the face image in the current frame is greater than or equal to a preset confidence threshold when the face key points in the previous frame original image are detected based on the CNN regression model, and the displacement between the face key points in the previous frame original image and the face key points in the current frame original image is less than or equal to a preset displacement threshold, the second transformation matrix from the face key points in the current frame original image to the reference point is determined according to the position of the face key points in the current frame original image and the position of the reference point.

[0028] As can be seen from the above scheme, in this embodiment of the application, the confidence level output when detecting the face image of the current frame based on the CNN regression model is greater than or equal to the preset confidence threshold, and the displacement between the face key points of the previous frame original image and the face key points of the current frame original image is less than or equal to the preset displacement threshold, will determine the second transformation matrix from the face key points in the current frame original image to the reference point. This avoids the reduction in the detection accuracy of the face key points in the current frame face image due to low confidence or large displacement, which in turn reduces the detection accuracy of the face key points in the current frame original image and the face key points in the next frame original image determined based on the second transformation matrix.

[0029] In a fifth possible implementation of the first aspect, the method further includes:

[0030] If the confidence level output by the CNN regression model when detecting the face image in the current frame is less than the preset confidence threshold, or if the displacement between the face key points in the previous frame and the face key points in the current frame is greater than the preset displacement threshold, the current frame is used as the new initial frame and the face key points in the current frame are detected again.

[0031] As can be seen from the above scheme, since the current frame original image is used as the new initial frame original image, the face image of the current frame is detected by the object detection model. The position of the facial key points in the current frame original image is only based on the position of the facial key points and the position of the face rectangle in the current frame face image, and no longer depends on the first transformation matrix corresponding to the previous frame original image. Therefore, using the current frame original image as the new initial frame original image and re-detecting the facial key points in the current frame original image will greatly improve the accuracy of facial key point detection.

[0032] Secondly, embodiments of this application provide a mobile face tracking device, the device comprising:

[0033] An acquisition unit is used to acquire a first transformation matrix, wherein the first transformation matrix is ​​a transformation matrix that transforms facial key points detected from the previous frame's original image to reference points, and the reference points are facial key points on a standard face image;

[0034] An alignment unit is used to align the original image of the current frame based on the first transformation matrix to obtain the face image of the current frame;

[0035] The detection unit is used to detect the face image of the current frame based on the convolutional neural network (CNN) regression model, and obtain the positions of the facial key points and the positions of the face bounding boxes in the face image of the current frame.

[0036] The first determining unit is used to determine the position of the facial key points in the original image of the current frame based on the position of the facial key points in the current frame face image, the position of the face rectangle in the current frame face image, and the transposed first transformation matrix.

[0037] The second determining unit is used to determine a second transformation matrix from the facial key points in the current frame original image to the reference point based on the facial key point positions in the current frame original image and the reference point positions, so as to detect the facial key point positions in the next frame original image based on the second transformation matrix, the next frame original image and the CNN regression model.

[0038] Wherein, when the previous frame original image is the initial frame original image, the first transformation matrix is ​​determined based on the initial frame face image detected by the target detection model from the initial frame original image, the CNN regression model, and the position of the reference point.

[0039] In a first possible implementation of the second aspect, the acquiring unit includes:

[0040] The first detection module is used to detect the face image of the initial frame based on the CNN regression model when the previous frame original image is the initial frame original image, and to obtain the position of the face key points and the position of the face rectangle in the initial frame face image.

[0041] The first determining module is used to determine the position of the facial key points in the original image of the initial frame based on the position of the facial key points in the initial frame face image and the position of the face rectangle in the initial frame face image;

[0042] The second determining module is used to determine the first transformation matrix based on the positions of facial key points in the original image of the initial frame and the positions of the reference points.

[0043] In a second possible implementation of the second aspect, the acquiring unit further includes:

[0044] The second detection module is used to perform face detection on the original image of the initial frame based on the target detection model to obtain the face bounding box of the original image of the initial frame; and to extract the face bounding box from the original image of the initial frame to obtain the face image of the initial frame.

[0045] In a third possible implementation of the second aspect, the second detection module is used to expand the face rectangle of the original image of the initial frame outward before extracting the face rectangle in the original image of the initial frame to obtain the face image of the initial frame, thereby obtaining an expanded face rectangle, wherein the expanded face rectangle is a sub-image of the original image of the initial frame; and extracting the expanded face rectangle to obtain the face image of the initial frame.

[0046] In a fourth possible implementation of the second aspect, the second determining unit is configured to determine, when the confidence level output by the CNN regression model when detecting the face image in the current frame is greater than or equal to a preset confidence threshold, and the displacement between the face key points in the previous frame original image and the face key points in the current frame original image is less than or equal to a preset displacement threshold, the second transformation matrix transforming the face key points in the current frame original image to the reference point based on the position of the face key points in the current frame original image and the position of the reference point.

[0047] In a fifth possible implementation of the second aspect, the device further includes:

[0048] The re-detection unit is used to re-detect the facial key points in the current frame original image when the confidence level output by the CNN regression model when detecting the face image in the current frame is less than the preset confidence threshold, or when the displacement between the facial key points in the previous frame original image and the facial key points in the current frame original image is greater than the preset displacement threshold.

[0049] As can be seen from the above scheme, the embodiments of this application only require a highly complex and computationally intensive target detection model when detecting facial key points in the initial frame original image. The detection of facial key points in subsequent frames original images only requires simple detection and mathematical calculation based on a lightweight CNN regression model and transformation matrix. Therefore, for mobile devices with limited computing power, the method provided by this application can meet the real-time requirements of face tracking and ensure accuracy under limited computing resources.

[0050] Thirdly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the method as described in any possible implementation of the first aspect.

[0051] Fourthly, embodiments of this application provide an electronic device, which includes:

[0052] One or more processors;

[0053] The processor is coupled to a storage device for storing one or more programs;

[0054] When one or more programs are executed by one or more processors, the electronic device performs the method as described in any possible implementation of the first aspect.

[0055] Fifthly, embodiments of this application provide a computer program product containing instructions that, when executed on a computer or processor, cause the computer or processor to perform the method described in any possible implementation of the first aspect. Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0057] Figure 1 is a flowchart illustrating a mobile face tracking method provided in an embodiment of this application;

[0058] Figure 2 is an example diagram of mobile face tracking provided in an embodiment of this application;

[0059] Figure 3 is a block diagram of a mobile face tracking device provided in an embodiment of this application;

[0060] Figure 4 is a schematic diagram of the structure of an electronic device or computer device provided in an embodiment of this application. Detailed Implementation

[0061] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0062] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The terms "comprising" and "having," and any variations thereof, in the embodiments and drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0063] Figure 1 is a flowchart illustrating a mobile face tracking method. This method can be applied to electronic devices or computer devices, specifically to applications on mobile terminals (i.e., mobile devices), such as applications in vehicles or mobile phones. The method may include the following steps:

[0064] S110: Obtain the first transformation matrix, where the first transformation matrix is ​​the transformation matrix of the facial key points detected from the previous frame original image to the reference points.

[0065] The reference points are the facial landmarks on the standard face image, including the center points of the two eyes, the center point of the nose tip, and the two corners of the mouth. The standard face image may include only the reference points and not the standard face model, or it may include both the reference points and the standard face model.

[0066] In practical applications, the original standard face image is usually the smallest bounding rectangle of the standard face model, for example, 112*112. To prevent the phenomenon of missing areas in subsequent face detection due to the reference standard face image being too small, the original standard face image can be expanded. The specific expansion size can be determined based on practical experience, for example, it can be expanded from 112*112 to 128*128.

[0067] Each frame of raw image captured by the camera includes at least one human face, and may also include other information such as background, animals, vehicles, etc.

[0068] The previous original image may be the initial frame original image or a non-initial frame original image. When the previous original image is the initial frame original image, the first transformation matrix is ​​determined based on the initial frame face image detected by the object detection model from the initial frame original image, the CNN (Convolutional Neural Network) regression model, and the position of the reference point.

[0069] Among these, the object detection model is a highly complex and computationally intensive model in related technologies. It can detect all objects included in each frame of the original image and track the desired objects. The method for detecting the initial frame face image from the initial frame original image based on the object detection model includes: performing face detection on the initial frame original image based on the object detection model to obtain the face bounding boxes of the initial frame original image; and cropping the face bounding boxes from the initial frame original image to obtain the initial frame face image.

[0070] When the standard face image is expanded, to prevent the problem of missing areas in face cropping, before cropping the face bounding box from the original initial frame image to obtain the initial frame face image, the face bounding box of the original initial frame image is expanded outward to obtain an expanded face bounding box, where the expanded face bounding box is a sub-image of the original initial frame image; then, the expanded face bounding box is cropped to obtain the initial frame face image. The size of the expanded face bounding box is the same as the size of the expanded standard face image.

[0071] Furthermore, in this embodiment, after detecting the face bounding box of the original image of the initial frame based on the target detection model, the face bounding box of the original image of the initial frame is first expanded outward, and then the expanded face bounding box is cropped as the face image of the initial frame. This makes the face region of the initial frame face image partially redundant, which can prevent the situation where the face region is missing when the subsequent frames are aligned according to the transformation matrix, thus ensuring the integrity of the face region and improving the accuracy of face key point detection in subsequent frames.

[0072] After obtaining the initial frame face image, the method for determining the first transformation matrix based on the initial frame face image, the CNN regression model, and the position of the reference point includes: detecting the initial frame face image based on the CNN regression model to obtain the positions of facial key points and the positions of facial bounding boxes in the initial frame face image; determining the positions of facial key points in the original initial frame image based on the positions of facial key points and the positions of facial bounding boxes in the initial frame face image; and determining the first transformation matrix based on the positions of facial key points and the positions of the reference point in the original initial frame image.

[0073] The CNN regression model is a lightweight regression model. In this embodiment, the CNN network can be pre-trained based on an aligned image sample set to obtain a CNN regression model capable of detecting facial key points and facial bounding boxes. The aligned image samples in the aligned image sample set are face sample images aligned to reference points, and the size of the aligned image samples is the same as the size of the expanded standard face image.

[0074] When the previous frame's original image is not the initial frame's original image, the calculation method for the first transformation matrix is ​​the same in principle as the calculation method for the second transformation matrix. For details, please refer to the calculation method for the second transformation matrix.

[0075] S120: Align the original image of the current frame with the first transformation matrix to obtain the face image of the current frame.

[0076] Aligning the original image of the current frame based on the first transformation matrix actually involves two steps: first, multiplying the first transformation matrix by the original image of the current frame so that the face pose in the original image of the current frame is aligned with the face pose represented by the reference point in the standard face image, and obtaining the face bounding box region of the original image of the current frame. Then, cropping the face bounding box region to obtain the face image of the current frame that is the same size as the standard face image.

[0077] As shown in Figure 2, when detecting the face image of the initial frame (i.e., the first frame) based on the original image of the initial frame, no transformation matrix is ​​involved, and there is no need to align the original image of the initial frame to the reference point. The face in the figure is still tilted. When detecting the face image of the subsequent frame (e.g., the second frame) based on the original image of the subsequent frame, since the object detection model is no longer used for detection, it is necessary to rely on the transformation matrix of the previous frame to align the original image of the initial frame to the reference point. The face in the figure is corrected and is no longer tilted.

[0078] S130: Detect the face image in the current frame based on the convolutional neural network (CNN) regression model, and obtain the positions of the facial key points and the positions of the face bounding boxes in the current frame face image.

[0079] After aligning the original image of the current frame and extracting the face image of the current frame, the face image of the current frame can be input into a lightweight CNN regression model to detect the location of facial key points and the location of the face bounding box in the face image of the current frame.

[0080] S140: Determine the position of the facial key points in the original image of the current frame based on the position of the facial key points in the current frame face image, the position of the facial rectangle in the current frame face image, and the first transformation matrix after transpose.

[0081] Since the current frame face image is a cropped image extracted from the original image after aligning it with the first transformation matrix, the current frame face image needs to be rotated to the pose of the face image in the original image to accurately detect the face key point positions in the original image. Therefore, in this embodiment, the first transformation matrix is ​​transposed to obtain the transposed first transformation matrix. Then, based on the transposed first transformation matrix, the face key point positions in the current frame face image, and the face bounding box positions in the current frame face image, the face key point positions in the original image are calculated. Specifically, since the face key point positions and face bounding box positions are marked in the current frame face image, multiplying the transposed first transformation matrix by the current frame face image yields the transposed face key point positions and face bounding box positions. Finally, the face key point positions in the original image can be determined based on the transposed face key point positions and face bounding box positions.

[0082] S150: Based on the positions of facial key points and reference points in the current frame original image, determine the second transformation matrix from the facial key points in the current frame original image to the reference points, so as to detect the positions of facial key points in the next frame original image based on the second transformation matrix, the next frame original image and the CNN regression model.

[0083] After determining the location of facial key points in the current frame's original image, if face tracking is to continue, a second transformation matrix can be determined based on the location of the facial key points in the current frame's original image and the location of the reference point. This matrix transforms the facial key points in the current frame's original image into the reference point. Then, based on the second transformation matrix, the next frame's original image, and the CNN regression model, the location of facial key points in the next frame's original image can be detected.

[0084] The method for detecting the location of facial key points in the next frame original image based on the second transformation matrix, the next frame original image, and the CNN regression model is the same as the method in steps S110-S140, and will not be repeated here.

[0085] The mobile face tracking method provided in this application only requires a highly complex and computationally intensive target detection model when detecting facial key points in the initial frame of the original image. In subsequent frames of the original image, facial key point detection only requires simple detection and mathematical calculation based on a lightweight CNN regression model and transformation matrix. Therefore, for mobile devices with limited computing power, the method provided in this application can meet the real-time requirements of face tracking while ensuring accuracy under limited computing resources.

[0086] In one implementation, after inputting the current frame face image into the CNN regression model, it outputs not only the locations of facial key points and face bounding boxes in the current frame face image, but also the confidence level of the facial key point locations. If the confidence level is too low, it will lead to a decrease in the number of facial key points in the current frame face image, thereby reducing the detection accuracy of facial key points in the current frame original image and the facial key points in the next frame original image determined based on the second transformation matrix. Furthermore, since the changes in a face are continuous, the differences between two adjacent frame face images are relatively small. Therefore, if the displacement between the facial key points in the previous frame original image and the facial key points in the current frame original image is too large, it also indicates that the detection accuracy of facial key points in the current frame face image is low.

[0087] To ensure the accuracy of facial landmark detection in the current frame and subsequent frames, this embodiment of the application determines the second transformation matrix from the facial landmarks in the current frame to the reference point only when the confidence level output by the CNN regression model for detecting the facial landmarks in the current frame is greater than or equal to a preset confidence threshold, and the displacement between the facial landmarks in the previous frame and the facial landmarks in the current frame is less than or equal to a preset displacement threshold.

[0088] The preset reliability threshold and preset displacement threshold can be determined based on practical experience. For example, the preset reliability threshold can be 0.5.

[0089] In one implementation, if the confidence level output by the CNN regression model when detecting the face image in the current frame is less than a preset confidence threshold, or if the displacement between the face key points in the previous frame and the face key points in the current frame is greater than a preset displacement threshold, the current frame is used as the new initial frame and the face key points in the current frame are re-detected.

[0090] The method for re-detecting facial keypoints in the current frame original image, using it as the new initial frame original image, is the same as the method for determining the first transformation matrix when using the previous frame original image as the initial frame original image. Specifically, after using the current frame original image as the new initial frame original image, the current frame face image can be detected first based on the object detection model, then the facial keypoint positions and face bounding box positions in the current frame face image can be detected based on the CNN regression model, and finally, the facial keypoint positions in the current frame original image can be determined based on the facial keypoint positions and face bounding box positions in the current frame face image.

[0091] Since the face image in the current frame is detected by the object detection model after the original image of the current frame is used as the new original image, the position of the facial key points in the current original image is only based on the position of the facial key points and the position of the face rectangle in the current original image, and no longer depends on the first transformation matrix corresponding to the previous original image. Therefore, using the original image of the current frame as the new original image and re-detecting the facial key points in the current original image will greatly improve the accuracy of facial key point detection.

[0092] Corresponding to the above method embodiments, another embodiment of this application provides a mobile face tracking device, as shown in FIG3, the device comprising:

[0093] The acquisition unit 210 is used to acquire a first transformation matrix, wherein the first transformation matrix is ​​a transformation matrix that transforms the facial key points detected from the previous frame original image to reference points, and the reference points are facial key points on the standard face image;

[0094] Alignment unit 220 is used to align the original image of the current frame based on the first transformation matrix to obtain the face image of the current frame;

[0095] The detection unit 230 is used to detect the face image of the current frame based on the convolutional neural network (CNN) regression model, and obtain the positions of the facial key points and the positions of the face rectangles in the face image of the current frame.

[0096] The first determining unit 240 is used to determine the position of the facial key points in the original image of the current frame based on the position of the facial key points in the current frame face image, the position of the face rectangle in the current frame face image, and the transposed first transformation matrix.

[0097] The second determining unit 250 is used to determine a second transformation matrix from the facial key points in the current frame original image to the reference point based on the facial key point positions in the current frame original image and the reference point positions, so as to detect the facial key point positions in the next frame original image based on the second transformation matrix, the next frame original image and the CNN regression model.

[0098] Wherein, when the previous frame original image is the initial frame original image, the first transformation matrix is ​​determined based on the initial frame face image detected by the target detection model from the initial frame original image, the CNN regression model, and the position of the reference point.

[0099] In one possible implementation, the acquisition unit 210 includes:

[0100] The first detection module is used to detect the face image of the initial frame based on the CNN regression model when the previous frame original image is the initial frame original image, and to obtain the position of the face key points and the position of the face rectangle in the initial frame face image.

[0101] The first determining module is used to determine the position of the facial key points in the original image of the initial frame based on the position of the facial key points in the initial frame face image and the position of the face rectangle in the initial frame face image;

[0102] The second determining module is used to determine the first transformation matrix based on the positions of facial key points in the original image of the initial frame and the positions of the reference points.

[0103] In one possible implementation, the acquisition unit 210 further includes:

[0104] The second detection module is used to perform face detection on the original image of the initial frame based on the target detection model to obtain the face bounding box of the original image of the initial frame; and to extract the face bounding box from the original image of the initial frame to obtain the face image of the initial frame.

[0105] In one possible implementation, the second detection module is configured to expand the face bounding box of the original initial frame image outward before obtaining the face image of the original initial frame by cropping the face bounding box in the original initial frame image, thereby obtaining an expanded face bounding box, wherein the expanded face bounding box is a sub-image of the original initial frame image; and crop the expanded face bounding box to obtain the face image of the original initial frame.

[0106] In one possible implementation, the second determining unit 250 is configured to, when the confidence level output by the CNN regression model when detecting the face image in the current frame is greater than or equal to a preset confidence threshold, and the displacement between the face key points in the previous frame original image and the face key points in the current frame original image is less than or equal to a preset displacement threshold, determine the second transformation matrix from the face key points in the current frame original image to the reference point based on the position of the face key points in the current frame original image and the position of the reference point.

[0107] In one possible implementation, the device further includes:

[0108] The re-detection unit is used to re-detect the facial key points in the current frame original image when the confidence level output by the CNN regression model when detecting the face image in the current frame is less than the preset confidence threshold, or when the displacement between the facial key points in the previous frame original image and the facial key points in the current frame original image is greater than the preset displacement threshold.

[0109] The mobile face tracking device provided in this application only requires a highly complex and computationally intensive target detection model when detecting facial key points in the initial frame of the original image. In subsequent frames of the original image, facial key point detection only requires simple detection and mathematical calculation based on a lightweight CNN regression model and transformation matrix. Therefore, for mobile devices with limited computing power, the method provided in this application can meet the real-time requirements of face tracking while ensuring accuracy under limited computing resources.

[0110] Based on the above method embodiments, another embodiment of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the above embodiments.

[0111] Based on the above method embodiments, another embodiment of this application provides an electronic device or computer device, as shown in FIG4, including:

[0112] One or more processors 310;

[0113] The processor 310 is coupled to a storage device 320, the storage device 320 being used to store one or more programs;

[0114] When the one or more programs are executed by the one or more processors 310, the electronic device or computer device performs the method as described in any of the above embodiments.

[0115] Based on the above method embodiments, another embodiment of this application provides a mobile terminal, which includes a mobile application, a camera, a processor, and a storage device;

[0116] A camera used to capture raw images containing human faces;

[0117] Storage device for storing mobile applications and original images;

[0118] The processor is used to execute programs in mobile applications to implement the methods described in any of the above embodiments.

[0119] Based on the above embodiments, another embodiment of this application provides a computer program product, which includes instructions that, when executed on a computer or processor, cause the computer or processor to perform the method described in any of the above embodiments.

[0120] The above-described apparatus embodiments correspond to the method embodiments and have the same technical effects. For detailed descriptions, please refer to the method embodiments. The apparatus embodiments are derived from the method embodiments; detailed descriptions can be found in the method embodiments section, and will not be repeated here. Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application.

[0121] Those skilled in the art will understand that the modules in the apparatus of the embodiments can be distributed in the apparatus of the embodiments as described in the embodiments, or they can be located in one or more devices different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0122] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A mobile face tracking method, characterized in that, The method includes: Obtain a first transformation matrix, wherein the first transformation matrix is ​​a transformation matrix that transforms the facial key points detected from the previous frame original image to reference points, and the reference points are facial key points on a standard face image; Align the original image of the current frame with the first transformation matrix to obtain the face image of the current frame; The face image in the current frame is detected based on the convolutional neural network (CNN) regression model to obtain the positions of facial key points and the positions of facial bounding boxes in the current frame face image. The positions of facial key points in the original image of the current frame are determined based on the positions of facial key points in the current frame face image, the positions of facial rectangles in the current frame face image, and the transposed first transformation matrix. Based on the positions of facial key points in the current frame original image and the position of the reference point, a second transformation matrix is ​​determined to transform the facial key points in the current frame original image to the reference point, so as to detect the positions of facial key points in the next frame original image based on the second transformation matrix, the next frame original image and the CNN regression model; Wherein, when the previous frame original image is the initial frame original image, the first transformation matrix is ​​determined based on the initial frame face image detected by the target detection model from the initial frame original image, the CNN regression model, and the position of the reference point.

2. The method according to claim 1, characterized in that, The first transformation matrix is ​​determined based on the initial frame face image detected from the original initial frame image by the object detection model, the CNN regression model, and the position of the reference point, including: The CNN regression model is used to detect the face image in the initial frame to obtain the positions of facial key points and the positions of face bounding boxes in the initial frame face image. The positions of facial key points in the original image of the initial frame are determined based on the positions of facial key points in the initial frame face image and the positions of facial rectangles in the initial frame face image. The first transformation matrix is ​​determined based on the positions of facial key points in the original image of the initial frame and the positions of the reference points.

3. The method according to claim 1, characterized in that, The initial frame face image is detected from the original initial frame image by the object detection model, including: Based on the target detection model, face detection is performed on the original image of the initial frame to obtain the face bounding box of the original image of the initial frame. The face bounding box is extracted from the original image of the initial frame to obtain the face image of the initial frame.

4. The method according to claim 3, characterized in that, Before obtaining the face image of the initial frame by cropping the face bounding box from the original image of the initial frame, the method further includes: The face rectangle of the original image of the initial frame is expanded outward to obtain an expanded face rectangle, wherein the expanded face rectangle is a sub-image of the original image of the initial frame; The step of extracting the face bounding box from the original image of the initial frame to obtain the face image of the initial frame includes: The expanded face rectangle is cropped to obtain the initial frame face image.

5. The method according to any one of claims 1-4, characterized in that, Based on the positions of facial key points in the current frame's original image and the position of the reference point, a second transformation matrix is ​​determined to transform the facial key points from the current frame's original image to the reference point, including: When the confidence level of the face image in the current frame is greater than or equal to a preset confidence threshold when the face key points in the previous frame original image are detected based on the CNN regression model, and the displacement between the face key points in the previous frame original image and the face key points in the current frame original image is less than or equal to a preset displacement threshold, the second transformation matrix from the face key points in the current frame original image to the reference point is determined according to the position of the face key points in the current frame original image and the position of the reference point.

6. The method according to claim 5, characterized in that, The method further includes: If the confidence level output by the CNN regression model when detecting the face image in the current frame is less than the preset confidence threshold, or if the displacement between the face key points in the previous frame and the face key points in the current frame is greater than the preset displacement threshold, the current frame is used as the new initial frame and the face key points in the current frame are detected again.

7. A mobile face tracking device, characterized in that, The device includes: An acquisition unit is used to acquire a first transformation matrix, wherein the first transformation matrix is ​​a transformation matrix that transforms facial key points detected from the previous frame's original image to reference points, and the reference points are facial key points on a standard face image; An alignment unit is used to align the original image of the current frame based on the first transformation matrix to obtain the face image of the current frame; The detection unit is used to detect the face image of the current frame based on the convolutional neural network (CNN) regression model, and obtain the positions of the facial key points and the positions of the face bounding boxes in the face image of the current frame. The first determining unit is used to determine the position of the facial key points in the original image of the current frame based on the position of the facial key points in the current frame face image, the position of the face rectangle in the current frame face image, and the transposed first transformation matrix. The second determining unit is used to determine a second transformation matrix from the facial key points in the current frame original image to the reference point based on the facial key point positions in the current frame original image and the reference point positions, so as to detect the facial key point positions in the next frame original image based on the second transformation matrix, the next frame original image and the CNN regression model. Wherein, when the previous frame original image is the initial frame original image, the first transformation matrix is ​​determined based on the initial frame face image detected by the target detection model from the initial frame original image, the CNN regression model, and the position of the reference point.

8. The apparatus according to claim 7, characterized in that, The acquisition unit includes: The first detection module is used to detect the face image of the initial frame based on the CNN regression model when the previous frame original image is the initial frame original image, and to obtain the position of the face key points and the position of the face rectangle in the initial frame face image. The first determining module is used to determine the position of the facial key points in the original image of the initial frame based on the position of the facial key points in the initial frame face image and the position of the face rectangle in the initial frame face image; The second determining module is used to determine the first transformation matrix based on the positions of facial key points in the original image of the initial frame and the positions of the reference points.

9. The apparatus according to claim 8, characterized in that, The acquisition unit further includes: The second detection module is used to perform face detection on the original image of the initial frame based on the target detection model to obtain the face bounding box of the original image of the initial frame; and to extract the face bounding box from the original image of the initial frame to obtain the face image of the initial frame.

10. The apparatus according to claim 9, characterized in that, The second detection module is used to expand the face rectangle in the original image of the initial frame outward before obtaining the face image of the initial frame by cropping the face rectangle in the original image of the initial frame, to obtain an expanded face rectangle, wherein the expanded face rectangle is a sub-image of the original image of the initial frame; and to crop the expanded face rectangle to obtain the face image of the initial frame.

11. The apparatus according to any one of claims 7-10, characterized in that, The second determining unit is configured to, when the confidence level output by the CNN regression model when detecting the face image in the current frame is greater than or equal to a preset confidence threshold, and the displacement between the face key points in the previous frame original image and the face key points in the current frame original image is less than or equal to a preset displacement threshold, determine the second transformation matrix from the face key points in the current frame original image to the reference point based on the position of the face key points in the current frame original image and the position of the reference point.

12. The apparatus according to claim 11, characterized in that, The device further includes: The re-detection unit is used to re-detect the facial key points in the current frame original image when the confidence level output by the CNN regression model when detecting the face image in the current frame is less than the preset confidence threshold, or when the displacement between the facial key points in the previous frame original image and the facial key points in the current frame original image is greater than the preset displacement threshold.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

14. An electronic device, characterized in that, The electronic device includes: One or more processors; The processor is coupled to a storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the electronic device performs the method as described in any one of claims 1-6.

Images