A face key point detection method capable of being trained by using a label-free video

Abstract

The application belongs to the technical field of computer vision, and discloses a face key point detection method capable of being trained by using unlabeled videos; bidirectional consistency of a video is learned by using RNN, and the RNN comprises a preceding RNN training module and a subsequent RNN training module; an initial frame of the video is learned by using a pre-trained face key point detector for detection or using calibrated face key points; the preceding RNN training uses a fusion feature of a current frame face image of the video and a key point of a previous frame as input to obtain a current frame key point position; the subsequent RNN training uses a fusion feature map of the current frame key point of the video and a next frame face image as input to obtain a next frame face key point position; an initial key point of the face key point is tracked for n times in the preceding way, and then is tracked for r times in the subsequent way in a corresponding reverse manner to obtain a tracking key point.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and more specifically, to a method for detecting facial landmarks that can be trained using unlabeled videos. Background Technology

[0002] With the rapid development of artificial intelligence-related technologies, computer vision is widely used in various aspects of industry and daily life. Face-related tasks are among the most widely applied technologies in the field of computer vision, with a wealth of application scenarios. Facial landmark detection is a prerequisite for face recognition, micro-expression recognition, virtual host expression cloning, video face beautification, and other face-related tasks. The accuracy of facial landmark detection directly affects the quality of the entire face-related task. Taking the face beautification function in short videos as an example, it is necessary to first locate hundreds or even thousands of key points in various parts of the face, and then perform makeup and deformation operations based on the location of the key points, which requires very high accuracy in key point detection. Therefore, facial landmark detection has a wide range of applications, and how to accurately detect facial landmarks directly affects the implementation of face-related technologies.

[0003] The existing technology is as follows: For example, CN112801043A discloses a real-time video face keypoint detection method based on deep learning. This method uses a convolutional neural network for single-frame keypoint detection. It employs depthwise separable convolution to improve the model's detection rate, uses boundary heatmaps as an additional subtask to enhance the constraint of the original network on the global face structure, thereby improving the detection accuracy, uses a loss function to address data imbalance in the heatmap, improving the model's generalization ability to large-pose samples with limited samples, and uses an optical flow loss function to enhance the smoothness between frames. During the detection process, for frames with extremely large angles that cause the confidence level to fall below the keypoint confidence threshold, 3DMM is used for fitting to obtain dense keypoint coordinates. The obtained dense keypoints are sampled at 68 points to minimize the projection error between frames, maintaining consistency with previous frames. This method has advantages such as real-time performance, utilization of global inter-frame information, and high accuracy in detecting large-pose faces.

[0004] For example, the authorization announcement number CN112597973A discloses a training method and a method for detecting facial landmarks, including the following steps: S1, each stage of the processing is divided into the following steps, which will perform key feature point location estimation and preparation of input data for the next stage of the fully connected network; S2, starting from the first stage, after n rounds of calculation, each round further fine-tunes the previous round to obtain the latest key feature point position, and introduces heatmaps and feature maps through visualization. We maintain the transmission of key information in each stage. The structure of this invention is scientific and reasonable, safe and convenient to use, and the processing process inherits the key information of the entire continuous frame of the video. Compared with the method of local area alignment processing, it can adapt to different head poses and initial images, and provide a basic guarantee for other efficient and high-quality business face processing.

[0005] The existing technology still has the following problems: 1. Facial landmark detection methods can be divided into traditional methods and deep learning methods. Traditional methods are fast and simple to implement, but their detection accuracy is often lower than that of deep learning methods. Therefore, deep learning methods have become the mainstream facial landmark detection technology today. However, deep learning methods require a large amount of labeled data for training. With the daily explosion of massive amounts of data, the manual cost of labeling for supervised training methods has become a major bottleneck in improving training accuracy. The lack of labeling results in a large amount of relevant data being unusable, causing a waste of massive amounts of data that cannot be used for facial landmark detection training. There is currently no publicly available method for training using unlabeled video clips. In view of this, the present invention proposes a facial landmark detection method that can be trained using unlabeled videos to solve the above problems. Summary of the Invention

[0006] To overcome the aforementioned deficiencies of the prior art and to achieve the above objectives, the present invention provides the following technical solution: a facial landmark detection method that can be trained using unlabeled videos, characterized in that it utilizes an RNN to learn bidirectional consistency of the video, comprising: RNNs include a pre-order RNN training module and a post-order RNN training module; The RNN learns by detecting facial landmarks in the initial frame of the video using a pre-trained facial landmark detector or by using calibrated facial landmarks. The preceding RNN training uses the RNN to learn the fusion features of the face image in the current frame of the video and the key points in the previous frame as input, and trains the temporal module to obtain the position of the key points in the current frame; The subsequent RNN training uses the RNN to learn the fusion features of the key points in the current frame of the video and the face image in the next frame as input, and trains the temporal module to obtain the position of the face key points in the next frame; The initial key points of the face are obtained by n preceding tracking steps, and then r following tracking steps are performed accordingly. The learning of the entire network is supervised by minimizing the error between the initial key points and the tracking key points, where r and n are both integers greater than 1.

[0007] Furthermore, the preceding RNN training module is responsible for processing the data during the forward playback of the RNN learning video. The training process includes the following methods: Step S101, Initialization: Set initial values ​​for the hidden states of the preceding RNN; Step S102, Frame-by-frame input: Input the video frames learned by the RNN into the preceding RNN one by one. Each input frame will trigger an iteration of the preceding RNN. Step S103, Forward Propagation: In the preorder RNN, the label or category of the current frame is passed to the next time step through forward propagation. This process continues until the RNN learning video is finished playing. Step S104, Backpropagation: At each time step, calculate the error between the current prediction result and the true label according to the loss function, and backpropagate the error to update the weights and biases of the preceding RNN; Step S105, Optimization: Use the gradient descent optimization algorithm to minimize the loss function to obtain the preceding RNN model.

[0008] Furthermore, the subsequent RNN training module processes the data from the reverse playback of the RNN learning video. The training process includes the following methods: Step S201, Initialization: Set initial values ​​for the hidden states of the subsequent RNN; Step S202, Frame-by-frame input: Input the video frames learned by the RNN into the subsequent RNN one by one. Each input frame will trigger an iteration of the subsequent RNN. The input sequence of the subsequent RNN is the reverse of the order when the RNN learns the video in the forward playback. Step S203, Backpropagation: In the post-order RNN, the label or category of the current frame is passed to the previous time step through backpropagation. This process continues until the video is finished playing. Step S204, Backpropagation: At each time step, calculate the error between the current prediction result and the true label according to the loss function, and backpropagate the error to update the weights and biases of the subsequent RNN; Step S205, Optimization: Use the gradient descent optimization algorithm to minimize the loss function to obtain the subsequent RNN model.

[0009] Furthermore, the fusion features of face images include spatial appearance features, geometric appearance features, and supplementary features.

[0010] Furthermore, methods for obtaining spatial appearance features include: The face is divided into regions according to the various organs of the face. Each region is used as a set of input features. All regions are then fed into a feature extraction convolution to extract spatial appearance features. The feature extraction uses a convolutional feature extraction layer with p parameters, where p values ​​range from 3 to 20.

[0011] Furthermore, methods for obtaining geometric appearance features include: The distance changes between each region of each organ part of the face and the key points within that region are normalized to the region center and then input into the feature extraction convolution. The extracted geometric feature original image is then used as the geometric appearance feature.

[0012] Furthermore, the original image with extracted geometric features is used as input feature extraction convolution as supplementary features; the spatial appearance features, geometric appearance features and supplementary features are fused through the spatial attention module to obtain the fused feature map of the face image.

[0013] Furthermore, the spatial attention module is a spatial transformer network (STN). The STN structure includes each STN module consisting of a Localisation net, a grid generator, and a sample. The Localisation net determines the parameters of the input transformation. Grid generator through The defined transformation method is used to find the mapping between output and input features. Sample selects the fused feature map of the input face image by combining position mapping and transformation parameters, and outputs it by combining bilinear interpolation. STN is an abbreviation for Spatial Transformer Network.

[0014] Furthermore, the Localisation net transforms the input features through convolution and fully connected operations, followed by a regression layer to regress the parameters. The Grid generator utilizes the output of the Localization net layer. After performing the corresponding spatial transformation on the feature map, the coordinates of each pixel position in the input feature map are: After STN processing, the coordinates of each pixel position are output as follows: .

[0015] Furthermore, the mapping relationship between the input and output feature maps is as follows: .

[0016] The technical effects and advantages of this invention are as follows: 1. By decomposing facial key point features using prior knowledge, the concept of fusion features for facial images is proposed. These features consist of three parts: spatial appearance features, geometric appearance features, and supplementary features. Spatial appearance features characterize the coordinate changes caused by facial rotation angle and scaling; geometric appearance features characterize the changes in key points within organ blocks caused by facial expression changes; and supplementary features are used to supplement learning. 2. To reduce feature redundancy, key features are extracted using a spatial attention mechanism, which is beneficial for detector learning. The method proposed in this invention can directly train existing detectors using unlabeled videos, thereby improving detection accuracy.

[0017] 3. A method for training using unlabeled videos is proposed. Two RNN temporal modules are used to learn the pre- and post-pre ... Attached Figure Description

[0018] Figure 1 This is a schematic diagram of a face landmark detection method that can be trained using unlabeled video according to the present invention; Figure 2 This is a schematic diagram illustrating the implementation process of a face landmark detection method that can be trained using unlabeled video according to the present invention. Figure 3 This is a schematic diagram of the pre-processor RNN training module of the present invention; Figure 4 This is a schematic diagram of the subsequent RNN training module of the present invention; Figure 5 This is a schematic diagram of the spatial attention module of the present invention; Figure 6 This is a schematic diagram of the electronic device of the present invention; Figure 7 This is a schematic diagram of the storage medium of the present invention; Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Example 1

[0021] Please see Figure 1As shown in the figure, the facial landmark detection method that can be trained using unlabeled video described in this embodiment is implemented by using RNN to learn bidirectional consistency of the video. The RNN includes two parts: a pre-trained RNN training module and a post-trained RNN training module. The RNN learns the initial frame of the video by using a pre-trained facial landmark detector or using calibrated facial landmarks. The pre-trained RNN training uses the fusion features of the current frame's facial image and the landmarks of the previous frame as input to train the timing module to obtain the landmark position of the current frame. The post-trained RNN training uses the fusion features of the current frame's landmarks and the next frame's facial image as input to train the timing module to obtain the landmark position of the next frame. The initial landmarks are obtained by n pre-training and r post-training respectively. The learning of the entire network is supervised by minimizing the error between the initial landmarks and the tracked landmarks. r and n are both integers greater than 1.

[0022] Please see Figure 2 As shown, the temporal module in the method is a shared parameter RNN module, including a pre-order RNN training module and a post-order RNN training module, which are used to learn the features of the changes in continuous video frames from the pre-order frame to the post-order frame. Because it is from the past to the future in time, it is called the pre-order module. Similarly, the change features from the pre-order frame to the post-order frame are called the post-order module. The timing module is based on simple assumptions: The original keypoint coordinates are used to obtain the coordinates of the next frame through tracking. The coordinates obtained by reverse tracking using the coordinates of the next frame should be the same as the original coordinates. This principle is referred to as the video bidirectional consistency feature in this patent. The keypoint coordinates of the initial frame are used for pre-training to obtain the keypoint coordinates of the next frame obtained through tracking. Then, the keypoint coordinates of the next frame obtained through tracking are used for post-tracking. The difference between the keypoint coordinates of the initial frame and the keypoint coordinates of the reverse tracked frame is used as the overall training function. When training the temporal module, the number of consecutive frames input needs to be set reasonably. Too few consecutive frames are not conducive to RNN learning temporal features, while too many consecutive frames will cause error accumulation and affect the training results. Therefore, it is preferable to set it to about 10 frames.

[0023] The preceding RNN training module is responsible for processing data during forward video playback. Please refer to [link / reference]. Figure 3 As shown, its working principle is similar to that of a traditional RNN. It receives an input sequence and learns patterns and rules in forward video playback through changes in the hidden states. During training, it gradually receives video frames as input and outputs the corresponding frame's label or category. The training process of a pre-RNN can be summarized in the following steps: (1) Initialization: Set initial values ​​for the hidden states of the preceding RNN, usually zero or random numbers; (2) Frame-by-frame input: The RNN learns video frames one by one and inputs them into the preceding RNN. Each input frame will trigger an iteration of the preceding RNN. (3) Forward propagation: In the preorder RNN, the label or category of the current frame is passed to the next time step through forward propagation. This process will continue until the RNN learns the video and finishes playing. (4) Backpropagation: At each time step, the error between the current prediction result and the true label is calculated according to the loss function, and the error is backpropagated to update the weights and biases of the preceding RNN; (5) Optimization: Minimize the loss function by using the gradient descent optimization algorithm to obtain the preceding RNN model.

[0024] The subsequent RNN training module is similar to the preceding RNN; please refer to [link / reference]. Figure 4 As shown, this is also a sequence used to process video data. The difference is that the post-order RNN processes data from the reverse playback of the video. Its purpose is to learn the patterns and rules in the reverse playback of the video so as to achieve consistent predictions when the video is played forward. The training process of the post-order RNN is similar to that of the pre-order RNN, including the following steps: (11) Initialization: Set initial values ​​for the hidden states of the subsequent RNN, usually zero or random numbers; (12) Frame-by-frame input: The RNN learns video frames one by one and inputs them into the subsequent RNN. Each input frame will trigger an iteration of the subsequent RNN. The input sequence of the subsequent RNN is the reverse of the order when the RNN learns the video in the forward playback. (13) Backpropagation: In a post-processing RNN, the label or category of the current frame is passed to the previous time step through backpropagation. This process continues until the RNN finishes learning the video. (14) Backpropagation: At each time step, calculate the error between the current prediction result and the true label according to the loss function, and backpropagate the error to update the weights and biases of the subsequent RNN; (15) Optimization: Minimize the loss function by using the gradient descent optimization algorithm to obtain the subsequent RNN model.

[0025] In this example, during training, several consecutive frames of face image video need to be input. The initial frame is detected using a pre-trained face keypoint detector or calibrated face keypoints. The preceding training uses the fusion features of the face image in the current frame and the keypoints of the previous frame as input. The fusion features of the face image include: spatial appearance features, geometric appearance features and supplementary features. The method for obtaining spatial appearance features includes: dividing the face into regions according to the various facial organs, using each region as a set of input features, and then inputting all regions together into a feature extraction convolution to extract spatial appearance features. The feature extraction uses a convolutional feature extraction layer with p parameters, where p ranges from 3 to 20. Because the input to this feature is each facial region, and it does not concern changes in key points within the region, it can focus on learning the coordinate changes caused by facial rotation angles and scaling.

[0026] The method for obtaining geometric appearance features includes: normalizing the distance changes between each region of each organ part of the face and the key points within that region to the region center to remove the influence of spatial appearance changes, inputting the result into the feature extraction convolution, using the extracted geometric feature original image as the geometric appearance feature, which is used to focus on learning the key point coordinate changes caused by facial expression changes, and inputting the extracted geometric feature original image into the feature extraction convolution as supplementary features; and fusing the spatial appearance features, geometric appearance features, and supplementary features through a spatial attention module to obtain a fused feature map of the face image.

[0027] The spatial attention module can learn the interrelationship matrix between feature maps, which are fused feature maps of face images. This module optimizes the spatial relationships between feature maps, improving the accuracy of subsequent keypoint prediction. The spatial attention module in this patent is based on Spatial Transformer Networks (STNs). For its main structure, please refer to [link to relevant documentation]. Figure 5 As shown; Each STN module consists of a Localization net, a Grid generator, and a Sample. The Localization net determines the parameters of the required transformation. Grid generator through The defined transformation method is used to find the mapping between output and input features. Sample combines position mapping and transformation parameters to select input features and then uses bilinear interpolation to output the results. Localisation Net processes the input feature map through a series of convolutional and fully connected operations, followed by a regression layer that regresses the parameters. The Grid generator utilizes the output of the Localization net layer. Perform a spatial transformation on the feature map, and let the coordinates of each pixel in the input feature map be: After STN processing, the coordinates of each pixel position are output as follows: Therefore, the mapping relationship between the input and output feature maps is as follows: .

[0028] This embodiment presents a face landmark detection method that can be trained using unlabeled videos. It utilizes an RNN to learn bidirectional consistency in the video. By combining a pre-trained RNN training module and a post-trained RNN training module, in the initial frame of video processing, a pre-trained face landmark detector is used to detect the location of face landmarks. The pre-training module uses the fused features of the current frame's face image and the landmarks of the previous frame as input to train the timing module and obtain the landmark locations of the current frame. The post-training module uses the fused feature map of the current frame's landmarks and the next frame's face image as input to train the timing module and obtain the landmark locations of the next frame. The initial landmarks are obtained through several pre-training iterations, followed by several post-training iterations in the opposite direction. The learning of the entire network is supervised by minimizing the error between the initial landmarks and the tracked landmarks.

[0029] This method considers the continuity of facial expression and pose changes across several consecutive frames. Therefore, it leverages the inherent features of continuous video frames, combining an RNN capable of learning temporal information with a CNN for feature extraction. This proposes a method that can be trained on unlabeled videos, demonstrating strong practical application value. Furthermore, this method proposes a way to decompose facial keypoint features using prior knowledge. Facial keypoint motion can be decomposed into two parts: rigid motion caused by scaling and rotation, and internal coordinate changes caused by expression variations. Decomposing the task along these two dimensions undoubtedly improves the detector's performance. In addition, to reduce feature redundancy, a spatial attention mechanism is used to extract key features, facilitating detector learning.

[0030] This embodiment presents a facial landmark detection method that can be trained using unlabeled video. It can be applied to platforms and systems that require facial landmark extraction, and potential future applications include: 1. Human-computer interaction scenarios such as face cloning; 2. Tasks based on facial landmark detection, such as fatigue driving and micro-expression detection; 3. Facial recognition security monitoring system; Potential value includes: a. Quickly extract key facial features; b. High accuracy in facial landmark detection.

[0031] Example 2

[0032] Please see Figure 6As shown, according to another aspect of this application, an electronic device 500 is also provided. The electronic device 500 may include one or more processors and one or more memories. The memories store computer-readable code that, when executed by the one or more processors, can perform a face landmark detection method trained using unlabeled video as described above.

[0033] The method or system according to the embodiments of this application can also be used by means of Figure 6 The architecture of the electronic device shown is used to implement this. For example... Figure 6 As shown, the electronic device 500 may include a bus 501, one or more CPUs 502, a read-only memory (ROM) 503, a random access memory (RAM) 504, a communication port 505 connected to a network, an input / output 506, a hard disk 507, etc. The storage device in the electronic device 500, such as the ROM 503 or the hard disk 507, may store a face key point detection method that can be trained using unlabeled video, as provided in this application. Furthermore, the electronic device 500 may also include a user interface 508. Of course, Figure 6 The architecture shown is merely exemplary and can be omitted as needed when implementing different devices. Figure 6 One or more components in the illustrated electronic device.

[0034] Example 3

[0035] Please see Figure 7 The diagram illustrates a computer-readable storage medium 600 according to one embodiment of this application. The computer-readable storage medium 600 stores computer-readable instructions. When executed by a processor, the computer-readable instructions can perform a face keypoint detection method based on an embodiment of this application, as described above with reference to the accompanying drawings, that can be trained using label-free video. The storage medium 600 includes, but is not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.

[0036] Furthermore, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, this application provides a non-transitory machine-readable storage medium storing machine-readable instructions that can be executed by a processor to perform instructions corresponding to the method steps provided in this application. When this computer program is executed by a central processing unit (CPU), it performs the functions defined in the method of this application. The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0037] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A facial landmark detection method that can be trained using unlabeled videos, characterized in that, The implementation utilizes RNN to learn bidirectional consistency in videos, including: RNNs include a pre-order RNN training module and a post-order RNN training module; The RNN learns by detecting facial landmarks in the initial frame of the video using a pre-trained facial landmark detector or by using calibrated facial landmarks. The preceding RNN training uses the RNN to learn the fusion features of the face image in the current frame of the video and the key points in the previous frame as input, and trains the temporal module to obtain the position of the key points in the current frame; The subsequent RNN training uses the RNN to learn the fused feature map of the key points in the current frame of the video and the face image in the next frame as input, and trains the temporal module to obtain the position of the face key points in the next frame; The initial key points of the face are obtained by n preceding tracking steps, and then r following tracking steps are performed accordingly. The learning of the entire network is supervised by minimizing the error between the initial key points and the tracking key points. r and n are both integers greater than 1. Among them, the fusion features of face images include spatial appearance features, geometric appearance features, and supplementary features; The face is divided into regions according to the various organs of the face. Each region is used as a set of input features. All regions are input together into the feature extraction convolution to extract spatial appearance features. Methods for obtaining geometric appearance features include: normalizing the distance changes between each region of each organ part of the face and the key points within the region to the region center, inputting it into the feature extraction convolution, and using the extracted geometric feature original image as the geometric appearance feature. The original image with extracted geometric features is input into the feature extraction convolution as a supplementary feature; the spatial appearance features, geometric appearance features and supplementary features are fused through the spatial attention module to obtain the fused feature map of the face image.

2. The facial landmark detection method that can be trained using unlabeled video according to claim 1, characterized in that, The preceding RNN training module is responsible for processing the data during the forward playback of the RNN learning video. The training process includes the following methods: Step S101, Initialization: Set initial values ​​for the hidden states of the preceding RNN; Step S102, Frame-by-frame input: Input the video frames learned by the RNN into the preceding RNN one by one. Each input frame will trigger an iteration of the preceding RNN. Step S103, Forward Propagation: In the preorder RNN, the label or category of the current frame is passed to the next time step through forward propagation. This process continues until the RNN learning video is finished playing. Step S104, Backpropagation: At each time step, calculate the error between the current prediction result and the true label according to the loss function, and backpropagate the error to update the weights and biases of the preceding RNN; Step S105, Optimization: Use the gradient descent optimization algorithm to minimize the loss function to obtain the preceding RNN model.

3. The facial landmark detection method that can be trained using unlabeled video according to claim 2, characterized in that, The subsequent RNN training module processes the data from the reverse playback of the RNN learning video. The training process includes the following methods: Step S201, Initialization: Set initial values ​​for the hidden states of the subsequent RNN; Step S202, Frame-by-frame input: Input the video frames learned by the RNN into the subsequent RNN one by one. Each input frame will trigger an iteration of the subsequent RNN. The input sequence of the subsequent RNN is the reverse of the order when the RNN learns the video in the forward playback. Step S203, Backpropagation: In the post-order RNN, the label or category of the current frame is passed to the previous time step through backpropagation. This process continues until the video is finished playing. Step S204, Backpropagation: At each time step, calculate the error between the current prediction result and the true label according to the loss function, and backpropagate the error to update the weights and biases of the subsequent RNN; Step S205, Optimization: Use the gradient descent optimization algorithm to minimize the loss function to obtain the subsequent RNN model.

4. The facial landmark detection method that can be trained using unlabeled video according to claim 1, characterized in that, Methods for obtaining spatial appearance features also include: The feature extraction uses a convolutional feature extraction layer with p parameters, where p ranges from 3 to 20.

5. A facial landmark detection method that can be trained using unlabeled video according to claim 1, characterized in that, The spatial attention module is a spatial transformer network (STN). The STN structure includes each STN module consisting of a Localization net, a grid generator, and a sample. The Localization net determines the parameters to be transformed. Grid generator through The defined transformation method is used to find the mapping between output and input features. Sample selects the fused feature map of the input face image by combining position mapping and transformation parameters, and outputs it by combining bilinear interpolation. STN is an abbreviation for Spatial Transformer Network.

6. A facial landmark detection method that can be trained using unlabeled video according to claim 5, characterized in that, The Localisation Net transforms the input features through convolution and fully connected operations, followed by a regression layer to regress the parameters. The Grid generator utilizes the output of the Localization net layer. After performing the corresponding spatial transformation on the feature map, the coordinates of each pixel position in the input feature map are: After STN processing, the coordinates of each pixel position are output as follows: .

7. A facial landmark detection method that can be trained using unlabeled video according to claim 6, characterized in that, The mapping relationship between the input and output feature maps is as follows: 。

Images