A face key point detector domain adaptation method based on spatial deformation

Abstract

The application provides a face key point detector domain self-adaption method based on spatial deformation, which is used for realizing generalization of face key point detection, and comprises the following steps: preparing a source domain data pair and a target domain data; setting an iteration number and an updating frequency of alternating optimization; training by using the source domain data pair to obtain a pre-trained face key point detector on the source domain; performing iteration, and in each iteration, a face transformer and a face key point detector are sequentially optimized and updated, and the face transformer is embedded with the same face key point detector; and outputting the face key point detector obtained through the last training. According to the technical scheme of the application, the generalization problem of the face key point detection model on the target domain and more unseen domains can be overcome, and the data distribution of the target domain can be better adapted.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of face detection, and particularly relates to a face key point detector domain self-adaption method based on spatial warping. BACKGROUND

[0002] Face key point detection is an important step in human face modeling, editing and generation. Face key point detection is very important in a variety of applications, including but not limited to face recognition, face stylization, three-dimensional face reconstruction, animation production, game development, virtual reality and augmented reality, etc. For example:

[0003] (1) Animation character recognition: Through face key point detection, different characters in animation can be recognized and distinguished, which is of great significance to the construction of animation database, character search and recommendation system.

[0004] (2) Animation content creation: Through face key point detection, animation producers can more accurately capture and replicate the expressions and movements of real humans, create realistic character expressions and mouth synchronization, and improve the realism of character performance.

[0005] (3) Interactive entertainment experience: In the field of games and virtual reality (VR), face labeling through face key point detection can make the virtual image of the player more vivid and personalized. The facial expressions of the player can be captured in real time and mapped to the virtual character, providing a more immersive gaming experience.

[0006] (4) Cross-object expression transfer: Face labeling through face key point detection can achieve cross-object expression redirection, i.e. transferring the expression information of an actor to different animation characters. This allows animators to create diverse character expressions and animation sequences using limited performance resources.

[0007] (5) Animation production efficiency improvement: In traditional animation production, animators need to manually create and adjust the expressions of characters in each frame, which is time-consuming and labor-intensive. Face labeling through face key point detection can automate this process, greatly improving the efficiency of animation production while maintaining the coherence and authenticity of expressions.

[0008] Spatial warping is a commonly used technique in the field of computer vision and image processing, which generates new perspectives or edits images by changing the positions of pixels in the image. This technique can maintain the content of the image while creating different visual effects or adapting to different application needs.

[0009] Domain Adaptation is an important research direction in the field of machine learning, and its core goal is to solve the challenges faced by models when migrating from one domain (source domain) to another domain (target domain). This challenge is usually manifested as the difference in data distribution between the source domain and the target domain, i.e., the problem of domain shift. In practical applications, due to environmental changes, data acquisition costs, and other factors, the data of the target domain is often difficult to obtain or label, which requires domain adaptation technology to improve the generalization ability of the model in the target domain.

[0010] Among them, "domain shift" refers to the distribution difference between training data (source domain) and data encountered in actual application (target domain) in machine learning and pattern recognition tasks. This difference may cause the model that performs well in the source domain to have decreased performance in the target domain, because the model may not be able to adapt to the data distribution in the target domain, thus failing to accurately predict or classify.

[0011] Domain shift can occur in many scenarios, such as:

[0012] (1) Environmental changes: In image recognition tasks, training data may be collected in indoor environments, while target domain data is in outdoor environments. Changes in environmental factors such as lighting and background will cause domain shift.

[0013] (2) Time changes: As time goes by, the distribution of data may change, such as changes in user behavior on social media, which may require the model to adapt to new data distribution.

[0014] (3) Differences in data sources: Different data sources may have different distribution characteristics, such as medical data from different hospitals, which may have domain shift due to differences in equipment, patient population, etc.

[0015] Existing domain adaptation methods mostly rely on alignment between feature spaces, aligning the feature spaces of source and target domains through some mapping, so that the model can still maintain good performance in the new feature space. There are roughly six kinds of methods:

[0016] Discrepancy-based methods: This method aligns by calculating the distribution difference of source and target domain data in the feature space. Common discrepancy measures include maximum mean difference, correlation pair, and KL divergence-based methods. The goal of these methods is to minimize the difference between the feature distributions of the source and target domains, so that the model can better generalize in the target domain.

[0017] Adversarial-based methods: Adversarial-based methods achieve alignment in feature space by introducing one or more discriminators. The goal of these discriminators is to distinguish between source and target domain features, while the goal of the domain adaptation network is to generate features that are indistinguishable between source and target domains in feature space. This approach typically involves adversarial training, including techniques such as Domain Adversarial Neural Network (DANN) and Multiple Adversarial Domain Adaptation (MADA).

[0018] Reconstruction-based methods: These methods achieve alignment in feature space by reconstructing source and target domain data. The reconstruction process can be in the form of autoencoders or generative adversarial networks (GANs), with the goal of making the reconstructed features consistent between source and target domains. An example of this approach is the Domain Separation Network (DSN).

[0019] Sample Generation-based methods: These methods assist training by generating samples that conform to the target domain distribution, such as using CycleGAN to generate samples that look like they came from the target domain. These generated samples can help the model better adapt to the feature distribution of the target domain.

[0020] Meta-learning methods: Meta-learning methods achieve domain adaptation by learning how to quickly adapt to new tasks. These methods typically involve learning a model that can capture commonalities between different tasks and quickly adjust itself to adapt to new domain data.

[0021] Graph-based methods: Graph-based methods achieve domain adaptation by constructing a graph representation of the data. These methods typically involve graph embedding techniques that map data points into a graph structure in a low-dimensional space, such that similar data points remain similar in the graph representation.

[0022] Most of the existing methods are proposed for classification or semantic segmentation problems, which are often not effective for point regression problems. For example, Yang and Soatto proposed a Fourier domain adaptive method (Fda) in 2020, Tsai et al. explored the learning method of structured output space adaptation in 2018, and Li et al. introduced the concept of bidirectional learning in 2019 to deal with the domain adaptation problem in semantic segmentation. At the same time, these methods and their theories are often based on specific assumptions, which will limit their adaptability in more general practical scenarios.

[0023] (1) Classification and semantic segmentation problems usually involve discrete labels or pixel-level classification tasks, and the output space of these problems is limited and explicit. Point regression problems such as face key point detection involve continuous numerical prediction, and the output space is infinite and continuous, which makes it more complex to align the feature distribution between the source domain and the target domain.

[0024] (2) Existing domain adaptation methods, such as the adversarial training-based framework, are more suitable for classification and segmentation tasks, because these methods can improve the generalization ability of the model by maximizing the consistency of the source domain and target domain features. For point regression problems such as face key point detection, the alignment of the feature space may not be sufficient to ensure the performance of the model on the target domain, and the changes in the numerical range and distribution shape of the features also need to be considered. SUMMARY

[0025] The present application aims to provide a face key point detector domain adaptation method based on spatial deformation, which can overcome the generalization problem of face key point detector in the target domain and more unseen domains, so that the face key point detector can better adapt to the data distribution of the target domain, and better adapt to the face key point detection needs in the fields of face recognition, face stylization, three-dimensional face reconstruction, animation production, game development, virtual reality and augmented reality, etc.

[0026] According to an aspect of the present application, a face key point detector domain adaptation method based on spatial deformation is provided for realizing the generalization of face key point detection, comprising:

[0027] Prepare source domain data pairs and target domain data, the source domain data pairs are a plurality of labeled real face data pairs, the target domain data are a plurality of unlabeled stylized face data, and the labeled real face data pairs include face pictures and corresponding feature point coordinates;

[0028] Set the number of iterations And the update frequency of alternating optimization ;

[0029] training the face warper using the source domain data pairs to minimize the distance error between the feature point coordinates predicted by the face landmark detector and the real feature point coordinates, to obtain a pre-trained face landmark detector on the source domain;

[0030] performing iterations, in each iteration, sequentially optimizing the update of the face warper and the face landmark detector, the face warper embedding the same face landmark detector;

[0031] training the face landmark detector using the target domain data pairs with pseudo labels and the source domain data pairs to minimize the distance error between the feature point coordinates predicted by the face landmark detector and the real feature point coordinates, to obtain the face landmark detector on the target domain. training ends, outputting the face landmark detector obtained in the last training.

[0032] According to some embodiments, the method further comprises: in each iteration, solving the optimization update problem of the face warper using epochs, and solving the optimization problem of the face landmark detector using epochs.

[0033] According to some embodiments, the optimization update problem of the face warper is solved using a thin plate spline interpolation method.

[0034] According to some embodiments, the optimization update problem of the face warper is solved using a thin plate spline interpolation method combined with minimizing the distance between the gradient field of the warped face data generated by the face warper from the source domain data and the gradient field of the target domain data.

[0035] According to some embodiments, the optimization problem of the face landmark detector is solved by training the face landmark detector using the source domain data pairs and the target domain data pairs with pseudo labels.

[0036] According to some embodiments, the pseudo labels of the target domain data are mapped from the target domain data using the face landmark detector obtained in the face warper optimization update process.

[0037] According to some embodiments, the number of iterations ranges from 50 to 60.

[0038] According to some embodiments, the update frequency of the alternating optimization .

[0039] According to another aspect of the present application, there is provided a spatial deformation based face landmark detector domain adaptation apparatus, comprising a non-volatile memory and one or more processors, the non-volatile memory storing executable code, the processors executing the executable code to implement the spatial deformation based face landmark detector domain adaptation method described above.

[0040] According to another aspect of the present application, a computer-readable storage medium is also provided, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed by a processor, the device where the storage medium is located is controlled to perform the above-mentioned spatial deformation-based facial key point detector domain adaptation method.

[0041] Compared with the prior art, this application has the following beneficial effects:

[0042] This application employs spatial deformation technology to transform the generalization challenge of facial landmark detection into an iterative optimization process for the face deformer and landmark detector. This approach achieves high-precision facial landmark detection while improving the model's adaptability and robustness across diverse scenarios and conditions. Through an iterative optimization process, this method ensures that the model maintains accurate and stable landmark detection in a variety of complex environments.

[0043] This method is a domain adaptation method for point regression problems. In view of the reality that it is difficult to obtain labeled data, it constructs high-confidence target domain data labels by utilizing labeled source domain data and unlabeled target domain data, enabling the target key point detector to perform effective self-supervised learning and reduce the distribution difference between source domain data and target domain data at the data level.

[0044] Compared with traditional domain adaptation methods for classification problems or semantic segmentation scenarios, this application can overcome the generalization problem of facial key point detection models in the target domain and more unseen domains, better adapt to the data distribution of the target domain, and thus better adapt to the facial key point detection needs in fields such as facial recognition, facial stylization, 3D facial reconstruction, animation production, game development, virtual reality and augmented reality.

[0045] It should be understood that the foregoing general description and the following detailed description are merely illustrative and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0047] Figure 1 A schematic diagram illustrating a domain adaptation method of a facial key point detector based on spatial deformation according to an example embodiment is shown.

[0048] Figure 2A flow chart illustrating a spatial transformation based face keypoint detector domain adaptation method according to an example embodiment is shown.

[0049] Figure 3 Experimental results of domain adaptation accuracy evaluation of a spatial transformation based face keypoint detector domain adaptation method according to an example embodiment are shown.

[0050] Figure 4 Experimental results of generalization ability test of a spatial transformation based face keypoint detector domain adaptation method according to an example embodiment are shown.

[0051] Figure 5 A block diagram illustrating a spatial transformation based face keypoint detector domain adaptation apparatus according to an example embodiment of the present application is shown. DETAILED DESCRIPTION

[0052] The embodiments of the present application will be described in detail below with reference to the drawings. It should be noted that the described embodiments are only some of the embodiments of the present application, but not all of the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0053] Those skilled in the art should know that the following specific embodiments or specific embodiments are a series of optimized settings listed by the present application to further explain the specific invention content, and these settings can be used together or in association with each other, unless the present application explicitly states that certain or a specific embodiment or embodiment cannot be associated with other embodiments or embodiments. At the same time, the following specific embodiments or embodiments are only the most optimized settings, and not as an understanding of the scope of protection of the present application.

[0054] The example embodiments of the present application will be described below with reference to the accompanying drawings.

[0055] Face keypoint detection problem description

[0056] Assumptions For picture space, For point space, where represents a human face picture composed of pixels, is the corresponding coordinate vector of key points, and the goal of this task is to learn a mapping to accurately identify and locate specific anatomical feature points on the face, such as the center of the eye, the corner of the eye, the tip of the nose, the corner of the mouth, the chin, etc. In mathematics, this problem can be formulated as the following optimization problem:

[0057]

[0058] in is the Frobenius-norm of the matrix, which aims to minimize the coordinate vector predicted by the model and the true label The squared distance error between Mapping for the goals we want to learn.

[0059] Domain Adaptation Problem Description

[0060] In this problem, we use labeled real faces (source domain )data , and unlabeled stylized faces (target domain )data , and aims to solve the key point detection in the source domain and target domain The generalization problem.

[0061] The domain adaptation method proposed in the present invention obtains a universal face key point detector by solving the above problems.

[0062] Domain Adaptation Problem for Facial Keypoint Detection Based on Spatial Deformation

[0063] This example proposes a training framework: Face Deformer and facial landmark detectors , where the same facial keypoint detector is embedded in the face deformer .

[0064] If we want to obtain a facial key point detector in the target domain, the most intuitive solution is to use the labeled target domain data for full supervision learning. However, in real life, data labels are difficult to obtain, so we cannot simply use formula (1) for training. However, we can use the labeled real face (source domain) data to input strong prior knowledge into the subsequent model learning. Through formula (1) and source domain data, , we get the pre-trained facial keypoint detector , and the current The distribution of facial features and facial structure have been learned from the source domain data.

[0065] Face Deformer The main purpose is to learn how to transform real face (source domain) data into cartoon face (target domain) through spatial deformation. Stylized face images Predict the corresponding key points . For each real face image The corresponding real label As the starting point of the deformation, The predicted As the end point of the deformation, the deformation operation is performed, that is, a "plane" is constructed and the "plane" is made to pass through All the points in the image. Through the thin plate spline interpolation method, the pixel on the warped image can be obtained as follows:

[0066]

[0067] where, The parameters corresponding to the deformation flow to be solved, is the inverse mapping from the pixel coordinates in the face image after deformation to the corresponding pixel positions on the real face (original image). In formula (2), is a non-parametric non-rigid deformation, is a predefined radial basis function kernel, which is defined as in this embodiment; is a linear function that captures the rigid transformation of the image.

[0068] If we know the pixel coordinates on the warped image, then the corresponding pixel coordinates on the real face can be obtained by solving the mapping ; further, we can obtain the pixel color corresponding to the original image as the pixel color on the warped image :

[0069]

[0070] For the generated warped image, on the one hand, we only care whether it can warp the real face to the position of the stylized face with high accuracy, so we use the gradient field information of the image to constrain the geometric shape of the warped face, thereby reducing the attention to other redundant color information. On the other hand, we require to be able to inversely map the predicted to the real label of the real face with high accuracy. Therefore, for a randomly sampled face image pair , we optimize the face deformer as follows:

[0071]

[0072] Among them, the first term of formula (4) aims to minimize the distortion of the face Gradient field and target domain face Gradient field The distance between them is used to make the geometric shapes of the two highly similar, further optimizing Accuracy in the target domain; in this embodiment, the Sobel operator is used to obtain the first-order gradient of the digital image based on the image space domain convolution. Minimizing the second term aims to update the parameters of the interpolation function in the face deformer , so that the deformed surface it generates fits all the data points , and as smooth as possible.

[0073] The facial key point detector mainly learns how to extract effective feature information from the input image and predict accurate key point coordinates. ,use As the starting point for this part of the optimization, and use Mapped key points As target domain data Pseudo labels of target domain data are formed , compared with the source domain data Mixed together as training data for this part, supervised learning of key point detector:

[0074]

[0075] The first term of formula (5) aims to minimize the estimation of the facial key point detector on the source domain data. and the true label on the source domain The distance error between the two, the second term ensures that the facial key point detector The result will not be far from the previous estimate is too far away, so this optimization process can be regarded as a proximal regularizer. In general, the purpose of formula (5) is to make By training on joint data from source and target domains, the semantic correspondence from faces to key points is learned.

[0076] Figure 1 A schematic diagram illustrating a domain adaptation method of a facial key point detector based on spatial deformation according to an example embodiment is shown.

[0077] The face key point detector domain adaptation method based on spatial deformation proposed in the application regards the domain adaptation problem of face key point detection as a joint optimization problem of face transformer and face key point detector. In the face transformer optimization problem, the source domain face is deformed based on the thin plate spline interpolation method (taking the real label of the source domain as the starting point and the prediction result of the target key point detector on the target domain data as the ending point) from the labeled source domain data pair and unlabeled target domain data, and the image deformation loss and the geometric difference between the image domains are taken as the objective function to optimize the module. In the face key point detector optimization problem, the labeled source domain data pair and the target domain data pair with pseudo labels (the pseudo labels are generated by the target key point detector obtained in the previous optimization) are used as the training data of the module to optimize the target model by minimizing the error between the model prediction and the real / pseudo label.

[0078] Figure 2 A flowchart of a face key point detector domain adaptation method based on spatial deformation according to an example embodiment is shown.

[0079] Referring to Figure 2 , the implementation of the scheme of the application is described based on the diagram shown in the figure, specifically a face key point detector domain adaptation method based on spatial deformation.

[0080] In S201, the required data is prepared, including a labeled real face data pair (source domain data pair), an unlabeled stylized face data (target domain data).

[0081] In S202, the iteration number and the update frequency of alternating optimization are set.

[0082] According to some embodiments, the iteration number ranges from 50 to 60.

[0083] According to some embodiments, the update frequency of alternating optimization .

[0084] In S203, the target model is initialized. Using the source domain data pair , the distance error between the coordinate vector predicted by the model and the real label is minimized by solving the optimization problem in formula (1), and a pre-trained face key point detector on the source domain is obtained, and is taken as the starting point for the following iteration.

[0085] In S204, it is determined whether the current number of iterations m is less than M. If so, the next step is executed; otherwise, the process jumps to step S207.

[0086] In S205, in In the iterations, use epochs to solve the optimization problem of the face deformer as described in formula (4), the face deformer (Embedded facial key point detector ) will be After the optimization is completed, we can get and .

[0087] According to some embodiments, a thin plate spline interpolation method is used to solve the optimization problem of the face deformer as described in formula (4), and the specific steps are as follows:

[0088] Assume that in the original shape points ,this After deformation, the points correspond to the new points , where each point is Use the deformation of a thin steel plate to simulate the 2D deformation, making sure that The points can be correctly matched to minimize the bending energy of the steel plate.

[0089] for The displacement of the two sets of points is In two directions, in fact, thin plate spline interpolation interpolates the coordinates of each dimension separately.

[0090] Solving the deformation function , so that the corresponding points match . And because we need to interpolate in two directions separately, Interpolation function in direction can be written as:

[0091]

[0092] in is a scalar, , and Defines the affine transformation between corresponding points; , function vector , is the radial basis function kernel. In the literature proposed by Kent, JT et al., it has been proved that this form of interpolation function minimizes the bending energy:

[0093]

[0094] In thin-plate spline interpolation function There are parameters, conditions Only , so you need to add Constraints:

[0095]

[0096]

[0097]

[0098] And respectively represent the point The coordinate value and Coordinate values.

[0099] So we can write the above equations:

[0100]

[0101] Where , Indicates the value of The Dimensional column vector, , , the parameters of the thin-plate spline interpolation function Can be obtained by solving the equation.

[0102] Let If Is a non-singular matrix, then Is also a non-singular matrix, and the parameters can be solved

[0103]

[0104] To apply these deformations to all other points , need and for the calculation of the control points used to calculate the deformation parameters together to construct the calculation matrix:

[0105]

[0106] Where , the result is:

[0107]

[0108] ​​​In this embodiment, the inverse mapping of the deformation process (from pixels on the warped image to pixels on the original image) is solved, that is where is the ground truth label corresponding to the real face, is any pixel point on the warped image, is the prediction result of the stylized face by the face key point detector . For any pixel position on the warped image, we can define the inverse mapping from the warped image to the original image by the thin-plate spline interpolation formula (note that the mapping here contains interpolation in both x and y directions):

[0109] where , , is the parameter of the deformation flow corresponding to the mapping, and the further expansion of the formula is referred to formula (2), and the solving method is as described above.

[0110] In S206, in the Mth iteration, the is fixed, and the optimization problem of the face key point detector as described in formula (5) is solved with epochs, and the face key point detector will be iteratively optimized in epochs, and after the optimization is completed, the is obtained, and the step S204 is returned. According to some embodiments, the face key point detector

[0111] is trained with source domain data with ground truth labels and target domain data with pseudo labels.

[0112] According to some embodiments, the face key point detector obtained in step S205 is used as the starting point of this part of optimization, and the key points mapped by the face key point detector are used as the pseudo labels of the target domain data , forming a target domain data pair , mixed with the source domain data pair , as the training data of this part, and the formula of the supervised learning of the key point detector is formula (5).

[0113] In S207, after the iteration is completed, the face key point detector obtained in the Mth iteration is output as the optimal face key point detector .

[0114] ​The training framework constructed in this embodiment includes two major modules: face deformer and facial landmark detectors , where the same facial keypoint detector is embedded in the face deformer The ultimate goal is a facial key point detector with strong generalization ability. That is to say, under this training framework, our optimization goal from beginning to end is the same , "Alternating Optimization Framework" means The optimization goal in the face deformer The optimization problem (Formula (4)) and the key point detector The optimization problem (Formula (5)) alternates between.

[0115] To demonstrate the practicality and feasibility of this method, we designed a series of experiments, including but not limited to evaluation on standard test sets and deployment in actual application scenarios.

[0116] The accuracy and generalization ability of the model were comprehensively evaluated by introducing datasets from three different domains. The selected datasets include real face images (source domain), cartoon-style faces, and artistic-style faces. The specific experimental results are as follows:

[0117] Domain adaptation accuracy evaluation experiment:

[0118] Figure 3 1 shows experimental results of domain adaptation accuracy evaluation of a domain adaptation method for a facial key point detector based on spatial deformation according to an example embodiment, where green dots represent true labels and red dots represent model prediction results.

[0119] exist Figure 3 In (a), this experiment selected real face images as the source domain and cartoon face images as the target domain. Figure 3 The experimental setup in (b) is similar to Figure 3 (a) Remains the same, but the target domain is changed to artistic face images.

[0120] In the experiment, multiple domain adaptation methods were implemented on the same facial key point detector to evaluate the accuracy of each method in the target domain. Figure 3 (a) (b), as can be seen from the enlarged detail images, compared with other domain adaptation methods, the predicted results of this invention are more dense and less prone to dispersion, and are closer to the true label (red dot), indicating that the facial key point detection model learned by this scheme has a deeper semantic understanding of the structure and distribution of facial features across domains.

[0121] Table 1 shows the Figure 3Comparison of various domain adaptation methods using the normalized mean error (NME) index under the experimental settings of (a) and (b).

[0122] where the normalized mean error (NME) calculates the L2 norm of the distance between each predicted point and the true value point divided by the distance between the inner corners of the eyes / inter-pupils , and is summed for each point, and finally divided by the total number of points. Therefore, the lower the value, the more accurate the prediction result.

[0123]

[0124] From Table 1, it can be seen that compared with other domain adaptation methods, the qualitative value of the present application is closer to the effect of the model in the all-dataset label-known (Oracle) case on three different styles of faces, indicating the effectiveness of the scheme.

[0125] Table 1 NME (normalized mean error) of various domain adaptation methods on three datasets

[0126]

[0127] where SLPT is a face key point detection method proposed by Jiahao Xia et al. in 2022, used to verify the stronger generalization ability of the present application compared to other domain adaptation methods. RevGrad is an unsupervised domain adaptation method based on gradient reversal layer proposed by Yaroslav Ganin et al. in 2015, CycleGAN learns the mapping between two domains through cycle consistency without paired samples, and is the work of Jun-Yan Zhu in 2017. Yunsheng Li further proposed BDL based on the idea of CycleGAN in 2019, combining forward and reverse transfer learning methods to utilize the complementary information between two domains to improve the performance of the model in the target domain. Yi-Hsuan Tsai proposed AdaptSegNet by learning to adjust the structure of the semantic segmentation output space to adapt to the differences between different domains, further improving the performance of the model in the target domain. Yanchao Yang proposed FDA in 2020, which aligns the feature distributions of the source and target domains in the frequency domain, thereby reducing the distribution difference between them.

[0128] Generalization ability test experiment:

[0129] Figure 4Fig. 6 shows experimental results of generalization ability test of the spatial deformation based face key point detector domain adaptation method according to an example embodiment, green points are real labels, and red points are model prediction results.

[0130] Figure 4 Further shown is the generalization ability of the selected method on an unknown domain (i.e. artistic face image dataset) under the experimental setup of (a). This test is of great significance for verifying the performance of the model on unseen data. Figure 3 As shown in (b), the face key point detector trained by the present application still has superior generalization ability on its unseen domain compared with other published methods, proving that the potential face distribution learned on the source domain and the target domain is effectively migrated to other domains.

[0131] Figure 4 The above experimental results show that the face key point detector model obtained by using the domain adaptation method proposed in the present application not only exhibits high accuracy in the source domain, but also shows good generalization ability in the target domain and the unknown domain.

[0132] The above experimental results show that the face key point detector model obtained by using the domain adaptation method proposed in the present application not only exhibits high accuracy in the source domain, but also shows good generalization ability in the target domain and the unknown domain.

[0133] In another implementation form of the present solution, the method can be implemented in the form of a device.

[0134] Figure 5 Fig. 7 shows a block diagram of a spatial deformation based face key point detector domain adaptation device according to an example embodiment of the present application, which includes a non-volatile memory and one or more processors, the non-volatile memory stores executable codes, and the processor executes the executable codes to implement the spatial deformation based face key point detector domain adaptation method in the above embodiments.

[0135] The present embodiment can be applied to any device with data processing capability, which can be a device or apparatus such as a computer. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a logical device, it is formed by reading the corresponding computer program instructions in the non-volatile memory to the memory for execution by the processor of the device with data processing capability where it is located. From the hardware perspective, as shown in Fig. 8, it is a hardware structure diagram of the device with data processing capability where the spatial deformation based face key point detector domain adaptation device of the present application is located. In addition to the processor, the memory, the network interface, and the non-volatile memory shown in Fig. 8, the device with data processing capability where the device is located in the embodiment usually includes other hardware according to the actual functions of the device with data processing capability, which will not be described here. Figure 5 Figure 5 The present embodiment can be applied to any device with data processing capability, which can be a device or apparatus such as a computer. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a logical device, it is formed by reading the corresponding computer program instructions in the non-volatile memory to the memory for execution by the processor of the device with data processing capability where it is located. From the hardware perspective, as shown in Fig. 8, it is a hardware structure diagram of the device with data processing capability where the spatial deformation based face key point detector domain adaptation device of the present application is located. In addition to the processor, the memory, the network interface, and the non-volatile memory shown in Fig. 8, the device with data processing capability where the device is located in the embodiment usually includes other hardware according to the actual functions of the device with data processing capability, which will not be described here.​​

[0136] The implementation process of the functions and roles of each unit in the above device is specifically described in the implementation process of the corresponding steps in the above method, which will not be repeated here.

[0137] For the device embodiment, since it basically corresponds to the method embodiment, the relevant part can be referred to the part of the method embodiment. The device embodiments described above are only illustrative, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place or distributed on multiple network units. Some or all of the modules can be selected to achieve the purpose of the present application according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0138] In another embodiment, the present application can be implemented by a computer readable storage medium, which stores a program. When the program is executed by a processor, the above-mentioned face key point detector domain adaptive method based on spatial deformation is realized.

[0139] The computer readable storage medium can be an internal storage unit of any data processing capable device, such as a hard disk or a memory, as described in any of the preceding embodiments. The computer readable storage medium can also be an external storage device of any data processing capable device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. Further, the computer readable storage medium can include both the internal storage unit and the external storage device of any data processing capable device. The computer readable storage medium is used to store the computer program and other programs and data required by the data processing capable device, and can also be used to temporarily store data that has been output or will be output.

[0140] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or replacements within the technical scope disclosed by the present application can be easily thought by those skilled in the art, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A spatial transformation based face keypoint detector domain adaptation method for achieving generalization to face keypoint detection, characterized in that, The method comprises: preparing source domain data pairs and target domain data, the source domain data pairs being a plurality of labeled real face data pairs, the target domain data being a plurality of unlabeled stylized face data, the labeled real face data pairs comprising face pictures and feature point coordinates corresponding to the face pictures; Setting the number of iterations and the update frequency of the alternating optimization ; training using the source domain data pairs to minimize distance errors between feature point coordinates predicted by a face key point detector and real feature point coordinates, to obtain a pre-trained face key point detector on the source domain; performing iterations, in each iteration, sequentially optimizing and updating a face warper and a face key point detector, the face warper being embedded with the same face key point detector; In each iteration, use The thin plate spline interpolation method is used to solve the optimization update problem of the face deformer in each epoch; in each iteration, the facial key point detector obtained in the face deformer optimization update process is used to map the target domain data to obtain the pseudo label of the target domain data, and the pseudo label of the target domain data is obtained by using the face key point detector obtained in the face deformer optimization update process. epochs, using the source domain data pairs and the target domain data pairs with pseudo labels to train the facial key point detector to solve an optimization problem of the facial key point detector; Iterate to the end of the first training, output the last face key point detector obtained by training.

2. The method of claim 1, wherein, solving the optimization and updating problem of the face warper by minimizing distances between gradient fields of warped face data and gradient fields of the target domain data, and combining a thin plate spline interpolation method, the warped face data being generated from the source domain data by the face warper.

3. The method of claim 1, wherein, the number of iterations is in the range of 50 to 60.

4. The method of claim 1, wherein, The update frequency of the alternating optimization . 5.A spatial warping based face key point detector domain adaptation apparatus, comprising a non-volatile memory and one or more processors, the non-volatile memory storing executable codes, and the processors executing the executable codes to implement a spatial warping based face key point detector domain adaptation method according to any one of claims 1-4.

6. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises a stored computer program, wherein the computer program, when executed by a processor, controls a device where the storage medium is located to perform a spatial warping based face key point detector domain adaptation method according to any one of claims 1-4.

Images