Face reorientation model, model training method and device

Abstract

The present disclosure provides a face reorientation model, a model training method and device. The method comprises: an encoder for feature extraction on a face image to obtain a face feature; an irrelevant attribute decomposition module for feature extraction on the face feature to obtain a detail feature corresponding to the face feature; the detail feature comprises an identity feature, a gaze feature and a head feature; a cross attribute separation module for angle rotation on a target feature in the detail feature; the target feature is the gaze feature or the head feature; and a decoder for generating a face image based on the angle-rotated detail feature. The face reorientation model, the model training method and device provided by the present disclosure can improve the sample set quality of a gaze estimation algorithm.

Description

Technical Field

[0001] This disclosure belongs to the field of image processing technology, and more specifically, relates to a face retargeting model, a face retargeting model training method and apparatus, an electronic device, and a readable storage medium. Background Technology

[0002] Existing gaze estimation algorithms can be categorized into model-based and appearance-based methods. Model-based estimation systems extract features from eye images to fit a geometric model predicting the gaze angle. However, they require dedicated eye-tracking cameras to capture near-field images of the eyes, limiting their widespread application. In contrast, appearance-based solutions rely on facial or eye images captured by ordinary RGB cameras, such as those in smartphones, tablets, and public displays, making them promising for use in various scenarios. State-of-the-art appearance-based gaze estimation systems benefit from the representation learning capabilities of deep neural networks, demonstrating significantly superior performance compared to their model-based counterparts.

[0003] However, deep neural networks require a large-scale labeled gaze dataset for training. Due to various limitations and conditions in the data acquisition process, the quality of the acquired data is not ideal, making it difficult to collect existing gaze estimation datasets and resulting in poor sample richness. Summary of the Invention

[0004] The purpose of this disclosure is to provide a face redirection model, a model training method, and an apparatus to improve the sample set quality of gaze estimation algorithms.

[0005] A first aspect of this disclosure provides a face redirection model, including: An encoder is used to extract features from a face image to obtain facial features; An irrelevant attribute decomposition module is used to extract features from the facial features to obtain detailed features corresponding to the facial features; the detailed features include identity features, gaze features, and head features; The cross-attribute separation module is used to perform angular rotation on the target features in the detailed features; the target features are gaze features or head features. A decoder is used to generate new face images based on detailed features after angular rotation.

[0006] A second aspect of this disclosure provides a method for training a face retargeting model, comprising: The first face image and the second face image are sequentially input into the encoder, the irrelevant attribute decomposition module, and the cross attribute separation module to obtain the first detail feature, the second detail feature, the first target feature after angle rotation, and the second target feature after angle rotation. The first detail feature is a detail feature corresponding to the first face image, and the second detail feature is a detail feature corresponding to the second face image; the first target feature is a target feature in the first detail feature, and the second target feature is a target feature in the second detail feature; the first face image and the second face image are face images with the same identity but different poses; The first target feature after angle rotation is swapped into the second detail feature to obtain the first detail feature after feature swapping; the second target feature after angle rotation is swapped into the first detail feature to obtain the second detail feature after feature swapping. The first detailed feature after feature swapping and the second detailed feature after feature swapping are input into the decoder to generate a third face image corresponding to the first face image and a fourth face image corresponding to the second face image; The parameters of each module in the face retargeting model are updated based on the similarity between the first face image and the third face image, as well as the similarity between the second face image and the fourth face image, so as to achieve the training of the face retargeting model.

[0007] A third aspect of this disclosure provides a face retargeting model training apparatus, comprising: The feature extraction module is used to sequentially input the first face image and the second face image into the encoder, the irrelevant attribute decomposition module, and the cross attribute separation module to obtain the first detail feature, the second detail feature, the first target feature after angle rotation, and the second target feature after angle rotation. The first detail feature is a detail feature corresponding to the first face image, and the second detail feature is a detail feature corresponding to the second face image; the first target feature is a target feature in the first detail feature, and the second target feature is a target feature in the second detail feature; the first face image and the second face image are face images with the same identity but different poses; The feature exchange module is used to exchange the first target feature after angle rotation into the second detail feature to obtain the first detail feature after feature exchange; and to exchange the second target feature after angle rotation into the first detail feature to obtain the second detail feature after feature exchange. The image generation module is used to input the first detailed feature after feature exchange and the second detailed feature after feature exchange into the decoder to generate a third face image corresponding to the first face image and a fourth face image corresponding to the second face image; The parameter update module is used to update the parameters of each module in the face retargeting model based on the similarity between the first face image and the third face image, and the similarity between the second face image and the fourth face image, so as to achieve the training of the face retargeting model.

[0008] A fourth aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the face retargeting model training method described above.

[0009] A fifth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the face retargeting model training method described above.

[0010] The beneficial effects of the face retargeting model, model training method, and apparatus provided in this disclosure are as follows: In this embodiment, facial features are first extracted from the face image using an encoder. Then, detailed features are further extracted from the facial features using an irrelevant attribute decomposition module. Next, the gaze or head features within the detailed features are rotated, while other detailed features remain unchanged. Finally, a new face image is generated based on the rotated detailed features. In this way, by changing the gaze or head features of the face image to generate a new image, data augmentation is performed on the existing gaze estimation sample set, thereby improving the quality of the sample set. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of a face retargeting model structure provided in an embodiment of the present disclosure; Figure 2 This is a flowchart illustrating a face retargeting model training method provided in an embodiment of the present disclosure. Figure 3 A schematic diagram illustrating the principle of a face retargeting model training method provided in an embodiment of this disclosure; Figure 4 This is a structural block diagram of a face retargeting model training device provided in an embodiment of the present disclosure; Figure 5 This is a schematic block diagram of an electronic device provided according to an embodiment of the present disclosure. Detailed Implementation

[0013] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, so as to provide a thorough understanding of the embodiments of this disclosure. However, those skilled in the art will understand that this disclosure may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this disclosure with unnecessary detail.

[0014] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following description will be provided in conjunction with the accompanying drawings and specific embodiments.

[0015] Please refer to Figure 1 , Figure 1 A schematic diagram of the structure of a face retargeting model provided in an embodiment of this disclosure includes: Encoder 11 is used to extract features from a face image to obtain face features.

[0016] The irrelevant attribute decomposition module 12 is used to extract features from facial features to obtain detailed features corresponding to the facial features; the detailed features include identity features, gaze features and head features.

[0017] The cross-attribute separation module 13 is used to rotate the target feature in the detail features; the target feature is the gaze feature or the head feature.

[0018] Decoder 14 is used to generate new face images based on detailed features after angular rotation.

[0019] In this embodiment, the encoder function can be implemented using existing feature extraction networks, such as convolutional neural networks (CNN), recurrent neural networks (RNN), deep residual neural networks (ResNet), etc., and there is no limitation here.

[0020] After obtaining facial features Subsequently, this embodiment further extracts facial features through an irrelevant attribute decomposition module. Detailed features within. Specifically, this can be achieved through a decomposer. From facial features Extracting facial identity information from the middle For the remaining information and identity information Here, two fully connected layers are used to extract gaze and head features respectively. and identity characteristics During the detailed feature extraction process, the extracted gaze features can be... An existing gaze estimator is used for auxiliary training to estimate the gaze direction, ensuring accurate gaze features are extracted. Similarly, the extracted head features An existing gaze estimator is used for auxiliary training to estimate head orientation, ensuring accurate extraction of head features. .

[0021] Based on the extracted identity features, gaze features, and head features, the gaze features or head features (i.e., target features) are rotated to obtain rotated detail features. New face images are then generated based on these rotated detail features to expand the face image sample set. Furthermore, the same method can be used to perform multiple rotations on the target features. For example, in the first rotation, the gaze features are rotated to obtain face images with different gaze directions; then, in the second rotation, the head features are rotated to obtain face images with different head directions, and so on. Multiple rotations can yield face images with different gaze angles and / or head angles.

[0022] As can be seen from the above, in this embodiment of the present disclosure, facial features are first extracted from the face image using an encoder. Then, detailed features are further extracted from the facial features using an irrelevant attribute decomposition module. Next, the gaze feature or head feature in the detailed features is rotated, while other detailed features remain unchanged. Finally, a new face image is generated based on the rotated detailed features. In this way, by changing the gaze feature or head feature of the face image to generate a new image, the existing gaze estimation sample set can be augmented, thereby improving the quality of the sample set. In one embodiment of this disclosure, angular rotation is performed on the target feature within the detailed features, including: The gaze feature is rotated at an angle based on a preset first rotation matrix, or the head feature is rotated at an angle based on a preset second rotation matrix.

[0023] In this embodiment, either the gaze feature or the head feature can be rotated. Specifically, a first rotation matrix can be calculated based on the gaze angle corresponding to the gaze feature and a preset target gaze angle, and then the gaze feature can be rotated using the first rotation matrix. Similarly, a second rotation matrix can be calculated based on the head angle corresponding to the head feature and a preset target head angle, and then the head feature can be rotated using the second rotation matrix.

[0024] In one embodiment of this disclosure, the first rotation matrix is ​​specifically:

[0025] in, Denotes the first rotation matrix. Let j represent the first rotation angle and g represent the line-of-sight characteristic.

[0026] The second rotation matrix is: , in, Denotes the second rotation matrix. Let k represent the second rotation angle and h represent the head features.

[0027] In this embodiment, a two-dimensional angle variable is defined. The rotation matrix is ​​calculated. The first rotation angle refers to the angle between the current gaze angle corresponding to the current gaze feature and the target gaze angle; the second rotation angle refers to the angle between the current head angle corresponding to the current head angle and the target head angle.

[0028] In one embodiment of this disclosure, the target feature is a feature randomly selected from gaze features and head features.

[0029] In this embodiment, a feature can be randomly selected from gaze features and head features as the target feature, and a new face image can be generated by rotating the target feature. This facilitates the generation of a sample set of face images with randomly distributed gaze and head angles, further enhancing the richness of the sample set.

[0030] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating a face retargeting model training method according to an embodiment of the present disclosure. The method includes: S101: Input the first face image and the second face image into the encoder, the irrelevant attribute decomposition module, and the cross attribute separation module in sequence to obtain the first detail feature, the second detail feature, the first target feature after angle rotation, and the second target feature after angle rotation.

[0031] The first detail feature is the detail feature corresponding to the first face image, and the second detail feature is the detail feature corresponding to the second face image; the first target feature is the target feature in the first detail feature, and the second target feature is the target feature in the second detail feature; the first face image and the second face image are face images with the same identity but different poses.

[0032] In this embodiment, the inventors, through extensive research, discovered that existing methods for synthesizing facial image data may lead to the dilution of local details when changing the gaze angle and head angle of the face. This is primarily because they ignore the coupling relationship between the gaze angle and head angle. Accordingly, this embodiment inputs facial images with the same identity but different poses (i.e., different gaze angles and / or head angles) into a face retargeting model for joint training. This leverages the potential connections between facial images with different gaze angles and head angles to further and completely decouple gaze features and head features, while keeping other features unaffected.

[0033] Taking joint training using two facial images as an example, such as Figure 3 As shown, this embodiment uses two first face images X with the same identity but different poses. a Second face image X b The images X are sequentially input into the model and encoded by the encoder. a X b The corresponding facial features. In the Irrelevant Attribute Decomposition (IAD) module, the first face image X a Facial features are decomposed into Second face image X b Facial features are decomposed into Next, in the cross-attribute separation module, based on the rotation matrix... For the first face image X a The line-of-sight features are rotated at an angle to obtain the first target feature after the angle rotation. Based on rotation matrix For the second face image X b The line-of-sight features are rotated at an angle to obtain the second target feature after angle rotation. .

[0034] Wherein, rotation matrix In the image, the rotation angle ab represents the rotation angle from the first face image X. a The viewing angle is rotated to the second face image X. b The angle traversed during the process of viewing the angle; rotation matrix In the image X, the rotation angle ba represents the angle from the second face image. b The viewing angle is rotated to the first face image X. a The angle that the line of sight passes through during the process.

[0035] S102: Swap the first target feature after angle rotation to the second detail feature to obtain the first detail feature after feature swap; swap the second target feature after angle rotation to the first detail feature to obtain the second detail feature after feature swap.

[0036] like Figure 3 As shown, in this embodiment, the first face image X a The first target feature is obtained by rotating the line-of-sight features at an angle. Second face image X b The second target feature is obtained by rotating the line-of-sight features at an angle. The first target feature Swap to second face image X b In the second detailed features, the gaze features of the second face image are simulated; simultaneously, the second target features are... Swap to the first face image X a The first detailed features are used to simulate the gaze features of the first face image.

[0037] S103: Input the first detail feature after feature exchange and the second detail feature after feature exchange into the decoder to generate a third face image corresponding to the first face image and a fourth face image corresponding to the second face image.

[0038] In this embodiment, the first face image X a Identity characteristics Second target features And the first face image X a Head features The input decoder can generate a new third-person face image. Simultaneously, the second face image X b Identity characteristics First target characteristics Second face image X b Head features The input decoder can generate a new fourth face image. .

[0039] It should be noted that, Figure 3 In the embodiment shown, the first face image X a Second face image X b Both were subjected to angular rotation of the gaze feature to obtain the first target feature and the second target feature, respectively. In other embodiments, the first face image X can also be selected as the first face image X. a The first target feature is obtained by rotating the gaze feature by angle, and the second face image X is obtained by rotating the gaze feature by angle. b The second target feature is obtained by rotating the head features of the first face image X at an angle; or by rotating the head features of the first face image X at an angle. a The head features are rotated to obtain the first target features, and the second face image X is processed. b The second target feature is obtained by rotating the head features of the first face image X at an angle; or by rotating the head features of the first face image X at an angle.a The first target feature is obtained by rotating the gaze feature by angle, and the second face image X is obtained by rotating the gaze feature by angle. b The second target feature is obtained by rotating the line-of-sight features at an angle.

[0040] S104: The parameters of each module in the face retargeting model are updated based on the similarity between the first face image and the third face image, as well as the similarity between the second face image and the fourth face image, in order to train the face retargeting model.

[0041] In this embodiment, the training effect of the face retargeting model can be judged by comparing the similarity between the first and third face images, as well as the similarity between the second and fourth face images. When the similarity between the first and third face images, as well as the similarity between the second and fourth face images, are all greater than a preset similarity, it indicates that a new face image can be generated by rotating individual gaze features or head features, and the face retargeting model training is complete. Otherwise, the parameters of each module in the face retargeting model, such as the encoder, irrelevant attribute decomposition module, cross attribute separation module, and decoder, can be adjusted for retraining.

[0042] As can be seen from the above, the embodiments of this disclosure fully decouple gaze features and head features by utilizing the potential connections between face images with the same identity but different poses, while keeping other features unaffected. Based on the complete decoupling of gaze features and head features, new images are generated by changing the gaze features and / or head features of the face image, and data augmentation is performed on the existing gaze estimation sample set, thereby improving the quality of the sample set. In one embodiment of this disclosure, the method further includes the following steps before inputting the feature-swapped first detail feature and the feature-swapped second detail feature into the decoder: The identity features in the first detailed feature and the identity features in the second detailed feature are swapped.

[0043] In this embodiment, the identity features in the first detail feature and the identity features in the second detail feature are the same features. By exchanging the same features in the two face images, interactive learning can be carried out, and each can learn the corresponding attribute features, which is beneficial for the separation and decoupling of attribute features.

[0044] In one embodiment of this disclosure, an orthogonal loss function is incorporated into the training of the irrelevant attribute decomposition module. The formula for calculating the orthogonal loss function is as follows: , in, Indicates orthogonal loss. This represents the product of elements.

[0045] In this embodiment, orthogonal loss is used to supervise the extraction of identity features, gaze features, and head features, ensuring that they are orthogonal to each other in the latent space. This facilitates better subsequent attribute editing and achieves decoupling.

[0046] In one embodiment of this disclosure, the face retargeting model training method further includes: The adversarial loss is calculated using the first formula; the first formula includes:

[0047] in, This represents the discriminator loss, where D represents the discriminator. This represents the generated image. E[ ] represents the true value of the image, and E[ ] represents the expected value calculation. This represents the generator loss.

[0048] pass Calculate the reconstruction loss; where, Let ||||1 represent the reconstruction loss, and let L1 norm be ||||1.

[0049] The label loss is calculated using the second formula; the second formula includes:

[0050] in, Indicates label loss, Labels are generated based on the viewing angle. Depend on Obtained through a fully connected layer express The corresponding label truth value; The label is generated based on the head angle. Depend on Obtained through a fully connected layer express The corresponding label truth value.

[0051] pass Calculate the total loss function L.

[0052] In this embodiment, a generator G (including an encoder and a decoder) is used to regenerate the face image, and a discriminator is used to determine the authenticity of the generated image. Adversarial loss is used to calculate the loss of the generator and the discriminator.

[0053] Reconstruction loss is used to calculate the generated image. and target image L1 reconstruction loss between.

[0054] After obtaining the gaze and head features, the corresponding gaze direction vector can be directly regressed through a fully connected layer. and head direction vector The label loss is used to calculate the estimated loss of pseudo-labels in the line of sight and head direction.

[0055] By weighting and summing the adversarial loss, reconstruction loss, label loss, and orthogonal loss, the total loss function L can be obtained. When the total loss function L is less than a set threshold, it indicates that the similarity between the regenerated third face image and the first face image, as well as the similarity between the regenerated fourth face image and the second face image, are all greater than the preset similarity, and the face retargeting model training is complete.

[0056] Corresponding to the face retargeting model training method in the above embodiment, Figure 4 This is a structural block diagram of a face retargeting model training apparatus provided according to an embodiment of the present disclosure. For ease of explanation, only the parts relevant to the embodiment of the present disclosure are shown. References Figure 4 The face retargeting model training device 20 includes: a feature extraction module 21, a feature exchange module 22, an image generation module 23, and a parameter update module 24. The feature extraction module 21 is used to input the first face image and the second face image into the encoder, the irrelevant attribute decomposition module and the cross attribute separation module in sequence to obtain the first detail feature, the second detail feature, the first target feature after angle rotation and the second target feature after angle rotation.

[0057] The first detail feature corresponds to the detail features of the first face image, and the second detail feature corresponds to the detail features of the second face image. The first target feature is the target feature within the first detail feature, and the second target feature is the target feature within the second detail feature. The first face image and the second face image are face images of the same person but different poses.

[0058] The feature exchange module 22 is used to exchange the first target feature after angle rotation to the second detail feature to obtain the first detail feature after feature exchange; and to exchange the second target feature after angle rotation to the first detail feature to obtain the second detail feature after feature exchange.

[0059] The image generation module 23 is used to input the first detail feature after feature exchange and the second detail feature after feature exchange into the decoder to generate a third face image corresponding to the first face image and a fourth face image corresponding to the second face image.

[0060] The parameter update module 24 is used to update the parameters of each module in the face retargeting model based on the similarity between the first face image and the third face image, as well as the similarity between the second face image and the fourth face image, so as to realize the training of the face retargeting model.

[0061] In one embodiment of this disclosure, the method further includes the following steps before inputting the feature-swapped first detail feature and the feature-swapped second detail feature into the decoder: The identity features in the first detailed feature and the identity features in the second detailed feature are swapped.

[0062] In one embodiment of this disclosure, the parameter update module 24 is further configured to: In the training of the irrelevant attribute decomposition module, an orthogonal loss function is calculated. The formula for the orthogonal loss function is as follows: , in, Indicates orthogonal loss. This represents the product of elements.

[0063] See Figure 5 , Figure 5 This is a schematic block diagram of an electronic device provided according to an embodiment of the present disclosure. Figure 5 The electronic device 300 in this embodiment may include one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memories 304 store computer programs, including program instructions. The processors 301 execute the program instructions stored in the memories 304. Specifically, the processors 301 are configured to invoke the program instructions to perform the functions of each module / unit in the above-described device embodiments, for example... Figure 4 The functions of modules 21 and 22 shown.

[0064] It should be understood that, in the embodiments of this disclosure, the processor 301 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0065] Input device 302 may include a touchpad, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint orientation information), a microphone, etc., and output device 303 may include a display (LCD, etc.), a speaker, etc.

[0066] The memory 304 may include read-only memory and random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include non-volatile random access memory. For example, the memory 304 may also store device type information.

[0067] In specific implementations, the processor 301, input device 302, and output device 303 described in this disclosure embodiment can execute the implementation methods described in the first and second embodiments of the face retargeting model training method provided in this disclosure embodiment, or they can execute the implementation methods of the electronic device described in this disclosure embodiment, which will not be repeated here.

[0068] In another embodiment of this disclosure, a computer-readable storage medium is provided. This computer-readable storage medium stores a computer program, which includes program instructions. When executed by a processor, the program instructions implement all or part of the processes in the methods described above. The computer program can also instruct related hardware to complete the process. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0069] The computer-readable storage medium can be an internal storage unit of the electronic device in any of the foregoing embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the electronic device. Furthermore, the computer-readable storage medium can include both internal and external storage units of the electronic device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0070] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0071] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the electronic devices and units described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0072] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces or units, or it may be an electrical, mechanical, or other form of connection.

[0073] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiments of this disclosure, depending on actual needs.

[0074] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0075] The above are merely specific embodiments of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this disclosure, and these modifications or substitutions should all be covered within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A method for training a face redirection model, wherein the face redirection model comprises: An encoder is used to extract features from a face image to obtain facial features; An irrelevant attribute decomposition module is used to extract features from the facial features to obtain detailed features corresponding to the facial features; the detailed features include identity features, gaze features, and head features; The cross-attribute separation module is used to perform angular rotation on the target features in the detailed features; the target features are gaze features or head features. A decoder for generating new face images based on detailed features after angular rotation, characterized in that the training method includes: The first face image and the second face image are sequentially input into the encoder, the irrelevant attribute decomposition module, and the cross attribute separation module to obtain the first detail feature, the second detail feature, the first target feature after angle rotation, and the second target feature after angle rotation. The first detail feature is a detail feature corresponding to the first face image, and the second detail feature is a detail feature corresponding to the second face image; the first target feature is a target feature in the first detail feature, and the second target feature is a target feature in the second detail feature; the first face image and the second face image are face images with the same identity but different poses; The first target feature after angle rotation is swapped into the second detail feature to obtain the first detail feature after feature swapping; the second target feature after angle rotation is swapped into the first detail feature to obtain the second detail feature after feature swapping. The first detailed feature after feature swapping and the second detailed feature after feature swapping are input into the decoder to generate a third face image corresponding to the first face image and a fourth face image corresponding to the second face image; The parameters of each module in the face retargeting model are updated based on the similarity between the first face image and the third face image, as well as the similarity between the second face image and the fourth face image, so as to achieve the training of the face retargeting model.

2. The face retargeting model training method as described in claim 1, characterized in that, The method further includes the following steps before the step of inputting the first detail feature after feature swapping and the second detail feature after feature swapping into the decoder: The identity features in the first detailed feature and the identity features in the second detailed feature are swapped.

3. The face retargeting model training method as described in claim 1, characterized in that, Also includes: An orthogonal loss function is incorporated into the training of the irrelevant attribute decomposition module. The formula for calculating the orthogonal loss function is as follows: ， in, E[] represents the orthogonal loss, and E[] represents the expected value calculation. Indicates identity characteristics, This refers to the features in the detailed features that are other than the identity features. Indicates visual characteristics, Indicates head features, This represents the product of elements.

4. The face retargeting model training method as described in claim 1, characterized in that, The angular rotation of the target features in the detailed features includes: The gaze feature is rotated at an angle based on a preset first rotation matrix, or the head feature is rotated at an angle based on a preset second rotation matrix.

5. The face retargeting model training method as described in claim 4, characterized in that, The first rotation matrix is ​​specifically: in, Denotes the first rotation matrix. Let j represent the first rotation angle and g represent the line-of-sight characteristic. The second rotation matrix is: in, Denotes the second rotation matrix. Let k represent the second rotation angle and h represent the head features.

6. The face retargeting model training method as described in claim 1, characterized in that, The target feature is a feature randomly selected from gaze features and head features.

7. A face retargeting model training apparatus, used to implement the face retargeting model training method according to any one of claims 1 to 6, characterized in that, include: The feature extraction module is used to sequentially input the first face image and the second face image into the encoder, the irrelevant attribute decomposition module, and the cross attribute separation module to obtain the first detail feature, the second detail feature, the first target feature after angle rotation, and the second target feature after angle rotation. The first detail feature is a detail feature corresponding to the first face image, and the second detail feature is a detail feature corresponding to the second face image; the first target feature is a target feature in the first detail feature, and the second target feature is a target feature in the second detail feature; the first face image and the second face image are face images with the same identity but different poses; The feature exchange module is used to exchange the first target feature after angle rotation into the second detail feature to obtain the first detail feature after feature exchange; and to exchange the second target feature after angle rotation into the first detail feature to obtain the second detail feature after feature exchange. The image generation module is used to input the first detailed feature after feature exchange and the second detailed feature after feature exchange into the decoder to generate a third face image corresponding to the first face image and a fourth face image corresponding to the second face image; The parameter update module is used to update the parameters of each module in the face retargeting model based on the similarity between the first face image and the third face image, and the similarity between the second face image and the fourth face image, so as to achieve the training of the face retargeting model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images