A method and device for training a face activation generation model and face activation

Abstract

Embodiments of the present specification disclose a face activation generation model training method, which comprises: acquiring a source image and a driving image, and inputting the source image and the driving image into the face activation generation model to obtain face action information of the source image and face action information of the driving image; performing feature encoding on the source image to obtain source image features; projecting the face action information of the driving image onto the face action information of the source image in a three-dimensional space to enable the action of the driving image to be migrated to the source image; generating a target activation image according to the face action information of the driving image, the face action information of the source image and the source image features; and training the face activation generation model with the minimization of the difference between the target activation image and the driving image as a training target. Embodiments of the present specification also disclose a face activation method. Accordingly, embodiments of the present specification disclose a face activation generation model training device and a face activation device.

Description

Technical Field

[0001] This specification relates to an image processing method, and more particularly to a method for training a face activation generation model. Background Technology

[0002] Mobile phones have become an essential communication tool for everyone. With the development of technology, the way mobile phones are used for identity verification has evolved from passwords to fingerprints and then to facial recognition. However, the security of mobile phones and other devices is facing challenges from various aspects.

[0003] In IIFAA biometric attack detection, sufficiently realistic facial attack materials need to be intelligently generated to conduct security assessments of mobile phone authentication functions, ensuring the completeness and effectiveness of facial security assessments. This attack method must first be universal, exhibiting stable attack power across different mobile phone black-box tests; secondly, it must have a high attack rate and strong offensive capabilities, thus distinguishing the strength of facial recognition algorithms against attacks and effectively differentiating the attack resistance of various tested mobile phone manufacturers' authentication functions; furthermore, this attack method needs to be intelligently generated through algorithms, unlike physically generated masks or ordinary Photoshop methods, making potential errors discovered in the facial recognition algorithm's model traceable, facilitating model improvement, and thereby enhancing model reliability and the interpretability of model prediction results.

[0004] Compared to current mainstream face activation algorithms such as FOMM, there is still a need for more effective methods to obtain attack materials through action transfer in order to reduce motion distortions that occur during actions such as turning the head, preserve intact facial identity information, and thus make the results obtained when evaluating the anti-attack capabilities of face recognition algorithms more reliable.

[0005] Therefore, we hope to obtain a new method for generating face animations that can produce more realistic face animation images, thereby more effectively evaluating face recognition algorithms. Summary of the Invention

[0006] One of the objectives of the embodiments in this specification is to provide a training method for a face activation generation model. This method can transfer specified actions to specified face images, preserve the identity information of the specified face to a large extent, and effectively test the anti-attack capability of face recognition algorithms.

[0007] In accordance with the above objectives, this specification provides an embodiment of a method for training a face activation generation model, the method comprising:

[0008] Obtain the source image and the driving image;

[0009] The source image and the driving image are input into the face activation generation model to obtain the face motion information of the source image and the face motion information of the driving image, respectively.

[0010] The source image is feature-encoded to obtain the source image features;

[0011] In three-dimensional space, the facial motion information of the driving image is projected onto the facial motion information of the source image, so that the motion of the driving image is transferred to the source image;

[0012] A target activation image corresponding to the source image is generated based on the facial motion information of the driving image, the facial motion information of the source image, and the features of the source image;

[0013] The face activation generation model is trained with the goal of minimizing the difference between the target activated image and the driving image.

[0014] In the embodiments of this specification, facial motion information projection in three-dimensional space is used to achieve motion transfer from the driving image to the source image. This correlates three-dimensional positional information with facial texture information, effectively improving the problem of motion distortion and allowing for more accurate and complete preservation of facial identity information in the source image. By minimizing the difference between the target activated image and the driving image, the realism of the target activated image is significantly improved. In summary, the target activated image generated by the facial activation generation model trained using this method can more effectively test the anti-attack capability of face recognition algorithms.

[0015] Furthermore, in some embodiments, the facial motion information includes three-dimensional facial pose information and facial key point information.

[0016] Furthermore, in some embodiments, projecting the facial motion information of the driving image onto the facial motion information of the source image in three-dimensional space, so that the motion of the driving image is transferred to the source image, specifically includes:

[0017] By comparing the three-dimensional facial pose information of the source image and the three-dimensional facial pose information of the driving image, the facial key point information of the source image and the facial key point information of the driving image are corrected to the same pose, and the source key points and driving key points in three-dimensional space are obtained.

[0018] The driving keypoints are projected onto the source keypoints to transfer the motion of the driving image to the source image.

[0019] Furthermore, in some embodiments, the three-dimensional facial pose information includes the head rotation angle.

[0020] Furthermore, in some embodiments, the face activation generation model is implemented through a generative adversarial network, which includes a discriminator and a generator;

[0021] The training of the face activation generation model, with the goal of minimizing the difference between the target activated image and the driving image, specifically includes:

[0022] The generator is input with the facial motion information of the driving image, the facial motion information of the source image, and the features of the source image, and outputs the target activated image corresponding to the source image.

[0023] A discriminator is used to determine whether the target activated image is the driving image;

[0024] The generation loss is calculated based on the target activated image and the driving image;

[0025] Calculate the discrimination loss based on the discrimination result of the discriminator;

[0026] The generative adversarial network is trained with minimizing the discrimination loss and the generation loss as training objectives, respectively.

[0027] Furthermore, in some embodiments, calculating the generation loss based on the target activated image and the driving image specifically includes:

[0028] Calculate a pixel-based first loss between the target activated image and the driving image;

[0029] Calculate the second loss based on the feature map between the target activated image and the driving image;

[0030] The first loss and the second loss are weighted and fused together to form the generated loss.

[0031] Furthermore, in some implementations, by comparing the 3D facial pose information of the source image and the 3D facial pose information of the driving image, the facial key point information of the source image and the facial key point information of the driving image are corrected to the same pose, and the source key points and driving key points in 3D space are obtained, specifically including:

[0032] By comparing the 3D facial pose information of the source image and the 3D facial pose information of the driving image, the facial key point information of the source image and the facial key point information of the driving image are rotated and / or offset corrected to the same pose, thus obtaining the source key points and driving key points in 3D space.

[0033] Furthermore, in some embodiments, by comparing the three-dimensional facial pose information of the source image and the three-dimensional facial pose information of the driving image, the facial key point information of the source image and the facial key point information of the driving image are rotated and / or offset corrected to the same pose to obtain the source key points and driving key points in three-dimensional space. Specifically, this includes:

[0034] By comparing the 3D facial pose information of the source image and the driving image, the facial key point information of the source image and the driving image are rotated and / or offset to the same pose using matrix transformation, thus obtaining the source key points and driving key points in 3D space.

[0035] Another objective of the embodiments of this specification is to provide a face activation method that can transfer specified actions to specified face images to generate realistic face activation images and retain the identity information of the specified face to a large extent, thereby effectively testing the anti-attack capability of face recognition algorithms.

[0036] In accordance with the above objectives, embodiments of this specification provide a method for activating a human face, the method comprising:

[0037] Acquire the source image to be activated and the driving video containing the target action, and extract several frames of driving images from the driving video;

[0038] For each obtained driving image, the source image and the driving image are input into the face activation generation model to generate the target activation image corresponding to the source image. The face activation generation model is trained using the steps described in any of the above methods.

[0039] All the obtained target animation images are concatenated to obtain the target animation video.

[0040] Another objective of the embodiments of this specification is to provide a face activation generation model training device that can transfer specified actions to specified face images, retain the identity information of the specified face to a large extent, and effectively test the anti-attack capability of face recognition algorithms.

[0041] In accordance with the above objectives, embodiments of this specification provide a face activation generation model training device, the device comprising:

[0042] The sample acquisition module is used to acquire source images and driving images;

[0043] The information extraction module is used to input the source image and the driving image into the face activation generation model to obtain the face action information of the source image and the face action information of the driving image, respectively; and to perform feature encoding on the source image to obtain source image features;

[0044] The generation module is used to project the facial motion information of the driving image onto the facial motion information of the source image in three-dimensional space, so that the motion of the driving image is transferred to the source image; and to generate a target activated image corresponding to the source image based on the facial motion information of the driving image, the facial motion information of the source image, and the features of the source image.

[0045] The training module is used to train the face activation generation model with the goal of minimizing the difference between the target activated image and the driving image.

[0046] Furthermore, in some embodiments, the facial motion information includes three-dimensional facial pose information and facial key point information.

[0047] Furthermore, in some embodiments, the generation module compares the three-dimensional facial pose information of the source image with the three-dimensional facial pose information of the driving image, and corrects the facial key point information of the source image and the driving image to the same pose, thereby obtaining the source key points and driving key points in three-dimensional space.

[0048] The driving keypoints are projected onto the source keypoints to transfer the motion of the driving image to the source image.

[0049] Furthermore, in some embodiments, the three-dimensional facial pose information includes the head rotation angle.

[0050] Furthermore, in some embodiments, the face activation generation model is implemented through a generative adversarial network, which includes a discriminator and a generator;

[0051] The training module specifically includes:

[0052] The generator is input with the facial motion information of the driving image, the facial motion information of the source image, and the features of the source image, and outputs the target activated image corresponding to the source image.

[0053] A discriminator is used to determine whether the target activated image is the driving image;

[0054] The generation loss is calculated based on the target activated image and the driving image;

[0055] Calculate the discrimination loss based on the discrimination result of the discriminator;

[0056] The generative adversarial network is trained with minimizing the discrimination loss and the generation loss as training objectives, respectively.

[0057] Furthermore, in some embodiments, the training module calculates a pixel-based first loss between the target activated image and the driving image;

[0058] Calculate the second loss based on the feature map between the target activated image and the driving image;

[0059] The first loss and the second loss are weighted and fused together to form the generated loss.

[0060] Furthermore, in some embodiments, the generation module compares the three-dimensional facial pose information of the source image and the three-dimensional facial pose information of the driving image, and then rotates and / or offsets the facial key point information of the source image and the driving image to the same pose to obtain the source key points and driving key points in three-dimensional space.

[0061] Furthermore, in some embodiments, the generation module compares the three-dimensional facial pose information of the source image and the three-dimensional facial pose information of the driving image, and uses matrix transformation to rotate and / or offset the facial key point information of the source image and the driving image to the same pose, thereby obtaining the source key points and driving key points in three-dimensional space.

[0062] Another objective of the embodiments of this specification is to provide a face activation device that can transfer a specified action to a specified face image, generate a realistic face activation image, and retain the identity information of the specified face to a large extent, thereby effectively testing the anti-attack capability of the face recognition algorithm.

[0063] In accordance with the above objectives, embodiments of this specification provide a face activation device, the device comprising:

[0064] The sample acquisition module is used to acquire the source image to be activated and the driving video containing the target action, and to extract several frames of driving images from the driving video.

[0065] An activation generation module is used to input the source image and the driving image into the face activation generation model for each obtained driving image frame to generate a target activation image corresponding to the source image, wherein the face activation generation model is trained using the steps described in any of the above methods;

[0066] The connection module is used to connect all the obtained target activation images to obtain the target activation video.

[0067] Another object of the embodiments of this specification is to provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the program to implement the steps described in any of the above methods.

[0068] The face activation generation model training and face activation method described in the embodiments of this specification have the following characteristics:

[0069] Beneficial effects:

[0070] By projecting facial motion information from the driving image to the source image in three-dimensional space to achieve motion transfer, and associating three-dimensional positional information with facial texture information, the motion distortion caused by head rotation can be effectively improved, and the facial identity information in the source image can be preserved more accurately and completely. By minimizing the difference between the target live image and the driving image, the realism of the target live image is greatly improved, and the generated target live image can more effectively test the anti-attack capability of the face recognition algorithm.

[0071] Keypoint alignment correction based on 3D facial pose information can project the original 2D keypoint information into the 3D facial pose information, transforming facial animation into pseudo-3D animation, which allows facial keypoint information and facial texture features to be better integrated, generating more realistic animated images.

[0072] The face activation generation model training and face activation device described in the embodiments of this specification also have the above-mentioned beneficial effects. Attached Figure Description

[0073] Figure 1 The flowchart of the face activation generation model training method described in the embodiments of this specification is shown as an example in one implementation.

[0074] Figure 2 The steps of the face activation generation model training method described in the embodiments of this specification are illustrated in one specific implementation.

[0075] Figure 3 The flowchart of the face activation method described in the embodiments of this specification is shown as an example in one implementation.

[0076] Figure 4 The diagram illustrates, by way of example, a structural schematic of the face activation generation model training device described in one embodiment of the present specification.

[0077] Figure 5 A schematic diagram of the structure of the face activation device described in one embodiment of this specification is shown as an example. Detailed Implementation

[0078] First, it should be noted that the terminology used in the embodiments of this invention is for the purpose of describing specific embodiments only and is not intended to limit the invention. The singular forms "a," "described," and "the" used in the embodiments of this invention and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. Furthermore, the embodiments and features described herein can be combined with each other without conflict.

[0079] It should be noted that the face activation process is to transfer the action or expression of the face in image B to the face in image A. Since expression changes and expression transfer can be expressed as changes in the movement of facial features and the movement transfer of facial features, the actions in the embodiments of this specification include not only movements such as shaking the head and nodding, but also expressions such as smiling, frowning, and closing the eyes.

[0080] The following detailed description of the face activation generation model training, face activation method, and apparatus of the present invention, in conjunction with the accompanying drawings and specific embodiments, shall be provided in further detail, but such detailed description shall not constitute a limitation thereof.

[0081] In one embodiment of this specification, a method for training a face activation generation model is proposed. Figure 1 The flowchart of the face activation generation model training method described in the embodiments of this specification is shown as an example in one implementation.

[0082] like Figure 1 As shown, it includes:

[0083] 100: Obtain the source image and the driving image.

[0084] The source and driving images used need to contain significantly different facial movements in order to more intuitively observe the effect of motion transfer. The source and driving images can be extracted from an existing set of images containing different facial movements, or they can come from images of different people or several video clips captured by acquisition devices such as cameras, where each video clip records the process of facial movement changes of the same person.

[0085] In some embodiments, a video segment is acquired each time as a training sample, from which two frames with different facial movements are extracted as source image A and driving image B, respectively. The purpose of face activation is to transfer the movements in driving image B to the face in source image A. For each round of training, the face activation generation model is trained using the same person's face as sample data, so that the model can be fully trained and thus converge more easily.

[0086] 102: Input the source image and the driving image into the face activation generation model to obtain the face motion information of the source image and the face motion information of the driving image, respectively.

[0087] In face activation generation models, general face action detection methods can be used to obtain face action information from both the source image and the driving image. Face action information can represent head movement and changes in the position of facial features in three-dimensional space, offering greater realism and representativeness compared to action information in two-dimensional space.

[0088] In some embodiments, facial motion information includes facial three-dimensional pose information and facial key point information.

[0089] Accordingly, face motion detection methods include 3D face pose detection methods and face landmark detection methods. 3D face pose information can reflect the head's posture in 3D space to determine the face's orientation; while face landmark information records positional information by adding key points to key facial features such as eyes, nose, and mouth, or other key facial areas, thereby representing face motion based on changes in the position of these key areas.

[0090] In some more specific embodiments, the three-dimensional pose information of the face includes the head rotation angle.

[0091] Facial 3D pose information can be represented using facial 3D pose information, which describes the angles of head rotation on three mutually perpendicular coordinate axes in 3D space. In other words, facial pose can reflect the orientation of the head and express head position information. These three mutually perpendicular coordinate axes are constructed based on the directions of nodding, shaking, and swaying head movements, forming an XYZ axis centered on the head. This axis can represent the rotation angles of the head on the three coordinate axes respectively, and is summarized as facial 3D pose information.

[0092] 104: Perform feature encoding on the source image to obtain the source image features.

[0093] The source image is input into a general image encoder for feature encoding, and the corresponding source image features are output. This allows for further processing of the source image, such as motion transfer, based on the source image features, and in conjunction with the image decoder, a target live image is generated.

[0094] 106: Project the facial motion information of the driving image onto the facial motion information of the source image in three-dimensional space, so that the motion of the driving image is transferred to the source image.

[0095] In some embodiments, projecting facial motion information from a driving image onto facial motion information from a source image in three-dimensional space, so that the motion of the driving image is transferred to the source image, specifically includes:

[0096] By comparing the three-dimensional facial pose information of the source image and the three-dimensional facial pose information of the driving image, the facial key point information of the source image and the facial key point information of the driving image are corrected to the same pose, and the source key points and driving key points in three-dimensional space are obtained.

[0097] The driving keypoints are projected onto the source keypoints to transfer the motion of the driving image to the source image.

[0098] Since facial 3D pose information represents the head's pose in 3D space, the head pose of the source image and the driving image can be obtained based on this. Facial keypoint information in the image corresponds to the head pose. To better integrate the action in the driving image with the face in the source image, keypoint correction is performed based on the differences in their head poses. This ensures that the keypoints in the source image and the face behind them are in the same pose, resulting in source keypoints and driving keypoints. Further comparison is then made to identify keypoints with the same and different positions. Keypoints with different positions represent changes in facial action and are the focus of the transfer. By projecting the driving keypoints onto the source keypoints and adjusting them according to the different positions, the facial action information to be transferred is obtained from the source image, completing the action transfer in the driving image. Unifying the head pose of the source image and the driving image facilitates adjusting the face to a pose that is easier for action transfer, resulting in better face activation effects, while avoiding the influence of different head poses on the activation results.

[0099] It should be noted that when correcting the facial key point information of the source image and the driving image to the same pose, the key points corresponding to the head pose of the source image can be fixed, so that the head pose of the source image is used as the target pose, and the head pose of the driving image is made consistent with the target pose by correcting the key points of the driving image; alternatively, the key points corresponding to the head pose of the driving image can be fixed, so that the head pose of the driving image is used as the target pose, and the head pose of the source image is made consistent with the target pose by correcting the key points of the source image; alternatively, a target pose that can better show the face activation effect can be preset, and then the head pose of the source image and the head pose of the driving image are corrected to that target pose.

[0100] In some more specific embodiments, by comparing the 3D facial pose information of the source image and the 3D facial pose information of the driving image, the facial key point information of the source image and the facial key point information of the driving image are corrected to the same pose, and the source key points and driving key points in 3D space are obtained, specifically including:

[0101] By comparing the 3D facial pose information of the source image and the 3D facial pose information of the driving image, the facial key point information of the source image and the facial key point information of the driving image are rotated and / or offset corrected to the same pose, thus obtaining the source key points and driving key points in 3D space.

[0102] When correcting facial key point information, the facial key points can be rotated to the key point position corresponding to the target pose, or the facial key points can be offset. Of course, the facial key points can also be rotated to the approximate key point position corresponding to the target pose first, and then the position of the facial key points can be further adjusted by offsetting.

[0103] In some more specific embodiments, by comparing the 3D facial pose information of the source image and the 3D facial pose information of the driving image, the facial key point information of the source image and the facial key point information of the driving image are rotated and / or offset corrected to the same pose to obtain the source key points and driving key points in 3D space. Specifically, this includes:

[0104] By comparing the 3D facial pose information of the source image and the driving image, the facial key point information of the source image and the driving image are rotated and / or offset to the same pose using matrix transformation, thus obtaining the source key points and driving key points in 3D space.

[0105] Based on the facial key point information of the source image and the driving image, a target pose for key point correction is selected, and the target key points corresponding to the target pose are obtained through a general key point detection method. Then, the affine transformation matrices for adjusting the facial key point information of the source image and the driving image to the target key points are determined respectively. The corresponding affine transformation matrices are used to correct the facial key point information of the source image and the driving image to the target pose.

[0106] 108: Generate the target activation image corresponding to the source image based on the facial motion information of the driving image, the facial motion information of the source image, and the features of the source image.

[0107] A target livened image is generated by combining source image features with facial motion information obtained through motion transfer using a general image decoder. In some embodiments, the image decoder can be constructed based on the image decoder in a GAN generative network.

[0108] 110: The face activation generation model is trained with the goal of minimizing the difference between the target activated image and the driving image.

[0109] Differences between images can be obtained by calculating the differences in pixel-by-pixel values. In some embodiments, the images to be compared can be further processed into images of other layers before calculating the differences in pixel-by-pixel values, for example, by extracting feature maps from the images to be compared. Preferably, calculating the differences between the target activated image and the driving image through multiple methods, and then using the obtained multiple losses to train the model, is beneficial to improving the training efficiency of the face activation generation model and accelerating the improvement of model performance.

[0110] In some embodiments, the face activation generation model is implemented through a generative adversarial network, which includes a discriminator and a generator.

[0111] The training of the face activation generation model, with the goal of minimizing the difference between the target activation image and the driving image, specifically includes:

[0112] The generator takes the facial motion information of the driving image, the facial motion information of the source image, and the features of the source image as inputs and outputs the target activated image corresponding to the source image.

[0113] A discriminator is used to determine whether the target activated image is a driven image;

[0114] Calculate the generation loss based on the target activation image and the driving image;

[0115] Calculate the discrimination loss based on the discrimination result of the discriminator;

[0116] The generative adversarial network was trained with the goal of minimizing both the discriminative loss and the generative loss.

[0117] Generative Adversarial Networks (GANs) consist of a discriminator and a generator. During GAN training, the discriminator is first trained to possess good discrimination capabilities; then the generator is trained so that its generated results can fool the discriminator, achieving higher fidelity. In the above embodiment, after obtaining the target live image, it is input into the discriminator to determine whether it is a genuine sample image or a tampered forged image, obtaining a discrimination result. The discrimination loss is then calculated based on the accuracy of the discrimination result, and the discriminator's discrimination capability is trained by minimizing the discrimination loss.

[0118] Preferably, in each training round of the face activation generation model, the source image and the driving image in the training samples are of the same person. Face activation transfers facial movements from the driving image to the source image, fusing them together. Ideally, the resulting target activated image should be identical to the driving image. Therefore, the generation loss is determined by calculating the difference between the driving image and the target activated image. By minimizing the generation loss, the generator's generation effect is optimized, making the fusion of the driving image and the source image more natural.

[0119] As the generator is continuously optimized, the generated target live-action images become more and more realistic, making it increasingly difficult for the discriminator to determine their authenticity. When the discriminator can no longer determine the authenticity of the target live-action images, the generative adversarial training of the face live-action generation model is completed.

[0120] During the iterative training process of continuously calculating the loss, the face activation generation model also learns and transfers the face texture features in the source image and the driving image. That is, the face activation process associates and fuses the face action information representing the three-dimensional position of the face with the face texture features to realize face activation in three-dimensional space and improve the problem of face motion distortion after action transfer.

[0121] In some more specific embodiments, a generative adversarial network (GAN) model is used to construct a face activation generation model.

[0122] In some more specific embodiments, calculating the generation loss based on the target activation image and the driving image specifically includes:

[0123] Calculate the pixel-based first loss between the target activated image and the driving image;

[0124] Calculate the feature map-based second loss between the target activated image and the driving image;

[0125] The first loss and the second loss are weighted and combined to form the generated loss.

[0126] Preferably, the first loss can be obtained by calculating the pixel-wise mean squared error loss between the target activated image and the driving image.

[0127] Preferably, the second loss is obtained by calculating the face similarity between the target activated image and the driving image. This can be achieved by extracting the feature vectors of the target activated image and the driving image respectively, and then calculating the face similarity using a combination of Arcface (also known as the additive angular interval loss function) and cross-entropy. Training the face activation generation model using Arcface can effectively increase the inter-class distance while maintaining the inter-class distance, and simultaneously reduce the intra-class distance, enabling the face activation generation model to generate more efficient and accurate activation results.

[0128] Figure 2 The steps of the face activation generation model training method described in the embodiments of this specification are illustrated in one specific implementation.

[0129] like Figure 2As shown, in some embodiments, two frames are extracted from a video of the same person as the source image and the driving image, respectively. Facial motion information is extracted from the source image and the driving image, respectively. Specifically, facial key points in both the source and driving images are extracted using a common facial key point detection method. Since the faces in the source and driving images are identical, only the facial key points in the source image are extracted and copied to the driving image. Then, the 3D facial poses in both the source and driving images are extracted using a common facial pose detection method. Based on a comparison of the 3D facial poses, the differences in head pose between the source and driving images are identified. Matrix transformation is used to correct the facial key points, ensuring that the head pose in the source and driving images is consistent, resulting in corrected source and driving key points.

[0130] Next, image features are extracted from the source image. These features, along with source keypoints and driving keypoints, are input into a generator constructed using an image decoder within a GAN-based generative network. Action transfer is achieved by projecting driving keypoints onto source keypoints, and the final target activated image is generated by combining the source image features. Then, the face activation generation model is iteratively trained with the goal of minimizing the difference between the target activated image and the driving image, resulting in a face activation generation model without motion distortion in the generated activated images.

[0131] The face activation generation model training method provided in this specification utilizes the projection of facial motion information from the driving image to the source image in three-dimensional space to achieve motion transfer, which can effectively improve the motion distortion problem caused by head rotation. By minimizing the difference between the target activated image and the driving image, the realism of the target activated image is greatly improved, and the facial identity information in the source image can be preserved more accurately and completely. The generated target activated image can more effectively test the anti-attack capability of the face recognition algorithm. Keypoint alignment correction based on facial three-dimensional pose information can project the original 2D keypoint information into the 3D facial three-dimensional pose information, thus transforming face activation into pseudo-3D activation, allowing facial keypoint information and facial texture features to be better integrated, generating a more realistic activated image.

[0132] In another embodiment of this specification, a face activation method is proposed. Figure 3 The flowchart of the face activation method described in the embodiments of this specification is shown as an example in one implementation.

[0133] like Figure 3 As shown, it includes:

[0134] 200: Obtain the source image to be activated and the driving video containing the target action, and extract several frames of driving images from the driving video.

[0135] When collecting samples, images can be directly input by installing a local image acquisition device, or images can be collected from a remote image acquisition device in the cloud, or a pre-prepared image set can be retrieved from the cloud and used directly as model input.

[0136] When performing face activation, the face in the source image and the face in the driving video may not be the same person. In other words, the face activation method can replace the face in the driving video with the target face in the source image.

[0137] 202: For each obtained driving image, input the source image and the driving image into the face activation generation model to generate the target activation image corresponding to the source image. The face activation generation model is trained using the steps described in any of the above methods.

[0138] The face activation generation model extracts facial motion information from the source image and the driving image, and corrects the facial motion information in three-dimensional space to a pose that is conducive to motion transfer, thereby generating a highly realistic target activation image. This model can be used more effectively to test the anti-attack capability of face recognition algorithms, solve the motion distortion problem, and preserve the facial identity information in the source image relatively completely.

[0139] 204: Connect all the obtained target activation images to obtain the target activation video.

[0140] By concatenating the generated corresponding target activated image frames based on the order of the extracted driving image frames, an activated video in which the face has been replaced with the face in the source image can be obtained.

[0141] In one embodiment of this specification, a face activation generation model training device is proposed. Figure 4 The diagram illustrates, by way of example, a structural schematic of the face activation generation model training device described in one embodiment of the present specification.

[0142] like Figure 4 As shown, it includes:

[0143] Sample acquisition module 30 is used to acquire source image and driving image;

[0144] The information extraction module 32 is used to input the source image and the driving image into the face activation generation model, and to obtain the face action information of the source image and the face action information of the driving image respectively; and to perform feature encoding on the source image to obtain the source image features;

[0145] The generation module 34 is used to project the facial motion information of the driving image onto the facial motion information of the source image in three-dimensional space, so that the motion of the driving image is transferred to the source image; and to generate the target activation image corresponding to the source image based on the facial motion information of the driving image, the facial motion information of the source image and the features of the source image.

[0146] Training module 36 is used to train the face activation generation model with the goal of minimizing the difference between the target activation image and the driving image.

[0147] The source and driving images acquired by the sample acquisition module need to contain significantly different facial movements to more intuitively observe the effect of motion transfer. The sample acquisition module can extract source and driving images from an existing set of images containing different facial movements, or it can capture images of different people or several video clips using acquisition devices such as cameras, where each video clip records the process of facial movement changes of the same person.

[0148] In some embodiments, the sample acquisition module acquires a video segment as a training sample each time, and extracts two frames with different facial movements from it as source image A and driving image B, respectively. The purpose of face activation is to transfer the movements in driving image B to the face in source image A. For each round of training, the face activation generation model is trained using the same person's face as sample data, so that the model can be fully trained and thus converge more easily.

[0149] The information extraction module can use general face motion detection methods to obtain face motion information from both the source image and the driving image. Face motion information can represent head movement and changes in the position of facial features in three-dimensional space, offering greater realism and representativeness compared to two-dimensional motion information.

[0150] In some embodiments, facial motion information includes three-dimensional facial pose information and facial landmark information. Accordingly, the facial motion detection method includes a three-dimensional facial pose detection method and a facial landmark detection method. Three-dimensional facial pose information can reflect the head's posture in three-dimensional space to determine the face's orientation; while facial landmark information records positional information by adding key points to key facial features such as the eyes, nose, and mouth, or other key facial areas, thereby representing facial motion based on changes in the position of these key areas.

[0151] In some more specific embodiments, facial 3D pose information includes head rotation angles. Facial 3D pose information can be represented using facial 3D pose information, which describes the angles of head rotation along three mutually perpendicular coordinate axes in three-dimensional space. In other words, facial pose can reflect the orientation of the head and express head position information. These three mutually perpendicular coordinate axes are constructed around the head, based on the directions of nodding, shaking, and tilting head movements, and can represent the head rotation angles along the three coordinate axes respectively, summarizing them into facial 3D pose information.

[0152] The information extraction module also inputs the source image into a general image encoder for feature encoding and outputs the corresponding source image features, so as to perform further processing such as motion transfer on the source image based on the source image features, and generate the target live image in conjunction with the image decoder.

[0153] In some more specific embodiments, the generation module compares the three-dimensional facial pose information of the source image with the three-dimensional facial pose information of the driving image, corrects the facial key point information of the source image and the facial key point information of the driving image to the same pose, and obtains the source key points and driving key points in three-dimensional space; the driving key points are projected onto the source key points to transfer the motion of the driving image to the source image.

[0154] Since facial 3D pose information represents the head's pose in 3D space, the head pose of the source image and the driving image can be obtained based on this. Facial keypoint information in the image corresponds to the head pose. To better integrate the action in the driving image with the face in the source image, the generation module corrects keypoints based on the differences in their head poses, ensuring that the keypoints in the source image and the face behind them are in the same pose. This yields source keypoints and driving keypoints, which are then compared to identify keypoints with the same or different positions. Keypoints with different positions represent changes in facial action and are the focus of the transfer. The generation module projects the driving keypoints onto the source keypoints and adjusts the source keypoints based on their different positions, thereby obtaining the facial action information to be transferred from the source image and completing the action transfer in the driving image. Unifying the head pose of the source image and the driving image facilitates adjusting the face to a pose that is easier for action transfer, resulting in better face activation and avoiding the impact of different head poses on the activation results.

[0155] In some more specific embodiments, the generation module compares the three-dimensional facial pose information of the source image with the three-dimensional facial pose information of the driving image, and then rotates and / or offsets the facial key point information of the source image and the driving image to the same pose to obtain the source key points and driving key points in three-dimensional space.

[0156] When the generation module corrects facial key point information, it can rotate the facial key points to the key point positions corresponding to the target pose, or it can offset the facial key points. Of course, it can also first rotate the facial key points to the approximate key point positions corresponding to the target pose, and then further adjust the positions of the facial key points through offset.

[0157] In some more specific embodiments, the generation module compares the three-dimensional facial pose information of the source image with the three-dimensional facial pose information of the driving image, and uses matrix transformation to rotate and / or offset the facial key point information of the source image and the driving image to the same pose, thereby obtaining the source key points and driving key points in three-dimensional space.

[0158] The generation module selects the target pose for keypoint correction based on the facial keypoint information of the source image and the driving image, and obtains the target keypoints corresponding to the target pose through a general keypoint detection method. Then, it determines the affine transformation matrix to adjust the facial keypoint information of the source image and the driving image to the target keypoints. The corresponding affine transformation matrix is ​​used to correct the facial keypoint information of the source image and the driving image to the target pose.

[0159] The generation module uses a general image decoder to combine source image features with facial motion information obtained from action transfer to generate a target livened image. In some embodiments, the image decoder in the generation module can be constructed based on the image decoder in a GAN generative network.

[0160] Differences between images can be obtained by calculating the differences in pixel-by-pixel values. In some embodiments, the training module can further process the images to be compared into images of other layers before calculating the differences in pixel-by-pixel values, such as extracting feature maps from the images to be compared. Preferably, the training module calculates the differences between the target activated image and the driving image through multiple methods, and then uses the obtained multiple losses to train the model, which helps to improve the training efficiency of the face activation generation model and accelerate the improvement of model performance.

[0161] In some embodiments, the face activation generation model is implemented through a generative adversarial network (GAN), which includes a discriminator and a generator; the training module specifically includes:

[0162] The generator takes the facial motion information of the driving image, the facial motion information of the source image, and the features of the source image as inputs and outputs the target activated image corresponding to the source image.

[0163] A discriminator is used to determine whether the target activated image is a driven image;

[0164] Calculate the generation loss based on the target activation image and the driving image;

[0165] Calculate the discrimination loss based on the discrimination result of the discriminator;

[0166] The generative adversarial network was trained with the goal of minimizing both the discriminative loss and the generative loss.

[0167] Generative Adversarial Networks (GANs) consist of a discriminator and a generator. During GAN training, the discriminator is first trained to develop strong discrimination capabilities; then the generator is trained to produce results that can fool the discriminator and achieve higher fidelity. In the above embodiment, after obtaining the target live image, the training module inputs it into the discriminator to determine whether it is a genuine sample image or a forged image, obtaining a discrimination result. The discrimination loss is then calculated based on the accuracy of the discrimination result, and the discriminator's discrimination capability is trained by minimizing the discrimination loss.

[0168] Preferably, in each training round of the face activation generation model, the source image and the driving image in the training samples are of the same person. The training module transfers the facial movements in the driving image to the source image through face activation, and fuses them with the face in the source image. Ideally, the obtained target activation image should be consistent with the driving image. Therefore, the training module determines the generation loss by calculating the difference between the driving image and the target activation image, and optimizes the generation effect of the generator by minimizing the generation loss, so that the driving image and the source image are fused more naturally.

[0169] As the generator is continuously optimized, the generated target live-action images become more and more realistic, making it increasingly difficult for the discriminator to determine their authenticity. When the discriminator can no longer determine the authenticity of the target live-action images, the generative adversarial training of the face live-action generation model is completed.

[0170] During the iterative training process of continuously calculating the loss, the face activation generation model also learns and transfers the face texture features in the source image and the driving image. That is, the face activation process associates and fuses the face action information representing the three-dimensional position of the face with the face texture features to realize face activation in three-dimensional space and improve the problem of face motion distortion after action transfer.

[0171] In some more specific embodiments, a generative adversarial network (GAN) model is used to construct a face activation generation model.

[0172] In some more specific embodiments, the training module is used to calculate a pixel-based first loss between the target activated image and the driving image; calculate a feature map-based second loss between the target activated image and the driving image; and weight and fuse the first loss and the second loss as the generation loss.

[0173] Preferably, the first loss can be obtained by calculating the pixel-wise mean squared error loss between the target activated image and the driving image through the training module.

[0174] Preferably, the second loss is obtained by calculating the face similarity between the target activated image and the driving image through the training module. This can be achieved by extracting the feature vectors of the target activated image and the driving image respectively, and then calculating the face similarity using a combination of Arcface (also known as the additive angular interval loss function) and cross-entropy. Training the face activation generation model using Arcface can effectively increase the inter-class distance while maintaining the inter-class distance, and simultaneously reduce the intra-class distance, enabling the face activation generation model to generate more efficient and accurate activation results.

[0175] In another embodiment of this specification, a face activation device is provided. Figure 5 A schematic diagram of the structure of the face activation device described in one embodiment of this specification is shown as an example.

[0176] like Figure 5 As shown, it includes:

[0177] The sample acquisition module 40 is used to acquire the source image to be activated and the driving video containing the target action, and to extract several frames of driving images from the driving video.

[0178] The activation generation module 42 is used to input the source image and the driving image into the face activation generation model for each obtained driving image frame to generate the target activation image corresponding to the source image. The face activation generation model is trained using the steps described in any of the above methods.

[0179] The connection module 44 is used to connect all the obtained target activation images to obtain the target activation video.

[0180] When collecting samples in the sample acquisition module, images can be directly input by installing a local image acquisition device, or images can be acquired from a remote image acquisition device retrieved from the cloud, or a pre-prepared image set can be retrieved from the cloud and used directly as model input. In the sample acquisition module, the face in the source image and the face in the driving video do not have to be the same person. In other words, using the face activation device, the face in the driving video can be replaced with the target face in the source image.

[0181] The face activation generation model in the activation generation module extracts the face action information from the source image and the driving image, and corrects the respective face action information in three-dimensional space into a pose that is conducive to action transfer, thereby generating a highly realistic target activation image. This can be used more effectively to test the anti-attack capability of face recognition algorithms, solve the problem of motion distortion, and preserve the face identity information in the source image relatively completely.

[0182] The connection module connects the generated corresponding target live image frames based on the order of the extracted driving image frames to obtain a live video in which the face has been replaced with the face in the source image.

[0183] In one embodiment of this specification, an electronic device is also proposed, including a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the program to implement the steps described in any of the above methods.

[0184] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0185] It should be noted that the above examples are merely specific embodiments of the present invention, and the present invention is obviously not limited to the above embodiments, with many similar variations. All modifications that can be directly derived or conceived by those skilled in the art from the content disclosed in this invention should fall within the protection scope of this invention.

Claims

1. A method for training a face activation generation model, the method comprising: Obtain the source image and the driving image; The source image and the driving image are input into the face activation generation model to obtain the face motion information of the source image and the face motion information of the driving image, respectively; the face motion information includes face three-dimensional pose information and face key point information; The source image is feature-encoded to obtain the source image features; In three-dimensional space, the facial motion information of the driving image is projected onto the facial motion information of the source image to transfer the motion of the driving image to the source image: by comparing the three-dimensional facial pose information of the source image and the three-dimensional facial pose information of the driving image, the facial key point information of the source image and the facial key point information of the driving image are corrected to the same pose to obtain the source key points and driving key points in three-dimensional space; the driving key points are projected onto the source key points to transfer the motion of the driving image to the source image. A target activation image corresponding to the source image is generated based on the facial motion information of the driving image, the facial motion information of the source image, and the features of the source image; The face activation generation model is trained with the goal of minimizing the difference between the target activated image and the driving image.

2. The face activation generation model training method as described in claim 1, wherein the three-dimensional pose information of the face includes the head rotation angle.

3. The face activation generation model training method as described in claim 1, wherein the face activation generation model is implemented through a generative adversarial network, and the generative adversarial network includes a discriminator and a generator; The training of the face activation generation model, with the goal of minimizing the difference between the target activated image and the driving image, specifically includes: The generator is input with the facial motion information of the driving image, the facial motion information of the source image, and the features of the source image, and outputs the target activated image corresponding to the source image. A discriminator is used to determine whether the target activated image is the driving image; The generation loss is calculated based on the target activated image and the driving image; Calculate the discrimination loss based on the discrimination result of the discriminator; The generative adversarial network is trained with minimizing the discrimination loss and the generation loss as training objectives, respectively.

4. The face activation generation model training method as described in claim 3, wherein calculating the generation loss based on the target activation image and the driving image specifically includes: Calculate a pixel-based first loss between the target activated image and the driving image; Calculate the second loss based on the feature map between the target activated image and the driving image; The first loss and the second loss are weighted and fused together to form the generated loss.

5. The face activation generation model training method as described in claim 1, by comparing the three-dimensional pose information of the face in the source image and the three-dimensional pose information of the face in the driving image, corrects the face key point information of the source image and the face key point information of the driving image to the same pose, and obtains the source key points and driving key points in three-dimensional space, specifically including: By comparing the 3D facial pose information of the source image and the 3D facial pose information of the driving image, the facial key point information of the source image and the facial key point information of the driving image are rotated and / or offset corrected to the same pose, thus obtaining the source key points and driving key points in 3D space.

6. The face activation generation model training method as described in claim 5, which compares the three-dimensional pose information of the face in the source image and the three-dimensional pose information of the face in the driving image, and then rotates and / or offsets the face key point information of the source image and the driving image to the same pose to obtain the source key points and driving key points in three-dimensional space, specifically includes: By comparing the 3D facial pose information of the source image and the driving image, the facial key point information of the source image and the driving image are rotated and / or offset to the same pose using matrix transformation, thus obtaining the source key points and driving key points in 3D space.

7. A method for activating a human face, the method comprising: Acquire the source image to be activated and the driving video containing the target action, and extract several frames of driving images from the driving video; For each obtained driving image, the source image and the driving image are input into the face activation generation model to generate a target activation image corresponding to the source image, wherein the face activation generation model is trained using the method described in any one of claims 1 to 6; All the obtained target animation images are concatenated to obtain the target animation video.

8. A face activation generation model training device, the device comprising: The sample acquisition module is used to acquire source images and driving images; The information extraction module is used to input the source image and the driving image into the face activation generation model to obtain the face action information of the source image and the face action information of the driving image, respectively; the face action information includes face three-dimensional pose information and face key point information; The source image is feature-encoded to obtain the source image features; A generation module is used to project the facial motion information of the driving image onto the facial motion information of the source image in three-dimensional space, so that the motion of the driving image is transferred to the source image. Specifically, the generation module compares the three-dimensional facial pose information of the source image and the driving image to correct the facial key point information of the source image and the driving image to the same pose, obtaining source key points and driving key points in three-dimensional space. The driving key points are then projected onto the source key points to transfer the motion of the driving image to the source image. The generation module generates a target activated image corresponding to the source image based on the facial motion information of the driving image, the facial motion information of the source image, and the features of the source image. The training module is used to train the face activation generation model with the goal of minimizing the difference between the target activated image and the driving image.

9. The face activation generation model training device as described in claim 8, wherein the three-dimensional pose information of the face includes the head rotation angle.

10. The face activation generation model training device as described in claim 8, wherein the face activation generation model is implemented through a generative adversarial network, and the generative adversarial network includes a discriminator and a generator; The training module specifically includes: The generator is input with the facial motion information of the driving image, the facial motion information of the source image, and the features of the source image, and outputs the target activated image corresponding to the source image. A discriminator is used to determine whether the target activated image is the driving image; The generation loss is calculated based on the target activated image and the driving image; Calculate the discrimination loss based on the discrimination result of the discriminator; The generative adversarial network is trained with minimizing the discrimination loss and the generation loss as training objectives, respectively.

11. The face activation generation model training device as described in claim 10, The training module calculates a pixel-based first loss between the target activated image and the driving image; Calculate a second loss based on feature maps between the target activated image and the driving image; then weight and fuse the first loss and the second loss to obtain the generation loss.

12. The face activation generation model training device as described in claim 8, The generation module compares the three-dimensional facial pose information of the source image and the three-dimensional facial pose information of the driving image, and then rotates and / or offsets the facial key point information of the source image and the driving image to the same pose to obtain the source key points and driving key points in three-dimensional space.

13. The face activation generation model training device as described in claim 12, The generation module compares the three-dimensional facial pose information of the source image and the three-dimensional facial pose information of the driving image, and uses matrix transformation to rotate and / or offset the facial key point information of the source image and the driving image to the same pose, thereby obtaining the source key points and driving key points in three-dimensional space.

14. A face activation device, the device comprising: The sample acquisition module is used to acquire the source image to be activated and the driving video containing the target action, and to extract several frames of driving images from the driving video. An activation generation module is used to input the source image and the driving image into the face activation generation model for each obtained driving image frame to generate a target activation image corresponding to the source image, wherein the face activation generation model is trained using the method described in any one of claims 1 to 6; The connection module is used to connect all the obtained target activation images to obtain the target activation video.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of any of the methods described in claims 1-7.

Images