Facial action unit migration method and device based on deformation autoencoder and decoupling exchange

Abstract

The present invention discloses a facial action unit migration method and device based on deformation autoencoder and decoupling exchange. First, the source image and the target image are extracted from the source domain data set and the target domain data set to form an original image pair to form a training data set; then the original image pair is respectively input into the deformation autoencoder to decouple the corresponding texture image and deformation field; then the exchanged source deformation field and the target texture image are input into the deformation field converter together to output a target new deformation field that can adapt to the target texture image; finally, the target texture image is deformed by using the target new deformation field to obtain a target new image with the facial action unit of the source image. The present invention uses a deformation autoencoder to decouple texture features and shape features, and realizes the migration of facial action units from one source domain image to another target domain image while retaining the target domain characteristics of the target image in terms of facial identity and texture, posture, etc.

Description

Technical Field

[0001] The invention relates to a facial action unit migration method and device based on a deformable autoencoder and decoupling exchange, and belongs to computer vision technology. Background Art

[0002] Facial action units represent the subtle expressions of a person's face. Facial Action Coding System (FACS) was first proposed by Ekman et al., a famous American emotional psychologist, in 1978, and was significantly improved in 2002. The Facial Action Coding System is divided into several independent and interconnected facial action units (AUs) based on the anatomical characteristics of the face. The facial expressions can be reflected through the action characteristics of these AUs and the main areas they control. Therefore, transferring facial action units is essentially transferring the subtle expressions of a person's face.

[0003] Facial expression transfer refers to mapping the captured real user's expression to another target image, so as to achieve the purpose of transferring the facial expression to the target image. In the existing scheme, expression transfer can be performed in a three-dimensional (3D) manner. First, a 3D model of the face head needs to be built based on the face image, and then image distortion and fusion are performed based on the corresponding 3D face model to finally achieve facial expression transfer.

[0004] However, constructing a face model using 3D modeling involves complex calculations. Therefore, a large amount of computing resources is consumed during the actual training process, resulting in high model training costs and greater training difficulty.

[0005] In addition, achieving expression migration from a source domain image to another target domain image still faces great challenges. It is difficult for the migrated target image to completely retain the original target domain characteristics such as facial identity and posture, and it is difficult to ensure the authenticity and vividness of the migrated expression. Summary of the invention

[0006] Purpose of the invention: In order to overcome the deficiencies in the prior art, the present invention provides a method and device for migrating facial action units based on deformation autoencoders and decoupled exchange, the goal of which is to migrate facial action units from a source domain image to another target domain image. Since the existing open source library for facial feature point detection can accurately annotate facial images with feature points, it can be determined that both the source image and the target image have accurate feature point labels. Facial action units represent the fine facial expressions, and migrating facial action units is essentially migrating the fine facial expressions.

[0007] Technical solution: In order to achieve the above purpose, the technical solution adopted by the present invention is:

[0008] A facial action unit migration method based on deformation autoencoder and decoupled exchange, comprising the following steps:

[0009] S01: Extract source images and target images from the source domain dataset and the target domain dataset to form original image pairs to form the training dataset, and preprocess the original image pairs. Each image is cropped to 256×256 size and further randomly mirrored to achieve data enhancement, which improves the generalization ability of the model to a certain extent. The preprocessed original image pairs are input into the deformation autoencoder.

[0010] S02: Decompose the image into texture features unrelated to expression and shape features related to expression using a deformation autoencoder, and use feature point positions to supervise the decoupling process of shape features. Then input the texture features into a texture decoder to obtain the corresponding texture image, and input the shape features into a shape decoder to obtain the corresponding deformation field.

[0011] S03: inputting the obtained source deformation field and the target texture image into the deformation field converter, and outputting a target new deformation field that can adapt to the target texture image; inputting the obtained target deformation field and the source texture image into the deformation field converter, and outputting a source new deformation field that can adapt to the source texture image;

[0012] S04: deforming the target texture image using the target new deformation field to obtain a reconstructed target image, that is, replacing the target new image with the expression of the source image; deforming the source texture image using the source new deformation field to obtain a reconstructed source image, that is, replacing the source new image with the expression of the target image;

[0013] S05: input the target new image and the source new image into the deformation autoencoder as the target image and the source image respectively, repeat the decoupling-exchange-reconstruction process of steps S02 to S04, and obtain the reconstructed target new image and the reconstructed source new image, that is, the new image pair similar to the original image pair after the second exchange of expression, which is used for the calculation of the objective function;

[0014] S06: Combine the losses of each network module in the deep neural network and use the training data set to train the entire deep neural network model;

[0015] S07: Input the given source image and target image into the trained deep neural network model to transfer the facial action units from the source image to the target image.

[0016] Specifically, in step S02, the deformable autoencoder adopts a deep learning architecture, including an encoder network, a feature point detection network and two decoder networks, the two decoder networks are a texture decoder network and a shape decoder network respectively; for the picture input into the deformable autoencoder, the encoder network is first used to decouple the input picture into texture features unrelated to expression and shape features related to expression; the obtained shape features are then input into the feature point detection network to predict the feature point positions of the image, and the feature point positions are used to supervise the process of decoupling the input picture into shape features; the obtained texture features are then input into the texture decoder network to generate a texture picture, and the shape features are input into the shape decoder network to generate a deformation field.

[0017] Since the face shape formed by dense facial feature points also controls the facial geometry and subtle expressions, and there is a priori positional relationship between the center of the facial action unit and the feature points, the feature points can be used to supervise the migration of subtle expressions; although the deformable autoencoder can perform feature decoupling and reconstruction without supervision, the deformation of shape features is closely related to the subtle expressions of the face, so a feature point detection network is needed to predict the feature point positions of the image and supervise the decoupling of shape features based on the feature point positions.

[0018] Specifically, the input of the feature point detection network is the shape features of the face image, and the output is the predicted feature point position. The loss function of the feature point detection network is defined as:

[0019]

[0020] in: and They represent the x-coordinate and y-coordinate of the ith feature point, i=1,…,n, where n is the total number of feature points on the face image; ||·||2 represents the L2 norm; d0 is the actual distance between the pupils of both eyes, It is used to normalize the loss function, which is helpful to eliminate the differences in some facial attributes between different samples (such as different sizes of faces, fatness and thinness, etc.); q is the actual feature point position. Thanks to the development of deep learning, the feature point detection network is more robust to interference such as occlusion and posture. Therefore, using the feature point position to supervise the decoupling of shape features can adapt to various scenarios.

[0021] Specifically, the encoder network takes a face image I as input and outputs a low-dimensional vector representation of texture and shape Z = [Z T ,Z S ]; among them, Z T is the texture feature, Z S is the shape feature; the input of the texture decoder network is the texture feature Z T, the low-dimensional vector Z is transformed through the texture decoder network T Transformed into a texture image T, i.e., a neutral expression and face shape; the input of the shape decoder network is the shape feature Z S , through the low-dimensional vector Z S Modeling the deformation field W: The process of modeling the deformation field W includes the following steps:

[0022] (1) Use the differential decoder to decode the shape feature Z S The local deformation information is decoded: for the position k on the texture image T, after being processed by the deformation field W, the new position becomes W(k)=(W x (k),W y (k)), W(k) indicates the index of position k on the texture image T in the deformation field W, W x (k) and W y (k) indicates the new position of position k on the texture image T in the x direction and the y direction, respectively, so as to realize the deformation of the texture image T by the deformation field W;

[0023] The differential decoder is used to generate the spatial gradient of the deformation field. x W x and y W y :▽ x W x The absolute value of represents the displacement of consecutive pixels in the x direction, ▽ x W x The positive and negative values ​​represent the displacement direction of consecutive pixels in the x direction, for example, ▽ x W x =2 means translation of two units in the positive direction of x, ▽ x W x =-1 means translation one unit in the negative direction of x; ▽ y W y The absolute value of represents the displacement of consecutive pixels in the y direction, ▽ y W y The positive and negative values ​​represent the displacement direction of consecutive pixels in the y direction; by controlling ▽ x W x and y W y Avoid new images Folding or over-stretching (over-stretching) to achieve the purpose of controlling the entire deformation field W; where: x and y Represents the horizontal and vertical components of the spatial gradient;

[0024] (2) Pass the output of the differential decoder through the ReLU activation function (▽ x W x andy W y ), ensure that adjacent pixels are offset in the same direction in the x and y directions; the purpose of the same direction offset is to ensure the rationality of the deformation and avoid left-right or upside-down);

[0025] (3) The output of the differential decoder is constructed into a smooth and regular deformation field W through the spatial data integration layer.

[0026] In order to convert the source image I s The expression of the target image I is transferred to t On the other hand, the deformation fields of the two images need to be exchanged; however, the source deformation field W s Cannot be directly applied to the target texture image T t , so a deformation field converter is introduced to exchange the source deformation field W s Correction is performed to obtain the target new deformation field Target new deformation field Should be consistent with the target deformation field W t Approach; at the same time, the target deformation field W t Cannot be directly applied to the source texture image T s , so a deformation field converter is introduced to exchange the target deformation field W t Correction is performed to obtain the new deformation field of the source New deformation field Should be the same as the source deformation field W s near.

[0027] The deformation field converter is a convolutional neural network with six convolutional layers. There is a maximum pooling layer after every two consecutive convolutional layers. The pooling kernel size of each maximum pooling layer is 2×2 and the step size is 2. The number of filters corresponding to every two convolutional layers in the six convolutional layers is the same. The number of filters corresponding to the first and second convolutional layers is 96, the number of filters corresponding to the third and fourth convolutional layers is 128, and the number of filters corresponding to the fifth and sixth convolutional layers is 160. The filter size is 3×3 and the step size is 1. The output of the third maximum pooling layer is the new deformation field.

[0028] Since the input is any two source images I s and target image I t , so there is no real expression exchange picture to directly constrain when training deep neural networks; in order to constrain the entire decoupling-exchange-reconstruction process, the source new picture and target new picture Input it into the deformation autoencoder again, repeat steps S02 to S04, and get a new source image and new pictures with new goals Since swapping expressions twice will return to the original expressions, they should be consistent with the source image I s and target image I t near.

[0029] The target texture image T t and the target deformation field W t The target image I t Input the texture image and deformation field generated by the deformation autoencoder, the target texture image T t The pixel value at position k is T t (k); Target new deformation field Control the deformation of the face, Indicate the new target image The pixel value at position k in the target texture image T t The position index on the image; bilinear interpolation is used to generate a new target image It means that it is asymptotically equal to;

[0030] The source texture image T s and the target deformation field W s Source Image I s Input the texture image and deformation field generated by the deformation self-encoder, the source texture image T s The pixel value at position k is T s (k); Source new deformation field Control the deformation of the face, Indicates source new picture The pixel value at position k in the source texture image T s The position index on the source image is generated by bilinear interpolation.

[0031] Specifically, the deformation field W and texture image T are decoupled from the input image I through the deformation autoencoder, and the new deformation field is obtained through the deformation field converter New deformation field Should be similar to the deformation field W, and reconstruct a new image Add new picture As the input image, repeat the process once to get a new new image, the new new image Should be similar to input image I in: Indicates new picture The pixel value at position k on the Indicate new picture The pixel value at position k on the texture image T is indexed by the position of the texture image. Indicates that the texture image T is indexed by position Should be given The pixel value of

[0032] Define the loss function L for image reconstruction rec for:

[0033]

[0034] The total variation of the differential deformation field in the x-direction and y-direction is used to penalize the fast-changing deformation in the local area to improve the smoothness and regularity of the deformation field W. The penalty function L of the deformation field W is defined smooth for:

[0035] L smooth =||▽ x W x ||1+||▽ y W y ||1 (3)

[0036] At the same time, when training deep neural networks, the deformation field W is required to approach the identical mapping deformation field W0, and the loss function L of the deformation field W is defined as warp for:

[0037]

[0038] To generate realistic new images Define the adversarial loss function L for image reconstruction adv for:

[0039]

[0040] Define the cycle consistency loss function L cycle for:

[0041]

[0042] Where: ||·||1 represents the L1 norm, ||·||2 represents the L2 norm; W0(k)=k, indicating that the index of position k on the texture image T in the identity mapping deformation field W0 is k; W(k) indicates the index of position k on the texture image T in the deformation field W; D(I) and Denotes that the discriminator D judges the input image I and the new image approximates the probability of the input image I, and -E[·] represents the expectation of the entropy of ·.

[0043] Formulas (3) and (4) are used to constrain the deep neural network to learn an effective deformation field W; deforming the texture image T generates a reasonable new image New pictures The target domain characteristics of the input image I, such as facial identity and posture, are retained; the goal of adversarial training is to make the discriminator D unable to distinguish new images and input image I; because input image I is also used as a reference image, D(I) should be close to 1; and the new image To synthesize pictures, during the adversarial training process, new pictures It will go from being unrealistic at the beginning to becoming more and more realistic later, so It will gradually change from an initial value close to 0 to a value close to 1.

[0044] Specifically, in step S06, by combining the losses of each network module in the deep neural network, the overall objective function of the entire deep neural network model is obtained as follows:

[0045]

[0046] Where: s , w , l , a and λ c To control the weights of each loss item, G includes the encoder network, texture decoder network, shape decoder network, deformation field converter and feature point detection network modules except the discriminator D.

[0047] A device for implementing any of the above-mentioned facial action unit migration methods based on deformation autoencoders and decoupled exchange, comprising an encoder, a texture decoder, a shape decoder, a discriminator, a deformation field converter, and a feature point detection network module;

[0048] The encoder network module is used to decouple the image into texture features unrelated to expression and shape features related to expression, and input them into subsequent network modules respectively;

[0049] The texture decoder network module converts low-dimensional texture features into texture images, i.e., neutral expressions and face shapes, which are used for subsequent image reconstruction;

[0050] The shape decoder network module uses a differential decoder to predict local deformation information, passes the output of the differential decoder through a ReLU module, and then uses a spatial integration layer to reconstruct the deformation field from the spatial gradient, and the deformation field is used for subsequent image reconstruction;

[0051] The discriminator network module is used to determine whether the reconstructed image is a real image;

[0052] The deformation field converter network module is used to correct the exchanged source deformation field so that it can be directly applied to the target texture image, thereby reconstructing the target image.

[0053] Beneficial effects: The facial action unit migration method and device based on deformable autoencoder and decoupling exchange provided by the present invention use a deformable autoencoder to achieve feature decoupling and image reconstruction, and achieve facial action unit migration from a source domain image to another target domain image by constructing a deep learning network architecture while retaining the original target domain characteristics such as facial identity and posture, showing stronger robustness and significantly improving the expressiveness of the model; the facial action unit migration framework proposed by the present invention is used to migrate the expressions of each frame in the source video to the same target image, so as to generate a target domain video. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 1 is a flow chart for implementing the method of the present invention;

[0055] Figure 2 It is a schematic diagram of the framework structure of the method of the present invention;

[0056] Figure 3 Schematic diagram of the positional relationship between the center of the facial action unit and the feature points. DETAILED DESCRIPTION

[0057] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0058] In the description of the present invention, it is necessary to understand that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0059] like Figure 1 FIG. 1 is a flow chart of a facial action unit migration method based on a deformable autoencoder and decoupled exchange, the method comprising the following steps:

[0060] S01: Extract source images and target images from the source domain dataset and the target domain dataset respectively to form original image pairs to form a training dataset, and preprocess the original image pairs.

[0061] Each image is cropped to 256×256 size and further randomly mirrored to achieve data augmentation, which improves the generalization ability of the model to a certain extent.

[0062] S02: Input the preprocessed original image pairs into the deformation autoencoder respectively; use the deformation autoencoder to decouple the image into texture features unrelated to expression and shape features related to expression, and use the feature point position to supervise the decoupling process of the shape features, then input the texture features into the texture decoder to obtain the corresponding texture image, and input the shape features into the shape decoder to obtain the corresponding deformation field.

[0063] The deformable autoencoder adopts a deep learning architecture, including an encoder network, a feature point detection network and two decoder networks, the two decoder networks are a texture decoder network and a shape decoder network respectively; for the picture input into the deformable autoencoder, the encoder network is first used to decouple the input picture into texture features unrelated to expression and shape features related to expression; the obtained shape features are then input into the feature point detection network to predict the feature point positions of the image, and the feature point positions are used to supervise the process of decoupling the input picture into shape features; the obtained texture features are then input into the texture decoder network to generate a texture picture, and the shape features are input into the shape decoder network to generate a deformation field.

[0064] like Figure 3 As shown in the figure, since the face shape formed by dense facial feature points also controls the face geometry and fine expressions, and there is a priori positional relationship between the center of the facial action unit and the feature points, the feature points can be used to supervise the migration of fine expressions; although the deformation autoencoder can perform feature decoupling and reconstruction without supervision, the deformation of the shape feature is closely related to the fine expression of the face, so a feature point detection network is needed to predict the feature point position of the image and supervise the decoupling of the shape feature based on the feature point position. The input of the feature point detection network is the shape feature of the face image, and the output is the predicted feature point position. The loss function of the feature point detection network is defined as:

[0065]

[0066] in: and They represent the x-coordinate and y-coordinate of the ith feature point, i=1,…,n, where n is the total number of feature points on the face image; ||·||2 represents the L2 norm; d0 is the actual distance between the pupils of both eyes, It is used to normalize the loss function, which is helpful to eliminate the differences in some facial attributes between different samples (such as different sizes of faces, fatness and thinness, etc.); q is the actual feature point position. Thanks to the development of deep learning, the feature point detection network is more robust to interference such as occlusion and posture. Therefore, using the feature point position to supervise the decoupling of shape features can adapt to various scenarios.

[0067] The encoder network takes a face image I as input and outputs a low-dimensional vector representation of texture and shape Z = [Z T ,Z S ]; among them, Z T is the texture feature, Z S is the shape feature; the input of the texture decoder network is the texture feature Z T , the low-dimensional vector Z is transformed through the texture decoder network T Transformed into a texture image T, i.e., a neutral expression and face shape; the input of the shape decoder network is the shape feature Z S , through the low-dimensional vector Z S Modeling the deformation field W: The process of modeling the deformation field W includes the following steps:

[0068] (21) Use the differential decoder to decode the shape feature Z S The local deformation information is decoded: for the position k on the texture image T, after being processed by the deformation field W, the new position becomes W(k)=(W x (k),W y (k)), W(k) indicates the index of position k on the texture image T in the deformation field W, W x (k) and W y (k) indicates the new position of position k on the texture image T in the x direction and the y direction respectively, thereby realizing the deformation of the texture image T by the deformation field W.

[0069] The differential decoder is used to generate the spatial gradient of the deformation field. x W x and y W y :▽ x W x The absolute value of represents the displacement of consecutive pixels in the x direction, ▽ x W x The positive and negative values ​​represent the displacement direction of consecutive pixels in the x direction, for example, ▽ x W x =2 means translation of two units in the positive direction of x, ▽ x W x =-1 means translation one unit in the negative direction of x; ▽ y W y The absolute value of represents the displacement of consecutive pixels in the y direction, ▽ y W y The positive and negative values ​​represent the displacement direction of consecutive pixels in the y direction; by controlling ▽ x W x and y W y Avoid new images Folding or over-stretching (over-stretching) to achieve the purpose of controlling the entire deformation field W; where: x and y Represents the horizontal and vertical components of the spatial gradient;

[0070] (22) The output of the differential decoder is passed through the ReLU activation function (▽ x W x and y W y ), ensure that adjacent pixels are offset in the same direction in the x and y directions; the purpose of the same direction offset is to ensure the rationality of the deformation and avoid left-right or upside-down);

[0071] (23) The output of the differential decoder is constructed into a smooth and regular deformation field W through the spatial data integration layer.

[0072] S03: input the obtained source deformation field and the target texture image into the deformation field converter together, and output a target new deformation field that can adapt to the target texture image; input the obtained target deformation field and the source texture image into the deformation field converter together, and output a source new deformation field that can adapt to the source texture image.

[0073] In order to convert the source image I s The expression of the target image I is transferred to t On the other hand, the deformation fields of the two images need to be exchanged; however, the source deformation field W s Cannot be directly applied to the target texture image T t , so a deformation field converter is introduced to exchange the source deformation field W s Correction is performed to obtain the target new deformation field Target new deformation field Should be consistent with the target deformation field W t Approach; at the same time, the target deformation field W t Cannot be directly applied to the source texture image T s , so a deformation field converter is introduced to exchange the target deformation field W t Correction is performed to obtain the new deformation field of the source New deformation field Should be the same as the source deformation field W s near.

[0074] The deformation field converter is a convolutional neural network with six convolutional layers. There is a maximum pooling layer after every two consecutive convolutional layers. The pooling kernel size of each maximum pooling layer is 2×2 and the step size is 2. The number of filters corresponding to every two convolutional layers in the six convolutional layers is the same. The number of filters corresponding to the first and second convolutional layers is 96, the number of filters corresponding to the third and fourth convolutional layers is 128, and the number of filters corresponding to the fifth and sixth convolutional layers is 160. The filter size is 3×3 and the step size is 1. The output of the third maximum pooling layer is the new deformation field.

[0075] S04: deforming the target texture image using the target new deformation field to obtain a reconstructed target image, that is, replacing the target new image with the expression of the source image; deforming the source texture image using the source new deformation field to obtain a reconstructed source image, that is, replacing the source new image with the expression of the target image.

[0076] The deformation field W and texture image T are decoupled from the input image I through the deformation autoencoder, and the new deformation field is obtained through the deformation field converter. New deformation field Should be similar to the deformation field W, and reconstruct a new image Add new picture As the input image, repeat the process once to get a new new image, the new new image Should be similar to input image I in: Indicates new picture The pixel value at position k on the Indicate new picture The pixel value at position k on the texture image T is indexed by the position of the texture image. Indicates that the texture image T is indexed by position Should be given The pixel value of .

[0077] Define the loss function L for image reconstruction rec for:

[0078]

[0079] The total variation of the differential deformation field in the x-direction and y-direction is used to penalize the fast-changing deformation in the local area to improve the smoothness and regularity of the deformation field W. The penalty function L of the deformation field W is defined smooth for:

[0080] L smooth =||▽ x W x ||1+||▽ y W y ||1 (3)

[0081] At the same time, when training deep neural networks, the deformation field W is required to approach the identical mapping deformation field W0, and the loss function L of the deformation field W is defined as warp for:

[0082]

[0083] To generate realistic new images Define the adversarial loss function L for image reconstruction adv for:

[0084]

[0085] Where: ||·||1 represents the L1 norm, ||·||2 represents the L2 norm; W0(k)=k, indicating that the index of position k on the texture image T in the identity mapping deformation field W0 is k; W(k) indicates the index of position k on the texture image T in the deformation field W; D(I) and Denotes that the discriminator D judges the input image I and the new image approximates the probability of the input image I, and -E[·] represents the expectation of the entropy of ·.

[0086] Formulas (3) and (4) are used to constrain the deep neural network to learn an effective deformation field W; deforming the texture image T generates a reasonable new image New pictures The target domain characteristics of the input image I, such as facial identity and posture, are retained; the goal of adversarial training is to make the discriminator D unable to distinguish new images and input image I; because input image I is also used as a reference image, D(I) should be close to 1; and the new image To synthesize pictures, during the adversarial training process, new pictures It will go from being unrealistic at the beginning to becoming more and more realistic later, so It will gradually change from an initial value close to 0 to a value close to 1.

[0087] S05: Input the target new image and the source new image into the deformable autoencoder as the target image and the source image respectively, and repeat the decoupling-exchange-reconstruction process of steps S02 to S04 to obtain the reconstructed target new image and the reconstructed source new image, that is, the new image pair that is similar to the original image pair after the expression is exchanged twice, which is used for the calculation of the objective function.

[0088] Since the input is any two source images I s and target image I t , so there is no real expression exchange picture to directly constrain when training deep neural networks; in order to constrain the entire decoupling-exchange-reconstruction process, the source new picture and target new picture Input it into the deformation autoencoder again, repeat steps S02 to S04, and get a new source image and new pictures with new goals Since swapping expressions twice will return to the original expressions, they should be consistent with the source image I s and target image I t Therefore, we define the cycle consistency loss function L cycle for:

[0089]

[0090] S06: Combine the losses of each network module in the deep neural network and use the training data set to train the entire deep neural network model.

[0091] By combining the losses of each network module in the deep neural network, the overall objective function of the entire deep neural network model is obtained as follows:

[0092]

[0093] Where: s , w , l , a and λ c To control the weights of each loss item, G includes the encoder network, texture decoder network, shape decoder network, deformation field converter and feature point detection network modules except the discriminator D.

[0094] S07: Input the given source image and target image into the trained deep neural network model to transfer the facial action units from the source image to the target image.

[0095] The method of the present invention can be completely implemented by computer without manual assistance; this shows that the present invention can realize batch automatic processing, which can greatly improve processing efficiency and reduce labor costs.

[0096] A device for implementing the above method comprises an encoder, a texture decoder, a shape decoder, a discriminator, a deformation field converter and a feature point detection network module; the encoder network module is used to decouple an image into texture features unrelated to expression and shape features related to expression, and input them into subsequent network modules respectively; the texture decoder network module converts low-dimensional texture features into texture images, i.e. neutral expression and face shape, and the texture images are used for subsequent image reconstruction; the shape decoder network module uses a differential decoder to predict local deformation information, transmits the output of the differential decoder through a ReLU module, and then uses a spatial integration layer to reconstruct a deformation field from spatial gradients, and the deformation field is used for subsequent image reconstruction; the discriminator network module is used to determine whether the reconstructed image is a real image; the deformation field converter network module is used to correct the exchanged deformation field so that it can be directly applied to a target texture image, thereby reconstructing the target image.

[0097] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0098] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0099] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any form, and any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A facial action unit migration method based on deformation autoencoder and decoupled exchange, characterized by: The steps include: S01: extract source images and target images from the source domain dataset and the target domain dataset respectively to form original image pairs to form a training dataset, preprocess the original image pairs, and input the preprocessed original image pairs into the deformation autoencoder; S02: Decompose the image into texture features unrelated to expression and shape features related to expression using a deformation autoencoder, and use feature point positions to supervise the decoupling process of shape features. Then input the texture features into a texture decoder to obtain the corresponding texture image, and input the shape features into a shape decoder to obtain the corresponding deformation field. S03: inputting the obtained source deformation field and the target texture image into the deformation field converter, and outputting a target new deformation field that can adapt to the target texture image; inputting the obtained target deformation field and the source texture image into the deformation field converter, and outputting a source new deformation field that can adapt to the source texture image; S04: deforming the target texture image using the target new deformation field to obtain a reconstructed target image, that is, replacing the target new image with the expression of the source image; deforming the source texture image using the source new deformation field to obtain a reconstructed source image, that is, replacing the source new image with the expression of the target image; S05: input the target new image and the source new image into the deformation autoencoder as the target image and the source image respectively, repeat the decoupling-exchange-reconstruction process of steps S02 to S04, and obtain the reconstructed target new image and the reconstructed source new image, that is, the new image pair similar to the original image pair after the second exchange of expression, which is used for the calculation of the objective function; S06: Combine the losses of each network module in the deep neural network and use the training data set to train the entire deep neural network model; S07: Input the given source image and target image into the trained deep neural network model to transfer the facial action units from the source image to the target image.

2. The facial action unit migration method based on deformation autoencoder and decoupled exchange according to claim 1, characterized in that: In the step S02, the deformable autoencoder adopts a deep learning architecture, including an encoder network, a feature point detection network and two decoder networks, wherein the two decoder networks are a texture decoder network and a shape decoder network respectively; for a picture input to the deformable autoencoder, the encoder network is first used to decouple the input picture into texture features unrelated to expression and shape features related to expression; the obtained shape features are then input into the feature point detection network to predict the feature point positions of the image, and the feature point positions are used to supervise the process of decoupling the input picture into shape features; The obtained texture features are then input into the texture decoder network to generate a texture image, and the shape features are input into the shape decoder network to generate a deformation field.

3. The facial action unit migration method based on deformation autoencoder and decoupled exchange according to claim 1, characterized in that: The input of the feature point detection network is the shape features of the face image, and the output is the predicted feature point position. The loss function of the feature point detection network is defined as: in: and They represent the x-coordinate and y-coordinate of the ith feature point, i=1,…,n, where n is the total number of feature points on the face image; ||·||2 represents the L2 norm; d0 is the actual distance between the pupils of both eyes, and q is the actual position of the feature point.

4. The facial action unit migration method based on deformation autoencoder and decoupled exchange according to claim 1, characterized in that: The encoder network takes a face image I as input and outputs a low-dimensional vector representation of texture and shape Z = [Z T ,Z S ]; among them, Z T is the texture feature, Z S is the shape feature; the input of the texture decoder network is the texture feature Z T , the low-dimensional vector Z is transformed through the texture decoder network T Transformed into a texture image T, i.e., a neutral expression and face shape; the input of the shape decoder network is the shape feature Z S , through the low-dimensional vector Z S Modeling the deformation field W: The process of modeling the deformation field W includes the following steps: (1) Use the differential decoder to decode the shape feature Z S The local deformation information is decoded: for the position k on the texture image T, after being processed by the deformation field W, the new position becomes W(k)=(W x (k),W y (k)), W(k) indicates the index of position k on the texture image T in the deformation field W, W x (k) and W y (k) indicates the new position of position k on the texture image T in the x direction and the y direction, respectively, so as to realize the deformation of the texture image T by the deformation field W; The differential decoder is used to generate the spatial gradient of the deformation field and The absolute value of represents the displacement of consecutive pixels in the x direction. The positive or negative value indicates the displacement direction of consecutive pixels in the x direction; The absolute value of represents the displacement of consecutive pixels in the y direction. The positive and negative values ​​indicate the displacement direction of consecutive pixels in the y direction; by controlling and Avoid folding or over-stretching; among them: and Represents the horizontal and vertical components of the spatial gradient; (2) The output of the differential decoder is passed through the ReLU activation function to ensure that adjacent pixels are offset in the same direction in the x and y directions; (3) The output of the differential decoder is constructed into a smooth and regular deformation field W through the spatial data integration layer.

5. The facial action unit migration method based on deformation autoencoder and decoupled exchange according to claim 1, characterized in that: The target texture image T t and the target deformation field W t The target image I t Input the texture image and deformation field generated by the deformation autoencoder, the target texture image T t The pixel value at position k is T t (k); Target new deformation field Control the deformation of the face, Indicate the new target image The pixel value at position k in the target texture image T t The position index on the image; bilinear interpolation is used to generate a new target image in: It means that it is asymptotically equal to; The source texture image T s and the target deformation field W s Source Image I s Input the texture image and deformation field generated by the deformation self-encoder, the source texture image T s The pixel value at position k is T s (k); Source new deformation field Control the deformation of the face, Indicates source new picture The pixel value at position k in the source texture image T s The position index on the source image is generated by bilinear interpolation.

6. The facial action unit migration method based on deformation autoencoder and decoupled exchange according to claim 1, characterized in that: The deformation field W and texture image T are decoupled from the input image I through the deformation autoencoder, and the new deformation field is obtained through the deformation field converter. And reconstruct a new picture Add new picture As the input image, repeat the process once to get a new new image in: Indicates new picture The pixel value at position k on the Indicate new picture The pixel value at position k on the texture image T is indexed by the position of the texture image. Indicates that the texture image T is indexed by position Should be given The pixel value of Define the loss function L for image reconstruction rec for: Define the penalty function L of the deformation field W smooth for: Define the loss function L of the deformation field W warp for: Define the adversarial loss function L for image reconstruction adv for: Define the cycle consistency loss function L cycle for: Where: ||·||1 represents the L1 norm, ||·||2 represents the L2 norm; W0(k)=k, indicating that the index of position k on the texture image T in the identity mapping deformation field W0 is k; W(k) indicates the index of position k on the texture image T in the deformation field W; D(I) and Denotes that the discriminator D judges the input image I and the new image approximates the probability of the input image I, and -E[·] represents the expectation of the entropy of ·.

7. The facial action unit migration method based on deformation autoencoder and decoupled exchange according to claim 1, characterized in that: In step S06, by combining the losses of each network module in the deep neural network, the overall objective function of the entire deep neural network model is obtained as follows: Where: s , w , l , a and λ c To control the weights of each loss item, G includes the encoder network, texture decoder network, shape decoder network, deformation field converter and feature point detection network modules except the discriminator D.

8. A device for implementing any one of the methods of claims 1 to 7, characterized in that: It includes encoder, texture decoder, shape decoder, discriminator, deformation field converter and feature point detection network modules; The encoder network module is used to decouple the image into texture features unrelated to expression and shape features related to expression, and input them into subsequent network modules respectively; The texture decoder network module converts low-dimensional texture features into texture images, i.e., neutral expressions and face shapes, which are used for subsequent image reconstruction; The shape decoder network module uses a differential decoder to predict local deformation information, passes the output of the differential decoder through a ReLU module, and then uses a spatial integration layer to reconstruct the deformation field from the spatial gradient, and the deformation field is used for subsequent image reconstruction; The discriminator network module is used to determine whether the reconstructed image is a real image; The deformation field converter network module is used to correct the exchanged source deformation field so that it can be directly applied to the target texture image, thereby reconstructing the target image.

Images