A face deformation method and system combining manifold constraints and FB-GAN

Abstract

The present invention provides a face deformation method and system combining manifold constraints and FB‑GAN. By using a generative adversarial neural network to design an end-to-end face image splicing adversarial network FB‑GAN, the function of generating high-resolution face images during face deformation is realized. The present invention generates high-resolution face images through a face deformation algorithm, solving the problem of "distorted" or "hollow" mouths caused by the mouth changing from closed to open in traditional deformation. The present invention solves the problem of poor splicing effect caused by large color differences in face splicing problems, and improves the authenticity of generated face images.

Description

Technical Field

[0001] The present invention belongs to the technical field of face image deformation, and in particular relates to a face deformation method and system combining manifold constraints and FB-GAN. Background Art

[0002] The goal of image deformation is to deform the target into the required shape and posture, which is usually achieved based on a set of different user-controlled handles, where the control handles generally include points, lines or polygonal meshes. Scheafer et al. proposed a rigid image deformation algorithm based on Moving Least Squares (MLS) constructed based on feature points. The algorithm constructs the deformation function through a linear function (such as a rigid transformation). The rigid deformation algorithm based on MLS makes the deformation "as rigid as possible" and can better maintain the global morphology and local details of the target. The algorithm is well applicable to the deformation of rigid objects (such as changing the posture of human body movement), but for non-rigid targets with coherent motion (such as dancing candles, breathing lungs, etc.), the "as rigid as possible" deformation principle shows limitations. Since the space constructed by the radial basis function has a strong approximation ability, it can not only approximate the function generated by the anisotropic problem, but also has a relatively simple representation and calculation. Among them, Gaussian function and thin plate spline function are commonly used choices for estimating spatial deformation functions in image deformation algorithms. Ma et al. proposed a mathematical model based on moving regularized least squares, using Gaussian kernel function to construct a reproducing kernel Hilbert space and define the deformation function in it to achieve vector field interpolation, thereby obtaining a realistic and smooth image deformation model. Zhou et al. used thin plate spline kernel function to estimate the deformation vector field under the moving regularized least squares estimation model. This algorithm is one of the more advanced non-rigid image deformation algorithms. The present invention proposes a moving manifold regularized least squares model (M) based on feature point guidance. 2 RLS-TPS-FG) considers both the feature control points and deformation control points in the image, and ensures that the deformation process can always better preserve the details in the source image through the popular regularization term.

[0003] The above deformation algorithms are suitable for deforming the existing parts of the image, but are not suitable for deforming something out of nothing. For example, in the process of face deformation, the mouth of a normal face is closed, and teeth should be generated when the mouth is open, but the existing image deformation algorithms cannot create teeth from nothing.

[0004] Image stitching refers to cropping a region of a source image (usually an object) and placing it at a specified location in a target image. The goal is to convert the stitched composite image into an image that has spatial and color consistency with the target image, making it as natural as possible.

[0005] Alpha blending is to blend the pixel values ​​of the source image and the target image according to the given Alpha blending vector value. This method first decomposes the source image pixels and the target image into three components according to the RGB color channels, then multiplies the three color components of the source image and the target image with the Alpha value and the negative Alpha value respectively, adds the results according to the corresponding color components and divides them by the maximum value of Alpha to get a fused image. Although Alpha blending is much better than direct copy and paste, the contents of the source image and the target image exist in the same area, resulting in a ghosting effect.

[0006] The human visual system is very sensitive to areas with sudden gradient changes, such as the edges of images. Based on this, a lot of work has been done to achieve image fusion in the gradient domain, so that a smooth transition boundary is produced in the fused image. The earliest work, Possion image editing, proposed to reconstruct the pixels of the fused area in the target image by constraining the gradient consistency of the source image and the target image. This method is the most popular image stitching fusion method in traditional stitching fusion. First, a gradient vector field is generated based on the gradient of the synthetic image, and then the Poisson equation is used to calculate and reconstruct each pixel in the stitching area from the information in the gradient vector field, so that there is a smaller gradient difference at the boundary of the stitching area. However, the images generated by this method often contain various artifacts and appear unrealistic, and traditional gradient-based algorithms are usually based on human prior knowledge. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a face deformation method and system combining manifold constraints and FB-GAN, which are used to generate high-resolution face images during the face deformation process.

[0008] The technical solution adopted by the present invention to solve the above technical problems is: a face deformation method combining manifold constraint and FB-GAN, comprising the following steps:

[0009] S0: Build a system, including an image stitching module, a face fusion network FB-GAN, and a face deformation model connected in sequence; input the source image and the target image into the image stitching module;

[0010] S1: The image stitching module cuts the open mouth part of the target image and covers it with the closed mouth part of the source image to obtain a stitched image and replace the source image;

[0011] S2: The face fusion network FB-GAN processes the spliced ​​image through color histogram constraints, gradient constraints, and perceptual loss constraints to obtain a high-resolution fused image that is semantically consistent with the spliced ​​image as a transition image for face deformation, which is used to eliminate the boundary problem and skin color difference problem of the spliced ​​area;

[0012] S3: Face Morphing Model via Feature-Guided Mobile Manifold Regularized Least Squares Model M 2 RLS-TPS-FG transforms the face shape of the fused image into the face shape of the target image.

[0013] According to the above scheme, in step S1, the specific steps are: suppose the target image x target , source image x source , mask x mask ; Mask x mask is obtained through annotation; the target image x target is the deformed target image with the mouth open; the source image x source is the source image with the mouth closed; * represents the product of corresponding pixels, then the spliced ​​image x composite for:

[0014] x composite =x target *x mask +x source *(1-x mask ).

[0015] Furthermore, in step S2, assuming that the generator of the face fusion network is G(·), the fused image x blend for:

[0016] x blend =G(x composite ).

[0017] Furthermore, in step S2, let the adversarial loss function L adv , target content loss function L content , Perceptual loss function L perceptual , gradient loss function L gradient , color histogram loss function L histogram , let λ adv , content , perceptual , gradient , hisogram Corresponding to the weights of the above loss functions, the loss function L of the face fusion network is g for:

[0018] L g =λ adv L adv +λ content L content +λ perceptual L perceptual +λ gradient L gradient +λ hisogram L histogram .

[0019] Furthermore, in step S2,

[0020] Assume that the discriminator of the face fusion network is D(·), then the adversarial loss L adv for:

[0021] L adv =maxE x∈χ [D(x blend )-D(x source )],

[0022] By optimizing the adversarial loss, the generator G(x) tends to generate high-resolution fused images x that are difficult for the discriminator D(x) to distinguish between true and false. blend ;

[0023] Suppose the target image x target The binary mask of the lips obtained from the facial feature points is x target_lip ,||.|| represents the norm, let:

[0024] L content_lip =||x blend *x targe_lip -x target *x targe_lip ||,

[0025] To ensure high resolution fused images x blend The mouth area and the target image x target The structural information of the mouth area is consistent; suppose:

[0026] L content_face =

[0027] ||x blend *(1-x targe_lip )-x source *(1-x targe_lip )||,

[0028] To ensure high resolution fused image x blend The area except the mouth and the source image x source The structural information of the area except the mouth is consistent; then the target content loss L content for:

[0029] L content =L content_face +L content_lip ;

[0030] Suppose the function used to obtain global features is is the Conv4-1 layer feature of VGG-16, then the perceptual loss L perceptual for:

[0031]

[0032] set up Represents the extracted gradient, which is obtained by convolution of the Soble operator, then the gradient loss L gradient for:

[0033]

[0034] Assume HM(·) represents color histogram matching, then the color histogram loss L histogram for:

[0035]

[0036] Furthermore, in step S3, the specific steps are:

[0037] S31: Construct a control point extraction module to extract the fused image x blend and the target image x target The feature points on the corresponding fusion face feature map x b_feature And the target face feature map x t_feature , and use the feature points in the feature map as deformation control points;

[0038] S32: Fusion face feature map x b_feature The feature points of the target face are taken as the deformation starting point, and the target face feature map x t_feature The feature point of is taken as the deformation end point, and the mobile popular regularized least squares model M based on feature point guidance is used. 2 RLS-TPS-FG performs deformation to obtain the final deformed image x deformed .

[0039] Furthermore, in step S3, the mobile popular regularized least squares model M 2 RLS-TPS-FG places the deformation function constructed by the TPS kernel in the reproducing kernel Hilbert space to solve the closed solution; according to the properties of the TPS kernel, given a face image x, let is a set of corresponding deformation control point pairs taken from the source image and the target image, where From the source image, From the target image; let is the SIFT feature control point from the source image (M>>N); the TPS kernel K is defined by the radial basis function, and for any point p on the two-dimensional image, we get:

[0040] K(p,p i )=||pp i || 2 log(||pp i ||),

[0041] Let c i is a column vector of size (D+1); A is a matrix of size 3×3; is the homogeneous coordinate corresponding to u; then any point u in the two-dimensional face image is divided into the global affine transformation function f u (u) and the local bending function g u (u) is:

[0042]

[0043]

[0044] Assume w i (p)=||up i || -2α Represents the weight coefficient; Let φ1 represent the smooth regularization term, which is used to ensure the smoothness of the deformation result; Let φ2 represent the manifold regularization term, which is used to ensure that more intrinsic structural information in the source image is retained during the deformation process; Let λ1 be used to control the complexity of the deformation, and λ2 be used to control the intrinsic geometric structure of the image; then minimize the moving manifold regularized least squares model M 2 The error functional of RLS-TPS-FG estimates the deformation function as:

[0045]

[0046] A face deformation system combining manifold constraints and FB-GAN is characterized by comprising an image splicing module, a face fusion network, and a face deformation model connected in sequence; the image splicing module is used to cut the open mouth part in the target image and cover it to the closed mouth part of the source image to obtain a spliced ​​image and replace the source image; the face fusion network is used to process the spliced ​​image through color histogram constraints, gradient constraints, and perceptual loss constraints to obtain a high-resolution fused image with semantic consistency with the spliced ​​image; the face deformation model is used to convert the face shape of the spliced ​​image into the face shape of the target image through a feature-guided moving manifold regularized least squares model.

[0047] Furthermore, the face fusion network adopts a generative adversarial network whose network architecture includes a generator and a discriminator; the network architecture of the generator includes convolution blocks, residual blocks, and deconvolution blocks, and the convolution kernels are all 4*4 convolutions. It adopts the form of an encoder and a decoder, and the encoder and the decoder are connected by jump connections to retain image detail features; the network architecture of the discriminator includes convolution blocks, and batch normalization BN and LeakyReLU function are used after each convolution layer except the first convolution layer and the last convolution layer. The first layer adopts a convolution layer and a Leaky ReLU function, and the last layer adopts a convolution layer; the convolution kernel of each convolution layer is 4*4, the convolution step size from the first to the third layer is 2, and the convolution step size of the fourth and fifth layers is 1.

[0048] A computer storage medium stores a computer program executable by a computer processor, wherein the computer program executes a face deformation method combining manifold constraints and FB-GAN.

[0049] The beneficial effects of the present invention are:

[0050] 1. The present invention is a face deformation method and system that combines manifold constraints and FB-GAN. By using a generative adversarial neural network to design an end-to-end face image splicing adversarial network FB-GAN, the function of generating high-resolution face images during face deformation is realized.

[0051] 2. The present invention generates a high-resolution face image through a face deformation algorithm, solving the problem of a "distorted" or "hollow" mouth caused by the face mouth changing from closed to open in traditional deformation.

[0052] 3. The present invention solves the problem of poor stitching effect caused by large color differences in face stitching, and improves the authenticity of the generated face image. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 is a flow chart of an embodiment of the present invention.

[0054] Figure 2 It is a schematic diagram of the generator structure of the generative adversarial network according to an embodiment of the present invention.

[0055] Figure 3 Schematic diagram of the discriminator structure of the generative adversarial network according to an embodiment of the present invention.

[0056] Figure 4 It is a schematic diagram of the overall framework of the generative adversarial network according to an embodiment of the present invention.

[0057] Figure 5 4 is a comparison diagram of face splicing test results according to an embodiment of the present invention.

[0058] Figure 6 4 is a comparison diagram of face deformation test results according to an embodiment of the present invention. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0060] An embodiment of the present invention includes an image stitching module, a face fusion network, and a face deformation model; the image stitching module is used to cut and paste the open mouth part in the target face image onto the source face image to replace the closed mouth in the source face image; the face fusion network is used to obtain a high-resolution fused image that is semantically consistent with the stitched image through color histogram constraints, gradient constraints, and perceptual loss constraints; the face deformation model is used to transform the fused face image into the shape of the target face through a feature-guided moving popular regularized least squares model.

[0061] See also Figure 1 The face deformation method of the present invention combining manifold constraint and FB-GAN comprises the following steps:

[0062] S1: construct an image stitching module, cut and paste the open mouth part of the target face image onto the source face image to replace the closed mouth in the source face image, and obtain a stitched image;

[0063] Given a target image x target , source image x source , mask x mask , where the mask x mask It is obtained by using LabelMe annotation, the target image x target Represents the deformed target image and provides the open mouth part, the source image x source is a closed mouth image; * represents the product of corresponding pixels, and the spliced ​​image x is calculated according to the following formula composite :

[0064] x composite =x target *x mask +x source *(1-x mask ).

[0065] In an embodiment of the present invention, high-resolution face images generated by StyleGAN2 are used as training sets and test sets, including 2000 pairs of images as training data sets and 100 pairs of images as test sets; the image size in the data set is 400×400 pixels.

[0066] S2: Construct a face fusion network. Let the generator of the face fusion network be G. Through color histogram constraints, gradient constraints, and perceptual loss constraints, a high-resolution fused image x with semantic consistency with the spliced ​​image is obtained. blend for:

[0067] x blend =G(x composite ),

[0068] See also Figure 4 ,The network architecture of the entire generative adversarial network consists of a generator and a discriminator.

[0069] like Figure 2 As shown, in the implementation of the present invention, the network of the generator includes a convolution block (Conv block), a residual block (Residual block) and a deconvolution block (DeConv block), and the convolution kernels are all 4*4 convolutions. The general structure is in the form of an encoder and a decoder, and the encoder and the decoder are connected by jumps to retain more detailed features.

[0070] like Figure 3 As shown, in the implementation process of the present invention, the network of the discriminator is composed of convolution blocks (Conv block), and batch normalization (BN) and leaky ReLU are applied after each convolution except the first and last layers, while the first layer uses convolution layer and leaky ReLU, and the last layer only uses convolution layer. The convolution kernel of each convolution layer in the network is 4*4, the convolution step length of the first to third layers is 2, and the convolution step length of the fourth and fifth layers is 1.

[0071] The loss function L of the entire network g It is expressed as:

[0072] L g =λ adv L adv +λ content L content +λ perceptual L perceptual +λ gradient L gradient +λ hisogram L histogram ,

[0073] The total loss is given by the adversarial loss L adv , target content loss L content , Perceptual loss L perceptual , gradient loss L gradient and color histogram loss L histogram Composition, where λ adv , content , perceptual , gradient and λ hisogram They correspond to the weights of each loss function respectively.

[0074] Let the discriminator be D(·), then the adversarial loss L adv It is expressed as:

[0075] L adv =maxE x∈χ [D(x blend )-D(x source )],

[0076] By optimizing the adversarial loss, the generator G(x) tends to generate a result image that is difficult for the discriminator to distinguish between true and false, that is, a high-resolution fused image x blend .

[0077] Target content loss L content It is expressed as:

[0078] L content =L content_face +L content_lip ,

[0079] in,

[0080] L content_face =

[0081] ||x blend *(1-x target_lip )-x source *(1-x target_lip )||,

[0082] To ensure the generated face image x blend Remove the mouth area and source image x source Except for the mouth area, the structural information is consistent;

[0083] L content_lip =||x blend *x targe_lip -x target *x targe_lip ||,

[0084] Used to ensure the generation of face image x blendThe mouth area of ​​the target image is consistent with the mouth area of ​​the target image in terms of structural information. target_lip It is the mask of the lips obtained from 68 feature points of the face.

[0085] Perceptual loss L perceptual It is expressed as:

[0086]

[0087] in The function is used to obtain the global features, which are the Conv4-1 layer features of VGG-16, x target_lip A binary mask representing the lips of the target image.

[0088] Gradient loss L gradient It is expressed as:

[0089]

[0090] in Represents the extracted gradient. In this paper, the gradient is obtained by convolution with the Soble operator.

[0091] Assume HM(·) represents color histogram matching, then the color histogram loss L histogram It is expressed as:

[0092]

[0093] In the embodiment of the present invention, 100 epochs are trained, wherein the batch size is 1.

[0094] S3: Constructing a face deformation model through feature-guided mobile popular regularized least squares model M 2 RLS-TPS-FG transforms the fused face image to the shape of the target face;

[0095] Mobile popular regularized least squares model M 2 RLS-TPS-FG places the deformation function constructed by the TPS kernel in the reproducing kernel Hilbert space and solves its closed solution. For any point u in a two-dimensional face image, according to the properties of TPS, it can be divided into a global affine transformation and a local bending function, the specific form is:

[0096]

[0097]

[0098] Where A is a matrix of size 3×3; is the homogeneous coordinate corresponding to u; c iis a column vector of size (D+1), and the TPS kernel K is defined by the radial basis function:

[0099] K(p,p i )=||pp i || 2 log(||pp i ||);

[0100] Minimize M 2 Error functional of the RLS-TPS-FG model, estimating the deformation function:

[0101]

[0102] where w i (p)=||up i || -2α represents the weight coefficient; φ1 represents the smooth regularization term, which ensures the smoothness of the deformation result; φ2 represents the manifold regularization term, which ensures that more intrinsic structural information in the source image can be retained during the deformation process; λ1 and λ2 control the deformation complexity and the intrinsic geometric structure of the image respectively.

[0103] The overall framework of the deformation task is as follows Figure 4 As shown, first according to the formula:

[0104] x composite =x target *x mask +x source *(1-x mask )

[0105] Calculate the stitched image x composite , where * represents the product of corresponding pixels. Next, in order to eliminate the spliced ​​image x composite In order to solve the boundary problem and skin color difference problem of the splicing area, the FB-GAN proposed in this paper is used to obtain the fused image x blend As the transition image of face deformation. FB-GAN solves the boundary problem by optimizing gradient loss, constrains color histogram loss to adjust the consistency of skin color, and optimizes image details by perceptual loss. So far, the first step of the algorithm is over.

[0106] After obtaining the fused image, extract the fused image x blend Face feature map x b_feature and the target image x target Face feature map x t_featureThe control point extraction module is constructed to extract 68 feature points on the spliced ​​face image and the target face image respectively. The 68 feature marker points in the feature map are regarded as deformation control points. The feature points of the spliced ​​face are used as the deformation starting point, and the 68 feature points of the target face are used as the deformation end point. The moving manifold regularized least squares method (M 2 RLS-TPS-FG) to get the final deformed image x deformed .

[0107] In the embodiment of the present invention, six parameters are involved in the deformation algorithm: parameter α determines the weight of each control point; parameter λ 11 and λ 12 Controls the manifold regularization term; parameter λ2 controls the smooth regularization term; parameter ∈ constructs the graph Laplacian matrix and calculates the weights of each edge in the graph; parameter T determines the number of points used for sparse approximation. All parameters are set as follows: α = 1, λ 11 =0.0015,λ 12 =0.15, λ2=1000, ∈=0.05, T=15.

[0108] To prove the effectiveness of the face fusion network proposed in this paper, the FB-GAN proposed in this paper is compared with the most commonly used Poisson Image Editing (PB) in traditional methods. The experimental results are as follows: Figure 5 shown. Figure 5 From left to right, each column represents the stitched image ( Figure 5 (a)), PB results ( Figure 5 (b)), FB-GAN fusion results ( Figure 5 (c)). It can be seen that the result produced by "copy and paste" has obvious artificial boundaries, and Poisson image editing can produce smooth boundaries, but it does not guarantee the consistency of skin color. In contrast, it can be seen that the FB-GAN proposed in this paper can produce consistent skin color, and the image is natural and realistic.

[0109] Figure 6 The experimental results of face deformation are shown. From left to right, each column represents the source face image ( Figure 6 (a)), deformed target face image (b), spliced ​​synthetic image (c), PB fusion result (d), FB-GAN fusion result (e), M 2 RLS-TPS-FG deformation results (f), PB+M 2 RLS-TPS-FG deformation results (g) and FB-GAN+M 2 RLS-TPS-FG deformation results (h).

[0110] To demonstrate the effectiveness of the two-step strategy proposed in this paper, we use only M 2 Deformation results of RLS-TPS-FG algorithm and PB+M 2 The results of RLS-TPS-FG were used as a comparative experiment. Figure 6 (c) It can be observed that there is a clear color difference in the mouth, which shows obvious signs of forgery. Although the PB fusion image has a significant improvement on this problem, when the color difference is large, it can still be observed that there is a clear boundary, such as Figure 6 As shown in (d), the fused image generated by the FB-GAN model is more realistic and natural. Figure 6 (e) The comparison of these two sets of experiments shows that the FB-GAN proposed in this paper can convert the face splicing synthetic image into a high-quality face image, and the image has the same face structure as the splicing synthetic image.

[0111] observe Figure 6 (f) It can be seen that only through the traditional deformation algorithm M 2 The facial contour lines generated by RLS-TPS-FG are smooth and natural, but some mouth deformation results show obvious "distortion" or "hollowness". There are also some mouths that appear normal but have no teeth, which leads to inconsistency with the target image.

[0112] observe Figure 6 (f) Figure 6 (g) with Figure 6 (h) It can be seen that PB+M 2 RLS-TPS-FG and FB-GAN+M 2 RLS-TPS-FG can make the deformed mouth consistent with the mouth structure of the target image, and the facial contour is smooth and natural. However, in comparison, the FB-GAN+M 2 RLS-TPS-FG ensures that the structure is similar while making the skin color around the mouth relatively harmonious. Figure 6 (f) Figure 6 (g) and Figure 6 (h), indicating the effectiveness of the deformation method based on the two-step strategy proposed in this paper; and the two groups of results only show differences in the mouth deformation results, and also show that the M proposed in the previous work 2 The RLS-TPS-FG algorithm is very stable.

[0113] It should be pointed out that, according to the needs of implementation, the various steps / components described in this application can be split into more steps / components, and two or more steps / components or partial operations of steps / components can be combined into new steps / components to achieve the purpose of the present invention.

[0114] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A face deformation method combining manifold constraint and FB-GAN, characterized by: The following steps are involved: S0: Build a system, including an image stitching module, a face fusion network FB-GAN, and a face deformation model connected in sequence; input the source image and the target image into the image stitching module; S1: The image stitching module cuts the open mouth part of the target image and covers it with the closed mouth part of the source image to obtain a stitched image and replace the source image; the specific steps are: Let the target image x target , source image x source , mask x mask ; Mask x mask is obtained through annotation; the target image x target is the deformed target image with the mouth open; the source image x source is the source image with the mouth closed; * represents the product of corresponding pixels, then the spliced ​​image x composite for: x composite =x target *x mask +x source *(1-x mask ); S2: The face fusion network FB-GAN processes the spliced ​​image through color histogram constraints, gradient constraints, and perceptual loss constraints to obtain a high-resolution fused image that is semantically consistent with the spliced ​​image as a transition image for face deformation, which is used to eliminate the boundary problem and skin color difference problem of the spliced ​​area; Assume that the generator of the face fusion network is G(·), then the fused image x blend for: x blend =G(x composite ); Assume the adversarial loss function L adv , target content loss function L content , Perceptual loss function L perceptual , gradient loss function L gradient , color histogram loss function L histogram , let λ adv , content , perceptual , gradient , hisogram Corresponding to the weights of the above loss functions, the loss function L of the face fusion network is g for: L g =λ adv l adv +λ content L content +λ perceptual L perceptual +λ gradient L gradient +λ hisogram L histogram ; S3: Face Morphing Model via Feature-Guided Mobile Manifold Regularized Least Squares Model M 2 RLS-TPS-FG converts the face shape of the fused image into the face shape of the target image; the specific steps are: S31: Construct a control point extraction module to extract the fused image x blend and the target image x target The feature points on the corresponding fusion face feature map x b_feature And the target face feature map x t_feature , and use the feature points in the feature map as deformation control points; S32: Fusion face feature map x b_feature The feature points of the target face are taken as the deformation starting point, and the target face feature map x t_feature The feature point of is taken as the deformation end point, and the mobile popular regularized least squares model M based on feature point guidance is used. 2 RLS-TPS-FG performs deformation to obtain the final deformed image x deformed ; Mobile popular regularized least squares model M 2 RLS-TPS-FG places the deformation function constructed by the TPS kernel in the reproducing kernel Hilbert space to solve the closed solution; according to the properties of the TPS kernel, given a face image x, let is a set of corresponding deformation control point pairs taken from the source image and the target image, where From the source image, From the target image; let is the SIFT feature control point from the source image (M>>N); the TPS kernel K is defined by the radial basis function, and for any point p on the two-dimensional image, we get: K(p,p i )=||p-p i || 2 log(||p-p i ||), Let c i is a column vector of size (D+1); A is a matrix of size 3×3; is the homogeneous coordinate corresponding to u; then any point u in the two-dimensional face image is divided into the global affine transformation function f u (u) and the local bending function g u (u) is: Assume w i (p)=||up i || -2α Represents the weight coefficient; Let φ1 represent the smooth regularization term, which is used to ensure the smoothness of the deformation result; Let φ2 represent the manifold regularization term, which is used to ensure that more intrinsic structural information in the source image is retained during the deformation process; Let λ1 be used to control the complexity of the deformation, and λ2 be used to control the intrinsic geometric structure of the image; then minimize the moving manifold regularized least squares model M 2 The error functional of RLS-TPS-FG estimates the deformation function as:

2. The face deformation method combining manifold constraint and FB-GAN according to claim 1, characterized in that: In the step S2, Assume that the discriminator of the face fusion network is D(·), then the adversarial loss L adv for: L adv =maxE x∈χ [D(x blend )-D(x source )], By optimizing the adversarial loss, the generator G(x) tends to generate high-resolution fused images x that are difficult for the discriminator D(x) to distinguish between true and false. blend ; Suppose the target image x target The binary mask of the lips obtained from the facial feature points is x target_lip ,||.|| represents the norm, let: L content_lip =||x blend *x targe_lip -x target *x targe_lip ||, To ensure high resolution fused images x blend The mouth area and the target image x target The structural information of the mouth area is consistent; set up: L content_face =||x blend *(1-x target_lip )-x source *(1-x target_lip )||, To ensure high resolution fused images x blend The area except the mouth and the source image x source The structural information of the area except the mouth is consistent; then the target content loss L content for: L content =L content_face +L content_lip ; Suppose the function used to obtain global features is is the Conv4-1 layer feature of VGG-16, then the perceptual loss L perceptual for: set up Represents the extracted gradient, which is obtained by convolution of the Soble operator, then the gradient loss L gradient for: Assume HM(·) represents color histogram matching, then the color histogram loss L histogram for:

3. A system for the face deformation method of combining manifold constraint and FB-GAN as claimed in any one of claims 1 to 2, characterized in that: It includes an image stitching module, a face fusion network, and a face deformation model connected in sequence; The image stitching module is used to cut the open mouth part of the target image and overlay it to the closed mouth part of the source image to obtain a stitched image and replace the source image; The face fusion network is used to process the spliced ​​image through color histogram constraints, gradient constraints, and perceptual loss constraints to obtain a high-resolution fused image that is semantically consistent with the spliced ​​image; The face deformation model is used to transform the face shape of the spliced ​​image to the face shape of the target image through a feature-guided moving manifold regularized least squares model.

4. The system according to claim 3, characterized in that: The face fusion network adopts a generative adversarial network with a network architecture including a generator and a discriminator; The network architecture of the generator includes convolution blocks, residual blocks, and deconvolution blocks. The convolution kernels are all 4*4 convolutions. It adopts the form of encoder and decoder. The encoder and decoder are connected through jump connections to retain image detail features. The network architecture of the discriminator includes convolution blocks. Batch normalization BN and Leaky ReLU function are used after each convolution layer except the first and last convolution layers. The first layer uses convolution layer and Leaky ReLU function, and the last layer uses convolution layer. The convolution kernel of each convolution layer is 4*4, the convolution step size from the first to the third layer is 2, and the convolution step size of the fourth and fifth layers is 1.

5. A computer storage medium, characterized in that: A computer program executable by a computer processor is stored therein, and the computer program executes a face deformation method combining manifold constraints and FB-GAN as described in any one of claims 1 to 2.

Images