Three-dimensional semantic noise driven face synthesis manipulation method and device

Abstract

The present invention provides a method and device for face synthesis and manipulation driven by three-dimensional semantic noise, wherein the method comprises: determining the semantics of a face to be processed; generating three-dimensional semantic noise corresponding to the semantics of the face to be processed; wherein the three-dimensional semantic noises corresponding to different semantics of the face are independent of each other; using a face synthesis and manipulation model, based on the three-dimensional semantic noise corresponding to the semantics of the face to be processed, generating a face synthesis image; or, based on the three-dimensional semantic noise corresponding to the semantics of the face to be processed, manipulating the semantics of the face to be processed in the manipulated face image to obtain a face manipulation image. The method and device for face synthesis and manipulation driven by three-dimensional semantic noise provided by the present invention can not only manipulate a certain semantics alone, but also generate images of various styles by resampling noise, thereby optimizing the effect of face synthesis and manipulation.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a three-dimensional semantic noise driven face synthesis manipulation method and device. Background Art

[0002] Ideal face portrait synthesis and manipulation refers to the task of synthesizing a real image with consistent semantic information based on a given semantic label map (face parsing map), and being able to perform diversified editing of the semantic content. Common face semantics include skin, eyes, and hair.

[0003] Facial portrait synthesis and manipulation have application scenarios and a broad market in many industries and enterprises. As an important component of "AI content generation" technology, it has gradually become a research hotspot in academia and industry. Facial portrait synthesis and manipulation technology is being widely used in many fields such as film and television, social networking, entertainment, e-commerce, scientific research, and many innovative application forms have emerged. In film and television production, this technology can be used to "one-click" for film and television actors to perform skin color lighting, hair color editing, and facial age manipulation, eliminating the need for complicated makeup and makeup changes; in entertainment applications, it can be used to create interactive experiences such as facial special effects, face-changing videos, and face animations; in social communications, in order to prevent personal identity privacy leakage, this technology can be used to create new facial images that are comparable to real faces or hide facial identity features; in advertising, the faces and virtual images synthesized using this technology can replace models, eliminating the traditional authorization to use other people's portraits; in scientific research, this technology can be used to generate image training models that are no different from real faces, solving problems such as insufficient data and face portrait privacy protection.

[0004] In recent years, with the development and application of generative adversarial networks in the field of image generation, significant progress has been made in the synthesis and manipulation of facial portraits. However, the current facial portrait synthesis and manipulation technology still faces two limitations. First, existing methods can often only synthesize real images based on semantic label maps, and cannot manipulate and edit a certain semantic. When artifacts appear in a certain semantic of the synthesized face or when you want to diversify the semantic style, existing methods are difficult to solve. Second, when the existing technology performs semantic editing on real images, it cannot well retain the information that does not need to edit the semantics, and it can only learn new semantic styles from reference images, making it difficult to achieve semantic style diversification. How to achieve semantically controllable and diverse-style facial portrait synthesis and manipulation is still a difficult problem that needs to be solved urgently. Summary of the invention

[0005] The present invention provides a three-dimensional semantic noise driven face synthesis manipulation method and device, which are used to solve the defects in the prior art that a certain semantic cannot be manipulated and edited, and the semantic style is difficult to diversify.

[0006] The present invention provides a three-dimensional semantic noise driven face synthesis manipulation method, comprising:

[0007] Determine the semantics of the face to be processed;

[0008] Generating three-dimensional semantic noise corresponding to the semantics of the face to be processed; wherein the three-dimensional semantic noises corresponding to different semantics of faces are independent of each other;

[0009] A face synthesis manipulation model is used to generate a face synthesis image based on the three-dimensional semantic noise corresponding to the face semantics to be processed; or, based on the three-dimensional semantic noise corresponding to the face semantics to be processed, the face semantics to be processed in the manipulated face image is manipulated to obtain a face manipulated image.

[0010] According to the present invention, a three-dimensional semantic noise driven face synthesis manipulation method is provided, wherein the face semantics to be processed in the manipulated face image is manipulated based on the three-dimensional semantic noise corresponding to the face semantics to be processed to obtain a face manipulated image, specifically comprising:

[0011] Based on the encoder in the face synthesis manipulation model, feature extraction is performed on the face image to be manipulated to obtain image features of the face image to be manipulated;

[0012] Based on the feature editor in the face synthesis manipulation model, the three-dimensional semantic noise corresponding to the semantics of the face to be processed is exchanged with the image feature coding of the semantics of the face to be processed in the image features to obtain the edited image features of the face image to be manipulated;

[0013] Based on the manipulated image generator in the face synthesis manipulation model, the manipulated face image is generated according to the edited image features.

[0014] According to a 3D semantic noise driven face synthesis and manipulation method provided by the present invention, the face synthesis and manipulation model is obtained by jointly training a face synthesis network, a face manipulation network and a face reconstruction network;

[0015] The face synthesis network includes a synthetic image generator; the synthetic image generator is used to perform face synthesis based on three-dimensional semantic noise corresponding to the semantics of the face;

[0016] The face manipulation network includes the feature editor and the manipulated image generator;

[0017] The face reconstruction network includes the encoder and a reconstructed image generator; the reconstructed image generator is used to perform image reconstruction based on the output result of the encoder;

[0018] The synthetic image generator, the manipulated image generator and the reconstructed image generator share parameters.

[0019] According to a 3D semantic noise driven face synthesis and manipulation method provided by the present invention, the loss function of the face synthesis and manipulation model includes style diversity loss, appearance matching loss, multi-class segmentation loss, reconstruction loss and adversarial generation loss;

[0020] The style diversity loss is used to maximize the difference between images synthesized by the synthetic image generator based on different three-dimensional semantic noises corresponding to the same face semantics;

[0021] The appearance matching loss is used to minimize the difference between the manipulated sub-image and the synthesized sub-image corresponding to the manipulated area, and minimize the difference between the manipulated sub-image and the reconstructed sub-image corresponding to the retained area; the manipulated sub-image and the synthesized sub-image corresponding to the manipulated area are respectively generated by the manipulated image generator and the synthesized image generator based on the same three-dimensional semantic noise corresponding to the manipulated area;

[0022] Multi-class segmentation loss is used to decouple the image feature encoding of the encoder for different facial semantic outputs;

[0023] The reconstruction loss is used to minimize the difference between the image generated by the reconstructed image generator and the input original image;

[0024] The adversarial generation loss is used to maximize the realism of the images generated by the synthetic image generator, the manipulated image generator, and the reconstructed image generator.

[0025] According to a 3D semantic noise driven face synthesis manipulation method provided by the present invention, the multi-class segmentation loss is specifically:

[0026]

[0027] in, and They respectively represent the semantic segmentation result and semantic label of the pixel at position (h, w) in the input image. The semantic segmentation result is obtained by semantic segmentation based on the image features of the input image. h and w represent the height and width of the semantic label map. P = {0, 1, ..., K-1} represents the semantic category.

[0028] According to a 3D semantic noise driven face synthesis manipulation method provided by the present invention, the edited image features of the face image to be manipulated are specifically:

[0029] F m =(1-M k )*F r +M k *N3D ;

[0030] Among them, F r and N 3D represent the image features of the face image to be manipulated and the three-dimensional semantic noise corresponding to all face semantics, M k A binary mask image representing the semantic k of the face to be processed.

[0031] According to a 3D semantic noise driven face synthesis manipulation method provided by the present invention, generating 3D semantic noise corresponding to the semantics of the face to be processed specifically includes:

[0032] Sampling multi-channel noise corresponding to the semantics of the face to be processed from a Gaussian distribution;

[0033] Based on the semantic label map, the noise elements of each channel in the multi-channel noise corresponding to the semantics of the face to be processed are spatially expanded to the shape and size of the corresponding area in the form of replication, so as to obtain the three-dimensional semantic noise corresponding to the semantics of the face to be processed.

[0034] The present invention also provides a three-dimensional semantic noise driven face synthesis manipulation device, comprising:

[0035] A face semantics determination unit, used to determine the semantics of a face to be processed;

[0036] A semantic noise generating unit, used for generating three-dimensional semantic noise corresponding to the semantics of the face to be processed; wherein the three-dimensional semantic noises corresponding to different semantics of faces are independent of each other;

[0037] A face synthesis and manipulation unit is used to generate a face synthesis image based on the three-dimensional semantic noise corresponding to the face semantics to be processed by using a face synthesis and manipulation model; or to manipulate the face semantics to be processed in the manipulated face image based on the three-dimensional semantic noise corresponding to the face semantics to be processed to obtain a face manipulation image.

[0038] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned three-dimensional semantic noise-driven face synthesis manipulation methods are implemented.

[0039] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described three-dimensional semantic noise-driven face synthesis manipulation methods.

[0040] The 3D semantic noise driven face synthesis and manipulation method and device provided by the present invention generate 3D semantic noise corresponding to the semantics of a face to be processed, wherein the 3D semantic noises corresponding to different semantics of a face are independent of each other, and then generate a face synthesis image based on the 3D semantic noise corresponding to the semantics of the face to be processed; or, based on the 3D semantic noise corresponding to the semantics of the face to be processed, manipulate the semantics of the face to be processed in the manipulated face image to obtain a face manipulation image, which not only retains the semantic information in the semantic label map to achieve the controllability of the semantics, so that a certain semantics can be manipulated separately, but also can generate diversified styles by resampling noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0042] Figure 1 A schematic diagram of the flow of the 3D semantic noise driven face synthesis and manipulation method provided by the present invention;

[0043] Figure 2 A schematic diagram of a training method for a face synthesis manipulation model provided by the present invention;

[0044] Figure 3 A schematic diagram of the structure of the 3D semantic noise driven face synthesis manipulation device provided by the present invention;

[0045] Figure 4 This is a schematic structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Figure 1 A schematic diagram of a flow chart of a 3D semantic noise driven face synthesis manipulation method provided by an embodiment of the present invention, such as Figure 1 As shown, the method includes:

[0048] Step 110, determining the semantics of the face to be processed.

[0049] Specifically, first determine the face semantics that need to be synthesized or manipulated. The face semantics include different parts of the face, such as eyes, nose, skin, etc. When face synthesis is required, the face semantics to be processed may include all face semantics, and when face manipulation is required, the face semantics to be processed may include one or more face semantics that need to be manipulated separately.

[0050] Step 120, generating three-dimensional semantic noise corresponding to the semantics of the face to be processed; wherein the three-dimensional semantic noises corresponding to different semantics of faces are independent of each other.

[0051] Specifically, the corresponding three-dimensional semantic noise is generated for the semantics of the face to be processed. It should be noted that the three-dimensional semantic noises corresponding to different facial semantics are independent of each other, so that the semantic information in the semantic label map is retained to achieve the controllability of the semantics, making it possible to manipulate a certain semantics separately, and a variety of styles can be generated by resampling the noise.

[0052] Step 130, using the face synthesis manipulation model, based on the three-dimensional semantic noise corresponding to the semantics of the face to be processed, a face synthesis image is generated; or, based on the three-dimensional semantic noise corresponding to the semantics of the face to be processed, the semantics of the face to be processed in the manipulated face image is manipulated to obtain a face manipulated image.

[0053] Specifically, the face synthesis and face manipulation model can be used to simultaneously achieve the two tasks of face synthesis and face manipulation. Specifically, based on the three-dimensional semantic noise corresponding to the semantics of the face to be processed, combined with the semantic label map of the face, a face synthesis image can be generated; or, based on the three-dimensional semantic noise corresponding to the semantics of the face to be processed, the semantics of the face to be processed in the manipulated face image can be manipulated to obtain a face manipulation image.

[0054] The method provided by the embodiment of the present invention generates three-dimensional semantic noise corresponding to the semantics of a face to be processed, wherein the three-dimensional semantic noises corresponding to different semantics of a face are independent of each other, and then generates a synthetic face image based on the three-dimensional semantic noise corresponding to the semantics of the face to be processed; or, based on the three-dimensional semantic noise corresponding to the semantics of the face to be processed, manipulates the semantics of the face to be processed in the manipulated face image to obtain a manipulated face image, which not only retains the semantic information in the semantic label map to achieve the controllability of the semantics, so that a certain semantics can be manipulated alone, but also can generate diversified styles by resampling noise.

[0055] Based on the above embodiment, based on the three-dimensional semantic noise corresponding to the semantics of the face to be processed, the semantics of the face to be processed in the manipulated face image is manipulated to obtain the manipulated face image, specifically including:

[0056] Based on the encoder in the face synthesis manipulation model, feature extraction is performed on the face image to be manipulated to obtain image features of the face image to be manipulated;

[0057] Based on the feature editor in the face synthesis manipulation model, the three-dimensional semantic noise corresponding to the semantics of the face to be processed is exchanged with the image feature coding of the semantics of the face to be processed in the image feature to obtain the edited image features of the face image to be manipulated;

[0058] Based on the manipulated image generator in the face synthesis manipulation model, a manipulated face image is generated according to the edited image features.

[0059] Specifically, the encoder in the face synthesis manipulation model can be used to extract features of the face image to be manipulated, and obtain the image features of the face image to be manipulated. The image features contain relevant information about the semantics of each face in the face image to be manipulated. Subsequently, based on the feature editor in the face synthesis manipulation model, the three-dimensional semantic noise corresponding to the semantics of the face to be processed is exchanged with the image feature encoding of the semantics of the face to be processed in the image features, so as to realize feature editing of the semantics of the face to be processed and obtain the edited image features of the face image to be manipulated. Then, based on the manipulated image generator in the face synthesis manipulation model, a face manipulation image can be generated according to the edited image features. In the generated face manipulation image, the image of the semantic part of the face to be processed has changed, but since only the features of the semantic part of the face to be processed have been edited, the image information consistent with the face image to be manipulated can be retained for the unmanipulated area, thereby retaining the content information that does not need to edit the semantics as much as possible, and achieving a natural and realistic manipulation effect.

[0060] Based on any of the above embodiments, the face synthesis and manipulation model is obtained by jointly training a face synthesis network, a face manipulation network and a face reconstruction network;

[0061] The face synthesis network includes a synthetic image generator; the synthetic image generator is used to synthesize faces based on three-dimensional semantic noise corresponding to the semantics of faces;

[0062] The face manipulation network contains a feature editor and a manipulated image generator;

[0063] The face reconstruction network includes an encoder and a reconstruction image generator; the reconstruction image generator is used to reconstruct an image based on the output result of the encoder;

[0064] The synthetic image generator, manipulated image generator, and reconstructed image generator share parameters.

[0065] Specifically, the face synthesis and manipulation model can be obtained by joint training of a ternary network consisting of a face synthesis network, a face manipulation network and a face reconstruction network. The face synthesis network includes a synthetic image generator, which is used to synthesize faces based on three-dimensional semantic noise corresponding to the semantics of the face; the face reconstruction network includes an encoder and a reconstructed image generator, the encoder is used to extract features from the input image, and the reconstructed image generator is used to reconstruct the image based on the output of the encoder; the face manipulation network includes a feature editor and a manipulated image generator, the feature editor is used to edit the output of the encoder for some semantics, and the manipulated image generator is used to generate a manipulated image based on the edited image features.

[0066] It should be noted that the synthetic image generator, manipulated image generator, and reconstructed image generator share parameters. Therefore, when jointly training the three networks, multi-task training can optimize the performance of the editor, synthetic image generator, and manipulated image generator, thereby improving the effects of face synthesis and face manipulation.

[0067] Based on any of the above embodiments, the loss function of the face synthesis manipulation model includes style diversity loss, appearance matching loss, multi-class segmentation loss, reconstruction loss and adversarial generation loss;

[0068] Among them, the style diversity loss is used to maximize the difference between the images synthesized by the synthetic image generator based on different 3D semantic noises corresponding to the same face semantics;

[0069] The appearance matching loss is used to minimize the difference between the manipulated sub-image and the synthesized sub-image corresponding to the manipulated area, and minimize the difference between the manipulated sub-image and the reconstructed sub-image corresponding to the retained area; the manipulated sub-image and the synthesized sub-image corresponding to the manipulated area are generated by the manipulated image generator and the synthesized image generator respectively based on the same three-dimensional semantic noise corresponding to the manipulated area;

[0070] Multi-class segmentation loss is used to decouple the encoder’s image feature encoding for different facial semantic outputs;

[0071] The reconstruction loss is used to minimize the difference between the image generated by the reconstructed image generator and the input original image;

[0072] The adversarial generation loss is used to maximize the realism of images generated by the synthetic image generator, the manipulated image generator, and the reconstructed image generator.

[0073] Specifically, the loss function of the face synthesis manipulation model can include style diversity loss, appearance matching loss, multi-class segmentation loss, reconstruction loss and adversarial generation loss.

[0074] Among them, the style diversity loss is used to maximize the difference between images synthesized by the synthetic image generator based on different three-dimensional semantic noises corresponding to the same facial semantics, so that the same facial semantics are resampled to obtain new three-dimensional semantic noises. The synthesized image has a large change in style, achieving style diversification.

[0075] The appearance matching loss is used to minimize the difference between the manipulated sub-image and the synthesized sub-image corresponding to the manipulated area, and minimize the difference between the manipulated sub-image and the reconstructed sub-image corresponding to the retained area; the manipulated sub-image and the synthesized sub-image corresponding to the manipulated area are generated by the manipulated image generator and the synthesized image generator respectively based on the same three-dimensional semantic noise corresponding to the manipulated area. Here, the manipulated area is the edited facial semantics, and the retained area is the other unedited facial semantics. The appearance matching loss minimizes the difference between the manipulated sub-image and the synthesized sub-image corresponding to the manipulated area, which can make the sub-images generated by the manipulated image generator and the synthesized image generator based on the same three-dimensional semantic noise as similar as possible; minimizes the difference between the manipulated sub-image and the reconstructed sub-image corresponding to the retained area, which can make the image information of the unedited semantics be retained as much as possible when manipulating the facial image, thereby optimizing the effect of facial manipulation.

[0076] The multi-class segmentation loss is used to decouple the image feature encoding of the encoder for different facial semantics outputs, thereby ensuring that after the image feature encoding of a certain facial semantics is exchanged with its corresponding three-dimensional semantic noise, it will only affect the image information of the facial semantics, and will not affect the image information of other facial semantics, thereby achieving independent manipulation of facial semantics.

[0077] The reconstruction loss is used to minimize the difference between the image generated by the reconstructed image generator and the input original image, optimize the encoder and the reconstructed image generator, so that the image features extracted by the encoder are accurate enough to accurately describe the characteristics of the input original image and reconstruct an image that is as similar as possible to the original image.

[0078] The adversarial generation loss is used to maximize the realism of the images generated by the synthetic image generator, the manipulated image generator, and the reconstructed image generator. Here, the discriminator can be used to distinguish the authenticity of the images generated by the synthetic image generator, the manipulated image generator, and the reconstructed image generator, and the discrimination result is fed back to the corresponding generator to guide the generation of the image, so as to prompt the corresponding generator to generate realistic face images.

[0079] Based on any of the above embodiments, the multi-class segmentation loss is specifically:

[0080]

[0081] in, and They represent the semantic segmentation result and semantic label of the pixel at position (h, w) in the input image respectively. The semantic segmentation result is obtained by semantic segmentation based on the image features of the input image. h and w represent the height and width of the semantic label map, and P = {0, 1, ..., K-1} represents the semantic category.

[0082] Specifically, a semantic segmentation branch can be added after the encoder, which performs semantic segmentation on the image features of the input image to obtain the semantic segmentation result of each pixel in the input image. Based on this, the multi-class segmentation loss can be calculated:

[0083]

[0084] in, and They represent the semantic segmentation result and semantic label of the pixel at position (h, w) in the input image respectively. The semantic segmentation result is obtained by semantic segmentation based on the image features of the input image. h and w represent the height and width of the semantic label map, and both can be 32. P = {0, 1, ..., K-1} represents the semantic category.

[0085] Here, by minimizing the multi-class segmentation loss, the encoder can be optimized so that the image features extracted by the encoder retain as much relevant information of each facial semantics as possible, avoiding the fusion of features of different facial semantics during the feature extraction process, thereby achieving feature decoupling of facial semantics.

[0086] Based on any of the above embodiments, step 120 specifically includes:

[0087] Sampling multi-channel noise corresponding to the semantics of the face to be processed from Gaussian distribution;

[0088] Based on the semantic label map, the noise elements of each channel in the multi-channel noise corresponding to the semantics of the face to be processed are spatially expanded to the shape and size of the corresponding area in the form of replication, so as to obtain the three-dimensional semantic noise corresponding to the semantics of the face to be processed.

[0089] Specifically, multi-channel noise corresponding to the semantics of the face to be processed can be randomly sampled from a Gaussian distribution, and then the noise elements of each channel in the multi-channel noise can be copied and expanded to the shape and size of the area corresponding to the semantics of the face to be processed, so as to obtain three-dimensional semantic noise consistent with the shape and size of the semantics of the face to be processed.

[0090] In addition, if face synthesis is required, the face semantics to be processed are usually all the semantics of the entire face. Therefore, multi-channel noise of the number of semantics in the semantic label map can also be sampled from the Gaussian distribution, and each semantic corresponds to a multi-channel noise. Among them, a 1×(N*C)-dimensional noise sampled from the Gaussian distribution can be directly cut into N 1×C-dimensional multi-channel noises, where N is the number of semantics, C is the number of channels, and C can be 8. Here, the above-mentioned cutting refers to cutting the 1×(N*C)-dimensional noise once every C dimensions, and cutting N-1 times to obtain N multi-channel noises. Based on the semantic label map, the noise elements of all channels in the multi-channel noise corresponding to each semantic are spatially expanded to the corresponding semantic shape size in the form of replication to form three-dimensional semantic noise of each semantic shape size; then, the three-dimensional semantic noise of all semantics is combined into a semantic noise map of the same size as the semantic label map according to the semantic position in the semantic label map for face synthesis.

[0091] On this basis, if face manipulation is required, the three-dimensional semantic noise of the face semantics to be processed can be segmented from the above semantic noise map to edit the image feature coding of the face semantics to be processed, thereby realizing the manipulation of the semantics.

[0092] Based on any of the above embodiments, the edited image features of the face image to be manipulated are specifically:

[0093] F m =(1-M k )*F r +M k *N 3D ;

[0094] Among them, F r and N 3D They represent the image features of the face image to be manipulated and the three-dimensional semantic noise corresponding to the semantics of all faces, respectively. k Represents the binary mask map of the face semantics k to be processed.

[0095] Specifically, feature editing can be achieved through the following formula:

[0096] F m =(1-M k )*F r +M k *N 3D ;

[0097] Among them, F r and N 3D They represent the image features of the face image to be manipulated and the three-dimensional semantic noise corresponding to the semantics of all faces, respectively. k Represents the binary mask map of the face semantics k to be processed.

[0098] Based on any of the above embodiments, Figure 2 A schematic diagram of a training method for a face synthesis manipulation model provided by an embodiment of the present invention, such as Figure 2 As shown, the method includes:

[0099] Input Gaussian noise, semantics to be edited, and face images;

[0100] According to the semantic label map, the noise corresponding to each semantics is expanded to the shape and size of the corresponding semantics;

[0101] Extract image features of the input face image using an encoder;

[0102] The expanded noise corresponding to each semantics is combined to form 3D semantic noise, and is input into a synthetic image generator to generate a semantic synthetic image;

[0103] The semantic features to be edited are exchanged with the corresponding noises to obtain the edited image feature expression, and the edited image feature expression is input into the manipulated image generator to generate the edited image;

[0104] After encoding the face image into feature expressions using the encoder, the feature expressions are input into the reconstructed image generator to reconstruct the input image. On the other hand, the semantic segmentation branch is used to perform semantic segmentation on the image feature expressions to calculate the multi-class segmentation loss.

[0105] A discriminator is used to judge the authenticity of semantically synthesized images, edited images, and reconstructed images, and then the model is trained based on style diversity loss, appearance matching loss, multi-class segmentation loss, reconstruction loss, and adversarial generation loss.

[0106] Based on any of the above embodiments, Figure 3 A schematic diagram of the structure of a 3D semantic noise driven face synthesis manipulation device provided by an embodiment of the present invention, as shown in FIG. Figure 3 As shown, the device includes: a face semantics determination unit 310, a semantic noise generation unit 320 and a face synthesis manipulation unit 330.

[0107] Wherein, the face semantics determination unit 310 is used to determine the face semantics to be processed;

[0108] The semantic noise generating unit 320 is used to generate three-dimensional semantic noise corresponding to the semantics of the face to be processed;

[0109] The face synthesis manipulation unit 330 is used to generate a face synthesis image based on the three-dimensional semantic noise corresponding to the face semantics to be processed by using the face synthesis manipulation model; or, based on the three-dimensional semantic noise corresponding to the face semantics to be processed, manipulate the face semantics to be processed in the manipulated face image to obtain a face manipulation image.

[0110] The device provided by the embodiment of the present invention generates three-dimensional semantic noise corresponding to the semantics of a face to be processed, wherein the three-dimensional semantic noises corresponding to different semantics of a face are independent of each other, and then generates a synthetic face image based on the three-dimensional semantic noise corresponding to the semantics of the face to be processed; or, based on the three-dimensional semantic noise corresponding to the semantics of the face to be processed, manipulates the semantics of the face to be processed in the manipulated face image to obtain a manipulated face image, which not only retains the semantic information in the semantic label map to achieve the controllability of the semantics, so that a certain semantics can be manipulated separately, but also can generate diversified styles by resampling noise.

[0111] Based on any of the above embodiments, based on the three-dimensional semantic noise corresponding to the semantics of the face to be processed, the semantics of the face to be processed in the manipulated face image is manipulated to obtain the manipulated face image, specifically including:

[0112] Based on the encoder in the face synthesis manipulation model, feature extraction is performed on the face image to be manipulated to obtain image features of the face image to be manipulated;

[0113] Based on the feature editor in the face synthesis manipulation model, the three-dimensional semantic noise corresponding to the semantics of the face to be processed is exchanged with the image feature coding of the semantics of the face to be processed in the image feature to obtain the edited image features of the face image to be manipulated;

[0114] Based on the manipulated image generator in the face synthesis manipulation model, a manipulated face image is generated according to the edited image features.

[0115] Based on any of the above embodiments, the face synthesis and manipulation model is obtained by jointly training a face synthesis network, a face manipulation network and a face reconstruction network;

[0116] The face synthesis network includes a synthetic image generator; the synthetic image generator is used to synthesize faces based on three-dimensional semantic noise corresponding to the semantics of faces;

[0117] The face manipulation network contains a feature editor and a manipulated image generator;

[0118] The face reconstruction network includes an encoder and a reconstruction image generator; the reconstruction image generator is used to reconstruct an image based on the output result of the encoder;

[0119] The synthetic image generator, manipulated image generator, and reconstructed image generator share parameters.

[0120] Based on any of the above embodiments, the loss function of the face synthesis manipulation model includes style diversity loss, appearance matching loss, multi-class segmentation loss, reconstruction loss and adversarial generation loss;

[0121] Among them, the style diversity loss is used to maximize the difference between the images synthesized by the synthetic image generator based on different 3D semantic noises corresponding to the same face semantics;

[0122] The appearance matching loss is used to minimize the difference between the manipulated sub-image and the synthesized sub-image corresponding to the manipulated area, and minimize the difference between the manipulated sub-image and the reconstructed sub-image corresponding to the retained area; the manipulated sub-image and the synthesized sub-image corresponding to the manipulated area are generated by the manipulated image generator and the synthesized image generator respectively based on the same three-dimensional semantic noise corresponding to the manipulated area;

[0123] Multi-class segmentation loss is used to decouple the encoder’s image feature encoding for different facial semantic outputs;

[0124] The reconstruction loss is used to minimize the difference between the image generated by the reconstructed image generator and the input original image;

[0125] The adversarial generation loss is used to maximize the realism of images generated by the synthetic image generator, the manipulated image generator, and the reconstructed image generator.

[0126] Based on any of the above embodiments, the multi-class segmentation loss is specifically:

[0127]

[0128] in, and They represent the semantic segmentation result and semantic label of the pixel at position (h, w) in the input image respectively. The semantic segmentation result is obtained by semantic segmentation based on the image features of the input image. h and w represent the height and width of the semantic label map, and P = {0, 1, ..., K-1} represents the semantic category.

[0129] Based on any of the above embodiments, the semantic noise generating unit 320 is specifically used for:

[0130] Sampling multi-channel noise corresponding to the semantics of the face to be processed from Gaussian distribution;

[0131] Based on the semantic label map, the noise elements of each channel in the multi-channel noise corresponding to the semantics of the face to be processed are spatially expanded to the shape and size of the corresponding area in the form of replication, so as to obtain the three-dimensional semantic noise corresponding to the semantics of the face to be processed.

[0132] Based on any of the above embodiments, the edited image features of the face image to be manipulated are specifically:

[0133] F m =(1-M k )*F r +M k *N 3D ;

[0134] Among them, F r and N 3D They represent the image features of the face image to be manipulated and the three-dimensional semantic noise corresponding to the semantics of all faces, respectively. k Represents the binary mask map of the face semantics k to be processed.

[0135] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4 As shown, the electronic device may include: a processor 410, a communication interface 420, a memory 430 and a communication bus 440, wherein the processor 410, the communication interface 420 and the memory 430 communicate with each other through the communication bus 440. The processor 410 may call the logic instructions in the memory 430 to execute the face synthesis manipulation method driven by three-dimensional semantic noise, the method comprising: determining the face semantics to be processed; generating three-dimensional semantic noise corresponding to the face semantics to be processed; wherein the three-dimensional semantic noises corresponding to different face semantics are independent of each other; using the face synthesis manipulation model, based on the three-dimensional semantic noise corresponding to the face semantics to be processed, generating a face synthesis image; or, based on the three-dimensional semantic noise corresponding to the face semantics to be processed, manipulating the face semantics to be processed in the manipulated face image to obtain a face manipulation image.

[0136] In addition, the logic instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0137] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the three-dimensional semantic noise-driven face synthesis and manipulation method provided by the above-mentioned methods, and the method includes: determining the semantics of a face to be processed; generating three-dimensional semantic noise corresponding to the semantics of the face to be processed; wherein the three-dimensional semantic noises corresponding to different facial semantics are independent of each other; using a face synthesis and manipulation model, based on the three-dimensional semantic noise corresponding to the semantics of the face to be processed, generating a face synthesis image; or, based on the three-dimensional semantic noise corresponding to the semantics of the face to be processed, manipulating the semantics of the face to be processed in the manipulated face image to obtain a face manipulation image.

[0138] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the above-mentioned three-dimensional semantic noise-driven face synthesis and manipulation methods, the method comprising: determining the semantics of a face to be processed; generating three-dimensional semantic noise corresponding to the semantics of the face to be processed; wherein the three-dimensional semantic noises corresponding to different facial semantics are independent of each other; using a face synthesis and manipulation model, based on the three-dimensional semantic noise corresponding to the semantics of the face to be processed, generating a face synthesis image; or, based on the three-dimensional semantic noise corresponding to the semantics of the face to be processed, manipulating the semantics of the face to be processed in the manipulated face image to obtain a face manipulation image.

[0139] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0140] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A 3D semantic noise driven face synthesis manipulation method, characterized in that: include: Determine the semantics of the face to be processed; Generating three-dimensional semantic noise corresponding to the semantics of the face to be processed; wherein the three-dimensional semantic noises corresponding to different semantics of faces are independent of each other; Generate a face synthesis image based on the three-dimensional semantic noise corresponding to the face semantics to be processed by using a face synthesis manipulation model; or manipulate the face semantics to be processed in the manipulated face image based on the three-dimensional semantic noise corresponding to the face semantics to be processed to obtain a face manipulation image; Based on the three-dimensional semantic noise corresponding to the semantics of the human face to be processed, the semantics of the human face to be processed in the human face image to be manipulated is manipulated to obtain a manipulated face image, specifically comprising: Based on the encoder in the face synthesis manipulation model, feature extraction is performed on the face image to be manipulated to obtain image features of the face image to be manipulated; Based on the feature editor in the face synthesis manipulation model, the three-dimensional semantic noise corresponding to the semantics of the face to be processed is exchanged with the image feature coding of the semantics of the face to be processed in the image features to obtain the edited image features of the face image to be manipulated; Based on the manipulated image generator in the face synthesis manipulation model, generating the face manipulated image according to the edited image features; The edited image features of the face image to be manipulated are specifically: F m =(1-M k )*F r +M k *N 3D ; Among them, F r and N 3D represent the image features of the face image to be manipulated and the three-dimensional semantic noise corresponding to all face semantics, M k A binary mask image representing the semantic k of the face to be processed; The generating of the three-dimensional semantic noise corresponding to the semantics of the face to be processed specifically includes: Sampling multi-channel noise corresponding to the semantics of the face to be processed from a Gaussian distribution; Based on the semantic label map, the noise elements of each channel in the multi-channel noise corresponding to the semantics of the face to be processed are spatially expanded to the shape and size of the corresponding area in the form of replication, so as to obtain the three-dimensional semantic noise corresponding to the semantics of the face to be processed.

2. The 3D semantic noise driven face synthesis manipulation method according to claim 1, characterized in that: The face synthesis and manipulation model is obtained by jointly training a face synthesis network, a face manipulation network and a face reconstruction network; The face synthesis network includes a synthetic image generator; the synthetic image generator is used to perform face synthesis based on three-dimensional semantic noise corresponding to the semantics of the face; The face manipulation network includes the feature editor and the manipulated image generator; The face reconstruction network includes the encoder and a reconstructed image generator; the reconstructed image generator is used to perform image reconstruction based on the output result of the encoder; The synthetic image generator, the manipulated image generator and the reconstructed image generator share parameters.

3. The 3D semantic noise driven face synthesis manipulation method according to claim 2, characterized in that: The loss functions of the face synthesis manipulation model include style diversity loss, appearance matching loss, multi-class segmentation loss, reconstruction loss and adversarial generation loss; The style diversity loss is used to maximize the difference between images synthesized by the synthetic image generator based on different three-dimensional semantic noises corresponding to the same face semantics; The appearance matching loss is used to minimize the difference between the manipulated sub-image and the synthesized sub-image corresponding to the manipulated area, and minimize the difference between the manipulated sub-image and the reconstructed sub-image corresponding to the retained area; the manipulated sub-image and the synthesized sub-image corresponding to the manipulated area are respectively generated by the manipulated image generator and the synthesized image generator based on the same three-dimensional semantic noise corresponding to the manipulated area; Multi-class segmentation loss is used to decouple the image feature encoding of the encoder for different facial semantic outputs; The reconstruction loss is used to minimize the difference between the image generated by the reconstructed image generator and the input original image; The adversarial generation loss is used to maximize the realism of the images generated by the synthetic image generator, the manipulated image generator, and the reconstructed image generator.

4. The 3D semantic noise driven face synthesis manipulation method according to claim 3, characterized in that: The multi-class segmentation loss is specifically: in, and They respectively represent the semantic segmentation result and semantic label of the pixel at position (h, w) in the input image. The semantic segmentation result is obtained by semantic segmentation based on the image features of the input image. h and w represent the height and width of the semantic label map. P = {0, 1, ..., K-1} represents the semantic category.

5. A 3D semantic noise driven face synthesis manipulation device, characterized in that: include: A face semantics determination unit, used to determine the semantics of a face to be processed; A semantic noise generating unit, used for generating three-dimensional semantic noise corresponding to the semantics of the face to be processed; wherein the three-dimensional semantic noises corresponding to different semantics of faces are independent of each other; A face synthesis manipulation unit is used to generate a face synthesis image based on the three-dimensional semantic noise corresponding to the face semantics to be processed by using the face synthesis manipulation model; or to manipulate the face semantics to be processed in the face image to be manipulated based on the three-dimensional semantic noise corresponding to the face semantics to be processed to obtain a face manipulation image; The face synthesis manipulation unit is used for: Based on the encoder in the face synthesis manipulation model, feature extraction is performed on the face image to be manipulated to obtain image features of the face image to be manipulated; Based on the feature editor in the face synthesis manipulation model, the three-dimensional semantic noise corresponding to the semantics of the face to be processed is exchanged with the image feature coding of the semantics of the face to be processed in the image features to obtain the edited image features of the face image to be manipulated; Based on the manipulated image generator in the face synthesis manipulation model, generating the face manipulated image according to the edited image features; The edited image features of the face image to be manipulated are specifically: F m =(1-M k )*F r +M k *N 3D ; Among them, F r and N 3D represent the image features of the face image to be manipulated and the three-dimensional semantic noise corresponding to all face semantics, M k A binary mask image representing the semantic k of the face to be processed; The semantic noise generating unit is used to: Sampling multi-channel noise corresponding to the semantics of the face to be processed from a Gaussian distribution; Based on the semantic label map, the noise elements of each channel in the multi-channel noise corresponding to the semantics of the face to be processed are spatially expanded to the shape and size of the corresponding area in the form of replication, so as to obtain the three-dimensional semantic noise corresponding to the semantics of the face to be processed.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the three-dimensional semantic noise driven face synthesis manipulation method as described in any one of claims 1 to 4 are implemented.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the three-dimensional semantic noise driven face synthesis manipulation method as described in any one of claims 1 to 4 are implemented.

Images