Face image editing method and device capable of keeping specific attributes, medium and equipment

Abstract

The invention relates to the technical field of face generation, in particular to a face image editing method and device capable of reserving specific attributes, a medium and equipment. The method comprises the following steps: acquiring original local semantic information and original rendering information corresponding to an original face image; obtaining a middle face feature image and a first loss value according to a face editing model; inputting the middle face feature image into a face analysis model to obtain a second loss value; and inputting the original face image into a face editing model, and controlling the face editing model to edit and obtain a target face image under the constraint of the first loss value and the second loss value. According to the method, the generation of the model is constrained by setting the original local semantic information and the original rendering information, so that the finally obtained face image retains the face attribute corresponding to the original local semantic information, and in addition, the face editing model is jointly constrained according to the first loss function and the second loss function, so that the user experience is improved. And the control capability of the face generation process is improved.

Description

Background Art

[0002] With the introduction of the Denoising Diffusion Probabilistic Model (DDPM), more and more researchers have paid attention to the powerful generation capabilities of the diffusion model. Compared with Generative Adversarial Networks (GAN), the diffusion model is not limited to the original image, the images it generates have better diversity, and the training is relatively stable, and there is no need to train two networks at the same time like the GAN. Therefore, the mainstream model of face attribute editing has gradually transitioned from StyleGAN based on the GAN to diffusion model-based diffusion face editing models such as StableDiffusion.

[0003] At present, the task of face attribute editing refers to changing facial features under the constraint of consistent identity to generate face images of different ages, expressions, postures, etc.

[0004] However, the study found that when using face editing data for data augmentation, researchers often expect to generate face images with specified attribute features. For example, in line of sight estimation, researchers expect to generate face images with specific eye shapes for targeted training. However, through current research, it is found that existing face editing models are oriented to identity-consistent face editing tasks, and no face editing models that retain specific facial attributes have been found. Summary of the invention

[0005] The technical problem to be solved by this application is: how to retain specific facial attributes when editing a face.

[0006] In view of the above technical problems, according to a first aspect of the present application, a method for editing a face image with specific attributes retained is provided, the method comprising:

[0007] Obtaining original local semantic information and original rendering information corresponding to the original face image; wherein the original local semantic information is used to describe feature information of a preset attribute category corresponding to the original face image; and the original rendering information is used to describe facial contour information and lighting information corresponding to the original face image;

[0008] According to the original face image, the original local semantic information, the original rendering information and the face editing model, an intermediate face feature image and a first loss value are obtained; wherein the first loss value is determined according to the difference between the real noise and the predicted noise in each time step in the process of generating the intermediate face feature image by the face editing model;

[0009] Inputting the intermediate facial feature image into the face parsing model to obtain a second loss value; wherein the second loss value is determined based on the difference between the original local semantic information and the intermediate local semantic information; the intermediate local semantic information is used to describe the feature information of the preset attribute category corresponding to the intermediate facial feature image;

[0010] The original face image is input into the face editing model, and the face editing model is controlled to edit the target face image under the constraints of the first loss value and the second loss value.

[0011] In an exemplary embodiment of the present application, the above-mentioned obtaining original local semantic information and original rendering information corresponding to the original face image includes:

[0012] According to the face parsing model, the original local semantic information corresponding to the original face image is obtained;

[0013] The original face image is input into the 3D face modeling model to obtain original rendering information corresponding to the original face image.

[0014] In an exemplary embodiment of the present application, the original face image has corresponding several attribute categories, and the preset attribute category is any one of the several attribute categories; the original local semantic information corresponding to the original face image is obtained according to the face parsing model, including:

[0015] Inputting the original face image into the face parsing model to obtain an original mask corresponding to the original face image; wherein the original mask includes a mask sub-region corresponding to each attribute category; and the mask sub-region has corresponding pixel point contour coordinates;

[0016] According to the pixel point contour coordinates of the mask sub-region corresponding to the preset attribute category, the region corresponding to the preset attribute category in the original face image is obtained;

[0017] Feature extraction is performed on the area corresponding to the preset attribute category in the original face image to obtain the original local semantic information.

[0018] In an exemplary embodiment of the present application, the above-mentioned original rendering information includes a shaded shape map and a surface normal vector map; the above-mentioned shaded shape map is used to describe the facial contour information corresponding to the original face image; the above-mentioned surface normal vector map is used to describe the lighting information corresponding to the original face image.

[0019] In an exemplary embodiment of the present application, the above-mentioned method of obtaining the intermediate face feature image and the first loss value according to the original face image, the original local semantic information, the original rendering information and the face editing model includes:

[0020] Encode the original face image to obtain the corresponding original latent space vector;

[0021] Performing feature fusion on the original rendering information to obtain an original rendering vector; wherein the original rendering vector has the same dimension as the original latent space vector;

[0022] According to the original latent space vector, the original rendering vector, the original local semantic information and the face editing model, an intermediate face feature image and a first loss value are obtained; wherein the original latent space vector is used as the input of the face editing model; the original rendering vector and the original local semantic information are used as input prompts of the face editing model.

[0023] In an exemplary embodiment of the present application, the above-mentioned method of obtaining the intermediate face feature image and the first loss value according to the original latent space vector, the original rendering vector, the original local semantic information and the face editing model includes:

[0024] Controlling the face editing model to add real noise and remove predicted noise to the original latent space vector according to the original rendering vector and the original local semantic information, so as to obtain a target latent space vector and a first loss value;

[0025] The above target latent space vector is decoded and features are extracted to obtain an intermediate face feature image.

[0026] In an exemplary embodiment of the present application, the first loss value is determined according to a first loss function. The first loss function L LDM Meet the following conditions:

[0027] L LDM =E t,z0,ε,T,M [||ε t -ε θ (z t ,t,local,exp)|| 2 ]

[0028] Where t represents the time step; z0 represents the image feature of the original latent space vector without noise; ε t represents the real noise at time t; ε θ represents the prediction noise; z t represents the image features of the original latent space vector at time t; local represents the original local semantic information control; exp represents the original rendering vector control;

[0029] The above second loss value is determined according to the second loss function. The second loss function L sim Meet the following conditions:

[0030]

[0031] Among them, A is the original local semantic information; B is the intermediate local semantic information.

[0032] According to a second aspect of the present application, a facial image editing device retaining specific attributes is provided, the device comprising:

[0033] An information acquisition unit, used to acquire original local semantic information and original rendering information corresponding to the original face image; wherein the original local semantic information is used to describe feature information of a preset attribute category corresponding to the original face image; and the original rendering information is used to describe facial contour information and lighting information corresponding to the original face image;

[0034] A first loss value determining unit is used to obtain an intermediate face feature image and a first loss value according to the original face image, the original local semantic information, the original rendering information and the face editing model; wherein the first loss value is determined according to the difference between the real noise and the predicted noise in each time step in the process of generating the intermediate face feature image according to the face editing model;

[0035] A second loss value determination unit is used to input the above-mentioned intermediate face feature image into the face parsing model to obtain a second loss value; wherein the above-mentioned second loss value is determined according to the difference between the original local semantic information and the intermediate local semantic information; the intermediate local semantic information is used to describe the feature information of the preset attribute category corresponding to the intermediate face feature image;

[0036] The target generation unit is used to input the original face image into the face editing model, and control the face editing model to edit the target face image under the constraints of the first loss value and the second loss value.

[0037] According to the third aspect of the present application, a non-transitory computer-readable storage medium is provided, in which at least one instruction or at least one program is stored, and the at least one instruction or at least one program is loaded and executed by a processor to implement the above-mentioned facial image editing method that retains specific attributes.

[0038] According to a fourth aspect of the present application, there is provided an electronic device comprising a processor and the above-mentioned non-transitory computer-readable storage medium.

[0039] This application has at least the following beneficial effects:

[0040] The face image editing method for retaining specific attributes provided by the present application first obtains the original local semantic information and original rendering information corresponding to the original face image. Among them, the original local semantic information is used to describe the feature information of the preset attribute category corresponding to the original face image; the original rendering information is used to describe the facial contour information and lighting information corresponding to the original face image. Secondly, according to the original face image, the original local semantic information, the original rendering information and the face editing model, an intermediate face feature image and a first loss value are obtained. Then, the intermediate face feature image is input into the face parsing model to obtain the second loss value. Finally, the original face image is input into the face editing model, and the face editing model is controlled to edit the target face image under the constraints of the first loss value and the second loss value. The face image editing method for retaining specific attributes provided by the present application constrains the generation of the model by setting the original local semantic information and the original rendering information, so that the final face image retains the facial attributes corresponding to the original local semantic information. In addition, the face editing model is constrained by the first loss function and the second loss function, thereby improving the control ability of the face generation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0042] Figure 1 A schematic diagram of a flow chart of a method for editing a facial image that retains specific attributes provided by an embodiment of the present application;

[0043] Figure 2 A schematic flow chart of a facial image editing method for retaining specific attributes provided by another embodiment of the present application;

[0044] Figure 3 A structural block diagram of a facial image editing device that retains specific attributes provided by one embodiment of the present application. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0046] like Figure 1As shown, an embodiment of the present application provides a method for editing a face image with retaining specific attributes, the method comprising:

[0047] Step S100, obtaining original local semantic information and original rendering information corresponding to the original face image; wherein the original local semantic information is used to describe feature information of a preset attribute category corresponding to the original face image; and the original rendering information is used to describe facial contour information and lighting information corresponding to the original face image.

[0048] Specifically, the original face image can be any face image randomly selected from a face image library. Further, step S100 also includes:

[0049] Step S110, obtaining original local semantic information corresponding to the original face image according to the face analysis model.

[0050] The above-mentioned face analysis model can segment the various parts of the face (left eye, right eye, nose, mouth, hair, etc.) as required. Here, the face analysis model can be any model that can realize face segmentation analysis known to those skilled in the art, and is not limited here.

[0051] Furthermore, the original face image has corresponding several attribute categories, and the preset attribute category is any one of the several attribute categories; as an example, the attribute category may be left eye, right eye, nose, mouth, hair, etc. The above step S110 includes:

[0052] Step S111, inputting the original face image into the face parsing model to obtain an original mask corresponding to the original face image; wherein the original mask includes a mask sub-region corresponding to each attribute category; and the mask sub-region has corresponding pixel point contour coordinates.

[0053] The input of the face parsing model is the original face image, and the output is the coordinates of each pixel in the preset face image, the organ category corresponding to each pixel, and the original mask corresponding to the original face image. That is, the segmentation of each attribute category of the face is realized according to the face parsing model, and the position of each attribute category (left eye, right eye, nose, mouth, hair, etc.) is determined. Then, the mask sub-area corresponding to each attribute category in the mask and the corresponding pixel point contour coordinates are matched to obtain the pixel point contour coordinates corresponding to each mask sub-area.

[0054] Step S112, obtaining the area corresponding to the preset attribute category in the original face image according to the outline coordinates of the pixel points of the mask sub-area corresponding to the preset attribute category.

[0055] Here, the area corresponding to the preset attribute category in the original face image is obtained according to the pixel point contour coordinates of the mask sub-area corresponding to the preset attribute category, that is, the pixel point contour coordinates of the mask sub-area corresponding to the preset attribute category are mapped to the area corresponding to the preset attribute category in the original face image.

[0056] Step S113, extracting features of the area corresponding to the preset attribute category in the original face image to obtain original local semantic information.

[0057] That is, the original local semantic information describes the feature information of the preset attribute category corresponding to the original face image. As an example: if the preset attribute category is nose, the original local semantic information describes the feature information of the nose area in the original face image.

[0058] Step S120: input the original face image into the 3D face modeling model to obtain original rendering information corresponding to the original face image.

[0059] Specifically, the 3D face modeling model in this embodiment is a 3D face modeling model with fixed weights. The 3D face modeling model can provide information such as lighting, posture, and expression for the face image generation process. At the same time, compared with the image mask, the 3D face modeling model can provide clearer facial contour information and can more clearly control the editing of face attributes. Through basic face modeling, only a single or a small amount of image data is needed to perform face editing tasks.

[0060] In addition, the above original rendering information includes a shaded shape map and a surface normal vector map; the above shaded shape map is used to describe the facial contour information corresponding to the original face image; the above surface normal vector map is used to describe the lighting information corresponding to the original face image.

[0061] Step S200, obtaining an intermediate facial feature image and a first loss value based on the original facial image, original local semantic information, original rendering information and the face editing model; wherein the above-mentioned first loss value is determined based on the difference between the actual noise and the predicted noise in each time step in the process of generating the intermediate facial feature image by the face editing model.

[0062] In one embodiment of the present application, the face editing model is a stable diffusion model (Stable Diffusion Model), which is a probabilistic framework based on the above diffusion model (Denoising Diffusion Probabilistic Model DDPM), and the training of the model includes two main parts: forward diffusion and reverse diffusion. In the forward diffusion stage, the model destroys the image by gradually introducing noise (real noise) until the image becomes completely random noise. In the reverse diffusion stage, the model uses a series of Markov chains to gradually remove the prediction noise and recover the data from Gaussian noise, thereby generating a high-quality image.

[0063] Furthermore, step S200 further includes:

[0064] Step S210, encoding the original face image to obtain the corresponding original latent space vector.

[0065] Here, the original face image is encoded by the image encoder to obtain the corresponding original latent space vector.

[0066] Step S220 , performing feature fusion on the original rendering information to obtain an original rendering vector; wherein the original rendering vector has the same dimension as the original latent space vector.

[0067] The u-net network is used to extract features from the colored shape map and the surface normal vector map at different scales and perform feature fusion, outputting feature information of the same dimension as the original latent space vector to obtain the original rendering vector.

[0068] U-Net is an important network in the field of image segmentation. It can obtain features of different receptive fields at the same level and pass them to the next level after fusion.

[0069] It should be noted that the dimensions of the original latent space vector, the original rendering vector, and the original local semantic information are all the same.

[0070] Step S230, obtaining an intermediate facial feature image and a first loss value based on the original latent space vector, the original rendering vector, the original local semantic information and the face editing model; wherein the original latent space vector is used as the input of the face editing model; and the original rendering vector and the original local semantic information are used as input prompts of the face editing model.

[0071] Specifically, step S230 also includes:

[0072] Step S231, controlling the above-mentioned face editing model to add real noise and remove predicted noise to the original latent space vector according to the original rendering vector and the original local semantic information, so as to obtain a target latent space vector and a first loss value.

[0073] First, Gaussian noise (real noise) is gradually added to the original latent space vector in each time step. As the noise is gradually added, the image data gradually becomes blurred and noisy until it finally approaches the standard Gaussian distribution, that is, the original latent space vector after noisy is obtained.

[0074] Furthermore, under the guidance of the original rendering vector and the original local semantic information, the stable diffusion model calculates the noise that needs to be removed from the original latent space vector after adding noise at each time step, and calculates the first loss function of the stable diffusion model. The first loss function is the mean square error between the noise predicted by the calculation model (removed noise) and the corresponding real noise (added noise). Furthermore, the first loss function L LDM Meet the following conditions:

[0075] L LDM =E t,z0,ε,T,M [||ε t -ε θ (z t ,t,local,exp)|| 2 ]

[0076] Where t represents the time step; z0 represents the image feature of the original latent space vector without noise; ε t represents the real noise at time t; ε θ represents the prediction noise; z t Represents the image features of the original latent space vector at time t; local represents the original local semantic information control; exp represents the original rendering vector control.

[0077] Step S232, decoding the target latent space vector and performing feature extraction to obtain an intermediate facial feature image.

[0078] In one embodiment, when decoding the target latent space vector and performing feature extraction, three layers of multi-scale features are extracted, and the final intermediate face feature image is obtained through convolution and continuous upsampling, so that the feature information corresponding to the extracted intermediate face feature image is richer.

[0079] Step S300, input the above-mentioned intermediate facial feature image into the face parsing model to obtain a second loss value; wherein the above-mentioned second loss value is determined based on the difference between the original local semantic information and the intermediate local semantic information; the intermediate local semantic information is used to describe the feature information of the preset attribute category corresponding to the intermediate facial feature image.

[0080] In one embodiment, the output of the face editing model (intermediate face feature image) is not directly used as the final output, but the intermediate face feature image is input into the face parsing model. On the one hand, by setting a second loss function, the similarity between the original local semantic information corresponding to the preset attribute category in the intermediate face feature image and the intermediate local semantic information preliminarily generated by the face editing model is determined. Furthermore, the second loss function L sim Meet the following conditions:

[0081]

[0082] Among them, A is the original local semantic information; B is the intermediate local semantic information.

[0083] When the similarity is greater, it means that the performance of the face editing model is closer to the expected output (that is, the feature information corresponding to the preset attribute category in the generated new face image is more similar to the feature information corresponding to the preset attribute category in the original face image, and is close to the same). In this embodiment, on the one hand, in order to improve the model's control ability over the face generation process, and on the other hand, in order to retain the feature information corresponding to the preset attribute category as much as possible, the first loss function and the second loss function are used to jointly constrain the output of the face editing model, that is, in the end, the comprehensive loss function L of the face editing model meets the following conditions:

[0084] L=L LDM +λL sim ;

[0085] Among them, λ is a hyperparameter determined by training.

[0086] Step S400: input the original face image into the face editing model, and control the face editing model to edit the target face image under the constraints of the first loss value and the second loss value.

[0087] Finally, the target face image is obtained. The feature information corresponding to the preset attribute category in the target face image is similar to the feature information corresponding to the preset attribute category in the original face image, that is, this embodiment achieves the purpose of retaining specific face attributes during face editing.

[0088] In addition, in another embodiment of the present application, after step S100, the method further includes:

[0089] S500, masking the original face image according to the original local semantic information to obtain a masked face image; wherein the masked area is an area of ​​the original face image that does not correspond to the preset attribute category;

[0090] S600, encoding the occluded face image to obtain a corresponding occlusion latent space vector.

[0091] S700, obtaining an intermediate facial feature image and a first loss value according to the occlusion latent space vector, the original rendering vector and the face editing model; wherein the occlusion latent space vector is used as the input of the face editing model; and the original rendering vector is used as the input prompt of the face editing model.

[0092] S800, inputting the occluded face image into the face editing model, and controlling the face editing model to edit and obtain the target face image under the constraint of the first loss value.

[0093] In this embodiment, the original face image is masked according to the original local semantic information, so that other areas except the original face image area corresponding to the preset attribute category are masked, and here, the masking method can be noise. Afterwards, the processed masked latent space vector is input into the model, and the parameters of the face editing model are adjusted at the same time, so that the face editing model only processes the masked part of the original face image. Therefore, in the output result obtained, the features of the original face image area corresponding to the preset attribute category are not processed by noise addition and denoising in the model, that is, the area corresponding to the preset attribute category in the generated target face image is basically the same as the area corresponding to the preset attribute category in the original face image.

[0094] This embodiment pre-processes the areas corresponding to non-preset attribute categories of the original image so that the areas corresponding to the preset attribute categories in the generated target facial image are basically the same as the areas corresponding to the preset attribute categories in the original facial image. On the one hand, the original rendering vector is used as an input prompt for the face editing model, which improves the model's control ability over the face generation process. On the other hand, the feature information corresponding to the preset attribute categories is retained.

[0095] This embodiment constrains the above-mentioned face editing model according to the comprehensive loss function L to obtain the target face image.

[0096] The following describes the face image editing process of retaining specific attributes of the present application according to a specific embodiment. Figure 2As shown in FIG. 1 , the original face image img is input into the face parsing model P(·) to obtain the original mask corresponding to the original face image; wherein the original mask includes a mask sub-region corresponding to each attribute category; the mask sub-region has a corresponding pixel point contour coordinate, and then the retained attribute c feature (local semantic information) is obtained according to the preset attribute category; at the same time, the original face image img is input into the 3D face modeling model, and the rendering image (original rendering information) corresponding to the original face image is obtained by changing the parameters to edit the illumination, posture, etc., and then multi-scale feature fusion is performed on it to obtain the multi-scale feature of the rendering image (original rendering vector); then, the original face image img is used as input, and the multi-scale feature of the rendering image (original rendering vector) is used to obtain the multi-scale feature of the rendering image. The target face image is then input into the face editing model P(·) to obtain the attribute c feature (intermediate local semantic information); wherein, the latent diffusion model is constrained according to a first loss value (determined according to the difference between the real noise and the predicted noise in each time step in the process of generating the intermediate face feature image by the face editing model) and a second loss value (the above-mentioned second loss value is determined according to the difference between the original local semantic information and the intermediate local semantic information; the intermediate local semantic information is used to describe the feature information of the preset attribute category corresponding to the intermediate face feature image).

[0097] In addition, although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0098] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0099] Please refer to Figure 3 As shown, an embodiment of the present application provides a facial image editing device 100 for retaining specific attributes, the device comprising:

[0100] The information acquisition unit 110 is used to acquire original local semantic information and original rendering information corresponding to the original face image; wherein the original local semantic information is used to describe feature information of a preset attribute category corresponding to the original face image; and the original rendering information is used to describe facial contour information and lighting information corresponding to the original face image.

[0101] The first loss value determination unit 120 is used to obtain an intermediate facial feature image and a first loss value based on the original facial image, original local semantic information, original rendering information and the face editing model; wherein the above-mentioned first loss value is determined based on the difference between the actual noise and the predicted noise in each time step in the process of generating the intermediate facial feature image by the face editing model.

[0102] The second loss value determination unit 130 is used to input the above-mentioned intermediate facial feature image into the face parsing model to obtain a second loss value; wherein the above-mentioned second loss value is determined based on the difference between the original local semantic information and the intermediate local semantic information; the intermediate local semantic information is used to describe the feature information of the preset attribute category corresponding to the intermediate facial feature image.

[0103] The target generation unit 140 is used to input the original face image into the face editing model, and control the face editing model to edit and obtain the target face image under the constraints of the first loss value and the second loss value.

[0104] In an exemplary embodiment of the present application, the information acquisition unit is obtained according to the following subunits:

[0105] The local semantic information acquisition subunit is used to obtain the original local semantic information corresponding to the original face image according to the face parsing model.

[0106] The rendering information acquisition subunit is used to input the original face image into the 3D face modeling model to obtain the original rendering information corresponding to the original face image.

[0107] In an exemplary embodiment of the present application, the above-mentioned original rendering information includes a shaded shape map and a surface normal vector map; the above-mentioned shaded shape map is used to describe the facial contour information corresponding to the original face image; the above-mentioned surface normal vector map is used to describe the lighting information corresponding to the original face image.

[0108] In an exemplary embodiment of the present application, the original face image has corresponding several attribute categories, and the preset attribute category is any one of the several attribute categories; the local semantic information acquisition subunit includes:

[0109] The mask acquisition component is used to input the original face image into the face parsing model to obtain the original mask corresponding to the original face image; wherein the original mask contains a mask sub-region corresponding to each attribute category; and the mask sub-region has corresponding pixel point contour coordinates.

[0110] The region determination component is used to obtain the region corresponding to the preset attribute category in the original face image according to the pixel point contour coordinates of the mask sub-region corresponding to the preset attribute category.

[0111] The local semantic information determination component is used to extract features from the area corresponding to the preset attribute category in the original face image to obtain the original local semantic information.

[0112] In an exemplary embodiment of the present application, the first loss value determining unit is obtained according to the following subunits:

[0113] The latent space vector acquisition subunit is used to encode the original face image to obtain the corresponding original latent space vector.

[0114] The rendering vector acquisition subunit is used to perform feature fusion on the original rendering information to obtain an original rendering vector; wherein the original rendering vector has the same dimension as the original latent space vector.

[0115] The first loss value determination subunit is used to obtain an intermediate facial feature image and a first loss value based on the original latent space vector, the original rendering vector, the original local semantic information and the face editing model; wherein the original latent space vector is used as the input of the face editing model; the original rendering vector and the original local semantic information are used as input prompts of the face editing model.

[0116] In an exemplary embodiment of the present application, the first loss value determining subunit includes:

[0117] The first loss value determination component is used to control the above-mentioned face editing model to add real noise and remove predicted noise to the original latent space vector according to the original rendering vector and the original local semantic information to obtain the target latent space vector and the first loss value.

[0118] The intermediate facial feature image determination component is used to decode the above-mentioned target latent space vector and extract features to obtain an intermediate facial feature image.

[0119] In an exemplary embodiment of the present application, the first loss value is determined according to a first loss function. The first loss function L LDM Meet the following conditions:

[0120] L LDM =E t,z0,ε,T,M [||εt -ε θ (z t ,t,local,exp)|| 2 ]

[0121] Where t represents the time step; z0 represents the image feature of the original latent space vector without noise; ε t represents the real noise at time t; ε θ represents the prediction noise; z t Represents the image features of the original latent space vector at time t; local represents the original local semantic information control; exp represents the original rendering vector control.

[0122] The above second loss value is determined according to the second loss function. The second loss function L sim Meet the following conditions:

[0123]

[0124] Among them, A is the original local semantic information; B is the intermediate local semantic information.

[0125] In an exemplary embodiment of the present application, an electronic device capable of implementing the above method is also provided.

[0126] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0127] The electronic device according to this embodiment of the present application is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0128] The electronic device is presented in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: the at least one processor mentioned above, the at least one storage device mentioned above, and a bus connecting different system components (including storage devices and processors).

[0129] The storage device stores program codes, which can be executed by the processor, so that the processor executes the steps described in the “exemplary method” section of the specification according to various exemplary embodiments of the present application.

[0130] The memory may include readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory, and may further include read only memory (ROM).

[0131] The storage may also include a program / utility having a set (at least one) of program modules, such program modules including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0132] The bus may represent one or more of several types of bus structures including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.

[0133] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may communicate with one or more devices that enable a user to interact with the electronic device, and / or may communicate with any device (e.g., routers, modems, etc.) that enables the electronic device to communicate with one or more other computing devices. Such communication may be performed through an input / output (I / O) interface. Furthermore, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter. The network adapter communicates with other modules of the electronic device through a bus. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0134] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0135] In an exemplary embodiment of the present application, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present application can also be implemented in the form of a program product, which includes a program code. When the above program product is run on a terminal device, the above program code is used to enable the above terminal device to execute the steps according to various exemplary implementations of the present application described in the above "Exemplary Method" section of the present specification.

[0136] The above program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0137] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0138] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0139] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0140] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0141] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0142] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for editing a face image while retaining specific attributes, characterized in that: The method comprises: Acquire original local semantic information and original rendering information corresponding to the original face image; wherein the original local semantic information is used to describe feature information of a preset attribute category corresponding to the original face image; and the original rendering information is used to describe facial contour information and lighting information corresponding to the original face image; According to the original face image, the original local semantic information, the original rendering information and the face editing model, an intermediate face feature image and a first loss value are obtained; wherein the first loss value is determined according to the difference between the real noise and the predicted noise in each time step in the process of generating the intermediate face feature image by the face editing model; Inputting the intermediate facial feature image into a facial parsing model to obtain a second loss value; wherein the second loss value is determined based on the difference between the original local semantic information and the intermediate local semantic information; the intermediate local semantic information is used to describe feature information of a preset attribute category corresponding to the intermediate facial feature image; The original face image is input into the face editing model, and the face editing model is controlled to edit the target face image under the constraints of the first loss value and the second loss value.

2. The method for editing a facial image while retaining specific attributes according to claim 1, characterized in that: The obtaining of original local semantic information and original rendering information corresponding to the original face image includes: According to the face parsing model, the original local semantic information corresponding to the original face image is obtained; The original face image is input into the 3D face modeling model to obtain original rendering information corresponding to the original face image.

3. The method for editing a facial image while retaining specific attributes according to claim 2, characterized in that: The original face image has corresponding several attribute categories, and the preset attribute category is any one of the several attribute categories; the original local semantic information corresponding to the original face image is obtained according to the face parsing model, including: Inputting the original face image into the face parsing model to obtain an original mask corresponding to the original face image; wherein the original mask includes a mask sub-region corresponding to each attribute category; and the mask sub-region has corresponding pixel point contour coordinates; According to the pixel point contour coordinates of the mask sub-region corresponding to the preset attribute category, the region corresponding to the preset attribute category in the original face image is obtained; Feature extraction is performed on the area corresponding to the preset attribute category in the original face image to obtain the original local semantic information.

4. The method for editing a facial image while retaining specific attributes according to claim 1, characterized in that: The original rendering information includes a shaded shape map and a surface normal vector map; the shaded shape map is used to describe the facial contour information corresponding to the original face image; the surface normal vector map is used to describe the lighting information corresponding to the original face image.

5. The method for editing a facial image while retaining specific attributes according to any one of claims 1 to 4, characterized in that: The method of obtaining an intermediate face feature image and a first loss value according to the original face image, the original local semantic information, the original rendering information and the face editing model includes: Encode the original face image to obtain the corresponding original latent space vector; Performing feature fusion on the original rendering information to obtain an original rendering vector; wherein the original rendering vector has the same dimension as the original latent space vector; According to the original latent space vector, the original rendering vector, the original local semantic information and the face editing model, an intermediate face feature image and a first loss value are obtained; wherein the original latent space vector is used as the input of the face editing model; the original rendering vector and the original local semantic information are used as input prompts of the face editing model.

6. The method for editing a facial image while retaining specific attributes according to claim 5, characterized in that: The method of obtaining an intermediate face feature image and a first loss value according to the original latent space vector, the original rendering vector, the original local semantic information and the face editing model includes: Controlling the face editing model to add real noise and remove predicted noise to the original latent space vector according to the original rendering vector and the original local semantic information, so as to obtain a target latent space vector and a first loss value; The target latent space vector is decoded and features are extracted to obtain an intermediate facial feature image.

7. The method for editing a facial image with a specific attribute preserved according to any one of claims 1 to 4 or 6, characterized in that: The first loss value is determined according to a first loss function. The first loss function L LDM Meet the following conditions: Where t represents the time step; z0 represents the image feature of the original latent space vector without noise; ε t represents the real noise at time t; ε θ represents the prediction noise; z t represents the image features of the original latent space vector at time t; local represents the original local semantic information control; exp represents the original rendering vector control; The second loss value is determined according to a second loss function, where the second loss function L sim Meet the following conditions: Among them, A is the original local semantic information; B is the intermediate local semantic information.

8. A facial image editing device that retains specific attributes, characterized in that: The device comprises: An information acquisition unit, used to acquire original local semantic information and original rendering information corresponding to the original face image; wherein the original local semantic information is used to describe feature information of a preset attribute category corresponding to the original face image; and the original rendering information is used to describe facial contour information and lighting information corresponding to the original face image; A first loss value determining unit is used to obtain an intermediate face feature image and a first loss value according to the original face image, the original local semantic information, the original rendering information and the face editing model; wherein the first loss value is determined according to the difference between the real noise and the predicted noise in each time step in the process of generating the intermediate face feature image according to the face editing model; A second loss value determination unit is used to input the intermediate facial feature image into the facial analysis model to obtain a second loss value; wherein the second loss value is determined according to the difference between the original local semantic information and the intermediate local semantic information; the intermediate local semantic information is used to describe the feature information of the preset attribute category corresponding to the intermediate facial feature image; The target generation unit is used to input the original face image into the face editing model, and control the face editing model to edit the target face image under the constraints of the first loss value and the second loss value.

9. A non-transitory computer-readable storage medium, wherein at least one instruction or at least one program is stored in the storage medium, characterized in that: The at least one instruction or the at least one program is loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: The invention comprises a processor and the non-transitory computer-readable storage medium as claimed in claim 9.

Images