Face editing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention is suitable for the technical field of images, and provides a face editing method and device, electronic equipment and a storage medium, and the method comprises the steps: extracting a first semantic mask and a second semantic mask in a to-be-edited face image; the first semantic mask comprises sub-masks corresponding to multiple semantic attributes respectively; the second semantic mask does not comprise a target sub-mask corresponding to the target semantic attribute needing to be edited; inputting the first semantic mask, the second semantic mask and an editing requirement corresponding to the target semantic attribute into a mask generation model for redrawing to obtain a third semantic mask; the mask generation model is used for reserving information corresponding to non-target semantic attributes in a third semantic mask based on the second semantic mask; and performing face editing on the to-be-edited face image based on the third semantic mask and an editing requirement to obtain an edited face image. By adopting the method, the refinement degree and controllability of face editing can be improved.

Description

Technical Field

[0001] The present application belongs to the field of image technology, and in particular relates to a face editing method, device, electronic device and storage medium. Background Art

[0002] Virtual digital human is a digital image close to the image of a human face created by editing a single face image through a computer. With the continuous breakthrough of artificial intelligence technology, the image, expression and expression of digital human are gradually becoming similar to real people.

[0003] Currently, when generating digital human images, the face image to be edited and the editing requirements are usually input into a pre-trained generative adversarial network algorithm model to directly generate the edited face image.

[0004] However, the existing editing methods of generative adversarial networks are difficult to achieve flexible editing of detailed features in digital faces. For example, when the adversarial network algorithm model processes the face image to be edited based on the editing requirements, the generated edited face image may not only be different from the face image to be edited in the image part corresponding to the editing requirements, but also have significant differences in other image parts that do not need to be edited. In other words, the degree of refinement of face editing in the existing technology is low. Summary of the invention

[0005] The embodiments of the present application provide a face editing method, device, electronic device and storage medium, which can solve the problem of low degree of refinement in face editing in the prior art.

[0006] In a first aspect, an embodiment of the present application provides a face editing method, the method comprising:

[0007] Extracting a first semantic mask and a second semantic mask from the face image to be edited; the first semantic mask includes sub-masks corresponding to a plurality of semantic attributes respectively; the second semantic mask does not include a target sub-mask corresponding to a target semantic attribute to be edited;

[0008] Inputting the first semantic mask, the second semantic mask, and the editing requirements corresponding to the target semantic attribute into the mask generation model for redrawing to obtain a third semantic mask; the mask generation model is used to retain information corresponding to the non-target semantic attribute in the third semantic mask based on the second semantic mask;

[0009] Based on the third semantic mask and the editing requirement, face editing is performed on the face image to be edited to obtain an edited face image.

[0010] In a second aspect, an embodiment of the present application provides a face editing device, the device comprising:

[0011] An extraction module is used to extract a first semantic mask and a second semantic mask from a face image to be edited; the first semantic mask includes sub-masks corresponding to a plurality of semantic attributes respectively; the second semantic mask does not include a target sub-mask corresponding to a target semantic attribute to be edited;

[0012] A redrawing module, used for inputting the first semantic mask, the second semantic mask and the editing requirements corresponding to the target semantic attribute into the mask generation model for redrawing to obtain a third semantic mask; the mask generation model is used for retaining information corresponding to the non-target semantic attribute in the third semantic mask based on the second semantic mask;

[0013] The editing module is used to perform face editing on the face image to be edited based on the third semantic mask and the editing requirement to obtain an edited face image.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method of the first aspect described above when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method of the first aspect described above is implemented.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute the method of the first aspect.

[0017] Compared with the prior art, the embodiment of the present application has the following beneficial effects: the electronic device can extract the first semantic mask of all the semantic information of the face image to be edited, and the second semantic mask that does not contain the target sub-mask corresponding to the target semantic attribute to be edited from the face image to be edited. Among them, since the second semantic mask does not include the semantic information corresponding to the target semantic attribute of the face image to be edited. Therefore, when the first semantic mask, the second semantic mask and the target semantic attribute are input into the mask generation model for redrawing, the mask generation model can reduce the image information loss corresponding to other semantic attributes that do not need to be edited during the redrawing process as much as possible based on the second semantic mask. Furthermore, it is possible to achieve accurate editing of the mask information corresponding to the target semantic attribute that needs to be edited, so that the third semantic mask after redrawing is different from the first semantic mask in the information on the target semantic attribute that needs to be edited. Finally, the electronic device can perform face editing on the face image to be edited based on the refined third semantic mask and the editing requirements corresponding to the target semantic attribute to obtain the edited face image. Based on this, the above method can accurately and controllably generate a refined intermediate product (third semantic mask) with different image information corresponding to the target semantic attribute to be edited, so that the face image to be edited can be edited based on the refined intermediate product, so that the final edited face image and the face image to be edited only differ in the image part corresponding to the editing requirements, and other image parts that do not need to be edited are as similar as possible. In turn, the refinement and controllability of face editing are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 This is a flowchart of a face editing method provided in one embodiment of the present application;

[0020] Figure 2 This is a schematic diagram of an application scenario of constructing a first semantic mask and a second semantic mask in a face editing method provided by an embodiment of the present application;

[0021] Figure 3 This is a schematic diagram of an implementation method for generating a third semantic mask in a face editing method provided in an embodiment of the present application;

[0022] Figure 4 It is a schematic diagram of the model structure of a mask generation model in a face editing method provided in an embodiment of the present application;

[0023] Figure 5 This is a schematic diagram of an application scenario of a face editing method in a face editing method provided in an embodiment of the present application;

[0024] Figure 6 This is a flowchart for implementing training of a first redrawing sub-model in a face editing method provided in an embodiment of the present application;

[0025] Figure 7 This is a schematic diagram of an implementation method for generating an edited face image in a face editing method provided in an embodiment of the present application;

[0026] Figure 8 It is a structural schematic diagram of a face editing device provided by an embodiment of the present application;

[0027] Fig. 9 It is a structural schematic diagram of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0028] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0029] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.

[0030] It should be noted that the information collection process (such as face image collection process, fingerprint information collection process, etc.) / feature extraction process involved in this application is performed with the user's knowledge and permission, that is, the information collection process / feature extraction process complies with the requirements of laws and regulations and does not constitute an act that harms the public interest.

[0031] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0032] Virtual digital human is a digital image close to the image of a human face created by editing a single face image through a computer. With the continuous breakthrough of artificial intelligence technology, the image, expression and expression of digital human are gradually becoming similar to real people.

[0033] Currently, when generating digital human images, the face image to be edited and the editing requirements are usually input into a pre-trained generative adversarial network algorithm model to directly generate the edited face image.

[0034] However, the existing editing methods of generative adversarial networks are difficult to achieve flexible editing of detailed features in digital faces. For example, when the adversarial network algorithm model processes the face image to be edited based on the editing requirements, the generated edited face image may not only be different from the face image to be edited in the image part corresponding to the editing requirements, but also have significant differences in other image parts that do not need to be edited. In other words, the degree of refinement of face editing in the existing technology is low.

[0035] Based on this, in order to improve the level of refinement of face editing, an embodiment of the present application provides a face editing method, which can be applied to electronic devices such as tablet computers, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, etc. The embodiment of the present application does not impose any restrictions on the specific types of electronic devices.

[0036] See also Figure 1 , Figure 1 A flowchart of a face editing method provided in an embodiment of the present application is shown, and the method comprises the following steps:

[0037] S101, extracting a first semantic mask and a second semantic mask from a face image to be edited; the first semantic mask includes sub-masks corresponding to a plurality of semantic attributes; the second semantic mask does not include a target sub-mask corresponding to a target semantic attribute to be edited.

[0038] In one embodiment, the facial image to be edited may be any facial image pre-stored in the electronic device, or may be a facial image selected from a preset facial library based on a user's selection operation, and there is no limitation to this.

[0039] It should be noted that, since the face in the face image to be edited needs to be edited, the face image to be edited at least includes a face portion.

[0040] In one embodiment, the face image to be edited can be processed according to a pre-trained image semantic segmentation model to obtain sub-masks corresponding to multiple semantic attributes, and then the first semantic mask and the second semantic mask can be generated based on the multiple sub-masks.

[0041] The semantic attributes include, but are not limited to, the attributes corresponding to various facial parts such as eyes, nose, ears, and hair, and may also include the attributes corresponding to accessories on the head such as earrings, hairpins, and glasses, without limitation. Based on this, it can be considered that the sub-masks corresponding to the semantic attributes include, but are not limited to, the sub-masks corresponding to the eyes, nose, ears, hair, earrings, hairpins, and glasses.

[0042] In one embodiment, the mask is usually a binary image, in which a pixel value of 0 indicates an area of ​​no interest, and a non-zero value (usually 255) indicates an area of ​​interest. In image segmentation, the mask can be used to mark and distinguish different areas in the image. Exemplarily, it can be used to distinguish various areas such as eyes, nose, ears, hair, etc. in the face image to be edited, without limitation.

[0043] For example, refer to Figure 2 , Figure 2 Schematic diagram of an application scenario of constructing a first semantic mask and a second semantic mask in a face editing method provided in an embodiment of the present application. Wherein, I is a face image to be edited, and n fine-grained unobstructed sub-masks M can be extracted from the face image to be edited. 1 ,M 2 ,...,M n . For example, M 1 It can be considered as the sub-mask corresponding to the hair, M 2 Mn can be considered as the sub-mask corresponding to earrings, and Mn can be considered as the sub-mask corresponding to mouth. The mask after the n sub-masks are superimposed is the first semantic mask mentioned above.

[0044] Exemplarily, the first semantic mask may be S 1 =f layer (M 1 , M 2 , ..., M n ). Where flayer represents the superposition of n sub-masks.

[0045] The second semantic mask is similar to the first semantic mask, except that the second semantic mask does not include the target sub-mask corresponding to the target semantic attribute to be edited. For example, when the target semantic attribute is hair, the second semantic mask is S 2 =f layer (M 2 , ..., M n ).

[0046] As an example, the second semantic mask may be as follows:

[0047] S k =flayer (M 1 , M 2 , ..., M k-1 , M k+1 ..., M n ), k∈[1,n].

[0048] Among them, S k represents the second semantic mask generated by not including the sub-mask corresponding to the k-th target semantic attribute.

[0049] S102, inputting the first semantic mask, the second semantic mask and the editing requirements corresponding to the target semantic attribute into the mask generation model for redrawing to obtain a third semantic mask; the mask generation model is used to retain the information corresponding to the non-target semantic attribute in the third semantic mask based on the second semantic mask.

[0050] In one embodiment, the editing requirement is a requirement for editing the image portion of the target semantic attribute. For example, when the target semantic attribute is hair, the corresponding editing requirement may be to lengthen the hair or to shorten the hair, which is not limited.

[0051] In one embodiment, the mask generation model is used to redraw the extracted first semantic mask based on the editing requirements, and based on the second semantic mask, the redrawn third semantic mask retains the information corresponding to the non-target semantic attribute as much as possible. That is, in the third semantic mask, the image portion corresponding to the target semantic attribute to be edited is redrawn from the original shape to the new shape, and the portions corresponding to other non-target semantic attributes remain unchanged.

[0052] Furthermore, when face editing is subsequently performed on the face image to be edited based on the third semantic mask, the generated edited face image can be made as similar as possible to the face image to be edited.

[0053] In one embodiment, the mask generation model may be a pre-set convolutional neural network, which is used to perform convolution processing on the first semantic mask and the second semantic mask. In this embodiment, the model structure of the mask generation model is not limited.

[0054] As an example, the mask generation model may include an autoencoder and a first redrawing submodel. The autoencoder includes a first encoding layer for encoding the second semantic mask respectively, and a first decoding layer symmetrical to the first encoding layer. The first redrawing network submodel may be a DIT network, a U-Net (deep learning network), which is not limited. In this embodiment, the first redrawing submodel is explained as U-Net as an example.

[0055] Among them, the network structure of U-Net is a symmetrical structure, and the symmetry is reflected in its overall U-shape, including the feature extraction layer on the left and the upsampling process on the right, which are symmetrical in structure. One of the main functions of U-Net in this embodiment is to generate noise with higher prediction accuracy.

[0056] In a specific embodiment, referring to Figure 3 , electronic equipment can be based on Figure 3 The steps S301-S303 shown generate a third semantic mask. The details are as follows:

[0057] S301 . Input a first semantic mask and an editing requirement into a first redrawing sub-model to obtain a first tensor after redrawing.

[0058] In one embodiment, the tensor may be a one-dimensional encoding vector or a multi-dimensional matrix, which is not limited. In this embodiment, the following description is made with the tensor being a multi-dimensional matrix. Specifically, the dimensions of each of the following tensors may be 4 dimensions, typically [B, C, H, W]. Wherein B represents the number of input images. C represents the number of color channels. For example, for an RGB image, C is typically 3. H represents the height of the input image, and W represents the width of the input image.

[0059] In one embodiment, the first redrawing network sub-model has been described above and will not be explained again. When training the first redrawing sub-model, the electronic device can use the first semantic mask of the face image to be redrawn and the preset random noise as training data, input them into the redrawing sub-model for training, and obtain a prediction tensor. Then, the training loss is calculated based on the prediction tensor and the preset tensor, and the redrawing sub-model is updated according to the training loss to obtain the first redrawing sub-model.

[0060] Alternatively, when the first semantic mask of the face image to be redrawn and the preset random noise are used as training data and input into the redrawing sub-model for training, the predicted random noise can be obtained. Then, the training loss is calculated based on the predicted random noise and the preset random noise, and the redrawing sub-model is updated according to the training loss to obtain the above-mentioned first redrawing sub-model. In this embodiment, the training method of the first redrawing sub-model is not limited.

[0061] It should be noted that when the editing requirement is input into the first redrawing sub-model, the first redrawing sub-model can not only determine the editing requirement, but also perform semantic recognition on the editing requirement to determine the target semantic attribute to be edited.

[0062] For example, when the editing requirement is “hair becomes longer”, the corresponding target semantic attribute can be identified as “hair”. Based on this, it can be considered that the first redrawing sub-model can generate a first tensor according to the first semantic mask, the editing requirement and the target semantic attribute.

[0063] As an example, in order to accurately generate the first tensor, the electronic device may first input the first semantic mask into the multi-layer first coding layer for processing to obtain a third tensor output by the multi-layer first coding layer. Then, the preset random noise is concatenated with the third tensor to obtain a new first semantic mask.

[0064] In one embodiment, the first coding layer can be considered as a node block that continuously deepens the network and the number of channels to extract shallow information and deep features in the first semantic mask. The first coding layer of each layer is used to encode the input data as the input of the first coding layer of the next layer. The input of the first coding layer of the first layer is the first semantic mask, and the output of the first coding layer of the last layer is the first tensor.

[0065] In one embodiment, the preset random noise may be randomly set, which is not limited.

[0066] As an example, see Figure 4 and Figure 5 , Figure 4 is a schematic diagram of a model structure of a mask generation model in a face editing method provided in an embodiment of the present application, Figure 5 It is a schematic diagram of an application scenario of a face editing method in a face editing method provided in one embodiment of the present application.

[0067] As an example, Figure 4 The preset random noise is not shown in the figure. Figure 5 Z t Among them, according to Figure 5 It can be seen that the electronic device can first input the first semantic mask into the multi-layer first encoding layer for processing, and obtain the detailed information corresponding to each semantic attribute in the face image to be edited. Then, it is spliced ​​with the preset random noise as a new first semantic mask, and input into the first redrawing sub-network model, so that the redrawn first tensor retains more detailed information in the encoding stage.

[0068] in, Figure 5 The first coding layer (ε) in is 3 layers, Figure 4 The first coding layer in is 5 layers. That is, the number of layers of the first coding layer can be set according to actual conditions and is not limited thereto. Figure 5 The semantic mask in can be considered as the first semantic mask and the second semantic mask. In order to ensure the clarity of the image, Figure 5 Only one semantic mask is used to represent it.

[0069] Among them, Figure 4For example, after the first semantic mask is input into the multi-layer first encoding layer for processing, the processed result can be input into the U-Net model for processing in combination with the editing requirements, and the output obtained is the first tensor.

[0070] S302: Input the second semantic mask into the first encoding layer of the multiple layers in the autoencoder for processing, and obtain a second tensor output by the first encoding layer of each layer.

[0071] In one embodiment, the multiple first coding layers in the autoencoder may be consistent with the first coding layer described in S101. That is, the electronic device may input the first semantic mask and the second semantic mask into the multiple first coding layers for processing.

[0072] It should be noted that, unlike the way of processing the first semantic mask, when the second semantic mask is input to the multiple first coding layers, the second tensor output by the first coding layer of each layer will be obtained, instead of only one first tensor in the end. In addition, when processing the second semantic mask, the first coding layer of each layer is used to encode the input data as the input of the first coding layer of the next layer. And, the input of the first coding layer of the first layer includes the second semantic mask.

[0073] The autoencoder further includes a first decoding layer symmetrical to the first encoding layer of each layer, so as to decode the first tensor and the second tensor generated in the above steps in the subsequent steps. Figure 4 , Figure 4 The order of the multiple first coding layers included in is represented by 5, 4, 3, 2, 1, and symmetrically, the order of the first decoding layers is 1, 2, 3, 4, 5. Among them, the first coding layer (1) is symmetrical with the first decoding layer (1), the first coding layer (2) is symmetrical with the second decoding layer (2), and so on.

[0074] As an example, see Figure 4 ,When the second semantic mask is input to multiple layers of the first encoding layer, each first encoding layer not only needs to pass the output second tensor to the first encoding layer of the next layer, but also needs to retain the second tensor.

[0075] S303: Input the first tensor and the second tensor of each layer to the first decoding layer of the multiple layers for decoding processing to obtain a third semantic mask.

[0076] In one embodiment, when decoding the first tensor and the second tensor, the input of the first decoding layer of each layer includes the output of the first decoding layer of the previous layer and the second tensor output by the first encoding layer symmetrical to the first decoding layer of the previous layer; the input of the first decoding layer of the first layer is the first tensor; the output of the first decoding layer of the last layer and the second tensor output by the first encoding layer symmetrical to the first decoding layer of the last layer are added as a third semantic mask.

[0077] It should be noted that the purpose of retaining the second encoding layer vector output by each second encoding layer and inputting the second tensor output by the first encoding layer symmetrical to the first decoding layer of the previous layer into the first decoding layer of the current layer when decoding the first tensor is that the first decoding layer of the current layer can transmit the image detail information corresponding to the semantic attributes that do not need to be edited in the encoding stage to the corresponding decoding stage when decoding the first tensor, and assist the first decoding layer in decoding and redrawing the first tensor, so as to keep the semantic mask that does not need to be edited in the third semantic mask unchanged as much as possible, thereby improving the accuracy of redrawing the third semantic mask. That is, in the decoding process, the image information loss corresponding to the semantic attributes that do not need to be edited in the encoding process is reduced as much as possible.

[0078] For example, refer to Figure 4 , Figure 4 The first decoding layer (1) in , since it is the first decoding layer, there is no first decoding layer in the previous layer. Based on this, it can be considered that the first decoding layer (1) is the first tensor output by the input U-Net model. For the first decoding layer (2), the second decoding layer (2) not only needs to obtain the output of the first decoding layer (1), but also needs to obtain the second vector output by the first encoding layer (1) symmetrical to the first decoding layer (1). Similarly, for the output of the first decoding layer (5) of the last layer, it is necessary to add the second tensor output by the first encoding layer (5) symmetrical to the first decoding layer (5) of the last layer to obtain the above-mentioned third semantic mask.

[0079] In another embodiment, in order to further improve the accuracy of the third semantic mask, before the second tensor is input to the first decoding layer, the first semantic mask may be subjected to mask extraction and inverse operation to obtain a fourth semantic mask. Then, the second tensor of each layer is convolved to obtain the fourth tensor corresponding to each layer. Finally, the fourth tensor of each layer is subjected to mask operation with the fourth semantic mask to obtain a new second tensor corresponding to each layer.

[0080] In one embodiment, the mask extraction is to extract the semantic mask corresponding to the target semantic attribute, and determine the pixel value corresponding to the target semantic attribute in the semantic mask as 1, and the pixel value corresponding to other non-target semantic attributes as 0. And, the inverse operation is to modify the pixel value 1 to the pixel value 0, and the pixel value 0 to the pixel value 1. That is, there are only two pixel values ​​in the fourth semantic mask, which are 1 and 0, respectively, and in the final fourth semantic mask, the pixel value corresponding to the target semantic attribute to be edited is 0, and the pixel value corresponding to the non-target semantic attribute is 1.

[0081] The convolution kernel and convolution step size for convolution processing of the second tensor of each layer can be set according to the actual situation and are not limited thereto. Convolution processing of the second tensor of each layer can further extract rich feature information and retain key information of the image. For example, edge information can be extracted and image information corresponding to each semantic attribute can be retained.

[0082] In one embodiment, the mask operation may be concatenation, or matrix operation may be performed on the fourth tensor of each layer and the fourth semantics to obtain a new second tensor of the corresponding layer.

[0083] Among them, the matrix corresponding to the fourth tensor of each layer can be set according to actual conditions and there is no limitation on this.

[0084] It should be noted that the purpose of performing mask extraction and inverse operation processing on the first semantic mask to obtain the fourth semantic mask is to weaken the image information corresponding to the target semantic attribute to be edited and to strengthen the image information corresponding to the non-target semantic attribute.

[0085] Based on this, by performing masking operation on the fourth tensor through the fourth semantic mask, the features corresponding to the target semantic attributes to be edited can be filtered again, and the first decoding layer can be more sensitive to the edge information in the first semantic mask. In turn, the accuracy of redrawing the third semantic mask can be improved. That is, the information corresponding to the non-target semantic attributes in the redrawn third semantic mask is as similar as possible to the first semantic mask, and the information corresponding to the target semantic attributes has been based on the target semantic mask.

[0086] In another embodiment, the electronic device may Figure 6 The steps S601-S604 shown train the first redrawing sub-model in the mask generation model. The details are as follows:

[0087] S601 : Acquire training data, where the training data includes a training semantic mask of a first training image, a training editing requirement, and training random noise.

[0088] In one embodiment, the first training image is an image containing a human face. The first training image can be obtained from the large-scale semantic segmentation dataset CelebAMask-HQ. The dataset contains 30,000 diverse real face images with semantic masks, a total of 372,767 instances, covering 18 different semantic attributes (e.g., hair, hat, glasses, etc.).

[0089] The training editing requirement may be an editing requirement corresponding to the semantic attribute to be edited in the first training image. For example, when the semantic attribute to be edited is hair, the training editing requirement may be a requirement that the hair in the redrawn training image changes from short hair to long hair, or from long hair to short hair. Based on this, it can also be considered that the training editing requirement also includes the training semantic attribute to be edited.

[0090] Furthermore, the method for obtaining the training semantic mask may be similar to the method for obtaining the first semantic mask or the second semantic mask described in the above step S101, and this is not limited. That is, the training semantic mask may include semantic masks corresponding to all semantic attributes extracted from the first training image, and may also include semantic masks corresponding to all non-training semantic attributes extracted from the first training image, and this is not limited.

[0091] It can be understood that when the training semantic mask includes a semantic mask generated by semantic attributes corresponding to all non-training editing requirements extracted from the first training image, the initial redrawing sub-model to be trained only needs to redraw an image that meets the training editing requirements.

[0092] In one embodiment, the training random noise can be randomly generated and used to add noise to the input training semantic mask in the initial redrawing sub-model. The initial redrawing sub-model generates predicted noise during the training process and performs denoising based on the predicted noise to redraw an image that meets the training editing requirements.

[0093] S602: Input the training data into the initial redrawing sub-model for model training to obtain predicted noise.

[0094] S603: Calculate a training loss value based on the prediction noise and the training random noise.

[0095] In order to make the trained initial redrawing sub-model more suitable for redrawing the third semantic mask in this embodiment, after obtaining the training semantic mask, the multi-layer first encoding layer in the autoencoder can also be used to encode the training semantic mask to obtain a training tensor. Then, the training tensor and the preset random noise are used as the input of the model.

[0096] As an example, the preset random noise may be Z t ∈R4×h×w , the training tensor corresponding to the training semantic mask can be E(S')∈R 4×h×w The electronic device can concatenate the preset random noise and the training tensor as the input of the initial redrawing sub-model. That is, the final input can be γ=[Z t ; E(S')]∈R (4+4)×h×w .

[0097] Among them, Z t represents the preset random noise at time step t, h and w are the width and height dimensions of the noise map respectively. E(S') represents the training semantic mask.

[0098] In addition, in order to match the input of the training data, during the training process, the initial redrawing sub-model can also expand the number of kernel channels of the first convolutional layer by adding zero-initialized weights to match the new input channel dimension to enhance the input capacity of the model.

[0099] And, at any time step t during the back-diffusion of the initial redrawn submodel, the denoised noise Based on the noise map Z obtained in the previous step t-1 (Denoising starts from step T and ends at step 0. For example, T is 1000 steps. When the noise map of step 999 needs to be predicted, the noise map obtained in the previous step is the noise map after denoising at step 1000) and the training editor requirements for prediction. Specifically, the predicted noise (i.e., the above noise map) can be as follows:

[0100] ρ=θ[Z t , E(S'), T, t];

[0101] In one embodiment, the electronic device may pre-store a training loss function for calculating a training loss value, which may be set by a staff member according to actual conditions, and is not limited thereto. Exemplarily, the training loss function may be defined as:

[0102]

[0103] Where ρ represents the prediction noise, S' represents the training semantic mask, and T is the total number of back-diffusion steps.

[0104] S604, iterating the model parameters of the initial redrawing sub-model based on the training loss value until the initial redrawing sub-model meets the convergence condition, thereby obtaining a first redrawing sub-model.

[0105] In one embodiment, the above-mentioned method of iterating model parameters based on the training loss value can be a gradient descent method, an optimization algorithm that adapts to the learning rate, and there is no limitation on this. Among them, the convergence conditions include but are not limited to the number of iterations reaching a preset number of times and the training loss value being less than a preset loss value, and there is no limitation on this.

[0106] S103: Based on the third semantic mask and the editing requirement, perform face editing on the face image to be edited to obtain an edited face image.

[0107] In one embodiment, the electronic device may pre-set a face editing model to perform model processing on the face image to be edited based on the third semantic mask and the editing requirements to generate an edited face image. Alternatively, based on a pre-set mask-to-image generation model, the third semantic mask is directly model processed to obtain an edited face image, which is not limited to this.

[0108] It is understandable that, since the redrawn third semantic mask only differs from the first semantic mask in the information on the target semantic attribute to be edited, when face editing is performed on the face image to be edited based on the refined third semantic mask and the editing requirements corresponding to the target semantic attribute, the final edited face image can be made to differ from the face image to be edited only in the image portion corresponding to the editing requirements, and the other image portions that do not need to be edited are as similar as possible. Thus, the refinement and controllability of face editing are improved.

[0109] As an example, the electronic device may use the face editing model to perform model processing on the face image to be edited based on the third semantic mask and the editing requirement to generate an edited face image. Figure 7 The steps S701-S702 shown generate an edited face image.

[0110] Details are as follows:

[0111] S701. Input the third semantic mask into the mask extraction sub-model in the face editing model to obtain the fifth semantic mask corresponding to the second decoding layer of each layer.

[0112] In one embodiment, the mask extraction sub-model is used to further extract corresponding semantic mask information from the third semantic mask to generate a fifth semantic mask, that is, to better extract the semantic information in the redrawn third semantic code.

[0113] Specifically, the mask extraction submodel includes multiple layers of second coding layers, and second decoding layers symmetrical to the second coding layers of each layer. The input of the second coding layer of each layer is the output of the second coding layer of the previous layer. Among them, the input of the second coding layer of the first layer is the third semantic mask. The input of the second decoding layer of each layer includes the output of the second coding layer symmetrical to the second decoding layer of the current layer. For example, the input of the second decoding layer of the last layer is the output of the second coding layer symmetrical to the second decoding layer of the last layer.

[0114] Specifically, refer to Figure 5, the second encoding layer takes 4 layers as an example. In the mask extraction sub-model, the input of the second encoding layer (4) of the first layer is the third semantic mask, and the input of the second encoding layer (3) of the second layer is the output of the second encoding layer (4). The output of the second encoding layer (1) of the fourth layer (the last layer) is the input of the second decoding layer (1) of the fourth layer. The input of the second decoding layer (2) of the third layer includes the output of the second decoding layer (1) of the fourth layer and the output of the second encoding layer (2) of the third layer.

[0115] In one embodiment, the number of layers of the second coding layer and the third coding layer can also be set according to actual conditions, and there is no limitation on this.

[0116] In one embodiment, referring to Figure 5 It can be seen that the network structure of the mask extraction sub-model can be a U-Net network structure.

[0117] S702: Input the fifth semantic mask of each layer, the target semantic attribute, the editing requirement and the face image to be edited into the second redrawing sub-model in the face editing model to obtain the edited face image.

[0118] In one embodiment, the second redrawing sub-model can redraw the face image to be edited based on the target semantic attributes and editing requirements, and during the redrawing process, correct the redrawn image in combination with the semantic information of the fifth semantic mask to obtain the edited face image.

[0119] The purpose of inputting the third semantic mask into the mask extraction submodel in the face editing model to obtain the fifth semantic mask corresponding to the second decoding layer of each layer is to obtain the image detail information corresponding to each semantic attribute of different layers (high and low layers) in the third semantic mask during the encoding process. In addition, when the corresponding fifth semantic mask is obtained by transmitting it to the corresponding second decoding layer for decoding, it can assist the second decoding layer in decoding and redrawing the image detail information, and can further improve the accuracy of redrawing the fifth semantic mask.

[0120] Based on this, according to the precise fifth semantic mask, the semantic mask corresponding to the edited face image finally generated can be made as similar as possible to the fifth semantic mask, so that the edited face image and the face image to be edited only differ in the image information corresponding to the target semantic attribute to be edited. Thus, the refinement and controllability of face editing are improved.

[0121] As an example, the second redrawing submodel has the same model structure as the mask extraction submodel. That is, the number of third encoding layers and third decoding layers included in the second redrawing submodel is the same as the number of second encoding layers and second decoding layers. The electronic device can input the target semantic attributes, editing requirements, and the face image to be edited into the multi-layer third encoding layer in the second redrawing submodel to obtain the fifth tensor. Then, the fifth tensor and the multi-layer fifth semantic mask are input into the multi-layer third decoding layer in the second redrawing submodel to obtain the edited face image.

[0122] Among them, the model structure of the second redrawing sub-model is the same as that of the mask extraction sub-model. Therefore, the network structure with the same model structure as that of the second redrawing sub-model can also be a U-Net network structure.

[0123] In one embodiment, the target semantic attributes and editing requirements are text content. Therefore, when inputting to the second multi-layer and third coding layer for encoding, the target semantic attributes and editing requirements can be edited using a text editor to generate corresponding text vectors. Figure 5 After that, the text vector and the image to be edited are input into the multi-layer third encoding layer for processing.

[0124] Among them, the input of the third encoding layer of each layer is the output of the third encoding layer of the previous layer, and the input of the third decoding layer of each layer is the output of the third decoding layer of the previous layer, and the fifth semantic mask output by the second decoding layer at the same layer as the third decoding layer of the current layer.

[0125] For example, refer to Figure 5 In the encoding process of the second redrawing sub-model, the input of the third encoding layer (4) of the first layer is the image to be edited and the text vector, and the input of the third encoding layer (3) of the second layer is the output of the third encoding layer (4) of the previous layer.

[0126] In the decoding process, since the third decoding layer (1) of the last layer does not have the third decoding layer of the previous layer, it can be considered that the input of the third decoding layer (1) of the last layer includes the output of the third encoding layer (1) of the last layer and the fifth semantic mask output by the second decoding layer (1) at the same layer (the last layer). Also, the input of the third decoding layer (2) of the third layer is the output of the third decoding layer (1) of the previous layer (the last layer) and the fifth semantic mask output by the second encoding layer (2) at the same layer (the third layer). The inputs of the remaining third decoding layers are deduced in the same way.

[0127] It can be understood that the fifth semantic mask is generated based on the third semantic mask, and compared with the first semantic mask corresponding to the image to be edited, the third semantic mask has the image portion corresponding to the target semantic attribute to be edited, which is redrawn from the original shape to a new shape, and the portions corresponding to other non-target semantic attributes remain unchanged. Therefore, it can be considered that the fifth semantic mask is also different from the first semantic mask in that only the image portion corresponding to the target semantic attribute to be edited exists. Based on this, by correcting the redrawn image in combination with the semantic information of the fifth semantic mask, the generated edited face image and the face image to be edited can be different only in the image portion corresponding to the target semantic attribute, thereby improving the refinement and controllability of face editing.

[0128] In this embodiment, the electronic device can extract the first semantic mask of all the semantic information of the face image to be edited, and the second semantic mask that does not contain the target sub-mask corresponding to the target semantic attribute to be edited from the face image to be edited. Among them, since the second semantic mask does not include the semantic information corresponding to the target semantic attribute of the face image to be edited. Therefore, when the first semantic mask, the second semantic mask and the target semantic attribute are input into the mask generation model for redrawing, the mask generation model can reduce the image information loss corresponding to other semantic attributes that do not need to be edited during the redrawing process as much as possible based on the second semantic mask. Furthermore, it is possible to achieve accurate editing of the mask information corresponding to the target semantic attribute that needs to be edited, so that the third semantic mask after redrawing is different from the first semantic mask in the information on the target semantic attribute that needs to be edited. Finally, the electronic device can perform face editing on the face image to be edited based on the refined third semantic mask and the editing requirements corresponding to the target semantic attribute to obtain the edited face image. Based on this, the above method can accurately and controllably generate a refined intermediate product (third semantic mask) with different image information corresponding to the target semantic attribute to be edited, so that the face image to be edited can be edited based on the refined intermediate product, so that the final edited face image and the face image to be edited only differ in the image part corresponding to the editing requirements, and other image parts that do not need to be edited are as similar as possible. In turn, the refinement and controllability of face editing are improved.

[0129] In another embodiment, in order to train the various sub-models and autoencoders in the mask generation model and the face editing model in the above embodiments, the training process can be divided into three stages.

[0130] First, in the first stage, the electronic device can train the autoencoder in the mask generation model in the NVIDIA A100-40G GPU (graphics processor for AI training). The number of training steps can be 30,000, the batch size can be 2, and the learning rate can be e -5 .

[0131] In the second stage, the first encoding layer and the first decoding layer in the autoencoder can be frozen. For example, the weights in the autoencoder are frozen. Then, the overall mask generation model is trained. This training can be performed on 4 NVIDIA A100-40G GPUs with a batch size of 48, 100,000 training steps, and a learning rate of e -5 And, during the training process, except for the first convolution layer of the first redrawing sub-model (U-Net), the model parameters of all convolution layers can be frozen. And, the model parameters of the first convolution layer are fine-tuned again to achieve the addition of specific objects (for example, preset random noise and concatenation of the third tensor). Specifically, refer to the input of the model in S603 above.

[0132] In the last stage, the face editing model can be trained based on the backbone network of Stable-Diffusion-1.5. Among them, the mask extraction submodel in the face editing model can be a U-Net structure of LDMs (latent diffusion model) that can be trained. During the training process, the mask extraction submodel can take the generated intermediate semantic mask (similar to the fifth semantic mask mentioned above) as a parameter and input it into the corresponding decoding layer in the second redrawing submodel. Among them, only the model parameters in the mask extraction submodel can be optimized during the training process. As an example, the loss function of the mask extraction submodel can be:

[0133]

[0134] Among them, L represents the loss value of the mask extraction sub-model, z0 represents the preset random noise at time 0, and c t represents the vector encoding of the training edit request, c m Represents the vector encoding corresponding to the above intermediate semantic mask.

[0135] See also Figure 8 , Figure 8 1 is a structural block diagram of a face editing device provided in an embodiment of the present application. The face editing device in this embodiment includes modules for executing Figure 1 , Figure 3 , Figure 6 as well as Figure 7 For details, please refer to the steps in the corresponding embodiment. Figure 1 , Figure 3 , Figure 6 as well as Figure 7 as well as Figure 1 , Figure 3 , Figure 6 as well as Figure 7For the convenience of explanation, only the parts related to this embodiment are shown. Figure 8 , the face editing device 800 may include: an extraction module 810, a redrawing module 820 and an editing module 830, wherein:

[0136] The extraction module 810 is used to extract the first semantic mask and the second semantic mask in the face image to be edited; the first semantic mask includes sub-masks corresponding to multiple semantic attributes respectively; the second semantic mask does not include the target sub-mask corresponding to the target semantic attribute to be edited.

[0137] The redrawing module 820 is used to input the editing requirements corresponding to the first semantic mask, the second semantic mask and the target semantic attribute into the mask generation model for redrawing to obtain a third semantic mask; the mask generation model is used to retain the information corresponding to the non-target semantic attribute in the third semantic mask based on the second semantic mask.

[0138] The editing module 830 is used to perform face editing on the face image to be edited based on the third semantic mask and the editing requirement to obtain an edited face image.

[0139] In one embodiment, the mask generation model includes an autoencoder and a first redrawing sub-model; the redrawing module 820 is further used to:

[0140] The first semantic mask and the editing requirement are input into the first redrawing sub-model to obtain the first tensor after redrawing; the second semantic mask is input into the first encoding layer of multiple layers in the autoencoder for processing to obtain the second tensor output by the first encoding layer of each layer; the first encoding layer of each layer is used to encode the input data as the input of the first encoding layer of the next layer; the input of the first encoding layer of the first layer includes the second semantic mask; the autoencoder also includes a first decoding layer symmetrical to the first encoding layer of each layer; the first tensor and the second tensor of each layer are input into the first decoding layers of multiple layers for decoding to obtain the third semantic mask; the input of the first decoding layer of each layer includes the output of the first decoding layer of the previous layer, and the second tensor output by the first encoding layer symmetrical to the first decoding layer of the previous layer; the input of the first decoding layer of the first layer is the first tensor.

[0141] In one embodiment, the face editing device 800 further includes:

[0142] The first input module is used to input the first semantic mask into the multi-layer first encoding layer for processing to obtain a third tensor output by the multi-layer first encoding layer.

[0143] The concatenation module is used to concatenate the preset random noise with the third tensor to obtain a new first semantic mask.

[0144] In one embodiment, the face editing device 800 further includes:

[0145] The processing module is used to perform mask extraction and inverse operation processing on the first semantic mask to obtain a fourth semantic mask; the pixel value corresponding to the mask representing the target semantic attribute in the fourth semantic mask is 0, and the pixel values ​​corresponding to the remaining masks are 1.

[0146] The convolution module is used to perform convolution processing on the second tensor of each layer to obtain the fourth tensor corresponding to each layer.

[0147] The operation module is used to perform mask operation on the fourth tensor of each layer and the fourth semantic mask respectively to obtain a new second tensor corresponding to each layer.

[0148] In one embodiment, the face editing apparatus 800 further includes the following modules to train the first redrawing sub-model:

[0149] The acquisition module is used to acquire training data, where the training data includes a training semantic mask of the first training image, a training editing requirement, and training random noise.

[0150] The second input module is used to input the training data into the initial redrawing sub-model for model training to obtain prediction noise.

[0151] The calculation module is used to calculate the training loss value based on the prediction noise and the training random noise.

[0152] The iteration module is used to iterate the model parameters of the initial redrawing sub-model based on the training loss value until the initial redrawing sub-model meets the convergence condition to obtain the first redrawing sub-model.

[0153] In one embodiment, the editing module 830 is further configured to:

[0154] The third semantic mask is input into the mask extraction sub-model in the face editing model to obtain the fifth semantic mask corresponding to the second decoding layer of each layer; the mask extraction sub-model includes multiple layers of second encoding layers, and a second decoding layer symmetrical to the second encoding layer of each layer; the input of the second encoding layer of each layer is the output of the second encoding layer of the previous layer; the input of the second decoding layer of each layer includes the output of the second encoding layer symmetrical to the second decoding layer of the current layer; the fifth semantic mask of each layer, the target semantic attributes, the editing requirements and the face image to be edited are input into the second redrawing sub-model in the face editing model to obtain the edited face image.

[0155] In one embodiment, the second redrawing sub-model has the same model structure as the mask extraction sub-model; the editing module 830 is further used to:

[0156] The target semantic attributes, editing requirements and the face image to be edited are input into the multi-layer third encoding layer in the second redrawing sub-model to obtain the fifth tensor; the fifth tensor and the multi-layer fifth semantic mask are input into the multi-layer third decoding layer in the second redrawing sub-model to obtain the edited face image; the input of the third encoding layer of each layer is the output of the third encoding layer of the previous layer, and the input of the third decoding layer of each layer is the output of the third decoding layer of the previous layer and the fifth semantic mask output by the second decoding layer at the same layer as the third decoding layer of the current layer.

[0157] When it is understood that Figure 8 In the structural block diagram of the face editing device shown in FIG. Figure 1 , Figure 3 , Figure 6 as well as Figure 7 The steps in the corresponding embodiments, and for Figure 1 , Figure 3 , Figure 6 as well as Figure 7 Each step in the corresponding embodiment has been explained in detail in the above embodiment. Figure 1 , Figure 3 , Figure 6 as well as Figure 7 as well as Figure 1 , Figure 3 , Figure 6 as well as Figure 7 The relevant descriptions in the corresponding embodiments are not repeated here.

[0158] Fig. 9 1 is a structural block diagram of an electronic device provided by an embodiment of the present application. Fig. 9 As shown, the electronic device 900 of this embodiment includes: a processor 910, a memory 920, and a computer program 930 stored in the memory 920 and executable on the processor 910, such as a program of a face editing method. When the processor 910 executes the computer program 930, the steps in each embodiment of the above-mentioned face editing method are implemented, such as Figure 1 Alternatively, the processor 910 implements the above when executing the computer program 930 Figure 8 The functions of each module in the corresponding embodiment are, for example, Figure 8 For details on the functions of each module shown, please refer to Figure 8 Related description in the corresponding embodiment.

[0159] Exemplarily, the computer program 930 may be divided into one or more modules, one or more modules are stored in the memory 920, and are executed by the processor 910 to implement the face editing method provided in the embodiment of the present application. One or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program 930 in the electronic device 900. For example, the computer program 930 may implement the face editing method provided in the embodiment of the present application.

[0160] The electronic device 900 may include, but is not limited to, a processor 910 and a memory 920. Those skilled in the art will appreciate that Fig. 9 It is only an example of the electronic device 900 and does not constitute a limitation of the electronic device 900. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0161] The processor 910 may be a central processing unit, or other general-purpose processors, digital signal processors, application-specific integrated circuits, off-the-shelf programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0162] The memory 920 may be an internal storage unit of the electronic device 900, such as a hard disk or memory of the electronic device 900. The memory 920 may also be an external storage device of the electronic device 900, such as a plug-in hard disk, a smart memory card, a flash memory card, etc. equipped on the electronic device 900. Furthermore, the memory 920 may also include both an internal storage unit of the electronic device 900 and an external storage device.

[0163] An embodiment of the present application provides a computer-readable storage medium, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the face editing method in the above-mentioned embodiments is implemented.

[0164] An embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device executes the face editing method in each of the above embodiments.

[0165] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A face editing method, characterized in that: The method comprises: Extracting a first semantic mask and a second semantic mask from a face image to be edited; the first semantic mask includes sub-masks corresponding to a plurality of semantic attributes respectively; the second semantic mask does not include a target sub-mask corresponding to a target semantic attribute to be edited; Inputting the first semantic mask, the second semantic mask, and the editing requirements corresponding to the target semantic attribute into a mask generation model for redrawing to obtain a third semantic mask; the mask generation model is used to retain information that does not correspond to the target semantic attribute in the third semantic mask based on the second semantic mask; Based on the third semantic mask and the editing requirement, face editing is performed on the face image to be edited to obtain an edited face image.

2. The method according to claim 1, characterized in that: The mask generation model includes an autoencoder and a first redrawing sub-model; the first semantic mask, the second semantic mask, and the editing requirements corresponding to the target semantic attribute are input into the mask generation model for redrawing to obtain a third semantic mask, including: Inputting the first semantic mask and the editing requirement into the first redrawing sub-model to obtain a first tensor after redrawing; The second semantic mask is input into the first encoding layer of multiple layers in the autoencoder for processing, so as to obtain a second tensor output by the first encoding layer of each layer; the first encoding layer of each layer is used to encode the input data as the input of the first encoding layer of the next layer; the input of the first encoding layer of the first layer includes the second semantic mask; the autoencoder also includes a first decoding layer symmetrical to the first encoding layer of each layer; The first tensor and the second tensor of each layer are input into the first decoding layers of multiple layers for decoding processing to obtain the third semantic mask; the input of the first decoding layer of each layer includes the output of the first decoding layer of the previous layer and the second tensor output by the first encoding layer symmetrical to the first decoding layer of the previous layer; the input of the first decoding layer of the first layer is the first tensor.

3. The method according to claim 2, characterized in that Before inputting the first semantic mask and the editing requirement into the first redrawing sub-model to obtain the redrawn first tensor, the method further includes: Inputting the first semantic mask into the first encoding layer of multiple layers for processing to obtain a third tensor output by the first encoding layer of multiple layers; The preset random noise is concatenated with the third tensor to obtain a new first semantic mask.

4. The method according to claim 2, characterized in that: Before inputting the first tensor and the second tensor of each layer to the first decoding layers of multiple layers for decoding processing to obtain the third semantic mask, the method further includes: Performing mask extraction and inverse operation processing on the first semantic mask to obtain a fourth semantic mask; in the fourth semantic mask, the pixel value corresponding to the mask representing the target semantic attribute is 0, and the pixel values ​​corresponding to the remaining masks are 1; Perform convolution processing on the second tensor of each layer respectively to obtain a fourth tensor corresponding to each layer; The fourth tensor of each layer is respectively subjected to a mask operation with the fourth semantic mask to obtain a new second tensor corresponding to each layer.

5. The method according to claim 2, characterized in that: The following steps are also included to train the first redrawing sub-model: Acquire training data, wherein the training data includes a training semantic mask of a first training image, a training editing requirement, and training random noise; Inputting the training data into the initial redrawing sub-model for model training to obtain predicted noise; Calculating a training loss value based on the prediction noise and the training random noise; Iterate the model parameters of the initial redrawing sub-model based on the training loss value until the initial redrawing sub-model meets the convergence condition, thereby obtaining the first redrawing sub-model.

6. The method according to any one of claims 1 to 5, characterized in that: The step of performing face editing on the face image to be edited based on the third semantic mask and the editing requirement to obtain an edited face image includes: The third semantic mask is input into the mask extraction submodel in the face editing model to obtain a fifth semantic mask corresponding to the second decoding layer of each layer; the mask extraction submodel includes multiple layers of second encoding layers, and a second decoding layer symmetrical to the second encoding layer of each layer; the input of the second encoding layer of each layer is the output of the second encoding layer of the previous layer; the input of the second decoding layer of each layer includes the output of the second encoding layer symmetrical to the second decoding layer of the current layer; The fifth semantic mask of each layer, the target semantic attribute, the editing requirement and the face image to be edited are input into the second redrawing sub-model in the face editing model to obtain the edited face image.

7. The method according to claim 6, characterized in that The second redrawing sub-model has the same model structure as the mask extraction sub-model; the fifth semantic mask of each layer, the target semantic attribute, the editing requirement and the face image to be edited are input into the second redrawing sub-model in the face editing model to obtain the edited face image, including: Inputting the target semantic attribute, the editing requirement and the face image to be edited into the multi-layer third encoding layer in the second redrawing sub-model to obtain a fifth tensor; The fifth tensor and the fifth semantic masks of multiple layers are input into the multiple third decoding layers in the second redrawing sub-model to obtain the edited face image; the input of the third encoding layer of each layer is the output of the third encoding layer of the previous layer, and the input of the third decoding layer of each layer is the output of the third decoding layer of the previous layer and the fifth semantic mask output by the second decoding layer at the same layer as the third decoding layer of the current layer.

8. A face editing device, characterized in that: The device comprises: An extraction module, used for extracting a first semantic mask and a second semantic mask from a face image to be edited; the first semantic mask includes sub-masks corresponding to a plurality of semantic attributes respectively; the second semantic mask does not include a target sub-mask corresponding to a target semantic attribute to be edited; A redrawing module, used for inputting the first semantic mask, the second semantic mask and the editing requirements corresponding to the target semantic attribute into a mask generation model for redrawing to obtain a third semantic mask; the mask generation model is used for retaining information not corresponding to the target semantic attribute in the third semantic mask based on the second semantic mask; An editing module is used to perform face editing on the face image to be edited based on the third semantic mask and the editing requirement to obtain an edited face image.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images