Face image generation method and device, medium and equipment

Abstract

The invention relates to the technical field of face generation, in particular to a face image generation method and device, a medium and equipment. Comprising the following steps: inputting a preset face image into a face analysis model to obtain a text prompt and an image prompt; obtaining an initial face feature image and a first loss value according to a preset face image, the text prompt, the image prompt and a face generation model; inputting the initial face feature image into a face analysis model to obtain a second loss value; and inputting a preset face image into the face generation model, and controlling the face generation model to generate a target face image under the constraint of the first loss value and the second loss value. According to the method, the face analysis model is used for assisting the face generation model to generate the face image, the first loss function and the second loss function are used for jointly constraining the face generation model, and the control capability of the face generation process is improved.

Description

Background Art

[0002] With the research and development of artificial intelligence technology, it has become possible for computer algorithm models to automatically generate images, and they have been widely used in film and television entertainment, artistic creation and other fields.

[0003] In the field of image generation, mainstream models based on deep learning include Generative Adversarial Networks (GAN), Variational AutoEncoder (VAE) and Diffusion Model (DDPM). Among them, the diffusion model has attracted the attention of researchers due to its advantages such as more realistic and diverse generated images and more stable training, and has shown excellent performance in most subtasks.

[0004] With the continuous optimization of generative models, face generation tasks have also made significant progress. Deep learning models can extract different facial attribute features by learning a large amount of face data, generate realistic new face images, and even change ambient light and facial expressions. Due to the advantages of diffusion models, face generation methods based on diffusion models have become the latest focus in the field of face generation. However, due to the complexity of facial structure, most existing methods have limited understanding of information description and adequacy of target feature extraction. When restoring images from noise, some subtle features will be ignored, and it is often impossible to accurately generate the facial images that users want, and even unreasonable images such as unnatural expressions and facial distortions will appear. Summary of the invention

[0005] The technical problem to be solved by this application is: how to improve the control capability over the face generation process.

[0006] In view of the above technical problem, according to the first aspect of the present application, a method for generating a face image is provided, the method comprising:

[0007] Input a preset face image into a face parsing model to obtain text prompts and image prompts; wherein the text prompts are used to describe all organ categories included in the preset face image and the pixel contour coordinates corresponding to each organ category; and the image prompts are used to describe the masks corresponding to each organ category in the preset face image;

[0008] According to the preset face image, text prompt, image prompt and face generation model, an initial face feature image and a first loss value are obtained; wherein the first loss value is determined according to the difference between the real noise and the predicted noise in each time step in the process of generating the initial face feature image by the pre-trained face generation model;

[0009] Inputting the initial facial feature image into the face parsing model to obtain a second loss value; wherein the second loss value is determined according to the difference between the preset mask and the initial mask; the preset mask is obtained by processing the preset facial image by the face parsing model; and the initial mask is obtained by processing the initial facial feature image by the face parsing model;

[0010] A preset face image is input into a face generation model, and the face generation model is controlled to generate a target face image under the constraints of a first loss value and a second loss value.

[0011] In an exemplary embodiment of the present application, the inputting of the preset face image into the face analysis model to obtain the text prompt includes:

[0012] Input a preset face image into a face analysis model to obtain the coordinates of each pixel point in the preset face image and the organ category corresponding to each pixel point;

[0013] According to the coordinates of each pixel point and the organ category corresponding to each pixel point, the contour coordinates of the pixel point corresponding to each organ category are obtained;

[0014] Construct a binary group according to each organ category and the corresponding pixel point contour coordinates to obtain a binary group corresponding to each organ category;

[0015] According to the bigram corresponding to each organ category, a text prompt is obtained.

[0016] In an exemplary embodiment of the present application, the step of inputting a preset face image into a face analysis model to obtain an image prompt includes:

[0017] According to the contour coordinates of the pixel points corresponding to each organ category, a mask corresponding to the preset face image is obtained; wherein the marking colors of the mask areas corresponding to any two organ categories in the mask are different;

[0018] The mask is determined as an image prompt corresponding to a preset face.

[0019] In an exemplary embodiment of the present application, the initial facial feature image and the first loss value are obtained according to the preset facial image, text prompt, image prompt and face generation model, including:

[0020] Encoding the preset face image, text prompt, and image prompt to obtain corresponding preset latent space vectors, text embedding vectors, and image embedding vectors;

[0021] An initial facial feature image and a first loss value are obtained according to the above-mentioned preset latent space vector, text embedding vector, image embedding vector and face generation model; wherein the above-mentioned preset latent space vector is used as the input of the face generation model; and the above-mentioned text embedding vector and the above-mentioned image embedding vector are used as input prompts of the face generation model.

[0022] In an exemplary embodiment of the present application, the above-mentioned encoding process of the preset face image, text prompt, and image prompt to obtain the corresponding preset latent space vector, text embedding vector, and image embedding vector includes:

[0023] Encoding the preset face image according to the image encoder to obtain a preset latent space vector;

[0024] Encode the above image prompts according to the image encoder to obtain a mask embedding vector;

[0025] The above text prompts are encoded according to the text encoder to obtain a text embedding vector.

[0026] In an exemplary embodiment of the present application, the initial face feature image and the first loss value are obtained according to the preset latent space vector, the text embedding vector, the image embedding vector and the face generation model, including:

[0027] Controlling the face generation model to add real noise and remove predicted noise to the preset latent space vector to obtain a target latent space vector and a first loss value;

[0028] The target latent space vector is decoded and features are extracted to obtain an initial facial feature image.

[0029] In an exemplary embodiment of the present application, the first loss value is determined according to a first loss function. The first loss function L LDM Meet the following conditions:

[0030]

[0031] Where t represents the time step; z0 represents the image feature of the preset latent space vector without noise; ε t represents the real noise at time t; ε θ represents the prediction noise; z t represents the image feature of the preset latent space vector at time t; T represents text condition control; M represents mask control;

[0032] The above second loss value is determined according to the second loss function. The second loss function L Dice Meet the following conditions:

[0033]

[0034] Wherein, X represents the initial mask, and Y represents the preset mask.

[0035] According to a second aspect of the present application, a facial image generation device is provided, the device comprising:

[0036] A prompt generating unit, used for inputting a preset face image into a face parsing model to obtain a text prompt and an image prompt; wherein the text prompt is used to describe all organ categories included in the preset face image and the pixel contour coordinates corresponding to each organ category; and the image prompt is used to describe the mask corresponding to each organ category in the preset face image;

[0037] An initial image acquisition unit, used to obtain an initial facial feature image and a first loss value according to a preset facial image, a text prompt, an image prompt, and a facial generation model; wherein the first loss value is determined according to a difference between a real noise and a predicted noise in each time step in a process of generating the initial facial feature image by a pre-trained facial generation model;

[0038] A loss function determination unit, used for inputting the initial face feature image into the face parsing model to obtain a second loss value; wherein the second loss value is determined according to the difference between the preset mask and the initial mask; the preset mask is obtained by the face parsing model processing the preset face image; the initial mask is obtained by the face parsing model processing the initial face feature image;

[0039] The target image generation unit is used to input a preset face image into a face generation model, and control the face generation model to generate a target face image under the constraints of a first loss value and a second loss value.

[0040] According to the third aspect of the present application, a non-transitory computer-readable storage medium is provided, in which at least one instruction or at least one program is stored, and the at least one instruction or at least one program is loaded and executed by a processor to implement the above-mentioned facial image generation method.

[0041] According to a fourth aspect of the present application, there is provided an electronic device comprising a processor and the above-mentioned non-transitory computer-readable storage medium.

[0042] This application has at least the following beneficial effects:

[0043] The facial image generation method provided by the present application first inputs a preset facial image into a facial analysis model to obtain text prompts and image prompts. Secondly, based on the preset facial image, text prompts, image prompts and facial generation model, an initial facial feature image and a first loss value are obtained; then, the above initial facial feature image is input into the facial analysis model to obtain a second loss value. Finally, the preset facial image is input into the facial generation model, and the facial generation model is controlled to generate a target facial image under the constraints of the first loss value and the second loss value. The present application utilizes a facial analysis model to assist a facial generation model in generating facial images, and utilizes the first loss function and the second loss function to jointly constrain the facial generation model, thereby improving the control capability over the facial generation process. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 A schematic diagram of a process for generating a facial image according to an embodiment of the present application;

[0046] Figure 2 A schematic diagram of a flow chart of a facial image generation method provided by another embodiment of the present application;

[0047] Figure 3 A schematic diagram of a process for generating text prompts and image prompts provided in one embodiment of the present application;

[0048] Figure 4 A structural block diagram of a facial image generation device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0050] like Figure 1 As shown, an embodiment of the present application provides a method for generating a face image, the method comprising:

[0051] Step S100, inputting a preset face image into a face parsing model to obtain text prompts and image prompts; wherein the text prompts are used to describe all organ categories included in the preset face image and the pixel point contour coordinates corresponding to each organ category; the image prompts are used to describe the masks corresponding to each organ category in the preset face image.

[0052] The preset face image may be any face image randomly selected from a face image library. The face parsing model may segment the various parts of the face (eyes, nose, mouth, eyebrows, etc.) as required. The face parsing model may be any model known to those skilled in the art that can realize face segmentation and parsing, and is not limited here.

[0053] The above inputs the preset face image into the face analysis model to obtain text prompts, including:

[0054] Step S110, inputting a preset face image into a face analysis model to obtain the coordinates of each pixel point in the preset face image and the organ category corresponding to each pixel point.

[0055] Specifically, the organ category may be nose, eyes, mouth, eyebrows, etc. The input of the pre-trained face parsing model is a preset face image, and the output is the coordinates of each pixel point in the preset face image and the organ category corresponding to each pixel point, that is, the segmentation of each facial organ is achieved according to the pre-trained face parsing model, and the position of each organ is determined.

[0056] Step S120 , obtaining the coordinates of the pixel contour corresponding to each organ category according to the coordinates of each pixel and the organ category corresponding to each pixel.

[0057] Specifically, by obtaining the coordinates of each pixel point and the corresponding organ category, all the pixel points corresponding to each organ category can be obtained, and then the contour coordinates of the pixel points corresponding to each organ category can be obtained.

[0058] Step S130 , constructing a binary group according to each organ category and the corresponding pixel point contour coordinates to obtain a binary group corresponding to each organ category.

[0059] Specifically, each organ category and the corresponding pixel point set are grouped into a tuple to realize the correspondence between the organ category and the pixel point set, and associate the two.

[0060] Step S140, obtaining a text prompt according to the bigram corresponding to each organ category.

[0061] Specifically, as an example: the corresponding text prompt is "an image containing {target}", where the target is all the bigrams corresponding to the preset face image.

[0062] In this embodiment, by constructing a binary group of organ categories and corresponding pixel points, the association between organ categories and corresponding pixel point sets is achieved, which can fully characterize the organ distribution of the preset face image and form an effective text prompt of the organ distribution of the preset face image.

[0063] In addition, the above-mentioned input of the preset face image into the face analysis model to obtain the image prompt includes:

[0064] Step S150, obtaining a mask corresponding to the preset face image according to the outline coordinates of the pixel points corresponding to each organ category; wherein the marking colors of the mask areas corresponding to any two organ categories in the mask are different.

[0065] Specifically, according to the coordinates of each pixel point in the pixel point contour coordinates corresponding to each organ category, a polygonal contour corresponding to each organ category is obtained, and then according to the polygonal contour corresponding to each organ category, a mask corresponding to the preset face image is obtained; wherein, the marking colors of the mask areas corresponding to any two organ categories in the mask are different.

[0066] Step S160: determining the mask as an image prompt corresponding to a preset face.

[0067] Step S200, obtaining an initial facial feature image and a first loss value based on a preset facial image, text prompts, image prompts, and a facial generation model; wherein the first loss value is determined based on the difference between the actual noise and the predicted noise in each time step in the process of generating the initial facial feature image by the pre-trained facial generation model.

[0068] In one embodiment of the present application, the face generation model is a stable diffusion model (Stable Diffusion Model), which is a probabilistic framework based on the above diffusion model (Denoising Diffusion Probabilistic Model DDPM), and the training of the model includes two main parts: forward diffusion and reverse diffusion. In the forward diffusion stage, the model destroys the image by gradually introducing noise (real noise) until the image becomes completely random noise. In the reverse diffusion stage, the model uses a series of Markov chains to gradually remove the prediction noise and recover the data from the Gaussian noise, thereby generating a high-quality image. Step S200 also includes:

[0069] Step S210, encoding the above-mentioned preset face image, text prompt, and image prompt to obtain corresponding preset latent space vectors, text embedding vectors, and image embedding vectors.

[0070] Specifically, there are different encoding methods for images and texts, as shown in the following steps:

[0071] Step S211, encoding the preset face image according to the image encoder to obtain a preset latent space vector.

[0072] Step S212: Encode the image prompt according to the image encoder to obtain a mask embedding vector.

[0073] Step S213, encoding the text prompt according to the text encoder to obtain a text embedding vector.

[0074] Step S220, obtaining an initial facial feature image and a first loss value according to the above-mentioned preset latent space vector, text embedding vector, image embedding vector and face generation model; wherein the above-mentioned preset latent space vector is used as the input of the face generation model; and the above-mentioned text embedding vector and the above-mentioned image embedding vector are used as input prompts of the face generation model.

[0075] Here, the preset latent space vector is used as the input of the face generation model; the text embedding vector and the image embedding vector are used as input prompts of the face generation model. That is, the image embedding vector is used as the layout information of face generation, and the text embedding vector is used as the text information of face generation.

[0076] In this embodiment, the text embedding vector and the image embedding vector are used together as input prompts for the face generation model, so that the input prompts of the two modalities complement each other, thereby achieving a better control effect on face generation.

[0077] Step S221, controlling the above-mentioned face generation model to add real noise and remove predicted noise to the above-mentioned preset latent space vector to obtain a target latent space vector and a first loss value.

[0078] First, Gaussian noise (real noise) is gradually added to the above-mentioned preset latent space vector in each time step. As the noise is gradually added, the image data gradually becomes blurred and noisy until it finally approaches the standard Gaussian distribution, that is, the preset latent space vector after noisy is obtained.

[0079] Furthermore, under the guidance of the text embedding vector and the image embedding vector, the stable diffusion model calculates the noise that needs to be removed from the preset latent space vector after adding noise at each time step, and calculates the first loss function of the stable diffusion model, wherein the stable diffusion model uses the U-Net structure to remove noise. U-Net is an important network in the field of image segmentation, which can obtain features of different receptive fields at the same level and pass them to the next layer after fusion. Therefore, in the process of removing noise, the multi-scale feature fusion map of the newly generated image can be extracted. Among them, the first loss value is determined according to the first loss function, which is the mean square error between the noise predicted by the calculation model (removed noise) and the corresponding real noise (added noise). The first loss function L LDM Meet the following conditions:

[0080] L LDM =E t,z0,ε,T,M [||ε t -ε θ (z t ,t,T,M)|| 2 ]

[0081] Where t represents the time step; z0 represents the image feature of the preset latent space vector without noise; ε t represents the real noise at time t; ε θ represents the prediction noise, that is, the noise removed at each time step; z t represents the image features of the preset latent space vector at time t; T represents text condition control; M represents mask control.

[0082] The noise is gradually removed to obtain the target latent space vector.

[0083] Step S222, decoding the target latent space vector and performing feature extraction to obtain an initial facial feature image.

[0084] Specifically, when decoding the target latent space vector and performing feature extraction, three layers of multi-scale features are extracted, and the final initial face feature image is obtained through convolution and continuous upsampling, so that the feature information corresponding to the extracted initial face feature image is richer.

[0085] Step S300, input the above-mentioned initial facial feature image into the face parsing model to obtain a second loss value; wherein the above-mentioned second loss value is determined based on the difference between a preset mask and an initial mask; the above-mentioned preset mask is obtained by the face parsing model processing the preset facial image; the above-mentioned initial mask is obtained by the face parsing model processing the initial facial feature image.

[0086] Specifically, the initial facial feature image is input into the face parsing model, and the similarity between the output initial mask and the preset mask is obtained to obtain the second loss value, which is determined according to the second loss function. The second loss function L Dice Meet the following conditions:

[0087]

[0088] Wherein, X represents the initial mask, and Y represents the preset mask.

[0089] Step S400: input a preset face image into a face generation model, and control the face generation model to generate a target face image under the constraints of a first loss value and a second loss value.

[0090] Therefore, in this embodiment, a comprehensive loss function L is obtained according to the first loss function and the second loss function, where L meets the following conditions:

[0091] L=L LDM +λL Dice

[0092] Among them, λ is a hyperparameter in training, which is used to adjust the influence of the perception part on the generation process.

[0093] This embodiment constrains the above-mentioned face generation model according to the comprehensive loss function L to obtain the target face image.

[0094] The following describes the face image generation process of the present application according to a specific embodiment. Figure 2 As shown, the original image img (preset face image) is input into the face parsing model P(·) to obtain text prompts and pixel-level masks (image prompts); wherein c1 is the first organ category, l1 is the set of pixels corresponding to the first organ category; c2 is the second organ category, l2 is the set of pixels corresponding to the second organ category; and so on, n is the number of organ categories. The text prompt is encoded according to the text editor, and the original image img (preset face image) and the pixel-level mask (image prompt) are encoded according to the image encoder to obtain text embedding T (text embedding vector), image latent space vector Z (preset latent space vector) and mask embedding M (image embedding vector), and then, according to the image generator G(·) (face generation model), a new image (target face image) is generated according to the comprehensive loss function obtained by the first loss function and the second loss function. Wherein, as Figure 3 As shown, it is a diagram showing the generation process of text prompts and image prompts.

[0095] In summary, this application proposes a method for using a face parsing model to assist in generation, which improves the control ability of perceptual features over the generation process, combines the advantages of text-generated images and mask-generated images, avoids text ambiguity, and optimizes the layout of detail processing, integrates the generation model and the segmentation model, and uses the generation loss and the segmentation loss to jointly influence face generation.

[0096] In addition, although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0097] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0098] Please refer to Figure 4 As shown, an embodiment of the present application provides a facial image generation device 100, the device comprising:

[0099] The prompt generation unit 110 is used to input a preset face image into a face parsing model to obtain text prompts and image prompts; wherein the above-mentioned text prompts are used to describe all organ categories included in the above-mentioned preset face image and the pixel point contour coordinates corresponding to each organ category; the above-mentioned image prompts are used to describe the masks corresponding to each organ category in the above-mentioned preset face image.

[0100] The initial image acquisition unit 120 obtains an initial facial feature image and a first loss value according to a preset facial image, text prompts, image prompts and a facial generation model; wherein the first loss value is determined based on the difference between the actual noise and the predicted noise in each time step in the process of generating the initial facial feature image by the pre-trained facial generation model.

[0101] The loss function determination unit 130 inputs the above-mentioned initial facial feature image into the face parsing model to obtain a second loss value; wherein the above-mentioned second loss value is determined based on the difference between a preset mask and an initial mask; the above-mentioned preset mask is obtained by the face parsing model processing the preset facial image; the above-mentioned initial mask is obtained by the face parsing model processing the initial facial feature image.

[0102] The target image generating unit 140 is used to input a preset face image into the face generating model, and control the face generating model to generate a target face image under the constraints of the first loss value and the second loss value.

[0103] In an exemplary embodiment of the present application, the above text prompt is obtained according to the following subunits:

[0104] The pixel coordinate acquisition subunit is used to input a preset face image into the face analysis model to obtain the coordinates of each pixel in the preset face image and the organ category corresponding to each pixel.

[0105] The pixel point set determination subunit is used to obtain the coordinates of the pixel point contour corresponding to each organ category according to the coordinates of each pixel point and the organ category corresponding to each pixel point.

[0106] The binary group acquisition subunit is used to construct a binary group according to each organ category and the corresponding pixel point contour coordinates to obtain the binary group corresponding to each organ category.

[0107] The text prompt acquisition subunit is used to obtain a text prompt according to the binary group corresponding to each organ category.

[0108] In an exemplary embodiment of the present application, the above-mentioned image prompt is obtained according to the following sub-units.

[0109] The mask image acquisition subunit is used to obtain a mask corresponding to a preset face image according to the outline coordinates of the pixel points corresponding to each organ category; wherein the marking colors of the mask areas corresponding to any two organ categories in the mask are different.

[0110] The image prompt acquisition subunit is used to determine the above mask as the image prompt corresponding to the preset face.

[0111] In an exemplary embodiment of the present application, the initial image acquisition unit includes:

[0112] The encoding subunit is used to encode the above-mentioned preset face image, text prompt, and image prompt to obtain the corresponding preset latent space vector, text embedding vector, and image embedding vector.

[0113] The initial image acquisition subunit is used to obtain an initial facial feature image and a first loss value according to the above-mentioned preset latent space vector, text embedding vector, image embedding vector and face generation model; wherein the above-mentioned preset latent space vector is used as the input of the face generation model; and the above-mentioned text embedding vector and the above-mentioned image embedding vector are used as input prompts of the face generation model.

[0114] In an exemplary embodiment of the present application, the encoding subunit includes:

[0115] The first encoding component is used to encode the above-mentioned preset face image according to the image encoder to obtain a preset latent space vector.

[0116] The second encoding component is used to encode the above image prompt according to the image encoder to obtain a mask embedding vector.

[0117] The third encoding component is used to encode the above text prompt according to the text encoder to obtain a text embedding vector.

[0118] In an exemplary embodiment of the present application, the initial image acquisition subunit includes:

[0119] The noise processing component is used to control the above-mentioned face generation model to add real noise and remove predicted noise to the above-mentioned preset latent space vector to obtain a target latent space vector and a first loss value.

[0120] The initial image acquisition component is used to decode the target latent space vector and extract features to obtain an initial facial feature image.

[0121] In an exemplary embodiment of the present application, the first loss value is determined according to a first loss function. The first loss function L LDM Meet the following conditions:

[0122]

[0123] Where t represents the time step; z0 represents the image feature of the preset latent space vector without noise; ε t represents the real noise at time t; ε θ represents the prediction noise; z t represents the image feature of the preset latent space vector at time t; T represents text condition control; M represents mask control;

[0124] The above second loss value is determined according to the second loss function. The second loss function L Dice Meet the following conditions:

[0125]

[0126] Wherein, X represents the initial mask, and Y represents the preset mask.

[0127] In an exemplary embodiment of the present application, an electronic device capable of implementing the above method is also provided.

[0128] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0129] The electronic device according to this embodiment of the present application is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0130] The electronic device is presented in the form of a general-purpose computing device. The components of the electronic device may include, but are not limited to: the at least one processor mentioned above, the at least one storage device mentioned above, and a bus connecting different system components (including storage devices and processors).

[0131] The storage device stores program codes, which can be executed by the processor, so that the processor executes the steps described in the “exemplary method” section of the specification according to various exemplary embodiments of the present application.

[0132] The memory may include readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory, and may further include read only memory (ROM).

[0133] The storage may also include a program / utility having a set (at least one) of program modules, such program modules including but not limited to: an operating system, one or more application programs, other program modules and program data, each of which or some combination may include the implementation of a network environment.

[0134] The bus may represent one or more of several types of bus structures including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures.

[0135] The electronic device may also communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may communicate with one or more devices that enable a user to interact with the electronic device, and / or may communicate with any device (e.g., routers, modems, etc.) that enables the electronic device to communicate with one or more other computing devices. Such communication may be performed through an input / output (I / O) interface. Furthermore, the electronic device may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) through a network adapter. The network adapter communicates with other modules of the electronic device through a bus. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0136] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0137] In an exemplary embodiment of the present application, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present application can also be implemented in the form of a program product, which includes a program code. When the above program product is run on a terminal device, the above program code is used to enable the above terminal device to execute the steps according to various exemplary implementations of the present application described in the above "Exemplary Method" section of the present specification.

[0138] The above program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0139] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0140] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0141] Program code for performing the operations of the present application may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0142] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present application, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0143] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0144] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for generating a face image, characterized in that: The method comprises: Input a preset face image into a face parsing model to obtain a text prompt and an image prompt; wherein the text prompt is used to describe all organ categories included in the preset face image and the pixel contour coordinates corresponding to each organ category; and the image prompt is used to describe the mask corresponding to each organ category in the preset face image; According to the preset face image, text prompt, image prompt and face generation model, an initial face feature image and a first loss value are obtained; wherein the first loss value is determined according to the difference between the real noise and the predicted noise in each time step in the process of generating the initial face feature image by the pre-trained face generation model; Inputting the initial facial feature image into the face parsing model to obtain a second loss value; wherein the second loss value is determined according to the difference between the preset mask and the initial mask; the preset mask is obtained by the face parsing model processing the preset facial image; the initial mask is obtained by the face parsing model processing the initial facial feature image; A preset face image is input into a face generation model, and the face generation model is controlled to generate a target face image under the constraints of a first loss value and a second loss value.

2. The method for generating a facial image according to claim 1, characterized in that: Input the preset face image into the face parsing model to obtain text prompts, including: Input a preset face image into a face analysis model to obtain the coordinates of each pixel point in the preset face image and the organ category corresponding to each pixel point; According to the coordinates of each pixel point and the organ category corresponding to each pixel point, the contour coordinates of the pixel point corresponding to each organ category are obtained; Construct a binary group according to each organ category and the corresponding pixel point contour coordinates to obtain a binary group corresponding to each organ category; According to the bigram corresponding to each organ category, a text prompt is obtained.

3. The method for generating a facial image according to claim 1, wherein: Input the preset face image into the face parsing model to obtain image prompts, including: According to the contour coordinates of the pixel points corresponding to each organ category, a mask corresponding to the preset face image is obtained; wherein the marking colors of the mask areas corresponding to any two organ categories in the mask are different; The mask is determined as an image prompt corresponding to a preset human face.

4. The method for generating a facial image according to any one of claims 1 to 3, characterized in that: The method of obtaining an initial facial feature image and a first loss value according to a preset facial image, a text prompt, an image prompt, and a facial generation model includes: Encoding the preset face image, text prompt, and image prompt to obtain corresponding preset latent space vectors, text embedding vectors, and image embedding vectors; An initial facial feature image and a first loss value are obtained according to the preset latent space vector, the text embedding vector, the image embedding vector and the face generation model; wherein the preset latent space vector is used as the input of the face generation model; and the text embedding vector and the image embedding vector are used as input prompts of the face generation model.

5. The method for generating a facial image according to claim 4, characterized in that: The encoding process of the preset face image, text prompt, and image prompt to obtain the corresponding preset latent space vector, text embedding vector, and image embedding vector includes: Encoding the preset face image according to an image encoder to obtain a preset latent space vector; encoding the image prompt according to an image encoder to obtain a mask embedding vector; The text prompt is encoded according to a text encoder to obtain a text embedding vector.

6. The method for generating a facial image according to claim 4, characterized in that: The step of obtaining an initial facial feature image and a first loss value according to the preset latent space vector, the text embedding vector, the image embedding vector and the face generation model includes: Controlling the face generation model to add real noise and remove predicted noise to the preset latent space vector to obtain a target latent space vector and a first loss value; The target latent space vector is decoded and features are extracted to obtain an initial facial feature image.

7. The method for generating a facial image according to claim 1, characterized in that: The first loss value is determined according to a first loss function. The first loss function L LDM Meet the following conditions: Where t represents the time step; z0 represents the image feature of the preset latent space vector without noise; ε t represents the real noise at time t; ε θ represents the prediction noise; z t represents the image feature of the preset latent space vector at time t; T represents text condition control; M represents mask control; The second loss value is determined according to a second loss function, where the second loss function L Dice Meet the following conditions: Wherein, X represents the initial mask, and Y represents the preset mask.

8. A facial image generating device, characterized in that: The device comprises: A prompt generating unit, used for inputting a preset face image into a face parsing model to obtain a text prompt and an image prompt; wherein the text prompt is used to describe all organ categories included in the preset face image and the contour coordinates of the pixel points corresponding to each organ category; and the image prompt is used to describe the mask corresponding to each organ category in the preset face image; An initial image acquisition unit, used to obtain an initial facial feature image and a first loss value according to a preset facial image, a text prompt, an image prompt, and a facial generation model; wherein the first loss value is determined according to a difference between a real noise and a predicted noise in each time step in a process of generating the initial facial feature image by a pre-trained facial generation model; A loss function determination unit, configured to input the initial face feature image into a face parsing model to obtain a second loss value; wherein the second loss value is determined according to a difference between a preset mask and an initial mask; the preset mask is obtained by processing the preset face image by the face parsing model; and the initial mask is obtained by processing the initial face feature image by the face parsing model; The target image generation unit is used to input a preset face image into a face generation model, and control the face generation model to generate a target face image under the constraints of a first loss value and a second loss value.

9. A non-transitory computer-readable storage medium, wherein at least one instruction or at least one program is stored in the storage medium, characterized in that: The at least one instruction or the at least one program is loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: The invention comprises a processor and the non-transitory computer-readable storage medium as claimed in claim 9.

Images