Face image generation method and device, and electronic equipment

Abstract

The embodiment of the invention provides a face image generation method. The face image generation method comprises the following steps: obtaining a cue word and a reference image; inputting the cue word and the reference image into a pre-trained face image generation model, wherein the face image generation model comprises a text coding module, a face processing module and a diffusion module; and generating a face image according to the cue word and the reference image through calculation of a diffusion model of the face image generation model. According to the face image generation method provided by the embodiment of the invention, the subsurface space tensor obtained by the text coding model is amplified, so that the generation quality of the face image of the diffusion model is improved, and the fidelity and editability of the face image are improved. The embodiment of the invention further provides a face image generation device and electronic equipment.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and more specifically, to a method, device, and electronic device for generating a facial image. Background Art

[0002] With the rapid development of artificial intelligence technology, image generation and editing technology has been widely used in various fields. However, the existing face image generation and editing technology still has many shortcomings, such as low quality of generated images, low face recognition, poor detail expression and other problems. As a cutting-edge diffusion model, the Stable Diffusion model has achieved remarkable achievements in the field of image generation, but it still needs further optimization and improvement to meet the specific needs of face image generation and editing. Summary of the invention

[0003] In response to the problems existing in the above-mentioned prior art, the embodiments of the present application provide a facial image generation method, device, and electronic device, which amplifies the latent space tensor obtained by the text encoding model, thereby improving the generation quality of the facial image of the diffusion model and improving the fidelity and editability of the facial image.

[0004] In a first aspect, an embodiment of the present application provides a method for generating a face image, comprising the following steps:

[0005] Get cue words and reference images;

[0006] Input the prompt word and the reference image into a pre-trained face image generation model, the face image generation model includes a text encoding module, a face processing module and a diffusion module, wherein the prompt word and the reference image are processed by the text encoding module to obtain a first latent space tensor, and the reference image is processed by the face processing module to obtain a second latent space tensor and a third latent space tensor, and the first latent space tensor, the second latent space tensor and the third latent space tensor constitute the conditions for inputting the diffusion module;

[0007] A face image is generated according to the prompt word and the reference image and by calculating a diffusion model of the face image generation model.

[0008] Furthermore, the face processing module includes a face identity submodule and a face detail submodule, the reference image is processed by the face identity submodule to obtain the second latent space tensor, and the reference image is processed by the face detail submodule to obtain the third latent space tensor.

[0009] Furthermore, the face identity submodule includes a face recognition module, a first self-attention module and a first multi-layer perceptron, and the reference image is processed by the face identity submodule to obtain the second latent space tensor, including:

[0010] The face of the reference image is mapped to a latent space vector expressing the face through the face recognition module, and the latent space vector is mapped to the latent space of the text encoding module through the first self-attention module and the first multi-layer perceptron to obtain the second latent space tensor.

[0011] Furthermore, the face detail submodule includes a CLIP visual model, a second self-attention module and a second multi-layer perceptron, and the reference image is processed by the face detail submodule to obtain the third latent space tensor, including:

[0012] The features of the face of the reference image processed by the CLIP visual model are input into the second self-attention module and the second multi-layer perceptron for processing, and mapped to the latent space of the text encoding module to obtain the third latent space tensor.

[0013] Furthermore, the generating of the face image according to the prompt word and the reference image and by calculation of the diffusion model of the face image generation model includes:

[0014] A face image is generated according to the prompt word and the reference image and through the reverse diffusion inference of the diffusion model of the face image generation model.

[0015] Furthermore, the training process of the face image generation model includes:

[0016] Constructing a training data set, including obtaining a preset number of face images, and generating description words according to the face images to form image-text pairs as the training data set;

[0017] The training data set is input into the diffusion model for training, and the diffusion model is optimized by a preset loss function to obtain the trained face image generation model.

[0018] Furthermore, the preset loss function L=L sd +λ diff L diff +λ id L id , where λ diff and λ id is the weight value of the preset loss function, L sd is the loss function of the standard diffusion model, L diffis the constraint on the difference between the training model and the reference model, L diff Used to maintain the facial similarity of the facial image.

[0019] In a second aspect, the embodiment of the present application further provides a facial image generation device, comprising:

[0020] A transition acquisition module is used to obtain the transition information of the video;

[0021] A sound effect reading module, used for reading the transition sound effect from the transition information; and

[0022] The sound effect loading module is used to load the transition sound effect according to the association relationship between the transition information and the transition sound effect.

[0023] In a third aspect, an embodiment of the present application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor is configured to implement the facial image generation method according to the first aspect when executing the program.

[0024] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, wherein the computer program is used to implement the facial image generation method according to the first aspect above.

[0025] The embodiments of the present application bring the following beneficial effects:

[0026] In the face image generation method provided in the embodiment of the present application, a prompt word and a reference image are first obtained, and then the prompt word and the reference image are input into a pre-trained face image generation model, and a face image is generated based on the prompt word and the reference image and through the calculation of the diffusion model of the face image generation model. The face image generation method provided in the embodiment of the present application improves the generation quality of the face image of the diffusion model by amplifying the latent space tensor obtained by the text encoding model, thereby improving the fidelity and editability of the face image. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying any creative work.

[0028] Figure 1 A schematic diagram of a process for generating a facial image according to an embodiment of the present invention;

[0029] Figure 2 A schematic diagram of the structure of the inference framework of the face image generation model adopted by the face image generation method provided in the embodiment of the present application;

[0030] Figure 3 A schematic diagram of the structure of a face identity submodule used in the face image generation method provided in an embodiment of the present application;

[0031] Figure 4 A schematic diagram of the structure of a face detail submodule used in the face image generation method provided in an embodiment of the present application;

[0032] Figure 5 A schematic diagram of the structure of an integrated self-attention module used in the face image generation method provided in an embodiment of the present application;

[0033] Figure 6 A structural block diagram of a facial image generation device provided in an embodiment of the present application;

[0034] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0035] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments described in the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0037] In the specification and claims of this application and the above-mentioned drawings, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "multiple" means two or more. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to the specific circumstances.

[0038] Figure 1 FIG. 1 is a flowchart of a method for generating a face image according to an embodiment of the present application. Figure 1 As shown, the face image generation method of the embodiment of the present application is used for gesture recognition training, including the following steps:

[0039] S101: Obtain prompt words and reference images;

[0040] Prompts are short sentences or questions used to stimulate creativity, guide thinking, or promote specific actions or reactions. Reference images are face images provided by users, so that the face images expected by users can be generated according to the prompts.

[0041] S102: inputting the prompt word and the reference image into a pre-trained face image generation model;

[0042] Specifically, after the user provides a prompt word and a reference image, the prompt word and the reference image need to be input into a pre-trained face image generation model to generate the face image desired by the user.

[0043] The face image generation model includes a text encoding module, a face processing module and a diffusion module, wherein the prompt word and the reference image are processed by the text encoding module to obtain a first latent space tensor, and the reference image is processed by the face processing module to obtain a second latent space tensor and a third latent space tensor, and the first latent space tensor, the second latent space tensor and the third latent space tensor constitute the conditions for inputting the diffusion module.

[0044] The latent space tensors in the diffusion model are vectors or tensors that represent the representation of an image in the latent space. They are gradually transformed into the target image through the diffusion process, and in the process, they capture the high-level features and structure of the image. In the diffusion model, the latent space tensor Z (and its iterative Z') plays a vital role. They not only capture the latent features and structure of the image, but are also gradually transformed into the target image through the diffusion process. This latent representation can be easily combined with other information (such as text embedding and time embedding), allowing the model to generate corresponding images based on the input conditions.

[0045] In the face image generation method provided in the embodiment of the present application, the latent space tensor obtained by the text encoding model is amplified, thereby improving the generation quality of the face image of the diffusion model, and improving the fidelity and editability of the face image.

[0046] S103: Generate a face image according to the prompt word and the reference image and by calculating a diffusion model of the face image generation model.

[0047] When the user provides a prompt word and a reference image and inputs the face image generation model, the face image generation model needs to be calculated or inferred through the diffusion model to generate the face image expected by the user.

[0048] Therefore, in the face image generation method provided in the embodiment of the present application, a prompt word and a reference image are first obtained, and then the prompt word and the reference image are input into a pre-trained face image generation model, and a face image is generated based on the prompt word and the reference image and through the calculation of the diffusion model of the face image generation model. The face image generation method provided in the embodiment of the present application amplifies the latent space tensor obtained by the text encoding model, thereby improving the generation quality of the face image of the diffusion model and improving the fidelity and editability of the face image.

[0049] Reference Figure 2 , the face image generation model adopted by the face image generation method provided in the embodiment of the present application includes a text encoding module (Text Encoder), a face processing module and a diffusion module, and the diffusion model is composed of a UNet module and a VAE (Variational Autoencoder) module. The UNet model can effectively capture the feature information of different scales in the image. The main function of the VAE encoding module is to calculate the low-dimensional mean and variance of the input data so as to sample from the latent variable space and generate new data. Conditional semantic information is more suitable for being introduced through the cross-attention part in UNet and affecting the diffusion process, while the self-attention part in UNet has a considerable influence on the spatial structure and texture information of the generated graph. Therefore, the embodiment of the present application has made a series of improvements around UNet. On the one hand, the latent space condition tensor obtained by the text encoder or text encoding module is amplified, and the relevant weights of the cross-attention inside UNet are updated accordingly. On the other hand, the self-attention module inside UNet is transformed, and the relevant weights of the self-attention part are updated accordingly.

[0050] Furthermore, in some embodiments of the present application, the face processing module includes a face identity submodule and a face detail submodule, the reference image is processed by the face identity submodule to obtain the second latent space tensor, and the reference image is processed by the face detail submodule to obtain the third latent space tensor.

[0051] Specifically, there are two inputs in the inference process, one is the prompt word, set as y, and the other is the face image provided by the user, called the reference image, set as x. y passes through the text encoder to obtain the latent space tensor c y =τ(y), where τ refers to the text encoder. The reference image x passes through the face segmentation module, the non-face area is set to random noise, and the corresponding latent space tensor z* is obtained through the VAE encoding module. The reference image x passes through the face recognition to the face identity submodule to obtain the latent space tensor c i =f i (x), where fi Refers to the face recognition and face identity submodules. The reference image x is processed by CLIP Vision (Contrastive Language-Image Pre-Training) and the face detail submodule to obtain the latent space tensor c d =f d (x), where f d Refers to the CLIPVision and face detail submodules, and the condition c=concat(c y , c i , c d ).

[0052] Further, refer to Figure 3 In some embodiments of the present application, the face identity submodule includes a face recognition module, a first self-attention module (SelfAttention) and a first multilayer perceptron (Multilayer Perceptron, MLP), and the reference image is processed by the face identity submodule to obtain the second latent space tensor, including:

[0053] The face of the reference image is mapped to a latent space vector expressing the face through the face recognition module, and the latent space vector is mapped to the latent space of the text encoding module through the first self-attention module and the first multi-layer perceptron to obtain the second latent space tensor.

[0054] The ID embedding module is a latent space vector that maps a face to a face by the face recognition module. This vector can be used to calculate face similarity and therefore contains information for identifying a face. This vector is mapped to the latent space of the text encoder through the first self-attention module and the first multi-layer perceptron (MLP). A point in the latent space of the text encoder can be regarded as a word. In the embodiment of the present application, four words can be used to express the face identity information. The converted face identity information is the c mentioned above. i .

[0055] Further, refer to Figure 4 In some embodiments of the present application, the face detail submodule includes a CLIP visual model, a second self-attention module, and a second multi-layer perceptron, and the reference image is processed by the face detail submodule to obtain the third latent space tensor, including:

[0056] The features of the face of the reference image processed by the CLIP visual model are input into the second self-attention module and the second multi-layer perceptron for processing, and mapped to the latent space of the text encoding module to obtain the third latent space tensor.

[0057] Specifically, although the face identity information contains information related to face recognition, some local details of the facial features, as well as hair, ears, neck and other areas are missing. On the one hand, these features do not contribute much to recognition, and on the other hand, hair, ears, neck and other areas are generally not within the feature extraction range of the face recognition module. Therefore, in the face detail submodule, the features of the last several layers of the CLIP visual model are used as the input features of this module. After two second multi-layer perceptrons (MLP) and the second self-attention module, they are mapped to the latent space of the text encoder. The converted face detail features are the c mentioned above. d For the features extracted from the visual model of CLIP, the embodiment of the present application uses the features of the last 6 layers to map to the latent space of the text encoder.

[0058] Furthermore, in some embodiments of the present application, generating a face image according to the prompt word and the reference image and by calculating a diffusion model of the face image generation model includes:

[0059] A face image is generated according to the prompt word and the reference image and through the reverse diffusion inference of the diffusion model of the face image generation model.

[0060] Specifically, the sampling step number of the diffusion model is set to T,z t represents the latent space variable at step t. The inference stage is a reverse diffusion process, starting from z T Start with z t Calculate z t-1 , step by step, we gradually get z 0 process. T Initialize to a sample of the standard Gaussian distribution, i.e. z T ~N(0,I). Suppose the predefined signal-to-noise ratio parameter of the planner is α t ,σ t T t =1.

[0061] For the tth step of the diffusion process, the reasoning process is as follows:

[0062] 1.z* t =α t z*+σ t ∈, where ∈~N(0,I);

[0063] 2. z tand z*t in batch mode, that is, connected on the 0th dimension, z' t ←concat(z t ,z*t),dim=0;

[0064] 3. z' t , t, c input UNet, that is, ∈θ(z' t , t, c), take out the first element of the 0th dimension (i.e., batch dimension) of the result tensor, denoted as u;

[0065] 4. When running UNet, its cross-attention module is divided into two parallel parts. The two parts share the input query sequence, but have their own independent key / value calculation units. The two parts receive c y and (c i ,c d ) as the input of the key / value calculation unit. Therefore, c is divided into two parts, one part is c y , part of it is (c i , c d ), the former obtains p through the cross attention module of UNet y , the latter is obtained by cross-attention calculation through the key / value calculation unit customized by the present invention x , α y p y +α x p x Passed to the next module as the output of the entire criss-cross attention module;

[0066] 5. According to the predefined planner, input z t , u, t, according to the logic of the planner, through simple algebraic calculation, we can get z t -1.

[0067] The parameter α in the above process y and α x They are the weight of the text prompt word and the weight of the face information. These two parameters are generally set to 1. In actual applications, end users can adjust them by themselves to obtain results that better meet their specific needs.

[0068] Further, refer again to Figure 2 In some embodiments of the present application, the training process of the face image generation model includes:

[0069] Constructing a training data set, including obtaining a preset number of face images, and generating description words according to the face images to form image-text pairs as the training data set;

[0070] The training data set is input into the diffusion model for training, and the diffusion model is optimized by a preset loss function to obtain the trained face image generation model.

[0071] Specifically, Figure 2 As shown in Figure 1, the whole training process is divided into two stages. The first stage is to adjust the weights, and the second stage is to adjust the weights. The training data set can be obtained from the Internet or open data sets. The pictures with multiple people or no people are removed through tools such as face detection. For example, about 10 million single-person pictures are used, and BLIP-2 (Bootstrapping Language-Image Pre-training, language-image guided pre-training model) is used to automatically generate description words for them to form image-text pairs as the training set of the diffusion model.

[0072] In the first stage of training, the self-attention module inside UNet uses a standard implementation, so when UNet is running, it is only different from the description in the previous reasoning process. The entire training process uses the standard SD training process, but there are three differences: 1. The structure is different, as mentioned above; 2. A reference model is required; 3. The loss function used is different. Training starts with the existing standard diffusion model. The model is first copied as the reference model, and the remaining one is transformed according to the previous architecture as the training model. Both models participate in the training, but the weights of the reference model are locked, and some weights in the training model are updated during the training process. The ∈ mentioned below theta refers to the training model, ∈ ref It refers to the reference model.

[0073] In the second phase of training, the self-attention module of the UNet of the training model is transformed as described above, that is, the structure and process are the same as those in the inference phase. In addition, the weights of the modules adjusted in the first phase are kept locked in this phase. Apart from these differences, the training process is the same as the first phase, including the loss function.

[0074] Reference Figure 5 In practice, it is found that after the first two modules are processed, the generated face has a good degree of recognition, but the subjective feeling of the face is still obviously lacking in realism, and some details such as skin texture or freckles, moles, etc., including the spatial position of these details, are not well grasped. The self-attention module has a significant impact on the spatial relationship and some texture details of the generated image, so the self-attention module of the diffusion model is modified, such as Figure 5As shown, z represents the standard Wensheng image data flow, and z* represents the data flow corresponding to the reference portrait image. Before entering the self-attention module and after leaving the self-attention module, both form a tensor in a batch manner, and the part outside the self-attention module still performs UNet related calculations in a standard manner. At the beginning of entering the self-attention module, the two tensors on the batch dimension of the input tensor are first adjusted to be connected in sequence on the word sequence dimension, and then through three linear operators Q, K and V, they are mapped to query (sequence), key (key) and value (value), and then the standard self-attention calculation is performed. The final result rearranges the two tensors on the word sequence dimension to the batch dimension. The weights of the three linear operators Q, K and V here need to be learned through subsequent training processes.

[0075] Furthermore, in some embodiments of the present application, the preset loss function L=L sd +λ diff L diff +λ id L id , where λ diff and λ id is the weight value of the preset loss function, L sd is the loss function of the standard diffusion model, L diff is the constraint on the difference between the training model and the reference model, L diff Used to maintain the facial similarity of the facial image.

[0076] Specifically, the overall loss function consists of three parts, namely, L = L sd +λ diff L diff +λ id L id , where λ diff and λ id is the weight value of the loss function. The three loss functions are: L sd is the loss function of the standard diffusion model, defined as L sd =E x0，∈，t (||∈-∈θ(zt, t, c)|| 2 2 ), where z t =αt z +σ t ∈, z is the tensor of the training image x after passing through the VAE encoder.

[0077] L diff A constraint on the difference between the training model and the reference model is defined as L diff =E x0,∈,tlogσ(s(zt,t,c,θ) 2 ),in:

[0078] The conditions accepted by the reference model here do not include the input of the two modules of face identity and face details, so the parameters it accepts are c y instead of c.

[0079] L id It is to maintain the similarity of the face. When t>0.2T, this loss function is defined as 0, and the quality of the generated image is not high. When t≤0.2T, it is defined as L id = cosine(f id (x),f id (f t2i (x,t,c)), where f id refers to the face recognition module, f t2i It refers to the entire Wensheng graph model, not just the UNet module.

[0080] Figure 6 FIG. 2 is a structural block diagram of a facial image generating device 200 provided in an embodiment of the present application. Figure 6 As shown, the face image generation device 200 of the embodiment of the present application includes: a picture and text acquisition module 210, a tensor generation module 220 and an image generation module 230, wherein:

[0081] The image and text acquisition module 210 is used to acquire prompt words and reference images;

[0082] A tensor generation module 220 is used to input the prompt word and the reference image into a pre-trained face image generation model, wherein the face image generation model includes a text encoding module, a face processing module and a diffusion module, wherein the prompt word and the reference image are processed by the text encoding module to obtain a first latent space tensor, and the reference image is processed by the face processing module to obtain a second latent space tensor and a third latent space tensor, and the first latent space tensor, the second latent space tensor and the third latent space tensor constitute the conditions for inputting the diffusion module; and

[0083] The image generation module 230 is used to generate a face image according to the prompt word and the reference image and by calculating the diffusion model of the face image generation model.

[0084] In the face image generation device provided in the embodiment of the present application, a prompt word and a reference image are first obtained, and then the prompt word and the reference image are input into a pre-trained face image generation model, and a face image is generated based on the prompt word and the reference image and through the calculation of the diffusion model of the face image generation model. The face image generation method provided in the embodiment of the present application improves the generation quality of the face image of the diffusion model by amplifying the latent space tensor obtained by the text encoding model, thereby improving the fidelity and editability of the face image.

[0085] It should be noted that the specific implementation method of the face image generation device of the embodiment of the present application is similar to the specific implementation method of the face image generation method of the embodiment of the present application. Please refer to the description of the method part for details, and no further details will be given here.

[0086] Figure 7 Schematic diagram of the structure of an electronic device 300 according to an embodiment of the present application.

[0087] like Figure 7 As shown, the electronic device 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from the storage part 302 to a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The CPU 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0088] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, a mouse, etc.; an output section 307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed, so that a computer program read therefrom is installed into the storage section 308 as needed.

[0089] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 309, and / or installed from a removable medium 311. When the computer program is executed by a central processing unit (CPU) 301, the above-mentioned functions defined in the electronic device of the present application are executed.

[0090] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electronic device, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0091] In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction-executing electronic device, apparatus, or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction-executing electronic device, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0092] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the processing receiving device, method and computer program product according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the aforementioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based electronic device that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0093] The units or modules involved in the embodiments of the present application may be implemented by software or hardware. The units or modules described may also be arranged in a processor, and the processor is used to implement the face image generation method when executing the program:

[0094] Get cue words and reference images;

[0095] Inputting the prompt word and the reference image into a pre-trained face image generation model;

[0096] A face image is generated according to the prompt word and the reference image and by calculating a diffusion model of the face image generation model.

[0097] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently and not be assembled into the electronic device. The above computer-readable storage medium stores one or more programs, and when the above programs are used by one or more processors to execute the face image generation method described in the present application:

[0098] Get cue words and reference images;

[0099] Inputting the prompt word and the reference image into a pre-trained face image generation model;

[0100] A face image is generated according to the prompt word and the reference image and by calculating a diffusion model of the face image generation model.

[0101] As another aspect, the present application further provides a computer program product, which may be included in the electronic device described in the above embodiment; or may exist independently without being installed in the electronic device. The above computer program product stores one or more programs, and when the above programs are used by one or more processors to execute the facial image generation method described in the present application:

[0102] Get cue words and reference images;

[0103] Inputting the prompt word and the reference image into a pre-trained face image generation model;

[0104] A face image is generated according to the prompt word and the reference image and by calculating a diffusion model of the face image generation model.

[0105] The above description is only a preferred embodiment of the present application, and does not limit the patent scope of the present application. All equivalent structural changes made by using the contents of the present application specification and drawings under the application concept of the present application, or directly / indirectly used in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A method for generating a face image, characterized in that: The following steps are involved: Get cue words and reference images; Input the prompt word and the reference image into a pre-trained face image generation model, the face image generation model includes a text encoding module, a face processing module and a diffusion module, wherein the prompt word and the reference image are processed by the text encoding module to obtain a first latent space tensor, and the reference image is processed by the face processing module to obtain a second latent space tensor and a third latent space tensor, and the first latent space tensor, the second latent space tensor and the third latent space tensor constitute the conditions for inputting the diffusion module; A face image is generated according to the prompt word and the reference image and by calculating a diffusion model of the face image generation model.

2. The method for generating a facial image according to claim 1, characterized in that: The face processing module includes a face identity submodule and a face detail submodule. The reference image is processed by the face identity submodule to obtain the second latent space tensor. The reference image is processed by the face detail submodule to obtain the third latent space tensor.

3. The method for generating a facial image according to claim 2, characterized in that: The face identity submodule includes a face recognition module, a first self-attention module and a first multi-layer perceptron. The reference image is processed by the face identity submodule to obtain the second latent space tensor, including: The face of the reference image is mapped to a latent space vector expressing the face through the face recognition module, and the latent space vector is mapped to the latent space of the text encoding module through the first self-attention module and the first multi-layer perceptron to obtain the second latent space tensor.

4. The method for generating a facial image according to claim 2, wherein: The face detail submodule includes a CLIP visual model, a second self-attention module, and a second multi-layer perceptron. The reference image is processed by the face detail submodule to obtain the third latent space tensor, including: The features of the face of the reference image processed by the CLIP visual model are input into the second self-attention module and the second multi-layer perceptron for processing, and mapped to the latent space of the text encoding module to obtain the third latent space tensor.

5. The method for generating a facial image according to claim 1, wherein: The step of generating a face image according to the prompt word and the reference image and by calculating a diffusion model of the face image generation model includes: A face image is generated according to the prompt word and the reference image and through the reverse diffusion inference of the diffusion model of the face image generation model.

6. The method for generating a facial image according to claim 1, characterized in that: The training process of the face image generation model includes: Constructing a training data set, including obtaining a preset number of face images, and generating description words according to the face images to form image-text pairs as the training data set; The training data set is input into the diffusion model for training, and the diffusion model is optimized by a preset loss function to obtain the trained face image generation model.

7. The method for generating a facial image according to claim 6, characterized in that: The preset loss function L=L sd +λ diff L diff +λ id L id , where λ diff and λ id is the weight value of the preset loss function, L sd is the loss function of the standard diffusion model, L diff is the constraint on the difference between the training model and the reference model, L diff Used to maintain the facial similarity of the facial image.

8. A facial image generating device, characterized in that: include: An image and text acquisition module, used to acquire prompt words and reference images; A tensor generation module, used for inputting the prompt word and the reference image into a pre-trained face image generation model, wherein the face image generation model includes a text encoding module, a face processing module and a diffusion module, wherein the prompt word and the reference image are processed by the text encoding module to obtain a first latent space tensor, and the reference image is processed by the face processing module to obtain a second latent space tensor and a third latent space tensor, and the first latent space tensor, the second latent space tensor and the third latent space tensor constitute the conditions for inputting the diffusion module; and The image generation module is used to generate a face image according to the prompt word and the reference image and by calculating a diffusion model of the face image generation model.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor is used to implement the method for generating a facial image according to any one of claims 1 to 7 when executing the program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is used to implement the facial image generation method according to any one of claims 1-7.

Images