Face image generation method and device, electronic equipment and storage medium
By acquiring face images and description text, using a combination of image feature extraction and text feature optimization, and combining diffusion model to generate face images, the distortion and distortion problems in face image generation are solved, and more accurate and realistic generation is achieved Effect.
Patent Information
- Application Number
- CN202411008312.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-25
- Publication Date
- 2024-12-03
AI Technical Summary
In the face image generation based on the diffusion model, due to the complex face structure, the generated face image is prone to distortion or distortion, making it difficult to achieve accurate generation.
By obtaining sample face images and description text, use preset image feature extraction models to extract image feature information, and optimize the text feature extraction model based on image feature information, generate target text feature information, and combine forward diffusion and reverse diffusion modules to perform humans Face generation, achieving accurate target face image generation.
It improves the accuracy of face image generation, reduces distortion and distortion, and achieves more realistic face image generation.
Smart Images

Figure CN119068071A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a method and device for generating a face image, an electronic device, and a storage medium. Background Art
[0002] Image generation based on diffusion model is a technology that uses diffusion model to generate realistic images. The image generation process includes two steps: forward diffusion and backward diffusion. First, a piece of text is input, and noise is gradually added to the text during the forward diffusion process. Then, the noise is gradually removed during the backward diffusion process, and finally the denoised text is decoded by the decoder to generate an image.
[0003] In the face image generation based on the diffusion model, due to the complex structure and features of the face, the generated face images are often distorted or distorted. Therefore, how to accurately generate face images has become an urgent problem to be solved. Summary of the invention
[0004] The main purpose of the embodiments of the present application is to propose a facial image generation method and device, an electronic device and a storage medium, aiming to accurately generate facial images.
[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a method for generating a face image, the method comprising:
[0006] Acquire a sample face image and a sample face description text; wherein the sample face description text is used to describe the sample face image;
[0007] Extracting image features from the sample face image using a preset image feature extraction model to obtain feature information of the sample face image;
[0008] Optimizing parameters of a preset text feature extraction model according to the sample face image feature information and the sample face description text to obtain a target text feature extraction model;
[0009] Acquire a target face description text, and perform text feature extraction on the target face description text using the target text feature extraction model to obtain target text feature information;
[0010] The target face generation model is used to generate the target face of the target text feature information to obtain a target face image.
[0011] In some embodiments, the step of optimizing parameters of a preset text feature extraction model according to the sample facial image feature information and the sample facial description text to obtain a target text feature extraction model includes:
[0012] Extracting features from the sample face description text using the preset text feature extraction model to obtain sample text feature information;
[0013] The parameters of the preset text feature extraction model are optimized according to the sample text feature information and the sample face image feature information to obtain the target text feature extraction model.
[0014] In some embodiments, the preset text feature extraction model includes a text encoding module and a self-attention module; the feature extraction of the sample face description text by the preset text feature extraction model to obtain sample text feature information includes:
[0015] Performing text encoding on the sample face description text by the text encoding module to obtain a sample text encoding;
[0016] The self-attention module is used to extract features from the sample text encoding to obtain feature information of the sample text.
[0017] In some embodiments, the target face generation model includes a forward diffusion module and a backward diffusion module; the step of performing face generation on the target text feature information using a preset target face generation model to obtain a target face image includes:
[0018] Performing noise processing on the target text feature information through the forward diffusion module to obtain target noise feature information;
[0019] The target face image is obtained by performing face generation on the target noise feature information through the reverse diffusion module.
[0020] In some embodiments, the reverse diffusion module includes an encoder module, a residual module and a decoder module; the step of performing face generation on the target noise feature information through the reverse diffusion module to obtain the target face image includes:
[0021] Characteristically encoding the target noise characteristic information through the encoder module to obtain target noise encoding information;
[0022] Performing residual connection on the target noise coding information through the residual module to obtain target noise residual information;
[0023] The target noise residual information is feature decoded by the decoder module to obtain the target face image.
[0024] In some embodiments, the method further includes pre-training the target face generation model, including:
[0025] Acquire a selected face image and a selected face description text; wherein the selected face description text is used to describe the selected face image;
[0026] Adding a preset linear layer to a preset image generation model to obtain a selected image generation model;
[0027] The linear layer in the selected image generation model is optimized in parameters according to the selected face image and the selected face description text to obtain the target face generation model.
[0028] In some embodiments, the step of obtaining the selected face image and the selected face description text includes:
[0029] Acquire an initial face image and an initial face description text; wherein the initial face description text is used to describe the initial face image;
[0030] Performing angle detection on the initial face image using a preset face angle detection model to obtain angle detection data;
[0031] Screening the initial face image according to a preset angle threshold and the angle detection data to obtain the selected face image;
[0032] The initial face description text is screened according to the selected face image to obtain the selected face description text.
[0033] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a facial image generation device, the device comprising:
[0034] An acquisition module, used to acquire a sample face image and a sample face description text; wherein the sample face description text is used to describe the sample face image;
[0035] A first extraction module, configured to extract image features from the sample face image using a preset image feature extraction model to obtain feature information of the sample face image;
[0036] An optimization module, used for optimizing parameters of a preset text feature extraction model according to the feature information of the sample face image and the sample face description text, to obtain a target text feature extraction model;
[0037] The second extraction module is used to obtain a target face description text, and perform text feature extraction on the target face description text through the target text feature extraction model to obtain target text feature information;
[0038] The generation module is used to generate a face for the target text feature information through a preset target face generation model to obtain a target face image.
[0039] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.
[0040] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.
[0041] The present application proposes a facial image generation method and device, an electronic device and a storage medium, which obtain a sample facial image and a sample facial description text, and then perform image feature extraction on the sample facial image through a preset image feature extraction model to obtain sample facial image feature information, and optimize the parameters of the preset text feature extraction model according to the sample facial image feature information and the sample facial description text, so that the preset text feature extraction model can fully learn the facial image feature information, thereby improving the expression ability of the sample facial description text from the text feature information; further, obtain a target facial description text, and perform text feature extraction on the target facial description text through a target text feature extraction model to generate target text feature information that can accurately express the target description text, and then perform face generation on the target text feature information through a target face generation model to achieve accurate generation of a target facial image. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 is a flow chart of a method for generating a facial image provided by an embodiment of the present application;
[0043] Figure 2 yes Figure 1 Flow chart of step S103 in FIG.
[0044] Figure 3 yes Figure 2 Flow chart of step S201 in FIG.
[0045] Figure 4 yes Figure 1 Flow chart of step S105 in FIG.
[0046] Figure 5 yes Figure 4 Flow chart of step S402 in FIG.
[0047] Figure 6 is a flowchart of a method for generating a facial image provided by another embodiment of the present application;
[0048] Figure 7 yes Figure 6Flow chart of step S601 in FIG.
[0049] Figure 8 is a schematic diagram of the structure of a selected image generation model provided in an embodiment of the present application;
[0050] Fig. 9 is a flowchart of a method for generating a facial image provided by another embodiment of the present application;
[0051] Fig.10 is a structural schematic diagram of a facial image generating device provided in an embodiment of the present application;
[0052] Fig.11 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION
[0053] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0054] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0056] First, some nouns involved in this application are analyzed:
[0057] Self-Attention Mechanism: A mechanism used in deep learning models to dynamically adjust the output by focusing on different parts of the input sequence. It is a core component of neural networks, especially transformer architectures. The main function of the self-attention module is to capture long-distance dependencies in the input data, enabling the model to understand and process the complex structure of sequence data. This mechanism is widely used in natural language processing and computer vision, such as machine translation, text generation, and image generation.
[0058] Text Encoding Module: It is a deep learning component that converts raw text data into numerical representations and is the core part of the natural language processing system. The main function of the text encoding module is to convert characters, words or sentences in the text into vector representations for subsequent processing and analysis by machine learning models. The text encoding module is widely used in various tasks, including text classification, sentiment analysis, machine translation, and text generation.
[0059] Forward Diffusion: In generative models, forward diffusion is a process of gradually adding noise to input data. This is part of how generative models learn data distributions, and by gradually adding noise during training, the model is able to recover the original data from varying degrees of noise, thereby improving the quality and diversity of generated data.
[0060] Reverse Diffusion: In generative models, reverse diffusion is a process of gradually recovering the original data from noise, and is a key step in generating images. In the forward diffusion process, noise is gradually added to the original data to form a noise distribution. In the reverse diffusion process, the model gradually removes noise and eventually generates a high-quality image.
[0061] Image generation based on diffusion model is a technology that uses diffusion model to generate realistic images. The image generation process includes two steps: forward diffusion and backward diffusion. First, a piece of text is input, and noise is gradually added to the text during the forward diffusion process. Then, the noise is gradually removed during the backward diffusion process, and finally the denoised text is decoded by the decoder to generate an image.
[0062] In the face image generation based on the diffusion model, due to the complex structure and features of the face, the generated face images are often distorted or distorted. Therefore, how to accurately generate face images has become an urgent problem to be solved.
[0063] Based on this, the embodiments of the present application provide a facial image generation method and device, an electronic device and a storage medium, aiming to accurately generate facial images.
[0064] The embodiments of the present application provide a facial image generation method and device, an electronic device, and a storage medium, which are specifically described through the following embodiments. First, the facial image generation method in the embodiments of the present application is described.
[0065] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0066] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0067] The facial image generation method provided in the embodiment of the present application relates to the field of image processing technology. The facial image generation method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the facial image generation method, etc., but is not limited to the above forms.
[0068] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.
[0069] Figure 1 is an optional flowchart of the face image generation method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105.
[0070] Step S101, obtaining a sample face image and a sample face description text; wherein the sample face description text is used to describe the sample face image;
[0071] Step S102, extracting image features from the sample face image using a preset image feature extraction model to obtain feature information of the sample face image;
[0072] Step S103, optimizing the parameters of a preset text feature extraction model according to the sample face image feature information and the sample face description text to obtain a target text feature extraction model;
[0073] Step S104, obtaining a target face description text, and performing text feature extraction on the target face description text through a target text feature extraction model to obtain target text feature information;
[0074] Step S105, performing face generation on the target text feature information through a preset target face generation model to obtain a target face image.
[0075] In steps S101 to S105 shown in the embodiment of the present application, a sample facial image and a sample facial description text are obtained, and then image features are extracted from the sample facial image through a preset image feature extraction model to obtain sample facial image feature information, and parameters of the preset text feature extraction model are optimized according to the sample facial image feature information and the sample facial description text, so that the preset text feature extraction model can fully learn the facial image feature information, thereby improving the expression ability of the sample facial description text from the text feature information; further, a target facial description text is obtained, and text features are extracted from the target facial description text through a target text feature extraction model to generate target text feature information that can accurately express the target description text, and then a target face generation model is used to generate a face from the target text feature information, thereby achieving accurate generation of a target facial image.
[0076] In step S101 of some embodiments, the sample face image includes a single image of the same person at multiple angles. The sample description text is a description text of the sample face image, including but not limited to skin color, distribution of facial features, skin condition, hair and hairstyle, facial expressions, glasses wearing status, beard status, makeup status and other detailed information. For example, the sample face features are skin color: medium tone; warm color, with slight freckles; eyebrows: curved shape, medium thickness, dark brown color; eyes: almond-shaped, about one eye width apart, iris color is brown; nose: straight nose bridge, round nose tip, slightly wide nose wings; mouth: thin upper lip, thicker lower lip, slightly upturned corners of the mouth; ears: medium size, slightly rounded earlobes; skin condition: smooth skin, slightly oily; hair and hairstyle: long straight dark brown hair, shoulder length, moderate hair volume; facial expressions: the image shows expressions such as smiling, calm and frowning; wearing frameless glasses, no beard, no makeup;
[0077] In step S102 of some embodiments, the preset image feature extraction model is a model for extracting feature information from an image, which may be a convolutional neural network (CNN) or a pre-trained visual attention model, and this application does not impose any specific restrictions. The sample face image feature information is the feature information of the sample face image, and the sample face image feature information retains the main information of the sample face image and removes irrelevant noise information in the sample face image.
[0078] See also Figure 2 In some embodiments, step S103 may include but is not limited to steps S201 to S202:
[0079] Step S201, extracting features from the sample face description text using a preset text feature extraction model to obtain sample text feature information;
[0080] Step S202, optimizing parameters of a preset text feature extraction model according to the sample text feature information and the sample face image feature information to obtain a target text feature extraction model.
[0081] In steps S201 to S202 shown in the embodiment of the present application, feature extraction is performed on the sample face description text through a preset text feature extraction model to obtain sample text feature information, and parameters of the preset text feature extraction model are optimized according to the sample text feature information and the sample face image feature information to obtain a target text feature extraction model, thereby improving the preset text feature extraction model's perception ability of text information and obtaining a target text feature extraction model that can accurately extract feature information from the text.
[0082] In step S201 of some embodiments, the preset text feature extraction model is a text feature extraction model, which is used to extract features from the sample face description text, obtain feature information in the sample face description text, and obtain sample text feature information.
[0083] See also Figure 3 In some embodiments, the preset text feature extraction model includes a text encoding module and a self-attention module, and step S201 may include but is not limited to steps S301 to S302:
[0084] Step S301, performing text encoding on the sample face description text by a text encoding module to obtain a sample text code;
[0085] Step S302: extract features from the sample text encoding through a self-attention module to obtain sample text feature information.
[0086] In steps S301 to S302 shown in the embodiment of the present application, the sample face description text is encoded by a text encoding module to obtain a sample text encoding, and the sample text encoding is extracted by a self-attention module to obtain sample text feature information, thereby obtaining sample feature information that can accurately express the sample face description text, and providing accurate data for subsequent parameter optimization of a preset text feature extraction model according to the sample feature information.
[0087] In step S301 of some embodiments, the text encoding module is a deep learning module that converts sample face description text into a numerical matrix representation.
[0088] In one embodiment, the text encoding module is Contrastive Language-Image Pre-Training (CLIP), and the sample face description text is input into the CLIP module for text encoding to obtain the sample text encoding.
[0089] In one embodiment, the text encoding module is a word embedding module, and the sample face description text is input into the word embedding module for text encoding to obtain the sample text encoding.
[0090] In step S302 of some embodiments, self-attention is a module used in a deep learning model to dynamically adjust the output by paying attention to different parts of the input sequence, and is used to focus on features in the sample text encoding.
[0091] In one embodiment, the text encoding module is a CLIP module, and the self-attention module is a long short-term memory module (LSTM). During pre-training, the CLIP module compares and learns text and images, that is, CLIP can convert text into a text encoding suitable for comparison with image features, and then use the self-attention module to obtain feature information in the text encoding, thereby accurately converting text into image features.
[0092] In step S202 of some embodiments, by calculating the loss value between the sample text feature information and the sample facial image feature information, and then optimizing the parameters of the preset text feature extraction model according to the loss value, the perception ability of the preset text feature extraction model for text information is improved, and a target text feature extraction model capable of accurately extracting feature information from the text is obtained.
[0093] In step S104 of some embodiments, the target face description text includes single images of the same person at multiple angles. The sample description text is a description text of the sample face image, including but not limited to detailed information such as skin color, facial features distribution, skin condition, hair and hairstyle, facial expression, glasses wearing status, beard status, makeup status, etc. The target text feature information is the feature information extracted from the target face description text by the target text feature extraction model, which retains the main information of the target face description text and removes irrelevant noise information in the target face description text.
[0094] See also Figure 4 In some embodiments, the target face generation model includes a forward diffusion module and a backward diffusion module, and step S105 may include but is not limited to steps S401 to S402:
[0095] Step S401, performing noise processing on the target text feature information through a forward diffusion module to obtain target noise feature information;
[0096] Step S402, performing face generation on the target noise feature information through a reverse diffusion module to obtain a target face image.
[0097] In steps S401 to S402 shown in the embodiment of the present application, the target text feature information is subjected to noise processing by a forward diffusion module to obtain target noise feature information, and then the target noise feature information is subjected to face generation by a reverse diffusion module to obtain a target face image, thereby realizing accurate generation of a face image according to the target text feature information.
[0098] In step S401 of some embodiments, the forward diffusion module is used to add noise to the target text feature information. The target text feature information is subjected to noise addition processing by the forward diffusion module to obtain target noise feature information as shown in formula (1):
[0099]
[0100] It should be noted that the noise addition process of the forward diffusion module to the target text feature information is a cyclic process. In one noise addition process, the number of iterations of the current cycle is t, and the number of iterations of the previous cycle is t-1. t is the target noise characteristic information obtained in the current cycle. t is the preset current cycle noise ratio. It should be noted that β t It can be a preset fixed value, or it can change with the number of iterations, and this application does not make any specific restrictions. t-1 is the target noise characteristic information obtained in the previous cycle. ∈ t is the current cyclic noise, which is a matrix whose dimension is consistent with the dimension of the target text feature information. Each value in the matrix is respectively obtained from the preset standard normal distribution. Randomly obtained from .
[0101] See also Figure 5 In some embodiments, the back diffusion module includes an encoder module, a residual module and a decoder module, and step S402 includes but is not limited to steps S501 to S503:
[0102] Step S501, feature encoding the target noise feature information through an encoder module to obtain target noise encoding information;
[0103] Step S502, performing residual connection on the target noise coding information through a residual module to obtain target noise residual information;
[0104] Step S503: feature decoding is performed on the target noise residual information through a decoder module to obtain a target face image.
[0105] In steps S501 to S503 shown in the embodiment of the present application, the target noise feature information is feature encoded by the encoder module to obtain the target noise encoding information, and the target noise encoding information is residually connected by the residual module to obtain the target noise residual information, and finally the target noise residual information is feature decoded by the decoder module, so as to accurately restore and generate the target face image based on the target noise feature information.
[0106] In step S501 of some embodiments, the encoder module is at least one downsampling layer, which continuously reduces the dimension of the target noise feature information through the downsampling layer, and extracts the main feature information therein to obtain the target noise coding information.
[0107] In step S502 of some embodiments, the residual module includes multiple linear layers, convolutional layers and activation functions, which are used to capture complex features in the target noise coding information, improve the perception ability of the features, and obtain the target noise residual information.
[0108] In step S503 of some embodiments, the decoder module is at least one upsampling layer, and the target noise residual information is continuously increased in dimension through the upsampling layer, so as to restore and merge the feature information extracted from the encoder module and the residual module into a new picture to obtain the target face image.
[0109] See also Figure 6 In some embodiments, the face image generation method further includes pre-training a target face generation model, specifically including but not limited to steps S601 to S603:
[0110] Step S601, obtaining a selected face image and a selected face description text; wherein the selected face description text is used to describe the selected face image;
[0111] Step S602, adding a preset linear layer to a preset image generation model to obtain a selected image generation model;
[0112] Step S603, optimizing the parameters of the linear layer in the selected image generation model according to the selected face image and the selected face description text to obtain a target face generation model.
[0113] In steps S601 to S603 shown in the embodiment of the present application, a selected face image and a selected face description text are obtained, and then a preset linear layer is added to a preset image generation model to obtain a selected image generation model. Finally, the linear layer in the selected image generation model is optimized according to the selected face image and the selected face description text to obtain a target face generation model, thereby improving the target face generation model's ability to perceive the description text, thereby accurately generating a face image.
[0114] See also Figure 7 In some embodiments, step S601 may include but is not limited to steps S701 to S704:
[0115] Step S701, obtaining an initial face image and an initial face description text; wherein the initial face description text is used to describe the initial face image;
[0116] Step S702, performing angle detection on the initial face image by using a preset face angle detection model to obtain angle detection data;
[0117] Step S703, screening the initial face image according to the preset angle threshold and the angle detection data to obtain a selected face image;
[0118] Step S704, screening the initial face description text according to the selected face image to obtain the selected face description text.
[0119] In steps S701 to S704 shown in the embodiment of the present application, an initial facial image and an initial facial description text are obtained, and then the initial facial image is angle detected by a preset facial angle detection model to obtain angle detection data. The initial facial image is then screened according to a preset angle threshold and the angle detection data to obtain a selected facial image. Finally, the initial facial description text is screened according to the selected facial image to obtain a selected facial description text, thereby filtering out images in the initial facial image whose angles are greater than the preset angle threshold, and retaining images suitable for training subsequent selected image generation models, thereby improving the training effect of subsequent selected image generation models.
[0120] In step S701 of some embodiments, the initial facial image includes a single image of the same person at multiple angles. The initial description text is a description text of the initial facial image, including but not limited to skin color, facial features, skin condition, hair and hairstyle, facial expression, glasses wearing status, beard status, makeup status and other detailed information.
[0121] In step S702 of some embodiments, a preset face angle detection model is used to analyze the posture of the face in the image relative to the camera, including the rotation angle and pitch angle of the head. The angle detection data is the rotation angle and pitch angle of the face in the image.
[0122] In step S703 of some embodiments, the preset angle threshold includes a rotation angle threshold and a pitch angle threshold. When the rotation angle of the face in the angle detection data is smaller than the rotation angle threshold of the preset angle threshold and the pitch angle of the face in the angle detection data is smaller than the pitch angle of the preset angle threshold, the initial face image is taken as the selected face image.
[0123] In step S704 of some embodiments, the selected facial image includes multiple images, each image uniquely corresponds to a description text, and the initial facial description text is filtered according to the selected facial image to obtain the selected facial description text corresponding to the selected facial image.
[0124] In step S602 of some embodiments, the linear layer is a low-rank adapter (Low-RankAdaptation.lora), the preset image generation model includes a preset forward diffusion module and a preset reverse diffusion module, the preset reverse diffusion module includes a preset encoder, a preset residual block and a preset decoder, and at least one fully connected layer is added to the side branch of the preset encoder, the side branch of the preset residual block and the side branch of the preset decoder, respectively, to obtain the selected image generation model.
[0125] See also Figure 8 , taking the addition of lora to the side branch of the preset encoder as an example, the output of the preset encoder before adding the fully connected layer is shown in formula (2):
[0126] h1=W·x (2),
[0127] Among them, h1 is the output of the preset encoder, W is the neural network parameter of the preset encoder, and x is the output of the preset forward diffusion module, that is, the input of the preset reverse diffusion module.
[0128] After adding lora to the side branch of the preset encoder, the input is as shown in formula (3):
[0129] h2=W·x+B·A·x (3),
[0130] Among them, h2 is the output of the side branch of the preset encoder after adding lora. Lora is composed of two fully connected layers. The network parameter of the first fully connected layer is A. After passing through the first fully connected layer, the dimension of x is reduced from d to r, and the main feature information in x is retained. The output of the first fully connected layer is input to the second fully connected layer. The network parameter of the second fully connected layer is B, and the matrix of r latitude is output. Finally, the output W·x of the preset encoder and the output B·A·x of lora are added to obtain the output h2 of the side branch of the preset encoder after adding lora.
[0131] In step S603 of some embodiments, the selected face description text is input into the linear layer in the selected image generation model to obtain a linear layer face image output, and a loss value is calculated based on the linear layer face image and the selected face image, and the linear layer is optimized based on the loss value to obtain a target face generation model. It should be noted that during the parameter optimization process, the neural network layers other than the linear layer in the selected image generation model are not trained, and the original pre-trained parameters are retained. Only the network parameters of the linear layer need to be trained, which speeds up the training speed.
[0132] See also Fig. 9In one embodiment, firstly, the sample face description text is input into the preset text feature extraction model to obtain the sample text feature information, and at the same time, the sample face image is input into the preset image feature extraction model to obtain the sample face image feature information, and then the loss value between the sample face image feature information and the sample text feature information is calculated, and the preset text feature extraction model is optimized according to the loss value to obtain the target text feature extraction model, thereby improving the perception ability of the target text feature extraction model to the face description text and improving the accuracy of the generated text features. Then, the target face description text is input into the target text feature extraction model to generate the target text feature information, and the face is generated by the target face generation model and the target text feature information to accurately generate the target face image.
[0133] See also Fig.10 The embodiment of the present application also provides a facial image generation device, which can implement the above facial image generation method, and the device includes:
[0134] The acquisition module 1001 is used to acquire a sample face image and a sample face description text; wherein the sample face description text is used to describe the sample face image;
[0135] The first extraction module 1002 is used to extract image features from the sample face image using a preset image feature extraction model to obtain feature information of the sample face image;
[0136] An optimization module 1003 is used to optimize the parameters of a preset text feature extraction model according to the sample face image feature information and the sample face description text to obtain a target text feature extraction model;
[0137] The second extraction module 1004 is used to obtain the target face description text, and perform text feature extraction on the target face description text through the target text feature extraction model to obtain target text feature information;
[0138] The generation module 1005 is used to generate a face for the target text feature information through a preset target face generation model to obtain a target face image.
[0139] The specific implementation of the face image generation device is basically the same as the specific implementation of the above-mentioned face image generation method, and will not be repeated here.
[0140] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned face image generation method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.
[0141] See also Fig.11 , Fig.11 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:
[0142] The processor 1101 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;
[0143] The memory 1102 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1102 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1102, and the processor 1101 calls and executes the face image generation method of the embodiment of this application;
[0144] Input / output interface 1103, used to implement information input and output;
[0145] The communication interface 1104 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);
[0146] A bus 1105 that transmits information between various components of the device (e.g., the processor 1101, the memory 1102, the input / output interface 1103, and the communication interface 1104);
[0147] The processor 1101 , the memory 1102 , the input / output interface 1103 and the communication interface 1104 are connected to each other in communication within the device via the bus 1105 .
[0148] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned facial image generation method is implemented.
[0149] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0150] The face image generation method, face image generation device, electronic device and storage medium provided in the embodiments of the present application obtain a sample face image and a sample face description text, then perform image feature extraction on the sample face image through a preset image feature extraction model to obtain sample face image feature information, and optimize the parameters of the preset text feature extraction model according to the sample face image feature information and the sample face description text, so that the preset text feature extraction model can fully learn the face image feature information, thereby improving the expression ability of the sample face description text from the text feature information; further, obtain a target face description text, and perform text feature extraction on the target face description text through a target text feature extraction model to generate target text feature information that can accurately express the target description text, and then perform face generation on the target text feature information through a target face generation model to achieve accurate generation of a target face image.
[0151] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0152] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.
[0153] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0154] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.
[0155] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0156] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0157] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0158] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0159] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0160] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.
[0161] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.
Claims
1. A method for generating a face image, characterized in that: The method comprises: Acquire a sample face image and a sample face description text; wherein the sample face description text is used to describe the sample face image; Extracting image features from the sample face image using a preset image feature extraction model to obtain feature information of the sample face image; Optimizing parameters of a preset text feature extraction model according to the sample face image feature information and the sample face description text to obtain a target text feature extraction model; Acquire a target face description text, and perform text feature extraction on the target face description text using the target text feature extraction model to obtain target text feature information; The target face generation model is used to generate the target face of the target text feature information to obtain a target face image.
2. The method according to claim 1, characterized in that The step of optimizing parameters of a preset text feature extraction model according to the sample face image feature information and the sample face description text to obtain a target text feature extraction model includes: Extracting features from the sample face description text using the preset text feature extraction model to obtain sample text feature information; The parameters of the preset text feature extraction model are optimized according to the sample text feature information and the sample face image feature information to obtain the target text feature extraction model.
3. The method according to claim 2, characterized in that The preset text feature extraction model includes a text encoding module and a self-attention module; the feature extraction of the sample face description text by the preset text feature extraction model to obtain sample text feature information includes: Performing text encoding on the sample face description text by the text encoding module to obtain a sample text encoding; The self-attention module is used to extract features from the sample text encoding to obtain feature information of the sample text.
4. The method according to claim 1, characterized in that: The target face generation model includes a forward diffusion module and a backward diffusion module; The step of generating a face for the target text feature information by using a preset target face generation model to obtain a target face image includes: Performing noise processing on the target text feature information through the forward diffusion module to obtain target noise feature information; The target face image is obtained by performing face generation on the target noise feature information through the reverse diffusion module.
5. The method according to claim 4, characterized in that The reverse diffusion module includes an encoder module, a residual module and a decoder module; the step of performing face generation on the target noise feature information through the reverse diffusion module to obtain the target face image includes: Characteristically encoding the target noise characteristic information through the encoder module to obtain target noise encoding information; Performing residual connection on the target noise coding information through the residual module to obtain target noise residual information; The target noise residual information is feature decoded by the decoder module to obtain the target face image.
6. The method according to claim 1, characterized in that The method further includes pre-training the target face generation model, including: Acquire a selected face image and a selected face description text; wherein the selected face description text is used to describe the selected face image; Adding a preset linear layer to a preset image generation model to obtain a selected image generation model; The linear layer in the selected image generation model is optimized in parameters according to the selected face image and the selected face description text to obtain the target face generation model.
7. The method according to claim 6, characterized in that The step of obtaining the selected face image and the selected face description text comprises: Acquire an initial face image and an initial face description text; wherein the initial face description text is used to describe the initial face image; Performing angle detection on the initial face image using a preset face angle detection model to obtain angle detection data; Screening the initial face image according to a preset angle threshold and the angle detection data to obtain the selected face image; The initial face description text is screened according to the selected face image to obtain the selected face description text.
8. A facial image generating device, characterized in that: The device comprises: An acquisition module, used to acquire a sample face image and a sample face description text; wherein the sample face description text is used to describe the sample face image; A first extraction module, configured to extract image features from the sample face image using a preset image feature extraction model to obtain feature information of the sample face image; An optimization module, used for optimizing parameters of a preset text feature extraction model according to the feature information of the sample face image and the sample face description text, to obtain a target text feature extraction model; The second extraction module is used to obtain a target face description text, and perform text feature extraction on the target face description text through the target text feature extraction model to obtain target text feature information; The generation module is used to generate a face for the target text feature information through a preset target face generation model to obtain a target face image.
9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the facial image generation method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the facial image generation method according to any one of claims 1 to 7 is implemented.
Citation Information
Cited By
Specific character video generation method, device and equipment and medium
CN119815147A
