Text-based multi-modal face generation method and device, equipment and storage medium

Abstract

The invention provides a text-based multi-modal face generation method and device, equipment and a storage medium, and belongs to the technical field of face image generation, and the method comprises the steps: determining a main body image based on a reference image and a main body mask corresponding to the reference image, and determining a decoupling global identity embedding feature based on the main body image. The reference image is an initial face image. And determining a multi-modal local identity embedding feature based on the reference image and the mask image corresponding to the reference image. The multi-modal local identity embedding features are text embedding class features. And determining a target generation face image based on the decoupled global identity embedding feature and the multi-modal local identity embedding feature. According to the text-based multi-modal face generation method and device, the equipment and the storage medium provided by the invention, the accuracy of text-to-face generation can be improved, and actual requirements are met.

Description

Technical Field

[0001] The present disclosure belongs to the technical field of facial image generation, and more specifically, to a text-based multimodal face generation method and apparatus, device, and storage medium. Background Art

[0002] With the continuous advancement of artificial intelligence (AI), particularly breakthroughs in deep learning, text-to-image generation technologies have made further progress. These technologies can generate corresponding images, including faces, based on given text descriptions. Personalized text-to-face generation technology has further evolved, aiming to generate facial images with specific identity features and personalized elements. Personalized text-to-face generation technology has significant value in areas such as artistic creation, 3D human body generation, and image animation.

[0003] However, existing technologies still face some problems. For example, while fine-tuning-based methods can transform visual concepts, they are time-consuming and prone to overfitting, reducing the editability of semantic cues. Due to the spatial characteristics of text embedding, generative models struggle to balance text-image and identity consistency, failing to meet practical needs. Secondly, methods based on training universal encoders rely on large-scale datasets to train mapping networks. While this improves inference efficiency, it struggles to learn facial details, resulting in poor identity consistency in text-to-face generation. Summary of the Invention

[0004] The purpose of this disclosure is to provide a text-based multimodal face generation method and apparatus, device, and storage medium to improve the identity consistency of text-to-face generation and meet practical needs.

[0005] A first aspect of the present disclosure provides a text-based multimodal face generation method, comprising: The subject image is determined based on the reference image and the subject mask corresponding to the reference image, and the decoupled global identity embedding feature is determined based on the subject image. The reference image is the initial face image.

[0006] A multimodal local identity embedding feature is determined based on a reference image and a mask image corresponding to the reference image. The multimodal local identity embedding feature is a text embedding feature.

[0007] The target face image is generated based on the disentangled global identity embedding features and the multimodal local identity embedding features.

[0008] A second aspect of the present disclosure provides a text-based multimodal face generation apparatus, comprising: The first calculation module is configured to determine a subject image based on a reference image and a subject mask corresponding to the reference image, and to determine a decoupled global identity embedding feature based on the subject image. The reference image is an initial face image.

[0009] The second calculation module is configured to determine a multimodal local identity embedding feature based on the reference image and the mask image corresponding to the reference image. The multimodal local identity embedding feature is a text embedding feature.

[0010] The face generation module is used to determine the target generation of face images based on the decoupled global identity embedding features and the multimodal local identity embedding features.

[0011] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, the steps of the above-mentioned text-based multimodal face generation method are implemented.

[0012] In a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, which stores a computer program. When the computer program is executed by a processor, the steps of the above-mentioned text-based multimodal face generation method are implemented.

[0013] The beneficial effects of the text-based multimodal face generation method, apparatus, device, and storage medium provided by the embodiments of the present disclosure are: On the one hand, the disclosed embodiment constructs a multimodal-guided identity constraint framework by decoupling the global identity embedding features and the multimodal local identity embedding features, which can more accurately capture identity information at different levels in facial images, ensure the stability of identity features and the retention of facial details, and have semantic editing capabilities. The combination of decoupled global identity embedding features and multimodal local identity embedding features makes the generated facial images more accurate and rich in identity feature restoration. In addition, by using text information to determine the multimodal local identity embedding features, users can accurately adjust the local details of the generated face by entering specific text, broadening the application scenarios of face generation.

[0014] On the other hand, the disclosed embodiment utilizes a mask balancing loss based on residual cross attention (corresponding to the subject mask and the relevant parts of the mask image), which can effectively separate the identity information from the non-identity related background and balance the influence of text cues and identity constraints.

[0015] On the other hand, the embodiments of the present disclosure also achieve smooth control of fine-grained semantic editing.

[0016] In summary, the embodiments of the present disclosure implement a method for personalized text-to-face generation based on multimodal learning to ensure stable identity consistency, so that the face generation result ensures stable identity consistency of the subject on the basis of aligning diverse text prompts. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 A flowchart of a text-based multimodal face generation method provided in one embodiment of the present disclosure; Figure 2 A flowchart of a text-based multimodal face generation method provided by another embodiment of the present disclosure; Figure 3 A schematic diagram of facial image generation and comparison provided by an embodiment of the present disclosure; Figure 4 A structural block diagram of a text-based multimodal face generation device provided in one embodiment of the present disclosure; Figure 5 A schematic block diagram of an electronic device provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present disclosure with unnecessary detail.

[0020] In order to make the purpose, technical solutions and advantages of the present disclosure more clear, specific embodiments will be described below with reference to the accompanying drawings.

[0021] Please refer to Figure 1 , Figure 1 This is a flowchart of a text-based multimodal face generation method provided in one embodiment of the present disclosure. The method may include S101 to S103.

[0022] S101: Determine a subject image based on a reference image and a subject mask corresponding to the reference image, and determine a decoupled global identity embedding feature based on the subject image. The reference image is an initial face image.

[0023] In this embodiment, determining the subject image based on the reference image and the subject mask corresponding to the reference image includes: A subject mask is determined based on a reference image.

[0024] The subject mask and the corresponding pixel positions of the reference image are calculated to obtain the subject image.

[0025] In this embodiment, determining the subject mask based on the reference image may include: extracting the subject mask corresponding to the reference image based on a face parsing model.

[0026] In this embodiment, calculating the subject mask and the corresponding pixel position of the reference image to obtain the subject image may include: performing pixel-level multiplication calculation on the subject mask and the corresponding pixel position of the reference image to obtain the subject image.

[0027] In this embodiment, the reference image is a face image. The subject mask corresponds to the reference image and divides the reference image into a face region and a background region, assigning a value of 0 to the background region and a value of 1 to the face region. The subject image is a face image with irrelevant background information removed.

[0028] like Figure 2 As shown, for example, the pre-trained face parsing model ParseNet is used to extract the subject mask of the reference image. ,Will The subject image is obtained by performing pixel-level multiplication calculation on the corresponding pixel position of the reference image.

[0029] In this embodiment, determining a decoupled global identity embedding feature based on a subject image includes: A first identity feature vector is determined based on the subject image and the first encoder.

[0030] Perform vector editing on the first identity feature vector to generate a second identity feature vector.

[0031] A disentangled global identity embedding feature is determined based on the second identity feature vector and the mapping network.

[0032] In this embodiment, performing vector editing on the first identity feature vector may include: The first identity feature vector is vector edited based on attribute direction and attribute strength.

[0033] In this embodiment, the first identity feature vector is a decoupled identity feature vector extracted from the subject image. The second identity feature vector is the first identity feature vector after vector editing. The attribute direction represents the facial attribute category, and the attribute strength represents the facial attribute parameter. Since a vector has both magnitude and direction, it can be controlled by adjusting parameters such as the attribute direction and attribute strength. A multidimensional mapping network is used to process the second identity feature vector, changing its data dimension to obtain a decoupled global identity embedding feature that conforms to the input format.

[0034] like Figure 2As shown, for example, the subject image is input into the pre-trained StyleGAN encoder to extract the decoupled identity feature vector , by adjusting the attribute direction of facial attributes and attribute strength (set to 0 during training), calculate the translated feature vector .Will Projected to a mapping network consisting of four linear layers to generate a global identity embedding for disentanglement .

[0035] S102: Determine a multimodal local identity embedding feature based on a reference image and a mask image corresponding to the reference image. The multimodal local identity embedding feature is a text embedding feature.

[0036] In this embodiment, determining the multimodal local identity embedding feature based on the reference image and the mask image corresponding to the reference image may include: A mask image is obtained based on the reference image.

[0037] Determining multimodal local identity embedding features based on mask images.

[0038] like Figure 2 As shown, in this embodiment, the mask image is a mask image that divides the reference image into regions with subject attributes. For example, the reference image is divided into local regions such as background, eyes, hair, nose, mouth, chin, and cheeks. The segmentation model BiSeNet can be used to segment the reference image to generate a mask image with clear division of the subject attribute regions.

[0039] For example, an animation studio uses a hand-drawn character sketch depicting a mysterious magician as a reference image. They obtain a mask image corresponding to this reference image, divide it into different regions, and extract features from each region, such as the magician's deep eyes, flowing hair, high nose, smiling mouth, firm chin, and round cheeks. These features are preprocessed to generate multimodal local identity embedding features (text embedding-like features). These multimodal local identity embedding features can guide subsequent production software to accurately generate facial images that match the magician's image, including the details and style of each local region, thus providing strong support for animation character creation.

[0040] S103: Determine a target face image based on the decoupled global identity embedding feature and the multimodal local identity embedding feature.

[0041] In this embodiment, determining a target to generate a facial image based on the decoupled global identity embedding feature and the multimodal local identity embedding feature may include: The decoupled global identity embedding features and multimodal local identity embedding features are input into the target diffusion model to generate the target generated face image.

[0042] like Figure 2 As shown, in a film and television post-production scenario, we have a photograph of a woman as a reference image, or the initial facial image. During the face generation process, we need to have the woman wear a police hat. Using the reference image and its corresponding subject mask, we can determine the subject image, highlighting key features such as the woman's face and hair while removing irrelevant background information. Feature extraction and analysis of the subject image yields disentangled global identity embedding features. These features capture the woman's overall appearance, such as facial contours and hairstyle. These disentangled global identity embedding features are used as identity constraints to ensure identity consistency.

[0043] The reference image and its corresponding mask image (the mask image divides the reference image's face into localized regions such as hair, eyes, mouth, and nose) are then used to determine the multimodal local identity embedding feature, a text embedding feature that captures information such as eye shape and color. The decoupled global identity embedding feature and the multimodal local identity embedding feature are then fed into a specific target diffusion model. The target diffusion model generates a target face image based on these features. While preserving the woman's original facial features, it adds a stylish police cap. The cap's style and wearing angle align with realistic visuals, meeting the creative demands of film and television production.

[0044] From the above, it can be concluded that, on the one hand, this embodiment constructs a multimodal-guided identity constraint framework by decoupling the global identity embedding features and the multimodal local identity embedding features to ensure the stability of identity features and the retention of facial details, and has semantic editing capabilities. The combination of decoupled global identity embedding features and multimodal local identity embedding features makes the generated facial images more accurate and rich in identity feature restoration. In addition, by using text information to determine the multimodal local identity embedding features, users can accurately adjust the local details of the generated face by entering specific text, broadening the application scenarios of face generation.

[0045] On the other hand, this embodiment utilizes the mask balancing loss based on residual cross attention (corresponding to the subject mask and the relevant parts of the mask image), which can effectively separate the identity information from the non-identity related background and balance the influence of textual cues and identity constraints.

[0046] On the other hand, this embodiment also achieves smooth control of fine-grained semantic editing.

[0047] In summary, this embodiment implements a method for personalized text-to-face generation based on multimodal learning to ensure stable identity consistency, so that the face generation result ensures stable identity consistency of the subject on the basis of aligning diverse text prompts.

[0048] like Figure 2 As shown, in one embodiment of the present disclosure, determining a multimodal local identity embedding feature based on a reference image and a mask image corresponding to the reference image includes: Facial cues and scene cues are determined based on a reference image, and a mask image is generated based on the reference image.

[0049] Data processing is performed on facial prompts and scene prompts to obtain text tag sequences, and feature extraction is performed on mask images to obtain facial attribute features.

[0050] The facial attribute features and text tag sequences are spliced ​​together to obtain a multimodal tag sequence, and the multimodal local identity embedding features are determined based on the multimodal tag sequence.

[0051] In this embodiment, data processing is performed on facial prompts and scene prompts to obtain a text tag sequence, and feature extraction is performed on the mask image to obtain facial attribute features, including: Concatenate the facial and scene cues to obtain text cues, which are then fed into the second encoder to obtain a text tag sequence. determining facial attribute features corresponding to the mask image based on the mask image and the third encoder; An attribute feature sequence is determined based on the facial attribute features and a first query feature; the first query feature is a query feature vector corresponding to the reference image.

[0052] In this embodiment, determining facial cues and scene cues based on the reference image may include: Determine the question description based on the reference image, input the question description into the visual language model, and obtain facial prompts and scene prompts.

[0053] In this embodiment, determining facial hints and scene hints based on the reference image may further include: The scene hints are determined based on the reference image and the generation constraints, which are personalized generation constraints for the reference image.

[0054] In this embodiment, the second encoder may be a text encoder. The third encoder may be an image encoder. The question description may include text of a question about the reference image. The facial prompt is multimodal data and may include a text feature description and image feature data of the reference image.

[0055] For example, the question description may include: "Describe the facial identity features of the character" and "Describe the character's posture, action, accessories, clothing and background environment", etc. The reference image and the question description are input into the visual language model LLaVA, which can generate facial prompts corresponding to the reference image. and scene prompts Generation constraints are personalized generation conditions that can be set by users according to specific needs.

[0056] For example, the reference image is segmented using the segmentation model BiSeNet to generate a mask image with clear division of the subject attribute area. The mask image is input into the image encoder to obtain the facial attribute features. ,Will The learnable query vector (i.e., the first query feature) is input into the 8-layer learnable attention network and continuously strengthened to obtain detailed facial attribute features of each part. The strengthened facial attribute features are processed to obtain an attribute feature sequence.

[0057] The attribute feature sequence and the text tag sequence are concatenated and input into the Multilayer Perceptron (MLP) network. The attribute features of each facial part (such as mouth, nose, eyebrows, and ears) in the attribute feature sequence are embedded into the corresponding prompt position of the text tag sequence through the attention network. The multimodal tag sequence (i.e., text tag sequence) after embedding the attribute feature sequence (i.e., image detail features) is processed by the eight-layer residual connection MLP network to generate the multimodal local identity embedding feature. .

[0058]

[0059] in, For text encoder, For the attention network, For facial prompts, For scene prompts, For facial attribute features, is the MLP network.

[0060] This embodiment combines facial cues and scene cues related to the reference image and incorporates detailed facial attribute features to more accurately capture image information. This embodiment takes generation constraints into account to meet user personalized needs. Different constraints can generate unique facial images in different scenarios. This embodiment uses an encoder, an attention network, and an MLP network to effectively fuse text and image features, making the multimodal local identity embedding feature more comprehensive and rich, improving the subsequent face generation effect.

[0061] In one embodiment of the present disclosure, determining a target for generating a facial image based on a decoupled global identity embedding feature and a multimodal local identity embedding feature includes: A first hidden state is determined based on the second query feature and the multimodal local identity embedding feature.

[0062] The first hidden state is residually connected to the decoupled global identity embedding feature to obtain the second hidden state.

[0063] The target is determined based on the second hidden state to generate a face image.

[0064] For example, given the second query feature , get the hidden state after text embedding cross attention (i.e., the first hidden state);

[0065]

[0066]

[0067]

[0068] Among them, z is the second query feature, is the first hidden state, 、 and are the query matrix, key matrix, and value matrix of the text cross attention module, respectively. is the weight matrix corresponding to Q, is the weight matrix corresponding to K, is the weight matrix corresponding to V, For the attention network, is the multimodal local identity embedding feature, is the attention score calculation formula, and d is the matrix vector dimension.

[0069] Will and The result of cross-attention as the key-value vector is connected to obtain the second hidden state of the reference image after the dual identity condition constraint. ;

[0070]

[0071]

[0072]

[0073] in, 、 and are the query, key, and value matrices in the identity feature, respectively. for The corresponding weight matrix, for The corresponding weight matrix, for The corresponding weight matrix, is a hyperparameter.

[0074] This example uses different queries for image and text hint cross attention.

[0075] This embodiment uses a specific computational approach to determine the first and second latent states, accurately fusing multimodal local identity embedding features with decoupled global identity embedding features. This helps comprehensively consider all identity-related information, from local details to overall features, when generating the target facial image. This results in more accurate and richer identity features for the generated facial image, avoiding information loss or confusion.

[0076] This embodiment utilizes dual identity constraints, adding decoupled global identity embedding features to the first hidden state to generate the second hidden state, effectively enhancing the grasp of the reference image's identity information. This dual constraint mechanism can better guide the target's facial image generation process, ensuring that the generated image is more consistent with expectations, improving image quality, and more realistically presenting the target subject's identity characteristics.

[0077] In one embodiment of the present disclosure, the text-based multimodal face generation method further includes: The initial diffusion model is trained based on the training data set to obtain the target diffusion model.

[0078] The training dataset includes disentangled global identity embedding features, multimodal local identity embedding features and noisy images.

[0079] The training process of the initial diffusion model uses the first loss function and the second loss function.

[0080] In this embodiment, training the initial diffusion model based on the training data set includes: First prediction data is generated based on the disentangled global identity embedding feature, the multimodal local identity embedding feature, and the noise image.

[0081] A target loss is determined based on the first prediction data, and an initial diffusion model is updated based on the target loss to obtain a target diffusion model.

[0082] In this embodiment, generating first prediction data based on the decoupled global identity embedding feature, the multimodal local identity embedding feature, and the noise image includes: Determine a noise image based on a reference image and real noise.

[0083] Generate predicted noise based on decoupled global identity embedding features, multimodal local identity embedding features, and the noise image, generate a noise reconstruction image based on decoupled global identity embedding features, multimodal local identity embedding features, and the noise image, and use the predicted noise and the noise reconstruction image as first prediction data.

[0084] In this embodiment, determining a target loss based on the first prediction data includes: Within a first loss step, calculate a first loss between the real noise and the predicted noise based on a first loss function. The first loss function is:

[0085] Where, represents the first loss, 0.4S, S) represents the first loss step, 0.4S < s < S represents within the first loss step range, S is the total number of denoising steps, s is the denoising index, and the value range of s is (0, S], is the real noise, is the predicted noise, is the noise decoded image after adding noise to the reference image at time t.

[0086] Within a second loss step, calculate a second loss between the subject image and the noise reconstruction image based on a second loss function. The second loss function is:

[0087] Where, represents the second loss, (0, represents the second loss step, 0 < s ≤ 0.4S represents within the second loss step range, is the decoded image obtained by decoding the predicted noise image, is the initial state before denoising.

[0088] Determine the target loss based on the first loss and the second loss.

[0089] In this embodiment, the denoising time step S can be set to 1000 steps (i.e., the total number of denoising steps), the initial value of s during the denoising process is 1000, and the loss functions within two different time steps of 0.6S (600 steps) and 0.4S (400 steps) are used to guide the diffusion generation process.

[0090] Exemplarily, within the first 600 steps of denoising (i.e., s decreases from 1000 to 400 one by one), use the mean square error between the predicted noise and the real noise as the first loss , let the first loss guide the initial diffusion model to gradually reduce the noise in the image, so as to determine the basic structure and element distribution for the generated image; In the last 400 steps of denoising, the subject mask Use prediction within the scope and reality The mean square error between replace , learning the pixel-level details of the reference image within the mask range.

[0091] In this embodiment, a third loss function is obtained based on the first loss function and the second loss function. The third loss function is:

[0092] in, Target loss.

[0093] In this embodiment, the method further includes: using a fourth loss function and a fifth loss function in the training process of the initial diffusion model.

[0094] In this embodiment, the fourth loss function is a mask editing loss function, which includes the loss of predicted noise and real noise in the subject mask. The mean square error in the range.

[0095] The fifth loss function is the diversity loss, which includes the noise loss when the multimodal local identity embedding feature and the multimodal local identity embedding feature are both used as conditional constraints and when only the multimodal local identity embedding feature is used as a conditional constraint.

[0096] In this embodiment, the text-based multimodal face generation method further includes constructing a comprehensive portrait dataset.

[0097] For example, to enhance the robustness of the initial diffusion model training, the synthetic portrait dataset construction process covers data collection, data cleaning, and prompt generation. The synthetic portrait dataset consists of real and generated images from sources including FFHQ, CelebA-HQ, and stylega2, totaling 181,431 training images.

[0098] 181K face images were processed using dlib alignment, and images with small faces, extreme angles, and incomplete facial regions were filtered out. The visual language model was asked to "describe the facial identity features of the person" and "describe the person's posture, action, accessories, clothing, and background environment," and facial prompts were obtained respectively. and scene prompts , to ensure the richness of image cues. The comprehensive portrait dataset has decoupled editable attribute vectors, providing fine-grained visual manipulation capabilities for model training.

[0099] This embodiment uses different loss functions in stages. Initially, it uses the mean squared error between predicted and real noise to determine the basic structure. Later, it learns pixel-level details within the mask. This precisely guides the diffusion generation process, ensuring that the generated image is well-structured and rich in detail. The target loss function comprehensively considers multiple factors, effectively balancing the influence of identity information and textual cues. This avoids overemphasizing one aspect in the generated image, ensuring that the generated face matches both the identity characteristics and the textual description.

[0100] The comprehensive portrait dataset in this embodiment contains a large number of images from various sources. After processing, it provides rich material for model training and enhances the model's generalization capabilities. This comprehensive portrait dataset also includes facial detail hints and editability, facilitating fine-grained visual manipulation of generated images, meeting diverse generation needs and overcoming the limitations of existing public datasets.

[0101] Corresponding to the text-based multimodal face generation method of the above embodiment, Figure 2 This is a structural block diagram of a text-based multimodal face generation device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 2 The text-based multimodal face generation device 20 includes: a first calculation module 21, a second calculation module 22 and a face generation module 23.

[0102] The first calculation module 21 is configured to determine a subject image based on a reference image and a subject mask corresponding to the reference image, and to determine a decoupled global identity embedding feature based on the subject image. The reference image is an initial face image.

[0103] The second calculation module 22 is configured to determine a multimodal local identity embedding feature based on the reference image and the mask image corresponding to the reference image. The multimodal local identity embedding feature is a text embedding feature.

[0104] The face generation module 23 is used to determine the target and generate a face image based on the decoupled global identity embedding feature and the multimodal local identity embedding feature.

[0105] In one embodiment of the present disclosure, the first calculation module 21 is specifically configured to determine a subject mask based on a reference image.

[0106] The subject mask and the corresponding pixel positions of the reference image are calculated to obtain the subject image.

[0107] In one embodiment of the present disclosure, the first calculation module 21 is further configured to determine a first identity feature vector based on the subject image and the first encoder.

[0108] Perform vector editing on the first identity feature vector to generate a second identity feature vector.

[0109] A disentangled global identity embedding feature is determined based on the second identity feature vector and the mapping network.

[0110] In one embodiment of the present disclosure, the second calculation module 21 is specifically configured to determine facial cues and scene cues based on a reference image, and generate a mask image based on the reference image.

[0111] Data processing is performed on facial prompts and scene prompts to obtain text tag sequences, and feature extraction is performed on mask images to obtain facial attribute features.

[0112] The facial attribute features and text tag sequences are spliced ​​together to obtain a multimodal tag sequence, and the multimodal local identity embedding features are determined based on the multimodal tag sequence.

[0113] In one embodiment of the present disclosure, the second calculation module 21 is further configured to concatenate the facial prompt and the scene prompt to obtain a text prompt, and input the text prompt into a second encoder to obtain a text tag sequence; determining facial attribute features corresponding to the mask image based on the mask image and the third encoder; An attribute feature sequence is determined based on the facial attribute features and a first query feature; the first query feature is a query feature vector corresponding to the reference image.

[0114] In one embodiment of the present disclosure, the face generation module 23 is specifically configured to determine the first hidden state based on the second query feature and the multimodal local identity embedding feature.

[0115] The first hidden state is residually connected to the decoupled global identity embedding feature to obtain the second hidden state.

[0116] The target is determined based on the second hidden state to generate a face image.

[0117] In one embodiment of the present disclosure, the text-based multimodal face generation apparatus 20 further includes: The model training module is used to train the initial diffusion model based on the training data set to obtain the target diffusion model.

[0118] The training dataset includes disentangled global identity embedding features, multimodal local identity embedding features and noisy images.

[0119] The training process of the initial diffusion model uses the first loss function and the second loss function.

[0120] See also Figure 5 , Figure 5 This is a schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Figure 3The electronic device 300 in the embodiment shown may include: one or more processors 301, one or more input devices 302, one or more output devices 303, and one or more memories 304. The processors 301, input devices 302, output devices 303, and memories 304 communicate with each other via a communication bus 305. The memory 304 is used to store computer programs, which include program instructions. The processor 301 is used to execute the program instructions stored in the memory 304. The processor 301 is configured to call the program instructions to execute the functions of the modules in the above-mentioned device embodiments, such as Figure 4 The functions of modules 21 to 23 are shown.

[0121] It should be understood that in the embodiments of the present disclosure, the processor 301 may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0122] The input device 302 may include a touchpad, a fingerprint collection sensor (for collecting user fingerprint information and fingerprint direction information), a microphone, etc. The output device 303 may include a display (LCD, etc.), a speaker, etc.

[0123] The memory 304 may include a read-only memory and a random access memory, and provides instructions and data to the processor 301. A portion of the memory 304 may also include a non-volatile random access memory. For example, the memory 304 may also store information about the device type.

[0124] In a specific implementation, the processor 301, input device 302, and output device 303 described in the embodiments of the present disclosure can execute the implementation methods described in the first and second embodiments of the text-based multimodal face generation method provided in the embodiments of the present disclosure, and can also execute the implementation methods of the electronic device 300 described in the embodiments of the present disclosure, which will not be repeated here.

[0125] In another embodiment of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, all or part of the process of the method in the above embodiment is implemented. The computer program can also be used to instruct related hardware to complete the process. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, the steps of each of the above method embodiments are implemented. The computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium.

[0126] The computer-readable storage medium can be an internal storage unit of the electronic device in any of the aforementioned embodiments, such as a hard disk or memory of the electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash memory card, etc. Furthermore, the computer-readable storage medium can include both an internal storage unit of the electronic device and an external storage device. The computer-readable storage medium is used to store computer programs and other programs and data required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or is about to be output.

[0127] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the composition and steps of each example according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this disclosure.

[0128] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the electronic devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0129] In the several embodiments provided in this application, it should be understood that the disclosed electronic devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces or units, or can be an electrical, mechanical or other form of connection.

[0130] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of these units may be selected based on actual needs to achieve the objectives of the embodiments of the present disclosure.

[0131] In addition, the functional units in the various embodiments of the present disclosure may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0132] The above are only specific embodiments of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or replacements within the technical scope disclosed in this disclosure, and such modifications or replacements should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A text-based multimodal face generation method, characterized in that: include: Determine a subject image based on a reference image and a subject mask corresponding to the reference image, and determine a decoupled global identity embedding feature based on the subject image; The reference image is an initial face image; Determining a multimodal local identity embedding feature based on the reference image and a mask image corresponding to the reference image; the multimodal local identity embedding feature is a text embedding feature; A target is determined based on the decoupled global identity embedding feature and the multimodal local identity embedding feature to generate a face image.

2. The text-based multimodal face generation method according to claim 1, characterized in that: The determining the subject image based on the reference image and the subject mask corresponding to the reference image includes: determining the subject mask based on the reference image; The subject mask and the corresponding pixel positions of the reference image are calculated to obtain the subject image.

3. The text-based multimodal face generation method according to claim 1, characterized in that: The step of determining a decoupled global identity embedding feature based on the subject image includes: determining a first identity feature vector based on the subject image and the first encoder; Performing vector editing on the first identity feature vector to generate a second identity feature vector; The decoupled global identity embedding feature is determined based on the second identity feature vector and a mapping network.

4. The text-based multimodal face generation method according to claim 1, characterized in that: The determining of a multimodal local identity embedding feature based on the reference image and a mask image corresponding to the reference image comprises: determining facial cues and scene cues based on the reference image, and generating the mask image based on the reference image; Performing data processing on the facial prompt and the scene prompt to obtain a text tag sequence, and performing feature extraction on the mask image to obtain facial attribute features; The facial attribute features and the text tag sequence are concatenated to obtain a multimodal tag sequence, and the multimodal local identity embedding features are determined based on the multimodal tag sequence.

5. The text-based multimodal face generation method according to claim 4, characterized in that: The step of performing data processing on the facial prompt and the scene prompt to obtain a text tag sequence, and performing feature extraction on the mask image to obtain facial attribute features includes: splicing the facial prompt and the scene prompt to obtain a text prompt, and inputting the text prompt into a second encoder to obtain a text tag sequence; Determining facial attribute features corresponding to the mask image based on the mask image and a third encoder; An attribute feature sequence is determined based on the facial attribute feature and a first query feature; the first query feature is a query feature vector corresponding to the reference image.

6. The text-based multimodal face generation method according to claim 1, characterized in that: The step of determining a target and generating a face image based on the decoupled global identity embedding feature and the multimodal local identity embedding feature comprises: determining a first hidden state based on a second query feature and the multimodal local identity embedding feature; Residual connection is performed between the first hidden state and the decoupled global identity embedding feature to obtain a second hidden state; The target is determined to generate a facial image based on the second hidden state.

7. The text-based multimodal face generation method according to claim 6, characterized in that: Also includes: The initial diffusion model is trained based on the training data set to obtain the target diffusion model; The training data set includes decoupled global identity embedding features, multimodal local identity embedding features and noisy images; The training process of the initial diffusion model uses a first loss function and a second loss function.

8. A text-based multimodal face generation device, characterized in that: include: A first computing module, configured to determine a subject image based on a reference image and a subject mask corresponding to the reference image, and determine a decoupled global identity embedding feature based on the subject image; The reference image is an initial face image; A second computing module is used to determine a multimodal local identity embedding feature based on the reference image and a mask image corresponding to the reference image; the multimodal local identity embedding feature is a text embedding feature; A face generation module is used to determine a target for generating a face image based on the decoupled global identity embedding feature and the multimodal local identity embedding feature.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images