An enhanced diffuse multi-modal face synthesis method and system

Abstract

The application provides an enhanced diffusion multi-modal face synthesis method and system, which obtains a pixel space face image through reverse estimation decoding, calculates text consistency loss and mask consistency loss, and introduces a time dynamic modulation coefficient, adaptively adjusts the constraint strength according to the current time step, avoids large time step noise interference, establishes explicit pixel-level alignment supervision for the generated results and multi-modal conditions, and significantly improves the fine-grained semantic consistency of hair color, accessories and expression attributes. In view of the weak modeling ability of rare attributes, a difference weight matrix is calculated based on the pixel frequency of each semantic attribute in the training data set, the error of the predicted velocity field and the target velocity field is weighted, a weighted flow matching loss is obtained, the model applies stronger gradient signals to the long-tail rare attribute area, and the deviation of uniform optimization to high-frequency attributes is effectively corrected. The two kinds of loss are jointly optimized to solve the technical defects of inaccurate semantic alignment and weak modeling ability of rare attributes.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence-generated content technology, and in particular to an enhanced diffusion multimodal facial synthesis method and system. Background Technology

[0002] Multimodal face synthesis refers to the technology of generating precisely matched facial images based on non-image modal conditions such as text descriptions and semantic segmentation masks. It has significant value in scenarios such as virtual digital humans, film and television special effects, and personalized interactions. Currently, diffusion models and flow matching probabilistic generation models have become the mainstream technical approaches in this field.

[0003] However, existing solutions still suffer from two core flaws: First, semantic alignment is inaccurate. Mainstream methods implicitly incorporate multimodal conditions into the denoising / probabilistic flow regression process through cross-attention or feature concatenation, lacking explicit modeling of fine-grained semantic relationships. This leads to biases in attributes such as hair color, accessories, and facial expressions. Second, they are weak at modeling rare attributes. Facial semantics exhibit a long-tail distribution, with very few samples of rare attributes such as pearl earrings and heterochromia. Uniform optimization mechanisms naturally favor high-frequency attributes, resulting in insufficient model response to these attributes and further exacerbating semantic bias.

[0004] The root of these problems lies in the fact that traditional noise prediction or flow matching loss only guarantees the overall consistency of the generated distribution, lacking pixel-level alignment constraints; there is a fundamental contradiction between the uniform optimization objective and the long-tailed data distribution, making it impossible for the model to learn different attributes differentially; and if pixel-level constraints are directly applied, unreliable estimates will be introduced at noisy time steps, interfering with training. These problems severely restrict the practical application of multimodal face synthesis technology. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide an enhanced diffusion multimodal face synthesis method and system to solve the problems of inaccurate semantic alignment and weak modeling ability for long-tail rare attributes in the prior art during multimodal face synthesis.

[0006] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0007] The first aspect of this invention discloses an enhanced diffusion multimodal face synthesis method, the method comprising:

[0008] Obtain a multimodal condition pair, which includes the original face image, text description, and semantic segmentation mask;

[0009] The original face image, the text description, and the semantic segmentation mask are preprocessed and input into a probabilistic generation model for multimodal fusion, and the predicted velocity field is output.

[0010] Based on the pixel frequency of each semantic attribute in the training dataset, a difference weight matrix is ​​calculated, and the error between the predicted velocity field and the target velocity field is weighted using the weight matrix to obtain a weighted flow matching loss; the training dataset contains multiple sample multimodal condition pairs; the sample multimodal condition pairs include original face images of samples, sample text descriptions, and sample semantic segmentation masks;

[0011] Based on the predicted velocity field, a face image is back-estimated and decoded into pixel space. The text consistency loss between the face image and the text description is calculated, as well as the mask consistency loss between the face image and the semantic segmentation mask.

[0012] Construct temporal dynamic modulation coefficients based on the current time step, and multiply the weighted sum of the text consistency loss and the mask consistency loss by the modulation coefficients to obtain the multimodal consistency guidance loss;

[0013] The weighted flow matching loss and the multimodal consistency guidance loss are added together to form the total training loss. The probabilistic generation model is then jointly optimized to obtain the trained multimodal face synthesis model.

[0014] Preferably, the preprocessing of the original face image, the text description, and the semantic segmentation mask, and the input into the probabilistic generation model for multimodal fusion, to output a predicted velocity field, includes:

[0015] The original face image is converted into a noisy latent representation, the text description is encoded into a first conditional representation, and the semantic segmentation mask is encoded into a second conditional representation;

[0016] The noise latent representation, the first conditional representation, and the second conditional representation are concatenated into a multimodal token sequence;

[0017] The multimodal token sequence is input into the probabilistic generation model, and the interaction and fusion between tokens are performed through the multi-head self-attention layer of the probabilistic generation model;

[0018] The fused features are mapped to a predicted velocity field through the output layer of the probabilistic generation model.

[0019] Preferably, the step of calculating the difference weight matrix based on the pixel frequency of each semantic attribute in the training dataset includes:

[0020] The total number of pixels covered by each semantic attribute category in the statistical training dataset; wherein, the semantic attribute category is the facial semantic component corresponding to the preset pixel label in the sample semantic segmentation mask;

[0021] For each spatial location in the sample semantic segmentation mask, obtain the attribute label corresponding to the spatial location;

[0022] The differential weight of the spatial location is calculated based on the total number of pixels corresponding to the attribute label and the sum of the total number of pixels covered by all semantic attribute categories.

[0023] The differential weights of each spatial location are arranged according to their spatial location to form a differential weight matrix.

[0024] Preferably, the step of back-estimating and decoding the face image into pixel space based on the predicted velocity field includes:

[0025] Obtain the value at the current time step, and the potential noise representation obtained by preprocessing the original face image;

[0026] Based on the predicted velocity field and the value of the current time step, a one-step back estimation is performed on the noise latent representation to obtain the denoised latent representation;

[0027] The denoised latent representation is reconstructed into a pixel-space face image using a decoder of a variational autoencoder.

[0028] Preferably, calculating the text consistency loss between the face image and the text description includes:

[0029] The first visual embedding of the face image is extracted using a pre-trained visual encoder;

[0030] The first text embedding of the text description is extracted using a pre-trained text encoder;

[0031] Calculate the cosine similarity between the first visual embedding and the first text embedding;

[0032] The difference between the cosine similarity and the value 1 is used as the text consistency loss.

[0033] Preferably, calculating the mask consistency loss between the face image and the semantic segmentation mask includes:

[0034] The face image is semantically segmented using a pre-trained face parsing model to obtain a predicted segmentation map;

[0035] Calculate the cross-entropy loss between the predicted segmentation map and the semantic segmentation mask;

[0036] Calculate the dice loss between the predicted segmentation map and the semantic segmentation mask;

[0037] The cross-entropy loss and the dice loss are weighted and summed to obtain the mask consistency loss.

[0038] Preferably, constructing the temporal dynamic modulation coefficients based on the current time step includes:

[0039] Get the value at the current time step;

[0040] Using the current time step as the independent variable, the temporal dynamic modulation coefficient is calculated using the logistic function; wherein the logistic function has a preset slope parameter and inflection point parameter.

[0041] A second aspect of this invention discloses an enhanced diffusion multimodal facial synthesis system, the system comprising:

[0042] An acquisition unit is used to acquire a multimodal condition pair, wherein the multimodal condition pair includes the original face image, text description, and semantic segmentation mask;

[0043] The predicted velocity field unit is used to preprocess the original face image, the text description and the semantic segmentation mask, and input them into the probabilistic generation model for multimodal fusion, and output the predicted velocity field.

[0044] The weighted flow matching loss calculation unit is used to calculate the difference weight matrix based on the pixel frequency of each semantic attribute in the training dataset, and use the weight matrix to weight the error between the predicted velocity field and the target velocity field to obtain the weighted flow matching loss; the training dataset contains multiple sample multimodal condition pairs; the sample multimodal condition pairs include the original face image of the sample, the text description of the sample, and the semantic segmentation mask of the sample;

[0045] The consistency loss calculation unit is used to back-estimate and decode a face image into pixel space based on the predicted velocity field, calculate the text consistency loss between the face image and the text description, and calculate the mask consistency loss between the face image and the semantic segmentation mask.

[0046] The multimodal consistency guidance loss calculation unit is used to construct temporal dynamic modulation coefficients based on the current time step, and multiply the weighted sum of the text consistency loss and the mask consistency loss by the modulation coefficients to obtain the multimodal consistency guidance loss.

[0047] The joint optimization unit is used to add the weighted flow matching loss and the multimodal consistency guidance loss as the total training loss, and to jointly optimize the probabilistic generation model to obtain the trained multimodal face synthesis model.

[0048] Preferably, the predicted velocity field unit includes:

[0049] The conversion module is used to convert the original face image into a noisy latent representation, and to encode the text description into a first conditional representation and the semantic segmentation mask into a second conditional representation;

[0050] The splicing module is used to splice the noise latent representation, the first conditional representation, and the second conditional representation into a multimodal token sequence;

[0051] The input module is used to input the multimodal token sequence into the probabilistic generation model, and the tokens interact and merge through the multi-head self-attention layer of the probabilistic generation model;

[0052] The output module is used to map the fused features into a predicted velocity field through the output layer of the probability generation model.

[0053] Preferably, the weighted flow matching loss calculation unit includes:

[0054] The statistics module is used to count the total number of pixels covered by each semantic attribute category in the training dataset; wherein, the semantic attribute category is the face semantic component corresponding to the preset pixel label in the sample semantic segmentation mask;

[0055] The first acquisition module is used to acquire the attribute label corresponding to each spatial location in the sample semantic segmentation mask;

[0056] The first calculation module is used to calculate the differential weight of the spatial location based on the total number of pixels corresponding to the attribute label and the sum of the total number of pixels covered by all semantic attribute categories;

[0057] The arrangement module is used to arrange the differential weights of each spatial location according to their spatial location to form a differential weight matrix.

[0058] Based on the above embodiments of the present invention, an enhanced diffusion multimodal face synthesis method and system is provided. This method obtains a pixel-space face image through reverse estimation decoding, calculates text consistency loss and mask consistency loss respectively, and introduces temporal dynamic modulation coefficients to adaptively adjust the constraint strength according to the current time step. While avoiding large time step noise interference, it establishes explicit pixel-level alignment supervision between the generated result and multimodal conditions, thereby significantly improving the fine-grained semantic consistency of attributes such as hair color, accessories, and expressions. Addressing the problem of weak modeling ability for rare attributes, this invention calculates a difference weight matrix based on the pixel frequency of each semantic attribute in the training dataset. It weights the error between the predicted velocity field and the target velocity field to obtain a weighted flow matching loss, enabling the model to apply a stronger gradient signal to long-tailed rare attribute regions, effectively correcting the bias of uniform optimization towards high-frequency attributes. The joint optimization of the two losses solves the technical defects of inaccurate semantic alignment and weak modeling ability for rare attributes. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0060] Figure 1 A flowchart of an enhanced diffusion multimodal face synthesis method provided in an embodiment of the present invention;

[0061] Figure 2 Another flowchart of an enhanced diffusion multimodal face synthesis method provided in an embodiment of the present invention;

[0062] Figure 3 This is a structural block diagram of an enhanced diffusion multimodal facial synthesis system provided in an embodiment of the present invention. Detailed Implementation

[0063] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0064] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0065] As the background technology shows, current diffusion models and flow matching in multimodal face synthesis lack pixel-level alignment constraints and their optimization mechanisms contradict long-tail distributions, resulting in insufficient fine-grained semantic bias and rare attribute modeling, which restricts their practical application.

[0066] Therefore, this invention provides an enhanced diffusion multimodal face synthesis method and system. It obtains a pixel-space face image through reverse estimation decoding, calculates text consistency loss and mask consistency loss separately, and introduces temporal dynamic modulation coefficients to adaptively adjust the constraint strength according to the current time step. While avoiding large time step noise interference, it establishes explicit pixel-level alignment supervision between the generated result and multimodal conditions, thereby significantly improving the fine-grained semantic consistency of attributes such as hair color, accessories, and expressions. Addressing the problem of weak modeling ability for rare attributes, this invention calculates a difference weight matrix based on the pixel frequency of each semantic attribute in the training dataset. It weights the error between the predicted velocity field and the target velocity field to obtain a weighted flow matching loss, enabling the model to apply a stronger gradient signal to long-tailed rare attribute regions, effectively correcting the bias of uniform optimization towards high-frequency attributes. The joint optimization of the two losses solves the technical defects of inaccurate semantic alignment and weak modeling ability for rare attributes.

[0067] See Figure 1 The diagram illustrates a flowchart of an enhanced diffusion multimodal face synthesis method provided by an embodiment of the present invention, the method comprising:

[0068] It should be noted that the core solution of this method The overall training process and architecture are as follows Figure 2 As shown, its complete workflow consists of three mutually cooperating core modules (namely (a) latent space modeling, (b) long-tail adaptive learning, and (c) explicit conditional consistency guidance), which together achieve joint optimization of "multimodal interaction in latent space, explicit constraints in pixel space, and long-tail adaptation in the training process".

[0069] Step S101: Obtain multimodal condition pairs, which include the original face image, text description, and semantic segmentation mask.

[0070] In the specific implementation step S101, a multimodal condition pair is obtained, which includes the original face image. Text description and semantic segmentation mask .

[0071] For example: Figure 2 The original face image shown in (a) of the latent space modeling module. Text description (like Figure 2The text shows "this woman has blond wavy long hair" and a semantic segmentation mask. .

[0072] Step S102: Preprocess the original face image, text description and semantic segmentation mask, and input them into the probabilistic generation model for multimodal fusion, and output the predicted velocity field.

[0073] In the specific implementation step S102, the original face image is subjected to noise addition and latent mapping processing, the text description and semantic segmentation mask are encoded, and the processed noise latent representation and conditional representation are jointly input into the probabilistic generation model for multimodal fusion, and the predicted velocity field is output.

[0074] It is understood that step S102 specifically includes the following processes (processes A1 to A4):

[0075] Process A1: Convert the original face image into a noisy latent representation, encode the text description into a first conditional representation, and encode the semantic segmentation mask into a second conditional representation.

[0076] It should be noted that encoders using variational autoencoders (VAEs) (such as...) Figure 2 The "shown" ") the original face image Compression into the initial latent representation Random noise (e.g., sampled from a preset noise distribution) is generated. Figure 2 (as shown by "noise"); based on the random sampling time step Random noise is superimposed onto the initial latent representation. The potential representation of noise is obtained. .

[0077] It is understandable that text encoders trained using pre-trained models (such as Contrastive Language–Image Pre-training (CLIP) text encoders) can achieve this. Figure 2 The "shown" Convert the text description into a sequence of text tokens. This serves as the first conditional representation. A lightweight linear projection layer is used to mask the semantic segmentation. Mapped to a masked token sequence , as a second condition.

[0078] Process A2: Concatenate the noise latent representation, the first conditional representation, and the second conditional representation into a multimodal token sequence.

[0079] It is understandable that noise is potentially represented First condition representation Second condition representation Concatenate them into a multimodal token sequence (token).

[0080] Process A3: Input the multimodal token sequence into the probabilistic generation model, and perform interaction and fusion between tokens through the multi-head self-attention layer of the probabilistic generation model.

[0081] It should be noted that the probabilistic generation model is based on a backbone network constructed using a Diffusion Transformer (DiT).

[0082] In implementing process A3, the multimodal token sequence (token) is input into the probabilistic generation model (e.g., Figure 2 As shown in “DiT-Blocks”, DiT Blocks utilizes its powerful multi-head self-attention mechanism to achieve deep interaction and fusion of multimodal token sequences in the latent space.

[0083] Process A4: The fused features are mapped to the predicted velocity field through the output layer of the probabilistic generation model.

[0084] Understandably, the predicted velocity field is This predicted velocity field is the core of the two subsequent innovative modules. These two modules are respectively... Figure 2 The examples shown (b) long-tail adaptive learning and (c) explicit conditional consistency guidance both work on this basis.

[0085] Step S103: Calculate the difference weight matrix based on the pixel frequency of each semantic attribute in the training dataset, and use the weight matrix to weight the error between the predicted velocity field and the target velocity field to obtain the weighted flow matching loss.

[0086] Understandably, to address the modeling bias issue of rare attributes caused by long-tail distribution, this embodiment introduces a differential gradient reweighting mechanism based on attribute frequency in the standard flow matching training objective. The process of this mechanism corresponds to... Figure 2 The part shown in (b) is long-tail adaptive learning.

[0087] In the specific implementation step S103, a difference weight matrix is ​​calculated based on the pixel frequency of each semantic attribute in the training dataset to apply a higher optimization intensity to rare attribute regions during training. The difference weight matrix is ​​multiplied by the element-wise error between the predicted velocity field and the target velocity field to obtain the weighted error; the mean of the sum of squares of the weighted error is calculated as the weighted flow matching loss.

[0088] It should be added that the specific process of calculating the difference weight matrix based on the pixel frequency of each semantic attribute in the training dataset includes (processes B1 to B4):

[0089] Procedure B1: Count the total number of pixels covered by each semantic attribute category in the training dataset.

[0090] It should be noted that the training dataset contains multiple sample multimodal conditional pairs; each sample multimodal conditional pair includes the original face image, sample text description, and sample semantic segmentation mask. The semantic attribute category is the face semantic component corresponding to the preset pixel label in the sample semantic segmentation mask.

[0091] Understandably, before training begins, a global statistical analysis is performed on the entire training dataset to obtain the total pixel frequency for each semantic attribute category. In other words, the total frequency of pixels for each semantic attribute category in the entire training dataset is calculated. The total number of pixels covered by (e.g., "hair", "eyes", "earrings", "necklace") This reflects the "commonness" of the attribute in the data.

[0092] For example Figure 2 The “attribute frequency calculation” shown refers to the frequency statistics of attributes. The pie chart is labeled as follows: hair (32%), background (28%), skin (25%), neck (4%), clothing (3%), nose (2%), and hat (1%).

[0093] Process B2: For each spatial location in the sample semantic segmentation mask, obtain the attribute label corresponding to the spatial location.

[0094] Process B3: Calculate the differential weight of spatial location based on the total number of pixels corresponding to the attribute labels and the sum of the total number of pixels covered by all semantic attribute categories.

[0095] It is understandable that for each sample semantic segmentation mask According to each spatial location Corresponding attribute tags Calculate a differential weight The calculation formula is:

[0096] (1).

[0097] In formula (1), This represents the total number of semantic attribute categories. The design of formula (1) ensures that the frequency of occurrence is low ( Rare attribute categories (with lower values) receive higher weight. High-frequency attribute categories receive lower weights.

[0098] Process B4: Arrange the differential weights of each spatial location according to their spatial location to form a differential weight matrix.

[0099] It is understandable that the differential weights of each spatial location are... According to spatial location Arranged to form a difference weight matrix ,like Figure 2 The weight matrix (w) is shown.

[0100] In practical applications, the difference weight matrix will be used. With the predicted velocity field (e.g.) Figure 2 The "Predicted" option is shown. ) and target velocity field (e.g. Figure 2 The "Target" shown The element-wise errors between the two sides are multiplied to obtain the weighted error; the mean of the sum of squares of the weighted errors is calculated as the weighted flow matching loss.

[0101] Specifically, the calculated weight matrix Applied to the standard flow matching loss function, a long-tailed adaptive optimized flow matching loss is formed. ):

[0102] (2).

[0103] In formula (2), This represents element-wise multiplication. This represents the target velocity. This mechanism ensures that during backpropagation, the model applies stronger gradient signals to rare attribute regions with high weights (such as pixel regions where earrings or necklaces are worn), thereby enabling them to be fully learned and effectively balancing the optimization intensity among different attributes.

[0104] Step S104: Perform inverse estimation and decoding of the face image into pixel space based on the predicted velocity field, calculate the text consistency loss between the face image and the text description, and calculate the mask consistency loss between the face image and the semantic segmentation mask.

[0105] Understandably, to accurately establish fine-grained semantic relationships between the generated results and input conditions, this embodiment overcomes the limitations of existing technologies that rely solely on implicit learning in the latent space. It innovatively constructs explicit consistency constraints with temporal dynamic modulation in the pixel space, and the related processes correspond to… Figure 2 The section shown is (c) explicit conditional consistency guidance.

[0106] In the specific implementation step S104, a one-step inverse estimation is performed based on the predicted velocity field to obtain the denoised latent representation, and the denoised latent representation is reconstructed into a face image in pixel space through the decoder of the variational autoencoder; then, the text consistency loss between the face image and the text description is calculated, and the mask consistency loss between the face image and the semantic segmentation mask is calculated.

[0107] It should be noted that the process of inversely estimating and decoding the face image into pixel space based on the predicted velocity field specifically includes the following steps (steps C1 to C3):

[0108] Process C1: Obtain the value of the current time step and the noise latent representation obtained by preprocessing the original face image.

[0109] Process C2: Based on the predicted velocity field and the value of the current time step, perform a one-step back estimation on the noisy latent representation to obtain the denoised latent representation.

[0110] Specifically, by subtracting the product of the current time step and the predicted velocity field from the noisy latent variable using formula (3), a single-step approximate estimate of the initial latent variable is obtained.

[0111] (3).

[0112] In formula (3), This is the latent representation after denoising; It is a noisy latent variable; This represents the value at the current time step. To predict the velocity field.

[0113] Process C3: The denoised latent representation is reconstructed into a pixel-space face image using a decoder from a variational autoencoder.

[0114] In other words, the VAE decoder will Reconstructed face image in pixel space .

[0115] Understandably, this involves calculating the text consistency loss between the face image and the text description. Specifically, it includes: extracting a first visual embedding of a face image through a pre-trained visual encoder; extracting a first text embedding of a text description through a pre-trained text encoder; calculating the cosine similarity between the first visual embedding and the first text embedding; and using the difference between the cosine similarity and the value 1 as the text consistency loss.

[0116] In other words, the pre-trained CLIP model is used to extract face images separately. Visual embedding and text description Text embedding The negative value of the cosine similarity between the two is calculated as the loss:

[0117] (4).

[0118] It should be noted that this text consistency loss ( Directly constrain the generated image to align with the text description in terms of semantic content.

[0119] Understandably, the mask consistency loss between the face image and the semantic segmentation mask is calculated. Specifically, this includes: performing semantic segmentation on face images using a pre-trained face parsing model to obtain a predicted segmentation map; calculating the cross-entropy loss between the predicted segmentation map and the semantic segmentation mask; calculating the dice loss between the predicted segmentation map and the semantic segmentation mask; and weighted summing the cross-entropy loss and the dice loss to obtain the mask consistency loss.

[0120] In other words, pre-trained face parsing models (such as Facial Representation Learning, FaRL) are used to analyze face images. Perform semantic segmentation to obtain the predicted segmentation map. The predicted segmentation map With a given semantic segmentation mask Compare and calculate the weighted portfolio loss:

[0121] (5).

[0122] In formula (5), For cross-entropy loss, The value is set to Dice loss; the weighting coefficients for both cross-entropy loss and dice loss are set to 0.5.

[0123] It should be noted that this mask consistency loss ( Ensure that the structure and layout of the generated image are highly consistent with the input mask conditions.

[0124] Step S105: Construct temporal dynamic modulation coefficients based on the current time step, and multiply the weighted sum of text consistency loss and mask consistency loss by the modulation coefficients to obtain the multimodal consistency guidance loss.

[0125] In the specific implementation step S105, the value of the current time step is obtained; using the current time step as the independent variable, the temporal dynamic modulation coefficient is calculated using the logistic function. The weighted sum of the text consistency loss and the mask consistency loss is multiplied by the modulation coefficient to obtain the multimodal consistency guidance loss. The weight coefficients of both the text consistency loss and the mask consistency loss are set to 0.5.

[0126] Understandably, the logistic function has preset slope and inflection point parameters. This ensures that the modulation coefficient is closer to 1 the smaller the current time step, and closer to 0 the larger the current time step. Specifically, the slope parameter is set to 10, and the inflection point parameter is set to 0.4.

[0127] In practical applications, considering the diffusion process, when time step Face images obtained based on single-step back estimation when the noise level is high (i.e., the noise intensity is high). It may not be reliable enough. If a strong constraint is applied directly at this time, noise will be introduced into the training. To this end, this embodiment designs a temporal dynamic modulation function β(t) based on the Sigmoid function, as shown in formula (6), to adaptively adjust the strength of the consistency constraint according to the current time step.

[0128] (6).

[0129] In formula (6), Control the slope of the curve. This is the inflection point. When When the value is small (and the estimate is reliable), Enhance consistency in guidance; when When it is large, Reduce or weaken constraints to avoid misleading decisions.

[0130] For details, see Figure 2 The "β(t) time-series dynamic modulation" shown in the graph can be seen as follows: the horizontal axis is time t, and the vertical axis is the weighting coefficient β. When When smaller, ,when When it is large, Decrease.

[0131] It should be noted that the multimodal consistency guidance loss is obtained by multiplying the weighted sum of the text consistency loss and the mask consistency loss by the modulation coefficients:

[0132] (7).

[0133] In formula (7), β(t) is the temporal dynamic modulation function; This represents the text consistency loss. For mask consistency loss; balance coefficient and All are set to 0.5.

[0134] Step S106: Add the weighted flow matching loss and the multimodal consistency guidance loss to obtain the total training loss, and perform joint optimization on the probabilistic generation model to obtain the trained multimodal face synthesis model.

[0135] In the specific implementation step S106, the weighted flow matching loss and the multimodal consistency guidance loss are added together as the total training loss, as shown in formula (8). The probabilistic generation model is jointly optimized so that the probabilistic generation model learns the flow matching prior in the latent space and the explicit semantic alignment in the pixel space at the same time, and automatically pays attention to the long-tail attributes during the training process, and finally obtains the trained multimodal face synthesis model.

[0136] (8).

[0137] In formula (8), Total training loss; For long-tail adaptive optimized flow matching loss; The loss is guided by multimodal consistency.

[0138] During the joint optimization process, the parameters of the probabilistic generation model are updated simultaneously to improve efficiency and robustness.

[0139] Specifically, when training the basic DiT model, only the visual embedding layer and some Transformer linear layers are fine-tuned using the Low-Rank Adaptation (LoRA) module, which significantly reduces the number of trainable parameters and computational overhead.

[0140] At the same time, with probability Independently and randomly discard text descriptions or semantic segmentation mask This enables the model to have robust generation capabilities for handling single-modal conditional inputs (text only or mask only), enhancing the practicality and flexibility of the solution.

[0141] In this embodiment of the invention, multimodal condition pairs are acquired and preprocessed, and a probabilistic generative model is used to output the predicted velocity field. A differential weight matrix is ​​calculated based on the pixel frequency of semantic attributes to obtain a weighted flow matching loss, achieving differentiated learning of long-tailed rare attributes. Face images are decoded through back-estimation, and text consistency loss and mask consistency loss are calculated separately. A multimodal consistency guidance loss is constructed using temporal dynamic modulation coefficients, achieving explicit semantic alignment in the pixel space. The two losses are added together for joint optimization, simultaneously addressing the problems of inaccurate semantic alignment and weak modeling ability for rare attributes. This effectively improves the fine-grained semantic accuracy and robustness to long-tailed attributes in multimodal face synthesis.

[0142] Corresponding to the enhanced diffusion multimodal face synthesis method provided in the above embodiments of the present invention, see also... Figure 3 The diagram shows a structural block diagram of an enhanced diffusion multimodal facial synthesis system provided by an embodiment of the present invention.

[0143] The system includes: an acquisition unit 301, a velocity field prediction unit 302, a weighted flow matching loss calculation unit 303, a consistency loss calculation unit 304, a multimodal consistency guidance loss calculation unit 305, and a joint optimization unit 306.

[0144] The acquisition unit 301 is used to acquire a multimodal condition pair, which includes the original face image, text description and semantic segmentation mask.

[0145] The predicted velocity field unit 302 is used to preprocess the original face image, text description and semantic segmentation mask, and input them into the probabilistic generation model for multimodal fusion, and output the predicted velocity field.

[0146] The weighted flow matching loss calculation unit 303 is used to calculate the difference weight matrix based on the pixel frequency of each semantic attribute in the training dataset, and use the weight matrix to weight the error between the predicted velocity field and the target velocity field to obtain the weighted flow matching loss; the training dataset contains multiple sample multimodal condition pairs; the sample multimodal condition pairs include the original face image of the sample, the text description of the sample, and the semantic segmentation mask of the sample.

[0147] The consistency loss calculation unit 304 is used to back-estimate and decode the face image into pixel space based on the predicted velocity field, calculate the text consistency loss between the face image and the text description, and calculate the mask consistency loss between the face image and the semantic segmentation mask.

[0148] The multimodal consistency guidance loss calculation unit 305 is used to construct temporal dynamic modulation coefficients based on the current time step, and multiply the text consistency loss and mask consistency loss by the weighted summation and modulation coefficients to obtain the multimodal consistency guidance loss.

[0149] The joint optimization unit 306 is used to add the weighted flow matching loss and the multimodal consistency guidance loss as the total training loss to jointly optimize the probabilistic generation model and obtain the trained multimodal face synthesis model.

[0150] Based on the aforementioned functional system, this invention acquires multimodal condition pairs through an acquisition unit and outputs a predicted velocity field through a predicted velocity field unit. A consistency loss calculation unit back-estimates and decodes the predicted velocity field into a pixel-space face image, calculating text consistency loss and mask consistency loss respectively, establishing explicit pixel-level alignment supervision. This effectively overcomes the fine-grained semantic bias caused by implicit modeling in existing technologies, ensuring accurate matching of attributes such as hair color, accessories, and expressions with input conditions. Simultaneously, a weighted flow matching loss calculation unit calculates a difference weight matrix based on the pixel frequency of each semantic attribute in the training dataset, weighting the error between the predicted and target velocity fields to obtain a weighted flow matching loss. This allows the model to apply stronger gradient signals to rare attribute regions, solving the problem of weak long-tail modeling ability caused by the uniform optimization mechanism's bias towards high-frequency attributes. A multimodal consistency guidance loss calculation unit introduces temporal dynamic modulation coefficients to avoid large time-step noise interference. A joint optimization unit adds the two losses for joint optimization, simultaneously improving semantic alignment accuracy and robustness to rare attributes, significantly improving the overall performance of multimodal face synthesis.

[0151] Combination Figure 3 The content shown is the prediction velocity field unit 302, which includes: a conversion module, a splicing module, an input module, and an output module.

[0152] The conversion module is used to convert the original face image into a noisy latent representation, and to encode the text description into a first conditional representation and the semantic segmentation mask into a second conditional representation.

[0153] The splicing module is used to splice the noise latent representation, the first conditional representation, and the second conditional representation into a multimodal token sequence.

[0154] The input module is used to input multimodal token sequences into the probabilistic generation model, and the tokens interact and merge through the multi-head self-attention layer of the probabilistic generation model.

[0155] The output module is used to map the fused features to a predicted velocity field through the output layer of the probabilistic generation model.

[0156] Combination Figure 3 The content shown is that the weighted flow matching loss calculation unit 303 includes: a statistics module, a first acquisition module, a first calculation module, and a permutation module.

[0157] The statistics module is used to count the total number of pixels covered by each semantic attribute category in the training dataset; where the semantic attribute category is the face semantic component corresponding to the preset pixel label in the sample semantic segmentation mask.

[0158] The first acquisition module is used to acquire the attribute label corresponding to each spatial location in the sample semantic segmentation mask.

[0159] The first calculation module is used to calculate the differential weight of spatial location based on the total number of pixels corresponding to the attribute label and the sum of the total number of pixels covered by all semantic attribute categories.

[0160] The arrangement module is used to arrange the differential weights of each spatial location according to their spatial location, forming a differential weight matrix.

[0161] Combination Figure 3 The consistency loss calculation unit 304 shown includes: a second acquisition module, a reverse estimation module, and a reconstruction module.

[0162] The second acquisition module is used to acquire the value of the current time step and the noise potential representation obtained by preprocessing the original face image.

[0163] The inverse estimation module is used to perform a one-step inverse estimation of the noisy latent representation based on the predicted velocity field and the value of the current time step, so as to obtain the denoised latent representation.

[0164] The reconstruction module is used to reconstruct the denoised latent representation into a pixel-space face image through a decoder of a variational autoencoder.

[0165] Combination Figure 3 The consistency loss calculation unit 304 shown includes: a first extraction module, a second extraction module, a second calculation module, and a difference calculation module.

[0166] The first extraction module is used to extract the first visual embedding of the face image through a pre-trained visual encoder.

[0167] The second extraction module is used to extract the first text embedding of the text description through a pre-trained text encoder.

[0168] The second calculation module is used to calculate the cosine similarity between the first visual embedding and the first text embedding.

[0169] The difference calculation module is used to take the difference between the cosine similarity and the value 1 as the text consistency loss.

[0170] Combination Figure 3 The consistency loss calculation unit 304 shown includes: a segmentation module, a third calculation module, a fourth calculation module, and a summation module.

[0171] The segmentation module is used to perform semantic segmentation on face images using a pre-trained face parsing model to obtain a predicted segmentation map.

[0172] The third computation module is used to calculate the cross-entropy loss between the predicted segmentation map and the semantic segmentation mask.

[0173] The fourth calculation module is used to calculate the dice loss between the predicted segmentation map and the semantic segmentation mask.

[0174] The summation module is used to perform a weighted summation of the cross-entropy loss and the dice loss to obtain the mask consistency loss.

[0175] Combination Figure 3 The multimodal consistency guided loss calculation unit 305 shown includes: a time step acquisition module and a fifth calculation module.

[0176] The time step acquisition module is used to obtain the value of the current time step.

[0177] The fifth calculation module is used to calculate the temporal dynamic modulation coefficients using the current time step as the independent variable and the logistic function; the logistic function has preset slope parameters and inflection point parameters.

[0178] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0179] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0180] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for enhancing diffusion-modal facial synthesis, characterized in that, The method includes: Obtain a multimodal condition pair, which includes the original face image, text description, and semantic segmentation mask; The original face image, the text description, and the semantic segmentation mask are preprocessed and input into a probabilistic generation model for multimodal fusion, and the predicted velocity field is output. Based on the pixel frequency of each semantic attribute in the training dataset, a difference weight matrix is ​​calculated, and the error between the predicted velocity field and the target velocity field is weighted using the weight matrix to obtain a weighted flow matching loss; the training dataset contains multiple sample multimodal condition pairs; the sample multimodal condition pairs include original face images of samples, sample text descriptions, and sample semantic segmentation masks; Based on the predicted velocity field, a face image is back-estimated and decoded into pixel space. The text consistency loss between the face image and the text description is calculated, as well as the mask consistency loss between the face image and the semantic segmentation mask. Construct temporal dynamic modulation coefficients based on the current time step, and multiply the weighted sum of the text consistency loss and the mask consistency loss by the modulation coefficients to obtain the multimodal consistency guidance loss; The weighted flow matching loss and the multimodal consistency guidance loss are added together to form the total training loss. The probabilistic generation model is then jointly optimized to obtain the trained multimodal face synthesis model.

2. The method according to claim 1, characterized in that, The preprocessing of the original face image, the text description, and the semantic segmentation mask, followed by inputting them into a probabilistic generation model for multimodal fusion, and outputting a predicted velocity field, includes: The original face image is converted into a noisy latent representation, the text description is encoded into a first conditional representation, and the semantic segmentation mask is encoded into a second conditional representation; The noise latent representation, the first conditional representation, and the second conditional representation are concatenated into a multimodal token sequence; The multimodal token sequence is input into the probabilistic generation model, and the interaction and fusion between tokens are performed through the multi-head self-attention layer of the probabilistic generation model; The fused features are mapped to a predicted velocity field through the output layer of the probabilistic generation model.

3. The method according to claim 1, characterized in that, The step of calculating the difference weight matrix based on the pixel frequency of each semantic attribute in the training dataset includes: The total number of pixels covered by each semantic attribute category in the statistical training dataset; wherein, the semantic attribute category is the facial semantic component corresponding to the preset pixel label in the sample semantic segmentation mask; For each spatial location in the sample semantic segmentation mask, obtain the attribute label corresponding to the spatial location; The differential weight of the spatial location is calculated based on the total number of pixels corresponding to the attribute label and the sum of the total number of pixels covered by all semantic attribute categories. The differential weights of each spatial location are arranged according to their spatial location to form a differential weight matrix.

4. The method according to claim 1, characterized in that, The step of back-estimating and decoding the face image into pixel space based on the predicted velocity field includes: Obtain the value at the current time step, and the potential noise representation obtained by preprocessing the original face image; Based on the predicted velocity field and the value of the current time step, a one-step back estimation is performed on the noise latent representation to obtain the denoised latent representation; The denoised latent representation is reconstructed into a pixel-space face image using a decoder of a variational autoencoder.

5. The method according to claim 1, characterized in that, The calculation of the text consistency loss between the face image and the text description includes: The first visual embedding of the face image is extracted using a pre-trained visual encoder; The first text embedding of the text description is extracted using a pre-trained text encoder; Calculate the cosine similarity between the first visual embedding and the first text embedding; The difference between the cosine similarity and the value 1 is used as the text consistency loss.

6. The method according to claim 1, characterized in that, The calculation of the mask consistency loss between the face image and the semantic segmentation mask includes: The face image is semantically segmented using a pre-trained face parsing model to obtain a predicted segmentation map; Calculate the cross-entropy loss between the predicted segmentation map and the semantic segmentation mask; Calculate the dice loss between the predicted segmentation map and the semantic segmentation mask; The cross-entropy loss and the dice loss are weighted and summed to obtain the mask consistency loss.

7. The method according to claim 1, characterized in that, The construction of temporal dynamic modulation coefficients based on the current time step includes: Get the value at the current time step; Using the current time step as the independent variable, the temporal dynamic modulation coefficient is calculated using the logistic function; wherein the logistic function has a preset slope parameter and inflection point parameter.

8. An enhanced diffusion multimodal facial synthesis system, characterized in that, The system includes: An acquisition unit is used to acquire a multimodal condition pair, wherein the multimodal condition pair includes the original face image, text description, and semantic segmentation mask; The predicted velocity field unit is used to preprocess the original face image, the text description and the semantic segmentation mask, and input them into the probabilistic generation model for multimodal fusion, and output the predicted velocity field. The weighted flow matching loss calculation unit is used to calculate the difference weight matrix based on the pixel frequency of each semantic attribute in the training dataset, and use the weight matrix to weight the error between the predicted velocity field and the target velocity field to obtain the weighted flow matching loss; the training dataset contains multiple sample multimodal condition pairs; the sample multimodal condition pairs include the original face image of the sample, the text description of the sample, and the semantic segmentation mask of the sample; The consistency loss calculation unit is used to back-estimate and decode a face image into pixel space based on the predicted velocity field, calculate the text consistency loss between the face image and the text description, and calculate the mask consistency loss between the face image and the semantic segmentation mask. The multimodal consistency guidance loss calculation unit is used to construct temporal dynamic modulation coefficients based on the current time step, and multiply the weighted sum of the text consistency loss and the mask consistency loss by the modulation coefficients to obtain the multimodal consistency guidance loss. The joint optimization unit is used to add the weighted flow matching loss and the multimodal consistency guidance loss as the total training loss, and to jointly optimize the probabilistic generation model to obtain the trained multimodal face synthesis model.

9. The system according to claim 8, characterized in that, The predicted velocity field unit includes: The conversion module is used to convert the original face image into a noisy latent representation, and to encode the text description into a first conditional representation and the semantic segmentation mask into a second conditional representation; The splicing module is used to splice the noise latent representation, the first conditional representation, and the second conditional representation into a multimodal token sequence; The input module is used to input the multimodal token sequence into the probabilistic generation model, and the interaction and fusion between tokens are performed through the multi-head self-attention layer of the probabilistic generation model; The output module is used to map the fused features into a predicted velocity field through the output layer of the probability generation model.

10. The system according to claim 8, characterized in that, The weighted stream matching loss calculation unit includes: The statistics module is used to count the total number of pixels covered by each semantic attribute category in the training dataset; wherein, the semantic attribute category is the face semantic component corresponding to the preset pixel label in the sample semantic segmentation mask; The first acquisition module is used to acquire the attribute label corresponding to each spatial location in the sample semantic segmentation mask; The first calculation module is used to calculate the differential weight of the spatial location based on the total number of pixels corresponding to the attribute label and the sum of the total number of pixels covered by all semantic attribute categories. The arrangement module is used to arrange the differential weights of each spatial location according to their spatial location to form a differential weight matrix.

Images