Consistency role zero sample generation method based on diffusion model

Abstract

According to the consistency role zero sample generation method based on the diffusion model, under the condition that no reference image exists, analysis from text to semantic features is achieved only by inputting a text sequence into a hierarchical semantic analysis module, and the understanding ability of the model for fine-grained attributes in text description is improved; aligning the text vector and the semantic feature vector through a comparison semantic alignment module to generate an initial image; a progressive attention guiding module is used for providing hierarchical attention guiding, different attributes can be adaptively concerned in the denoising process of an initial image, and the quality and consistency of generated results are guaranteed; and a semantic consistency regularization module is used for preventing semantic drift in the generation process, so that an image corresponding to the text sequence is generated.

Description

Technical Field

[0001] The present invention relates to the technical field of image generation, and in particular to a method for generating consistent character zero samples based on a diffusion model. Background Art

[0002] The diffusion model is one of the most important generative models in the field of deep learning in recent years. It converts random noise into high-quality images through an iterative denoising process. In the character generation task, the existing technology has the following problems and limitations:

[0003] Existing diffusion models have difficulty maintaining the consistency of characters during multiple generation processes, including attributes such as facial features, clothing styles, and body shapes that are prone to change. Traditional diffusion models represented by denoising diffusion probabilistic models (DDPM) and implicit diffusion models (DDIM) mainly focus on the quality of a single generation and lack a consistency constraint mechanism across the generation process.

[0004] Most consistency generation methods require reference images as conditions. For example, InstantID uses facial feature encoding technology to ensure identity consistency in different scenarios and generate identity features close to the reference image; IP-Adapter-FaceID uses face embedding as a conditional control to adjust the style and features of the generated content to match the target identity; both methods rely on facial features as identity constraints, which limits their application in pure text-driven scenarios.

[0005] Existing models do not have sufficient understanding of the character attributes in text descriptions. For example, StoryMaker introduces location-aware attention to enhance feature fusion capabilities, and performs intelligent processing based on the location of features in space to improve the coherence and consistency of generated content. Although this method introduces an attribute parsing mechanism, it often only focuses on surface features and lacks understanding and modeling of deep semantics.

[0006] In terms of model architecture, existing technologies have the problem of representation space fragmentation. There is a lack of effective alignment and fusion mechanism between text space, image space and semantic space, resulting in a semantic gap between the generated results and the text description. Summary of the invention

[0007] The present invention provides a method for generating consistent characters by zero samples based on a diffusion model, the purpose of which is to achieve consistent generation of characters only through text description in the absence of a reference image.

[0008] In order to achieve the above object, the present invention provides a method for generating zero samples of consistent roles based on a diffusion model, comprising:

[0009] Step 1, obtaining the text sequence to be processed;

[0010] Step 2, constructing a diffusion model, the diffusion model includes a hierarchical semantic parsing module for extracting semantic features, a contrastive semantic alignment module for aligning text space and semantic feature space, a progressive attention guidance module for providing attention guidance, and a semantic consistency regularization module for generating images, and training the diffusion model to obtain an image generation model;

[0011] Step 3: Input the text sequence to be processed into the image generation model for generation, and obtain an image corresponding to the text sequence to be processed.

[0012] Further, the hierarchical semantic parsing module includes a word embedding unit, an attribute attention unit, and a semantic pooling unit;

[0013] The word embedding unit is used to encode the text sequence to be processed and obtain the text vector;

[0014] The attribute attention unit is used to perform attention calculation on the text vector to obtain the appearance feature matrix, posture feature matrix, and style feature matrix;

[0015] The semantic pooling unit is used to perform pooling processing on the appearance feature matrix, posture feature matrix, and style feature matrix respectively to obtain the semantic feature set and input it into the contrast semantic alignment module, the progressive attention guidance module, and the semantic consistency regularization module respectively.

[0016] Specifically, the computational expression of the semantic feature set is:

[0017] S=(Sapp,Spose,Sstyle)

[0018]

[0019] Where S represents the semantic feature set, Sapp represents the appearance feature matrix, Spose represents the posture feature matrix, Sstyle represents the style feature matrix, Pool(·) represents the attention weighted pooling function, σ(·) represents the activation function, FNNapp represents the feedforward neural network of appearance features, FNNpose represents the feedforward neural network of posture features, FNNstyle represents the feedforward neural network of style features, Happ represents the attention feature map of appearance attributes, Hpose represents the attention feature map of posture attributes, Hstyle represents the attention feature map of style attributes, d represents the dimension, Represents the real number space identity.

[0020] Further, the contrastive semantic alignment module includes a feature projection unit, an InFoNCE loss calculation unit, a mutual information maximization unit, and an alignment loss calculation unit;

[0021] The input end of the feature projection unit is connected to the output end of the word embedding unit and the output end of the semantic pooling unit respectively, the output end of the feature projection unit is connected to the input end of the InFoNCE loss calculation unit, the output end of the InFoNCE loss calculation unit is connected to the input end of the mutual information maximization unit, the output end of the mutual information maximization unit is connected to the input end of the alignment loss calculation unit, and the output end of the alignment loss calculation unit is connected to the second input end of the progressive attention guidance module.

[0022] Furthermore, the alignment loss calculation unit is used to calculate the alignment loss value, and the text projection vector is aligned with the semantic feature projection vector through the alignment loss value to generate the initial image. The calculation expression is:

[0023] L a lign=Lcsa+λmiLmi

[0024]

[0025] Lmi=-I(z s ;z t )

[0026] Among them, L a lign represents the alignment loss value, Lcsa represents the contrast loss value, λmi represents the mutual information loss weight, Lmi represents the mutual information maximization loss value, E represents the expectation operator, exp(·) represents the natural exponential function, z s represents the semantic feature projection vector, s represents the semantic feature vector, represents the text projection vector that matches the semantic feature projection vector, represents the text projection vector that does not match the semantic feature projection vector, t represents the text vector, τ represents the temperature parameter, sim(·) represents the cosine similarity function, I(z s ;z t ) represents the semantic feature projection vector z s With the text projection vector z t The mutual information between them.

[0027] Further, the progressive attention guidance module includes a multi-attribute attention unit, a time-varying weight calculation unit, an attention fusion unit, and a diffusion update unit;

[0028] The input end of the multi-attribute attention unit is connected to the output end of the alignment loss calculation unit and the output end of the semantic pooling unit respectively, the input end of the multi-attribute attention unit is connected to the input end of the time-varying weight calculation unit, the output end of the time-varying weight calculation unit is connected to the input end of the attention fusion unit, the output end of the attention fusion unit is connected to the input end of the diffusion update unit, and the output end of the diffusion update unit is connected to the second input end of the semantic consistency regularization module.

[0029] Furthermore, the multi-attribute attention unit is used to perform multi-attribute attention calculation on the initial image, and the calculation expression is:

[0030]

[0031] Among them, Attapp represents the appearance attention feature map, Attpose represents the pose attention feature map, Attstyle represents the style attention feature map, x represents the initial image, Sapp represents the appearance feature matrix, Spose represents the pose feature matrix, Sstyle represents the style feature matrix, Q represents the query matrix, K represents the key matrix, V represents the value matrix, d represents the attention feature dimension, and T represents the matrix transpose.

[0032] Furthermore, the attention fusion unit is used to fuse the attribute attention feature map of the initial image with the time-varying weight coefficient of each attribute attention to obtain the fused attention. The calculation expression is:

[0033] PAG(x,S)=λapp(t)·Attapp+λpose(t)·Attpose+λstyle(t)·Attstyle

[0034] Among them, PAG represents the fused attention feature map, λapp(t) represents the time-varying weight coefficient of the appearance attention feature map, λpose(t) represents the time-varying weight coefficient of the pose attention feature map, λstyle(t) represents the time-varying weight coefficient of the style attention feature map, and t represents the diffusion time step.

[0035] Specifically, the semantic consistency regularization module is used to:

[0036] Construct multiple constraints for the denoised image based on the semantic feature set;

[0037] The denoised image is regenerated into an image corresponding to the text sequence to be processed based on multiple constraints.

[0038] Furthermore, the multiple constraints include feature reconstruction constraints, KL divergence constraints, cycle consistency constraints, and regularization constraints.

[0039] The above scheme of the present invention has the following beneficial effects:

[0040] The present invention obtains a text sequence to be processed; constructs a diffusion model including a hierarchical semantic parsing module, a contrastive semantic alignment module, a progressive attention guidance module, and a semantic consistency regularization module, and trains the diffusion model to obtain an image generation model; inputs the text sequence to be processed into the image generation model for generation, and obtains an image corresponding to the text sequence to be processed; compared with the prior art, the present invention implements text-to-semantic feature parsing in the absence of a reference image only by inputting the text sequence into a hierarchical semantic parsing module, thereby significantly improving the model's ability to understand fine-grained attributes in text descriptions; aligns a text vector with a semantic feature vector through a contrastive semantic alignment module to generate an initial image; uses a progressive attention guidance module to provide hierarchical attention guidance, and can adaptively focus on different attributes during the denoising process of the initial image, thereby ensuring the quality and consistency of the generated result; and uses a semantic consistency regularization module to prevent semantic drift during the generation process, thereby generating an image corresponding to the text sequence.

[0041] Other beneficial effects of the present invention will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of a flow chart of an embodiment of the present invention;

[0043] Figure 2 Schematic diagram of the structure of the diffusion model in the embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the technical problems, technical solutions and advantages to be solved by the present invention clearer, the following will be described in detail with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0046] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installation", "connection" and "connection" should be understood in a broad sense, for example, it can be a locking connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0047] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0048] In view of the existing problems, the present invention provides a method for generating zero samples of consistent roles based on a diffusion model.

[0049] like Figure 1 , 2 As shown, an embodiment of the present invention provides a method for generating zero samples of a consistent role based on a diffusion model, comprising:

[0050] Step 1, obtaining the text sequence to be processed;

[0051] Step 2, constructing a diffusion model, the diffusion model includes a hierarchical semantic parsing module for extracting semantic features, a contrastive semantic alignment module for aligning text space and semantic feature space, a progressive attention guidance module for providing attention guidance, and a semantic consistency regularization module for generating images, and training the diffusion model to obtain an image generation model;

[0052] Step 3: Input the text sequence to be processed into the image generation model for generation, and obtain an image corresponding to the text sequence to be processed.

[0053] In this embodiment of the present invention, the text sequence to be processed is obtained as “A young female scientist with shoulder-length black hair and glasses, wearing a white lab coat, standing confidently in a professional pose, realistic style”.

[0054] Most preferably, the hierarchical semantic parsing module for parsing the input text sequence includes a word embedding unit, an attribute attention unit, and a semantic pooling unit;

[0055] The word embedding unit is used to encode the text sequence to be processed and obtain the text vector;

[0056] The attribute attention unit is used to perform attention calculation on the text vector to obtain the appearance feature matrix, posture feature matrix, and style feature matrix;

[0057] The semantic pooling unit is used to perform pooling processing on the appearance feature matrix, posture feature matrix, and style feature matrix respectively to obtain the semantic feature set and input it into the contrast semantic alignment module, the progressive attention guidance module, and the semantic consistency regularization module respectively.

[0058] Specifically, the encoding expression of the word embedding unit is:

[0059]

[0060] Where E represents the text vector, WordEmded(·) represents the encoding function, which is used to map discrete text tokens into continuous vector representations, T represents the input text sequence, L represents the length of the text sequence, and d represents the dimension of the text vector. represents the real matrix space, Represents the real number space identity.

[0061] Specifically, the attribute attention unit consists of three parallel attribute-specific attention layers, namely, appearance attention layer, pose attention layer, and style attention layer;

[0062] The attention calculation for the text vector includes:

[0063] The appearance attention is calculated for the text vector, and the calculation expression is:

[0064] Happ=MultiHead(Q=WappE,K=E,V=E)

[0065] Where Happ represents the attention feature map of appearance attributes, MultiHead(·) represents the multi-head attention function, Wapp represents the query matrix of appearance attributes, which is initialized to a normal distribution N∈(0,0.02) with a dimension of dk×d=64, used for dimensionality reduction to improve computational efficiency, Q represents the query matrix, K represents the key matrix, and V represents the value matrix;

[0066] Perform posture attention calculation on the text vector, and the calculation expression is:

[0067] Hpose=MultiHead(Q=WposeE,K=E,V=E)

[0068] Among them, Hpose represents the attention feature map of the posture attribute, Wpose represents the query matrix of the posture attribute, which is initialized to a normal distribution N∈(0,0.02) with a dimension of dk×d=64, which is used for dimensionality reduction to improve computational efficiency;

[0069] The style attention is calculated for the text vector, and the calculation expression is:

[0070] Hstyle=MultiHead(Q=WstyleE,K=E,V=E)

[0071] Among them, Hstyle represents the attention feature map of the style attribute, Wstyle represents the query matrix of the style attribute, which is initialized to a normal distribution N∈(0,0.02) with a dimension of dk×d=64, which is used for dimensionality reduction to improve computational efficiency.

[0072] In this embodiment of the present invention, the specific calculation process of the multi-head attention function is:

[0073] MultiHead(Q,K,V)=Concat(head1,…,headh)WO

[0074] in,

[0075] headi=Attention(QWQ i ,KWK i ,VWV i )

[0076]

[0077] Concat(·) represents the tensor concatenation operation, h represents the number of attention heads, usually 8 or 16, which divides the attention mechanism into 8 or 16 parallel subspaces, and concatenates the outputs of h heads in the feature dimension. WO represents the output projection matrix with a size of (h×dv)×d, Attention(·) represents the attention operation, and WQ i represents the query matrix of the i-th head, dimension dk×d, i∈[1,h], WK i represents the key matrix of the i-th head, dimension dk×d, i∈[1,h], WV i represents the value matrix of the i-th head, dimension dk×d, i∈[1,h], represents a scaling factor to prevent the dot product attention score from being too large, and softmax(·) represents a normalization function to convert the attention score into a probability distribution.

[0078] Specifically, the semantic pooling unit is used to pool the attention feature maps of appearance attributes, the attention feature maps of posture attributes, and the attention feature maps of style attributes, respectively, to obtain the appearance feature matrix, the posture feature matrix, and the style feature matrix. The pooling expression is:

[0079]

[0080] Among them, Sapp represents the appearance feature matrix, Spose represents the posture feature matrix, Sstyle represents the style feature matrix, Pool(·) represents the attention weighted pooling function, σ(·) represents the activation function, FNNapp represents the feedforward neural network of appearance features, FNNpose represents the feedforward neural network of posture features, FNNstyle represents the feedforward neural network of style features, and d represents the dimension, which is set to 128;

[0081] The calculation expression of the pooling operation is:

[0082] Pool(H)=Σαi·hi

[0083] in,

[0084] αi=softmax(w T tanh(Whi+b))

[0085] In the formula, H represents the input feature matrix with a dimension of L×d, αi represents the attention weight of the i-th feature vector, and w T represents the transpose of the weight vector, hi represents the i-th eigenvector in the input feature matrix, W represents the pooling transformation matrix, w represents the attention weight vector, b represents the bias vector, and tanh(·) represents the hyperbolic tangent activation function.

[0086] Specifically, the calculation expression of the semantic feature set is:

[0087] S=(Sapp,Spose,Sstyle)

[0088] Wherein, S represents a semantic feature set, the semantic feature set includes multiple semantic feature matrices, and each semantic feature matrix includes multiple semantic feature vectors.

[0089] The embodiment of the present invention first normalizes the text sequence of the example; unifies the special characters, punctuation marks, etc. in the text, and normalizes the text length; truncates the overly long text, and fills the overly short text to ensure the consistency of the length of all input texts, and then inputs the processed text sequence into the hierarchical semantic parsing module to identify and extract three types of key information contained in the text: character appearance features (such as hairstyle, skin color, clothing, etc.), posture features (such as standing posture, expression, action, etc.) and style features (such as artistic style, scene atmosphere, etc.), and obtains:

[0090] Appearance attribute characteristics:

[0091] Gender: female

[0092] Age: young

[0093] Hairstyle: shoulder-length black hair

[0094] Outfit: glasses, white lab coat

[0095] Posture attribute features: standing confidently, professional pose

[0096] Style attribute characteristics: realistic style.

[0097] Most preferably, the contrastive semantic alignment module for aligning the text space and the semantic space through the contrastive learning framework includes a feature projection unit, an InFoNCE loss calculation unit, a mutual information maximization unit, and an alignment loss calculation unit;

[0098] The input end of the feature projection unit is connected to the output end of the word embedding unit and the output end of the semantic pooling unit respectively, the output end of the feature projection unit is connected to the input end of the InFoNCE loss calculation unit, the output end of the InFoNCE loss calculation unit is connected to the input end of the mutual information maximization unit, the output end of the mutual information maximization unit is connected to the input end of the alignment loss calculation unit, and the output end of the alignment loss calculation unit is connected to the second input end of the progressive attention guidance module.

[0099] Specifically, the feature projection unit is used to project the text vector and the semantic feature vector into the same latent space, expressed as:

[0100] z t =Proj t (t) = W t t+b t

[0101] z s =Proj s (s) = W s s+b s

[0102] Among them, z t represents the text projection vector, t represents the text vector, and the dimension is dt, z s represents the semantic feature projection vector, s represents the semantic feature vector, the dimension is ds, Proj t (·) represents the projection function of the text vector, Proj s (·) represents the projection vector of the semantic feature vector, W t Represents the text projection matrix, with a size of dz×dt, W s represents the semantic projection matrix, with a size of dz×ds, b t, b s Both represent projection bias vectors, with dimension dz and are initialized to 0.

[0103] Specifically, the InFoNCE loss calculation unit is used to calculate the contrast loss value between the text projection vector and the semantic feature projection vector. The calculation expression is:

[0104]

[0105] Where Lcsα represents the contrast loss value, E represents the expectation operator, exp(·) represents the natural exponential function, and z s represents the semantic feature projection vector, s represents the semantic feature vector, represents the text projection vector that matches the semantic feature projection vector, represents the text projection vector that does not match the semantic feature projection vector, t represents the text vector, τ represents the temperature parameter used to control the smoothness of the distribution, sim(·) represents the cosine similarity function, and ||·|| represents the L2 norm of the vector.

[0106] Specifically, the mutual information maximization unit is used to calculate the mutual information between the semantic feature vector and the text vector. The calculation expression is:

[0107] Lmi=-I(z s ;z t )≈-E[log(p θ (z t |z s ))]+E[log(p(z t ))]

[0108] p θ (z t |z s )=N(μθ(z s ),σθ 2 (z s ))

[0109] Among them, I(z s ;z t ) represents the semantic feature projection vector z s With the text projection vector z t The mutual information between them, μθ(z s ) represents the mean network, z s Mapped to the mean of the conditional distribution, σθ 2 (z s ) represents the variance network, z s Mapped to the variance of the conditional distribution, p(z t) represents the marginal distribution of the text vector, θ represents the neural network parameters, including the weights of the mean network and the variance network, and p θ (z t |z s ) represents the conditional probability distribution, which is modeled as a Gaussian distribution.

[0110] Specifically, the alignment loss calculation unit is used to calculate the alignment loss value, and the text projection vector is aligned with the semantic feature projection vector through the alignment loss value to generate the initial image. The calculation expression is:

[0111] L a lign=Lcsa+λmiLmi

[0112] Among them, L a lign represents the alignment loss value, which is used to measure the alignment effect between text and semantics, λmi represents the mutual information loss weight, Lmi represents the mutual information maximization loss value, E represents the expectation operator, exp(·) represents the natural exponential function, z s represents the semantic feature projection vector, s represents the semantic feature vector, represents the text projection vector that matches the semantic feature projection vector, represents the text projection vector that does not match the semantic feature projection vector, t represents the text vector, τ represents the temperature parameter, and sim(·) represents the cosine similarity function.

[0113] Most preferably, the progressive attention guidance module includes a multi-attribute attention unit, a time-varying weight calculation unit, an attention fusion unit, and a diffusion update unit;

[0114] The input end of the multi-attribute attention unit is connected to the output end of the alignment loss calculation unit and the output end of the semantic pooling unit respectively, the input end of the multi-attribute attention unit is connected to the input end of the time-varying weight calculation unit, the output end of the time-varying weight calculation unit is connected to the input end of the attention fusion unit, the output end of the attention fusion unit is connected to the input end of the diffusion update unit, and the output end of the diffusion update unit is connected to the second input end of the semantic consistency regularization module.

[0115] Specifically, the multi-attribute attention unit is used to perform multi-attribute attention calculation on the initial image, and the calculation expression is:

[0116]

[0117] Among them, Attapp represents the appearance attention feature map, Attpose represents the pose attention feature map, Attstyle represents the style attention feature map, x represents the initial image, Sapp represents the appearance feature matrix, Spose represents the pose feature matrix, Sstyle represents the style feature matrix, Q represents the query matrix, K represents the key matrix, V represents the value matrix, d represents the attention feature dimension, and T represents the matrix transpose.

[0118] Specifically, the time-varying weight calculation unit is used to calculate the time-varying weight coefficient of each attention feature map, and the calculation expression is:

[0119] λapp(t)=sigmoid(w app ·t+b app )

[0120] λpose(t)=sigmoid(w pose ·t+b pose )

[0121] λstyle(t)=sigmoid(w style ·t+b style )

[0122] Where t represents the diffusion time step, ranging from [0, T], λapp(t) represents the time-varying weight coefficient of the appearance attention feature map, which varies with the diffusion time step t, λpose(t) represents the time-varying weight coefficient of the pose attention feature map, which varies with the diffusion time step t, λstyle(t) represents the time-varying weight coefficient of the style attention feature map, which varies with the diffusion time step t, w represents the time weight coefficient, b represents the time bias term, and sigmoid(·) represents the activation function, which compresses the output to the (0, 1) interval.

[0123] Specifically, the attention fusion unit is used to fuse the attribute attention feature map of the initial image with the time-varying weight coefficient of each attribute attention to obtain the fused attention. The calculation expression is:

[0124] PAG(x,S)=λapp(t)·Attapp+λpose(t)·Attpose+λstyle(t)·Attstyle

[0125] Among them, PAG represents the fused attention feature map and t represents the diffusion time step.

[0126] Specifically, the diffusion update unit is used to diffusely update the initial image through the diffusion update rule, and the update expression is:

[0127] xt-1=μθ(xt,t)+σθ(t)·∈+γ(t)·PAG(xt,S)

[0128] γ(t)=α·(1-cos(πt / T))

[0129] Where xt represents the image at time t, μθ(·) represents the mean prediction of the denoising network, σθ(·) represents the noise amplitude coefficient, ranging from (0,1), ∈ represents random noise, which obeys the standard normal distribution, γ(t) represents the attention guidance strength, using cosine scheduling, α represents the guidance strength coefficient, T represents the total number of diffusion steps, and cos(·) represents the cosine function.

[0130] In this embodiment of the present invention, the initial image is a 512×512 random noise image, the total diffusion steps are 1000 steps, and the attention guidance process includes: t>800: emphasizing the overall layout, 800>t>400: refining the appearance features, t<400: optimizing local details.

[0131] It should be noted that in the embodiment of the present invention, the attribute attention unit and the multi-attribute attention unit are both configured with parameters through the attention controller.

[0132] Specifically, the semantic consistency regularization module is used to:

[0133] Construct multiple constraints for the denoised image based on the semantic feature set;

[0134] The denoised image is regenerated into an image corresponding to the text sequence to be processed based on multiple constraints.

[0135] Most preferably, the multiple constraints include feature reconstruction constraints, KL divergence constraints, cycle consistency constraints, and regularization constraints.

[0136] The embodiment of the present invention performs a text map based on the text sequence used for example, and obtains a 512×512 color image. The image features are consistent with the image of a young female scientist, maintaining a professional posture, and showing a realistic style.

[0137] Specifically, the expression of the feature reconstruction constraint that ensures the semantic consistency of the generated image is:

[0138] Lrec=||Enc(x)-S||2

[0139] Among them, Enc(·) represents the semantic encoder, x represents the generated image, and S represents the semantic feature set.

[0140] Specifically, the expression for the KL divergence constraint used to align the posterior and prior distributions is:

[0141]

[0142] p(z|S)=N(μψ(S),σψ2 (S)

[0143] Among them, KL(·||·) represents the KL divergence, which is used to measure the difference between two distributions. represents the posterior distribution, which is parameterized by the encoder, and p(z|S) represents the prior distribution, which is determined by the semantic features. represents the network parameters of the posterior distribution, ψ represents the network parameters of the prior distribution, Represents the mean and variance network of the posterior distribution, μψ, σψ 2 represents the mean and variance network of the prior distribution, and Lkl represents the final KL divergence value.

[0144] Specifically, the expression of the cycle consistency constraint used to ensure the consistency of generation and reconstruction is:

[0145]

[0146] Where G(·) represents the generator network, represents the feature extractor, the intermediate layer features of VGG-16, ||·||1 represents the L1 norm, λfeat represents the weight coefficient of feature reconstruction loss, and G(Enc(x)) represents the regenerated image.

[0147] Specifically, the expression of the regularization constraint is:

[0148] Lscr=Lrec+λklLkl+λcycLcyc

[0149] Among them, λkl represents the KL divergence weight, which adjusts the distribution alignment strength, ranging from 0.01-0.1, and λcyc represents the cycle consistency weight, which controls the reconstruction strength, ranging from 0.5-2.0.

[0150] In the embodiment of the present invention, the diffusion model adjusts the parameters of the model by calculating the combined loss value through the loss function during the training process, thereby obtaining the image generation model, wherein the expression of the loss function is:

[0151] L=Ldiff+λ1L align +λ2Lscr

[0152] Among them, Ldiff represents the difference loss, L align represents the alignment loss and Lscr represents the regularization loss.

[0153] The embodiment of the present invention obtains a text sequence to be processed; constructs a diffusion model including a hierarchical semantic parsing module, a contrastive semantic alignment module, a progressive attention guidance module, and a semantic consistency regularization module, and trains the diffusion model to obtain an image generation model; inputs the text sequence to be processed into the image generation model for generation, and obtains an image corresponding to the text sequence to be processed; compared with the prior art, the embodiment of the present invention implements text-to-semantic feature parsing in the absence of a reference image only by inputting the text sequence into a hierarchical semantic parsing module, thereby significantly improving the model's ability to understand fine-grained attributes in text descriptions; aligns the text vector with the semantic feature vector through the contrastive semantic alignment module to generate an initial image; uses the progressive attention guidance module to provide hierarchical attention guidance, which can adaptively focus on different attributes during the denoising process of the initial image, thereby ensuring the quality and consistency of the generated results; and uses the semantic consistency regularization module to prevent semantic drift during the generation process, thereby generating an image corresponding to the text sequence.

[0154] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for generating zero samples of consistent roles based on a diffusion model, characterized in that: include: Step 1, obtaining the text sequence to be processed; Step 2, constructing a diffusion model, the diffusion model includes a hierarchical semantic parsing module for extracting semantic features, a contrast semantic alignment module for aligning text space and semantic feature space, a progressive attention guidance module for providing attention guidance, and a semantic consistency regularization module for generating images, and training the diffusion model to obtain an image generation model; Step 3: input the text sequence to be processed into the image generation model for generation, and obtain an image corresponding to the text sequence to be processed.

2. The method for generating consistent role zero samples based on a diffusion model according to claim 1, characterized in that: The hierarchical semantic parsing module includes a word embedding unit, an attribute attention unit, and a semantic pooling unit; The word embedding unit is used to encode the text sequence to be processed to obtain a text vector; The attribute attention unit is used to perform attention calculation on the text vector to obtain an appearance feature matrix, a posture feature matrix, and a style feature matrix; The semantic pooling unit is used to perform pooling processing on the appearance feature matrix, the posture feature matrix, and the style feature matrix respectively to obtain a semantic feature set and input it into the contrast semantic alignment module, the progressive attention guidance module, and the semantic consistency regularization module respectively.

3. The method for generating consistent role zero samples based on a diffusion model according to claim 2, characterized in that: The calculation expression of the semantic feature set is: S=(Sapp,Spose,Sstyle) Where S represents the semantic feature set, Sapp represents the appearance feature matrix, Spose represents the posture feature matrix, Sstyle represents the style feature matrix, Pool(·) represents the attention weighted pooling function, σ(·) represents the activation function, FNNapp represents the feedforward neural network of appearance features, FNNpose represents the feedforward neural network of posture features, FNNstyle represents the feedforward neural network of style features, Happ represents the attention feature map of appearance attributes, Hpose represents the attention feature map of posture attributes, Hstyle represents the attention feature map of style attributes, d represents the dimension, Represents the real number space identity.

4. The method for generating zero samples of consistent roles based on a diffusion model according to claim 3, characterized in that: The contrastive semantic alignment module includes a feature projection unit, an InFoNCE loss calculation unit, a mutual information maximization unit, and an alignment loss calculation unit; The input end of the feature projection unit is connected to the output end of the word embedding unit and the output end of the semantic pooling unit respectively, the output end of the feature projection unit is connected to the input end of the InFoNCE loss calculation unit, the output end of the InFoNCE loss calculation unit is connected to the input end of the mutual information maximization unit, the output end of the mutual information maximization unit is connected to the input end of the alignment loss calculation unit, and the output end of the alignment loss calculation unit is connected to the second input end of the progressive attention guidance module.

5. The method for generating consistent role zero samples based on a diffusion model according to claim 4, characterized in that: The alignment loss calculation unit is used to calculate the alignment loss value, and the text projection vector is aligned with the semantic feature projection vector by the alignment loss value to generate an initial image. The calculation expression is: L a lign=Lcsa+λmiLmi Lmi=-I(z s with t ) Among them, L a lign represents the alignment loss value, Lcsa represents the contrast loss value, λmi represents the mutual information loss weight, Lmi represents the mutual information maximization loss value, E represents the expectation operator, exp(·) represents the natural exponential function, z s represents the semantic feature projection vector, s represents the semantic feature vector, represents the text projection vector that matches the semantic feature projection vector, represents the text projection vector that does not match the semantic feature projection vector, t represents the text vector, τ represents the temperature parameter, sim(·) represents the cosine similarity function, I(z s ; z t ) represents the semantic feature projection vector z s With the text projection vector z t The mutual information between them.

6. The method for generating consistent role zero samples based on a diffusion model according to claim 5, characterized in that: The progressive attention guidance module includes a multi-attribute attention unit, a time-varying weight calculation unit, an attention fusion unit, and a diffusion update unit; The input end of the multi-attribute attention unit is connected to the output end of the alignment loss calculation unit and the output end of the semantic pooling unit respectively, the input end of the multi-attribute attention unit is connected to the input end of the time-varying weight calculation unit, the output end of the time-varying weight calculation unit is connected to the input end of the attention fusion unit, the output end of the attention fusion unit is connected to the input end of the diffusion update unit, and the output end of the diffusion update unit is connected to the second input end of the semantic consistency regularization module.

7. The method for generating zero samples of consistent roles based on a diffusion model according to claim 6, characterized in that: The multi-attribute attention unit is used to perform multi-attribute attention calculation on the initial image, and the calculation expression is: Among them, Attapp represents the appearance attention feature map, Attpose represents the pose attention feature map, Attstyle represents the style attention feature map, x represents the initial image, Sapp represents the appearance feature matrix, Spose represents the pose feature matrix, Sstyle represents the style feature matrix, Q represents the query matrix, K represents the key matrix, V represents the value matrix, d represents the attention feature dimension, and T represents the matrix transpose.

8. According to the method for generating consistent role zero samples based on diffusion model in claim 7, the attention fusion unit is used to fuse the attribute attention feature map of the initial image with the time-varying weight coefficient of each attribute attention to obtain the fused attention, and the calculation expression is: PAG(x,S)=λapp(t)·Attapp+λpose(t)·Attpose+λstyle(t)·Attstyle in, PAG represents the fused attention feature map, λapp(t) represents the time-varying weight coefficient of the appearance attention feature map, λpose(t) represents the time-varying weight coefficient of the pose attention feature map, λstyle(t) represents the time-varying weight coefficient of the style attention feature map, and t represents the diffusion time step.

9. The method for generating consistent role zero samples based on a diffusion model according to claim 6, characterized in that: The semantic consistency regularization module is used to: constructing multiple constraints for the denoised image based on the semantic feature set; The denoised image is regenerated into an image corresponding to the text sequence to be processed based on multiple constraints.

10. The method for generating zero samples of consistent roles based on a diffusion model according to claim 9, characterized in that: The multiple constraints include feature reconstruction constraints, KL divergence constraints, cycle consistency constraints, and regularization constraints.

Images