Image generation method, electronic device, and computer-readable storage medium

Abstract

An image generation method is disclosed. The method comprises: constructing a plurality of noisy image tokens, text tokens corresponding to a text prompt, and reference tokens of a plurality of reference subjects into a unified multimodal attention sequence, and determining noise-to-text attention and attention quality based on the sequence; for each reference subject, averaging the text tokens in the text prompt that describe the reference subject in the noise-to-text attention to obtain a text prior, and averaging the text tokens that do not describe any reference subject to obtain a background prior; based on the query features of the noisy image tokens, the background prior, and the text priors of each reference subject, a neural network is used to obtain an assignment weight; according to the assignment weight, the attention quality of the noisy image tokens on each reference subject is biased, so as to redistribute the attention quality among the plurality of reference subjects; based on the redistributed multimodal attention, the noisy image tokens are iteratively denoised to generate an image containing the plurality of reference subjects.

Description

Technical Field

[0001] This disclosure relates to the field of image generation technology, and in particular to an image generation method, electronic device, and computer-readable storage medium. Background Technology

[0002] Personalized image generation aims to synthesize images containing user-specified subjects based on text prompts. With evolving application needs, multi-subject personalized generation (generating multiple user-specified reference subjects within the same image while maintaining text controllability and consistency in reference appearance) is receiving increasing attention.

[0003] Diffusion models, especially diffusion transformers (DiT), provide a modeling approach for multi-condition fusion. In diffusion transformers based on multimodal attention, text, noisy images, and reference images can be uniformly organized into an attention space and interact with each other, making it easy to introduce multiple reference images as additional conditions into the model.

[0004] However, as the number of reference subjects increases, the performance of personalized generation methods in related technologies typically declines significantly, manifested as weakened text alignment capabilities and reduced consistency in reference appearance. Analysis of the attention patterns in multimodal attention reveals that this degradation primarily stems from two aspects: first, text-reference mismatch, where the text of a subject and its corresponding reference conditions focus on inconsistent image regions; second, subject confusion, where the generated region of a subject may focus on multiple visually or semantically similar reference subjects, resulting in inconsistent appearances, and in severe cases, even subject fusion or the loss of the target subject.

[0005] In related technologies, to alleviate the above problems, some methods introduce external layout information (such as bounding boxes and segmentation maps) or rely on masks obtained from external segmentation to spatially constrain the generated regions of each subject; other methods fit the attention map to the mask through regression loss during the training phase to constrain the attention. However, the former relies on additional external masks or segmentation results during the inference phase, increasing the input burden and potentially limiting the flexibility of generation; the latter mainly regularizes the existing attention during the training phase, lacking an explicit, learnable redistribution mechanism for the attention of each reference branch; and when adjusting the attention to the reference, the above methods often change the overall attention quality of the reference branch relative to the text branch and the image branch, which poses a risk of compromising the stability of the image-text alignment of the pre-trained model.

[0006] Therefore, how to alleviate subject confusion and text-reference mismatch in multi-subject personalized generation without relying on external masks, while maintaining the image-text alignment capability of the pre-trained model, is an urgent technical problem to be solved. Summary of the Invention

[0007] This disclosure provides an image generation method, an electronic device, and a storage medium. The method utilizes noise-to-text attention to construct a text prior corresponding to a reference subject and a background prior corresponding to the background. A neural network is then used to obtain allocation weights based on the query features of noisy image words and the aforementioned priors. A bias is then applied to the attention quality based on these allocation weights, thereby redistributing the attention quality of noisy image words among multiple reference subjects.

[0008] This disclosure provides an image generation method, comprising: acquiring multiple reference subjects and text prompts, each reference subject corresponding to a reference image, and at least a portion of the descriptive text in the text prompts being descriptive text used to describe the reference images; constructing a unified multimodal attention sequence by multiple noisy image words, text words corresponding to the text prompts, and reference words corresponding to the reference images of each reference subject; wherein, the multiple noisy image words are multiple image words obtained by dividing the latent representation of the image to be generated in the latent space, and the multiple noisy image words correspond to different spatial positions of the image to be generated; determining noise-to-text attention and attention quality based on the unified multimodal attention sequence; wherein, noise-to-text attention represents the attention weight of multiple noisy image words to text words, and attention quality is the sum of the attention weights of one noisy image word to all reference words of a reference subject; for each reference subject, in the noise-to-text attention, each of the multiple noisy image words is assigned a text word weight. The attention weights of each noisy image word in the text prompt corresponding to the text word describing the reference subject are averaged to obtain the text prior of the reference subject. In the noise-to-text attention process, the attention weights of each noisy image word in the text prompt corresponding to the text word not describing any reference subject are averaged to obtain the background prior. The background prior corresponds to the background in the image to be generated that does not belong to any reference subject. Based on the query features of each noisy image word, the background prior, and the text prior of each reference subject, a neural network is used to obtain the assigned weights of multiple noisy image words. These assigned weights include the weight for each noisy image word to match the background and the weight for matching each reference subject. According to the assigned weights, a bias is applied to the attention quality, thereby redistributing the attention quality of multiple noisy image words among multiple reference subjects, resulting in a redistributed multimodal attention. Based on the redistributed multimodal attention, multiple noisy image words are iteratively denoised to generate an image containing multiple reference subjects.

[0009] According to one or more embodiments of this disclosure, a neural network is used to obtain the weights for multiple noisy image terms, including: concatenating the query features of each noisy image term with the prior p along the channel dimension to obtain the concatenated features z=[h||p], where h is the query feature of each noisy image term, and p is the prior obtained by stacking the background prior and the text prior of each reference subject; rearranging the concatenated features z into a two-dimensional latent space grid, inputting it into a convolutional network, and outputting the residual Δ corresponding to each noisy image term; based on the prior p and the residual Δ, according to the formula... The assigned weight α is obtained.

[0010] According to one or more embodiments of this disclosure, the residual Δ is expressed as follows: Determined; where z is the concatenated feature, t is the current denoising time step, and l is the layer index of the current attention layer. Let be a convolutional network, and Δ be the residual.

[0011] According to one or more embodiments of this disclosure, biasing attention quality based on assigned weights includes: biasing the attention score of a reference word pointing from the i-th noisy image word to the c-th reference subject. ,in, The attention score is the subject-specific bias corresponding to the i-th noisy image term and the c-th reference subject; the attention score is the attention score before softmax normalization in the unified multimodal attention sequence, where i is the index of the noisy image term and c is the index of the reference subject. The weight for matching the i-th noisy image term with the c-th reference subject, where C is the number of reference subjects.

[0012] According to one or more embodiments of this disclosure, applying a bias to attention quality based on assigned weights further includes: for each noisy image word, after applying the bias, further applying a shared offset shared among the multiple reference subjects to the attention score of the reference word pointing to the multiple reference subjects of the noisy image word, such that after applying the shared offset, the total attention quality of the noisy image word to the multiple reference subjects is equal to the total attention quality before applying the bias.

[0013] According to one or more embodiments of this disclosure, the shared offset is calculated as follows: Determined; among them, is the shared offset corresponding to the i-th noisy image term; logit(·) is the log-odds function, and i is the index of the noisy image term; The total attention quality of the i-th noisy image term to multiple reference subjects before applying bias; Let be the total attention quality of the i-th noisy image term for multiple reference subjects when a bias has been applied but a shared offset has not yet been applied.

[0014] According to one or more embodiments of this disclosure, the neural network is pre-trained in the following manner: based on a true foreground mask, each noisy image term is divided into foreground terms belonging to a reference subject, background terms not belonging to any reference subject, and the remaining ambiguous terms, wherein the following conditions are met: The noisy image words are foreground words, satisfying The noisy image words are considered background words, and the remaining noisy image words are considered blurred words; the first... The target of each noisy image word was determined as follows: Foreground words As its main reference subject, for background lexical units Background; Based on the assigned weights output by the neural network and the target, the cross-entropy loss is calculated using the following formula. Negative losses Interval loss Total loss :

[0015]

[0016]

[0017] Based on the total loss Adjust the parameters of the neural network; in, For indexing noisy image terms, Index for reference subjects; For the first The true foreground mask of the reference subject in the th... The value at each noisy image word. , These are the first threshold and the second threshold, respectively; The set of foreground lexical units and the set of background lexical units. The set of foreground words; For the first The target of a noisy image terminology, For the first Noisy image word matching its target The weight, For the first The noisy image word matching The weight of each reference subject, For intervals, , , These are the weights of the cross-entropy loss, the negative loss, and the interval loss, respectively.

[0018] This disclosure also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the image generation method of any of the above embodiments.

[0019] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image generation method of any of the above embodiments.

[0020] This disclosure has at least the following beneficial effects: by constructing text priors and background priors from internal model noise to text attention, the dependence on external masks or external segmentation results can be reduced during the inference stage; by obtaining assigned weights based on the query features of noisy image words and the aforementioned priors through a neural network, the degree of matching with the background or different reference subjects can be dynamically determined for different spatial locations; by applying a bias to the attention quality according to the assigned weights, the attention of noisy image words to mismatched reference subjects can be suppressed, while the attention to matching reference subjects can be enhanced; by maintaining the total attention quality of reference subjects through shared offsets, the impact on text alignment stability can be reduced while redistributing attention quality among reference subjects. Attached Figure Description

[0021] Figure 1 A flowchart of an image generation method provided in this disclosure embodiment; Figure 2 This is a schematic flowchart of a multimodal attention routing method provided in an embodiment of the present disclosure; Figure 3 A schematic diagram of the specific structure of a neural network; and Figure 4 This is a schematic diagram of the electronic device structure according to an embodiment of the present disclosure. Detailed Implementation

[0022] The technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only for illustrating this disclosure and are not intended to limit the scope of protection of this disclosure. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0023] In this disclosure, a lexical unit can be the Chinese representation of a token. A noisy image lexical unit can be an image lexical unit obtained by partitioning the latent representation of the image to be generated in the latent space. A reference subject can be a subject whose appearance or identity features the user wishes to retain in the generated image; each reference subject corresponds to a reference image. Reference lexical units can be lexical units encoded from the reference image and used in multimodal attention calculations. Text lexical units can be lexical units encoded from text prompts. The background can be a portion of the image to be generated that does not belong to any reference subject.

[0024] Multimodal Attention Routing (MAR) in this embodiment refers to the redistribution of attention quality from noisy image words to different reference subjects within a unified multimodal attention framework that includes noisy image words, text words, and reference words from multiple reference subjects. This redistribution can be achieved by applying a bias to the attention score before softmax normalization, which is a soft bias method rather than hard masking of a particular reference subject.

[0025] like Figure 1 As shown, in step S100, multiple reference subjects and text prompts are obtained. Each reference subject corresponds to a reference image. At least a portion of the descriptive text in the text prompt is used to describe the reference image. For example, the text prompt may include text fragments describing the first reference subject, the second reference subject, and the background environment, respectively. The text fragments used to describe the reference subjects may correspond to corresponding intervals in the text lexical sequence, for example, the descriptive text of the c-th reference subject corresponds to the text lexical interval []. , Text terms that do not fall within the descriptive text range of any reference subject in the text prompt can be used to form context-related semantic information.

[0026] In some embodiments, the text prompt may further include subject placeholders corresponding to each reference subject, for example... <p1> 、 <p2>The text placeholders are used to distinguish text fragments describing different reference subjects. After being encoded by a text encoder, the subject placeholders form corresponding text lexical units, which can be included within the text lexical unit interval corresponding to the descriptive text of the corresponding reference subject, thus participating in the determination of the text prior for that reference subject. The lexical representation of the subject placeholders can be a learnable representation, and its specific settings can depend on the personalized image generation model used. The method disclosed herein does not require the use of subject placeholders as a necessary condition; without using subject placeholders, the corresponding text prior can be directly determined based on the text lexical unit interval corresponding to the text fragments describing each reference subject.

[0027] In some embodiments, text prompts can be encoded into text terms T by a text encoder; reference images of each reference subject can be encoded into reference terms by an image encoder, a reference image encoder, or a conditional encoding module in a generative model. Where c is the reference subject index and C is the number of reference subjects. For the current denoising time step t, the latent representation of the image to be generated in the latent space can be... The image is divided into N noisy image terms X, and the N noisy image terms correspond to different spatial positions of the image to be generated.

[0028] In step S200, multiple noisy image terms, text terms corresponding to text prompts, and reference terms corresponding to the reference images of each reference subject are constructed into a unified multimodal attention sequence. In one embodiment, the unified multimodal attention sequence S can be represented as:

[0029] Where X represents noisy image terms, and T represents text terms. Let || represent the reference lexical of the c-th reference subject, and || denote concatenation along the lexical dimension. This unified multimodal attention sequence can be input into the multimodal attention layer of a diffusion transformer or a stream-matching diffusion transformer. It should be noted that the diffusion transformer and unified sequence concatenation can serve as the generative model environment upon which the method of this disclosure relies. The main improvement of this disclosure lies in how to redistribute the attention quality among reference subjects based on noise-to-text attention, text priors, and neural network outputs.

[0030] In the multimodal attention layer, a query Q, a key K, and a value V can be generated based on a unified multimodal attention sequence S, and attention calculation can be performed.

[0031] Where d is the channel dimension of the query or key. (Previous softmax normalization) This can be called the attention score; the result after softmax normalization is the attention weight.

[0032] Specifically, for the i-th noisy image term and the m-th conditional term, the attention score before normalization can be expressed as:

[0033] in, The attention score between the query vector of the i-th noisy image term and the key vector of the m-th condition term. Let i be the query vector for the i-th noisy image term. Let i be the key vector of the m-th conditional term, where i is the index of the noisy image term and m is the index of the conditional term.

[0034] In step S300, noise-to-text attention and attention quality are determined based on the unified multimodal attention sequence. Noise-to-text attention represents the attention weights of multiple noisy image terms to text terms. In some embodiments, noise-to-text attention can be obtained by extracting a submatrix from the multimodal attention matrix when noisy image terms are used as queries and text terms are used as keys. Noise-to-text attention can also be called a cross-attention map or... When a multimodal attention layer includes multiple attention heads, the noise-to-text attention from the multiple attention heads can be averaged to obtain the noise-to-text attention used for subsequent prior construction.

[0035] Attention quality can be used to characterize the total amount of attention weights assigned to a reference subject for a noisy image term. For the i-th noisy image term and the c-th reference subject, its attention quality... It can be represented as:

[0036] in, Let c be the set of keys corresponding to the reference lexical units of the c-th reference subject. Let be the attention weight of the i-th noisy image word to the m-th word. Therefore, the attention quality of a noisy image word to a certain reference subject is not the weight of a single reference word, but the sum of the attention weights of the noisy image word to all reference words of that reference subject.

[0037] In step S400, for each reference subject, in the noise-to-text attention process, the attention weights of each noisy image word from multiple noisy image words to the text words corresponding to the descriptive text describing the reference subject in the text prompt are averaged to obtain the text prior of the reference subject. For the c-th reference subject, its descriptive text corresponds to the text word interval [ , If the reference subject's textual prior can be represented as:

[0038] in, This represents the noise-to-text attention weights for all noisy image terms to the k-th text term. The above text prior... It can be viewed as a spatial prior map related to the c-th reference subject, where each spatial location corresponds to a noisy image term representing the average attention level of the descriptive text of that reference subject.

[0039] In step S400, during the noise-to-text attention process, the attention weights of each noisy image word to text words in the text prompt that do not describe any reference subject are averaged to obtain the background prior. The background prior corresponds to the background in the image to be generated that does not belong to any reference subject. In some embodiments, the background prior may also be referred to as the base text prior. The background prior is not a reference subject, nor does it correspond to a reference image; its role is to accommodate the spatial location of the background region or that does not belong to any reference subject, so that subsequent weight assignment can include the background branch and multiple reference subject branches.

[0040] In this way, both text prior and background prior originate from the noise-to-text attention within the generative model, rather than from the foreground mask or segmentation mask of the external input during the inference stage. Therefore, during the inference stage, the text-space correspondence already formed within the model can be used to guide the attention allocation among reference subjects.

[0041] In step S500, based on the query features of each noisy image term, the background prior, and the text prior of each reference subject, a neural network is used to obtain the assigned weights for multiple noisy image terms. This neural network can also be called a router or a joint spatial router. In some embodiments, the neural network can be a lightweight convolutional network or a convolutional tower, used to predict the weights for each noisy image term to match the background and to match each reference subject based on spatial location, text prior, and background prior.

[0042] like Figure 1 and Figure 2 As shown, in some embodiments, the query features of each noisy image term are concatenated with the prior p along the channel dimension to obtain the concatenated features (step S510):

[0043] Where h represents the query feature for each noisy image term. This query feature can be the query feature obtained by projecting the noisy image term onto the query in the current multimodal attention layer. p represents the prior obtained by stacking the background prior and the text prior for each reference subject, for example... In some embodiments, p includes C reference subject branches and one background branch, so the weights assigned to the subsequent output of the neural network may include reference subject components corresponding to C reference subjects and one background component.

[0044] In step S520, the concatenated feature z is rearranged into a two-dimensional latent space grid and then input into the convolutional network. Since multiple noisy image terms correspond to different spatial locations in the image to be generated, the concatenated feature z can be rearranged into a two-dimensional latent space grid feature of H×W according to the spatial arrangement of the noisy image terms, where H×W=N. In this way, the convolutional network can use local spatial neighborhood information to predict the weight allocation for different spatial locations.

[0045] In some embodiments, the convolutional network is modulated by the current denoising time step t and the layer index l of the current attention layer, such that the residual Δ is determined by the following formula:

[0046] in, Let be a convolutional network, and θ be the network parameters. Modulation refers to how the convolutional network adjusts its processing of the concatenated feature z based on the current denoising time step t and the layer index l of the current attention layer. As an example, the denoising time step t and the layer index l can be encoded as embedding vectors, and then applied to the intermediate features of the convolutional network through feature addition, channel scaling, FiLM modulation, or adaptive normalization. The above modulation methods are merely implementation examples; any method that enables the convolutional network to output the corresponding residual based on the current denoising stage and the attention layer position is acceptable.

[0047] In step S530, the convolutional network outputs a residual Δ corresponding to each noisy image term. This residual is used to correct the logarithmic reference quantity formed by the text prior and the background prior. In some embodiments, the branch dimension of Δ corresponds to the branch dimension of the prior p, that is, it includes C reference subject branches and one background branch.

[0048] In step S540, based on the prior p and the residual Δ, the allocation weight α is obtained according to the following formula:

[0049] Softmax normalizes each noisy image term separately along the branch dimension. The assigned weights α, also known as the routing distribution, include the weights for matching each noisy image term to the background and the weights for matching each reference subject.

[0050] Here, weights are assigned. It is the output of a neural network (router), normalized across C+1 branches consisting of the background branch and C reference subject branches; it differs from the attention score in a multimodal attention layer. Attention weight And the attention quality obtained by summing attention weights .

[0051] In the above embodiments, the background branch is used to characterize image regions that do not belong to any reference subject, and can be used for background lexical classification during the training phase and suppression of incorrect reference assignments during the inference phase. Since the background branch does not correspond to a specific reference image, it does not necessarily correspond to a set of reference lexical keys; therefore, when generating attention score biases based on the assigned weights, biases for reference lexical units can be generated based on the reference subject components corresponding to C reference subjects, while the background components are used as background absorption terms or training supervision branches.

[0052] In step S600, a bias is applied to the attention quality according to the assigned weights, thereby redistributing the attention quality of multiple noisy image terms among multiple reference subjects to obtain the redistributed multimodal attention. This bias can be applied to the attention score before softmax normalization in the unified multimodal attention sequence. By changing the attention score, the attention weights and attention quality after softmax are thus changed.

[0053] In step S610, a bias is applied to the attention score of the reference word pointing from the i-th noisy image word to the c-th reference subject:

[0054] in, Let i be the index of the noisy image term and c be the index of the reference subject, and c be the index of the noisy image term. The weight is assigned to match the i-th noisy image term with the c-th reference subject, where C is the number of reference subjects. This bias can be applied to the attention score of the i-th noisy image term to all reference terms of the c-th reference subject. If it is greater than 1 / C, then A positive value can enhance the attention of the i-th noisy image term to the c-th reference subject; if... If less than 1 / C, then A negative value can suppress the attention of the i-th noisy image term to the c-th reference subject.

[0055] This method applies a soft bias to the attention score before normalization. Unlike hard masking or hard gating, soft bias does not directly delete the attention connection of a specific reference subject. Instead, it adjusts the relative attention distribution between different reference subjects within the original structure of unified multimodal attention. Therefore, the model can still retain the multimodal fusion capabilities already present in the pre-trained model.

[0056] After applying the aforementioned bias, in step S620, for each noisy image lexical, a shared offset can be further applied to the attention scores of the reference lexicals pointing to multiple reference subjects, shared among the multiple reference subjects. This is done so that after applying the shared offset, the total attention quality of the noisy image lexical to the multiple reference subjects is equal to the total attention quality before applying the bias. This shared offset is applied jointly to multiple reference subject branches of the same noisy image lexical, thus not altering the relationship between the multiple reference subjects. Instead of a fixed relative proportion, it adjusts the overall attention quality of the reference subject branch as a whole relative to text lexical units and other lexical units.

[0057] In step S630, the shared offset is determined by the following formula:

[0058] in, is the shared offset corresponding to the i-th noisy image term; logit(·) is the log-odds function. The total attention quality of the i-th noisy image term to multiple reference subjects before applying bias; Let be the overall attention quality of the i-th noisy image term towards multiple reference subjects, with bias applied but before shared offset applied. The overall attention quality of the i-th noisy image term towards multiple reference subjects can be expressed as:

[0059] A shared offset is applied to reference lexical units of multiple reference subjects. Then, the softmax weights of the reference main branch are multiplied by the total weights. The corresponding factor allows the overall attention quality after applying the shared offset to return to the level before the bias was applied. In other words, The main decision is "which reference subject should be given more attention for the i-th noisy image term". It is mainly used to maintain a balance in "how much attention the i-th noisy image lexical generally pays attention to the reference subject". As a result, attention quality can be redistributed among multiple reference subjects, while reducing perturbations to the attention balance between text branches and other branches.

[0060] In this embodiment, the overall attention quality of the reference branch is recovered by a word-by-word shared offset applied to the pre-normalized attention scores corresponding to all reference lexical units. Let

[0061] in, To apply attention weights before subject-specific biases are applied. Attention weights are applied after subject-specific biases have been applied but before shared offsets have been applied. This is the set of key indices corresponding to the reference terms of all reference subjects.

[0062]

[0063] The actual bias added to each reference attention score is:

[0064] in, This is the bias applied to the attention score between the i-th noisy image term and all reference terms of the c-th reference subject.

[0065] make This represents the attention score before the bias is applied. After the actual bias is applied, the attention score is updated according to the following formula:

[0066] in, This is the set of key indices corresponding to the reference lexical of the c-th reference subject; This is the set of key indices corresponding to the reference words of all reference subjects; The remaining set of key indexes that do not receive the bias. Each set satisfies:

[0067] Based on the updated attention score, the final attention weights are obtained after softmax normalization:

[0068] in, This applies the actual bias and completes the softmax normalized attention weights. The shared bias ensures that the total attention quality across all reference subjects satisfies:

[0069] In this case, the overall balance of pre-trained multimodal attention can be maintained.

[0070] In practical calculations, to avoid the logit function's value becoming unstable at 0 or 1, it can be adjusted... and Numerical truncation is performed, for example, limiting it to the range [ε, 1-ε], where ε is a decimal greater than 0. This numerical processing does not change the technical meaning of the shared offset used to maintain the overall attention quality.

[0071] In step S700, based on the reallocated multimodal attention, multiple noisy image terms are iteratively denoised to generate an image containing multiple reference subjects. In some embodiments, the generation model can repeatedly perform at least some of the operations in S200 to S600 above at multiple denoising time steps, so that noisy image terms in different denoising stages and different attention layers can dynamically adjust their attention allocation to multiple reference subjects.

[0072] In some embodiments, the generative model can be a velocity field prediction model, which, at the current denoising time step t, is based on a noisy latent representation. Output velocity field prediction with conditional information c The conditional information c may include text prompts, reference images, and the reassigned multimodal attention obtained by the method of this disclosure. The generative model may update the latent representation based on the prediction results until the final latent representation is obtained, which is then decoded by the image decoder to obtain the generated image.

[0073] like Figure 2 As shown, in the data stream of multimodal attention routing, the text prior and background prior of each reference subject are first obtained from the noise to text attention. Then, these priors are combined with the query features of noisy image words to form concatenated features, which are input into the neural network to obtain residuals and assigned weights. Subsequently, the assigned weights are converted into attention score biases of the reference subjects, and the overall attention quality of the reference subjects is maintained by sharing the offset. Figure 2 The "joint space router" in the text is a specific implementation of a neural network. Figure 2 The "residual" in this context represents the word-by-word meta-correction of the neural network output. Figure 2 The "assignment weight α" in the text corresponds to the route distribution.

[0074] Figure 2 In <p1> 、 <p2>The examples illustrate subject placeholders corresponding to different reference subjects; the term "learnable" in the illustration indicates that the lexical representation of the subject placeholder can be used as a learnable parameter. This illustration corresponds to the aforementioned optional implementation and does not imply that this disclosure must use subject placeholders.

[0075] like Figure 3 As shown, in one specific implementation, the neural network includes an input projection layer, multiple joint spatial residual blocks, and an output projection layer. The input projection layer can be implemented by 1×1 convolutions to perform channel transformation on the concatenated features z. The joint spatial residual blocks can include channel-wise 3×3 convolutions, group normalization, SiLU activation, and pointwise 1×1 convolutions. The features processed by the joint spatial residual blocks can be further grouped and normalized, and then modulated based on the current denoising time step t and the layer index l of the current attention layer. The modulated features are then activated by SiLU and convolved with 1×1 to output the residual Δ. Figure 3 This mainly corresponds to the process of inputting concatenated features into a convolutional network, the modulation of the convolutional network by time steps and layer indices, and the output of residuals; the generation of concatenated features z and the normalization of weights α can be combined with... Figure 1 and Figure 2 understand.

[0076] In some embodiments, the number of joint spatial residual blocks can be two, or it can be set to one, three or more depending on the model size. The number of channels, kernel size, normalization method and activation function of the convolutional network can also be adjusted according to the specific generative model. As long as the neural network can determine the residual amount of weight allocation based on the query features of noisy image terms, text priors and background priors, it can be used as the neural network implementation in this disclosure.

[0077] During the training phase, the aforementioned neural network can be pre-trained. The training phase can use a real foreground mask to supervise the neural network, but the inference phase does not require an external foreground mask. This difference is not contradictory: the real foreground mask in the training phase provides a supervisory signal, enabling the neural network to learn how to assign weights based on text priors and query features; in the inference phase, the trained neural network directly outputs weights based on internal model noise and text attention.

[0078] In some embodiments, a true foreground mask This corresponds to the foreground region of the c-th reference subject in the training sample target image. Based on the true foreground mask, each noisy image term is divided into foreground terms belonging to the reference subject, background terms not belonging to any reference subject, and the remaining blurred terms. Specifically, noisy image terms satisfying the following formula can be divided into foreground terms:

[0079] Noisy image terms that satisfy the following formula can be classified as background terms:

[0080] in, , These are the preset first threshold and the second threshold, respectively.

[0081] The remaining noisy image terms can be classified as fuzzy terms. Fuzzy terms can include boundary regions, overlapping regions, or regions that are difficult to reliably assign to a single reference subject or background. Fuzzy terms can be excluded from loss calculation to reduce the impact of uncertain annotations on training.

[0082] For the i-th noisy image term, its target can be determined as For foreground words, It can be its dominant reference subject, for example, making The largest reference subject; for background lexical units, This can serve as background. The weights α and the target are assigned based on the neural network output. Cross-entropy loss can be calculated. Negative losses Interval loss Total loss :

[0083]

[0084]

[0085]

[0086] in, The set of foreground lexical units and the set of background lexical units. The set of foreground words; For the first The target of a noisy image terminology, For the first Noisy image word matching its target The weight, For the first The noisy image word matching The weight of each reference subject, For intervals, , , These are the weights of the cross-entropy loss, the negative loss, and the interval loss, respectively.

[0087] Cross-entropy loss is used to incentivize the neural network to assign foreground words to their corresponding reference subjects and background words to the background. Negative loss further penalizes insufficient assignment of foreground words to the target reference subject. Margin loss widens the weight gap between the target reference subject and the strongest competing reference subjects, thereby reducing subject confusion. During training, the total loss can be used to determine the appropriate weights. Adjust the parameters of the neural network.

[0088] In some embodiments, the true foreground mask can come from an annotated training dataset, or it can be obtained from an existing image segmentation model, manual annotation, or a semi-automatic annotation process. Threshold , Interval m, and loss weight , , The parameters can be set according to the training data and the size of the generated model. The above values ​​are adjustable hyperparameters, and different values ​​can be used without changing the technical solution disclosed herein.

[0089] In some embodiments, the neural network can be used as a lightweight add-on module in conjunction with a pre-trained generative model. During training, the parameters of the neural network can be trained; alternatively, depending on the specific engineering implementation, it may be chosen whether to freeze the generative model base or to train a small number of other adaptive parameters simultaneously. Regardless of the training scope used, as long as the inference stage can assign weights based on the query features of noise-to-text attention and noisy image terms, and apply a bias to the attention quality accordingly, it falls within the technical concept scope of the embodiments of this disclosure.

[0090] In one inference implementation, given the text prompt "The first reference subject is on the left, the second reference subject is on the right, and the background is an indoor scene," and inputting reference images of the first and second reference subjects, the model reads the noise-to-text attention from noisy image words to text words at the current denoising time step. The model averages the intervals of text words describing the first reference subject to obtain the first reference subject text prior; averages the intervals of text words describing the second reference subject to obtain the second reference subject text prior; and averages the text words not describing either reference subject to obtain the background prior. The neural network assigns weights based on these priors and the query features of the noisy image words. Noisy image words located in the left subject region receive a larger first reference subject weight, noisy image words located in the right subject region receive a larger second reference subject weight, and noisy image words in the background region receive a larger background weight. Subsequently, a bias is applied to the pre-normalized attention scores to enhance attention to matching reference subjects and suppress attention to mismatched reference subjects, thereby reducing the probability of the two reference subject appearances blending.

[0091] In an embodiment of the apparatus not shown, an image generation device may be provided. This image generation device may include an acquisition module, a sequence construction module, an attention determination module, a prior determination module, an allocation weight determination module, a bias application module, and a denoising generation module. The acquisition module acquires multiple reference subjects and text cues. The sequence construction module constructs a unified multimodal attention sequence. The attention determination module determines noise-to-text attention and attention quality. The prior determination module determines the text prior and background prior for each reference subject. The allocation weight determination module uses a neural network to obtain allocation weights for multiple noisy image terms. The bias application module applies a bias to the attention quality based on the allocation weights and may further apply a shared offset. The denoising generation module iteratively denoises based on the reassigned multimodal attention to generate an image containing multiple reference subjects.

[0092] It should be understood that the formula symbols in the above specific embodiments are only used to illustrate the technical solutions of the embodiments of this disclosure, and do not limit the variable names or code names in implementation. Those skilled in the art can adjust the network structure, normalization method, modulation method, training data source, hyperparameter values ​​and computing devices without departing from the technical concept of this disclosure.

[0093] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure.

[0094] Reference Figure 4 An electronic device 400 according to embodiments of the present disclosure may include a processor 410 and a memory 420. The processor 410 may include (but is not limited to) a central processing unit (CPU), a digital signal processor (DSP), a microcomputer, a field-programmable gate array (FPGA), a system-on-a-chip (SoC), a microprocessor, an application-specific integrated circuit (ASIC), etc. The memory 420 may store computer programs to be executed by the processor 410. The memory 420 includes high-speed random access memory and / or a non-volatile computer-readable storage medium. When the processor 410 executes the computer program stored in the memory 420, the above-described image generation method can be implemented.

[0095] According to embodiments of this disclosure, a computer-readable storage medium is also provided, the computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described image generation method.

[0096] Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store computer programs and any associated data, data files, and data structures in a non-transitory manner and to provide the computer programs and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer programs. In one example, the computer programs and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer programs and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0097] While some embodiments of this disclosure have been shown and described, those skilled in the art will understand that modifications may be made to these embodiments without departing from the principles and spirit of this disclosure, which are defined by the claims and their equivalents. < / p1> < / p1>

Claims

1. An image generation method, characterized in that, The image generation method includes: Multiple reference subjects and text prompts are obtained, each reference subject corresponds to a reference image, and at least a portion of the descriptive text in the text prompt is descriptive text used to describe the reference image; A unified multimodal attention sequence is constructed by combining multiple noisy image lexical units, text lexical units corresponding to the text prompts, and reference lexical units corresponding to the reference images of each reference subject; wherein, the multiple noisy image lexical units are multiple image lexical units obtained by dividing the latent representation of the image to be generated in the latent space, and the multiple noisy image lexical units correspond to different spatial positions of the image to be generated; Based on the unified multimodal attention sequence, noise-to-text attention and attention quality are determined; wherein, noise-to-text attention represents the attention weight of the plurality of noisy image words to the text words, and the attention quality is the sum of the attention weights of one of the plurality of noisy image words to all reference words of a reference subject; For each reference subject, in the noise-to-text attention process, the attention weights of each noisy image word in the plurality of noisy image words to the text words corresponding to the descriptive text describing the reference subject in the text prompt are averaged to obtain the text prior of the reference subject; and in the noise-to-text attention process, the attention weights of each noisy image word to the text words in the text prompt that do not describe any reference subject are averaged to obtain the background prior; wherein, the background prior corresponds to the background in the image to be generated that does not belong to any reference subject; Based on the query features of each noisy image word, the background prior, and the text prior of each reference subject, a neural network is used to obtain the assigned weights of the multiple noisy image words. The assigned weights include the weight of each noisy image word matching the background and the weight of each reference subject. According to the allocation weight, a bias is applied to the attention quality, thereby redistributing the attention quality of the multiple noisy image units among the multiple reference subjects to obtain the redistributed multimodal attention; Based on the redistributed multimodal attention, the multiple noisy image terms are iteratively denoised to generate an image containing the multiple reference subjects.

2. The image generation method according to claim 1, characterized in that, The method of obtaining the allocation weights of the multiple noisy image terms using a neural network includes: The query features of each noisy image term are compared with priors. By stitching along the channel dimension, the stitching feature is obtained. ,in, For each noisy image term, the query features are... The prior is obtained by stacking the background prior and the text prior of each reference subject; The splicing feature After being rearranged into a two-dimensional latent space grid, the data is input into a convolutional network, which outputs the residual value corresponding to each noisy image term. ; Based on the prior and the residual amount The allocation weights are obtained by the following formula. : in, Assign weights to the given values.

3. The image generation method according to claim 2, characterized in that, The residual Determine by the following formula: in, For the splicing feature, The current denoising time step, The layer index of the current attention layer. For the convolutional network, Let be the residual.

4. The image generation method according to claim 1, characterized in that, Applying a bias to the attention quality based on the assigned weights includes: A bias is applied to the attention score of the i-th noisy image term pointing to the c-th reference term of the reference subject: in, The attention score is the attention score before softmax normalization in the unified multimodal attention sequence, where i is the index of the noisy image word and c is the index of the reference subject. This is defined as the subject-specific bias corresponding to the i-th noisy image word and the c-th reference subject. The weight for matching the i-th noisy image term with the c-th reference subject, where C is the number of the plurality of reference subjects.

5. The image generation method according to claim 4, characterized in that, The step of applying a bias to the attention quality based on the assigned weights further includes: For each noisy image word, after applying the bias, the attention score of the reference word pointing to the plurality of reference subjects is further applied with a shared offset shared among the plurality of reference subjects, so that after applying the shared offset, the total attention quality of the noisy image word to the plurality of reference subjects is equal to the total attention quality before applying the bias.

6. The image generation method according to claim 5, characterized in that, The shared offset is determined by the following formula: in, This represents the shared offset corresponding to the i-th noisy image term; Let be the log-odds function, and i be the index of the noisy image term; The total attention quality of the i-th noisy image term to the plurality of reference subjects before the bias is applied; The total attention quality of the i-th noisy image term to the plurality of reference subjects when the bias has been applied but the shared offset has not yet been applied.

7. The image generation method according to claim 1, characterized in that, The neural network is pre-trained in the following manner: Based on the true foreground mask, each noisy image term is divided into foreground terms belonging to the reference subject, background terms not belonging to any reference subject, and the remaining blurred terms, wherein the following conditions are met: The noisy image words are foreground words, satisfying The noisy image words are background words, and the rest of the noisy image words are blurred words. The first The target of each noisy image word was determined as follows: Foreground words As its main reference subject, for background lexical units For the background; Based on the assigned weights output by the neural network and the target, the cross-entropy loss is calculated using the following formula. Negative losses Interval loss Total loss : Based on the total loss Adjust the parameters of the neural network; in, For indexing noisy image terms, For reference subjects, index; For the first The true foreground mask of the reference subject in the th... The value at each noisy image word. , These are the first threshold and the second threshold, respectively; The set of foreground lexical units and the set of background lexical units. The set of foreground words; For the first The target of a noisy image terminology, For the first Noisy image word matching its target The weight, For the first The noisy image word matching The weight of each reference subject, For intervals, , , These are the weights of the cross-entropy loss, the negative loss, and the interval loss, respectively.

8. An electronic device, characterized in that, The method includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the image generation method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the image generation method according to any one of claims 1 to 7.

Images