Multi-subject personalized image generation method, system, device and storage medium

Abstract

This invention discloses a method, system, device, and storage medium for generating personalized images for multiple subjects. These are corresponding solutions. The solutions include: generating a diffusion blueprint based on text priors and recording initial noise, which extracts the model's inherent semantic-level spatial layout priors from plain text prompts; introducing a region-constrained image cross-attention mechanism, based on the spatial bias of the blueprint mask, forcing the visual features of each subject's reference image to only be effective within the mask area specified by the diffusion blueprint, preventing cross-boundary interaction of subject appearance features; utilizing the reversible noise addition-denoising property of the diffusion model and combining it with the recorded initial noise to construct a mixed initial noise; furthermore, delaying the injection of background image information in the later stages, ultimately achieving a natural fusion of foreground and background; the entire solution is built entirely based on a pre-trained model, requiring no fine-tuning for specific subjects or scenes, and possessing good versatility and scalability.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and computer vision technology, and in particular to a method, system, device and storage medium for generating personalized images from multiple subjects. Background Technology

[0002] In recent years, text-to-image (T2I) generation techniques, represented by diffusion models, have made significant progress, enabling the generation of high-quality natural images based on text prompts. Diffusion models are deep generative models based on a denoising process. Their core principle operates through two stages: in the forward pass, Gaussian noise is progressively added to the training image until it is transformed into pure random noise; in the backward pass (i.e., the generation process), the model learns to start from the noise, progressively denoise it based on the given text description, and reconstruct a high-quality image that conforms to semantics. This model achieves cross-modal generation from natural language descriptions to realistic images.

[0003] Personalized image generation, also known as customized generation or subject-driven image generation, is a task aimed at generating images of a specific subject placed in a completely new scene, pose, or style, based on a few (usually 3-5) reference images of that subject provided by the user. The subject in this context includes things like personal items, pets, and unique IP (intellectual property) images. The core technology lies in enabling a pre-trained general-purpose image generation model to learn and faithfully reproduce the key visual features of a specific subject, while maintaining the model's original scene composition and stylistic generation capabilities.

[0004] Current mainstream technologies can be mainly divided into two categories:

[0005] (1) Model fine-tuning method: This method uses a small number of reference images to fine-tune all or part of the parameters of a pre-trained diffusion model, binding a specific subject to a unique identifier (such as "my_cat"), so that the model internalizes the identifier as a new concept and learns new subject features. Although the generated quality is high, a complete copy of the model or incremental weights needs to be stored for each new concept, resulting in huge training and storage costs. A typical example is DreamBooth (a fine-tuning technique for personalized text-to-image generation).

[0006] (2) Embedded Learning Method: This method fixes all parameters of the pre-trained model and typically initializes a placeholder, namely a pseudo-word, and sets its corresponding concept vector as a trainable parameter. During training, this vector is optimized by backpropagation by minimizing the reconstruction error between the generated image and the reference image, so that it can accurately represent the concept of the specific subject. During generation, the concept can be invoked by using the pseudo-word in the prompt. This method has low storage overhead, but because it relies on only a single word vector to carry complex visual details, the generation fidelity and generalization ability are often limited. A typical example is Text Inversion (a personalized training technique used in image generation models).

[0007] However, in multi-subject personalized generation tasks based on a specific reference image (i.e., simultaneously customizing multiple specific characters, objects, or backgrounds), existing technologies still face the following major technical bottlenecks in practical applications:

[0008] (1) High computation and storage costs: Most current methods require resource-intensive model fine-tuning for each new concept, resulting in high generation and deployment costs and making it difficult to support large-scale online applications.

[0009] (2) Spatial Layout Misalignment Between Text Guidance and Image Conditions: Existing techniques struggle to maintain responsiveness to spatial layout instructions in text prompts while simultaneously introducing reference images. Research has found that text attention maps typically possess clear semantic boundaries. When image conditions are forcibly injected, the strong signals from the image often disrupt the reasonable spatial layout originally established by the text in the model. For example... Figure 1 As shown, the text prompt reads: "A cat and a dog are playing on the grass." However, the image attention map often appears divergent and overlapping. This causes the text and image to fail to work in harmony, resulting in misaligned, overlapping, or missing subject positions, making accurate spatial modeling difficult. Figure 1 In Hyperparameters that control the weights to balance text attention features and image attention features.

[0010] (3) Attribute leakage and confusion of visual features in multi-subject generation: Existing methods typically employ a global cross-attention mechanism to inject the visual features of the reference image into the entire image generation process. This performs reasonably well in single-subject generation, but in multi-subject scenarios (e.g., a cat and a dog playing on the grass), due to the lack of explicit constraints on the effective range of visual features, the features of different subjects are prone to interaction and interference. A common phenomenon is attribute leakage, such as incorrectly rendering the cat's fur texture onto the dog, or merging two subjects to generate a mixture that is neither a cat nor a dog, resulting in a serious decrease in the consistency of identity in multi-subject generation and an inability to accurately respond to user commands.

[0011] (4) In addition to the above three technical bottlenecks, if the user specifies the background style (i.e. the input information includes a background reference image), there is also the problem of the foreground and background blending failure or stiffness. The reason is that this situation belongs to the full scene customization task. If the whole image is generated directly, the complex background texture will often destroy the main features of the foreground. If simple layer overlay or local redraw is used, the foreground subject often lacks lighting, shadow and style that match the background environment, resulting in the generated result showing obvious collage or floating feeling, lacking the visual harmony of the overall scene.

[0012] Therefore, there is an urgent need for a multi-subject personalized image generation technology that can automatically extract spatial layout information from text semantics and guide the image generation process accordingly.

[0013] In view of this, the present invention is hereby proposed. Summary of the Invention

[0014] The purpose of this invention is to provide a method, system, device and storage medium for generating personalized images of multiple subjects, which can achieve accurate identity preservation of multiple subject features and natural integration with the background, and obtain personalized images that simultaneously possess multi-subject fidelity and scene style consistency.

[0015] The objective of this invention is achieved through the following technical solution:

[0016] A method for generating personalized images from multiple subjects, comprising:

[0017] Obtain input information, including: text prompts containing multiple subjects and subject reference images;

[0018] The text prompts are denoised and sampled to obtain a prior image containing the spatial layout, and the corresponding initial noise is recorded; the prior image is semantically segmented in combination with the main subject in the text prompt to obtain the binary mask corresponding to each subject, which is used as a diffusion blueprint.

[0019] Feature extraction is performed on each subject reference image and text prompt, and the features are mapped to corresponding key-value pairs;

[0020] Based on the key-value pairs of each subject reference image, a regional constraint is applied to the cross-attention mechanism with the corresponding diffusion blueprint as the spatial bias term, and the attention features of all subject reference images are calculated comprehensively. Combined with the key-value pairs of the text prompt, the text attention features are calculated through the cross-attention mechanism. Combining the attention features of all subject reference images and the text attention features, the initial noise is gradually denoised, and finally the denoised vector is obtained. Based on the denoised vector, the multi-subject personalized image is decoded.

[0021] A multi-subject personalized image generation system for implementing the aforementioned method includes:

[0022] An input information acquisition unit is used to acquire input information, including: text prompts containing multiple subjects and subject reference images;

[0023] A pre-trained diffusion model is used to denoise sample text-containing prompts to obtain a prior image containing spatial layout and record the corresponding initial noise.

[0024] An open vocabulary segmentation model is used to perform semantic segmentation on prior images by combining the main body in the text prompts, obtain the binary mask corresponding to each main body, and use it as a diffusion blueprint;

[0025] The feature extraction and mapping unit is used to extract features from each subject reference image and text prompt, and map them into corresponding key-value pairs.

[0026] The pre-trained diffusion model is also used to apply regional constraints to the cross-attention mechanism based on the key-value pairs of each subject reference image, with the corresponding diffusion blueprint as the spatial bias term, and to comprehensively calculate the attention features of all subject reference images; combined with the key-value pairs of the text prompt, the text attention features are calculated through the cross-attention mechanism; and the initial noise is gradually denoised by combining the attention features of all subject reference images and the text attention features, and finally the denoised vector is obtained.

[0027] The decoding unit is used to decode multi-subject personalized images based on the denoised vectors.

[0028] A processing device includes: one or more processors; and a memory for storing one or more programs;

[0029] When the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method.

[0030] A readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method.

[0031] As can be seen from the technical solution provided by the present invention above: the diffusion blueprint generation and initial noise recording based on text priors can extract the model's inherent semantic-level spatial layout priors from plain text prompts, providing spatial constraints for subsequent generation, and recording initial noise; furthermore, a region-constrained image cross-attention mechanism is proposed: introducing a spatial bias based on blueprint masks, transforming traditional global attention into region-specific local attention, forcing the visual features of each subject reference image to only take effect within the mask area specified by the diffusion blueprint, preventing cross-boundary interaction of the appearance features of each subject; in addition, the entire solution is built entirely based on a pre-trained model, requiring no fine-tuning for specific subjects or scenes, and possesses good versatility and scalability. Attached Figure Description

[0032] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a visualization diagram of the cross-attention between text and images in a multi-subject scenario, which serves as the background technology of this invention.

[0034] Figure 2 This is a flowchart of a multi-subject personalized image generation method provided in an embodiment of the present invention.

[0035] Figure 3 This is a schematic diagram of the overall framework of a multi-subject personalized image generation method provided in an embodiment of the present invention.

[0036] Figure 4 This is a schematic diagram showing a visual comparison between the present method and existing methods in a single-subject generation task provided by an embodiment of the present invention.

[0037] Figure 5 This is a schematic diagram showing a visual comparison between the present method and existing methods in a multi-agent generation task provided by an embodiment of the present invention.

[0038] Figure 6 The image shows a scene-level personalized customization effect provided in the embodiment of the present invention.

[0039] Figure 7 The following are illustrations showing other multi-entity combinations and scenario-level customized implementation effects provided in the embodiments of the present invention.

[0040] Figure 8 This is a schematic diagram of a multi-subject personalized image generation system provided in an embodiment of the present invention.

[0041] Figure 9 This is a schematic diagram of a processing device provided in an embodiment of the present invention. Detailed Implementation

[0042] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the protection scope of the present invention.

[0043] First, the following explanations are provided for the terms that may be used in this article:

[0044] The terms "comprising," "including," "containing," "having," or other similar semantic descriptions should be interpreted as non-exclusive inclusion. For example, including a technical feature element (such as raw material, component, ingredient, carrier, dosage form, material, size, part, component, mechanism, device, step, process, method, reaction conditions, processing conditions, parameter, algorithm, signal, data, product or article of manufacture, etc.) should be interpreted as including not only the expressly listed technical feature element, but also other technical feature elements that are not expressly listed and are well-known in the art.

[0045] The following provides a detailed description of a multi-subject personalized image generation method, system, device, and storage medium provided by the present invention. Contents not described in detail in the embodiments of the present invention are prior art known to those skilled in the art. Where specific conditions are not specified in the embodiments of the present invention, they are performed according to conventional conditions in the art or conditions recommended by the manufacturer. Where the manufacturers of the reagents or instruments used in the embodiments of the present invention are not specified, they are all conventional products that can be purchased commercially.

[0046] Example 1

[0047] This invention provides a method for generating personalized images for multiple subjects, which mainly includes: (1) obtaining input information, including: text prompts containing multiple subjects and subject reference images; (2) denoising and sampling the text prompts to obtain a prior image containing spatial layout and recording the corresponding initial noise; performing semantic segmentation on the prior image in combination with the subjects in the text prompts to obtain binary masks corresponding to each subject and using them as diffusion blueprints; (3) extracting features from each subject reference image and the text prompts respectively and mapping them to corresponding key-value pairs; (4) applying regional constraints to the cross-attention mechanism based on the key-value pairs of each subject reference image, with the corresponding diffusion blueprint as the spatial bias term, and comprehensively calculating the attention features of all subject reference images; combining the key-value pairs of the text prompts, calculating the text attention features through the cross-attention mechanism, combining the attention features of all subject reference images and the text attention features, gradually denoising the initial noise, and finally obtaining the denoised vector; and decoding the personalized images for multiple subjects based on the denoised vectors.

[0048] Furthermore, if the input information also includes a background reference image, then the above is adjusted as follows: (A1) Extract features from the reference image and map them into corresponding key-value pairs; (A2) Construct a mixed initial noise based on the initial noise, the diffusion blueprint corresponding to each subject, and the background reference image; (A3) Divide the denoising process into a foreground subject generation stage and a delayed background injection stage according to the time step; wherein, in the foreground subject generation stage: based on the key-value pairs of each subject reference image, apply regional constraints to the cross-attention mechanism with the corresponding diffusion blueprint as the spatial bias term, and comprehensively calculate the attention features of all subject reference images; combine the key-value pairs of the text prompt, calculate the text attention features through the cross-attention mechanism, and combine the attention features of all subject reference images and the text attention features to gradually denoise the mixed initial noise; in the delayed background injection stage: combine the key-value pairs of the background reference image, calculate the background reference image attention features through the cross-attention mechanism, and continue to perform denoising processing based on the information required in the foreground subject generation stage, and finally obtain the denoised vector.

[0049] Based on the above description, the method of the present invention adopts the same technical processing path depending on whether the input information includes a background reference image. Generally speaking, if no background reference image is included, then: only feature extraction is required for each subject reference image and the text prompt, and in the denoising process, the initial noise is denoised based on the information of the text information and the subject reference image. During this process, the complete background is generated by combining the information of the text information and the subject reference image. If a background reference image is included, then additional features of the background reference image need to be extracted, and a mixed initial noise is constructed based on the initial noise, the diffusion blueprint corresponding to each subject, and the background reference image. At the same time, the mixed initial noise also needs to be constructed by combining the background reference image. Furthermore, the denoising process is also divided into a foreground subject generation stage and a delayed background injection stage according to the time step.

[0050] like Figure 2 The diagram illustrates the main workflow of a multi-subject personalized image generation method. This workflow provides an example where the input information includes a background reference image, and it primarily includes the following steps:

[0051] Step 1: Obtain input information.

[0052] In this embodiment of the invention, the input information includes: a text prompt containing multiple subjects and a subject reference image; and in this example, the input information also includes a background reference image.

[0053] Step 2: Generation of diffusion blueprint and noise recording based on text prior.

[0054] In this embodiment of the invention, the text prompt containing multiple subjects is denoised and sampled to obtain a prior image containing spatial layout, and the corresponding initial noise is recorded; the prior image is semantically segmented in combination with the subjects in the text prompt to obtain a binary mask corresponding to each subject, which is used as a diffusion blueprint.

[0055] In this embodiment of the invention, the step of denoising and sampling a text prompt containing multiple subjects to obtain a prior image containing the spatial layout and recording the corresponding initial noise includes: for a text prompt containing N subjects, performing denoising and sampling using a pre-trained diffusion model: the pre-trained diffusion model uses the text prompt as a condition to sample a random noise from a standard Gaussian distribution as the initial noise. And perform denoising to generate a priori image. The spatial layout contained in the prior image covers the position, scale, and orientation relationships of N subjects.

[0056] In this embodiment of the invention, the step of semantically segmenting the prior image based on the subject in the text prompt to obtain a binary mask corresponding to each subject, and using it as a diffusion blueprint, includes: based on an open vocabulary segmentation model, using the name of each subject as a query, performing semantic segmentation on the prior image to obtain binary masks corresponding to N subjects. ,in 1 represents the main area, and 0 represents a non-main area. Let H be the binary mask for the i-th subject, and H and W be the height and width of the prior image, respectively; let the binary mask corresponding to each subject be used as the diffusion blueprint of the corresponding subject.

[0057] Step 3: Feature extraction and mapping.

[0058] In this embodiment of the invention, features are extracted from each subject reference image, background reference image and text prompt, and mapped to corresponding key-value pairs.

[0059] Of course, if the input information only contains reference images of each subject and text prompts, then only feature extraction and mapping of these two types of information are involved.

[0060] In this step, the appropriate image encoder and text encoder can be selected according to the actual situation to realize the above feature extraction of images and text, and matched with the corresponding linear layer mapping to key-value pairs.

[0061] Step 4: Construct mixed initial noise.

[0062] In this embodiment of the invention, a hybrid initial noise is constructed based on the initial noise, the diffusion blueprint corresponding to each subject, and the background reference image: the background reference image is inverted to the initial time step T using an inverse diffusion process to obtain background noise containing background texture and lighting information. ; Calculate the background area ;in, Let be the binary mask for the i-th subject, which serves as the diffusion blueprint for the i-th subject. Indicates all The union of the initial noise; the mixed initial noise is constructed by the following formula. :

[0063] ;

[0064] in, Initial noise, It represents the Hadamardi (or Hadama) stack.

[0065] Of course, if the input information only contains reference images of each subject and text prompts, then the initial noise obtained in step 2 above will be used as the input noise for the subsequent denoising process.

[0066] Step 5: Denoising and Image Generation.

[0067] In this embodiment of the invention, if the input includes a background reference image, a background-conditional delayed injection strategy is used. In the foreground subject generation stage: based on the key-value pairs of each subject reference image, a regional constraint is applied to the cross-attention mechanism using the corresponding diffusion blueprint as a spatial bias term, and the attention features of all subject reference images are comprehensively calculated. Combining the key-value pairs of the text prompt, the text attention features are calculated through the cross-attention mechanism. The initial mixed noise is then progressively denoised by combining the attention features of all subject reference images with the text attention features. In the delayed background injection stage: combining the key-value pairs of the background reference image, the attention features of the background reference image are calculated through the cross-attention mechanism, and denoising is further performed based on the information required in the foreground subject generation stage, ultimately obtaining a denoised vector. A multi-subject personalized image is then decoded based on the denoised vector.

[0068] If the input does not contain a background reference image, this part directly obtains the final denoised vector through the foreground subject generation stage, and the input noise at this time is the initial noise obtained in step 2 above, thereby decoding the multi-subject personalized image.

[0069] In this embodiment of the invention, the step of applying regional constraints to the cross-attention mechanism based on the key-value pairs of each subject reference image, with the corresponding diffusion blueprint as the spatial bias term, and comprehensively calculating the attention features of all subject reference images includes:

[0070] For the i-th subject, its reference image, after feature extraction, is mapped to key-value pairs. Using the corresponding diffusion blueprint as a spatial bias term, a regional constraint is imposed on the cross-attention mechanism, and the attention weight is calculated using the following formula. :

[0071] ;

[0072] ;

[0073] in, For key vectors, Let log be a value vector, and log be the logarithmic function. The binary mask for the i-th subject serves as the diffusion blueprint for the i-th subject; These are hyperparameters used to... Logarithmic space softening is performed to obtain the processed mask image. ; softmax is a normalized exponential function; Key vector Dimension; Represents the key vector transpose; is the constraint strength coefficient; Q is the query vector obtained by mapping the initial noise after denoising (or mixed initial noise if the input information includes a background reference image);

[0074] The attention features of all subject reference images are calculated using the following formula:

[0075] ;

[0076] in, The attention features of all subject reference images (image attention features with region constraints), where N is the number of subjects.

[0077] In this embodiment of the invention, the denoising process is defined as starting from the initial time step T and continuing until the final time step 0; a time step threshold is set. , The time stage threshold is used for the foreground subject generation stage, which is the initial time step. In this stage, at each time step, the attention features of all the calculated subject reference images and the text attention features are weighted and summed to obtain fused features. These fused features are then used to denoise the mixed initial noise processed in the previous time step. If the current time step is the initial time step T, then the mixed initial noise processed in the previous time step is the constructed mixed initial noise. The delayed background injection stage is... In this stage, at each time step, the attention features of all the subject reference images and the background reference image are added together, and then weighted and summed with the text attention features to obtain the fusion features. The fusion features are then used to denoise the mixed initial noise after the denoising process of the previous time step.

[0078] The above-mentioned scheme provided by the embodiments of the present invention has the following advantages: (1) The layout prior automatic extraction mechanism based on diffusion blueprint and the initial noise recording can extract the semantic-level spatial layout prior of the model from the plain text prompt, provide spatial constraints for subsequent generation, and record the initial noise state; (2) Region constraint image cross attention mechanism: introduce spatial bias based on blueprint mask, transform the traditional global attention into region-specific local attention, force the visual features of each subject reference image to only take effect within the mask area specified by the diffusion blueprint, and prevent the appearance features of each subject from crossing the boundary and interacting; (3) utilize the noise addition-denoising reversibility of the diffusion model and combine it with the recorded initial noise, so that the constructed mixed initial noise carries the style texture of the background and the structural prior of the foreground; (4) Background condition delayed injection strategy: delay the injection of background image information in the later stage to prevent the strong background signal (such as complex texture) from destroying the integrity of the foreground, and finally achieve the natural fusion of the foreground and the background; (5) The whole scheme is completely based on the pre-trained model, without the need for fine-tuning of specific subjects or scenes, and has good universality and scalability.

[0079] To more clearly demonstrate the technical solution and its effects provided by the present invention, the method provided by the embodiments of the present invention will be described in detail below with reference to specific examples.

[0080] I. Overall Overview of the Plan

[0081] This invention provides a method for generating personalized images from multiple subjects, achieving accurate identity preservation and natural integration with the background. Its core technical points are as follows:

[0082] 1. Text-Prior-Based Diffusion Blueprint Generation and Noise Recording Mechanism: This pioneering paradigm of text-prior-guided layout generation utilizes a frozen pre-trained diffusion model (such as Stable Diffusion XL) to denoise and sample plain text prompts, thereby obtaining a prior image containing the position, scale, and pose relationships of multiple subjects, and recording the initial noise generated to produce this prior image. Subsequently, an open vocabulary segmentation model (such as Grounding-SAM) is used to automatically extract binary masks corresponding to the semantic segmentation of each subject, forming a set of diffusion blueprints, providing precise spatial constraints for subsequent multi-subject feature injection.

[0083] Among them, Stable Diffusion XL (SDXL for short) is a stable diffusion model, and XL indicates an enhanced version; Grounding-SAM is an open-vocabulary image segmentation model that supports text prompts.

[0084] 2. Region-constrained image cross-attention mechanism: Each image condition is restricted to its semantically corresponding region to prevent attribute leakage and structural chaos. For example, a spatial bias term based on blueprint mask is introduced in the image cross-attention layer of IP-Adapter (an image cue adapter that injects image conditions into a diffusion model). Through attention masking, the visual features (Value) of each subject reference image are forced to interact with the latent features (Query) only within the spatial region specified by the blueprint, blocking the leakage of visual features to non-target regions and realizing precise decoupling and independent control of the visual features of multiple subject reference images.

[0085] 3. Background-aware hybrid initial noise construction strategy: If the input information contains a background reference image, the background reference image is mapped back to the latent space by using the reversibility of the diffusion model through DDIM (a method that converts the real image back to the noise latent variable through the reverse diffusion process) and mixed with the recorded initial noise (foreground initial noise) to achieve a natural transition and fusion of background style and foreground subject in terms of light and shadow, tone and other styles.

[0086] 4. Design a background condition delayed injection strategy. If the input information contains a background reference image, only conditions containing text and subject information are injected in the early stage of denoising (i.e., the foreground subject generation stage). In the later stage (i.e., the delayed background injection stage), the background conditions are injected in a delayed manner to prevent complex background details from interfering with the formation of the early foreground layout structure, thereby improving the generation quality.

[0087] 5. Achieve training-free multi-subject generation and scene-level customization: The entire solution is built entirely on a pre-trained model, requiring no fine-tuning of specific subjects or scenes, and possesses good versatility and scalability; In addition to the subject reference image, users can upload background reference images, and through the above-mentioned mixed initial noise construction and background conditional delayed injection strategy, achieve natural control of multiple subjects and background scenes.

[0088] II. Detailed introduction of the entire plan.

[0089] like Figure 3 As shown, the overall framework for implementing the above-described method of the present invention is illustrated. This framework is named the BPComposer framework. Figure 3 The input information in the example shown includes a background reference image. The following sections will first provide a detailed introduction to the core aspects of this invention, and then introduce a multi-subject personalized image generation strategy.

[0090] 1. A diffusion blueprint generation and noise recording mechanism based on text priors: This mechanism aims to extract the model's inherent semantic-level spatial layout priors from plain text prompts, provide spatial constraints for subsequent generation, and record the initial noise state of the process.

[0091] Specifically: Given a text prompt containing N target subjects, a pre-trained diffusion model (text-to-image diffusion model) is invoked for denoising sampling. During this process, the sampling is performed only conditioned on the text prompt, following a standard Gaussian distribution. Sample random noise as initial noise It then begins denoising to generate a priori image. Record the corresponding initial noise as The prior image contains the diffusion model's understanding of the potential spatial layout of the scene described by the text prompt, based on its internal prior knowledge.

[0092] For example, the text prompt could be: A cat and a dog are playing on the grass.

[0093] Subsequently, an open-source open vocabulary segmentation model was invoked, using the name of each subject in the text prompt (e.g., cat, dog) as the query vector, to process the prior image. Perform semantic segmentation and automatically extract binary masks corresponding to N subjects. ,in 1 represents the main subject region, and 0 represents the non-subject region. This set of binary masks constitutes the diffusion blueprint (spatial layout blueprint) that guides the subsequent multi-subject synthesis.

[0094] 2. Region-constrained image cross-attention mechanism: Introducing a spatial bias based on diffusion blueprints, the traditional global attention is transformed into region-specific local attention, which forces the reference image features of each subject to be effective only within the mask area specified by the diffusion blueprint, preventing the features of each subject from interacting outside the boundaries.

[0095] Specifically, to achieve precise feature control across multiple subjects, this invention introduces a spatial bias based on a diffusion blueprint into the image cross-attention layer of the pre-trained diffusion model. For the i-th subject reference image provided by the user, its visual features are first extracted by an image encoder and then mapped to key-value pairs through a linear layer. To ensure that the visual concept is effective only within the spatial region specified in its blueprint, this invention imposes a regional constraint on the image cross-attention mechanism, introducing a blueprint mask as a spatial bias when calculating the attention score. The attention weight calculation formula is as follows:

[0096] ;

[0097] ;

[0098] in, The constraint strength coefficient, The larger the value, the more strictly the model confines attention to the mask region. τ is a minimum value (e.g., 10). -5 ),right Logarithmic space softening is performed to obtain The meanings of the other parameters can be found in the previous text, and will not be repeated here.

[0099] When the location is inside the mask ( )hour, Attention weights are unaffected when the position is outside the mask. )hour, The attention weights approach zero after processing by the Softmax function. This mechanism forces the attention weights of the i-th subject to... It can only participate in calculations within its corresponding spatial region, thus effectively preventing attribute leakage between multiple entities.

[0100] 3. Background-aware hybrid initial noise construction strategy: If the input information contains a background reference image, the background reference image is inverted back to the initial latent space by utilizing the noise addition-denoising reversibility of the diffusion model, and combined with the recorded initial noise, so that the constructed hybrid initial noise carries both the style texture of the background and the structural prior of the foreground.

[0101] Specifically: for the background reference image Using DDIM Inversion, it is inverted back to the initial time step T to obtain background noise containing background texture and lighting information. Simultaneously, retrieve the prior image generated from the previous text. Initial noise used at time The initial noise contains layout information of the foreground subject.

[0102] Calculate the background region mask (i.e., the entire map area minus all) (union of the initial noise) to construct the mixed initial noise :

[0103] ;

[0104] in, This represents the Hadamard product (element-wise multiplication). At the latent space level, this strategy ensures that at the starting point of denoising, the background region carries user-specified style information, while the foreground region retains the layout priors of the generated text, achieving an organic fusion between the two.

[0105] If the input information does not contain a background reference image, this step is unnecessary; the initial noise obtained in the aforementioned process can be directly used as the input noise for denoising.

[0106] 4. Design a background condition delayed injection strategy. If the user's input includes a background reference image, then in the early stage (i.e., the foreground subject generation stage), only text and subject image-related information are injected to establish a stable screen layout and subject identity; in the later stage (i.e., the delayed background injection stage), background image-related information is injected in a delayed manner to prevent excessively strong background signals (such as complex textures) from destroying the integrity of the foreground, and ultimately achieve a natural fusion of the foreground subject and the background.

[0107] Specifically, in order to solve the problem of strong background signal interference in the generation of foreground subjects, this invention adopts a time step t-based delayed injection strategy.

[0108] The denoising process is defined as moving from t=T (full noise latent representation) to t=0 (sharp image latent representation). If the user's input includes a background reference image, a time step threshold is set. The denoising process is divided into two stages. The foreground subject generation stage of the denoising process (…) Only text and region-constrained subject information are injected to prioritize confirming the subject's location and maintaining its identity; the background injection stage is delayed. While maintaining the information injected during the foreground subject generation stage, the injection of background information is activated to perform the final style rendering and texture filling of the scene.

[0109] This coarse-to-fine injection strategy effectively balances the text's control over the layout with the image's fidelity in terms of detail, avoiding compositional collapse caused by introducing complex visual features too early, and achieving a natural transition and fusion between the foreground and background.

[0110] 5. Multi-subject personalized image generation strategy.

[0111] Since this invention relates to both text and image modalities, a decoupled cross-attention mechanism is employed to process text and image features in parallel, and the aforementioned region constraints are introduced in the image branch.

[0112] For the text branch: The input text prompt is encoded using a frozen text encoder aligned with the pre-trained weights of the SDXL model to obtain global text features. These features are then mapped to key vectors through the frozen linear layers in the SDXL model. Sum value vector Cross-attention is calculated as follows:

[0113] ;

[0114] in, Text attention features; Represents the key vector The transpose of; since the key vector dimensions of the text and the reference image are the same in the embodiments of the invention, the symbol d is used to represent them uniformly; the meanings of the other symbols are as described above.

[0115] For example, the text encoder could be CLIP Text Encoder, which is the text encoder in the CLIP (Contrastive Language-Image Pretrained) model.

[0116] For the image branch: the core visual features of the reference subject are extracted using the frozen IP-Adapter image branch, and mapped to the key vector of the i-th image condition through two lightweight linear layers of the IP-Adapter. Sum value vector Combining the region constraint mechanism described above, the cross-attention calculation for all subject reference images is as follows:

[0117] .

[0118] The background reference image is also processed using a similar method for feature extraction and mapping to obtain the corresponding key vector. Sum value vector The cross-attention between all subject reference images and background reference images is calculated as follows:

[0119] ;

[0120] in, For image attention features, the image here includes all subject reference images and background reference images; Represents the key vector The transpose of .

[0121] To achieve effective collaboration between text and image features, the two types of attention features are weighted and summed to obtain the fused features. And used in the corresponding stage of the denoising process:

[0122] ;

[0123] in, The hyperparameters control the weights to balance text attention features and image attention features. The above formula corresponds to the foreground subject generation stage; if it is the delayed background injection stage, then... Replace with .

[0124] In summary, the BPComposer framework described in this invention is a training-free framework, with each part built based on a pre-trained model, requiring no additional training or fine-tuning. A two-part generation strategy is employed during inference:

[0125] Part 1: Text-Prior-Based Diffusion Blueprint Generation and Noise Recording. Input text prompts are used, and a pre-trained model performs text-guided denoising generation. First, a prior image is generated, then an open-vocabulary segmentation model is called to extract diffusion blueprints for each subject, and initial noise is cached. See the previous section for details.

[0126] Part Two: Personalized Image Generation.

[0127] If the input information contains a background reference image, a mixed initial noise is constructed based on the cached initial noise, the diffusion blueprints corresponding to each subject, and the background reference image. This initial noise is then used as the starting point for the denoising process (i.e., the input noise for the denoising process). In each step of the denoising process, the text attention features and the attention features of all subject reference images are calculated in parallel. The time step determines whether to inject background attention features. If it is the foreground subject generation stage, no background attention features are injected; if it is the delayed background injection stage, background attention features are injected. This process is repeated to gradually denoise the mixed initial noise, and the final decoded output is a personalized image that simultaneously possesses multi-subject fidelity and scene style consistency.

[0128] If the input information does not contain a background reference image, the denoising process only involves the foreground subject generation stage, and the denoising process is performed starting from the initial noise.

[0129] III. Example Introduction.

[0130] This embodiment is based on the Stable Diffusion XL (SDXL) model architecture, combined with the IP-Adapter image encoder and the Grounding-SAM model. All models are deployed on a server equipped with a single NVIDIA RTX 3090 GPU, using PyTorch 2.1 and FP16 precision throughout to reduce latency and memory usage. NVIDIA RTX 3090 is an NVIDIA graphics processor; PyTorch is a deep learning framework; and FP16 is a 16-bit floating-point format. Using Stable Diffusion XL as the base model, it integrates the CLIP text encoder, IP-Adapter image encoder, Grounding-SAM model, and DDIM inversion module. All components are kept frozen, achieving true training-free operation.

[0131] In the example, the main implementation steps are as follows:

[0132] 1. Blueprint generation and initial noise recording.

[0133] The input text prompt is used with a fixed random seed, and the SDXL model is run for 50 steps of denoising inference. This process does not use any reference image, but only relies on the text prompt to generate a highly consistent prior image, recording and caching the initial sampled standard Gaussian noise (i.e., initial noise). Then, the Grounding-SAM model is called to detect the noun subjects in the text prompt and output the corresponding binary mask. This binary mask is the diffusion blueprint generated subsequently.

[0134] Those skilled in the art will understand that a fixed random seed ensures the determinism of the prior image generated from the same text prompt and its extracted spatial layout mask, thus providing a stable spatial constraint benchmark for subsequent conditional generation. Based on this, the denoising trajectory reusing this deterministic initial noise starting point can maintain spatial structure alignment with the aforementioned layout prior, achieving reproducible guidance from the model's internal prior to controllable generation.

[0135] 2. Feature extraction and encoding.

[0136] For user-provided text prompts, features are extracted using CLIP Text Encoder and mapped into serialized key-value pairs using a pre-trained linear projection layer of SDXL. This process keeps the parameters of SDXL frozen.

[0137] For the user-provided subject reference image, its high-dimensional visual features are extracted by the IP-Adapter image encoder and mapped into serialized key-value pairs through the linear projection layer of the IP-Adapter. This process keeps the parameters of the IP-Adapter and SDXL frozen.

[0138] 3. Construct by mixing initial noise.

[0139] If the user provides a specific background reference image, it is backsampled 50 times using DDIM Inversion to obtain background noise. The initial mixed noise is then constructed according to the method described above.

[0140] 4. Controlled denoising generation.

[0141] The image denoising process consists of 50 steps during generation. If the user's input contains a background reference image, then in the first 60% (i.e., ... The first 40% of the time step is used as the foreground / subject generation stage, injecting text and subject-related information into the SDXL model. The last 40% of the time step is used as the delayed background injection stage, which injects additional background information while retaining the information injected in the foreground / subject generation stage. The final denoised vector is then decoded by a VAE (Variational Autoencoder) to output the image.

[0142] For any parts not detailed in the above example, please refer to the preceding text or existing technologies; they will not be elaborated upon here.

[0143] IV. Performance Description.

[0144] To illustrate the performance of the above-described solution of the present invention, a comparison with existing methods was conducted through testing.

[0145] This invention employs two sets of test benchmarks to cover diverse generation requirements. For personalized generation of a single subject, the publicly available DreamBench++ dataset was used. This dataset contains 110 different subjects, each with 9 complex text prompts. A total of 3960 images were generated for large-scale evaluation.

[0146] To address the lack of multi-agent complex interaction scenarios in existing datasets, this invention constructs a dedicated multi-agent test set containing 81 agent combinations. This test set selects 24 representative agents from the DreamBench, DreamBench++, and MSBench datasets, and uses GPT-4o to generate test cue words containing different scenes, spatial arrangements, and atmosphere descriptions. Four cue words are generated for each combination, resulting in a total of 1296 images. This fills the gap in evaluating multi-agent complex interactions and ensures that the test results accurately reflect the model's performance in handling spatial relationships and semantic interactions. GPT-4o is a versatile, multimodal large language model.

[0147] To evaluate the model's adherence to text instructions, the CLIP-T metric was used to calculate the similarity between the generated image and the input text prompt. This metric is calculated using image features extracted by the CLIP model's image encoder. To evaluate the consistency between the generated subject and the reference image, the CLIP-I and DINOv2 metrics were used. DINOv2, due to its stronger robustness, was used as a key criterion for measuring the degree of visual feature preservation. The DINOv2 metric (a self-supervised visual pre-trained model) is an evaluation metric calculated from the image features extracted by DINOv2. Furthermore, to quantify the overall quality and aesthetics of the generated images, this invention introduces two metrics: PickScore and HPSv2. These metrics are trained based on large-scale human preference data and can effectively simulate human subjective judgment of image quality. PickScore uses a dedicated preference model (Pick-a-Pic) to score the generated images, and its score directly reflects the probability of the image winning relative to a given text prompt in human preference selection. Its core evaluation is the semantic alignment quality between the generated image and the text prompt. HPSv2 is an extended version of PickScore. While inheriting the core text alignment evaluation, it introduces an independent aesthetic scoring branch, simultaneously outputting text alignment scores and aesthetic quality scores, providing a more comprehensive and reliable evaluation of the generated images.

[0148] The existing methods involved in the test include: (1) FreeCustom: a method for generating custom images based on reference concepts and combining multiple concepts; this method introduces a multi-reference self-attention mechanism and a weighted masking strategy, enabling the generated images to focus more on reference concepts; (2) MS-Diffusion: a method for personalized generation of zero-shot images of multiple subjects. This method solves the detail fidelity problem through layout guidance and multi-subject cross-attention mechanism; (3) BLIP-Diffusion: a subject-driven image generation model that supports multimodal control of subject images and text prompts. This method introduces a novel multimodal encoder that is trained to provide subject representations; (4) -ECLIPSE: An efficient method for generating personalized text images for multiple subjects. By running directly in the pre-trained CLIP latent space, it bypasses the complex parameter tuning costs within the diffusion model and achieves personalized generation of multiple subjects; (5)SSR-Encoder: A method for selectively and faithfully extracting feature representations of specified subjects from reference images; It extracts fine-grained subject representations without fine-tuning through token-to-patch alignment and detail-preserving encoders; (6)IP-Adapter: An image cue adapter that injects image conditions into the diffusion model.

[0149] The method provided by this invention has produced significant beneficial effects in the field of personalized image generation, opening up new paths for applications such as film and television-level visual content creation, virtual scene construction, and personalized wallpaper generation. Its main advantages are as follows:

[0150] 1. Achieve high-quality single / multi-subject image generation: While maintaining consistency in the appearance of the subjects, improve the consistency of text and the overall visual appeal of the images.

[0151] In single-subject generation tasks, existing mainstream methods often fail to adapt to complex scenes, and their visual appeal needs improvement. This invention, while maintaining the subject's identity characteristics, flexibly changes the subject's posture, perspective, and composition based on text prompts, preserving not only object details but also accurately recreating the context described in the text. Comparative results are as follows: Figure 4 As shown. Figure 4 Each row in the diagram corresponds to a single subject generation task. The leftmost part is the subject reference image, and each column to its right shows the output results of each method. The text below is the prompt text. For example, "A photograph of a catlounging in a sunny window" is the prompt text for each single subject generation task in the first row. It should be noted that the language of the text prompts involved in this invention can be selected according to the actual situation. Only an example of English text prompts is provided here. Figure 4 Each column.

[0152] In multi-subject generation tasks, existing methods such as FreeCustom often suffer from attribute leakage, such as incorrectly rendering the texture of one animal onto another, or exhibiting anomalies in subject blending. While MS-Diffusion solves some layout problems by using preset bounding boxes, its rigid constraints limit natural interaction between subjects, resulting in stiff images, and it still doesn't fully adhere to the semantics of the text (e.g., generating three dogs when requested). In contrast, the images generated by this invention maintain the independent identity characteristics of each subject while achieving a natural and coherent composition. The comparison results are as follows... Figure 5 As shown. Figure 5 Each row in the table corresponds to a multi-subject generation task. The left side shows the subject reference image, the right side shows the text prompt, and the middle shows the output results of each method. Similarly, only examples of English text prompts are provided here.

[0153] 2. Overall Generation Results: Compared to existing methods, experiments on the DreamBench++ dataset and a self-built multi-topic dataset show that this method outperforms mainstream methods (including IP-Adapter, BLIP-Diffusion, etc.) in metrics such as CLIP-T, DINOv2, PickScore, and HPSv2. (ECLIPSE, SSR-Encoder, FreeCustom, and MS-Diffusion) The results are shown in Table 1.

[0154] Table 1: Quantitative Comparison Results Between Our Method and Existing Methods

[0155]

[0156] In single-subject generation tasks, while methods such as IP-Adapter perform reasonably well in terms of subject fidelity, they often sacrifice responsiveness to text commands. In contrast, this invention achieves a CLIP-T score of 35.08, ranking first among all compared methods, and also achieves the highest scores on PickScore (23.05) and HPSv2 (32.26), which reflect human preferences.

[0157] In the more challenging multi-subject generation task, the advantages of this invention are even more pronounced. Experimental data shows that this invention achieves a CLIP-T score of 35.77, significantly outperforming [previous technologies]. The results of ECLIPSE (34.55) and MS-Diffusion (33.17) demonstrate that the blueprint-based generation strategy can more effectively solve the text omission problem in multi-agent scenarios. Furthermore, this invention also ranks first in PickScore (22.89) and HPSv2 (33.33) in multi-agent tasks, indicating that the layout generated using the model's prior knowledge is more aesthetically pleasing and better reflects user intent than images generated using traditional forced constraints.

[0158] 3. Supports personalized backgrounds: A delayed background injection strategy enables a natural blending of background style and foreground subject, supporting complete scene customization. For example... Figure 6 and Figure 7 As shown.

[0159] 4. No training required, with good versatility: The framework is built entirely on a pre-trained model and is suitable for a variety of subject and scenario combinations.

[0160] Through the above description of the embodiments, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the above embodiments can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.), including several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0161] Example 2

[0162] This invention also provides a multi-subject personalized image generation system, which is mainly used to implement the methods provided in the foregoing embodiments, such as... Figure 8 As shown, the system mainly includes:

[0163] An input information acquisition unit is used to acquire input information, including: text prompts containing multiple subjects and subject reference images;

[0164] A pre-trained diffusion model is used to denoise and sample text prompts to obtain a prior image containing spatial layout and record the corresponding initial noise.

[0165] An open vocabulary segmentation model is used to perform semantic segmentation on prior images by combining the main body in the text prompts, obtain the binary mask corresponding to each main body, and use it as a diffusion blueprint;

[0166] The feature extraction and mapping unit is used to extract features from each subject reference image and text prompt, and map them into corresponding key-value pairs.

[0167] The pre-trained diffusion model is also used to apply regional constraints to the cross-attention mechanism based on the key-value pairs of each subject reference image, with the corresponding diffusion blueprint as the spatial bias term, and to comprehensively calculate the attention features of all subject reference images; combined with the key-value pairs of the text prompt, the text attention features are calculated through the cross-attention mechanism; and the initial noise is gradually denoised by combining the attention features of all subject reference images and the text attention features, and finally the denoised vector is obtained.

[0168] The decoding unit is used to decode multi-subject personalized images based on the denoised vectors.

[0169] Furthermore, if the input information also includes a background reference image, then:

[0170] The feature extraction and mapping unit is also used to extract features from the reference image and map them into corresponding key-value pairs;

[0171] Furthermore, it also includes a hybrid initial noise construction unit for constructing hybrid initial noise based on the initial noise, the diffusion blueprint corresponding to each subject, and the background reference image;

[0172] Furthermore, the denoising process of the pre-trained diffusion model is divided into a foreground subject generation stage and a delayed background injection stage according to the time step. In the foreground subject generation stage: based on the key-value pairs of each subject reference image, the cross-attention mechanism is subjected to regional constraints with the corresponding diffusion blueprint as the spatial bias term, and the attention features of all subject reference images are calculated comprehensively. Combined with the key-value pairs of the text prompt, the text attention features are calculated through the cross-attention mechanism. The initial noise is gradually denoised by combining the attention features of all subject reference images and the text attention features. In the delayed background injection stage: combined with the key-value pairs of the background reference image, the attention features of the background reference image are calculated through the cross-attention mechanism. Denoising is further performed on the basis of the information required in the foreground subject generation stage, and finally the denoised vector is obtained.

[0173] Since the main technical details of this system have been described in detail in previous embodiments, they will not be repeated here.

[0174] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the system can be divided into different functional modules to complete all or part of the functions described above.

[0175] Example 3

[0176] The present invention also provides a processing device, such as Figure 9 As shown, it mainly includes: one or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in the foregoing embodiments.

[0177] Furthermore, the processing device also includes at least one input device and at least one output device; in the processing device, the processor, memory, input device, and output device are connected via a bus.

[0178] In this embodiment of the invention, the specific types of the memory, input device, and output device are not limited; for example:

[0179] Input devices can be touchscreens, image acquisition devices, physical buttons, or mice, etc.

[0180] The output device can be a display terminal;

[0181] The memory can be random access memory (RAM) or non-volatile memory, such as disk storage.

[0182] Example 4

[0183] The present invention also provides a readable storage medium storing a computer program that, when executed by a processor, implements the method provided in the foregoing embodiments.

[0184] In this embodiment of the invention, the readable storage medium is a computer-readable storage medium and can be disposed in the aforementioned processing device, for example, as a memory in the processing device. Furthermore, the readable storage medium can also be any medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0185] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims. The information disclosed in the background section is intended only to enhance the understanding of the overall background technology of the present invention and should not be construed as an admission or implication in any way that such information constitutes prior art known to those skilled in the art.

Claims

1. A method for generating personalized images from multiple subjects, characterized in that, include: Obtain input information, including: text prompts containing multiple subjects and subject reference images; The text prompts are denoised and sampled to obtain a prior image containing the spatial layout, and the corresponding initial noise is recorded; the prior image is semantically segmented in combination with the main subject in the text prompt to obtain the binary mask corresponding to each subject, which is used as a diffusion blueprint. Feature extraction is performed on each subject reference image and text prompt, and the features are mapped to corresponding key-value pairs; Based on the key-value pairs of each subject reference image, a regional constraint is applied to the cross-attention mechanism with the corresponding diffusion blueprint as the spatial bias term, and the attention features of all subject reference images are calculated comprehensively. Combined with the key-value pairs of the text prompt, the text attention features are calculated through the cross-attention mechanism. Combining the attention features of all subject reference images and the text attention features, the initial noise is gradually denoised, and finally the denoised vector is obtained. Based on the denoised vector, the multi-subject personalized image is decoded.

2. The multi-subject personalized image generation method according to claim 1, characterized in that, Also includes: If the input information also includes a background reference image; Then: feature extraction is performed on the reference image, and it is mapped to the corresponding key-value pairs; A mixed initial noise is constructed based on the initial noise, the diffusion blueprint corresponding to each subject, and the background reference image; Furthermore, the denoising process is divided into a foreground subject generation stage and a delayed background injection stage according to the time step; among them, the foreground subject generation stage: based on the key-value pairs of each subject reference image, the corresponding diffusion blueprint is used as a spatial bias term to apply regional constraints to the cross-attention mechanism, and the attention features of all subject reference images are calculated in a comprehensive manner. By combining the key-value pairs of the text prompts and using a cross-attention mechanism, the text attention features are calculated. The attention features of all subject reference images are combined with the text attention features to progressively denoise the mixed initial noise. In the delayed background injection stage, the background reference image attention features are calculated by combining the key-value pairs of the background reference image and using a cross-attention mechanism. Denoising is then performed based on the information required in the foreground subject generation stage to finally obtain the denoised vector.

3. The multi-subject personalized image generation method according to claim 2, characterized in that, The denoising process, which is divided into a foreground subject generation stage and a delayed background injection stage according to time steps, includes: The denoising process is defined as starting from the initial time step T and continuing to the final time step 0; a time step threshold is set. , This represents the threshold for the time phase. The foreground subject generation stage is the initial time step. In this stage, at each time step, the attention features of all the subject reference images and the text attention features are weighted and summed to obtain the fusion features, and the fusion features are used to denoise the mixed initial noise after the denoising process of the previous time step; if the current time step is the initial time step T, the mixed initial noise after the denoising process of the previous time step is the constructed mixed initial noise. Delayed background injection phase is In this stage, at each time step, the attention features of all the subject reference images and the background reference image are added together, and then weighted and summed with the text attention features to obtain the fusion features. The fusion features are then used to denoise the mixed initial noise after the denoising process of the previous time step.

4. The multi-subject personalized image generation method according to claim 1, characterized in that, The step of denoising and sampling the text prompt to obtain a priori image containing the spatial layout, and recording the corresponding initial noise, includes: For a text prompt containing N subjects, a pre-trained diffusion model is used to perform denoising sampling: the pre-trained diffusion model uses the text prompt as a condition and samples a random noise from a standard Gaussian distribution as the initial noise. And perform denoising to generate a priori image. The spatial layout contained in the prior image covers the position, scale, and orientation relationships of N subjects.

5. The multi-subject personalized image generation method according to claim 1, characterized in that, The step of semantically segmenting the prior image based on the subject in the text prompt to obtain a binary mask corresponding to each subject, and using it as a diffusion blueprint, includes: Based on an open vocabulary segmentation model, the name of each subject is used as the query to perform semantic segmentation on the prior image, obtaining binary masks corresponding to N subjects. ,in 1 represents the main area, and 0 represents a non-main area. Let H be the binary mask for the i-th subject, and H and W be the height and width of the prior image, respectively. The binary mask corresponding to each subject is used as the diffusion blueprint for that subject.

6. The multi-subject personalized image generation method according to claim 2, characterized in that, The construction of the mixed initial noise based on the initial noise, the diffusion blueprint corresponding to each subject, and the background reference image includes: The background reference image is inverted to the initial time step T using an inverse diffusion process to obtain background noise containing background texture and lighting information. ; Calculate the background area ;in, Let be the binary mask for the i-th subject, which serves as the diffusion blueprint for the i-th subject. Indicates all The union of the initial noise; the mixed initial noise is constructed by the following formula. : ； in, Initial noise, It represents the Hadamardi (or Hadama) stack.

7. The multi-subject personalized image generation method according to claim 1, characterized in that, The key-value pairs based on each subject reference image are used to apply regional constraints to the cross-attention mechanism with the corresponding diffusion blueprint as a spatial bias term, and the attention features of all subject reference images are calculated in a comprehensive manner. For the i-th subject, its reference image, after feature extraction, is mapped to key-value pairs. Using the corresponding diffusion blueprint as a spatial bias term, a regional constraint is imposed on the cross-attention mechanism, and the attention weight is calculated using the following formula. : ； ； in, For key vectors, Let log be a value vector, and log be the logarithmic function. The binary mask for the i-th subject serves as the diffusion blueprint for the i-th subject; These are hyperparameters used to... Logarithmic space softening is performed to obtain the processed mask image. ; softmax is a normalized exponential function; Key vector Dimension; Represents the key vector Transpose of; Q represents the constraint strength coefficient; Q is the query vector obtained by mapping the initial noise after denoising. The attention features of all subject reference images are calculated using the following formula: ； in, The attention features for all subject reference images are denoted by N, where N is the number of subjects.

8. A multi-subject personalized image generation system, characterized in that, To implement the method according to any one of claims 1 to 7, comprising: An input information acquisition unit is used to acquire input information, including: text prompts containing multiple subjects and subject reference images; A pre-trained diffusion model is used to denoise sample text-containing prompts to obtain a prior image containing spatial layout and record the corresponding initial noise. An open vocabulary segmentation model is used to perform semantic segmentation on prior images by combining the main body in the text prompts, obtain the binary mask corresponding to each main body, and use it as a diffusion blueprint; The feature extraction and mapping unit is used to extract features from each subject reference image and text prompt, and map them into corresponding key-value pairs. The pre-trained diffusion model is also used to apply regional constraints to the cross-attention mechanism based on the key-value pairs of each subject reference image, with the corresponding diffusion blueprint as the spatial bias term, and to comprehensively calculate the attention features of all subject reference images; combined with the key-value pairs of the text prompt, the text attention features are calculated through the cross-attention mechanism; and the initial noise is gradually denoised by combining the attention features of all subject reference images and the text attention features, and finally the denoised vector is obtained. The decoding unit is used to decode multi-subject personalized images based on the denoised vectors.

9. A processing device, characterized in that, include: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method as described in any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that, When a computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images