A multi-object scene-oriented training-free conditional diffusion image generation method
By constructing global branches, object-local branches, and extended vision branches, and combining cross-scale semantic alignment and cyclic feature injection, the problem of object-level semantic binding and global structure coordination in multi-object scenarios is solved, generating high-quality, semantically consistent images.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2026-05-27
- Publication Date
- 2026-06-23
Smart Images
Figure CN122265425A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and deep learning technology, and specifically relates to a training-free conditional diffusion image generation method for multi-object scenes. Background Technology
[0002] In recent years, text-to-image diffusion models have made significant progress in image generation tasks, capable of generating high-quality, diverse images based on textual prompts. Building upon this generative capability, researchers have further incorporated spatial cues such as depth maps and Canny edge maps into the generation process, enabling the model to understand the underlying structure of the scene and achieve more precise pixel-level control. This controllable generation capability is of great value for practical applications such as professional design, advertising creativity, and animation production.
[0003] However, as users demand increasingly fine-grained control, relying solely on text prompts and single-conditional images is no longer sufficient to meet the generation requirements of complex multi-object scenarios. Specifically, when multiple similar objects exist in the global spatial structure, existing methods are still prone to semantic leakage. For example, if a user wants to render a specific color, category, or attribute to a target region, the model may incorrectly propagate that attribute to other similar objects, leading to failed object attribute binding. For small-scale objects, due to the limited resolution of local spatial regions, fine-grained structure is further lost after latent variable encoding and downsampling, often resulting in inaccurate local semantics, missing objects, or blurred details in the generated results. Therefore, complex multi-object conditional generation not only requires global spatial structure control but also precise association between the local semantics and spatial location of each object.
[0004] Existing research typically mitigates these issues by introducing layout information to achieve object-level control in complex scenes, primarily including training-free methods and trained methods. Training-free methods usually employ pixel-level feature aggregation, latent variable transfer, or attention modulation to control object position and attributes, offering the advantage of not requiring retraining of the base model. However, differences in feature distribution among different objects can easily lead to distribution shifts during fusion, introducing visual artifacts into the final image. Attention-based modulation methods also suffer from insufficient stability when handling small-scale or dense objects. Trained methods typically use bounding box-guided adapters or layout control modules to control object position and appearance, achieving certain results under specific models and data distributions. However, they introduce additional trainable parameters, increasing training complexity and resource overhead. Furthermore, adapters often struggle to generalize to other base models or new conditional generation backbones, resulting in high transfer costs for applications requiring rapid integration with diffusion Transformer models such as FLUX and StableDiffusion 3. Although the aforementioned control methods that introduce layout information can play a role in controlling fine-grained attributes such as object position, color, and style, they are still prone to problems such as semantic leakage, edge artifacts, and local-global inconsistency when used in conjunction with global spatial conditions.
[0005] In summary, current research on image generation methods for complex multi-object scenarios faces two main scientific challenges: First, the problem of object-level semantic consistency. When multiple similar, small-scale, or densely packed objects exist in the global spatial conditions, existing methods struggle to accurately bind object-level textual descriptions to corresponding spatial regions, easily leading to attribute leakage, object omissions, local semantic mismatches, and missing details of small targets. Second, the difficulty in coordinating and integrating local object features with the global spatial structure. Existing training-free methods are prone to feature distribution shifts, edge artifacts, and local-global inconsistencies when integrating local object features into the global generation process. Furthermore, the timing of feature injection lacks adaptive control; too little feature injection results in insufficient semantic activation, while too much disrupts image naturalness and global structural consistency.
[0006] Therefore, existing methods still lack a training-free solution that can achieve high-precision object-level semantic binding and ensure image generation quality under global structural constraints. The aim is to improve the performance of the model in multi-object complex scene generation tasks without fine-tuning the basic generation model. Summary of the Invention
[0007] In view of the above, the purpose of this invention is to provide a training-free conditional diffusion image generation method for multi-object scenes. By constructing a global branch, an object local branch, and an expanded field of view branch, and sequentially performing cross-scale semantic alignment and cyclic feature injection during the denoising iteration process, the local object features are refined step by step, and the optimized object-level features are adaptively fused into the global generation process. Under the training-free paradigm without model fine-tuning or parameter updates, the method can simultaneously satisfy the precise binding of global spatial structure constraints and object-level semantic description, thereby generating high-quality multi-object scene images with semantic consistency and clear local details.
[0008] To achieve the above-mentioned objectives, the present invention provides the following technical solution: In a first aspect, embodiments of the present invention provide a training-free conditional diffusion image generation method for multi-object scenarios, comprising the following steps: Extract global text semantic features, object-level text semantic features, and global structure graph. Perform local cropping and extended cropping on the global structure graph to obtain object local structure graph and extended vision structure graph. A conditional diffusion model is constructed, which includes global branches, object local branches, and extended vision branches. Each branch is guided by a global structure diagram and global text semantic features, an object local structure diagram and object-level text semantic features, and an extended vision structure diagram and object-level text semantic features, respectively. During the denoising iteration of the conditional diffusion model, cross-scale semantic alignment and cyclic feature injection are performed independently for each object to update the global branch features. Specifically, cross-scale semantic alignment injects the fine-grained semantic features of the local branch of the object into the extended vision branch and achieves spatial coordinate alignment. Cyclic feature injection injects the background features of the global branch into the non-object region of the extended vision branch and back-injects the object-level features of the extended vision branch into the global branch. The final global branch features of the iterative output are decoded to generate the final image.
[0009] Preferably, the step of extracting global text semantic features, object-level text semantic features, and a global structure graph, and performing local and extended cropping on the global structure graph to obtain an object-level local structure graph and an extended view structure graph, includes: Text features are extracted from global text prompts and object-level text descriptions to obtain global text semantic features and object-level text semantic features, respectively. Structural features are extracted from the spatial condition map to obtain a global structure map. Based on the bounding boxes of each object, the global structure map is locally cropped to obtain a local object structure map. Based on the extended view bounding boxes of each object, the global structure map is expanded and cropped to obtain an extended view structure map.
[0010] Preferably, during cross-scale semantic alignment, the step of injecting fine-grained semantic features of the object's local branches into the expanded view branch and achieving spatial coordinate alignment includes: The local branch of the object is based on the local structure graph of the object and the semantic features of the object-level text. It generates fine-grained semantic features through an attention mechanism and aligns the feature space coordinates of the local branch to the feature space coordinates of the extended vision branch through positional encoding. The fine-grained semantic features and their aligned positional encoding are then injected into the extended vision branch. The extended vision branch is based on the extended vision structure graph, the semantic features of the object-level text, and the fine-grained semantic features. It performs feature fusion through an attention mechanism and outputs the extended vision branch features at the object level.
[0011] Preferably, during cyclic feature injection, injecting the background features of the global branch into the non-object region of the expanded view branch includes: Based on the extended field of view bounding box, the background features of the corresponding region are cropped from the global branch features. After fusing the background features with the extended field of view branch features using the preset object region mask in the extended field of view branch, the features are injected only into the non-object regions indicated by the mask, and the updated extended field of view branch features are output.
[0012] Preferably, during cyclic feature injection, the step of back-injecting the object-level features of the expanded view branch to the global branch includes: At the output of each network layer in the global branch, the extended vision branch features are zero-padding the spatial boundaries to match the spatial resolution of the global branch, and then fused with the global branch features to output the updated global branch features.
[0013] Preferably, the method further includes an object-level feature injection early stopping step: At each denoising time step, object-level features of the same object region are extracted from the global branch and the extended field of view branch respectively. After high-frequency information is extracted by a high-pass filter, the similarity is calculated. When the similarity exceeds a preset threshold, the feature injection of the object in subsequent time steps is terminated.
[0014] Preferably, the global branch, the object local branch, and the expanded vision branch share the same pre-trained conditional diffusion model backbone parameters, and no parameter updates are performed during the denoising iteration process.
[0015] Secondly, the present invention provides a training-free conditional diffusion image generation system for multi-object scenes, implemented using the aforementioned training-free conditional diffusion image generation method for multi-object scenes, comprising: The input parsing module is used to extract global text semantic features, object-level text semantic features, and global structure graph. It performs local cropping and extended cropping on the global structure graph to obtain the object local structure graph and the extended field structure graph. The multi-branch building module is used to build a conditional diffusion model that includes a global branch, an object-local branch, and an extended vision branch. Each branch is guided by a global structure graph and global text semantic features, an object-local structure graph and object-level text semantic features, and an extended vision structure graph and object-level text semantic features, respectively. The cross-scale semantic alignment module is used to inject fine-grained semantic features of the local branches of an object into the extended vision branch and achieve spatial coordinate alignment during the denoising iteration of the conditional diffusion model. The recurrent feature injection module is used to inject the background features of the global branch into the non-object region of the extended vision branch during the denoising iteration of the conditional diffusion model, and to inject the object-level features of the extended vision branch back into the global branch. The image decoding output module is used to decode the final global branch features of the iterative output to generate the final image.
[0016] Preferably, the system further includes an object-level adaptive early stopping module, which is used to extract object-level features of the same object region from the global branch and the extended vision branch respectively at each denoising time step, extract high-frequency information through a high-pass filter and calculate similarity. When the similarity exceeds a preset threshold, the feature injection of the object in subsequent time steps is terminated.
[0017] Thirdly, an electronic device provided by an embodiment of the present invention includes a memory and one or more processors. The memory is used to store a computer program, and the processor is used to implement the above-described training-free conditional diffusion image generation method for multi-object scenarios when executing the computer program.
[0018] Compared with the prior art, the beneficial effects of the present invention include at least the following: (1) This invention aligns the local semantics of an object with the surrounding spatial context through the cascade design of object local branches and extended vision branches, effectively ensuring the object-level control precision in multi-object scenarios, enabling the attribute information of different objects to be accurately bound to the corresponding spatial regions, alleviating attribute leakage and local semantic mismatch, and improving the image generation quality.
[0019] (2) This invention injects the features of the local branch of the object into the extended vision branch through cross-scale semantic alignment, and uses the position encoding of the extended vision coordinate space to align the local features, which can enhance the detailed expression of objects at different scales and local object regions and solve the problem of insufficient local semantic details.
[0020] (3) This invention achieves bidirectional feedback from the global background to the extended field of view branch and from the extended field of view object features to the global branch through cyclic feature injection, which reduces edge artifacts caused by feature splicing or local replacement, and makes the local object features and the global background structure coordinated and integrated, resulting in a more natural image.
[0021] (4) The present invention uses an object-level adaptive early stopping strategy to independently control the injection duration of each object based on the high-frequency structural similarity, thereby avoiding edge artifacts and image quality degradation caused by improper timing of object-level feature injection, while reducing unnecessary inference overhead.
[0022] (5) This invention is compatible with spatial conditions such as depth maps and Canny edge maps, and can be integrated as a plug-and-play module into various diffusion Transformer-based condition generation frameworks such as FLUX-Depth, FLUX-Canny, EasyControl, OminiControl, Stable Diffusion 3 plus ControlNet, and has good versatility and engineering application value. Attached Figure Description
[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0024] Figure 1 This is a flowchart illustrating the training-free conditional diffusion image generation method for multi-object scenarios provided in this embodiment of the invention. Figure 2 This is a schematic diagram of the framework of the training-free conditional diffusion image generation method for multi-object scenes provided in the embodiments of the present invention; Figure 3 This is the qualitative comparison result between our method and the existing baseline method under condition input 1; Figure 4 This is a qualitative comparison result between our method and the existing baseline method under conditional input 2; Figure 5 This is a comparison chart showing the improvement effect of this method on the existing baseline method under condition input 3; Figure 6 This is a schematic diagram of the structure of a training-free conditional diffusion image generation system for multi-object scenarios provided in an embodiment of the present invention. Detailed Implementation
[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.
[0026] The inventive concept of this invention is as follows: Addressing the problems encountered in existing technologies regarding semantic leakage, loss of small-scale object details, inconsistency between local object features and global structure fusion, and difficulty in adaptively controlling feature injection timing in complex multi-object scenarios, this invention provides a training-free conditional diffusion image generation method for multi-object scenarios. By constructing global branches, object-local branches, and expanded field-of-view branches, the global spatial structure can be decoupled from the local semantics of multiple objects, allowing the attribute information of each object to be independently bound to its corresponding spatial region, avoiding attribute error propagation and local semantic mismatch between multiple objects. By sequentially performing cross-scale semantic alignment and cyclic feature injection during the denoising iteration process, the fine-grained local features of objects can be bidirectionally aligned and adaptively fused with the global background context, effectively suppressing attribute leakage and edge artifacts while ensuring the integrity of local details of small-scale objects. Finally, without updating model parameters, the method achieves progressive refinement and adaptive fusion of local semantics, effectively improving the generation quality and semantic consistency in multi-object scenarios.
[0027] like Figure 1 As shown in the embodiment, a training-free conditional diffusion image generation method for multi-object scenes is provided, which specifically includes the following steps: S1: Extract global text semantic features, object-level text semantic features, and global structure graph. Perform local cropping and extended cropping on the global structure graph to obtain object local structure graph and extended vision structure graph.
[0028] In the embodiment, a global text prompt is received. Spatial Conditions Diagram Object-level bounding box collection and object-level text description collection Among them, global text suggestions Spatial conditional graphs are textual descriptions used to guide the model in generating the desired results. A set of object-level bounding boxes used to control the overall structure of the generated image, either as a depth condition map or a Canny condition map. Spatial condition diagram The set of bounding boxes of the objects contained therein, and the set of object-level text descriptions. Spatial condition diagram S The text description of the objects contained therein.
[0029] global text hints and text description of each object The tokens are processed by a word segmenter to obtain a global text token sequence and an object-level text token sequence. Then, the global text semantic features and object-level text semantic features are extracted using a T5 text encoder and a CLIP text encoder, respectively.
[0030] Furthermore, regarding the spatial condition map Structural features are extracted to create a global structure map. This is based on the bounding boxes at each object level. Local cropping of the global structure diagram yields the object's local structure diagram. The bounding boxes of each object level are then... The bounding box after expanding the view is obtained by performing a view expansion operation (expanding outwards proportionally along the height and width axes). , This indicates a field-of-view expansion operation. This represents the scaling factor, which is set to 0.5 in this example, based on the bounding boxes of each extended field of view. The global structure graph is expanded and pruned to obtain an expanded view structure graph.
[0031] S2 constructs a conditional diffusion model that includes global branches, object-local branches, and extended vision branches. Each branch is guided by a global structure diagram and global textual semantic features, an object-local structure diagram and object-level textual semantic features, and an extended vision structure diagram and object-level textual semantic features, respectively.
[0032] In this embodiment, based on a pre-trained FLUX-Depth conditional diffusion model, a global branch, an object-local branch, and an extended vision branch are constructed, each containing a multi-layer diffusion Transformer (DiT) network. The global branch is guided by a global structure graph and global textual semantic features; the object-local branch by object-local structure graphs and object-level textual semantic features; and the extended vision branch by extended vision structure graphs and object-level textual semantic features. This achieves progressive refinement and adaptive fusion of local object features through cross-scale semantic alignment and cyclic feature injection without updating model parameters. This allows the global branch to gradually absorb the fine-grained semantics of each object while maintaining consistency with the global background. Feature extraction for each branch image is performed using a VAE encoder. During the initialization phase, the latent variables of the global branch... Latent variables of local branches of the object obtained from standard Gaussian noise sampling. Latent variables of the extended perspective branch By respectively through The cropping region is obtained by cropping, where the cropped region corresponds to the bounding box of each object and its expanded bounding box. , In the final stage, the final network layer outputs of the local branch and the expanded vision branch are discarded, and only the final network layer output of the global branch is retained as the final global branch feature of the iterative output.
[0033] S3, during the denoising iteration of the conditional diffusion model, performs cross-scale semantic alignment independently for each object to inject fine-grained semantic features of the object's local branches into the extended vision branch and achieve spatial coordinate alignment.
[0034] Given the strong generative capabilities of pre-trained conditional diffusion models, existing object-level generation methods typically extract object conditional features first, then directly fuse these features into the global generation stream using bounding boxes. However, these methods still have inherent limitations when introducing global spatial conditions to guide the generation process. For small-scale objects, the local conditional input itself has limited resolution, and the downsampling of the VAE encoder further exacerbates this problem, leading to a severe loss of fine-grained spatial details. Therefore, the generated objects often lack fine details and are difficult to semantically align accurately with their textual descriptions, thus affecting the effectiveness of subsequent multi-object feature fusion. Although expanding the field of view can alleviate this resolution bottleneck to some extent, it inevitably introduces background information and interference from potential neighboring objects, and when guided only by object cues, this interference weakens the consistency between text and image. Therefore, this invention proposes cross-scale semantic alignment, explicitly injecting object features into the expanded field of view branch to enhance object semantics.
[0035] In the embodiments, such as Figure 2 As shown in (a), an attention-based feature fusion mechanism is used to achieve cross-scale semantic alignment. Specifically, the object local branch, based on the object local structure graph and object-level textual semantic features, is encoded through layer normalization and linear layers, and then fine-grained semantic features are generated through an attention mechanism. The fine-grained semantic features corresponding to each object are used to obtain its query. ,key Sum And will query ,key Sum The data is fed into the DiT layer corresponding to the expanded field of view branch. In this layer, these features are compared with the first DiT layer of the expanded field of view branch. Features corresponding to each object , and Concatenating along the sequence dimension is represented as: , , , , in, and These represent the rotation position codes for the expanded view branch and the object local branch, respectively. This indicates a splicing operation. This indicates that the position encoding is explicitly aligned using the extended view branch as the reference coordinate space, specifically according to the first... The object-level bounding box of an object The spatial extent, encoded from the rotational position of the extended field of view branch. Extract the corresponding spatial index subset from the middle as the rotation position encoding of the local branch of the object. Through this strict relative alignment, the expanded field-of-view branch can accurately locate and focus on local prior information. Subsequently, the spliced features... , , Feature fusion is performed in the DiT layer of the extended vision branch. It's worth noting that textual and image features interact intensively in the DiT layer of the extended vision branch, and the textual features themselves also encode some visual information. Simultaneously, the extended vision structure map and object-level textual semantic features are input into the extended vision branch to fully preserve the semantic information in the object's local branches.
[0036] Through the above cross-scale semantic alignment operation, the local branch features of the object are given their true relative positions in the extended field coordinate space, so that the extended field branches can accurately locate and absorb the local semantics of the object.
[0037] S4. During the denoising iteration of the conditional diffusion model, a loop feature injection is performed independently for each object to inject the background features of the global branch into the non-object region of the extended vision branch, and to inject the object-level features of the extended vision branch back into the global branch to update the global branch features.
[0038] Considering that the expanded vision branch is still guided by the object's own descriptive cues, directly integrating optimized features that rely solely on local semantics into the global features can easily lead to visual inconsistencies. Therefore, this invention proposes a recurrent feature injection method, which involves independently injecting background features for each object. This allows the expanded vision branch to perceive the global context while preserving object details, and independently performs object-level feature back-injection for each object, thereby achieving adaptive fusion of local semantics and global structure, avoiding the inconsistencies caused by direct splicing or replacement.
[0039] S4.1, Background Feature Injection.
[0040] In the embodiments, such as Figure 2 As shown in (b), based on the extended field of view bounding box, the latent variables are derived from the global branch space aligned with the extended region space. Extracting background features , represented as: , in, Indicates based on the extended view bounding box Spatial range from global branch latent variables The background features of the corresponding region are extracted.
[0041] In the expanded field of view branch, background features are included. Original features of the extended vision branch Weighted fusion is represented as: , in, This represents the preset object region mask in the extended view branch (1 for object regions and 0 for non-object regions). This indicates element-wise multiplication. This represents the updated expanded view branch features. This fusion method ensures that background features are injected only into the non-object regions indicated by the mask, while the object regions of the original features remain unchanged.
[0042] By performing the background feature injection operation described above, we can avoid the inconsistency between the extended vision branch and the global branch due to the fact that the extended vision branch is only constrained by object-level text prompts and cross-scale semantic alignment. By injecting global background information into the non-object part of the extended vision region through the background feature injection strategy, the extended vision branch can perceive the context of the global branch while retaining the local semantic features of the object region.
[0043] S4.1, Object-level feature back-injection.
[0044] In the embodiment, after cross-scale semantic alignment and background feature injection, object-level features in the expanded field-of-view branch can effectively fuse object-level semantics and spatial context, thereby extracting these features and accurately injecting them into the corresponding regions of the global branch. For example... Figure 2 As shown in (c), at the output of each DiT network layer in the global branch, a set of object-level features extracted from the extended vision branch is generated. With global branch features Perform weighted fusion and output the updated global branch features. , represented as: , in, The first branch extracted from the expanded viewpoint The characteristics of an object Indicates the total number of objects. This represents the weighting coefficient, which is 0.5 in this example. Due to the spatial resolution mismatch between the expanded view branch and the global branch, the extracted features need to be processed before performing the weighted summation. Zero-fill along its spatial boundaries.
[0045] Through the above object-level feature back-injection operation, the object-level features that integrate the local semantics and global context of the object in the extended vision branch can be accurately transferred to the corresponding spatial region of the global branch. This allows the global branch to gradually refine the local details of each object while maintaining the overall structural consistency, thereby effectively improving the semantic fidelity and visual coordination of the generated image in multi-object scenes.
[0046] S5. At each denoising time step, object-level features of the same object region are extracted from the global branch and the extended field of view branch respectively. After high-frequency information is extracted by a high-pass filter, the similarity is calculated. When the similarity exceeds a preset threshold, the feature injection of the object in subsequent time steps is terminated.
[0047] Considering that the guiding steps of object-level feature back-injection have a significant impact on the final image quality, insufficient back-injection will lead to insufficient semantic activation, while excessive back-injection will compromise global consistency. To achieve a trade-off, this invention proposes an object-level adaptive early stopping strategy to dynamically determine the feature injection termination time step for each object.
[0048] In the embodiment, in the first The denoising time step uses the first denoising step in the global branch respectively. object region mask of an object And the branch of expanding vision object region mask of an object Extract corresponding local latent features from the global branch and the expanded vision branch. and These features are then filtered by a high-pass filter. Processing, represented as: , in, High-frequency features representing local latent features of global branches. High-frequency features representing local latent features of extended field-of-view branches.
[0049] Then, the cosine similarity among these high-frequency features was calculated. To dynamically determine the termination of the injection: , in, Represents the cosine similarity function. This represents a predefined similarity threshold, with a default value of 0.85. If... This indicates that the object's edge and outline structure in the global branch is sufficiently consistent with the expanded view branch, thus the object... The feature injection operation will be performed at time step Termination at this point can effectively avoid steps at this termination time. Subsequent object-level feature backinjection leads to manifold interference in the original generation and introduces visual artifacts. This strategy not only significantly reduces inference costs but also ensures the quality of the final generated image.
[0050] S6 decodes the final global branch features output by the iteration to generate the final image.
[0051] In the embodiments, in all After the denoising time step is completed, the final global branch feature output by the global branch is received and input into the VAE decoder or other image decoder to obtain the final image.
[0052] Furthermore, the significant advantages of this invention in object-level image generation tasks are confirmed through a comprehensive evaluation of the following quantitative and qualitative experiments.
[0053] like Figure 3 and Figure 4 The figures show qualitative comparisons between our method and existing baseline methods (including FLUX, Regional-Prompting-FLUX, 3DIS, DreamRenderer, GrounDit, and NoiseCollage) under conditions 1 and 2, respectively. While the FLUX method can generate high-quality images, it often struggles to adhere to spatial layout. Methods such as 3DIS and DreamRenderer occasionally lead to semantic leakage, especially when controlling small-scale objects. Furthermore, Regional-FLUX and NoiseCollage cannot adaptively adjust their guiding step size, resulting in visual artifacts in complex scenes. In contrast, our method demonstrates superior spatial fidelity and object-level accuracy, with generated objects precisely aligned to the input description and maintaining clear boundaries even in overlapping scenes. Notably, our method excels at preserving the semantic attributes of each object without sacrificing overall image realism, and the robust control capability of this invention is further validated by the high visual quality achieved for small objects. In summary, these qualitative results confirm that our method effectively balances high-fidelity image synthesis with strict object-level controllability, and outperforms existing methods in complex multi-object generation.
[0054] Tables 1 and 2 show the quantitative evaluation results of our proposed method compared to existing baseline methods (including FLUX, Regional-FLUX, 3DIS, DreamRenderer, GrounDit, and NoiseCollage). Tests were conducted on the COCO-POS and COCO-MIG benchmarks. Each sample used multiple random seeds to generate images, and evaluation was performed using three categories of metrics: spatial fidelity, semantic consistency, and image quality. Spatial fidelity metrics included mean Intersection over Union (mIoU), mean mean accuracy (mAP), object success ratio (OSR), and image success ratio (ISR); semantic consistency metrics included global CLIP score (G-CLIP), local CLIP score (L-CLIP), Human Preference Score Version 3 (HPSv3), and Pick model score; image quality metrics included Fraser Initial Distance (FID). The results demonstrate that our proposed method achieves a good balance between accurately adhering to spatial conditions and maintaining spatial fidelity.
[0055] Table 1. Quantitative evaluation results of our method compared to existing baseline methods.
[0056] Table 2. Quantitative evaluation results of our method and existing baseline methods in complex multi-object scenarios.
[0057] As shown in Table 1, in the depth map modality, our method achieves the best FID (16.34) and exhibits strong aesthetic quality, ranking second in HPSv3 metrics (6.145). Although NoiseCollage achieves slightly higher spatial alignment metrics, such as OSR and mIoU, in this modality, this is mainly due to the dense background prior provided by the depth map. Such dense priors are beneficial for achieving stricter layout alignment, but often sacrifice overall image quality, as reflected in its higher FID (20.42). In the more challenging Canny edge map modality, the robustness of our method is even more pronounced. Because the Canny condition lacks the dense spatial cues contained in the depth map, multiple baseline models struggle to maintain structural integrity. In comparison, our method demonstrates leading performance in this modality, achieving the highest spatial fidelity, such as an OSR of 50.93 and an ISR of 21.37. It also performs well in human preference-related evaluations, with a Pick score of 0.153 and the second-best FID (18.19).
[0058] As shown in Table 2, to further verify the effectiveness of this method in complex multi-object scenarios, fine-grained evaluation results of the multi-object generation task are presented. As the scene complexity increases from 2 objects (… Increased to 6 objects () The performance of methods such as Regional-FLUX and GrounDit showed a significant decline. In contrast, our method typically maintained the highest or second-highest success rate under both spatial conditions. Especially under the Canny constraint, our method outperformed the baseline methods at all object count levels, ultimately achieving the highest average OSR (48.9) and ISR (12.0).
[0059] Regarding model adaptability, Table 3 shows the performance of our method when integrated with state-of-the-art FLUX-based conditional generation methods OminiControl and EasyControl. Quantitative results show that when applied to OminiControl, our method produces a significant increase in ISR (from 18.13 to 26.02) and mAP (from 44.25 to 56.63). The performance improvement in EasyControl is even more pronounced; after integrating our method's strategy, OSR increased from 46.28 to 63.66, and ISR nearly doubled to 34.77. These continuous improvements, achieved while maintaining stable G-CLIP and L-CLIP scores, highlight how our method effectively addresses the inherent local structural ambiguity and object omissions in current methods. Figure 5 The qualitative visualization results shown further confirm the above results, clearly demonstrating the ability of this method to coordinate complex multi-object layouts and maintain geometric fidelity.
[0060] Table 3. Quantitative evaluation results when this method is integrated with existing baseline methods.
[0061] Furthermore, to demonstrate that this method can be generalized to other DiT-based conditional diffusion models, as shown in Table 4, a quantitative comparison was performed using StableDiffusion3 (SD3) with ControlNet as the backbone. The results show that this method significantly outperforms the baseline methods of StableDiffusion3 and DreamRenderer in terms of spatial fidelity. Overall, this method achieves good object-level controllability on the SD3 architecture while maintaining highly competitive overall aesthetic quality (e.g., HPSv3 score of 5.435).
[0062] Table 4. Quantitative evaluation results of this method extended to other DiT-based conditional diffusion models.
[0063] Based on the same inventive concept, such as Figure 6 As shown, this embodiment of the invention also provides a training-free conditional diffusion image generation system 600 for multi-object scenarios, including: an input parsing module 610, a multi-branch construction module 620, a cross-scale semantic alignment module 630, a cyclic feature injection module 640, an object-level adaptive early stopping module 650, and an image decoding output module 660.
[0064] The input parsing module 610 is used to extract global text semantic features, object-level text semantic features, and global structure graph. It performs local cropping and extended cropping on the global structure graph to obtain object local structure graph and extended field structure graph.
[0065] The multi-branch construction module 620 is used to construct a conditional diffusion model that includes a global branch, an object local branch, and an extended vision branch. Each branch is guided by a global structure diagram and global textual semantic features, an object local structure diagram and object-level textual semantic features, and an extended vision structure diagram and object-level textual semantic features, respectively.
[0066] The cross-scale semantic alignment module 630 is used to inject fine-grained semantic features of the local branches of an object into the extended vision branch and achieve spatial coordinate alignment during the denoising iteration of the conditional diffusion model.
[0067] The cyclic feature injection module 640 is used to inject the background features of the global branch into the non-object region of the extended vision branch during the denoising iteration of the conditional diffusion model, and to inject the object-level features of the extended vision branch back into the global branch to update the global branch features.
[0068] The object-level adaptive early stopping module 650 is used to extract object-level features of the same object region from the global branch and the extended vision branch at each denoising time step. After extracting high-frequency information through a high-pass filter, the similarity is calculated. When the similarity exceeds a preset threshold, the feature injection of the object in subsequent time steps is terminated.
[0069] The image decoding output module 660 is used to decode the final global branch features of the iterative output to generate the final image.
[0070] Based on the same inventive concept, embodiments of the present invention also provide an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-described training-free conditional diffusion image generation method for multi-object scenarios when executing the computer program.
[0071] It should be noted that the training-free conditional diffusion image generation system and electronic device for multi-object scenarios provided in the above embodiments belong to the same inventive concept as the training-free conditional diffusion image generation method for multi-object scenarios. The specific implementation process is detailed in the embodiments of the training-free conditional diffusion image generation method for multi-object scenarios, and will not be repeated here.
[0072] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A training-free conditional diffusion image generation method for multi-object scenes, characterized in that, Includes the following steps: Extract global text semantic features, object-level text semantic features, and global structure graph. Perform local cropping and extended cropping on the global structure graph to obtain object local structure graph and extended vision structure graph. A conditional diffusion model is constructed, which includes global branches, object local branches, and extended vision branches. Each branch is guided by a global structure diagram and global text semantic features, an object local structure diagram and object-level text semantic features, and an extended vision structure diagram and object-level text semantic features, respectively. During the denoising iteration of the conditional diffusion model, cross-scale semantic alignment and cyclic feature injection are performed independently for each object to update the global branch features. Specifically, cross-scale semantic alignment injects the fine-grained semantic features of the local branch of the object into the extended vision branch and achieves spatial coordinate alignment. Cyclic feature injection injects the background features of the global branch into the non-object region of the extended vision branch and back-injects the object-level features of the extended vision branch into the global branch. The final global branch features of the iterative output are decoded to generate the final image.
2. The training-free conditional diffusion image generation method for multi-object scenes according to claim 1, characterized in that, The process of extracting global text semantic features, object-level text semantic features, and a global structure graph, and then performing local and extended cropping on the global structure graph to obtain object-level local structure graphs and extended view structure graphs, includes: Text features are extracted from global text prompts and object-level text descriptions to obtain global text semantic features and object-level text semantic features, respectively. Structural features are extracted from the spatial condition map to obtain a global structure map. Based on the bounding boxes of each object, the global structure map is locally cropped to obtain a local object structure map. Based on the extended view bounding boxes of each object, the global structure map is expanded and cropped to obtain an extended view structure map.
3. The training-free conditional diffusion image generation method for multi-object scenes according to claim 1, characterized in that, When performing cross-scale semantic alignment, the step of injecting fine-grained semantic features of the local branch of the object into the expanded view branch and achieving spatial coordinate alignment includes: The local branch of the object is based on the local structure graph of the object and the semantic features of the object-level text. It generates fine-grained semantic features through an attention mechanism and aligns the feature space coordinates of the local branch to the feature space coordinates of the extended vision branch through positional encoding. The fine-grained semantic features and their aligned positional encoding are then injected into the extended vision branch. The extended vision branch is based on the extended vision structure graph, the semantic features of the object-level text, and the fine-grained semantic features. It performs feature fusion through an attention mechanism and outputs the extended vision branch features at the object level.
4. The training-free conditional diffusion image generation method for multi-object scenes according to claim 1, characterized in that, During cyclic feature injection, the step of injecting the background features of the global branch into the non-object region of the extended view branch includes: Based on the extended field of view bounding box, the background features of the corresponding region are cropped from the global branch features. After fusing the background features with the extended field of view branch features using the preset object region mask in the extended field of view branch, the features are injected only into the non-object regions indicated by the mask, and the updated extended field of view branch features are output.
5. The training-free conditional diffusion image generation method for multi-object scenes according to claim 1, characterized in that, During cyclic feature injection, the step of back-injecting object-level features from the expanded view branch to the global branch includes: At the output of each network layer in the global branch, the extended vision branch features are zero-padding the spatial boundaries to match the spatial resolution of the global branch, and then fused with the global branch features to output the updated global branch features.
6. The training-free conditional diffusion image generation method for multi-object scenes according to claim 1, characterized in that, It also includes an early stopping step for object-level feature injection: At each denoising time step, object-level features of the same object region are extracted from the global branch and the extended field of view branch respectively. After high-frequency information is extracted by a high-pass filter, the similarity is calculated. When the similarity exceeds a preset threshold, the feature injection of the object in subsequent time steps is terminated.
7. The training-free conditional diffusion image generation method for multi-object scenes according to claim 1, characterized in that, The global branch, object local branch, and extended vision branch share the same pre-trained conditional diffusion model backbone parameters and are not updated during the denoising iteration process.
8. A training-free conditional diffusion image generation system for multi-object scenes, implemented using the method described in any one of claims 1 to 7, characterized in that, include: The input parsing module is used to extract global text semantic features, object-level text semantic features, and global structure graph. It performs local cropping and extended cropping on the global structure graph to obtain the object local structure graph and the extended field structure graph. The multi-branch building module is used to build a conditional diffusion model that includes a global branch, an object-local branch, and an extended vision branch. Each branch is guided by a global structure graph and global text semantic features, an object-local structure graph and object-level text semantic features, and an extended vision structure graph and object-level text semantic features, respectively. The cross-scale semantic alignment module is used to inject fine-grained semantic features of the local branches of an object into the extended vision branch and achieve spatial coordinate alignment during the denoising iteration of the conditional diffusion model. The recurrent feature injection module is used to inject the background features of the global branch into the non-object region of the extended vision branch during the denoising iteration of the conditional diffusion model, and to inject the object-level features of the extended vision branch back into the global branch. The image decoding output module is used to decode the final global branch features of the iterative output to generate the final image.
9. The training-free conditional diffusion image generation system for multi-object scenarios according to claim 8, characterized in that, It also includes an object-level adaptive early stopping module, which is used to extract object-level features of the same object region from the global branch and the extended vision branch at each denoising time step. After extracting high-frequency information through a high-pass filter, the similarity is calculated. When the similarity exceeds a preset threshold, the feature injection of the object in subsequent time steps is terminated.
10. An electronic device comprising a memory and one or more processors, the memory being used to store a computer program, characterized in that, The processor is used to implement the training-free conditional diffusion image generation method for multi-object scenes as described in any one of claims 1 to 7 when executing a computer program.