Fine-grained occlusion control image generation method based on region awareness
By using a support region-aware approach to estimate and refine the object's support region, and combining this with transmittance calculation, the problem of inaccurate occlusion relationships in complex multi-object scenes is solved, achieving more accurate occlusion control and fine-grained structure preservation.
Patent Information
- Application Number
- CN202611601800.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-16
- Publication Date
- 2026-11-17
AI Technical Summary
Existing image generation methods struggle to accurately estimate the supporting regions of objects in complex multi-object scenes, leading to inaccurate occlusion relationships. This is especially true for foreground objects with holes, sparse, elongated, or continuous thin structures, which can easily cause excessive occlusion of objects behind them, loss of internal gaps, and structural distortion. Furthermore, there is a lack of effective evaluation methods.
By obtaining the text description and spatial bounding box of the object, the occlusion relationship is determined. The segmentation path is constructed using the object-level attention response and bounding box, and the support region is estimated and refined. Combined with the transmittance calculation, the visibility of the object is dynamically adjusted to generate a fine-grained occlusion control image.
It improves the accuracy of occlusion control, maintains the integrity and fine-grained structure of objects, provides test scenarios and evaluation methods for multiple objects and multiple layers of occlusion, and improves the comprehensive evaluation of occlusion generation effects.
Smart Images

Figure CN123118831A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and computer vision technology, specifically to a method for generating fine-grained occlusion-controlled images based on region awareness. Background Technology
[0002] In recent years, diffusion models have made significant progress in text-to-image generation, capable of generating images with high visual quality and semantic consistency based on text descriptions. To further improve the controllability of the generation process, existing techniques typically utilize spatial constraint information such as bounding boxes and layout maps to control the position and extent of different objects in the generated image. These methods can improve object localization and overall layout to some extent, but they mainly focus on "where the object is located," lacking fine-grained modeling of visibility relationships among multiple objects, such as "who occludes whom" and "how much is occluded."
[0003] In complex multi-object scenes, relying solely on textual descriptions or coarse-grained spatial conditions is often insufficient to reliably represent the hierarchical and local occlusion relationships between objects. When multiple objects overlap, the generated results are prone to issues such as incorrect processing order, excessive occlusion of objects behind them, object boundary adhesion, and loss of fine-grained structure. This is especially true for foreground objects with holes, sparse or locally discontinuous structures, such as fences, meshes, wire meshes, spider webs, beaded curtains, leaves, and lace. The actual occlusion area differs significantly from the rectangular area covered by the bounding box. Directly using the bounding box or coarse-grained attention response as the object's support region can incorrectly identify holes or gaps that should remain visible as foreground occlusion areas.
[0004] Existing latent space rendering methods combine latent features of different objects according to a preset occlusion order and use transmittance to describe the influence of foreground objects on the visibility of background objects, thus achieving multi-object occlusion control without additional training. However, the transmittance calculation of such methods depends on the object's support region. When the support region is approximated by a bounding box or coarse-grained attention map, the spatial extent of the foreground object is easily overestimated. Since the transmittance of background objects is directly affected by the support regions of the objects in front of them, the support region estimation error will further propagate into the transmittance calculation, causing background objects to be incorrectly suppressed in places where they should be visible, resulting in over-occlusion, thickening of fine structures, disappearance of internal holes, and structural distortion.
[0005] On the other hand, directly estimating the object's support region in the final occlusion state also presents challenges. Foreground objects can occlude parts of the structure of objects behind them, making it difficult to reliably recover the complete spatial extent and local structural cues of each object from a single generation result. Therefore, how to obtain a more accurate object support region while maintaining a given occlusion order, and how to enable this support region to dynamically participate in the transmissivity and visibility calculations during the diffusion generation process, are key issues in improving the accuracy of fine-grained occlusion control in complex scenes.
[0006] Furthermore, existing image generation evaluation methods primarily focus on semantic consistency between text and image, spatial relationships between objects, or relatively simple foreground-background depth relationships, lacking targeted evaluation for local visibility errors in complex multi-object occlusion scenarios. Especially when foreground objects with holes, sparse, elongated, or sheet-like structures participate in occlusion, relying solely on semantic similarity or overall spatial relationships is insufficient to accurately reflect whether background objects are incorrectly occluded, whether objects exist completely, whether the occlusion contact area is reasonable, and whether fine-grained structures are preserved. Simultaneously, existing test scenarios for occlusion control still have shortcomings in terms of the number of objects, occlusion levels, and coverage of fine-grained foreground structures, making it difficult to comprehensively evaluate the actual performance of complex occlusion control methods.
[0007] Therefore, it is necessary to propose a novel image generation method that can finely estimate the supporting regions of objects in complex multi-object scenes without retraining the diffusion model, and reintroduce the obtained supporting regions into the latent space rendering process to reduce the transmittance deviation caused by coarse-grained spatial constraints, thereby achieving more accurate occlusion order, local visibility, and fine-grained structure preservation. Simultaneously, it is also necessary to establish test scenarios and evaluation methods that can cover multiple objects, multiple layers of occlusion, and fine-grained foreground structures, in order to comprehensively evaluate the complex occlusion generation effect from aspects such as occlusion correctness, object integrity, contact rationality, and structure preservation. Summary of the Invention
[0008] To overcome the aforementioned technical problems, the present invention aims to provide a fine-grained occlusion control image generation method based on support region awareness, which solves the problem of inaccurate occlusion relationships caused by overly coarse estimation of object support regions in complex multi-object scenes in existing controllable image generation methods. In particular, it solves the problem that when foreground objects with holes, sparse, slender or continuous thin structures exist, it is easy to cause excessive occlusion of background objects, disappearance of internal gaps and structural distortion.
[0009] The objective of this invention can be achieved through the following technical solutions: Specifically, it provides a fine-grained occlusion control image generation method based on region awareness, including the following steps: S1: Obtain the text descriptions, spatial bounding boxes, and occlusion relationships between multiple objects, and determine the processing order of each object from front to back based on the occlusion relationships; S2: Using the spatial bounding box of each object as the initial support region, perform the first latent space rendering under the condition of reducing the occlusion intensity of the foreground object to obtain the intermediate sketch and the corresponding object-level attention response; S3: Based on the intermediate sketch, construct two segmentation paths using object-level attention response and spatial bounding box to obtain candidate support regions. Select the one with the larger score of the scoring function as the initial support region. The scoring function is based on at least segmentation confidence, attention consistency and geometric rationality. S4: Based on the object's preset structure type, the initial support region is refined using structure-aware processing and sequential consistency processing based on the processing order to obtain the refined support region; S5: Perform a second latent space rendering on the refined support region, and dynamically merge it with the current object's attention during the diffusion denoising step to obtain the dynamic support region; S6: Based on the dynamic support region, recalculate the transmittance of each object according to the processing order; S7: Based on the dynamic support region, transmittance, and processing order, the latent features of each object are weighted and combined, and the final image is output after diffusion denoising.
[0010] As a further aspect of the present invention: In step S2, reducing the occlusion intensity of the foreground object involves multiplying the final target opacity of each object by an opacity scaling factor less than 1 to obtain the actual opacity used in the sketching stage. ; in, The opacity actually used in the sketching stage. This is the opacity scaling factor. , The goal is opaque.
[0011] As a further aspect of the present invention: In step S3, a segmentation cue for the current object is constructed based on the object-level attention response and input into the segmentation model to obtain the first candidate support region. The bounding box guidance path specifically involves using the spatial bounding box of the object as a segmentation cue to segment the corresponding object in the intermediate sketch, thereby obtaining a second candidate support region. When processing the current object, the support region of the identified foreground object is used as the avoidance region to prevent the segmentation model from incorrectly classifying the regions of other objects into the current object.
[0012] As a further aspect of the present invention: in step S3, the scoring function The calculation formula is: ; in, This represents the mask confidence score output by the segmentation model. This indicates the consistency between the candidate support region and the object's attention response; , , , and These are used to reflect the geometric rationality constraints of positive prompts, negative prompts, area size, avoidance areas, and severe area overlap, respectively. , , , , , = (1.0, 0.20, 0.10, 0.70, 0.20, 0.90).
[0013] As a further aspect of the present invention: In step S4, for objects with holes, meshes, or locally sparse features, the structure-aware refinement specifically includes: The Canny edge detection method extracts local fine structures, performs dilation and closing operations on the edges at a first preset scale to connect local broken regions, deletes isolated connected regions with an area smaller than a preset area threshold, and combines object attention response for gating to remove regions that are semantically inconsistent with the current object. Finally, the boundaries of the supporting regions are feathered.
[0014] As a further aspect of the present invention: the low and high thresholds of the Canny edge detection are set to 50 and 130, respectively; the dilation kernel size is 2; the closing kernel size is 3; isolated connected components with an area less than 80 pixels are deleted; the object-level attention response threshold is set to 0.22; and the boundary feathering kernel size is set to 7. For lace objects, the low and high thresholds of the Canny edge detection are set to 18 and 95, respectively; the dilation kernel size is 2; the closing kernel size is 3; the minimum area of the retained connected components is 30 pixels; the object-level attention response threshold is set to 0.18; the feathering kernel size is 13; and the Gaussian feathering standard deviation is set to 3.0.
[0015] As a further aspect of the present invention: In step S4, after the structural awareness refinement of a single object is completed, sequential consistency processing is also performed according to the processing order obtained in step S1: when processing objects located behind, the regions already allocated to objects in front are excluded from the support regions of objects behind, so as to reduce the conflict of support regions between different occlusion layers.
[0016] As a further aspect of the present invention: the dynamic fusion in step S5 specifically refers to: for each diffusion step The refined support area obtained in the first phase Attention to the object after shaping in the current step To merge: ; in, For the reliable geometric support area recovered in the first phase, For the object attention after the current diffusion step, The fusion weights between the two in the current step, It is element-wise multiplication. For diffusion steps The dynamic support area that actually participates in rendering.
[0017] As a further aspect of the present invention: In step S5, before fusion, the object attention in the current step is further subjected to type-aware shaping, and the shaping formula is: ; in, Adjusting parameters for attention power order The parameter representing the minimum response level of attention. For the attention of the object after plastic surgery, Attention to the object in the current step.
[0018] As a further aspect of the present invention: in step S6, the transmittance is calculated according to the following formula: ; in: For the object The object in front In the dynamic support region of the current diffusion step; For object Rendering density; For object The cumulative transmittance corresponding to the current step; The weighted combination of latent features is performed according to the following formula: ; Among them, the normalization term for , For object In the current diffusion steps Dynamic support area, For object Rendering density, For latent features of objects in the diffusion layer, For the spread of the first The latent features of objects in the layer, where N is the number of objects; in each subsequent diffusion step, steps S5 to S7 are repeated until all denoising steps are completed, and finally the latent representation is decoded to obtain the generated image, which is the final image.
[0019] The beneficial effects of this invention are: 1. This invention combines object support region estimation with transmittance calculation: In response to the problem that coarse-grained support regions in existing latent space rendering methods can easily cause transmittance estimation errors, the object support region is first estimated and refined, and then reintroduced into the transmittance calculation process, so that the object visibility can be adjusted according to the actual spatial structure.
[0020] 2. This invention uses intermediate sketches with reduced occlusion intensity to recover object structural information: before the support region is estimated, the foreground occlusion intensity is temporarily reduced to weaken the excessive suppression of the rear object by the coarse-grained foreground support region, so that the occluded object and the fine-grained foreground structure can obtain more sufficient visibility and spatial structural information.
[0021] 3. This invention employs a dual-path support region estimation and structure-aware refinement mechanism: combining semantic localization information provided by object-level attention response and geometric prior information provided by bounding boxes, candidate support regions are obtained through attention-guided paths and bounding box-guided paths, and more reliable object support regions are selected. Based on this, the support regions are refined according to the characteristics of different fine-grained foreground structures: for fine-grained structures with porous features, the focus is on preserving their internal gaps and local sparsity; for slender, continuous, or sheet-like structures, the focus is on maintaining the connectivity of the support regions to reduce the risk of breakage in fine structures. Attached Figure Description
[0022] The invention will now be further described with reference to the accompanying drawings.
[0023] Figure 1 This is the overall algorithm flowchart of the fine-grained occlusion control image generation method based on region awareness of the present invention; Figure 2 This is a schematic diagram of the two-stage architecture of the fine-grained occlusion control image generation method based on region awareness of the present invention. Figure 3 This is a schematic diagram of dual-path support region estimation for the fine-grained occlusion control image generation method based on support region awareness of the present invention. Figure 4 This is a schematic diagram of the dynamic fusion mechanism of the fine-grained occlusion control image generation method based on region awareness of the present invention. Detailed Implementation
[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0025] Example 1 See Figures 1-4 A method for generating fine-grained occlusion-controlled images based on region awareness is disclosed, including the following steps: S1: Scene standardization; First, read the object information from the scene to be generated.
[0026] Assume there are a total of An object, whose collection is represented as: ; Each object corresponds to at least an object text description and a spatial bounding box; the scene also includes occlusion relationships between objects, such as: "the barbed wire is in front of the character", "the character is in front of the background".
[0027] Convert these explicit occlusion pairs or the context relationships within the text into a directed occlusion graph: ; In this context, directed edges represent the front-to-back occlusion relationship between two objects.
[0028] Then the occlusion map A topological sort is performed to obtain the processing order of objects from front to back. Subsequent support region processing, transmittance calculation, and latent feature combination are all performed in this order.
[0029] When the input is only a natural language description, a large language model can be used to parse the original text into object descriptions, object bounding boxes, and occlusion relationships between objects, and then the above normalization process can be performed.
[0030] S2: Generate an intermediate sketch with low occlusion intensity using a coarse support region; After obtaining the object order, instead of directly generating the final result, the first latent space rendering is performed to obtain an intermediate sketch for reconstructing the object structure.
[0031] At this point, the region formed by the spatial bounding boxes of each object is used as the initial coarse support region.
[0032] For objects Let the target opacity corresponding to the final generation stage be . The sketch stage will temporarily reduce it to: ; in, The final target opacity of the object; This is the opacity scaling factor during the sketching stage; The opacity used in the actual sketching stage.
[0033] In a default implementation, for foreground objects with holes or locally sparse features, the opacity scaling factor during the sketching stage... It is set to 0.45, and this scaling is only used for the intermediate sketch generation in the first stage.
[0034] because Therefore, the occlusion effect of foreground objects in the sketching stage is weaker than that in the final generation stage.
[0035] For example, when a fence is in front of a character, if the entire fence boundary is heavily occluded, the character may disappear in large quantities; by reducing the fence opacity, the character and the fence's own mesh, lines, and other structures can retain more visual information in the intermediate sketch.
[0036] Therefore, the sketch generated in this step is not the final output image, but is specifically used for the next step of restoring the actual supporting area of the object.
[0037] Simultaneously, the object-level attention response corresponding to each object during the sketch generation process is saved for subsequent segmentation.
[0038] S3: Dual-path objects support region estimation; For each object, segmentation cues are constructed by combining the geometric priors provided by its bounding box and the semantic localization information provided by the object-level attention response, and two complementary support region estimation paths are formed: the attention-guided path and the bounding box-guided path.
[0039] The first category is object bounding boxes. Bounding boxes can determine the approximate location of an object, thus providing coarse-grained geometric constraints. The second category is object-level attention responses. Attention responses reflect the position in the currently generated result that corresponds to the semantic meaning of the object's text, thus providing object-level semantic localization information.
[0040] Based on the two types of information mentioned above, two segmentation paths are established respectively: The first path is the attention-guided path. It constructs segmentation cues for the current object based on the object-level attention response and applies them to the segmentation model, allowing the segmentation results to utilize the semantic response positions of the object more effectively, thereby recovering fine-grained regions of the object. First candidate support region: .
[0041] The second path is a bounding box-guided path. Using the geometric extent provided by the object's bounding box as a segmentation cue, the corresponding objects in the intermediate sketch are segmented to obtain a relatively conservative object region. Second candidate support region: .
[0042] In one specific implementation, object support region extraction employs SAMViT-H.
[0043] In addition, when processing a particular object, the following are also used: negative cues generated by other objects; and an avoidance mask formed by the already identified foreground object region. Their purpose is to prevent the segmentation model from incorrectly classifying regions of other objects into the current object.
[0044] For example, in a scenario where a fence obscures a person, when extracting the person, it is necessary to avoid areas that already belong to the fence from re-entering the person's supporting area, thereby reducing confusion between the supporting areas of objects.
[0045] S4: Candidate results support regional scoring and selection; The two paths yield the following results: , ; The candidate support regions obtained from the two paths were then evaluated separately. For the object Candidate support areas A deterministic masked candidate scoring function is used:
[0046] in, This represents the mask confidence score output by the segmentation model. This indicates the consistency between the candidate support region and the object's attention response; , , , and These are used to reflect geometric rationality constraints such as positive prompts, negative prompts, area size, avoidance areas, and severe area overlap.
[0047] In one implementation, the fixed weights for each item are set as follows: ( , , , , , = (1.0, 0.20, 0.10, 0.70, 0.20, 0.90) The threshold for determining severe overlap is set to 0.30.
[0048] The above scores are calculated for the two candidate support regions obtained from the attention-guided path and the bounding box-guided path, respectively, and the candidate with the larger score is selected: ; ; Thus, the object is obtained. The initial support area.
[0049] S5: Further refine the support region based on the object structure: After obtaining the initial support area, the foreground objects that need fine-grained processing are divided into different structure types according to the pre-given object categories or structure types in the input scene.
[0050] For objects with holes, meshes, or local sparse features, further refinement is achieved by utilizing image edges, object attention, and connected region information based on the initial support region. Specifically, local fine structures can first be extracted using Canny edge detection, and then edges are dilated and closed using a first preset scale to connect local broken regions. Subsequently, isolated connected regions with excessively small areas are deleted, and gating is performed using the object attention response to remove regions that are semantically inconsistent with the current object. Finally, the boundaries of the support region are feathered to reduce artifacts caused by hard boundaries while maintaining internal holes and local sparsity.
[0051] In one implementation of the chain network object, the Canny low and high thresholds are set to 50 and 130, respectively, the dilation kernel size is 2, the closing kernel size is 3, isolated connected regions with an area less than 80 pixels are deleted, the object-level attention response threshold is set to 0.22, and the boundary feathering kernel size is set to 7.
[0052] For continuous thin structures, continuous coverage of the supporting region is maintained through a first preset scale dilation, closing operation, isolated region filtering, and boundary feathering, reducing the probability of the fine structure being segmented into multiple isolated regions. In one implementation of a lace object, the Canny low and high thresholds are set to 18 and 95, respectively, the dilation kernel size is 2, the closing operation kernel size is 3, the minimum preserved connected region area is 30 pixels, the object-level attention response threshold is set to 0.18, the feathering kernel size is 13, and the Gaussian feathering standard deviation is set to 3.0.
[0053] The above parameters can be adjusted according to the object's structure category. The purpose is not to simply expand the support area, but to avoid obvious breaks in the continuous fine structure while preserving the holes and local sparsity.
[0054] After refining the structure of a single object, sequential consistency processing is performed according to the processing order obtained in step S1: when processing objects located later, the regions already allocated to objects in front are excluded from the supporting regions of objects in back.
[0055] For example, the order of objects is: Then process It is necessary to consider that it already belongs to Area; processing This requires considering the foreground region that has already been determined. This operation reduces support region conflicts between different occlusion layers and ensures that the support region results are consistent with the global occlusion order.
[0056] The first stage concludes with the final refined support regions for each object: .
[0057] S6: Re-inject the refined support region into the diffusion generation process; The first phase of the acquisition Re-inject the second latent space rendering.
[0058] Assuming in the diffusion denoising step In the middle, object The current object attention is: .
[0059] Since different fine-grained structures have different requirements for attention response, the object attention is first subjected to type-aware shaping: ; in, Adjusting parameters for attention power order The parameter representing the minimum response level of attention. Attention to the reshaped object.
[0060] The parameters are set according to the fine-grained structure type of the object. In a default implementation, for porous-thin objects with porous and locally sparse features, the attention power-law adjustment parameter and the minimum response level parameter are set as follows: ; For elongated, continuous, or sheet-like thin objects, set them as follows: ; When different types of fine-grained foreground objects exist simultaneously in the same scene, each object adopts the corresponding parameter configuration according to its own structure type to meet the requirements of hole preservation and continuous support respectively.
[0061] Different parameter settings are used for different structural types to suppress unreliable weak attention responses while retaining the spatial response required for different fine-grained structures.
[0062] The purpose of this step is to avoid relying solely on the static support region obtained in the first stage, and also to avoid relying entirely on the attention that may change during the diffusion process, but rather to prepare for combining the two in the next step.
[0063] S7: Dynamic fusion refinement supports region and object attention; For each diffusion step The refined support area obtained in the first phase Attention after the current step of the reshaping To merge: ; in, For the reliable geometric support area recovered in the first phase, For the semantic attention of the current diffusion step, The fusion weights between the two in the current step, It is element-wise multiplication. For diffusion steps The dynamic support area that actually participates in rendering.
[0064] Fusion weights A deterministic linear scheduling method that varies with the number of denoising steps is adopted. Let the total number of denoising steps be... The current noise reduction step is as follows: ,but: ; In a default implementation, , , Therefore, in the early stages of denoising, the dynamic support region is more constrained by the first-stage refined support region; as the denoising steps progress, the weight of the support region gradually decreases, and the role of object attention increases accordingly, enabling the model to gradually restore local structure and semantic details while maintaining stable occlusion relationships.
[0065] therefore, It is not completely fixed from beginning to end, but changes dynamically with the diffusion process.
[0066] S8: Recalculate transmittance using dynamic support regions; After obtaining the dynamic support region for the current step, recalculate the object transmittance according to the processing order.
[0067] For objects Its transmittance is calculated as follows: .
[0068] in: For the object The object in front In the dynamic support region of the current diffusion step; For object Rendering density; For object The cumulative transmittance corresponding to the current step.
[0069] The practical meaning is: only locations that truly belong to the support area of the object in front should have their visibility of objects behind significantly reduced. For example, for holes in a fence, because the refined fence support area has a lower value at the hole location or is not part of the effective fence area, these locations will not continuously reduce the transmittance of the person behind as when using the entire bounding box, and the person can be displayed normally through the fence holes.
[0070] This is the core process by which the present invention solves the problem of "excessive occlusion of fine-grained foreground".
[0071] S9: Complete the latent feature combination based on the updated transmittance; Latent features of objects in the diffusion layer The weighted combination is based on the order of the objects, the dynamic support area, and the recalculated transmittance.
[0072] The latent feature combination form is: ; The normalization term for: .
[0073] Therefore, the final contribution of each object to the latent features is simultaneously controlled by the current object's supporting region, the cumulative transmittance caused by the foreground object, and the object's own rendering density.
[0074] An object in front occludes an object behind it according to its actual supporting area, while its holes, gaps, or unoccupied positions can preserve the latent features of the object behind it.
[0075] S10: Repeat the denoising process and output the final image; Repeat the following steps in subsequent diffusion processes: Step 1: Obtain the attention of the current object; Step Two: Attention Reduction; Step 3: Integration with refined support areas; Step 4: Update the dynamically supported regions; Step 5: Recalculate the transmittance; Step 6: Combine latent features of the object; Continue until all diffusion denoising steps are completed.
[0076] Finally, the latent representation is decoded to obtain a generated image that is semantically consistent with the input text, satisfies the preset processing order, and preserves fine-grained structure.
[0077] To reduce the randomness of diffusion sampling, multiple candidate results can be generated under the same scenario conditions during inference, and the final selection result can be retained from them. This process does not require additional model training.
[0078] In one implementation, the present invention is based on the LaRender latent space rendering framework of SDXL, and uses IterComp pre-trained weights as the default generation backbone; the object opacity is converted to rendering density and an inverse density scheduling is adopted; the diffusion sampling adopts the DPM-Solver multi-step scheduler, the number of denoising steps is 25, the guiding scale is 7.0, and the generation resolution is 1024×1024; the object support region extraction adopts SAMViT-H, combined with a point cueing method to avoid region awareness.
[0079] To verify the technical effect of the present invention, experiments were conducted on the constructed CompOcc complex occlusion benchmark. The benchmark contains a total of 480 complex occlusion scenes, each containing 2 to 5 objects, specifically including 132 two-object scenes, 156 three-object scenes, 174 four-object scenes, and 18 five-object scenes, and covers sparse hole structures, continuous thin structures, and mixed fine-grained structures.
[0080] In the overall quantitative evaluation, the Scalar, OccTerm, QualTerm, PairScore, PresenceScore, StructScore, ContactScore, and CLIP scores of the method of this invention are 0.874, 0.636, 1.112, 0.616, 0.020, 0.135, 1.590, and 0.324, respectively. For direct comparison, the corresponding indices for LaRender+IterComp are 0.981, 0.746, 1.215, 0.691, 0.056, 0.149, 1.734, and 0.278, respectively. Except for CLIP, lower scores for all other indices are better. Experimental results show that this invention improves overall occlusion control, object front-and-back relationships, object existence, contact rationality, and text semantic consistency.
[0081] As scene complexity increases, the advantages of this invention become more apparent. In a scene containing four objects, the Scalar is reduced from 0.960 (LaRender+IterComp) to 0.828, a reduction of approximately 13.8%. In a mixed-thin scene with complex fine-grained structures, the Scalar is reduced from 0.942 to 0.834, demonstrating that this invention can effectively reduce visibility errors caused by multi-layered occlusion chains and complex fine-grained foreground structures.
[0082] Ablation experiments further showed that the final Scalar result of the complete method was 0.679; after removing the support region-guided transmittance refinement, it increased to 0.983, and after removing the attention-guided support region estimation, it increased to 0.924. This indicates that support region estimation, support region reinjection, and transmittance refinement together constitute a complete technical process, rather than a simple combination of several independent steps.
[0083] In the qualitative results, compared with methods using coarse-grained support regions, this invention can more stably preserve holes, fine lines, and locally visible areas in objects such as fences, grids, spider webs, and beaded curtains, reducing phenomena such as excessive occlusion of objects behind, abnormal adhesion between objects in front and behind, and thickening of fine structures. User research further evaluated 80 participants and 20 sets of images. The invention achieved a comprehensive average ranking of 1.344 across six indicators: occlusion order, object integrity, contact naturalness, fine structure preservation, text consistency, and overall quality. The average Spearman correlation coefficient between the automatic indicators and manual ranking reached 0.917.
[0084] The foregoing has provided a detailed description of one embodiment of the present invention, but this description is merely a preferred embodiment and should not be construed as limiting the scope of the invention. All equivalent variations and modifications made within the scope of the claims of this invention should still fall within the patent coverage of this invention.
Claims
1. A method for generating images with fine-grained occlusion control based on region awareness, characterized in that: Includes the following steps: S1: Obtain the text descriptions, spatial bounding boxes, and occlusion relationships between multiple objects, and determine the processing order of each object from front to back based on the occlusion relationships; S2: Using the spatial bounding box of each object as the initial support region, perform the first latent space rendering under the condition of reducing the occlusion intensity of the foreground object to obtain the intermediate sketch and the corresponding object-level attention response; S3: Based on the intermediate sketch, construct two segmentation paths using object-level attention response and spatial bounding box to obtain candidate support regions. Select the one with the larger score of the scoring function as the initial support region. The scoring function is based on at least segmentation confidence, attention consistency and geometric rationality. S4: Based on the object's preset structure type, the initial support region is refined using structure-aware processing and sequential consistency processing based on the processing order to obtain the refined support region; S5: Perform a second latent space rendering on the refined support region, and dynamically merge it with the current object's attention during the diffusion denoising step to obtain the dynamic support region; S6: Based on the dynamic support region, recalculate the transmittance of each object according to the processing order; S7: Based on the dynamic support region, transmittance, and processing order, the latent features of each object are weighted and combined, and the final image is output after diffusion denoising.
2. The fine-grained occlusion control image generation method based on region-aware support according to claim 1, characterized in that, In step S2, reducing the occlusion intensity of the foreground object involves multiplying the final target opacity of each object by an opacity scaling factor less than 1 to obtain the actual opacity used in the sketching stage. in, The opacity actually used in the sketching stage. This is the opacity scaling factor. , The target opacity is represented by the subscript i, which indicates the i-th object.
3. The fine-grained occlusion control image generation method based on region awareness according to claim 1, characterized in that, In step S3, a segmentation cue for the current object is constructed based on the object-level attention response and input into the segmentation model to obtain the first candidate support region. The bounding box guidance path specifically involves using the spatial bounding box of the object as a segmentation cue to segment the corresponding object in the intermediate sketch, thereby obtaining a second candidate support region. .
4. The fine-grained occlusion control image generation method based on region awareness according to claim 1, characterized in that, In step S3, the scoring function The calculation formula is: in, This represents the mask confidence score output by the segmentation model. This indicates the consistency between the candidate support region and the object's attention response; , , , and These are used to reflect the geometric rationality constraints of positive prompts, negative prompts, area size, avoidance areas, and severe area overlap, respectively. , , , , , These are the corresponding weights.
5. The fine-grained occlusion control image generation method based on region awareness according to claim 1, characterized in that, In step S4, for objects with holes, meshes, or locally sparse features, the structure-aware refinement specifically includes: The Canny edge detection method extracts local fine structures, performs dilation and closing operations on the edges at a first preset scale to connect local broken regions, deletes isolated connected regions with an area smaller than a preset area threshold, and combines object attention response for gating to remove regions that are semantically inconsistent with the current object. Finally, the boundaries of the supporting regions are feathered.
6. The fine-grained occlusion control image generation method based on region awareness according to claim 5, characterized in that, The Canny edge detection low and high thresholds are set to 50 and 130, respectively. The dilation kernel size is 2, the closing kernel size is 3, isolated connected components with an area less than 80 pixels are deleted, the object-level attention response threshold is set to 0.22, and the boundary feathering kernel size is set to 7. For lace objects, the Canny edge detection low and high thresholds are set to 18 and 95, respectively. The dilation kernel size is 2, the closing kernel size is 3, the minimum area of the retained connected components is 30 pixels, the object-level attention response threshold is set to 0.18, the feathering kernel size is 13, and the Gaussian feathering standard deviation is set to 3.
0.
7. The fine-grained occlusion control image generation method based on region awareness according to claim 1, characterized in that, In step S4, after the structure-aware refinement of a single object is completed, sequential consistency processing is performed according to the processing order obtained in step S1: when processing objects located behind, the regions already assigned to objects in front are excluded from the support regions of objects behind, in order to reduce support region conflicts between different occlusion layers.
8. The fine-grained occlusion control image generation method based on region awareness according to claim 1, characterized in that, The dynamic fusion described in step S5 specifically refers to: for each diffusion step The refined support area obtained in the first phase Attention to the object after shaping in the current step To merge: in, This represents the reliable geometric support region recovered in the first phase. For the object attention after the current diffusion step, The fusion weight between the two in the current step. It is element-wise multiplication. For diffusion steps The dynamic support region that actually participates in rendering, with the subscript i representing the i-th object.
9. The fine-grained occlusion control image generation method based on region awareness according to claim 8, characterized in that, In step S5, before fusion, type-aware shaping is performed on the object attention of the current step. The shaping formula is as follows: in, The parameter is adjusted to the power of attention. The parameter representing the minimum level of attention response. Attention to the object after reshaping in the current step. Attention to the object in the current step.
10. The fine-grained occlusion control image generation method based on region awareness according to claim 1, characterized in that, In step S6, the transmittance is calculated according to the following formula: in, For the object The object in front In the current diffusion steps Dynamic support area; For object Rendering density; For object In the current diffusion steps The corresponding cumulative transmittance; In step S7, the weighted combination of latent features is performed according to the following formula: Among them, the normalization term for , For object In the current diffusion steps Dynamic support area, For object Rendering density, For latent features of objects in the diffusion layer, For the spread of the first Latent features of objects in a layer, where N is the number of objects; In subsequent diffusion steps, steps S5 to S7 are repeated until all denoising steps are completed, and finally the latent representation is decoded to obtain the generated image.
