A diffusion model-based maskless image synthesis method and system

Abstract

The application belongs to the technical field of image processing, and discloses a diffusion model-based unmasked image synthesis method and system, which comprises the following steps: inputting a background image and a noise image containing a target object into a pre-trained latent diffusion model; the latent diffusion model performs a reverse denoising process under the guidance of a condition vector to adaptively synthesize the target object in the background image, thereby obtaining a synthesized image; wherein the condition vector is obtained in the following manner: encoding a reference image containing the target object into an object embedding vector, and generating a set of learnable background cues according to the object embedding vector; concatenating the object embedding vector and the background cues; and performing spatial mapping on the concatenated feature representation to obtain the condition vector. The application can overcome the defects of the existing image synthesis technology, such as complicated process, low efficiency, the need for a large amount of manual intervention, the lack of realism and harmony in the synthesis result, and the inability of the foreground object to adaptively adjust according to the background environment.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and more specifically, relates to a maskless image synthesis method and system based on a diffusion model. Background Technology

[0002] Image compositing is a fundamental task in computer vision and image editing. Its goal is to extract one or more foreground objects from a source image and naturally integrate them into a target background image to create a visually realistic and harmonious new image. This technology has wide and significant applications in numerous fields, including film special effects, advertising production, virtual reality, augmented reality, e-commerce product displays, and creative design.

[0003] However, traditional image compositing methods typically rely on a series of complex and fragmented manual or semi-automatic editing steps. These steps usually include, but are not limited to: precise segmentation and extraction of foreground objects (e.g., obtaining alpha matte through matting techniques), spatial placement and geometric transformation of objects in the background image (e.g., scaling, rotation, perspective correction), image harmonization (e.g., adjusting the color distribution, light intensity and direction, shadows, hue, saturation, style, etc. of foreground objects to match the background environment), shadow generation and rendering when necessary, and edge smoothing and blending.

[0004] This multi-step, fragmented processing workflow has significant limitations: First, achieving high-quality image compositing requires executing multiple independent editing tasks sequentially, a time-consuming and labor-intensive process that demands high levels of professional skill and artistic competence from the operator, making rapid, batch processing difficult. Second, because each editing step is typically optimized independently or based on simplified assumptions, it's difficult to fully consider the complex interactions between foreground and background across multiple dimensions, including lighting conditions, global illumination transmission, perspective relationships, depth of field effects, and semantic environment. This often results in composite images lacking sufficient realism and overall harmony, easily exhibiting visual inconsistencies such as lighting mismatches, color discord, missing or incorrect shadows, harsh edges, and objects appearing "floating" or "pasted." Third, traditional methods typically treat the foreground object to be composited as a relatively fixed visual entity, primarily adjusting external attributes (such as color and brightness) to match the background. They generally cannot intelligently and adaptively adjust the foreground object's intrinsic attributes (such as pose, shape, and partial lighting effects) based on the specific environment of the target background (such as specific lighting directions, occlusion relationships of surrounding objects, the overall atmosphere of the scene, or semantic logic). For example, when composited from a side-view car reference image onto a background requiring a frontal view of the car, traditional methods cannot automatically adjust the car's pose. Finally, many methods require precise foreground object masks (alpha matte) or other auxiliary information (such as precise placement) as input, which further increases the complexity of the operation and the requirements for preliminary data preparation.

[0005] In recent years, significant progress has been made in image generation and editing with the development of deep learning technologies, particularly Generative Adversarial Networks (GANs) and Generative Diffusion Models. Some methods have attempted to leverage deep learning for specific subtasks such as image harmonization and shadow generation. However, integrating these subtasks into an end-to-end, automated, and efficient synthesis workflow capable of adaptively adjusting foreground objects remains a significant challenge. While Generative Diffusion Models have demonstrated powerful capabilities in generating high-fidelity images and performing controlled editing, effectively applying them to build a unified image synthesis framework that intelligently understands context and enables foregrounds to adapt to background changes is a pressing issue that needs to be addressed.

[0006] Therefore, existing technologies still have significant shortcomings and room for improvement in achieving simple and unified processes, high levels of automation, realistic and natural composite effects, and intelligent adaptation of the foreground to the background environment. A new technical solution is urgently needed to overcome these challenges. Summary of the Invention

[0007] In view of the above-mentioned defects or improvement needs of existing technologies, the present invention provides a maskless image synthesis method and system based on a diffusion model. Its purpose is to overcome the defects of existing image synthesis technology, such as cumbersome process, low efficiency, need for a lot of manual intervention, lack of realism and harmony in the synthesis results, and especially the inability of foreground objects to adaptively adjust according to the background environment.

[0008] To achieve the above objectives, this invention provides a maskless image synthesis method based on a diffusion model, comprising:

[0009] Background image and noisy images containing target objects The input is fed into a pre-trained latent diffusion model, which performs an inverse denoising process guided by a conditional vector c to adaptively synthesize the target object into the background image. In the process, a composite image containing the target object is obtained; wherein the condition vector c is obtained in the following ways:

[0010] Reference image containing the target object Encoded as object embedding vector And according to the background image Generate a set of learnable background cue words; embed the object into a vector. The concatenated feature representation is obtained by concatenating the background cue words; the concatenated feature representation is then spatially mapped to obtain a condition vector c used to guide the reverse denoising process of the latent diffusion model.

[0011] Furthermore, the potential diffusion model includes an encoder, a denoising network, and a decoder;

[0012] The encoder is used to process the background image. and noisy images containing target objects Feature encoding is performed to obtain the latent representation of the background image. Potential representation of noisy images ;

[0013] The denoising network is obtained by modifying the number of input channels of the standard latent diffusion model denoising network, so that the modified denoising network can simultaneously receive the latent representation of the background image. Potential representation of noisy images Guided by the conditional vector c, a reverse denoising process is performed to obtain the latent representation of the synthesized image. ;

[0014] The decoder is used for the latent representation of the synthesized image. Decoding is performed to obtain the synthesized image.

[0015] Furthermore, the denoising network is a denoising network based on the U-Net structure; the latent diffusion model performs a reverse denoising process under the guidance of the conditional vector c, including:

[0016] The conditional vector c is used as the key and value of each cross-attention layer in the U-Net structure, and the spatial features corresponding to the U-Net structure are used as the query. The denoising process is guided by the cross-attention mechanism.

[0017] Furthermore, the CLIP image encoder is used to process the reference image containing the target object. Encoded as object embedding vector .

[0018] Furthermore, a spatial mapping module is used to spatially map the concatenated feature representation to obtain a conditional vector c used to guide the reverse denoising process of the latent diffusion model; wherein, the spatial mapping module includes at least one attention layer, or the spatial mapping module is a multilayer perceptron.

[0019] Furthermore, the dataset used for pre-training the potential diffusion model is constructed as follows:

[0020] For each original image in the dataset with a target object segmentation mask M Image restoration techniques are used to restore the original image. The region marked by the target object segmentation mask M is repaired to obtain the background image missing the target object. Based on the bounding box of the target object segmentation mask M, from the original image... The region containing the target object is cropped out and used as a reference image. ; the original image As training samples ( , The corresponding real image The training triplet was obtained. , , );

[0021] Original images in the dataset with target object segmentation mask M The corresponding training triplet ( , , The dataset used for pre-training the potential diffusion model constitutes the dataset used for this purpose.

[0022] Furthermore, prior to pre-training the potential diffusion model, the following steps are also included:

[0023] Each training sample in the dataset used for pre-training the potential diffusion model ( , Data augmentation can be performed using at least one of elastic deformation, random rotation, and Gaussian blur.

[0024] The present invention also provides a maskless image synthesis system based on a diffusion model, including a computer-readable storage medium and a processor;

[0025] The computer-readable storage medium is used to store executable instructions;

[0026] The processor is used to read executable instructions stored in the computer-readable storage medium and execute the maskless image synthesis method based on the diffusion model described above.

[0027] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the maskless image synthesis method based on the diffusion model as described in any of the preceding claims.

[0028] The present invention also provides a computer program product, including a computer program that, when run on a computer, causes the computer to execute the maskless image synthesis method based on the diffusion model described above.

[0029] In summary, the above-described technical solutions conceived in this invention can achieve the following beneficial effects:

[0030] (1) This invention provides a process-efficient image synthesis method based on a diffusion model, which can process the input background image end-to-end. and noisy images containing target objects And output the final synthesized image. Specifically, this invention combines cue words representing background context information generated from the target background image with object embeddings extracted from the reference foreground image, and then maps them into conditional vectors that can be effectively utilized by the diffusion model. This effectively injects background context information into the conditional guidance, precisely guiding the denoising process to generate content that conforms to the characteristics of the reference object and adapts to the background. In the reverse denoising process of the diffusion model, the background image is specifically introduced. This allows the model to directly utilize the underlying visual information of the background during the generation process. The precise guidance of the conditional vector and the synergistic effect of utilizing the underlying visual information of the background enable the model to adaptively adjust the pose, size, and appearance features of the foreground object based on the contextual information of the background image, allowing it to blend naturally with the background scene. This results in high-quality, highly realistic synthetic results and significantly improves process efficiency. The method of this invention requires no mask or other auxiliary information, reducing operational complexity and the requirements for preliminary data preparation. Therefore, this invention solves the shortcomings of existing image synthesis techniques, such as cumbersome processes, low efficiency, the need for extensive manual intervention, lack of realism and harmony in the synthesized results, and, in particular, the inability of foreground objects to adaptively adjust to the background environment.

[0031] (2) Furthermore, in order to enable the denoising network to directly utilize the underlying visual information of the background, the number of input channels of the standard latent diffusion model denoising network was modified in this invention. The modified denoising network can simultaneously receive the latent representation of the background image. Potential representation of noisy images This allows the model to perceive pixel-level / latent-level information of the background image at each step of denoising.

[0032] (3) Further, in the reverse denoising process of the diffusion model, the conditional vector c is used as the key and value of each cross-attention layer in the U-Net structure, and the spatial features corresponding to the U-Net structure are used as the query. The denoising process is guided by the cross-attention mechanism, which enables the model to simultaneously and adaptively focus on the background region (the contextual information of the background is injected into the conditional vector c), while the attention corresponding to the object embedding is focused on the foreground region where the object needs to be generated. Thus, the target object from the reference image is not only placed in the appropriate region of the background image, but its appearance (such as lighting, color, style) and even pose are adaptively adjusted and fused according to the environment of the background image.

[0033] (4) Preferably, a CLIP image encoder is used to encode the reference image containing the target object. Encoded as object embedding vector The feature space of the CLIP image encoder has good alignment with the text space, which is helpful for subsequent integration with the diffusion model.

[0034] (5) Furthermore, since it is very difficult and costly to obtain large-scale, high-quality (reference image, background image, real synthetic image) paired data, this invention automatically constructs a large number of (reference image, background image, real image) training triples using existing, easily accessible datasets with object segmentation masks, thus solving the challenge of lacking large-scale paired synthetic data.

[0035] (6) Preferably, specific data augmentation operations, such as elastic deformation, random rotation and Gaussian blur, are applied to the reference image and background image during training to disrupt the direct correspondence and force the model to learn deeper object feature representation and scene adaptive synthesis capabilities.

[0036] In summary, this invention overcomes the limitations of traditional methods that separate the foreground and background synthesis processes, innovatively integrating multiple key tasks such as object placement, image harmonization, and shadow generation into a unified end-to-end framework. By introducing a background cueing mechanism and a spatial mapping module, the model can adaptively adjust the pose, size, and appearance features of the foreground object, allowing it to blend naturally with the background scene, significantly improving process efficiency. Furthermore, this invention provides an effective self-supervised training method to address the lack of large-scale paired synthesis data when training such models. Attached Figure Description

[0037] Figure 1 The flowchart illustrates a maskless image synthesis method based on a diffusion model, as provided in an embodiment of the present invention.

[0038] Figure 2 This is a schematic diagram of the overall architecture of the UNICOM framework provided in an embodiment of the present invention.

[0039] Figure 3 This is an example of background cue word attention map visualization provided in an embodiment of the present invention.

[0040] Figure 4 This is a schematic diagram of the self-supervised training data generation process provided in an embodiment of the present invention.

[0041] Figure 5 This is an example of the diversity results generated in an embodiment of the present invention. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0043] Example 1

[0044] Figure 1 This is an overall flowchart of a high-efficiency image synthesis method based on a diffusion model provided by an embodiment of the present invention. The method in this embodiment uses a reference image containing the target object. and target background image As input, the core processing is accomplished by a unified image synthesis framework based on a latent diffusion model (hereinafter referred to as the UNICOM framework). This framework can process the input end-to-end and output the final synthesized image. The key is that this framework does not simply paste objects from the reference image onto the background. Instead, it combines cue words generated from the target background image with object embeddings extracted from the reference foreground image, and then maps them into a conditional vector that the diffusion model can effectively utilize. This conditional vector contains contextual information about the background image. Based on this contextual information, the target object is adaptively adjusted (including but not limited to lighting, color, style, viewpoint, pose, etc.) to harmoniously blend it into the background scene, generating high-quality, highly realistic composite results.

[0045] A pre-trained latent diffusion model (such as the StableDiffusion model based on the U-Net architecture) is used as the core generative backbone network. Figure 2 This is a schematic diagram of the overall architecture of the UNICOM framework provided in this embodiment of the invention. The core of this framework is a pre-trained latent diffusion model. In a specific implementation, the pre-trained latent diffusion model can be Stable Diffusion v2.1 or a similar model, which typically includes an encoder (such as a VAE Encoder), a denoising network based on a U-Net structure, and a decoder (such as a VAE Decoder). To achieve efficient adaptive image synthesis, this embodiment makes key modifications and extensions to the standard latent diffusion model:

[0046] (1) Latent space processing: Input background image and noisy images The target background image is encoded into the latent space using an encoder (e.g., VAE Encoder) of a diffusion model to obtain the latent representation of the target background image. (Encoded information) and noisy images Potential representation .

[0047] (2) U-Net Input Modification: To enable the denoising network to directly utilize the low-level visual information of the background, the input of the denoising network is modified so that it can simultaneously receive the encoded information from the target background image and the potential representation of the current noise as input. This embodiment modifies the number of input channels in the first convolutional layer of the U-Net structure (denoising network). For example, if the original U-Net input is 4 channels... The modified U-Net input layer then receives 8 channels of input, i.e. and The result of splicing along the channel dimension [ , This allows the model to perceive pixel-level / latent-level information of the background image at each step of denoising. In this embodiment of the invention, the denoising network based on the U-Net structure includes multiple spatial feature extraction modules connected in sequence. Each spatial feature extraction module includes a convolutional layer, a self-attention layer, and a cross-attention layer connected in sequence. The denoising network based on the U-Net structure is used to perform a conditionally guided reverse denoising process. In the reverse denoising process at the first time step, the input of the convolutional layer in the first spatial feature extraction module (i.e., the first convolutional layer mentioned above) is... and The result of splicing along the channel dimension [ , The output of the denoising network based on the U-Net structure is the predicted noise containing the synthesized image at the current time step. .

[0048] (3) Conditional Information Generation and Injection: In order to guide the model to synthesize specific content that is related to the reference image object and is harmonious with the background, this embodiment designs a special conditional generation mechanism:

[0049] (3-1) Object Feature Encoding: A powerful image encoder, preferably a CLIP (Contrastive Language–Image Pre-training) model, is used to encode the input reference image. Encode the object to extract an object embedding vector that represents the core semantics and key visual features of the target object. The CLIP image encoder was chosen because its feature space aligns well with the text space, facilitating its subsequent integration with diffusion models (typically text-conditional). In this embodiment, the parameters of the CLIP image encoder are typically frozen during training to fully utilize its powerful pre-trained feature extraction capabilities; the U-Net parameters of the diffusion model, the background cue word parameters (described below), and the spatial mapping module parameters are adjusted during training, while the CLIP encoder parameters remain frozen.

[0050] (3-2) Background Cueing Mechanism: To effectively pass the global contextual information of the background image to the cross-attention layer, which typically relies on serialization conditions (such as text token sequences), this embodiment introduces a set of learnable background cue words (tokens). These are learnable background cue words generated based on the target background image. This set of learnable background cue words is a set of learnable embedding vectors that learn to represent the background contextual information during training. These cue words are associated with the object embedding... Dimensionally compatible (consistent number of channels) embedding vectors learn to capture key information relevant to the context during training. These learnable background tokens are then used in conjunction with object embeddings. The features are combined (e.g., concatenated along the sequence dimension) to obtain the concatenated object embeddings. The concatenated feature representations are then processed by the spatial mapping module to obtain a conditional vector c. This conditional vector serves as the key and value input to each cross-attention layer in the U-Net structure, while the spatial features in U-Net act as the query. For example... Figure 3 As shown in the attention map visualization, these background cue words can adaptively focus on the background region after training, while the attention corresponding to the object embedding is focused on the foreground region where the object needs to be generated, thereby effectively injecting the contextual information of the background into the conditional vector c, realizing the effective injection of the background context; the conditional vector c is used to guide the synthesized image to be close to the real image.

[0051] (3-3) Spatial Mapping Module: Object Embedding with Only Image Encoder Output (or even if the object is embedded) The combined object embedding (obtained by combining background cue words) may still differ from the text embedding conditions used during the original pre-training of the diffusion model in terms of format (e.g., sequence length) and semantic space. To bridge this gap, this embodiment introduces a spatial mapping module. This module further processes the combined features (object embeddings obtained from the combination of object embeddings and background cue words), mapping the combined object embeddings into conditional vectors that the diffusion model can effectively utilize. Specifically, it maps the combined object embeddings to a conditional space compatible (with the same number of channels) across the cross-attention layers in the U-Net-based denoising network of the diffusion model. In this embodiment, the spatial mapping module can be implemented through one or more attention layers, a multilayer perceptron (MLP), or other linear / nonlinear transformations, aiming to align the feature space of the diffusion model and generate the final conditional vector used to guide the serialization process of diffusion. The conditional vector c is input as the main conditional signal to each cross-attention layer of the U-Net, precisely guiding the denoising process to generate content that conforms to the reference object features and adapts to the background. Specifically, the conditional vector c is used as the key and value of each cross-attention layer in the denoising network based on the U-Net structure, and the spatial features output by the self-attention layer in each spatial feature extraction module are used as the query.

[0052] (4) Condition-guided denoising process: In the reverse denoising process of the diffusion model, the predicted noise at each time step At that time, U-Net not only receives the modified input [ , It also receives a conditional vector c generated by the spatial mapping module. This conditional vector c is used as an intermediate layer feature of the U-Net via a cross-attention mechanism, while the latent representation of the target background image is... The modified U-Net input path provides underlying spatial information, allowing background information to participate in each step of the denoising prediction; that is... The input allows the diffusion model to directly utilize the underlying visual information of the background during the generation process. The two work together to guide the model gradually away from noise. Recover the latent representation of the target synthetic image .

[0053] (5) Image decoding (end-to-end synthesized image output): The final latent representation is obtained by performing a complete conditionally guided backdiffusion process. The final, high-resolution synthetic image is generated by decoding using a diffusion model decoder (such as a VAE decoder). The target object is adaptively composited into the background image. In this composite image, the target object originating from the reference image is not only placed in an appropriate area of ​​the background image, but its appearance (such as lighting, color, style) and even pose are adaptively adjusted and blended according to the environment of the background image.

[0054] Furthermore, since obtaining large-scale, high-quality paired data (reference image, background image, and real synthetic image) is very difficult and costly, this embodiment employs a self-supervised training strategy. This strategy utilizes existing, readily available datasets with object segmentation masks to automatically construct a large number of training triples (reference image, background image, and real image) to train the aforementioned diffusion model, thus addressing the challenge of lacking large-scale paired synthetic data. Figure 4 As shown, the training strategy is as follows:

[0055] (1) Automatic data construction: For each original image in the dataset containing the segmentation mask M

[0056] (1-1) Generate background image Using image inpainting techniques, such as the LAMA (Resolution-robust Large Mask Inpainting with Fourier Convolutions) model, for... The area marked by the mask M is repaired (content filling) to obtain the background image that is missing the target object. To achieve better repair results, the mask M can be appropriately inflated before repair to provide richer context for the repair model.

[0057] (1-2) Generate reference image Based on the object mask M or its corresponding bounding box, from the original image A region containing only or primarily the target object is cropped out from the image and used as a reference image. When cropping, some context margins can be preserved.

[0058] (1-3) Set realistic image The original, unmodified, complete image As the "real" synthetic target corresponding to this training sample, i.e. = .

[0059] Through the above steps, a large number of training triples are automatically constructed. , , ).

[0060] (2) Specific data augmentation: In order to further improve the robustness of the model and prevent it from learning a simple pixel-level copy-paste strategy, during the training process, reference images are augmented. (As needed, it can also be adjusted) Specific data augmentation techniques are applied, such as elastic deformation, random rotation, and Gaussian blur. The purpose of these augmentations is not merely to increase data diversity, but more importantly, to disrupt it. and The direct pixel-level correspondence and simple geometric relationships between objects in the model force it to go beyond simply learning "copy and paste" operations; it must instead develop an understanding of these relationships. Deep features of objects and learn to... These features are "resynthesized" in the environment, enabling the model to learn deeper object feature representations and scene-adaptive synthesis capabilities.

[0061] During model training, the latent diffusion model is used to add data to real images. Predictive loss for noise on the latent representation (added to the real image) The model is optimized by considering the loss between the noise in the latent representation and the predicted noise generated by the noisy network at each time step. During optimization, the model can adjust the lighting, color, style, or pose of the target object based on the context of the background image. Examples of the diverse results generated are as follows: Figure 5As shown. This self-supervised learning framework enables the model to be effectively trained without the need for manually labeled paired data, significantly improving the feasibility and practicality of the method.

[0062] Example 2

[0063] This invention provides a maskless image synthesis system based on a diffusion model, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the maskless image synthesis method based on the diffusion model in Embodiment 1 above.

[0064] The relevant technical solutions are the same as above, and will not be repeated here.

[0065] Example 3

[0066] This invention provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the maskless image synthesis method based on the diffusion model in Embodiment 1 above.

[0067] Specifically, the memory may include high-speed random access memory, as well as non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital (SD) cards, flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0068] The relevant technical solutions are the same as above, and will not be repeated here.

[0069] Example 4

[0070] This invention provides a computer program product, including a computer program that, when run on a computer, causes the computer to perform the steps of the maskless image synthesis method based on the diffusion model in Embodiment 1 above.

[0071] The relevant technical solutions are the same as above, and will not be repeated here.

[0072] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A maskless image synthesis method based on a diffusion model, characterized in that, include: Background image and noisy images containing target objects The input is fed into a pre-trained latent diffusion model, which performs an inverse denoising process guided by a conditional vector c to adaptively synthesize the target object into the background image. In the process, a composite image containing the target object is obtained; wherein the condition vector c is obtained in the following ways: Reference image containing the target object Encoded as object embedding vector And according to the background image Generate a set of learnable background cue words; embed the object into a vector. The concatenated feature representation is obtained by concatenating the background cue words; the concatenated feature representation is then spatially mapped to obtain a conditional vector c used to guide the reverse denoising process of the latent diffusion model. The denoising network for the potential diffusion model is a denoising network based on the U-Net structure; The concatenated feature representation is spatially mapped to obtain a condition vector c used to guide the reverse denoising process of the latent diffusion model. This includes mapping the concatenated feature representation to the condition space of each cross-attention layer in the denoising network to align the feature space of the latent diffusion model and generate a serialized condition vector c used to guide the diffusion process. The latent diffusion model performs a reverse denoising process under the guidance of the condition vector c, including: using the condition vector c as the key and value of each cross-attention layer in the U-Net structure, using the spatial features corresponding to the U-Net structure as the query, and guiding the denoising process through the cross-attention mechanism.

2. The maskless image synthesis method based on a diffusion model according to claim 1, characterized in that, The potential diffusion model includes an encoder, a denoising network, and a decoder; The encoder is used to process the background image. and noisy images containing target objects Feature encoding is performed to obtain the latent representation of the background image. Potential representation of noisy images ; The denoising network is obtained by modifying the number of input channels of the standard latent diffusion model denoising network, so that the modified denoising network can simultaneously receive the latent representation of the background image. Potential representation of noisy images Guided by the conditional vector c, a reverse denoising process is performed to obtain the latent representation of the synthesized image. ; The decoder is used for the latent representation of the synthesized image. Decoding is performed to obtain the synthesized image.

3. The maskless image synthesis method based on a diffusion model according to claim 1, characterized in that, The CLIP image encoder is used to extract the reference image containing the target object. Encoded as object embedding vector .

4. The maskless image synthesis method based on a diffusion model according to claim 1, characterized in that, The spliced ​​feature representation is spatially mapped using a spatial mapping module to obtain a condition vector c used to guide the reverse denoising process of the latent diffusion model; wherein, the spatial mapping module includes at least one attention layer, or the spatial mapping module is a multilayer perceptron.

5. The maskless image synthesis method based on a diffusion model according to any one of claims 1-4, characterized in that, The dataset used for pre-training the potential diffusion model is constructed in the following manner: For each original image in the dataset with a target object segmentation mask M Image restoration techniques are used to restore the original image. The region marked by the target object segmentation mask M is repaired to obtain the background image missing the target object. ; Based on the bounding box of the target object segmentation mask M, from the original image The region containing the target object is cropped out and used as a reference image. ; the original image As training samples ( , The corresponding real image The training triplet was obtained. , , ); Original images in the dataset with target object segmentation mask M The corresponding training triplet ( , , The dataset used for pre-training the potential diffusion model constitutes the dataset used for this purpose.

6. The maskless image synthesis method based on the diffusion model according to claim 5, characterized in that, Before pre-training the potential diffusion model, the following steps are also included: Each training sample in the dataset used for pre-training the potential diffusion model ( , Data augmentation can be performed using at least one of elastic deformation, random rotation, and Gaussian blur.

7. A maskless image synthesis system based on a diffusion model, characterized in that, Includes computer-readable storage media and processors; The computer-readable storage medium is used to store executable instructions; The processor is used to read executable instructions stored in the computer-readable storage medium and execute the maskless image synthesis method based on the diffusion model according to any one of claims 1-6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the maskless image synthesis method based on the diffusion model as described in any one of claims 1-6.

9. A computer program product, characterized in that, Includes a computer program that, when run on a computer, causes the computer to perform the maskless image synthesis method based on a diffusion model as described in any one of claims 1-6.

Images