Maskless image synthesis method and system based on diffusion model

Abstract

The invention belongs to the technical field of image processing, and discloses a maskless image synthesis method and system based on a diffusion model, and the method comprises the steps: inputting a background image and a noise image containing a target object into a pre-trained potential diffusion model, and enabling the potential diffusion model to execute a reverse denoising process under the guidance of a condition vector, the target object is synthesized in a self-adaptive mode, and a synthesized image is obtained; wherein the condition vector acquisition mode comprises the following steps: encoding a reference image containing a target object into an object embedding vector, and generating a group of learnable background cue words according to the object embedding vector; splicing with the background prompt word; and performing spatial mapping on the spliced feature representation to obtain a condition vector. The method can overcome the defects that in an existing image synthesis technology, the process is tedious, the efficiency is low, a large amount of manual intervention is needed, the synthesis result lacks the sense of reality and harmony, and especially the foreground object cannot be adaptively adjusted according to the background environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and more specifically, relates to a maskless image synthesis method and system based on a diffusion model. Background Art

[0002] Image synthesis is a fundamental task in computer vision and image editing. Its goal is to extract one or more foreground objects from a source image and naturally blend them into a target background image to create a new, visually realistic and harmonious image. This technology has broad and important applications in a wide range of fields, including film special effects, advertising production, virtual reality, augmented reality, e-commerce product display, and creative design.

[0003] However, traditional image synthesis methods typically rely on a series of complex and discrete manual or semi-automatic editing steps. These steps typically include, but are not limited to: precise segmentation and extraction of foreground objects (e.g., obtaining alpha matte through cutout techniques), spatial placement and geometric transformation of objects within the background image (e.g., scaling, rotation, and perspective correction), image harmonization (e.g., adjusting the foreground object's color distribution, lighting intensity and direction, shadows, hue, saturation, and style to match the background environment), shadow generation and rendering when necessary, and smooth edge blending.

[0004] This multi-step, fragmented processing pipeline has significant limitations. First, achieving high-quality image synthesis requires executing multiple independent editing tasks sequentially. This process is time-consuming and labor-intensive, requiring high levels of operator expertise and artistic accomplishment, and hinders rapid, batch processing. Second, because each editing step is typically optimized independently or based on simplified assumptions, it is difficult to fully consider the complex interactions between foreground and background across multiple dimensions, such as lighting conditions, global illumination transmission, perspective relationships, depth of field effects, and semantic context. As a result, the final synthesized image often lacks sufficient realism and overall harmony, prone to visual incongruities such as lighting mismatches, color disharmony, missing or incorrect shadows, hard edges, and the appearance of objects appearing "floating" or "pasted." Third, traditional methods typically treat the synthesized foreground object as a relatively fixed visual entity, primarily adjusting its external properties (such as color and brightness) to match the background. These methods typically fail to intelligently and adaptively adjust the foreground object's intrinsic properties (such as pose, shape, and partial lighting effects) to the specific context of the target background (such as specific lighting direction, occlusion relationships with surrounding objects, the overall scene atmosphere, or semantic logic). For example, when synthesizing a reference image of a car from the side onto a background that requires a frontal view of the car, traditional methods cannot automatically adjust the car's pose. Finally, many methods require an accurate foreground object mask (alpha matte) or other auxiliary information (such as precise placement) as input, further increasing the complexity of the operation and the requirements for upfront data preparation.

[0005] In recent years, the development of deep learning techniques, particularly generative adversarial networks (GANs) and generative diffusion models, has led to significant progress in image generation and editing. Some methods have attempted to leverage deep learning for specific subtasks such as image harmonization and shadow generation. However, integrating these subtasks into an efficient, end-to-end, automated synthesis pipeline that can adaptively adjust foreground objects remains a significant challenge. While generative diffusion models have demonstrated powerful capabilities for generating high-fidelity images and performing controllable editing, effectively applying them to build a unified image synthesis framework that intelligently understands context and enables adaptive foreground-to-background changes remains a pressing challenge.

[0006] Therefore, the existing technology still has obvious shortcomings and room for improvement in achieving image synthesis with simple and unified processes, high degree of automation, realistic and natural synthesis effects, and the ability of the foreground to intelligently adapt to the background environment. A new technical solution is urgently needed to overcome these challenges. Summary of the Invention

[0007] In response to the above-mentioned defects or improvement needs of the existing technology, the present invention provides a maskless image synthesis method and system based on a diffusion model, which aims to overcome the defects of existing image synthesis technology, such as cumbersome procedures, low efficiency, requirement for a lot of manual intervention, lack of realism and harmony in the synthesis results, and especially the inability of foreground objects to adaptively adjust according to the background environment.

[0008] To achieve the above object, the present invention provides a maskless image synthesis method based on a diffusion model, comprising: Set the background image and a noisy image containing the target object The input is fed into the pre-trained latent diffusion model, which performs a reverse denoising process under the guidance of the conditional vector c to adaptively synthesize the target object into the background image. , a synthetic image containing the target object is obtained; wherein the method of obtaining the conditional vector c includes: The reference image containing the target object Encoded as object embedding vector , and according to the background image Generate a set of learnable background prompt words; embed the object into a vector and the background prompt words to obtain a concatenated feature representation; and spatially map the concatenated feature representation to obtain a conditional vector c for guiding the reverse denoising process of the latent diffusion model.

[0009] Furthermore, the latent diffusion model includes an encoder, a denoising network, and a decoder; The encoder is used to and a noisy image containing the target object Perform feature encoding to obtain the potential representation of the background image and the latent representation of the noisy image ; The denoising network is obtained by modifying the number of input channels of the standard latent diffusion model denoising network so that the modified denoising network can simultaneously receive the latent representation of the background image and the latent representation of the noisy image , and perform the reverse denoising process under the guidance of the conditional vector c to obtain the potential representation of the synthetic image ; The decoder is used to generate a latent representation of the synthesized image Decoding is performed to obtain the synthesized image.

[0010] Furthermore, the denoising network is a denoising network based on a U-Net structure; the potential diffusion model performs a reverse denoising process under the guidance of the conditional vector c, including: The conditional vector c is used as the key and value of each cross-attention layer in the U-Net structure, and the spatial feature corresponding to the U-Net structure is used as the query. The denoising process is guided by the cross-attention mechanism.

[0011] Furthermore, a CLIP image encoder is used to convert the reference image containing the target object into Encoded as object embedding vector .

[0012] Furthermore, a spatial mapping module is used to spatially map the spliced ​​feature representation to obtain a conditional vector c for guiding the reverse denoising process of the latent diffusion model; wherein the spatial mapping module includes at least one attention layer, or the spatial mapping module is a multi-layer perceptron.

[0013] Furthermore, the dataset used for pre-training of the latent diffusion model is constructed in the following way: For each original image in the dataset with the target object segmentation mask M , use image restoration technology to restore the original image The area marked by the target object segmentation mask M is repaired to obtain the background image with the target object missing ; According to the bounding box of the target object segmentation mask M, from the original image Cut out the area containing the target object as the reference image ; The original image As training samples ( , ) The corresponding real image , and obtain the training triples ( , , ); Each original image in the dataset with the target object segmentation mask M The corresponding training triplet ( , , ) constitutes the dataset used for pre-training of the latent diffusion model.

[0014] Furthermore, before pre-training the potential diffusion model, the method further includes: For each training sample in the dataset used for pre-training the latent diffusion model ( , ) performs data enhancement, including at least one of elastic deformation, random rotation, and Gaussian blur.

[0015] The present invention also provides a diffusion model-based maskless image synthesis system, comprising a computer-readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is configured to read the executable instructions stored in the computer-readable storage medium to execute any one of the above-mentioned maskless image synthesis methods based on the diffusion model.

[0016] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method for synthesizing a maskless image based on a diffusion model as described above is implemented.

[0017] The present invention also provides a computer program product, comprising a computer program. When the computer program is run on a computer, the computer is enabled to execute any one of the above-mentioned methods for maskless image synthesis based on a diffusion model.

[0018] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects: (1) The present invention provides a process-efficient image synthesis method based on a diffusion model, which can process the input background image end-to-end. and a noisy image containing the target object , and output the final composite image Specifically, the present invention combines the prompt words representing the background context information generated based on the target background image with the object embedding extracted from the reference foreground image, and then maps them into conditional vectors that can be effectively utilized by the diffusion model, thereby effectively injecting the background context information into the conditional guidance, accurately guiding the denoising process to generate content that conforms to the characteristics of the reference object and adapts to the background. In the reverse denoising process of the diffusion model, the background image is specifically introduced. , so that the underlying visual information of the background can be directly utilized during the generation process. The precise guidance of the conditional vector and the synergistic use of the underlying visual information of the background enable the model to adaptively adjust the posture, size and appearance characteristics of the foreground target according to the contextual information of the background image, so that it can be naturally integrated with the background scene, thereby generating high-quality, highly realistic synthesis results, and significantly improving the efficiency of the process. The method of the present invention does not require masks or other auxiliary information, reducing the complexity of operations and the requirements for preliminary data preparation. Therefore, the present invention solves the defects of existing image synthesis technologies, such as cumbersome processes, low efficiency, the need for a lot of manual intervention, lack of realism and harmony in synthesis results, and especially the inability of foreground objects to be adaptively adjusted according to the background environment.

[0019] (2) Furthermore, in order to enable the denoising network to directly utilize the underlying visual information of the background, the present invention modifies the number of input channels of the standard latent diffusion model denoising network. The modified denoising network can simultaneously receive the potential representation of the background image. and the latent representation of the noisy image , which enables the model to perceive the pixel-level / latent-level information of the background image at each step of denoising.

[0020] (3) Furthermore, in the reverse denoising process of the diffusion model, the conditional vector c is used as the key and value of each cross-attention layer in the U-Net structure, and the spatial features corresponding to the U-Net structure are used as the query. The denoising process is guided by the cross-attention mechanism, which enables the model to adaptively pay attention to the background area at the same time (the contextual information of the background is injected into the conditional vector c), while the attention corresponding to the object embedding is focused on the foreground area where the object needs to be generated, so that the target object from the reference image is not only placed in the appropriate area of ​​the background image, but also its appearance (such as lighting, color, style) and even posture are adaptively adjusted and integrated according to the environment of the background image.

[0021] (4) As a preference, a CLIP image encoder is used to convert the reference image containing the target object into Encoded as object embedding vector ,The feature space of the CLIP image encoder has a good alignment with the text space, which facilitates the subsequent ,combination with the diffusion model.

[0022] (5) Furthermore, since it is very difficult and costly to obtain large-scale, high-quality (reference image, background image, real synthetic image) paired data, the present invention automatically constructs a large number of (reference image, background image, real image) training triplets using existing, easily accessible datasets with object segmentation masks, thus solving the challenge of lacking large-scale paired synthetic data.

[0023] (6) Preferably, specific data augmentation operations are applied to the reference image and background image during training, such as elastic deformation, random rotation, and Gaussian blur, in order to destroy the direct correspondence and force the model to learn deeper object feature representation and scene adaptive synthesis capabilities.

[0024] Overall, this invention overcomes the limitations of traditional methods that separate the foreground and background synthesis processes, innovatively integrating multiple key tasks such as object placement, image harmonization, and shadow generation into a unified end-to-end framework. By introducing a background cueing mechanism and a spatial mapping module, the model can adaptively adjust the posture, size, and appearance characteristics of foreground objects to naturally blend with the background scene, significantly improving process efficiency. Furthermore, this invention provides an effective self-supervised training method to address the lack of large-scale paired synthetic data required to train such models. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a flowchart of a maskless image synthesis method based on a diffusion model provided by an embodiment of the present invention.

[0026] Figure 2 A schematic diagram of the overall architecture of the UNICOM framework provided in an embodiment of the present invention.

[0027] Figure 3 This is an example of visualizing the attention map of background cue words provided by an embodiment of the present invention.

[0028] Figure 4 A schematic diagram of the self-supervised training data generation process provided by an embodiment of the present invention.

[0029] Figure 5 This is an example of the diversity results generated in the embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only intended to illustrate the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0031] Example 1 Figure 1 This is a flow chart of an efficient image synthesis method based on a diffusion model provided by an embodiment of the present invention. The method of this embodiment uses a reference image containing a target object. and the target background image The core processing is done by a unified image synthesis framework based on the latent diffusion model (hereinafter referred to as the UNICOM framework). This framework can process the input end-to-end and output the final synthesized image. The key is that this framework does not simply paste the object in the reference image onto the background. Instead, it combines the cue words generated based on the target background image with the object embedding extracted from the reference foreground image, and then maps them into a conditional vector that can be effectively utilized by the diffusion model. This conditional vector contains the contextual information of the background image. Based on the contextual information of the background image, the target object is adaptively adjusted (including but not limited to lighting, color, style, perspective, posture, etc.) to make it harmoniously integrate into the background scene, generating high-quality and highly realistic synthesis results.

[0032] A pre-trained potential diffusion model (such as the StableDiffusion model based on the U-Net architecture) is used as the core to generate the backbone network. Figure 2 The following is a schematic diagram of the overall architecture of the UNICOM framework provided by an embodiment of the present invention. The core of this framework is a pre-trained latent diffusion model. In a specific implementation, this pre-trained latent diffusion model can be Stable Diffusion v2.1 or a similar model. It typically includes an encoder (such as a VAE encoder), a denoising network based on a U-Net structure, and a decoder (such as a VAE decoder). To achieve efficient adaptive image synthesis, this embodiment makes key modifications and extensions to the standard latent diffusion model: (1) Latent space processing: input background image and noisy images They are encoded into the latent space through the encoder of the diffusion model (such as VAE Encoder) to obtain the latent representation of the target background image (encoded information) and noisy image potential representation of .

[0033] (2) U-Net input modification: In order to allow the denoising network to directly utilize the underlying visual information of the background, the input of the denoising network is modified so that it can simultaneously receive the encoded information from the target background image and the potential representation of the current noise as input. This embodiment modifies the number of input channels of the first convolutional layer of the U-Net structure (denoising network). For example, if the original U-Net input is 4 channels , then the modified U-Net input layer receives 8-channel input, that is and The result of splicing in the channel dimension [ , ]. This enables the model to perceive the pixel-level / latent-level information of the background image at each step of denoising. In an embodiment of the present invention, the denoising network based on the U-Net structure includes a plurality of sequentially connected spatial feature extraction modules, each of which includes a sequentially connected convolutional layer, a self-attention layer, and a cross-attention layer. The denoising network based on the U-Net structure is used to perform a conditionally guided reverse denoising process. In the reverse denoising process at the first time step, the input of the convolutional layer in the first spatial feature extraction module (that is, the first convolutional layer mentioned above) is and The result of splicing in the channel dimension [ , The output of the denoising network based on the U-Net structure is the predicted noise containing the synthetic image at the current time step. .

[0034] (3) Conditional Information Generation and Injection: To guide the model to synthesize specific content that is relevant to the reference image object and harmonious with the background, this embodiment designs a special conditional generation mechanism: (3-1) Object feature encoding: Use a powerful image encoder, preferably a CLIP (Contrastive Language–Image Pre-training) model, to encode the input reference image. Encode and extract an object embedding vector that can represent the core semantics and key visual features of the target object The CLIP image encoder was chosen because its feature space aligns well with the text space, facilitating subsequent integration with a diffusion model (typically conditioned on text). In this embodiment, the CLIP image encoder parameters are typically frozen during training to fully leverage its powerful pre-trained feature extraction capabilities. The training process adjusts the diffusion model's U-Net parameters, the background cue word parameters described below, and the spatial mapping module parameters, while the CLIP encoder parameters remain frozen.

[0035] (3-2) Background cue mechanism: In order to effectively pass the global context information of the background image to the cross-attention layer that usually relies on serialization conditions (such as text token sequences), this embodiment introduces a set of learnable background cue words (tokens), that is, a set of learnable background cue words (tokens) generated according to the target background image. This set of learnable background cue words is a set of learnable embedding vectors that learn to represent background context information during training. These cue words are embedded in the object. Dimensionally compatible (consistent number of channels) embedding vectors that learn to capture key information related to the background scene during training. These learnable background prompt words (tokens) are embedded in the object Combine (for example, concatenate in the sequence dimension) to obtain the concatenated object embedding. The concatenated feature representation is processed by the spatial mapping module to obtain the conditional vector c, which is used as the key (Key) and value (Value) input of each cross-attention layer in the U-Net structure, and the spatial feature in the U-Net is used as the query (Query). Figure 3 As shown in the attention map visualization, these background prompt words can adaptively focus on the background area after training, while the attention corresponding to the object embedding is concentrated on the foreground area where the object needs to be generated, thereby effectively injecting the background context information into the conditional vector c, realizing the effective injection of background context; the conditional vector c is used to guide the synthetic image to be close to the real image.

[0036] (3-3) Spatial Mapping Module: Only the object embedding output by the image encoder (or even if the object is embedded The combined object embedding (derived from the contextual cue words) may still differ from the text embedding conditions used in the original pre-training of the diffusion model in terms of format (e.g., sequence length) and semantic space. To bridge this gap, this embodiment introduces a spatial mapping module. This module further processes the combined features (from the object embedding and the contextual cue words), mapping the combined object embedding into a conditional vector that can be effectively utilized by the diffusion model. Specifically, the combined object embedding is mapped into a conditional space that is compatible (with the same number of channels) across the cross-attention layers in the diffusion model's U-Net-based denoising network. In this embodiment, the spatial mapping module can be implemented using one or more attention layers, multi-layer perceptrons (MLPs), or other linear / nonlinear transformations to align the feature space of the diffusion model and generate a sequenced conditional vector that is ultimately used to guide the diffusion process. The conditional vector c serves as the primary conditional signal input to each cross-attention layer of the U-Net, accurately guiding the denoising process to generate content that conforms to the reference object features and adapts to the background. Specifically, the conditional vector c is used as the key (Key) and value (Value) of each cross-attention layer in the denoising network based on the U-Net structure, and the spatial features output by the self-attention layer in each spatial feature extraction module are used as the query (Query).

[0037] (4) Condition-guided denoising process: In the reverse denoising process of the diffusion model, the noise is predicted at each time step. When U-Net receives not only the modified input [ , ], and also receives the conditional vector c generated by the spatial mapping module. The conditional vector c acts on the intermediate layer features of U-Net through the cross attention mechanism, and the potential representation of the target background image The modified U-Net input path provides the underlying spatial information, so that the background information is involved in the denoising prediction at each step; that is, The input of , enables the diffusion model to directly utilize the underlying visual information of the background during the generation process. The two work together to guide the model to gradually Recover the latent representation of the target synthetic image .

[0038] (5) Image decoding (end-to-end synthetic image output): By performing the complete conditionally guided back-diffusion process, the final latent representation is obtained Decode it through the decoder of the diffusion model (e.g., VAE Decoder) to generate the final, high-resolution synthetic image , the target object is adaptively synthesized into the background image. In this synthesized image, the target object from the reference image is not only placed in the appropriate area of ​​the background image, but its appearance (such as lighting, color, style) and even posture are adaptively adjusted and integrated according to the environment of the background image.

[0039] Furthermore, since obtaining large-scale, high-quality (reference image, background image, real synthetic image) paired data is very difficult and costly, this embodiment adopts a self-supervised training strategy that can automatically construct a large number of (reference image, background image, real image) training triplets using existing, easily available datasets with object segmentation masks to train the above-mentioned diffusion model, thereby addressing the challenge of lacking large-scale paired synthetic data. Figure 4 As shown, the training strategy is as follows: (1) Automatic data construction: For each original image in the dataset containing the segmentation mask M (1-1) Generate background image :Use image inpainting technology, such as LAMA (Resolution-robust Large Mask Inpainting with Fourier Convolutions) model, to The area marked by the mask M is repaired (content filled) to obtain the background image with the target object missing In order to obtain better restoration effects, the mask M can be properly expanded before restoration to provide richer context for the restoration model.

[0040] (1-2) Generate reference image :According to the object mask M or its corresponding bounding box, from the original image Cut out an area that only or mainly contains the target object from the image as the reference image . A certain context margin can be preserved when cropping.

[0041] (1-3) Set the real image : The original, unmodified complete image As the “real” synthetic target corresponding to the training sample, that is, = .

[0042] Through the above steps, a large number of training triplets are automatically constructed ( , , ).

[0043] (2) Specific data enhancement: In order to further improve the robustness of the model and prevent it from learning a simple pixel-level copy-paste strategy, during the training process, the reference image (If necessary, you can also ) Apply specific data enhancement techniques, such as elastic deformation, random rotation, and Gaussian blur. The purpose of these enhancements is not only to increase data diversity, but more importantly to destroy and The direct correspondence and simple geometric relationships between objects at the pixel level force the model to not only learn "copy and paste" operations, but also to understand deep features of objects in the These features are "resynthesized" in the environment, so that the model can learn deeper object feature representation and scene adaptability synthesis capabilities.

[0044] During model training, the latent diffusion model is added to the real image. The prediction loss of the noise on the latent representation (added to the real image The model is optimized by the loss between the noise on the potential representation and the predicted noise generated by the noise network at each time step. During the optimization process, the model can adjust the lighting, color, style or posture of the target object according to the context of the background image. Examples of the generated diverse results are shown below. Figure 5 This self-supervised learning framework enables the model to be effectively trained without manually annotated paired data, significantly improving the feasibility and practicality of the method.

[0045] Example 2 An embodiment of the present invention provides a diffusion model-based maskless image synthesis system, including a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, the steps of the diffusion model-based maskless image synthesis method in the above-mentioned embodiment 1 are implemented.

[0046] The relevant technical solutions are the same as above and will not be repeated here.

[0047] Example 3 An embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the maskless image synthesis method based on the diffusion model in the above-mentioned embodiment 1 are implemented.

[0048] Specifically, the memory may include a high-speed random access memory and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0049] The relevant technical solutions are the same as above and will not be repeated here.

[0050] Example 4 An embodiment of the present invention provides a computer program product, including a computer program. When the computer program is run on a computer, the computer is caused to execute the steps of the maskless image synthesis method based on the diffusion model in the above embodiment 1.

[0051] The relevant technical solutions are the same as above and will not be repeated here.

[0052] It will be easily understood by those skilled in the art that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A maskless image synthesis method based on a diffusion model, characterized in that: include: Set the background image and a noisy image containing the target object The input is fed into the pre-trained latent diffusion model, which performs a reverse denoising process under the guidance of the conditional vector c to adaptively synthesize the target object into the background image. , a synthetic image containing the target object is obtained; wherein the method of obtaining the conditional vector c includes: The reference image containing the target object Encoded as object embedding vector , and according to the background image Generate a set of learnable background prompt words; embed the object into a vector and the background prompt words to obtain a concatenated feature representation; and spatially map the concatenated feature representation to obtain a conditional vector c for guiding the reverse denoising process of the latent diffusion model.

2. The maskless image synthesis method based on the diffusion model according to claim 1, characterized in that: The latent diffusion model includes an encoder, a denoising network and a decoder; The encoder is used to and a noisy image containing the target object Perform feature encoding to obtain the potential representation of the background image and the latent representation of the noisy image ; The denoising network is obtained by modifying the number of input channels of the standard latent diffusion model denoising network so that the modified denoising network can simultaneously receive the latent representation of the background image and the latent representation of the noisy image , and perform the reverse denoising process under the guidance of the conditional vector c to obtain the potential representation of the synthetic image ; The decoder is used to generate a latent representation of the synthesized image Decoding is performed to obtain the synthesized image.

3. The maskless image synthesis method based on the diffusion model according to claim 2, characterized in that: The denoising network is a denoising network based on a U-Net structure; The latent diffusion model performs a reverse denoising process under the guidance of the conditional vector c, including: The conditional vector c is used as the key and value of each cross-attention layer in the U-Net structure, and the spatial feature corresponding to the U-Net structure is used as the query. The denoising process is guided by the cross-attention mechanism.

4. The maskless image synthesis method based on the diffusion model according to claim 1, characterized in that: The CLIP image encoder is used to convert the reference image containing the target object into Encoded as object embedding vector .

5. The maskless image synthesis method based on the diffusion model according to claim 1, characterized in that: A spatial mapping module is used to spatially map the spliced ​​feature representation to obtain a conditional vector c for guiding the reverse denoising process of the latent diffusion model; wherein the spatial mapping module includes at least one attention layer, or the spatial mapping module is a multi-layer perceptron.

6. The maskless image synthesis method based on a diffusion model according to any one of claims 1 to 5, characterized in that: The dataset used for pre-training of the latent diffusion model is constructed as follows: For each original image in the dataset with the target object segmentation mask M , use image restoration technology to restore the original image The area marked by the target object segmentation mask M is repaired to obtain the background image with the target object missing ; According to the bounding box of the target object segmentation mask M, from the original image Cut out the area containing the target object as the reference image ; The original image As training samples ( , ) The corresponding real image , and obtain the training triples ( , , ); Each original image in the dataset with the target object segmentation mask M The corresponding training triplet ( , , ) constitutes the dataset used for pre-training of the latent diffusion model.

7. The maskless image synthesis method based on the diffusion model according to claim 6, characterized in that: Before pre-training the latent diffusion model, the following steps are also included: For each training sample in the dataset used for pre-training the latent diffusion model ( , ) for data enhancement, including at least one of elastic deformation, random rotation, and Gaussian blur.

8. A maskless image synthesis system based on a diffusion model, characterized in that: comprising a computer-readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is configured to read the executable instructions stored in the computer-readable storage medium to execute the maskless image synthesis method based on the diffusion model according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the maskless image synthesis method based on the diffusion model according to any one of claims 1 to 7 is implemented.

10. A computer program product, characterized in that The invention comprises a computer program, which, when running on a computer, enables the computer to execute the maskless image synthesis method based on the diffusion model according to any one of claims 1 to 7.

Images