Multi-target image diffusion generation method based on image content guidance

Abstract

The invention provides a multi-target image diffusion generation method, device and equipment based on image content guidance. The method comprises the following steps: constructing a multi-target image diffusion generation model based on image content guidance; the method comprises the following steps: taking a single multi-target image as a condition image, segmenting the condition image to obtain a segmented instance image, extracting global image information of the condition image by using a first encoder, and extracting local image information of the segmented instance image by using a second encoder; a single multi-target image is processed by using a VAE encoder to obtain hidden space features, global image information, local image information and the hidden space features are input into a U-shaped network, a denoised encoded feature image is output, the encoded feature image is processed by using a VAE decoder to obtain a reconstructed image, and the reconstructed image is subjected to image reconstruction. According to the multi-target image diffusion generation model provided by the invention, high-quality images in which the number of objects in each category is the same as or even more than that of condition images but the layouts are different can be randomly generated.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method, apparatus and device for generating multi-target image diffusion based on image content guidance. Background Art

[0002] Over the past decade, deep neural networks have achieved tremendous success in a wide range of computer vision tasks. A key prerequisite for this success is the collection of large-scale training images. However, in real-world scenarios, even for a specific single task, collecting enough images to build a dataset is often costly and time-consuming. A promising solution to this problem is to use generative models for image enhancement, which aim to randomly generate a large number of synthetic images based on a small number of manually collected images to quickly build a dataset. Following this idea, various effective image generation models have been proposed. Among them, diffusion models have received increasing attention in image generation and enhancement due to their strong generation and generalization capabilities. Typically, given some hints related to the scene content, diffusion models can directly generate high-quality images with such content.

[0003] In practical applications, generating images of multiple objects in each scene is more critical, and the spatial relationships are more complex. Although multi-object image generation has made some progress recently, most of these methods have obvious limitations due to the increased difficulty of generation. Summary of the Invention

[0004] The main purpose of this application is to provide a method, device and equipment for multi-target image diffusion generation based on image content guidance, aiming to use a single multi-target image instead of text prompts as the condition of the diffusion model to reduce the category bias caused by text descriptions, thereby better extracting and injecting multi-object information into the diffusion process.

[0005] To achieve the above-mentioned objectives, the present application provides a multi-target image diffusion generation method based on image content guidance, including: constructing a multi-target image diffusion generation model based on image content guidance based on a first CLIP encoder, a second CLIP encoder, a VAE encoder, a U-shaped network and a VAE decoder; taking a single multi-target image as a conditional image, and segmenting the conditional image to obtain a segmented instance image, using the first encoder to extract the global image information of the conditional image, and using the second encoder to extract the local image information of the segmented instance image; using the VAE encoder to process the single multi-target image to obtain latent space features, inputting the global image information, local image information and latent space features into the U-shaped network, outputting a denoised encoded feature image, and using the VAE decoder to process the encoded feature image to obtain a reconstructed image with a category and quantity close to that of the single multi-target image but a different layout, wherein the multi-target image diffusion generation model is obtained based on training.

[0006] Optionally, the training process of the multi-target image diffusion generation model includes: training a U-shaped network of the multi-target image diffusion generation model.

[0007] Optionally, it also includes: adding an open set target detector to the multi-target image diffusion generation model; and separately training the U-shaped network of the multi-target image diffusion generation model after adding the open set target detector.

[0008] Optionally, the multi-target image diffusion generation model after adding the open-set target detector includes: a U-shaped network, and a VAE encoder, a first encoder, and a second encoder connected in parallel with the input of the U-shaped network; and a VAE decoder and an open-set target detector connected in series with the output of the U-shaped network.

[0009] Optionally, the U-shaped network of the multi-target image diffusion generation model is trained, including: obtaining a single multi-target sample image; obtaining a conditional sample image and a segmented instance sample image based on the single multi-target sample image; encoding the single multi-target sample image using a VAE encoder to obtain latent space feature information; extracting first global feature information of the conditional sample image using a first CLIP encoder, and extracting first local feature information of the segmented instance sample image using a second CLIP encoder; extracting second global feature information of the conditional sample image using the first CLIP encoder, and extracting second local feature information of the segmented instance sample image using the second CLIP encoder; connecting the first global feature information and the second global feature information in dimension to obtain third global feature information, and connecting the first local feature information and the second local feature information in dimension to obtain third local feature information; connecting the third global feature information and the third local feature information through a sequence to obtain a fusion feature information; input the fused feature information, latent space feature information and generated noise feature information into the U-shaped network, and output the encoded sample image; use the VAE decoder to decode the encoded sample image to obtain the restored sample image; use the open set target detector to process the restored sample image to obtain supervision information, wherein the supervision information includes the category list and the number of targets corresponding to each category in the category list, as well as text prompts and position information constructed based on the name of each category in the category list; and based on the supervision information, use the open set target detector to detect the category information of the restored sample image; construct a category loss function based on the supervision information and the category information; construct a mean square loss function based on a single multi-target sample image and the restored sample image; construct a total loss function based on the category loss function and the mean square loss function; based on the total loss function training, train the U-shaped network of the multi-target image diffusion generation model after adding the open set target detector separately to obtain the trained multi-target image diffusion generation model.

[0010] Optionally, the fusion feature, latent space feature and generated noise feature are input into the U-shaped network, and the expression used to output the encoded sample image is:

[0011]

[0012] Among them, ∈ θ represents the noise characteristics, α t , and σ t All represent the hyperparameters of the multi-target image diffusion generation model, z~N(0,I), represents the signal-to-noise ratio control term, Represents the fusion features and latent space features of the input, x i Represents an encoded sample image.

[0013] Optionally, the expression of the category loss function is:

[0014]

[0015] in, Represents the restored sample image x detected by the open set detector i c j The confidence results of all targets in the category, τ represents the threshold hyperparameter, where the threshold hyperparameter is determined according to the number of targets corresponding to each category in the category list.

[0016] Optionally, it also includes: approximately decomposing the weight matrix of the multi-target image diffusion generation model into a first low-rank matrix and a second low-rank matrix, wherein the respective ranks of the first low-rank matrix and the second low-rank matrix are much smaller than the rank of the weight matrix; initializing the first low-rank matrix with a random Gaussian distribution with a mean of 0; initializing the second low-rank matrix to all zeros; training the initialized first low-rank matrix and the initialized second low-rank matrix to obtain a trained U-shaped network of the multi-target image diffusion generation model.

[0017] In addition, to achieve the above-mentioned purpose, the present application also provides a multi-target image diffusion generation device based on image content guidance, including: a model construction module, used to construct a multi-target image diffusion generation model based on image content guidance based on a first CLIP encoder, a second CLIP encoder, a VAE encoder, a U-shaped network and a VAE decoder; an image generation module, used to take a single multi-target image as a conditional image, and segment the conditional image to obtain a segmented instance image, use the first encoder to extract the global image information of the conditional image, and use the second encoder to extract the local image information of the segmented instance image; use the VAE encoder to process the single multi-target image to obtain latent space features, input the global image information, local image information and latent space features into the U-shaped network, output a denoised encoded feature image, use the VAE decoder to process the encoded feature image, and obtain a reconstructed image with a category and quantity close to that of the single multi-target image but a different layout, wherein the multi-target image diffusion generation model is obtained based on training.

[0018] To achieve the above-mentioned objectives, the present application also provides an electronic device, which includes: at least one processor, a memory, and an input / output unit; wherein the memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the multi-target image diffusion generation method based on image content guidance provided in any of the aforementioned embodiments.

[0019] The embodiments of the present application propose a method, device and equipment for multi-target image diffusion generation based on image content guidance. A multi-target image diffusion generation model based on image content guidance is constructed based on a first CLIP encoder, a second CLIP encoder, a VAE encoder, a U-shaped network and a VAE decoder; a single multi-target image is used as a conditional image, and the conditional image is segmented to obtain a segmented instance image, the global image information of the conditional image is extracted using the first encoder, and the local image information of the segmented instance image is extracted using the second encoder; the single multi-target image is processed by the VAE encoder to obtain latent space features, the global image information, the local image information and the latent space features are all input into the U-shaped network, and a denoised encoded feature image is output, and the encoded feature image is processed by the VAE decoder to obtain a reconstructed image with a category and quantity close to that of the single multi-target image but a different layout, wherein the multi-target image diffusion generation model is obtained based on training. The multi-target image diffusion generation model proposed in the present application can randomly generate high-quality images with the same or even more objects in each category as the conditional image but with different layouts, thereby ensuring the diversity of image enhancement. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart illustrating an embodiment of a multi-target image diffusion generation method guided by image content provided by this application;

[0021] Figure 2 This is an overall structural diagram of an embodiment of a multi-target image diffusion generation method guided by image content in this application.

[0022] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0023] It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application.

[0024] In practical applications, the existing technology is more critical in generating images of multiple objects in each scene, and the spatial relationships are more complex. Although multi-target image generation has made some progress recently, most of these methods have obvious limitations due to the increased difficulty of generation. For example, one existing technology directly uses a simple description containing the category name to generate data, resulting in deviations in style, size, etc. between the category targets in the generated image and the original data. Furthermore, the second existing technology attempts to replace or add targets to the original image through image editing to enhance the image. Although this makes the enhanced image as realistic as possible, the amount of information added is limited because the layout, background and most targets of the image have not changed.

[0025] In order to overcome the shortcomings of the existing technology, the present application provides a multi-target image diffusion generation method based on image content guidance, which uses a single multi-target image instead of a text prompt as the condition of the diffusion model to reduce the category bias caused by the text description. In order to better extract and inject multi-object information into the diffusion process, the present application proposes a local-global semantic fusion strategy, which uses a pre-trained CLIP model to extract semantic knowledge from the entire conditional image and its local cropping respectively. On the other hand, in order to further control the number of objects and layout diversity in the generated image, the present application further proposes a counting loss based on a reward model to explicitly limit the number of objects in each category in the generated image without imposing any restrictions on their spatial layout. In this way, the proposed model can randomly generate high-quality images with the same or even more objects in each category as the conditional image but with different layouts, thereby ensuring the diversity of image enhancement.

[0026] Reference Figure 1 The first embodiment of the present application provides a multi-target image diffusion generation method based on image content guidance. The multi-target image diffusion generation method based on image content guidance may include:

[0027] S10, constructing a multi-target image diffusion generation model guided by image content based on the first CLIP encoder, the second CLIP encoder, the VAE encoder, the U-shaped network and the VAE decoder;

[0028] S20. Take a single multi-target image as a conditional image, and segment the conditional image to obtain segmented instance images, use the first encoder to extract global image information of the conditional image, and use the second encoder to extract local image information of the segmented instance image; use the VAE encoder to process the single multi-target image to obtain latent space features, input the global image information, local image information and latent space features into the U-shaped network, output the denoised encoded feature image, use the VAE decoder to process the encoded feature image, and obtain a reconstructed image with a category and quantity close to the single multi-target image but a different layout, wherein the multi-target image diffusion generation model is obtained based on training.

[0029] In one embodiment of the present application, the training process of the multi-target image diffusion generation model may include:

[0030] Training a U-shaped network for a multi-target image diffusion generative model.

[0031] In one embodiment of the present application, the method for generating multi-target image diffusion based on image content guidance may further include:

[0032] Adding an open set target detector to the multi-target image diffusion generation model;

[0033] The U-shaped network of the multi-target image diffusion generation model after adding the open set target detector is trained separately.

[0034] refer to Figure 2 In one embodiment of the present application, the multi-target image diffusion generation model after adding the open set target detector may include:

[0035] A U-shaped network, and a VAE encoder, a first encoder, and a second encoder connected in parallel to the input of the U-shaped network; and a VAE decoder and an open-set target detector connected in series with the output of the U-shaped network.

[0036] In one embodiment of the present application, the training of the U-shaped network of the multi-target image diffusion generation model may include the following execution process:

[0037] S201, obtaining a single multi-target sample image;

[0038] It should be noted that in large-scale multi-target image enhancement, for a set of N samples D = {(x i ,y i ),i=1,…,N}, where Indicates inclusion categories of input images, and for each category c j It contains o j goals. represent The category and box information of each target, and The goal of the task is to generate a set of enhanced images D = {x i ,i=1,…,N}, where For each category c in the input image j can find a c in the enhanced image l Correspondingly, and o l ≥o j .

[0039] S202, obtaining a conditional sample image and a segmented instance sample image based on a single multi-target sample image;

[0040] S203, using a VAE encoder to encode a single multi-target sample image to obtain latent space feature information;

[0041] A single multi-target sample image includes a natural image And its corresponding category label y i , where H and W represent the height and width of the image respectively. First, the image is input into the encoder E to encode it into the latent space

[0042]

[0043] in, d=4.

[0044] To adapt the latent diffusion model from text-guided image generation to image-guided image generation, we utilize the same text Image encoder E trained jointly with paired text-image data img The image condition is encoded so that the obtained features remain in the same feature space without fine-tuning all the parameters of the diffusion model. The original text encoder uses the latent space of text condition to capture the semantic relationship between the text context, that is:

[0045] C(p text )=E text (T(p text )0#(2)

[0046] in, is the output conditional feature, bs is the text condition p text, seq is the sequence length, emb is the feature dimension, and T is the word segmenter. To further clarify the instance that needs to be enhanced, we crop it from the image, merge it with the original image and input it into the image encoder to extract features, that is:

[0047]

[0048] in, For image conditions, the Crop function uses the image x that you want to enhance i Bounding box information b i Crop the instances you want to enhance. To better understand the image context, we use the parameter pad to control the pixels cropped outward. Through the above operations, we express information that is difficult to control with text, such as quantity and category, through image information, highlighting its importance in the condition. To reduce computational complexity, we only need all the features of the original image, that is, the first global feature information and the second global feature information in this application. For each cropped image, we only need its category features, that is, the first local feature information and the second local feature information in this application:

[0049]

[0050] in, P is an image processor that processes the input for batch training. Specifically, for the input image condition The image processor fixes the number of cropped instances to M. When the number of instances is less than M, it uses zero tensors for expansion. Otherwise, it randomly selects M instances for training. The above process of extracting global information and local information may include steps S204 to S207.

[0051] S204, using the first CLIP encoder to extract first global feature information of the conditional sample image, and using the second CLIP encoder to extract first local feature information of the segmented instance sample image;

[0052] The first CLIP encoder is a ViT-L / 14 image encoder, and the second CLIP encoder is a ViT-bigG / 14 image encoder.

[0053] S205, using the first CLIP encoder to extract second global feature information of the conditional sample image, and using the second CLIP encoder to extract second local feature information of the segmented instance sample image;

[0054] S206: Concatenate the first global feature information and the second global feature information in a dimension to obtain third global feature information, and concatenate the first local feature information and the second local feature information in a dimension to obtain third local feature information;

[0055] S207, connecting the third global feature information and the third local feature information through a sequence to obtain fused feature information;

[0056] S208, inputting the fused feature information, the latent space feature information and the generated noise feature information into the U-shaped network, and outputting the encoded sample image;

[0057] By using the proposed global-local image condition module, the present application can improve the complexity of the generated image. In order to further ensure that the complexity will not degenerate, specifically, for the input image x i , we obtain the image x by denoising it in one step during training i :

[0058]

[0059] where ∈ θ is the noise prediction network, α t 、 and σ t The hyperparameters set for DDPM (denoising diffusion probability model), z~N(0,I) are used to adjust the signal-to-noise ratio.

[0060] S209, using a VAE decoder to decode the encoded sample image to obtain a restored sample image;

[0061] S210, using an open set object detector to process the restored sample image to obtain supervision information, wherein the supervision information includes a category list and the number of objects corresponding to each category in the category list, and text prompts and location information constructed based on the names of each category in the category list;

[0062] S211, based on the supervision information, using an open set target detector to detect the category information of the restored sample image;

[0063] S212, a category loss function constructed based on supervision information and category information;

[0064] S213, constructing a mean square loss function based on the single multi-target sample image and the restored sample image;

[0065] S214. Construct a total loss function based on the category loss function and the mean square loss function;

[0066] Specifically, this application then uses the image annotation to construct supervision information. The counting loss does not constrain the bounding box position but only the number of categories to improve the diversity of layout generation. categories, first count the number of targets in each category and get a one-to-one corresponding category list and number list

[0067]

[0068] Among them, the name(·) function is used to get the category name, and the len(·) function is used to count the number. Then use a period to connect Each category in the reward model constructs a text prompt S i Since some categories have more than one word, record each category in S i The location information is Used to obtain the results of the reward model.

[0069] Finally, we use the pre-trained open vocabulary object detector, also known as the open set detector, as the reward model according to S i Detect the category in the image, for category c j , take the highest confidence sample according to the number in the input image and calculate the loss of each category according to the set threshold hyperparameter τ:

[0070]

[0071] in, The image x obtained by reward model detection i c j The confidence result of all targets in the category. Dividing the loss by the number of all instances gives the final count loss of the generated image:

[0072]

[0073] Since the denoised image may also contain a lot of noise at the beginning of training, we set a hyperparameter λ to control the starting step size for calculating the counting loss. The total training loss of this application can be expressed as:

[0074]

[0075]

[0076] Among them, λ is a hyperparameter for adjusting the loss weight.

[0077] S215. Based on the total loss function training, the U-shaped network of the multi-target image diffusion generation model after adding the open set target detector is trained separately to obtain the trained multi-target image diffusion generation model.

[0078] After the diffusion model is trained, this application uses the training set images to generate the potential space diffusion, and the denoising result z i , use the decoder D to restore it to the image space:

[0079] x i =D(x i )#(12)

[0080] In one embodiment of the present application, the diffusion model adopts the stable diffusion model SDXL, and the reward model adopts the pre-trained Grounding DINO tiny model.

[0081] Exemplarily, this application also includes a specific implementation process of a multi-target image diffusion generation method guided by image content:

[0082] 1. Data preprocessing

[0083] For a given dataset D, each sample consists of a natural image and its corresponding class label y, where H and W represent the height and width of the image, respectively. This application first resizes the image to a 512×512 resolution. Furthermore, random cropping and random horizontal flipping are used for training input images to enhance the model's generalization capabilities.

[0084] 2. Model structure and initialization

[0085] This application uses a low-rank matrix adaptation method to fine-tune the model, and the diffusion model uses the stable diffusion model SDXL. The low-rank matrix decomposes and reconstructs part of the model matrix to reduce the scale of parameter updates, thereby reducing computational costs and improving fine-tuning efficiency. Specifically, the low-rank matrix adaptation method approximately decomposes the model's weight matrix W into low-rank matrices A and B, namely:

[0086] W+ΔW=W+AB,#(13)

[0087] The ranks of A and B are much smaller than the rank of the original weight matrix, that is:

[0088] rank(A)< <rank(W),rank(B)<<rank(W) (14)

[0089] This effectively reduces the number of parameters that need to be trained.

[0090] For matrix initialization, the first low-rank matrix A is initialized using a random Gaussian distribution with a mean of zero and a small variance. The second low-rank matrix B is initialized to an all-zero matrix, ensuring that the low-rank matrix adaptation method does not affect the behavior of the original model during the initial fine-tuning phase. This initialization method ensures the stability of the fine-tuning process, allowing the model to maintain its original generative capabilities during the initial training phase. As training progresses, the low-rank matrix gradually adjusts the model weights to adapt to the requirements of the new task, ultimately resulting in a U-shaped network for the individually trained multi-target image diffusion generative model.

[0091] 3. Model training and generation

[0092] The denoising network of the model is pre-trained using batch training using the entire training set to better generate diverse images. Specifically, the other modules of the diffusion model are frozen to supervise the results of the denoising network training. This application uses the AdamW optimizer, sets the learning rate to 1e-4, and trains with 32 training samples per batch. Model training is to solve the following optimization problem:

[0093]

[0094] in, is the training loss, N B is the number of training samples in each batch, and θ is the parameter of the diffusion model denoising network. Finally, this application uses the training set images of each dataset as conditions and uses the Euler scheduler to sample the trained diffusion model for 50 steps, ultimately generating a large amount of diverse image data.

[0095] The beneficial effects of this application are as follows:

[0096] This application proposes a new multi-objective image enhancement framework, which aims to perform large-scale and diversified enhancements on images that conform to the original category distribution. Specifically, this application proposes a global-local image condition module that expresses the condition of the number of categories, which is difficult to express in text, using batched images, thereby improving the fidelity and diversity of the generated images. In order to further increase the complexity of the generated images, this application proposes a counting loss based on a reward model, which prevents image degradation by comparing the number of each category in the generated image with the original image.

[0097] Based on the above embodiments, the present application also provides a multi-target image diffusion generation device guided by image content, including a model construction module and an image generation module, wherein the model construction module is used to construct a multi-target image diffusion generation model guided by image content based on a first CLIP encoder, a second CLIP encoder, a VAE encoder, a U-shaped network and a VAE decoder; the image generation module is used to take a single multi-target image as a conditional image, and segment the conditional image to obtain a segmented instance image, use the first encoder to extract the global image information of the conditional image, and use the second encoder to extract the local image information of the segmented instance image; use the VAE encoder to process the single multi-target image to obtain latent space features, input the global image information, local image information and latent space features into the U-shaped network, output a denoised encoded feature image, and use the VAE decoder to process the encoded feature image to obtain a reconstructed image with a category and quantity close to that of the single multi-target image but a different layout, wherein the multi-target image diffusion generation model is obtained based on training.

[0098] It is not difficult to find that this embodiment is a system embodiment corresponding to the above-mentioned method embodiment, and this embodiment can be implemented in conjunction with the above-mentioned method embodiment. The relevant technical details and technical effects mentioned in the above-mentioned embodiments are still valid in this embodiment, and to reduce repetition, they are not repeated here. Accordingly, the relevant technical details mentioned in this embodiment can also be applied to the above-mentioned embodiments.

[0099] It is worth mentioning that all modules involved in this embodiment are logical modules. In actual applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. In addition, to highlight the innovation of this application, this embodiment does not introduce units that are not closely related to solving the technical problems proposed by this application. However, this does not mean that other units do not exist in this embodiment.

[0100] Based on the above embodiments, the present application also provides an electronic device to achieve the above purpose, which includes: at least one processor, a memory and an input and output unit; wherein the memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the multi-target image diffusion generation method based on image content guidance provided by any of the above embodiments.

[0101] The above are only preferred embodiments of the present application and do not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A multi-target image diffusion generation method based on image content guidance, characterized in that: include: Based on the first CLIP encoder, the second CLIP encoder, the VAE encoder, the U-shaped network and the VAE decoder, a multi-target image diffusion generation model guided by image content is constructed; A single multi-target image is used as the conditional image, and the conditional image is segmented to obtain segmented instance images. The first encoder is used to extract the global image information of the conditional image, and the second encoder is used to extract the local image information of the segmented instance image. The single multi-target image is processed by the VAE encoder to obtain latent space features. The global image information, local image information and latent space features are all input into the U-shaped network, and the denoised encoded feature image is output. The encoded feature image is processed by the VAE decoder to obtain a reconstructed image with categories and quantities close to the single multi-target image but different layout. Among them, the multi-target image diffusion generation model is obtained based on training.

2. The multi-target image diffusion generation method based on image content guidance according to claim 1, characterized in that: The training process of the multi-target image diffusion generation model includes: Training a U-shaped network for a multi-target image diffusion generative model.

3. The multi-target image diffusion generation method based on image content guidance according to claim 2, characterized in that: The U-shaped network of the multi-target image diffusion generation model is trained, comprising: Adding an open set target detector to the multi-target image diffusion generation model; The U-shaped network of the multi-target image diffusion generation model after adding the open set target detector is trained separately.

4. The multi-target image diffusion generation method based on image content guidance according to claim 3, characterized in that: The multi-target image diffusion generation model after adding the open set target detector includes: A U-shaped network, and a VAE encoder, a first encoder, and a second encoder connected in parallel to an input of the U-shaped network; And, the VAE decoder and open set object detector are sequentially connected in series with the output of the U-shaped network.

5. The multi-target image diffusion generation method based on image content guidance according to claim 4, characterized in that: The U-shaped network of the multi-target image diffusion generation model after adding the open set target detector is trained separately, including: Acquire a single multi-target sample image; Based on a single multi-target sample image, a conditional sample image and a segmented instance sample image are obtained; The VAE encoder is used to encode a single multi-target sample image to obtain latent space feature information; Extracting first global feature information of the conditional sample image using a first CLIP encoder, and extracting first local feature information of the segmented instance sample image using a second CLIP encoder; Extracting second global feature information of the conditional sample image using the first CLIP encoder, and extracting second local feature information of the segmented instance sample image using the second CLIP encoder; Connecting the first global feature information and the second global feature information in a dimension to obtain third global feature information, and connecting the first local feature information and the second local feature information in a dimension to obtain third local feature information; Connecting the third global feature information and the third local feature information through a sequence to obtain fused feature information; The fused feature information, latent space feature information and generated noise feature information are input into the U-shaped network, and the encoded sample image is output; Use the VAE decoder to decode the encoded sample image to obtain the restored sample image; Using an open set object detector to process the restored sample image to obtain supervision information, where the supervision information includes a category list and the number of objects corresponding to each category in the category list, as well as text prompts and location information constructed based on the category names in the category list; Based on the supervision information, an open set target detector is used to detect the category information of the restored sample image; Category loss function constructed based on supervision information and category information; Construct a mean square loss function based on a single multi-target sample image and the restored sample image; Construct a total loss function based on the category loss function and the mean square loss function; Based on the total loss function training, the U-shaped network of the multi-target image diffusion generation model after adding the open set target detector is trained separately to obtain the trained multi-target image diffusion generation model.

6. The multi-target image diffusion generation method based on image content guidance according to claim 5, characterized in that: The fusion features, latent space features and generated noise features are input into the U-shaped network, and the expression used to output the encoded sample image is: Among them, ∈ θ represents the noise characteristics, α t , and σ t All represent the hyperparameters of the multi-target image diffusion generation model, z~N(0,I), represents the signal-to-noise ratio control term, Represents the fusion features and latent space features of the input, x i Represents an encoded sample image.

7. The multi-target image diffusion generation method based on image content guidance according to claim 5, characterized in that: The expression of the category loss function is: in, Represents the restored sample image x detected by the open set detector i c j The confidence results of all targets in the category, τ represents the threshold hyperparameter, where the threshold hyperparameter is determined according to the number of targets corresponding to each category in the category list.

8. The multi-target image diffusion generation method based on image content guidance according to claim 1, characterized in that: Also includes: The weight matrix of the multi-target image diffusion generation model is approximately decomposed into a first low-rank matrix and a second low-rank matrix, wherein the rank of the first low-rank matrix and the second low-rank matrix is ​​much smaller than the rank of the weight matrix; Initialize the first low-rank matrix with a random Gaussian distribution with mean 0; Initialize the second low-rank matrix to all zeros; The initialized first low-rank matrix and the initialized second low-rank matrix are trained to obtain a trained U-shaped network of the multi-target image diffusion generation model.

9. A multi-target image diffusion generation device based on image content guidance, characterized in that: include: A model construction module is used to construct a multi-target image diffusion generation model guided by image content based on the first CLIP encoder, the second CLIP encoder, the VAE encoder, the U-shaped network and the VAE decoder; An image generation module is used to use a single multi-target image as a conditional image, segment the conditional image to obtain segmented instance images, use a first encoder to extract global image information of the conditional image, and use a second encoder to extract local image information of the segmented instance image; use a VAE encoder to process the single multi-target image to obtain latent space features, input the global image information, local image information and latent space features into a U-shaped network, output a denoised encoded feature image, and use a VAE decoder to process the encoded feature image to obtain a reconstructed image with a category and quantity close to the single multi-target image but a different layout, wherein the multi-target image diffusion generation model is obtained based on training.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor, memory, and input-output unit; The memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the multi-target image diffusion generation method based on image content guidance according to any one of claims 1 to 8.

Images