A high-quality image synthesis method based on diffusion model

Abstract

The application provides a high-quality image synthesis method based on a diffusion model, which gradually adds and removes noise through a diffusion model, and fuses multiple foreground and background images into a synthesized image with high realism; the method introduces an edge contour correction model, uses the edge contour information of the foreground as a prompt to stabilize the diffusion process of the edge contour information, and ensures that the synthesized image edge is not excessively smooth in the case of having overlapping objects; the method also introduces a color conversion model based on ControlNet, processes the color information of the foreground and the background, feeds back to the ControlNet, and then serves as a prompt for the diffusion model decoder, so that the color of the synthesized image is closer to the real situation in the case of having multiple foreground images; in addition, the application can also synthesize high-quality X-ray images with overlapping prohibited objects, and can synthesize images in more complex situations.

Description

Technical Field

[0001] This invention relates to the field of image processing and fusion technology, and more specifically, to a high-quality image synthesis method based on a diffusion model. Background Technology

[0002] In modern airport security operations, the rapid and efficient identification of prohibited items in baggage using X-ray machines is crucial. Improving the performance of X-ray detection systems requires continuous optimization of detection algorithms and enhanced operator accuracy, both of which depend on high-quality image data. However, high-quality X-ray image datasets containing diverse prohibited items are relatively scarce. Due to insufficient suitable training data, developing efficient and accurate automatic detection algorithms has become a challenge. Although image synthesis techniques such as GANs and Stable Diffusion offer a possible solution for synthesizing X-ray images, problems such as large color differences between foreground and background images and unnatural edge transitions still exist, failing to achieve satisfactory realism.

[0003] Image synthesis methods based on deep neural networks have made significant progress in recent years. These methods typically use large amounts of data to train models to learn how to effectively harmonize images. For example, Deep Image Harmonization Networks (DIH) automatically adjust foreground objects to match the statistical features of the background by predicting global color transformations. In recent years, image harmonization techniques have seen new developments with the rise of synthetic models such as Generative Adversarial Networks (GANs) and Variational Autoencoders (VAEs). These models are capable of synthesizing high-quality images and, through end-to-end training, better capture the complex interactions between foreground and background. These methods have demonstrated superior performance, particularly in style transfer and content preservation. However, despite the breakthroughs achieved by GANs in image synthesis, they still face challenges in maintaining the realism and detail of image content. Furthermore, the instability of GANs during training often leads to inconsistent quality in synthesized images.

[0004] Against this backdrop, diffusion models, as an emerging image synthesis technique, offer a novel approach. Compared to traditional GANs and other synthesis models, diffusion models synthesize images through a stepwise noise addition and removal process, which is more controllable and results in greater model stability during training. Diffusion models have demonstrated significant potential in image harmonization, offering not only superior image quality but also marked advantages in model training and stability.

[0005] Existing patent documents disclose an image editing method and system for image harmonization based on a diffusion model, including: collecting images to construct a foreground dataset and a background dataset, and constructing a synthetic image set; constructing an adaptive encoder using a pixel inverse reconstruction layer and downsampling blocks, and obtaining a composite feature map based on a foreground mask map and the synthetic image; improving the U-Net encoder structure in the diffusion model, obtaining a denoised feature map using the improved diffusion model, fusing the denoised feature map and the composite feature map, processing the fused feature map using RFFT and IRFFT, and mixing the processed feature map with the corresponding synthetic image using a foreground mask map to obtain a global feature map; inputting the global feature map into a VGG model for training, and simultaneously using a style loss function to process the output. The optimized global feature map is obtained through optimization. Although the existing technology is also based on the diffusion model for image synthesis, the existing technology still has the following defects: 1) The edges are not smooth and lack local adaptability. In the edge area, local image characteristics such as texture and lighting changes need to be considered. The existing technology does not adapt well to these local characteristics, resulting in unnatural edge transitions. This also makes it difficult for the technology to synthesize multiple foreground images with overlapping areas. 2) The colors in the synthesized image at the foreground position are not realistic enough and color shifts may occur. The foreground and background may come from different lighting conditions. The overall brightness and contrast vary greatly. Local lighting differences can also significantly affect color perception, resulting in inconsistency. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies in synthesizing images, such as uneven edges and low color realism, this invention provides a high-quality image synthesis method based on a diffusion model. This method can effectively fuse images of prohibited items into luggage X-ray scan images while maintaining clear outlines and high color realism. In addition, this invention can also synthesize high-quality X-ray images with overlapping prohibited items.

[0007] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0008] A high-quality image synthesis method based on a diffusion model includes the following steps:

[0009] S1: Obtain the background image and at least one foreground image, and extract the mask image of each foreground image;

[0010] Each of the foreground images is pasted into the background image to obtain a preliminary fused image, and the edge contour map of the preliminary fused image is extracted;

[0011] S2: Establish an edge contour correction model, which includes an adaptive encoder and an edge feature fusion module connected in sequence;

[0012] The mask map of each foreground image and the edge contour map of the preliminary fused image are input together into the adaptive encoder to obtain the contour feature map;

[0013] The preliminary fused image is input into the U-Net encoder of the diffusion model to obtain intermediate feature maps;

[0014] The contour feature map and the intermediate feature map are input together into the edge feature fusion module to obtain the contour fusion feature map;

[0015] S3: Establish a color conversion model, which includes a Lab space color transformation module and a ControlNet module connected in sequence;

[0016] The background image, along with each foreground image and its mask, are input into the Lab color transformation module to obtain a composite color adaptation map.

[0017] The synthesized color adaptation map is input into the ControlNet module to obtain the color adaptation feature map;

[0018] S4: Use the contour fusion feature map as a cue for the U-Net encoder of the diffusion model, and use the color adaptation feature map as a cue for the U-Net decoder of the diffusion model to obtain the improved diffusion model.

[0019] The preliminary fused image is input into the improved diffusion model to obtain a high-quality synthesized image.

[0020] Preferably, in step S1, the background image is specifically an X-ray image of luggage; the foreground image is specifically an X-ray image of contraband.

[0021] Preferably, in step S1, the edge contour map of the preliminary fused image is extracted using the Canny edge detector.

[0022] Preferably, in step S2, the adaptive encoder includes a Pixel Unshuffle downsampling block, a first feature extraction block, a first downsampling block, a second feature extraction block, a second downsampling block, a third feature extraction block, a third downsampling block, and a fourth feature extraction block connected in sequence, and each feature extraction block includes a convolutional layer and two residual blocks connected in sequence.

[0023] Preferably, in step S2, the edge feature fusion module includes a first Transformer layer, a second Transformer layer, and a fully connected layer;

[0024] The input to the first Transformer layer is an intermediate feature map, the input to the second Transformer layer is a contour feature map, and the outputs of the first and second Transformer layers are respectively connected to the input of the fully connected layer.

[0025] The output of the fully connected layer is concatenated with the intermediate feature map and the contour feature map to obtain the contour fusion feature map.

[0026] Preferably, in step S3, the step of obtaining the synthesized color adaptation map using the Lab color space transformation module includes:

[0027] S3.1: Combine the background image with the mask image of each foreground image to obtain the image of each foreground position in the background;

[0028] S3.2: Convert the images of each foreground position in the background and each foreground image from the RGB color space to the Lab color space respectively, and obtain the background Lab image and each foreground Lab image respectively;

[0029] S3.3: Match the background Lab image with the mean and standard deviation of each foreground Lab image in the Lab color space to obtain the color adaptation map of each foreground Lab image respectively;

[0030] S3.4: Convert the color adaptation map of each foreground Lab image from the Lab color space to the RGB color space, and stitch them together with the background image to obtain the image in the background after color adaptation transformation at the foreground position, which is used as the composite color adaptation map.

[0031] Preferably, step S3.3 includes:

[0032] The mean value μ of the background Lab image and each foreground Lab image in the Lab color space is calculated using the color mean calculation formula. C The formula for calculating the average color value is as follows:

[0033]

[0034] Among them, C i This represents the value of the i-th pixel, where n is the number of pixels.

[0035] According to the mean μ C Calculate the corresponding standard deviation σ C The specific calculation formula is as follows:

[0036]

[0037] The color adaptation map for each foreground Lab image is obtained using the following formula:

[0038]

[0039] Among them, L ′ a ′ and b ′ L, a, and b are the luminance channel value, A color channel value, and B color channel value of the color adaptation map of the foreground Lab image, respectively; L, a, and b are the luminance channel value, A color channel value, and B color channel value of the foreground Lab image, respectively. and These represent the mean values ​​of the brightness channel, the A color channel, and the B color channel of the background Lab image, respectively. and These represent the mean values ​​of the brightness channel, the A color channel, and the B color channel of the foreground Lab image, respectively. and The standard deviations of the brightness channel, A color channel, and B color channel of the background Lab image are: and These represent the standard deviations of the luminance channel, A color channel, and B color channel of the foreground Lab image, respectively.

[0040] Preferably, in step S3, the step of obtaining the color adaptation feature map using the ControlNet module includes:

[0041] First, a pre-defined encoder is used to convert the synthetic color adaptation map into a feature space, matching its size and dimensions to the input layer of the diffusion model, as shown in the following formula:

[0042] c f =E(c)

[0043] Among them, c f The prompt condition is E(c); E(c) is the preset encoder; c is the synthesized color adaptation map.

[0044] Copy the weights Θ of the U-Net encoder in the diffusion model to form a new trainable copy Θ. c :

[0045] The ControlNet module includes, in sequence: a first zero convolutional layer, a trainable copy Θ, and so on. c And the second zero convolutional layer;

[0046] The prompt condition c f As input to the ControlNet module, the ControlNet module outputs a color adaptation feature map y. c Specifically:

[0047] y c =F(x;Θ)+Z(F(x+Z(c)) f ;Θ z1 );Θ c );Θ z2 )

[0048] Where F represents the U-Net encoder of the diffusion model, x is the input of the U-Net encoder, and Θ is the parameter of the U-Net encoder; Z represents the zero convolutional layer, and Θ... z1 Θ represents the parameters of the first zero convolutional layer. z2 These are the parameters of the second zero convolutional layer.

[0049] Preferably, in the U-Net encoder of the diffusion model, the intermediate layer of the U-Net encoder outputs the intermediate feature map, and the contour fusion feature map output by the edge feature fusion module is input into the cross attention layer of the U-Net encoder, and then mapped to the intermediate layer of the U-Net encoder as a cue for the U-Net encoder.

[0050] In the U-Net decoder of the diffusion model, a ControlNet module is set after each decoding block of the U-Net decoder. The color adaptation feature map output by the ControlNet module is used as a cue for the U-Net decoder of the diffusion model to obtain the improved diffusion model.

[0051] Preferably, the diffusion model is the Stable Diffusion diffusion model.

[0052] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0053] This invention provides a high-quality image synthesis method based on a diffusion model. First, a background image and at least one foreground image are acquired, and the mask image of each foreground image is extracted. Each foreground image is pasted into the background image to obtain a preliminary fused image, and the edge contour map of the preliminary fused image is extracted. Next, an edge contour correction model is established, comprising an adaptive encoder and an edge feature fusion module connected in sequence. The mask image of each foreground image and the edge contour map of the preliminary fused image are input together into the adaptive encoder to obtain a contour feature map. The preliminary fused image is input into the U-Net encoder of the diffusion model to obtain an intermediate feature map. The contour feature map and the intermediate feature map are then input together into the edge feature fusion module. In this module, a contour fusion feature map is obtained; then, a color conversion model is built, which includes a Lab space color transformation module and a ControlNet module connected in sequence; the background image, as well as each foreground image and its mask image, are input into the Lab space color transformation module to obtain a synthetic color adaptation map; the synthetic color adaptation map is input into the ControlNet module to obtain a color adaptation feature map; finally, the contour fusion feature map is used as a cue for the U-Net encoder of the diffusion model, and the color adaptation feature map is used as a cue for the U-Net decoder of the diffusion model to obtain an improved diffusion model; the preliminary fused image is input into the improved diffusion model to obtain a high-quality synthetic image.

[0054] This invention constructs an edge contour correction model, using the edge contour map of the foreground as a condition, thereby stabilizing the diffusion process of edge contour information and ensuring that the edges of the foreground do not become jagged or uneven in the background. Simultaneously, this invention constructs a color conversion model, combining images in the Lab color space as cues, thereby obtaining images with more realistic colors. This invention can effectively fuse multiple foreground images into a background image while maintaining clear contours and highly realistic colors. Furthermore, this invention can also synthesize high-quality X-ray images with overlapping contraband, enabling the synthesis of images in more complex situations. Attached Figure Description

[0055] Figure 1 This is a flowchart of a high-quality image synthesis method based on a diffusion model provided in Example 1.

[0056] Figure 2 This is an overall architecture diagram of a high-quality image synthesis method based on a diffusion model provided in Example 2.

[0057] Figure 3 This is a diagram of the adaptive encoder architecture provided in Example 2.

[0058] Figure 4 This is a diagram of the edge feature fusion module architecture provided in Example 2.

[0059] Figure 5 This is a schematic diagram of the color conversion model provided in Example 2.

[0060] Figure 6 This is a schematic diagram of a ControlNet block provided in Example 2.

[0061] Figure 7 This is a schematic diagram of the ControlNet module structure provided in Example 2.

[0062] Figure 8 This is a structural diagram of the improved Stable Diffusion model provided in Example 2.

[0063] Figure 9 This is a demonstration diagram of high-quality image synthesis based on a diffusion model provided in Example 2. Detailed Implementation

[0064] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0065] To better illustrate this embodiment, some parts in the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions;

[0066] It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings.

[0067] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0068] Example 1

[0069] like Figure 1 As shown, this embodiment provides a high-quality image synthesis method based on a diffusion model, including the following steps:

[0070] S1: Obtain the background image and at least one foreground image, and extract the mask image of each foreground image;

[0071] Each of the foreground images is pasted into the background image to obtain a preliminary fused image, and the edge contour map of the preliminary fused image is extracted;

[0072] S2: Establish an edge contour correction model, which includes an adaptive encoder and an edge feature fusion module connected in sequence;

[0073] The mask map of each foreground image and the edge contour map of the preliminary fused image are input together into the adaptive encoder to obtain the contour feature map;

[0074] The preliminary fused image is input into the U-Net encoder of the diffusion model to obtain intermediate feature maps;

[0075] The contour feature map and the intermediate feature map are input together into the edge feature fusion module to obtain the contour fusion feature map;

[0076] S3: Establish a color conversion model, which includes a Lab space color transformation module and a ControlNet module connected in sequence;

[0077] The background image, along with each foreground image and its mask, are input into the Lab color transformation module to obtain a composite color adaptation map.

[0078] The synthesized color adaptation map is input into the ControlNet module to obtain the color adaptation feature map;

[0079] S4: Use the contour fusion feature map as a cue for the U-Net encoder of the diffusion model, and use the color adaptation feature map as a cue for the U-Net decoder of the diffusion model to obtain the improved diffusion model.

[0080] The preliminary fused image is input into the improved diffusion model to obtain a high-quality synthesized image.

[0081] In the specific implementation process, firstly, a background image and at least one foreground image are acquired, and the mask image of each foreground image is extracted; each foreground image is pasted into the background image to obtain a preliminary fused image, and the edge contour map of the preliminary fused image is extracted;

[0082] Next, an edge contour correction model is established, which includes an adaptive encoder and an edge feature fusion module connected in sequence. The mask map of each foreground image and the edge contour map of the preliminary fused image are input into the adaptive encoder to obtain the contour feature map. The preliminary fused image is input into the U-Net encoder of the diffusion model to obtain the intermediate feature map. The contour feature map and the intermediate feature map are input into the edge feature fusion module to obtain the contour fused feature map.

[0083] Next, a color conversion model is established, which includes a Lab space color transformation module and a ControlNet module connected in sequence. The background image, as well as each foreground image and its mask image, are input into the Lab space color transformation module to obtain a synthetic color adaptation map. The synthetic color adaptation map is then input into the ControlNet module to obtain a color adaptation feature map.

[0084] Finally, the contour fusion feature map is used as a cue for the U-Net encoder of the diffusion model, and the color adaptation feature map is used as a cue for the U-Net decoder of the diffusion model to obtain the improved diffusion model; the preliminary fused image is input into the improved diffusion model to obtain a high-quality synthetic image.

[0085] This method constructs an edge contour correction model and uses the edge contour map of the foreground as a condition to stabilize the diffusion process of edge contour information, ensuring that the edges of the foreground do not become jagged or uneven in the background. At the same time, this method constructs a color conversion model and uses images in the Lab color space as cues to obtain images with more realistic colors. This method can effectively fuse multiple foreground images into the background image while maintaining clear contours and highly realistic colors.

[0086] Example 2

[0087] This embodiment provides a high-quality image synthesis method based on a diffusion model, including the following steps:

[0088] S1: Obtain a background image and at least one foreground image, and extract the mask image of each foreground image; the background image is specifically an X-ray image of luggage; the foreground image is specifically an X-ray image of contraband.

[0089] Each of the foreground images is pasted into the background image to obtain a preliminary fused image, and the edge contour map of the preliminary fused image is extracted using the Canny edge detector;

[0090] S2: Establish an edge contour correction model, which includes an adaptive encoder and an edge feature fusion module connected in sequence;

[0091] The mask map of each foreground image and the edge contour map of the preliminary fused image are input together into the adaptive encoder to obtain the contour feature map;

[0092] The preliminary fused image is input into the U-Net encoder of the diffusion model to obtain intermediate feature maps;

[0093] The contour feature map and the intermediate feature map are input together into the edge feature fusion module to obtain the contour fusion feature map;

[0094] S3: Establish a color conversion model, which includes a Lab space color transformation module and a ControlNet module connected in sequence;

[0095] The background image, along with each foreground image and its mask, are input into the Lab color transformation module to obtain a composite color adaptation map.

[0096] The synthesized color adaptation map is input into the ControlNet module to obtain the color adaptation feature map;

[0097] S4: Use the contour fusion feature map as a cue for the U-Net encoder of the diffusion model, and use the color adaptation feature map as a cue for the U-Net decoder of the diffusion model to obtain the improved diffusion model.

[0098] The preliminary fused image is input into the improved diffusion model to obtain a high-quality synthesized image;

[0099] In step S2, the adaptive encoder includes a Pixel Unshuffle downsampling block, a first feature extraction block, a first downsampling block, a second feature extraction block, a second downsampling block, a third feature extraction block, a third downsampling block, and a fourth feature extraction block connected in sequence, and each feature extraction block includes a convolutional layer and two residual blocks connected in sequence.

[0100] In step S2, the edge feature fusion module includes a first Transformer layer, a second Transformer layer, and a fully connected layer.

[0101] The input to the first Transformer layer is an intermediate feature map, the input to the second Transformer layer is a contour feature map, and the outputs of the first and second Transformer layers are respectively connected to the input of the fully connected layer.

[0102] The output of the fully connected layer is concatenated with the intermediate feature map and the contour feature map to obtain the contour fusion feature map;

[0103] In step S3, the step of obtaining the synthetic color adaptation map using the Lab space color transformation module includes:

[0104] S3.1: Combine the background image with the mask image of each foreground image to obtain the image of each foreground position in the background;

[0105] S3.2: Convert the images of each foreground position in the background and each foreground image from the RGB color space to the Lab color space respectively, and obtain the background Lab image and each foreground Lab image respectively;

[0106] S3.3: Match the background Lab image with the mean and standard deviation of each foreground Lab image in the Lab color space to obtain the color adaptation map of each foreground Lab image respectively;

[0107] S3.4: Convert the color adaptation map of each foreground Lab image from Lab color space to RGB color space, and stitch them together with the background image to obtain the image in the background after color adaptation transformation at the foreground position, which is used as the composite color adaptation map;

[0108] Step S3.3 includes:

[0109] The mean value μ of the background Lab image and each foreground Lab image in the Lab color space is calculated using the color mean calculation formula. C The formula for calculating the average color value is as follows:

[0110]

[0111] Among them, C i This represents the value of the i-th pixel, where n is the number of pixels.

[0112] According to the mean μ C Calculate the corresponding standard deviation σ C The specific calculation formula is as follows:

[0113]

[0114] The color adaptation map for each foreground Lab image is obtained using the following formula:

[0115]

[0116] Among them, L ′ a ′ and b ′ L, a, and b are the luminance channel value, A color channel value, and B color channel value of the color adaptation map of the foreground Lab image, respectively; L, a, and b are the luminance channel value, A color channel value, and B color channel value of the foreground Lab image, respectively. and These represent the mean values ​​of the brightness channel, the A color channel, and the B color channel of the background Lab image, respectively. and These represent the mean values ​​of the brightness channel, the A color channel, and the B color channel of the foreground Lab image, respectively. and The standard deviations of the brightness channel, A color channel, and B color channel of the background Lab image are: and These represent the standard deviations of the brightness channel, A color channel, and B color channel of the foreground Lab image, respectively.

[0117] In step S3, the step of obtaining the color adaptation feature map using the ControlNet module includes:

[0118] First, a pre-defined encoder is used to convert the synthetic color adaptation map into a feature space, matching its size and dimensions to the input layer of the diffusion model, as shown in the following formula:

[0119] c f =E(c)

[0120] Among them, c f The prompt condition is E(c); E(c) is the preset encoder; c is the synthesized color adaptation map.

[0121] Copy the weights Θ of the U-Net encoder in the diffusion model to form a new trainable copy Θ. c :

[0122] The ControlNet module includes, in sequence: a first zero convolutional layer, a trainable copy Θ, and so on. c And the second zero convolutional layer;

[0123] The prompt condition c f As input to the ControlNet module, the ControlNet module outputs a color adaptation feature map y. c Specifically:

[0124] y c =F(x;Θ)+Z(F(x+Z(c)) f ;Θ z1 );Θ c );Θ z2 )

[0125] Where F represents the U-Net encoder of the diffusion model, x is the input of the U-Net encoder, and Θ is the parameter of the U-Net encoder; Z represents the zero convolutional layer, and Θ... z1 Θ represents the parameters of the first zero convolutional layer. z2 These are the parameters of the second zero convolutional layer;

[0126] In the U-Net encoder of the diffusion model, the intermediate layer of the U-Net encoder outputs the intermediate feature map, and the contour fusion feature map output by the edge feature fusion module is input into the cross attention layer of the U-Net encoder, and then mapped to the intermediate layer of the U-Net encoder as a cue for the U-Net encoder.

[0127] In the U-Net decoder of the diffusion model, a ControlNet module is set after each decoding block of the U-Net decoder. The color adaptation feature map output by the ControlNet module is used as a cue for the U-Net decoder of the diffusion model to obtain the improved diffusion model.

[0128] The diffusion model mentioned is specifically the Stable Diffusion diffusion model.

[0129] In the specific implementation process, such as Figure 2 The diagram shown illustrates the overall architecture of this method, which includes the following processes:

[0130] 1) Acquire X-ray images of the luggage as the background and at least one foreground image representing prohibited items;

[0131] This step mainly involves two types of input; the first is fusing X-ray images, which involves pasting the foreground image onto the background image: denoted as I c ,in H and W are the height and width of the image, respectively; C is the number of channels; next is the mask image of the foreground; denoted as M, where H and W are the height and width of the image, respectively; the fused X-ray image I c I. Obtaining fused edge contour maps based on the Canny edge detector e ,in

[0132] 2) Input the fused edge contour map and foreground mask image into the adaptive encoder;

[0133] like Figure 3 The diagram shows the architecture of the adaptive encoder. The first step involves stitching the edge contour map with the mask image of the foreground image to obtain a contour stitched image. The second step uses this contour stitched image through a pixel unshuffle downsampling block to obtain... The third step is to construct an EM module, which consists of a convolutional layer and two residual blocks. The adaptive encoder has four EM modules. As input, the signal passes through four EM modules, ultimately yielding the output. It can be represented as:

[0134]

[0135]

[0136]

[0137]

[0138] Where DS() represents the downsampling block and EM() represents the EM module; the resulting The resolutions are 64×64, 32×32, 16×16 and 8×8 respectively; this resolution corresponds one-to-one with the U-Net encoder in the diffusion model;

[0139] 3) Using the edge feature fusion module, the output of the adaptive encoder is combined with that of the U-Net encoder in the diffusion model to obtain the edge contour fusion feature map, which is then input into the U-Net encoder;

[0140] like Figure 4 The diagram shows the architecture of the edge feature fusion model. The edge feature fusion module designs different fusion strategies for features of different resolutions input from the adaptive encoder and the U-Net encoder in the diffusion model; for shallow features with high resolution... and Directly with the output of the U-Net encoder in the diffusion model and Addition; for deep features and Requires the output of the U-Net encoder and The input is then fused into the edge feature fusion module.

[0141] first, Element-wise multiplication with the foreground mask M yields As a query in the Transformer layer, it is then mapped from high-dimensional features to a one-dimensional vector, where the dimension of the foreground mask M needs to be the same as the dimension of the vector. Correspondingly, before element-wise multiplication, the foreground mask M needs to be downsampled to obtain the result. Dimensional correspondence The formula is shown below:

[0142]

[0143] Here, Flatten() represents the transformation from a 2D feature map to a 1D feature sequence. Represents element-wise product;

[0144] Element-wise multiplication with the foreground inverse mask 1-M yields As the key and value in the Transformer layer;

[0145]

[0146] Here, Flatten() represents the transformation from a 2D feature map to a 1D feature sequence. Represents element-wise product;

[0147] Received and The formula is entered into the transformer as shown below:

[0148]

[0149] Similarly, the output of the U-Net encoder in the diffusion model and Repeat the above The operation yields... and Then input it into the transformer layer to get

[0150] The output of the Transformer layer is obtained by combining the adaptive encoder with the U-Net encoder in the diffusion model. and The formula for splicing is as follows:

[0151]

[0152] Where FC() represents a fully connected layer. This can be represented as a connection between two vectors;

[0153] Finally, output the U-Net encoder. Contour composite feature map and the background of the contour feature map Combining them, we can ultimately obtain The formula is expressed as follows:

[0154]

[0155] Fold() represents the collapsing of the connection between two vectors into a 2D foreground image. Represents element-wise product;

[0156] In summary, the overall formula for the edge feature fusion module is as follows:

[0157]

[0158] 4) Change the colors of the foreground and background images;

[0159] like Figure 5 As shown, using the foreground and background images, the foreground image is converted to the Lab color space, and the background and foreground images are composited using masks to extract the foreground position image from the background image; using the Lab to RGB space conversion algorithm, the background Lab image and the foreground Lab image are obtained; the two obtained images are then subjected to color mean and standard deviation matching operations; the goal of this operation is to adjust the color distribution of the foreground to make it similar to the color distribution of the background image;

[0160] The formula for color mean is shown below.

[0161]

[0162] Where C i This represents the value of the i-th pixel.

[0163] The formula for matching the standard deviation of color means is shown below;

[0164]

[0165] The formula is as follows: Subtract the mean color value of the background image from the pixel value, multiply by the ratio of the standard deviation of the foreground and background images, and then add the mean color value of the foreground image.

[0166]

[0167] Among them, L ′ a ′ and b ′ L, a, and b are the luminance channel value, A color channel value, and B color channel value of the color adaptation map of the foreground Lab image, respectively; L, a, and b are the luminance channel value, A color channel value, and B color channel value of the foreground Lab image, respectively. and These represent the mean values ​​of the brightness channel, the A color channel, and the B color channel of the background Lab image, respectively. and These represent the mean values ​​of the brightness channel, the A color channel, and the B color channel of the foreground Lab image, respectively. and The standard deviations of the brightness channel, A color channel, and B color channel of the background Lab image are: and These represent the standard deviations of the brightness channel, A color channel, and B color channel of the foreground Lab image, respectively.

[0168] The color distribution of the obtained foreground image is converted from Lab space back to Rgb space, and then composited into the background image to obtain a composite image with color adaptation transformation at the foreground position;

[0169] 5) Use the color adaptation transformation to synthesize the graph, which is then output to the diffusion model based on ControlNet as further hints for the generation;

[0170] The ControlNet workflow is as follows: First, an encoder is used to transform the color-adapted transformation synthetic map input into a feature space, matching its size and dimensions to the input layer of Stable Diffusion, as a hint c. f The input to Control Net is c fBecause in Stable Diffusion, the original input image first passes through a simple encoder to map the input to a space before being input to the U-Net encoder, a similar operation is used here to form a similar input;

[0171] c f =E(c)

[0172] The weights of the U-Net encoder in the diffusion model are copied to form a new trainable copy Θ. c ;

[0173] Θ c =copy(Θ)

[0174] A zero-convolutional layer is added before each trainable copy, taking the color adaptation transformation composite image as input. The formula for the zero-convolutional layer is as follows:

[0175] Z(x;Θ z )=W·x+b

[0176] Where Θ z This represents the parameters of this zero convolutional layer;

[0177] Each ControlNet block contains a zero convolutional layer, a weight copy of the U-Net encoder, and another zero convolutional layer, such as... Figure 6 As shown; the formula is as follows:

[0178] y c =F(x;Θ)+Z(F(x+Z(c)) f ;Θ z1 );Θ c );Θ z2 )

[0179] The generated feature y c This is achieved by adding a pass condition c to the original feature x of the U-Net encoder. f The adjusted features are obtained by inserting a ControlNet structure after each decoding block of Stable Diffusion, which combines the color adaptation transformation features with the feature map of the main model through zero convolutional layers, resulting in the feature y. c It is then combined with the U-Net decoder as a cue for the decoder; its overall architecture is as follows: Figure 7 As shown;

[0180] like Figure 7As shown, the basic structure of the ControlNet module is as follows: ControlNet accepts two inputs, one is Condition (obtained by color adaptation of the foreground and background); the second is input (the input of the U-Net encoder in the backbone network of Stable Diffusion); the ControlNet module contains a total of 12 SD Encoder Blocks and one SD Middle Block. The number of Encoder Blocks in ControlNet is the same as the number of Encoder Blocks in the UNet decoder, and the parameters of the module are the same as those in UNet, which is equivalent to a copy used for further training of Condition;

[0181] like Figure 6 As shown, this is the detail of each Block. It accepts two inputs: c (from the previous block) and x. x has two inputs: prompt (features from the edge feature fusion module in this embodiment) and time embedding (the number of diffusion reconstructions). Each block includes a zero convolutional layer, an Encoder block copied from the Diffusion U-Net module, and finally another zero convolutional layer. The output is then sent to the Diffusion U-Net decoder.

[0182] 6) Input the fused image into the diffusion model to synthesize a high-quality X-ray image;

[0183] like Figure 8 As shown, the diffusion model used in this method consists of two stages; in the first stage, the diffusion model first trains an autoencoder and an autodecoder; the encoder first encodes the image into the latent space z. ′ In 0 = ε(I), the decoder then reconstructs them into the original image. In the second stage, the autoencoder is frozen, moving towards z. ′ Gaussian noise ∈ is gradually added to 0 to obtain z t ′ At this time, z t ′ It is Gaussian noise; the formula is shown below.

[0184]

[0185] Here, z t ′ z represents the data state at time t. ′ 0 represents the original data, and ∈ represents the data from the standard normal distribution. The noise obtained from sampling, while αt It is a coefficient that controls the noise level, and it usually decreases gradually as t increases (i.e., the noise level increases);

[0186] The obtained noise is input into the U-Net of the diffusion model. In the U-Net, a cross-attention layer is added and mapped to the intermediate layer of the U-Net, as shown in the following formula.

[0187]

[0188]

[0189] in, Indicates noise input. Represented as Mapped to an intermediate representation via a domain-specific encoder;

[0190] The formula for the noise denoising process is shown below.

[0191]

[0192] Where ∈ θ (z t (,t) is a U-Net network model used to predict noise data z. t Given time step t, the distribution of noise ∈;

[0193] After T rounds of denoising, z0 is obtained, and z0 is input to the decoder. In this process, the final synthesized X-ray image output is obtained;

[0194] like Figure 9 The image shown is a schematic diagram of synthesizing two foreground X-ray images of contraband items and one background X-ray image of luggage using this method. It can be seen that a high-quality X-ray image with overlapping items was finally obtained.

[0195] This embodiment proposes a method for synthesizing X-ray images of airport logistics, aiming to synthesize high-quality X-ray detection images with complex overlapping items, serving as a dataset for model training and for personnel to learn the use of X-ray detection images to identify hazardous materials. The method employs a diffusion model-based image synthesis approach, fusing foreground and background images into a highly realistic composite image through progressive noise addition and removal. An edge contour correction module is introduced, using foreground edge contour information as cues to stabilize the diffusion process of edge contour information, ensuring smooth edge transitions in the synthesized image even with overlapping items. Furthermore, a ControlNet-based color conversion module is incorporated, processing the color information of the foreground and background and feeding it back to ControlNet as cues for the diffusion model decoder, ensuring that the synthesized image's colors are closer to reality even with multiple foreground images. This method can be used as a dataset for training detection models, effectively addressing the limitations of existing X-ray image datasets in terms of quantity and diversity.

[0196] The same or similar labels correspond to the same or similar parts;

[0197] The terms used to describe positional relationships in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent.

[0198] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A high-quality image synthesis method based on a diffusion model, characterized in that, Includes the following steps: S1: Obtain the background image and at least one foreground image, and extract the mask image of each foreground image; Each of the foreground images is pasted into the background image to obtain a preliminary fused image, and the edge contour map of the preliminary fused image is extracted; S2: Establish an edge contour correction model, which includes an adaptive encoder and an edge feature fusion module connected in sequence; The mask map of each foreground image and the edge contour map of the preliminary fused image are input together into the adaptive encoder to obtain the contour feature map; The preliminary fused image is input into the U-Net encoder of the diffusion model to obtain intermediate feature maps; The contour feature map and the intermediate feature map are input together into the edge feature fusion module to obtain the contour fusion feature map; S3: Establish a color conversion model, which includes a Lab space color transformation module and a ControlNet module connected in sequence; The background image, along with each foreground image and its mask, are input into the Lab color transformation module to obtain a composite color adaptation map. The synthesized color adaptation map is input into the ControlNet module to obtain the color adaptation feature map; S4: Use the contour fusion feature map as a cue for the U-Net encoder of the diffusion model, and use the color adaptation feature map as a cue for the U-Net decoder of the diffusion model to obtain the improved diffusion model. The preliminary fused image is input into the improved diffusion model to obtain a high-quality synthesized image.

2. The high-quality image synthesis method based on a diffusion model according to claim 1, characterized in that, In step S1, the background image is specifically an X-ray image of luggage; the foreground image is specifically an X-ray image of contraband.

3. The high-quality image synthesis method based on a diffusion model according to claim 1, characterized in that, In step S1, the edge contour map of the preliminary fused image is extracted using the Canny edge detector.

4. The high-quality image synthesis method based on a diffusion model according to claim 1, characterized in that, In step S2, the adaptive encoder includes a Pixel Unshuffle downsampling block, a first feature extraction block, a first downsampling block, a second feature extraction block, a second downsampling block, a third feature extraction block, a third downsampling block, and a fourth feature extraction block connected in sequence, and each feature extraction block includes a convolutional layer and two residual blocks connected in sequence.

5. The high-quality image synthesis method based on a diffusion model according to claim 1, characterized in that, In step S2, the edge feature fusion module includes a first Transformer layer, a second Transformer layer, and a fully connected layer. The input to the first Transformer layer is an intermediate feature map, the input to the second Transformer layer is a contour feature map, and the outputs of the first and second Transformer layers are respectively connected to the input of the fully connected layer. The output of the fully connected layer is concatenated with the intermediate feature map and the contour feature map to obtain the contour fusion feature map.

6. The high-quality image synthesis method based on a diffusion model according to claim 1, characterized in that, In step S3, the step of obtaining the synthetic color adaptation map using the Lab space color transformation module includes: S3.1: Combine the background image with the mask image of each foreground image to obtain the image of each foreground position in the background; S3.2: Convert the images of each foreground position in the background and each foreground image from the RGB color space to the Lab color space respectively, and obtain the background Lab image and each foreground Lab image respectively; S3.3: Match the background Lab image with the mean and standard deviation of each foreground Lab image in the Lab color space to obtain the color adaptation map of each foreground Lab image respectively; S3.4: Convert the color adaptation map of each foreground Lab image from the Lab color space to the RGB color space, and stitch them together with the background image to obtain the image in the background after color adaptation transformation at the foreground position, which is used as the composite color adaptation map.

7. A high-quality image synthesis method based on a diffusion model according to claim 6, characterized in that, Step S3.3 includes: The mean value μ of the background Lab image and each foreground Lab image in the Lab color space is calculated using the color mean calculation formula. C The formula for calculating the average color value is as follows: Among them, C i This represents the value of the i-th pixel, where n is the number of pixels. According to the mean μ C Calculate the corresponding standard deviation σ C The specific calculation formula is as follows: The color adaptation map for each foreground Lab image is obtained using the following formula: Among them, L ′ a ′ and b ′ L, a, and b are the luminance channel value, A color channel value, and B color channel value of the color adaptation map of the foreground Lab image, respectively; L, a, and b are the luminance channel value, A color channel value, and B color channel value of the foreground Lab image, respectively. and These represent the mean values ​​of the brightness channel, the A color channel, and the B color channel of the background Lab image, respectively. and These represent the mean values ​​of the brightness channel, the A color channel, and the B color channel of the foreground Lab image, respectively. and The standard deviations of the brightness channel, A color channel, and B color channel of the background Lab image are: and These represent the standard deviations of the luminance channel, A color channel, and B color channel of the foreground Lab image, respectively.

8. The high-quality image synthesis method based on a diffusion model according to claim 1, characterized in that, In step S3, the step of obtaining the color adaptation feature map using the ControlNet module includes: First, a pre-defined encoder is used to convert the synthetic color adaptation map into a feature space, matching its size and dimensions to the input layer of the diffusion model, as shown in the following formula: c f =E(c) Among them, c f The prompt condition is E(c); E(c) is the preset encoder; c is the synthesized color adaptation map. Copy the weights Θ of the U-Net encoder in the diffusion model to form a new trainable copy Θ. c : The ControlNet module includes, in sequence: a first zero convolutional layer, a trainable copy Θ, and so on. c And the second zero convolutional layer; The prompt condition c f As input to the ControlNet module, the ControlNet module outputs a color adaptation feature map y. c Specifically: y c =F(x:Θ)+Z(F(x+Z(c f ;Θ z1 ):Θ c ):Θ z2 ) Where F represents the U-Net encoder of the diffusion model, x is the input of the U-Net encoder, and Θ is the parameter of the U-Net encoder; Z represents the zero convolutional layer, and Θ... z1 Θ represents the parameters of the first zero convolutional layer. z2 These are the parameters of the second zero convolutional layer.

9. The high-quality image synthesis method based on a diffusion model according to claim 1, characterized in that, In the U-Net encoder of the diffusion model, the intermediate layer of the U-Net encoder outputs the intermediate feature map, and the contour fusion feature map output by the edge feature fusion module is input into the cross attention layer of the U-Net encoder, and then mapped to the intermediate layer of the U-Net encoder as a cue for the U-Net encoder. In the U-Net decoder of the diffusion model, a ControlNet module is set after each decoding block of the U-Net decoder. The color adaptation feature map output by the ControlNet module is used as a cue for the U-Net decoder of the diffusion model to obtain the improved diffusion model.

10. A high-quality image synthesis method based on a diffusion model according to any one of claims 1 to 9, characterized in that, The diffusion model mentioned is specifically the Stable Diffusion diffusion model.

Images