Multi-scale controllable image generation method and system based on submerged space diffusion model
By introducing a multi-scale content-style conditional injection mechanism into the latent space diffusion model, the problems of insufficient controllability and multi-task adaptability of image generation in existing technologies are solved, realizing high-quality, low-cost image generation and conversion, which is suitable for hardware environments with limited computing resources.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING NORMAL UNIVERSITY
- Filing Date
- 2026-01-12
- Publication Date
- 2026-05-05
AI Technical Summary
Existing latent space diffusion models have shortcomings in controllability and multi-task adaptability. They are difficult to simultaneously take into account the detailed texture of images and global semantics. The generated results are prone to content distortion or awkward style fusion. Furthermore, they have poor multi-task adaptability, high development costs, and weak generalization ability.
A multi-scale content-style conditional injection mechanism is adopted to perform diffusion modeling in the latent space, gradually recover the latent space features through a denoising network, and fuse the content conditional image and style conditional image at multiple feature scales to generate the target image.
It improves the controllability and generation quality of the image generation process, is suitable for scenarios with high requirements for detail and semantic consistency, reduces computational complexity and storage overhead, supports image generation under unpaired conditions, reduces data acquisition and annotation costs, and improves the model's versatility and adaptability.
Smart Images
Figure CN121982145A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision, deep learning, and image generation and editing, and in particular to a multi-scale controllable image generation method and system based on a latent space diffusion model. Background Technology
[0002] Image generation and editing technology is an important research direction in the field of computer vision, and it is widely used in digital content creation, medical image processing, film and television production, and other scenarios. Early image generation methods mainly relied on manual rules or traditional machine learning models, which made it difficult to simultaneously ensure generation quality and control accuracy. With the development of deep learning technology, generative adversarial networks (GANs) have been widely used for image generation tasks. However, these methods are prone to problems such as pattern collapse and unstable generation results during training, which limits their application in high-reliability scenarios.
[0003] In recent years, diffusion models have gradually become a core technology in the field of image generation due to their high-quality generation capabilities. Traditional diffusion models directly perform noise addition and denoising processes in pixel space, which can generate images with rich details, but suffer from high computational complexity and slow inference speed. To address this deficiency, the latent space diffusion model (LDM) was proposed. LDM compresses the image to a low-dimensional latent space through a variational autoencoder (VAE) and then performs the diffusion process in the latent space, which significantly reduces computational costs while preserving high generation quality.
[0004] However, existing latent space diffusion models still have shortcomings in terms of controllability and multi-task adaptability: First, the conditional guidance mechanism is singular, and most LDMs only support conditional injection at a single scale, making it difficult to simultaneously take into account the detailed texture and global semantics of the image, resulting in problems such as "content distortion" or "stiff style fusion" in the generated results.
[0005] Second, it has poor multi-task adaptability. For different tasks such as image coordination and style transfer, the condition module needs to be redesigned, which results in high development costs and weak generalization ability.
[0006] Third, the balance between content and style is difficult to control. In tasks such as style transfer and medical image conversion, situations such as "style covering content" or "content losing style" are likely to occur, which cannot meet the dual requirements of "fidelity" and "controllability" in practical applications. Summary of the Invention
[0007] The purpose of this invention is to overcome the shortcomings of the prior art and provide a multi-scale controllable image generation method and system based on a latent space diffusion model.
[0008] The objective of this invention can be achieved through the following technical solutions:
[0009] A multi-scale controllable image generation method based on a latent space diffusion model includes the following processing stages:
[0010] In the first stage, the input image, content condition image, and style condition image corresponding to the target image processing task are obtained; and the images are subjected to necessary preprocessing operations to obtain a multi-source condition image suitable for subsequent feature modeling.
[0011] The second stage involves mapping the input image to the latent space and performing a diffusion modeling process in the latent space, where the denoising network gradually recovers the latent space features during the backdiffusion process. Simultaneously, the content conditional image and style conditional image are input into the conditional injector, and the content features and style features are fused at multiple feature scales. The constructed multi-scale conditional feature representation is injected into the denoising network to guide the backdiffusion process of the latent space.
[0012] The third stage: Based on the latent space features obtained from the latent space denoising process and the multi-scale conditional feature representation, the latent space features are conditionally controlled to generate a target image with the same spatial resolution as the input image.
[0013] Furthermore, when the target image processing task is an image coordination task, the input image is a composite image, the content condition image is the foreground region of the composite image, and the style condition image is the complete background image after image inpainting; the image inpainting is used to supplement the background information in the composite image that is occluded by the foreground.
[0014] Furthermore, the preprocessing includes using Gaussian filtering to remove salt-and-pepper noise and Gaussian noise from the input image, and using adaptive histogram equalization to enhance image details; the resolution unification process uses bilinear interpolation to adjust the content conditional image and style conditional image to the same resolution as the input image.
[0015] Furthermore, the expression for the stepwise noise addition operation is:
[0016]
[0017] In the formula, for Features after adding noise for Features after adding noise Parameters for controlling noise intensity for Gaussian noise injected step by step.
[0018] Furthermore, the conditional injector is a multi-scale content-style conditional injector, which standardizes and fuses content features and style features at different scales. The fusion process can satisfy the following expression:
[0019]
[0020] In the formula, For the first The fusion characteristics For the first The stylistic features of the layers For the first The content characteristics of the layer This represents the function for calculating standard deviation. This represents the function for calculating the mean.
[0021] Furthermore, the latent space mapping module is a VAE encoder, and the decoder is a VAE decoder; the input features of each layer of the decoder can satisfy the following relationship:
[0022]
[0023]
[0024] In the formula, for The input features of the decoder layer, For the first Layer fusion characteristics, For the decoder Output features of the -1 layer This indicates a zero-initialization convolution operation. For the latent input features of the decoder, For the latent fusion characteristics of MCSI, These are the initial latent space features.
[0025] Furthermore, the MCSI and VAE encoders have the same number of layers and initial weight configuration. The feature fusion process covers shallow texture features and deep semantic features. Shallow features are used to preserve image detail texture information, while deep features are used to ensure image semantic consistency. The skip connection is used to precisely align and superimpose the multi-scale fused features with the corresponding layer features of the decoder.
[0026] Furthermore, the denoising network is a pre-trained denoising U-Net network, and its optimization objective satisfies the following expression:
[0027]
[0028] The initial latent space features output by the VAE encoder. This is information that integrates content and style conditions. It is Gaussian noise. For pre-training the denoising U-Net network, for Latent space features after adding noise To add noise steps, For the expected calculation, It is the square of the L2 norm.
[0029] Furthermore, when the target image processing task is a style transfer task, a white balance correction task, or a medical image conversion task:
[0030] When it is a style transfer task that preserves the original style, both the input image and the content condition image are the original images to be transferred, and the style condition image is the target style image. The target image retains the semantic content of the original images to be transferred and has the style features of the target style image.
[0031] When performing a white balance correction task, both the input image and the content condition image are images with color casts, and the style condition image is a reference image with normal white balance. The reference image and the image with color casts do not need to satisfy a pairing relationship.
[0032] When performing a medical image conversion task, the input image and the content condition image are both source modal medical images, and the style condition image is a target modal medical image. The source modality and the target modality include any two combinations of T1-weighted MRI images, T1-weighted enhanced MRI images, T2-weighted MRI images, and T2-Flair MRI images.
[0033] The present invention also provides a system for implementing the multi-scale controllable image generation method based on the latent space diffusion model as described above, comprising:
[0034] Multi-source image acquisition and feature preprocessing module: used to acquire input images, content-conditional images and style-conditional images, and perform preprocessing operations such as noise removal and resolution unification;
[0035] Multi-scale conditional fusion and guided denoising module: used to map the input image to the latent space and perform noise addition and denoising processing, while performing multi-scale feature fusion on content conditions and style conditions;
[0036] Conditionally controlled injection and target image reconstruction module: used to generate and output the target image at the original resolution by fusing denoising features and multi-scale fusion features through the decoder.
[0037] Compared with the prior art, the present invention has the following advantages:
[0038] (1) This invention introduces a multi-scale content-style conditional injection mechanism in the latent space diffusion model, fuses content features and style features at different resolution layers, and injects the fused features into the corresponding layer of the decoder through skip connections, so that the generated image can maintain global semantic consistency while taking into account local texture details, thereby effectively improving the controllability and generation quality in the image generation process. It is suitable for image generation scenarios with high requirements for detail and semantic consistency.
[0039] (2) This invention unifies image processing tasks into a combination of content-conditional images and style-conditional images as input, enabling the same model structure to adapt to various image generation or conversion tasks. Without adjusting the main network structure, tasks such as image coordination, style transfer, white balance correction, and medical image conversion can be completed simply by configuring different conditional images, thereby reducing the development and maintenance costs of the model in multi-task applications and improving the model's versatility and adaptability.
[0040] (3) To address the problem of balancing content semantics and style features in existing methods, this invention standardizes content features and introduces a feature transfer mechanism based on style feature statistics. This allows content information and style information to maintain a relatively independent and controllable relationship during the fusion process, thereby ensuring the integrity of content semantics while achieving effective transfer of style features and reducing content distortion or style weakening.
[0041] (4) Based on the latent space diffusion model architecture, the present invention performs the diffusion process in the low-dimensional latent space. Compared with the direct diffusion operation in the pixel space, it can effectively reduce the computational complexity and storage overhead. At the same time, during the model training process, the parameters of some pre-trained modules can be fixed, and only the condition injection related modules are updated, thereby shortening the model training time and improving the training efficiency. It is suitable for deployment and application in hardware environments with limited computing resources.
[0042] (5) This invention supports image generation and conversion tasks under non-paired conditions. The style condition image and the input image do not need to satisfy a one-to-one correspondence, thereby reducing the dependence on paired training data, reducing data collection and annotation costs, and can still maintain a relatively stable generation effect under non-paired data conditions. It has good generalization ability and practical application value. Attached Figure Description
[0043] Figure 1 This is a flowchart illustrating the multi-scale controllable image generation method based on the latent space diffusion model provided in this embodiment of the invention.
[0044] Figure 2 This is a flowchart illustrating a multi-source image acquisition and feature preprocessing process provided in an embodiment of the present invention;
[0045] Figure 3 This is a flowchart illustrating a multi-scale conditional fusion and guided denoising process provided in an embodiment of the present invention:
[0046] Figure 4 This is a flowchart illustrating the process of a conditionally controlled injection and target image reconstruction module provided in an embodiment of the present invention. Detailed Implementation
[0047] The principles and features of the present invention are described below with reference to the accompanying drawings. The listed embodiments are only used to explain the present invention and are not intended to limit the scope of the present invention.
[0048] Example 1
[0049] like Figure 1 As shown, this embodiment provides a multi-scale controllable image generation method based on a latent space diffusion model, characterized by including the following processing stages:
[0050] In the first stage S1, the input image, content condition image, and style condition image corresponding to the target image processing task are acquired; and necessary preprocessing operations are performed on the images to obtain a multi-source condition image suitable for subsequent feature modeling.
[0051] First, the image generation task instruction is received, clarifying that the core task requirement is to eliminate style conflicts between the foreground and background in a composite image. Based on this task scenario, the rules for the three core inputs are determined:
[0052] Input image: Select a composite image of foreground and background stitched together. It must contain obvious stylistic conflict features. The data can be from public datasets or custom-collected and stitched images.
[0053] Content-conditional image: The foreground slice in the input composite image is used as the content condition, and the semantic segmentation algorithm is used to accurately extract the content, ensuring that the core semantic content is complete and undisturbed.
[0054] Style conditional image: Using the restored complete background as the style condition, the foreground occlusion area is filled by the mask restoration algorithm to obtain a background image with a unified style and no clutter.
[0055] Optionally, for custom-captured composite images, the original foreground and background materials need to be captured using a high-definition camera and manually stitched together to form a composite image with stylistic conflicts; uniform lighting must be ensured during the capture process to avoid additional style interference caused by lighting differences.
[0056] To improve the robustness of subsequent latent space feature extraction, targeted quality optimization operations are performed on the input image to eliminate noise interference and enhance detailed features.
[0057] The first step, noise detection and removal: such as Figure 2 As shown, noise detection is first performed on the input image to determine the type and intensity of noise. For common photosensitive noise, a Gaussian filtering algorithm is used for denoising. By reasonably setting the filter window size and standard deviation parameters, noise is effectively removed while preserving the core texture details of the foreground and background, avoiding blurring of details caused by over-filtering.
[0058] Preferably, if mixed noise is detected, a combination of Gaussian filtering and median filtering can be used to specifically eliminate different types of noise.
[0059] The second step is detail enhancement: the contrast of the denoised input image is evaluated. If it is determined to be a low-contrast image, an adaptive histogram equalization algorithm is applied to enhance local contrast and texture details. By setting reasonable contrast limits and block size parameters, accurate equalization of local image regions is achieved, enhancing edge textures and background details, and providing richer feature information for subsequent multi-scale feature extraction.
[0060] Optionally, if the image has significant brightness deviation, a brightness correction step can be added before enhancement to optimize the overall brightness balance of the image.
[0061] To ensure that MCSI can accurately capture and fuse multi-source features and avoid feature misalignment caused by dimensional differences, firstly, a dimensional alignment benchmark is determined. The resolution of the input image after quality enhancement and noise processing is used as the benchmark, and a uniform target resolution is preferably set to ensure a balance between efficiency and quality in subsequent latent space mapping and feature extraction. If the original resolution of the input image is too high or too low, the input image can be initially scaled, and then the scaled resolution can be used as the benchmark.
[0062] Secondly, a resolution unification operation is performed: for content-conditional images and style-conditional images, the size is adjusted by selecting an appropriate interpolation algorithm based on their scaling ratio with the baseline resolution.
[0063] Preferably, when the scaling ratio is small, a bilinear interpolation algorithm is used to balance computational efficiency and image smoothness, ensuring that style and content features are not significantly distorted.
[0064] Optionally, if the scaling ratio is large, bicubic interpolation or Lanczos interpolation algorithms are preferred. By using multi-neighbor pixel weighted calculation, the texture details and edge sharpness of the image are better preserved, and problems such as blurring and jagged edges after scaling are avoided.
[0065] During the adjustment process, the image aspect ratio must be strictly maintained. If the original aspect ratio deviates significantly from the baseline, first crop the central area of the image according to the baseline ratio to ensure that core features are not lost, and then perform interpolation scaling. Finally, perform pixel value normalization and format unification.
[0066] Phase 2 S2, as Figure 3 As shown, the input image is mapped to the latent space, and a diffusion modeling process is performed in the latent space, wherein the denoising network gradually recovers the latent space features during the backdiffusion process; at the same time, the content conditional image and style conditional image are input into the conditional injector, and the content features and style features are fused at multiple feature scales. The constructed multi-scale conditional feature representation is injected into the denoising network to guide the backdiffusion process of the latent space.
[0067] Preferably, the input image, after multi-scale spatial dimension alignment, is input into the latent space mapping module. A pre-trained VAE encoder is preferred, employing a depthwise convolutional structure. By appropriately setting the convolutional kernel size and stride, dimensionality compression is achieved while preserving the core semantic and texture features of the input image, ultimately outputting a fixed-dimensional initial latent space feature representation. .
[0068] Optionally, to enhance feature representation capabilities, a self-attention mechanism can be introduced deep into the encoder to strengthen feature extraction from core regions such as the foreground subject and key background areas, ensuring the initial latent space features are fully utilized. It can accurately represent the core information of the input image; for single-channel scenarios such as medical images, a lightweight VAE encoder adapted to single-channel input can be selected to reduce redundant calculations.
[0069] Regarding the above initial latent space characteristics Perform a stepwise noise-adding operation to construct a noise evolution sequence.
[0070] Preferably, a cosine noise scheduling strategy is adopted, which can make the noise intensity change smoothly with the number of diffusion steps, thereby improving the stability of subsequent denoising modeling.
[0071] Optionally, for scenarios with high requirements for generation speed, a linear noise scheduling strategy can be used to simplify the calculation process.
[0072] The noise addition process follows the Markov chain property, where each noise addition step generates noisy latent space features based on the previous feature state and the current noise intensity. Its core formula is as follows:
[0073]
[0074] In the formula, for Features after adding noise for Features after adding noise Parameters for controlling noise intensity, for Gaussian noise injected step by step.
[0075] Preferably, the noise addition process can be simplified by calculating the cumulative product. First, calculate... cumulative value The formula is as follows:
[0076]
[0077] Based on cumulative value You can directly get any number of steps. Noisy features and initial features The relationship is straightforward and requires no iterative steps; the formula is as follows:
[0078]
[0079] In the formula, for Noisy latent space features This simplified method, using standard Gaussian noise, can significantly improve the efficiency of noise addition processing.
[0080] Optionally, the total number of diffusion steps can be dynamically adjusted according to the task's requirements for generation quality: in high-fidelity generation scenarios, the total number of diffusion steps can be increased to further optimize noise evolution details; in fast inference scenarios, the total number of diffusion steps can be decreased to significantly improve processing speed. In addition, an adaptive noise addition mechanism can be introduced to increase the local noise addition intensity for semantically complex regions (such as regions with rich foreground details) and decrease the noise addition intensity for semantically simple regions (such as solid color backgrounds) to improve the targeting of noise addition.
[0081] The dimension-aligned content conditional image and style conditional image are simultaneously input into MCSI. This injector preferably employs a multi-scale feature extraction network that matches the VAE encoder structure, ensuring that the extracted conditional features are compatible with the latent space feature scales, and extracting content features at multiple feature scales. With style characteristics The shallow network focuses on extracting texture detail features (such as foreground edge texture and background material texture), while the deep network focuses on extracting global semantic features (such as foreground category and background scene type), ensuring that the extracted features cover the complete information dimension from details to the whole.
[0082] Preferably, a coordinate attention module can be introduced into the key layer of the feature extraction network to enhance feature capture of the core region and suppress interference from irrelevant regions.
[0083] Optionally, for devices with low computing power, a lightweight feature extraction network can be used to reduce the amount of computation through depthwise separable convolutions while retaining the core feature extraction capabilities; the extracted features can also be L2 normalized to improve the generalization and stability of the features.
[0084] At each of the above feature scales, the extracted content features With style characteristics Perform fusion processing to generate multi-scale fusion conditional features of the corresponding scale. A standardized fusion strategy is preferred, as shown in the formula below. First, mean-variance standardization is performed on both content features and style features. Then, mean-variance transfer of style features is achieved through feature weighted fusion, which can effectively avoid the problems of "style covering content" or "content losing style".
[0085]
[0086] In the formula, For the first Layer fusion characteristics, For the first The stylistic features of the layers For the first The content characteristics of the layer This represents the function for calculating standard deviation. This represents the function for calculating the mean.
[0087] Optionally, the fusion weights of content features and style features can be dynamically adjusted according to task requirements: for image coordination tasks, the weight of style features can be appropriately increased to strengthen background style guidance; for style transfer tasks, the weights can be balanced to take into account both semantic preservation and style transfer effects; for high-fidelity content generation tasks, the weight of content features can be increased to prioritize content integrity. In addition, a scale-adaptive fusion mechanism can be introduced, using adaptive weight fusion for shallow texture features and fixing the content weight as the priority for deep semantic features, further improving the targeting of fusion.
[0088] Multi-scale fusion conditional features at various scales The denoising network is injected through skip connections and a pre-trained U-Net network is used. This network has a strong feature reconstruction capability and can accurately recover clean features in the latent space.
[0089] Optionally, for high-resolution generation scenarios, the U-Net++ network can be used to improve feature recovery accuracy through deep supervision; for low-computing-power scenarios, a lightweight U-Net can be used to reduce inference costs.
[0090] During the denoising process, the denoising network combines multi-scale fusion conditional features. With the current diffusion time step Encoding information to predict injected noise in noisy features. Then, the clean latent space features are recovered by reverse denoising using the denoising formula. The core denoising formula is as follows:
[0091]
[0092] In the formula, The characteristics of the restored clean latent space. for step The cumulative product value is calculated. As the diffusion steps advance, the latent space features are gradually optimized to simultaneously align with content semantics and style guidance. A cross-attention mechanism is introduced at the decoding end of the denoising network to strengthen the interaction between multi-scale fusion conditional features and latent space features, thereby improving the accuracy of conditional guidance.
[0093] The preferred training optimization objective for the denoising network is MSE loss, which improves denoising accuracy and the effectiveness of conditional guidance by minimizing the difference between predicted and actual noise. The loss function formula is as follows:
[0094]
[0095] The initial latent space features output by the VAE encoder. This is information that integrates content and style conditions. It is Gaussian noise. For pre-training the denoising U-Net network, for Latent space features after adding noise To add noise steps, For the expected calculation, It is the square of the L2 norm.
[0096] Optionally, for task scenarios with texture blurring issues, a combination of MSE loss and perceptual loss can be used. The perceptual loss uses the feature distance extracted by the VGG network to improve the texture detail richness of the generated features. For small sample training scenarios, contrastive learning loss can also be introduced to enhance the discriminativeness of the features.
[0097] After completing all the preset diffusion denoising steps, the final stable latent space features are output.
[0098] Preferably, a feature consistency verification step can be added before output. By calculating the L2 distance between adjacent denoised features, the stability of the output features is ensured. If the threshold is not reached, an additional 10-20 denoising iterations can be added until the features are stable. This feature fully integrates the semantic constraints of content conditions and the style features of style conditions, and has good representational consistency. It can be directly used as the core input for the third-stage target image reconstruction, providing a reliable latent space feature foundation for the subsequent generation of a style-unified and semantically complete target image.
[0099] The third stage S3: Based on the latent space features obtained from the latent space denoising process and the multi-scale conditional feature representation, the latent space features are conditionally controlled to generate a target image with the same spatial resolution as the input image.
[0100] like Figure 4 As shown, the fusion conditional features generated in the second stage at various scales are... By injecting skip connections into the corresponding resolution layer of the decoder, it is ensured that conditional features can continuously guide feature reconstruction throughout the entire decoding process.
[0101] Preferably, the decoder adopts a transposed convolutional upsampling structure symmetrical to the VAE encoder, with the number of input channels in each layer matching the number of output channels in the corresponding layer of the encoder, ensuring compatibility of feature dimensions; before injection, the fused conditional features are fused through a 1×1 zero-initialization convolution operation. The number of channels is adjusted to match the number of input channels of the corresponding layer in the decoder to avoid feature interference caused by channel dimension mismatch.
[0102] The decoding operation is performed using the VAE decoder, and the latent space mapping module corresponds to the VAE encoder. Both have the same number of layers and consistent initial weight configurations to ensure compatibility between feature encoding and decoding. The input features of each layer of the decoder are constructed strictly according to the following formula to achieve precise alignment and overlay of multi-scale fused features and decoder features:
[0103]
[0104]
[0105] In the formula, for The input features of the decoder layer, For the first Layer fusion characteristics, For the decoder The output features of the layer This indicates a zero-initialization convolution operation. For the latent input features of the decoder, For the latent fusion characteristics of MCSI, These are the initial latent space features.
[0106] The above fusion process is achieved through skip connections. The multi-scale fusion features are precisely aligned with and superimposed on the corresponding layer features of the decoder, covering both shallow texture features and deep semantic features: shallow fusion features can preserve the image's detailed texture information, ensuring the richness of details in the generated image; deep fusion features can guarantee the semantic consistency of the image, avoiding semantic distortion problems.
[0107] Optionally, LayerNorm normalization can be added to the input features of each layer after feature stacking to further improve decoding stability.
[0108] Based on the input features of each layer of the constructed decoder, the VAE decoder performs layer-by-layer transposed convolutional upsampling operations to gradually improve the feature resolution. After each upsampling layer, a nonlinearity is introduced through the ReLU activation function to enhance the feature representation capability; a self-attention module is introduced in the high-resolution feature layer to enhance the reconstruction accuracy of local texture details and avoid edge blurring and texture distortion. The upsampling process follows the rule of "doubling the resolution and halving the number of channels," gradually restoring the resolution of the initial latent space features to the target resolution consistent with the input image.
[0109] The features output from the last layer of the decoder are mapped by the Sigmoid activation function to normalize the feature values, and then converted to the pixel value range of an 8-bit RGB image through linear scaling to obtain the generated image.
[0110] Example 2
[0111] This embodiment provides a system for a multi-scale controllable image generation method based on a latent space diffusion model, as shown in Embodiment 1, comprising:
[0112] Multi-source image acquisition and feature preprocessing module: used to acquire input images, content-conditional images and style-conditional images, and perform preprocessing operations such as noise removal and resolution unification;
[0113] Multi-scale conditional fusion and guided denoising module: used to map the input image to the latent space and perform noise addition and denoising processing, while performing multi-scale feature fusion on content conditions and style conditions;
[0114] Conditionally controlled injection and target image reconstruction module: used to generate and output the target image at the original resolution by fusing denoising features and multi-scale fusion features through the decoder.
[0115] It should be noted that the specific content and beneficial effects of the system in this application can be found in the above method embodiments, and will not be repeated here.
[0116] The above description is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the protection scope of the present invention.
Claims
1. A multi-scale controllable image generation method based on a latent space diffusion model, characterized in that, The processing includes the following stages: In the first stage, the input image, content condition image, and style condition image corresponding to the target image processing task are obtained; and the images are subjected to necessary preprocessing operations to obtain a multi-source condition image suitable for subsequent feature modeling. The second stage involves mapping the input image to the latent space and performing a diffusion modeling process in the latent space, where the denoising network gradually recovers the latent space features during the backdiffusion process. Simultaneously, the content conditional image and style conditional image are input into the conditional injector, and the content features and style features are fused at multiple feature scales. The constructed multi-scale conditional feature representation is injected into the denoising network to guide the backdiffusion process of the latent space. The third stage: Based on the latent space features obtained from the latent space denoising process and the multi-scale conditional feature representation, the latent space features are conditionally controlled to generate a target image with the same spatial resolution as the input image.
2. The multi-scale controllable image generation method based on the latent space diffusion model according to claim 1, characterized in that, When the target image processing task is an image coordination task, the input image is a composite image, the content condition image is the foreground region of the composite image, and the style condition image is the complete background image after image restoration processing. Image inpainting is used to supplement background information that has been obscured by the foreground in a composite image.
3. The multi-scale controllable image generation method based on the latent space diffusion model according to claim 1, characterized in that, Preprocessing includes using Gaussian filtering to remove salt-and-pepper noise and Gaussian noise from the input image, and enhancing image details through adaptive histogram equalization; resolution unification processing uses bilinear interpolation to adjust the content-conditioning image and style-conditioning image to the same resolution as the input image.
4. The multi-scale controllable image generation method based on the latent space diffusion model according to claim 1, characterized in that, The expression for the stepwise noise addition operation is: In the formula, for Features after adding noise for Features after adding noise Parameters for controlling noise intensity, for Gaussian noise injected step by step.
5. The multi-scale controllable image generation method based on the latent space diffusion model according to claim 1, characterized in that, The conditional injector is a multi-scale content-style conditional injector that standardizes and fuses content features and style features at different scales. The fusion process can satisfy the following expression: In the formula, For the first Layer fusion characteristics, For the first The stylistic features of the layers For the first The content characteristics of the layer This represents the function for calculating standard deviation. This represents the function for calculating the mean.
6. The multi-scale controllable image generation method based on the latent space diffusion model according to claim 1, characterized in that, The latent space mapping module is the encoder of the VAE, and the decoder is the decoder of the VAE; the input features of each layer of the decoder can satisfy the following relationship: In the formula, for The input features of the decoder layer, For the first Layer fusion characteristics, For the decoder Output features of the -1 layer This indicates a zero-initialization convolution operation. For the latent input features of the decoder, For the latent fusion characteristics of MCSI, These are the initial latent space features.
7. The multi-scale controllable image generation method based on the latent space diffusion model according to claims 5-6, characterized in that, The MCSI and VAE encoders have the same number of layers and initial weight configuration. The feature fusion process covers shallow texture features and deep semantic features. Shallow features are used to preserve image detail texture information, while deep features are used to ensure image semantic consistency. The skip connection is used to accurately align and superimpose the multi-scale fused features with the corresponding layer features of the decoder.
8. The multi-scale controllable image generation method based on the latent space diffusion model according to claim 6, characterized in that, The denoising network is a pre-trained denoising U-Net network, and its optimization objective satisfies the following expression: In the formula, The initial latent space features output by the VAE encoder. This is information that integrates content and style conditions. It is Gaussian noise. For pre-training the denoising U-Net network, for Latent space features after adding noise To add noise steps, For the expected calculation, It is the square of the L2 norm.
9. The multi-scale controllable image generation method based on the latent space diffusion model according to claim 1, characterized in that, When the target image processing task is a fidelity style transfer task, white balance correction task, or medical image conversion task: When it is a style transfer task that preserves the original style, both the input image and the content condition image are the original images to be transferred, and the style condition image is the target style image. The target image retains the semantic content of the original images to be transferred and has the style features of the target style image. When performing a white balance correction task, both the input image and the content condition image are images with color casts, and the style condition image is a reference image with normal white balance. The reference image and the image with color casts do not need to satisfy a pairing relationship. When performing a medical image conversion task, the input image and the content condition image are both source modal medical images, and the style condition image is a target modal medical image. The source modality and the target modality include any two combinations of T1-weighted MRI images, T1-weighted enhanced MRI images, T2-weighted MRI images, and T2-Flair MRI images.
10. A system for implementing the multi-scale controllable image generation method based on the latent space diffusion model as described in any one of claims 1-9, characterized in that, include: Multi-source image acquisition and feature preprocessing module: used to acquire input images, content-conditional images and style-conditional images, and perform preprocessing operations such as noise removal and resolution unification; Multi-scale conditional fusion and guided denoising module: used to map the input image to the latent space and perform noise addition and denoising processing, while performing multi-scale feature fusion on content conditions and style conditions; Conditionally controlled injection and target image reconstruction module: used to generate and output the target image at the original resolution by fusing denoising features and multi-scale fusion features through the decoder.