Unified underlying vision pre-training method based on multi-scale diffusion model

Abstract

The invention discloses a unified underlying vision pre-training method based on a multi-scale diffusion model, and the method comprises the steps: constructing and training a multi-scale degradation robust variational auto-encoder (VAE) which is used for extracting the multi-scale hidden space representation of degradation robustness; training a degradation invariant image feature encoder for extracting visual semantic features irrelevant to degradation types; based on a pre-trained diffusion model backbone network, in combination with the robust hidden space representation and the visual semantic features, constructing and training a condition-controllable bottom layer visual diffusion model, and modeling a diffusion process by adopting an improved stochastic differential equation (SDE); and integrating the multi-scale degradation robust VAE with a bottom-layer visual diffusion model, and constructing a large multi-scale bottom-layer visual pre-training model. According to the method, unified pre-training of various underlying degeneration is realized, and the fidelity and robustness of image restoration are improved.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a unified underlying visual pre-training method based on a multi-scale diffusion model. Background Art

[0002] Image generation models based on deep learning techniques have achieved remarkable success in computer vision tasks, such as image restoration, image enhancement, and super-resolution. Many existing advanced methods use theories based on diffusion models, which generate or restore images by simulating a forward process of gradually adding noise to the image and learning an inverse denoising process.

[0003] Current underlying visual diffusion models often draw on the architecture of text-to-image generation models, such as performing diffusion and denoising within the latent space of a variational autoencoder (VAE). These models often rely on a text encoder to provide a description of the image content or instructions for the target restoration effect, which serves as a condition to control the denoising behavior of the diffusion model.

[0004] However, the existing technical solutions still have the following deficiencies:

[0005] 1. Limitations of Text Control: The core goal of the underlying vision task is to faithfully restore image content and remove degradations, rather than creating new content based on text descriptions. Over-reliance on text conditioning can cause the restored image to deviate from the original input image in detail. This problem is particularly prominent when text descriptions struggle to accurately capture complex image degradation features or inherent image content.

[0006] 2. Interference of degradation on semantic understanding: If attempts are made to automatically extract text descriptions from degraded low-quality images as a condition, the image degradation itself (such as blur, noise, rain or snow) may seriously interfere with the accurate extraction of semantic content, thereby generating inappropriate or erroneous text descriptions and misleading the restoration process.

[0007] 3. Mismatch between model and task characteristics: Many existing underlying visual diffusion models directly adopt the stochastic differential equation (SDE) or deterministic differential equation (ODE) modeling approach used for image generation tasks. For example, in SDEs, the end point of the diffusion process (i.e., the starting point of the denoising process) is often modeled as pure Gaussian noise. This modeling approach essentially treats the underlying visual task as a conditional image generation problem, failing to fully utilize the rich information inherent in low-quality images and failing to specifically design for the core goal of "de-degradation."

[0008] 4. Insufficient utilization of multi-scale information: Image degradation and content characteristics often exhibit different characteristics at different scales. Some existing single-scale diffusion models may find it difficult to simultaneously restore both global structure and local details.

[0009] 5. Generalization ability to unknown degradations: The performance and robustness of existing models may be limited when dealing with new degradations that have not been seen during training or complex combinations of multiple unknown degradations.

[0010] Therefore, it is necessary to develop a pre-training model and method that can overcome the above-mentioned defects and perform unified underlying visual tasks more accurately and robustly. Summary of the Invention

[0011] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and provide a unified underlying visual pre-training method based on a multi-scale diffusion model, aiming to improve the fidelity of image restoration and robustness to various complex degradations, and enable the model to more effectively learn the "de-degradation" sampling process from low-quality images to high-quality images.

[0012] In order to achieve the above object, the present invention adopts the following technical solutions:

[0013] In a first aspect, the present invention provides a unified underlying visual pre-training method based on a multi-scale diffusion model, comprising the following steps:

[0014] Constructing and training a multi-scale degradation robust variational autoencoder, wherein the multi-scale degradation robust variational autoencoder is used to extract a multi-scale latent space representation that is robust to degradation from an input degraded image;

[0015] Training a degradation-invariant image feature encoder, wherein the degradation-invariant image feature encoder is used to extract visual semantic features that are independent of degradation type from the degraded image;

[0016] Based on a pre-trained diffusion model backbone network, a conditionally controllable underlying visual diffusion model is constructed and trained, using the robust latent space representation extracted by the multi-scale degradation robust variational autoencoder as a first condition and the visual semantic features extracted by the degradation-invariant image feature encoder as a second condition. The training of the underlying visual diffusion model uses an improved stochastic differential equation to model the diffusion process, and the improved stochastic differential equation introduces a drift term based on the latent space features of the degraded image during the forward noisy process.

[0017] The multi-scale degradation robust variational autoencoder is integrated with the underlying visual diffusion model to construct a multi-scale underlying visual pre-training large model, so that image restoration can be performed at different scales using the corresponding robust latent space representation and the visual semantic features.

[0018] As a preferred technical solution, the training of the multi-scale degradation robust variational autoencoder includes:

[0019] The degraded images and their corresponding high-quality image pairs are used for training, and a loss function consisting of reconstruction loss, perceptual loss, and KL divergence regularization term is adopted.

[0020] As a preferred technical solution, when training the scale-degradation robust variational autoencoder, the parameters of its corresponding decoder are fixed.

[0021] As a preferred technical solution, the degradation-invariant image feature encoder is trained using a contrastive learning method by maximizing the feature similarity between pairs of images with the same content but different degradation types, and minimizing the feature similarity between pairs of images with different contents.

[0022] As a preferred technical solution, the forward noise addition process of the improved stochastic differential equation is expressed as:

[0023]

[0024] Among them, z t is the state of the image feature in the latent space at time step t, z LQ is the latent space feature of the degraded image obtained by the VAE encoder, β(t) is the time-dependent noise scheduling function, and h(t) is the function related to β(t) and the diffusion coefficient. Related functions, w t Describes the standard Brownian motion, ζ is the control of the z LQ The coefficient introduced by the intensity of information uncertainty;

[0025] The forward sampling formula is expressed as:

[0026]

[0027] Among them, z0 is the initial latent space feature corresponding to the high-quality image, and ∈ is random noise.

[0028] As a preferred technical solution, the training goal of the underlying visual diffusion model is to minimize the difference between the noise it predicts and the noise actually added in the forward noisy process.

[0029] As a preferred technical solution, the multi-scale underlying visual pre-training large model is processed using a diffusion model with different parameter quantities or network structures at different scales.

[0030] In a second aspect, the present invention provides a unified underlying visual pre-training system based on a multi-scale diffusion model, which is applied to the unified underlying visual pre-training method based on the multi-scale diffusion model, including a multi-scale degradation robust variational autoencoder construction module, a degradation invariant image feature encoder training module, a conditionally controllable underlying visual diffusion model construction module, and a multi-scale underlying visual pre-training large model integration module;

[0031] The multi-scale degradation robust variational autoencoder building module is used to construct and train a multi-scale degradation robust variational autoencoder, and the multi-scale degradation robust variational autoencoder is used to extract a multi-scale latent space representation that is robust to degradation from an input degraded image;

[0032] The degradation-invariant image feature encoder training module is used to train a degradation-invariant image feature encoder, and the degradation-invariant image feature encoder is used to extract visual semantic features that are independent of degradation type from the degraded image;

[0033] The conditionally controllable underlying visual diffusion model construction module is used to construct and train a conditionally controllable underlying visual diffusion model based on a pre-trained diffusion model backbone network, in combination with the robust latent space representation extracted by the multi-scale degradation robust variational autoencoder as a first condition, and the visual semantic features extracted by the degradation-invariant image feature encoder as a second condition; the training of the underlying visual diffusion model adopts an improved stochastic differential equation to model the diffusion process, and the improved stochastic differential equation introduces a drift term based on the latent space features of the degraded image during the forward noisy process;

[0034] The multi-scale underlying visual pre-training large model integration module is used to integrate the multi-scale degradation robust variational autoencoder with the underlying visual diffusion model to construct a multi-scale underlying visual pre-training large model, so that image restoration can be performed at different scales using the corresponding robust latent space representation and the visual semantic features.

[0035] In a third aspect, the present invention provides an electronic device, comprising:

[0036] at least one processor; and,

[0037] a memory communicatively connected to the at least one processor; wherein,

[0038] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the unified underlying visual pre-training method based on the multi-scale diffusion model.

[0039] In a fourth aspect, the present invention provides a computer-readable storage medium storing a program, which, when executed by a processor, implements the unified underlying visual pre-training method based on a multi-scale diffusion model.

[0040] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0041] 1. Improving restoration fidelity and detail consistency: This invention introduces a degradation-invariant image feature encoder to replace the text encoder, directly using the visual content of the input image itself as a guide, avoiding detail distortion or content inconsistency that may be caused by inaccurate text descriptions or differences in image and text modalities.

[0042] 2. Enhanced robustness to complex degradation: The degradation-robust VAE in this paper is designed to extract purer, degradation-insensitive image content representations, while the degradation-invariant image feature encoder focuses on extracting stable semantic information. The combination of these two makes the entire model more robust to a variety of known and even unknown complex degradations.

[0043] 3. More optimized diffusion process modeling: The SDE modeling method proposed in this paper, which is suitable for underlying visual tasks, introduces low-quality image information in the forward diffusion process, allowing the model to directly learn the conversion trajectory from "low quality" to "high quality" rather than simple conditional generation. This is more in line with the essence of image restoration tasks and is conducive to model learning to remove degraded features.

[0044] 4. Effective use of multi-level semantic information: The multi-scale architecture of the present invention enables the model to process image features at different granularities, taking into account both global structural consistency and local detail recovery, thereby improving the overall restoration quality.

[0045] 5. Improved generalization for unknown degradations: Since the model is dedicated to learning degradation-independent image content representations and semantic features, and repairing them directly from low-quality image features, it is expected to show better generalization performance for unknown degradation types that are not fully covered in the training data. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0047] Figure 1 Schematic diagram of the unified underlying visual pre-training method based on the multi-scale diffusion model of the present invention;

[0048] Figure 2This is a model architecture diagram of the multi-scale degradation robust variational autoencoder (VAE) in the method of the present invention.

[0049] Figure 3 This is a model architecture diagram of the robust VAE in the method of the present invention.

[0050] Figure 4 is a schematic diagram of a degradation-invariant image encoder in the method of the present invention;

[0051] Figure 5 This is the underlying diffusion model architecture diagram in the method of the present invention;

[0052] Figure 6 4 is a block diagram of a unified underlying visual pre-training system based on a multi-scale diffusion model according to an embodiment of the present invention;

[0053] Figure 7 2 is a structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0055] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0056] Please refer to Figure 1 This embodiment provides a unified underlying visual pre-training method based on a multi-scale diffusion model, including the following steps:

[0057] S1. Construct and train a multi-scale degradation robust variational autoencoder (VAE), which is used to extract a multi-scale latent space representation that is robust to degradation from an input degraded image.

[0058] For further information, please refer to Figure 2 and Figure 3This example first constructs a multi-scale, high-resolution VAE model. The VAE's encoder compresses the input image into a high-dimensional latent space, while the decoder reconstructs the image from the latent space representation. Existing diffusion models typically perform image generation and denoising in the VAE's latent space. Therefore, the properties of the VAE itself are fundamental to the subsequent construction and training of the underlying diffusion model.

[0059] Subsequently, a degradation-robust variational autoencoder is fine-tuned based on the pre-trained multi-scale VAE model. By pairing degraded and high-quality images, the encoder learns how to remove noise and extract degradation-independent latent space representations. The multi-scale autoencoder adopts a U-Net structure, with both the encoder and decoder divided into multiple layers. The encoder continuously reduces dimensionality with increasing depth, while the decoder does the opposite. Each intermediate-layer feature of the encoder passes through a corresponding fully connected layer to obtain a predicted mean and variance, constructing a probability distribution of latent space features. The sampled latent space features are then concatenated with the features output by the previous decoder layer and input into the next decoder layer for subsequent decoding. Features at different semantic levels are obtained from multiple layers, and visual features are compressed at multiple levels to improve the detail and quality of image reconstruction.

[0060] In a specific embodiment, the VAE adopts a hierarchical structure similar to U-Net. The encoder E contains multiple downsampling modules (e.g., E1, E2, E3) to gradually compress the input image x into latent space features of different scales. The decoder D contains multiple upsampling modules (e.g., D1, D2, D3) to reconstruct the image x^ from the latent space features. The features output by each intermediate layer of the encoder are passed through a small multi-layer perceptron (MLP) network to predict the mean μ and logarithmic variance log(σ) of the corresponding latent variable. 2 ). From this Gaussian distribution N(μ,σ 2 ) will be concatenated with the upsampled features of the corresponding level of the decoder and input to the next layer of the decoder.

[0061] The core goal of the robust encoder training process is to improve its robustness to different types of image degradation. This paper selects a variety of common degradation types, including but not limited to Gaussian blur, low resolution, rain and snow, JPEG compression distortion, and constructs a comprehensive training set of degraded image pairs. By introducing a multi-degradation cross-training strategy, the encoder can still extract latent space representations that can be reconstructed into high-quality images under various complex degradation conditions. To achieve degradation robustness, a large number of paired image data are used for training, each pair of data consists of a high-quality clear image x HQ and a degraded image x synthesized by it LQThe degradation types include but are not limited to Gaussian blur, low resolution, rainy and snowy weather, JPEG compression distortion and other degradations and their combinations. The training goal is to enable the encoder E to be able to transform the degraded image x into LQ Extract as close as possible to the corresponding high-quality image x HQ The latent space representation extracted from , and this representation can be reconstructed into a high-quality image x by the decoder D HQ .

[0062] Loss Function This can include:

[0063] 1. Reconstruction loss: such as L1 loss or MSE loss, which measures the reconstructed image With original high quality image x HQ The differences between:

[0064] 2. Perceptual loss: Use a pre-trained deep convolutional network (such as VGG-19) to extract the activation values ​​of the reconstructed image and the high-quality image at different feature layers and calculate the difference between them:

[0065]

[0066] Among them, φ i Represents the i-th feature layer of the VGG network.

[0067] 3. KL divergence regularization term: constrains the distribution of latent variables output by the encoder to be close to the standard normal distribution:

[0068]

[0069] The total loss is: During the fine-tuning stage, the parameters of the decoder D can be fixed and only the parameters of the encoder E can be optimized, prompting the encoder to learn to generate a latent space that is more robust to degradation.

[0070] S2. Train a degradation-invariant image feature encoder, which is used to extract visual semantic features from the degraded image that are independent of the degradation type. This step aims to solve the semantic bias and modality gap problems caused by the over-reliance on text encoders in existing methods. By training a special image encoder to extract stable visual semantic features that are independent of the degradation type, a robust degraded image semantic understanding channel is constructed. The degradation-invariant image feature encoder is trained using a contrastive learning strategy. The training data consists of positive sample pairs of images containing the same content but undergoing different types of degradation (including single degradation and mixed degradation), and negative samples of images with completely different content. The encoder based on the Swin V2 architecture maps the input image to a feature vector of fixed dimension.

[0071] The goal of contrastive learning is to maximize the similarity (e.g., cosine similarity) in feature space between positive pairs (i.e., images with the same content but different degradation) while minimizing their feature similarity with negative pairs (images with different content). In this way, the encoder is trained to focus on the intrinsic semantic content of the image and ignore its surface degradation, thereby extracting visual features that are invariant to degradation and can stably represent the core semantics of the image. These visual semantic features are then used as conditional information to replace the text embedding in traditional diffusion models, guiding the restoration process of the underlying visual diffusion model.

[0072] Please refer to Figure 4 , this embodiment trains an image encoder This allows similar feature vectors to be extracted for images with the same content but different degradation types, while significantly different feature vectors are extracted for images with different content. The training data is constructed as follows: Each batch contains N samples. Some of these samples are "positive pairs," meaning images with the same content but different random degradations applied. Others are "negative pairs," meaning images with content completely different from the positive samples.

[0073] Image Encoder Map the input image x into a feature vector Calculate the cosine similarity between the two to get the similarity matrix

[0074]

[0075] The similarity matrix M is converted into a probability distribution using the sigmoid function, and the model is optimized using the contrastive learning loss function. This enables it to extract semantic features that are not related to degradation for images with various degradations or even mixed degradations, so as to effectively guide the diffusion model to remove the degradation features contained in the image. Therefore, the loss function of this stage of training is:

[0076]

[0077] By optimizing this loss over a large number of images with different contents and different degradation types, Able to learn semantic features that are invariant to degradation.

[0078] S3. Based on the pre-trained diffusion model backbone network, combined with the robust latent space representation extracted by the multi-scale degradation robust variational autoencoder as the first condition, and the visual semantic features extracted by the degradation invariant image feature encoder as the second condition, a conditionally controllable underlying visual diffusion model is constructed and trained; the training of the underlying visual diffusion model adopts an improved stochastic differential equation to model the diffusion process, and the improved stochastic differential equation introduces a drift term based on the latent space features of the degraded image in the forward noisy process.

[0079] This step builds on a pretrained image-generating diffusion model backbone network (e.g., a stable diffusion model based on the U-Net architecture) and deeply integrates the degradation-robust VAE trained in the previous step with the degradation-invariant image feature encoder to construct a conditionally controllable latent space diffusion model optimized for the underlying visual task characteristics. The specific construction method is as follows: First, the low-quality image to be inpainted is fed into the encoder of the degradation-robust VAE trained in step S1 to extract its robust latent space representation. This representation is fed into the diffusion model as a spatial control condition (e.g., by injecting it into each layer of the diffusion model U-Net backbone using a ControlNet-like adapter structure), providing the U-Net with prior information about the image structure and low-level degradation patterns. Second, the low-quality image to be inpainted is fed into the degradation-invariant image feature encoder trained in step S2 to extract its visual semantic features. This visual semantic feature replaces the text-based conditional embedding in the original diffusion model and is integrated into the attention module of the diffusion model U-Net backbone through methods such as cross-attention mechanisms, providing high-level semantic guidance for the inpainting process.

[0080] Based on a degradation-robust variational autoencoder and a degradation-invariant image feature encoder, the present invention can construct a multi-structure conditionally controllable latent space underlying visual diffusion model designed for underlying visual tasks. The diffusion model is trained in the latent space provided by the robust variational autoencoder, which is more friendly to underlying visual tasks. In the conditionally controllable diffusion model, the forward noise addition process can be represented by the following stochastic differential equation:

[0081] dz t =f(t,Z t )dt+g(t)dw t ,

[0082] Among them, z t is the state of the latent space feature corresponding to the image x at time step t, f(t,z t ) is the drift term of the stochastic differential equation, g(t) is the diffusion term, and w tDescribes standard Brownian motion (obeying Gaussian distribution). The underlying visual diffusion model of the present invention is built based on the existing pre-trained image generation model SDXL, using the VP (variance preserving) SDE paradigm. Its forward noise addition process can be expressed by the following stochastic differential equation:

[0083]

[0084] β(t)∈[β min ,β max ] is a time-dependent noise scheduling function that determines the diffusion trajectory. Let You can get the above VP SDE z t Analytical solution:

[0085]

[0086] Where z0 is the original image and ∈ is random noise. Existing cultural graph models and underlying visual models based on diffusion algorithms use this formula to construct image generation or image de-degradation models, implementing the reverse SDE process:

[0087]

[0088] Among them, c T is the text control condition, It is a diffusion model with time-step input. Most existing diffusion-based models use the deterministic differential equation (ODE) paradigm instead of the SDE paradigm. This is because the underlying visual tasks do not require high randomness, and it can also accelerate the reverse process, completing the image restoration process more quickly.

[0089]

[0090] The ODE solver can further accelerate this process, resulting in higher quality images at a faster speed.

[0091] The key innovation lies in the improvement of the stochastic differential equations that the diffusion process itself relies on for the underlying visual task. The traditional SDE model usually gradually diffuses the latent space features of high-quality images into pure Gaussian noise during the forward noise addition process. The improved SDE proposed in this invention explicitly introduces the latent space features z based on low-quality (LQ) images during the forward noise addition process. LQ The forward SDE can be specifically expressed as:

[0092]

[0093]

[0094] Among them, h(t) is a The related function is used to describe the diffusion trajectory from low-quality images to high-quality images, and ζ is a key hyperparameter used to control the low-quality image information z LQ The uncertainty intensity and information content in the forward diffusion trajectory. Accordingly, the analytical solution of the improved SDE (i.e., the forward sampling formula for the forward sampling from z0 and z LQ Generate noisy samples z at any time t t ) can be expressed as:

[0095]

[0096] in, is standard Gaussian noise. This improved SDE modeling method changes the learning goal of the diffusion model from simply "generating images from noise (guided by text conditions)" to a more direct "de-sampling from noisy low-quality image features to high-quality image features." Therefore, the corresponding inverse denoising (i.e., image restoration) process is no longer from pure Gaussian noise z t Instead, it starts from a feature state (i.e. z T Still partly derived from z LQ ) and gradually and guidedly recover the target high-quality image feature z0. The training goal is to enable the diffusion model (U-Net) to accurately predict the high-quality and clear feature z0.

[0097] Please refer to Figure 5 (Single scale example), this embodiment is based on a pre-trained diffusion model U-Net backbone network p θ (z t ,t,c1,c2) ​​for modification and fine-tuning, where c1 is the spatial condition and c2 is the content semantic condition.

[0098] Conditional input:

[0099] Spatial condition c1: The low-quality image x to be repaired LQ Input into the encoder of the degradation robust VAE trained in Example 1 to obtain its robust latent space representation Will It is fed into the U-Net backbone as a spatial control condition (the output of each downsampling module connected to the U-Net through zero convolutional layers and crisscross attention layers).

[0100] Content condition c2: low-quality image x LQ Input to the degradation invariant image feature encoder trained in Example 2 Get its visual semantic features This feature replaces the text embedding in the original diffusion model and is injected into each Transformer module of the U-Net backbone through the cross-attention mechanism.

[0101] Improved SDE modeling and training:

[0102] The improved SDE described in the invention summary is used for forward denoising and backward denoising. HQ , first obtain its initial latent space feature z0 through the conventional VAE encoder. At the same time, the low-quality image x LQ Its latent space feature z is also obtained through the VAE encoder LQ The present invention is based on the analytical solution of the underlying visual SDE process:

[0103]

[0104] Get the degenerate latent variable z at any time t t The diffusion model needs to be based on z t And the guidance of spatial conditions c1 and content conditions c2, predict the latent variable z corresponding to the high-quality image HQ Based on a multi-scale degradation-robust variational autoencoder, the present invention constructs a corresponding multi-scale underlying visual pre-training large model, which performs image restoration operations simultaneously from multiple different semantic levels, with each level being responsible for restoring image features at different levels.

[0105] S4. Integrate the multi-scale degradation robust variational autoencoder with the underlying visual diffusion model to construct a multi-scale underlying visual pre-training large model, so that image restoration can be performed at different scales using the corresponding robust latent space representation and the visual semantic features.

[0106] This step deeply integrates the multi-scale degradation-robust VAE constructed in step S1 with the underlying visual diffusion model optimized in step S3 to construct an end-to-end multi-scale underlying visual pre-training large model. Specifically, the encoder of the multi-scale VAE extracts degradation-robust latent space features of varying resolutions at different downsampling levels (scales). These feature maps at different scales can be fed separately (or after appropriate adaptation) into multiple parallel-connected diffusion models at corresponding scales for denoising and restoration. Diffusion models of different scales can be designed with different network parameter counts and structural complexity based on the characteristics of the feature hierarchies they process. For example, models with larger parameters and wider receptive fields can be used to process deeper scales containing more abstract semantic information and complex degradation patterns; whereas, relatively lightweight models can be used to process shallower scales containing more image detail and local texture information. This multi-scale design enables the model to simultaneously refine image features at different levels (for example, maintaining global structural consistency, restoring mid-range texture, and enhancing high-frequency details), effectively balancing overall computational complexity with restoration effectiveness. Finally, the features of each scale after multi-scale diffusion processing are fused and reconstructed through the decoder of multi-scale VAE to generate the final high-quality output image.

[0107] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0108] Based on the same idea as the unified underlying visual pre-training method based on the multi-scale diffusion model in the above-mentioned embodiment, the present invention also provides a unified underlying visual pre-training system based on the multi-scale diffusion model, which can be used to execute the above-mentioned unified underlying visual pre-training method based on the multi-scale diffusion model. For ease of explanation, the structural diagram of the embodiment of the unified underlying visual pre-training system based on the multi-scale diffusion model only shows the parts related to the embodiment of the present invention. Those skilled in the art will understand that the illustrated structure does not constitute a limitation of the device, and may include more or fewer components than shown in the diagram, or combine certain components, or arrange the components differently.

[0109] See also Figure 6 In another embodiment of the present application, a unified underlying visual pre-training system 100 based on a multi-scale diffusion model is provided, the system comprising a multi-scale degradation robust variational autoencoder construction module 101, a degradation invariant image feature encoder training module 102, a conditionally controllable underlying visual diffusion model construction module 103, and a multi-scale underlying visual pre-training large model integration module 104;

[0110] The multi-scale degradation robust variational autoencoder construction module 101 is used to construct and train a multi-scale degradation robust variational autoencoder, wherein the multi-scale degradation robust variational autoencoder is used to extract a multi-scale latent space representation that is robust to degradation from an input degraded image;

[0111] The degradation invariant image feature encoder training module 102 is used to train a degradation invariant image feature encoder, wherein the degradation invariant image feature encoder is used to extract visual semantic features that are independent of degradation type from the degraded image;

[0112] The conditionally controllable underlying visual diffusion model construction module 103 is configured to construct and train a conditionally controllable underlying visual diffusion model based on a pre-trained diffusion model backbone network, using the robust latent space representation extracted by the multi-scale degradation robust variational autoencoder as a first condition and the visual semantic features extracted by the degradation-invariant image feature encoder as a second condition; the training of the underlying visual diffusion model uses an improved stochastic differential equation to model the diffusion process, and the improved stochastic differential equation introduces a drift term based on the latent space features of the degraded image during the forward noisy process;

[0113] The multi-scale underlying visual pre-training large model integration module 104 is used to integrate the multi-scale degradation robust variational autoencoder with the underlying visual diffusion model to construct a multi-scale underlying visual pre-training large model, so that image restoration can be performed at different scales using the corresponding robust latent space representation and the visual semantic features.

[0114] It should be noted that the unified underlying visual pre-training system based on the multi-scale diffusion model of the present invention corresponds one-to-one to the unified underlying visual pre-training method based on the multi-scale diffusion model of the present invention. The technical features and beneficial effects described in the above-mentioned embodiment of the unified underlying visual pre-training method based on the multi-scale diffusion model are all applicable to the embodiment of the unified underlying visual pre-training based on the multi-scale diffusion model. For specific contents, please refer to the description in the embodiment of the method of the present invention. No further details will be given here. This is hereby declared.

[0115] In addition, in the implementation of the unified underlying visual pre-training system based on the multi-scale diffusion model in the above-mentioned embodiment, the logical division of each program module is only an example. In actual application, the above-mentioned functions can be distributed to different program modules as needed, for example, for the convenience of corresponding hardware configuration requirements or software implementation. That is, the internal structure of the unified underlying visual pre-training system based on the multi-scale diffusion model is divided into different program modules to complete all or part of the functions described above.

[0116] See also Figure 7In one embodiment, an electronic device for implementing a unified underlying visual pre-training method based on a multi-scale diffusion model is provided. The electronic device 200 may include a first processor 201, a first memory 202 and a bus, and may also include a computer program stored in the first memory 202 and executable on the first processor 201, such as a unified underlying visual pre-training program 203 based on a multi-scale diffusion model.

[0117] In which, the first memory 202 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a magnetic memory, a magnetic disk, an optical disk, etc. In some embodiments, the first memory 202 can be an internal storage unit of the electronic device 200, such as a mobile hard disk of the electronic device 200. In other embodiments, the first memory 202 can also be an external storage device of the electronic device 200, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 200. Furthermore, the first memory 202 can also include both an internal storage unit of the electronic device 200 and an external storage device. The first memory 202 can not only be used to store application software and various types of data installed on the electronic device 200, such as the code of the unified underlying visual pre-training program 203 based on the multi-scale diffusion model, but can also be used to temporarily store data that has been output or is to be output.

[0118] In some embodiments, the first processor 201 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including a combination of one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and various control chips. The first processor 201 is the control core (Control Unit) of the electronic device, connecting the various components of the entire electronic device using various interfaces and lines, and executing or executing programs or modules stored in the first memory 202, as well as calling data stored in the first memory 202, to perform various functions of the electronic device 200 and process data.

[0119] Figure 7 Only the electronic device with components is shown, and it can be understood by those skilled in the art that Figure 7The structure shown does not constitute a limitation on the electronic device 200 , and the electronic device 200 may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0120] The unified underlying visual pre-training program 203 based on the multi-scale diffusion model stored in the first memory 202 of the electronic device 200 is a combination of multiple instructions. When running in the first processor 201, it can achieve the following:

[0121] Constructing and training a multi-scale degradation robust variational autoencoder, wherein the multi-scale degradation robust variational autoencoder is used to extract a multi-scale latent space representation that is robust to degradation from an input degraded image;

[0122] Training a degradation-invariant image feature encoder, wherein the degradation-invariant image feature encoder is used to extract visual semantic features that are independent of degradation type from the degraded image;

[0123] Based on a pre-trained diffusion model backbone network, a conditionally controllable underlying visual diffusion model is constructed and trained, using the robust latent space representation extracted by the multi-scale degradation robust variational autoencoder as a first condition and the visual semantic features extracted by the degradation-invariant image feature encoder as a second condition. The training of the underlying visual diffusion model uses an improved stochastic differential equation to model the diffusion process, and the improved stochastic differential equation introduces a drift term based on the latent space features of the degraded image during the forward noisy process.

[0124] The multi-scale degradation robust variational autoencoder is integrated with the underlying visual diffusion model to construct a multi-scale underlying visual pre-training large model, so that image restoration can be performed at different scales using the corresponding robust latent space representation and the visual semantic features.

[0125] Furthermore, if the modules / units integrated in the electronic device 200 are implemented as software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard drive, a magnetic disk, an optical disk, a computer memory, or a read-only memory (ROM).

[0126] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0127] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0128] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A unified underlying visual pre-training method based on a multi-scale diffusion model, characterized by: The steps include: Constructing and training a multi-scale degradation robust variational autoencoder, wherein the multi-scale degradation robust variational autoencoder is used to extract a multi-scale latent space representation that is robust to degradation from an input degraded image; Training a degradation-invariant image feature encoder, wherein the degradation-invariant image feature encoder is used to extract visual semantic features that are independent of degradation type from the degraded image; Based on a pre-trained diffusion model backbone network, a conditionally controllable underlying visual diffusion model is constructed and trained, using the robust latent space representation extracted by the multi-scale degradation robust variational autoencoder as a first condition and the visual semantic features extracted by the degradation-invariant image feature encoder as a second condition. The training of the underlying visual diffusion model uses an improved stochastic differential equation to model the diffusion process, and the improved stochastic differential equation introduces a drift term based on the latent space features of the degraded image during the forward noisy process. The multi-scale degradation robust variational autoencoder is integrated with the underlying visual diffusion model to construct a multi-scale underlying visual pre-training large model, so that image restoration can be performed at different scales using the corresponding robust latent space representation and the visual semantic features.

2. The unified underlying visual pre-training method based on the multi-scale diffusion model according to claim 1 is characterized in that: The training of the multi-scale degradation robust variational autoencoder comprises: The degraded images and their corresponding high-quality image pairs are used for training, and a loss function consisting of reconstruction loss, perceptual loss, and KL divergence regularization term is adopted.

3. The unified underlying visual pre-training method based on the multi-scale diffusion model according to claim 1 is characterized in that: When training the scale-degradation robust variational autoencoder, the parameters of its corresponding decoder are fixed.

4. The unified underlying visual pre-training method based on the multi-scale diffusion model according to claim 1 is characterized in that: The degradation-invariant image feature encoder is trained using a contrastive learning method by maximizing the feature similarity between pairs of images with the same content but different degradation types, and minimizing the feature similarity between pairs of images with different contents.

5. The unified underlying visual pre-training method based on the multi-scale diffusion model according to claim 1 is characterized in that: The forward noise addition process of the improved stochastic differential equation is expressed as: Among them, z t is the state of the image feature in the latent space at time step t, z LQ is the latent space feature of the degraded image obtained by the VAE encoder, β(t) is the time-dependent noise scheduling function, and h(t) is the function related to β(t) and the diffusion coefficient. Related functions, w t Describes the standard Brownian motion, ζ is the control of the z LQ The coefficient introduced by the intensity of information uncertainty; The forward sampling formula is expressed as: Among them, z0 is the initial latent space feature corresponding to the high-quality image, and ∈ is random noise.

6. The unified underlying visual pre-training method based on the multi-scale diffusion model according to claim 1, characterized in that: The training goal of the underlying visual diffusion model is to minimize the difference between its predicted noise and the noise actually added during the forward noisy process.

7. The unified underlying visual pre-training method based on the multi-scale diffusion model according to claim 1, characterized in that: The multi-scale underlying visual pre-training large model is processed using diffusion models with different parameter amounts or network structures at different scales.

8. A unified underlying visual pre-training system based on a multi-scale diffusion model, characterized by: A unified underlying visual pre-training method based on a multi-scale diffusion model applied to any one of claims 1-7, comprising a multi-scale degradation robust variational autoencoder construction module, a degradation invariant image feature encoder training module, a conditionally controllable underlying visual diffusion model construction module, and a multi-scale underlying visual pre-training large model integration module; The multi-scale degradation robust variational autoencoder building module is used to construct and train a multi-scale degradation robust variational autoencoder, and the multi-scale degradation robust variational autoencoder is used to extract a multi-scale latent space representation that is robust to degradation from an input degraded image; The degradation-invariant image feature encoder training module is used to train a degradation-invariant image feature encoder, and the degradation-invariant image feature encoder is used to extract visual semantic features that are independent of degradation type from the degraded image; The conditionally controllable underlying visual diffusion model construction module is used to construct and train a conditionally controllable underlying visual diffusion model based on a pre-trained diffusion model backbone network, in combination with the robust latent space representation extracted by the multi-scale degradation robust variational autoencoder as a first condition, and the visual semantic features extracted by the degradation-invariant image feature encoder as a second condition; the training of the underlying visual diffusion model adopts an improved stochastic differential equation to model the diffusion process, and the improved stochastic differential equation introduces a drift term based on the latent space features of the degraded image during the forward noisy process; The multi-scale underlying visual pre-training large model integration module is used to integrate the multi-scale degradation robust variational autoencoder with the underlying visual diffusion model to construct a multi-scale underlying visual pre-training large model, so that image restoration can be performed at different scales using the corresponding robust latent space representation and the visual semantic features.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can execute the unified underlying visual pre-training method based on the multi-scale diffusion model as described in any one of claims 1-7.

10. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the unified underlying visual pre-training method based on a multi-scale diffusion model described in any one of claims 1 to 7 is implemented.

Images