Real scene image super-resolution reconstruction method based on diffusion model

Abstract

The application discloses a real scene image super-resolution reconstruction method based on a diffusion model, relates to the technical field of power management, and through construction of a physical transmission model of semi-transparent medium coupling degradation, accurate disassembly of three physical components of a real imaging process, establishment of accurate mathematical expression, deep binding of the light transmission physical process of the semi-transparent medium and the forward noise injection process of the diffusion model, matching of exclusive time step intervals for different physical components, realization of deep binding of the physical process and the diffusion model, construction of independent decoupling branches of the three components in the diffusion hidden space, combination of accurate division of the time step intervals, accurate decoupling of the three components, construction of a multi-constraint joint loss function, introduction of a physical transmission constraint in diffusion model training, and guarantee of decoupling rationality and reconstruction accuracy.

Description

Technical Field

[0001] This invention relates to the field of power management technology, specifically to a method for super-resolution reconstruction of real-scene images based on a diffusion model. Background Technology

[0002] Image super-resolution reconstruction, as a core underlying technology in the field of computer vision, is dedicated to restoring low-resolution images to high-resolution images. It has a wide range of critical applications in many fields such as security monitoring, vehicle vision, remote sensing imaging, and consumer electronics. With the continuous development of computer vision and deep learning technologies, image super-resolution reconstruction technology is also constantly improving to meet the growing demand for image quality in various fields.

[0003] However, existing diffusion super-resolution methods are designed for simple scenarios such as ideal downsampling degradation and Gaussian noise degradation. They suffer from numerous performance defects in real-world shooting environments. In real-world shooting environments, when the shooting path contains semi-transparent media such as glass, water mist, gauze, haze, or rain, complex light scattering and transmission coupling effects occur during the imaging process. The acquired LR image will exhibit multiple coupling degradations simultaneously. For such coupled degraded images, existing technologies have the following core problems: First, the degradation modeling is severely mismatched. The scattering interference from semi-transparent media is generally equated to ordinary additive Gaussian noise, lacking a physical model mathematical expression. This results in a significant discrepancy between the degradation modeling and the actual imaging process, leading to a substantial decrease in reconstruction accuracy. Second, precise component decoupling is impossible. Real coupled degraded images are composed of media scattering components, ambient stray light components, and scene intrinsic reflection components. Existing methods cannot effectively separate these three components, and cannot distinguish media transmission during super-resolution reconstruction. First, the inherent reflected light from the scene makes it impossible to simultaneously achieve artifact removal and detail preservation in the reconstruction results. Second, the diffusion process is completely disconnected from physical degradation. The current forward noise injection process of diffusion super-resolution has no physical constraints and is only a general Gaussian noise injection, which cannot correspond to the real degradation process of semi-transparent media. Third, the processing flow is cumbersome and has poor generalization ability. It usually adopts a cascaded scheme of preprocessing + super-resolution. First, the medium interference is removed by preprocessing modules such as dehazing, dereflection, and descattering, and then the preprocessed image is super-reconstructed. This not only increases the complexity of the processing flow, but also easily introduces the cumulative effect of preprocessing errors. Fourth, the decoupling process has no physical constraints and is prone to producing spurious solutions. Existing deep learning-based decoupling methods are mostly pure data-driven black box models. The decoupling process has no prior mathematical constraints of optical physics, which easily leads to spurious solutions that do not conform to the laws of light transmission. This results in unnatural artifacts in the reconstruction results, insufficient visual realism, and failure to meet the application requirements of real-world scenarios. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a real-scene image super-resolution reconstruction method based on a diffusion model. This method constructs a physical transport model with degraded coupling in a semi-transparent medium, accurately decomposes the three physical components of the real imaging process, establishes a precise mathematical expression, and deeply binds the physical process of light transport in the semi-transparent medium with the forward noise injection process of the diffusion model. It matches dedicated time step intervals for different physical components, achieving deep binding between the physical process and the diffusion model. Independent decoupling branches for the three components are constructed in the diffusion latent space, and precise decoupling of the three components is achieved through precise division of the time step intervals. A multi-constraint joint loss function is constructed, and physical transport constraints are introduced into the diffusion model training to ensure both rational decoupling and reconstruction accuracy.

[0005] To solve the above-mentioned technical problems, this invention provides the following technical solution: a real-scene image super-resolution reconstruction method based on a diffusion model, the method comprising: S1: Construct a physical transmission model for coupling degradation in a semi-transparent medium, decompose the coupled degradation image captured in the real scene into three independent physical components: medium scattering component, ambient stray light component, and scene intrinsic component, and establish a precise mathematical expression for the coupled degradation image. S2: Construct a forward process of the diffusion model with physical process constraints. Through the forward noise addition formula of the diffusion model with physical constraints, the physical process of light transmission in the semi-transparent medium is deeply bound to the forward noise injection process of the diffusion model. Independent noise mapping time step intervals are matched for the three components respectively, and a forward noise addition process with physical constraints is established. S3: Construct a diffusion UNet backbone network with three components decoupled in the latent space. In the latent space of the diffusion model, construct in parallel a decoupling branch for the medium scattering component, a decoupling branch for the ambient stray light component, and a decoupling branch for the scene intrinsic component. Configure an independent time step embedding module for each branch with a corresponding time step interval to achieve independent encoding and decoupling of the three components in the latent space. S4: Construct a multi-constraint joint loss function, integrate the diffusion noise prediction loss, medium physical transmission constraint loss, and super-resolution image fidelity loss through the general formula of the multi-constraint joint loss function, and perform end-to-end training and optimization of the diffusion UNet backbone network based on the multi-constraint joint loss function; S5: Input the low-resolution image with semi-transparent medium interference into the trained model. Through the backsampling process, remove the medium scattering component and environmental stray light component interference in the corresponding time step interval. Then, complete the super-resolution reconstruction based on the pure scene intrinsic features and output a high-resolution clear image end-to-end.

[0006] Furthermore, in S1, a physical transport model for degradation caused by coupling in a semi-transparent medium is constructed, and its model formula is as follows: ,in, Image pixel coordinates, The resulting low-resolution image is a product of coupling degradation. The intrinsic reflection components of the scene, i.e., a clear scene image without any media interference; The transmittance diagram of the medium has a value range of [0, 1], which characterizes the light transmission capability of a semi-transparent medium. The intrinsic transmission component of the scene, that is, the actual reflected light signal of the scene through the semi-transparent medium; The medium scattering component characterizes the interference signal formed by the scattering effect of the semi-transparent medium on incident light; The ambient stray light component represents the low-frequency interference signal formed by global atmospheric light and stray light reflected from the surface of the medium in the shooting environment.

[0007] Furthermore, in S2, the formula for the forward noise addition of physical constraints is: ,in, for Noisy images at time steps; Given an input image hour, Conditional probability distribution of noisy images at each time step; It follows a Gaussian distribution; The preset noise scheduling coefficient varies with time step. Increases monotonically and decreases; It is the identity matrix; This represents the large time step interval corresponding to the medium scattering component. This represents the mid-time step interval corresponding to the ambient stray light component. The three intervals are the small time step intervals corresponding to the scene's intrinsic components. They are continuous and do not overlap, covering all time steps of the diffusion model.

[0008] Furthermore, in S2, the three time step intervals are set based on the physical characteristics of degradation in the semi-transparent medium: the medium scattering component and the ambient stray light component belong to low-frequency, large-scale interference, corresponding to the high-noise distribution of the diffusion model in the large time step; the scene intrinsic components belong to high-frequency detail signals, corresponding to the low-noise distribution of the diffusion model in the small time step; the total time step of the diffusion model is set to... ,but .

[0009] Furthermore, in S3, the diffused UNet backbone network is a conditional diffused network structure, specifically including: Input encoding module: used to map the coupled, degraded, low-resolution input image to the latent space through convolutional layers to obtain initial latent features; Three-component decoupling branch module: Three independent decoupling branches are set up in parallel in the latent space, namely the medium scattering component decoupling branch, the ambient stray light component decoupling branch, and the scene intrinsic component decoupling branch. Each branch adopts a residual convolution block structure and is configured with an independent time step embedding module for the corresponding time step interval, which outputs the latent features of the medium scattering component, the latent features of the ambient stray light component, and the latent features of the scene intrinsic component, respectively. Time-step gated fusion module: Based on the current input time step, it generates gate weights for three branches, performs weighted fusion of the latent features of the three components, and obtains fused latent features; UNet encoding / decoding module: It adopts a symmetrical encoder-decoder structure, sets up skip connections, and inputs the conditional features of the low-resolution image into each layer of the encoder and decoder through hierarchical injection. It completes noise encoding and decoding based on the fused latent features. Noise prediction head: Convolutional layers are used to map the decoded latent features into a noise prediction map with the same number of channels as the input image; Super-resolution reconstruction head: Composed of upsampling convolutional layers and pixel reconstruction modules, it is used to perform upsampling reconstruction of high-resolution images based on pure scene intrinsic latent features at the backsampling end.

[0010] Furthermore, the medium physical transmission constraint loss includes component summation constraint loss, non-negative constraint loss, and atmospheric light smoothing constraint loss; wherein the component summation constraint loss is used to constrain the superposition of the three components obtained by decoupling to be consistent with the input image, the non-negative constraint loss is used to constrain the transmittance obtained by decoupling and the pixel value of each component to be non-negative, and the atmospheric light smoothing constraint loss is used to constrain the environmental stray light component to be a globally smooth low-frequency component.

[0011] Furthermore, in S4, the general formula for the multi-constraint joint loss function is: ,in, Total loss; , , These are preset weighting coefficients used to balance the optimization priorities of each loss term; This is the noise prediction loss, used to constrain the noise prediction accuracy of the model; This is the physical transmission constraint loss of the medium, used to ensure that the decoupling process conforms to the laws of optical physics; This is the super-resolution image fidelity loss, used to constrain the consistency between the super-resolution reconstruction result and the ground truth image.

[0012] Furthermore, in S4, the specific steps for end-to-end training and optimization of the diffusion UNet backbone network based on the multi-constraint joint loss function are as follows: The weight ratios of each component of the multi-constraint joint loss function are determined according to the task characteristics of super-resolution with semi-transparent media degradation. The diffusion noise prediction loss, the medium physical transmission constraint loss, and the super-resolution image fidelity loss are weighted and fused. The diffusion noise prediction loss calculates the error between the actual injected noise and the network predicted noise during the forward diffusion process using mean square error. The medium physical transmission constraint loss integrates three types of sub-losses: component summation constraint, pixel non-negativity constraint, and environmental stray light smoothing constraint, ensuring that the decoupled scattering, stray light, and intrinsic components conform to the semi-transparent medium... The physical laws of high-resolution image transmission are investigated, and the super-resolution image fidelity loss integrates pixel-level error loss, deep feature perception loss, and image style loss. Subsequently, the AdamW optimizer is used to set the initial learning rate and weight decay coefficient, and the learning rate is dynamically adjusted by combining a cosine annealing learning rate scheduling strategy. On paired image datasets with interference from multiple semi-transparent media, all parameters of the diffusion UNet backbone network are iteratively updated in an end-to-end mode. During training, random horizontal flipping, random cropping, and brightness fine-tuning data augmentation operations are performed on the input images. The model reconstruction performance is evaluated on the validation set by peak signal-to-noise ratio and structural similarity index at fixed rounds, and the network parameters with the best performance on the validation set are retained.

[0013] Furthermore, in S5, the backsampling process specifically includes: The input is a coupled, degraded, low-resolution image to be processed. Initial latent features are obtained through the input encoding module, and the time step of backsampling is initialized. ; when At that time, the noise at the current time step is predicted by decoupling the medium scattering component, and a backsampling denoising step is performed to remove the interference of the medium scattering component. Decrease successively; when At that time, the noise at the current time step is predicted by decoupling the environmental stray light component, and a backsampling denoising step is performed to remove the interference of the environmental stray light component. Decrease successively; when At that time, the noise at the current time step is predicted by decoupling the scene intrinsic components, and a backsampling denoising step is performed to recover the detailed features of the scene intrinsic components. Decrease successively; when When the value decreases to 0, the pure intrinsic latent features of the scene are obtained. The high-resolution image is then upsampled and reconstructed using a super-resolution reconstruction head, and the final super-resolution clear image is output.

[0014] Beneficial effects Compared with existing technologies, this real-scene image super-resolution reconstruction method based on a diffusion model has the following advantages: This invention deeply binds the physical process of light transmission through the medium with the diffusion model, matching dedicated time step intervals for different components, thus accurately corresponding the diffusion process with the actual degradation process and significantly improving the anti-interference capability. It constructs three independent decoupled branches in the diffusion latent space, combined with precise time step division, achieving complete separation of medium interference and intrinsic scene signals, thoroughly resolving the contradiction between artifact residue and detail loss. The introduction of a multi-physical constraint joint loss function ensures that the decoupling process conforms to the laws of optical physics from multiple dimensions, effectively avoiding spurious solutions while ensuring the fidelity of super-resolution reconstruction. The end-to-end processing flow significantly reduces processing complexity and exhibits strong robustness to various semi-transparent medium interference scenarios, significantly improving generalization ability.

[0015] Other advantages, objectives and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination or study, or may be learned from the practice of the invention. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0017] Figure 1 This is a flowchart of a real-scene image super-resolution reconstruction method based on a diffusion model; Figure 2 This is a flowchart of step S4 in the real-scene image super-resolution reconstruction method based on the diffusion model. Detailed Implementation

[0018] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0019] This invention provides a real-scene image super-resolution reconstruction method based on a diffusion model. By constructing a physical transport model with degraded coupling in a semi-transparent medium, it accurately decomposes the three physical components of the real imaging process, establishes a precise mathematical expression, and deeply binds the physical process of light transport in the semi-transparent medium with the forward noise injection process of the diffusion model. It matches a dedicated time step interval for different physical components, achieving a deep binding between the physical process and the diffusion model. It constructs independent decoupling branches for the three components in the diffusion latent space, and achieves precise decoupling of the three components by combining the precise division of the time step interval. It constructs a multi-constraint joint loss function and introduces physical transport constraints in the training of the diffusion model to ensure the rationality of decoupling and the accuracy of reconstruction.

[0020] S1: Construct a physical transmission model for coupling degradation in a semi-transparent medium, decompose the coupled degradation image captured in the real scene into three independent physical components: medium scattering component, ambient stray light component, and scene intrinsic component, and establish a precise mathematical expression for the coupled degradation image. The mathematical expression for the physical transport model of degradation caused by coupling in a semi-transparent medium is: In the formula, Image pixel coordinates, To capture coupled, degraded, low-resolution images, it supports input of 3-channel RGB color images or single-channel grayscale images with pixel dimensions of H×W×C, where H is the image height, W is the image width, and C is the number of image channels. The intrinsic reflection component of the scene, i.e., an ideal, clear, high-resolution scene image without any semi-transparent medium interference and without resolution loss, is the target signal that needs to be reconstructed in this embodiment. This is a medium transmittance map, which is strongly correlated with the concentration of the semi-transparent medium and the light transmission distance. It is a single-channel image with the same size as the input image. The value range of each pixel is [0, 1]. The closer the value is to 1, the stronger the medium's ability to transmit light and the smaller the scene signal attenuation. The closer the value is to 0, the stronger the medium's scattering effect and the more severe the scene signal attenuation. The intrinsic transmission component of the scene is the effective scene light signal that finally reaches the camera imaging surface after the reflected light emitted by the scene target is attenuated by the semi-transparent medium. The medium scattering component is the superposition of forward and backscattered signals that directly enter the camera imaging system after ambient light is scattered by suspended particles in a semi-transparent medium. The ambient stray light component includes global atmospheric light in the shooting environment, stray light formed by direct strong light sources, and specular reflection stray light from semi-transparent medium interfaces. It belongs to the low-frequency bias interference of global smoothing and manifests as overall brightness shift and contrast reduction in the image.

[0021] S2: Construct a forward process of the diffusion model with physical process constraints. Through the forward noise addition formula of the diffusion model with physical constraints, the physical process of light transmission in the semi-transparent medium is deeply bound to the forward noise injection process of the diffusion model. Independent noise mapping time step intervals are matched for the three components respectively, and a forward noise addition process with physical constraints is established. The total diffusion time step of the preset diffusion model In this embodiment The number of steps can be set to 1000, or adjusted to 500, 2000, or other suitable values ​​depending on the task requirements. Based on the noise characteristics of the diffusion model and the physical properties of degradation in the semi-transparent medium, all time steps are divided into three continuous and non-overlapping time step intervals: The large time step of the diffusion model corresponds to high variance Gaussian noise, which represents the large-scale, low-frequency structural information of the image; the small time step corresponds to low variance Gaussian noise, which represents the high-frequency details and texture information of the image.

[0022] Medium scattering components For large-scale, strongly interfering low-frequency signals, the corresponding diffusion model has a high noise distribution over a large time step, matching a large time step interval. ; Ambient stray light component The globally smooth intermediate frequency bias signal corresponds to the moderate noise distribution at the time step in the diffusion model, matching the intermediate time step interval. ; Scene intrinsic components It contains high-frequency signals such as image edges, textures, and details, corresponding to the low-noise distribution of the diffusion model at small time steps, and matches the small time step interval. .

[0023] The division of the three time step intervals satisfies: In this embodiment, when the total time step When =1000, set For 0-200 steps, For 201-600 steps, The range is 601-1000; the interval division can be dynamically adjusted according to the medium type and interference intensity. For example, in strong scattering interference scenarios, it can be expanded. The step size ratio of the interval improves the decoupling accuracy of the scattering components.

[0024] Based on the above time step interval division, a piecewise physical constraint positive noise addition formula is constructed, which binds the three physical components to the noise injection process of the diffusion model one by one. The conditional probability distribution formula for positive noise addition is as follows: In the formula, for The noisy image at each time step has the same size as the input image I; Given an input image hour, Conditional probability distribution of noisy images at each time step; It follows a multidimensional Gaussian distribution; Given a preset noise scheduling coefficient, this embodiment employs a linear noise scheduling strategy. With time step The increase of is monotonically decreasing. The value range is [0.0001, 0.9999], and a cosine noise scheduling strategy can also be used to adapt to different degradation scenarios; It is an identity matrix with dimensions matching the pixel dimensions of the input image, and the covariance matrix used to constrain the Gaussian distribution is isotropic. This represents the large time step interval corresponding to the medium scattering component. This represents the mid-time step interval corresponding to the ambient stray light component. This represents the small time step interval corresponding to the scene's intrinsic components.

[0025] The positive noise addition logic for each interval is as follows: Within a large time step interval, the forward noise injection process only injects noise into the entire input image I, corresponding to the noise mapping process of the medium scattering components, binding scattering interference with the high noise distribution; in In the mid-time step interval, the forward noise addition process injects reference Gaussian noise unrelated to the image content, corresponding to the global bias noise mapping of the ambient stray light component; in Within a small time step interval, the positive noise addition process only affects the scene's intrinsic components. Inject noise to bind high-frequency details in the scene with low-noise distribution.

[0026] This segmented forward noise addition process achieves a precise mapping between the three physical components and the time step of the diffusion model, providing strict physical constraints for component decoupling in the subsequent reverse process.

[0027] S3: Construct a diffusion UNet backbone network with three components decoupled in the latent space. In the latent space of the diffusion model, construct in parallel a decoupling branch for the medium scattering component, a decoupling branch for the ambient stray light component, and a decoupling branch for the scene intrinsic component. Configure an independent time step embedding module for each branch with a corresponding time step interval to achieve independent encoding and decoupling of the three components in the latent space. The diffusion UNet backbone network specifically comprises six core modules: an input encoding module, a three-component decoupling branch module, a time-step gating fusion module, a UNet-encoder / decoder module, a noise prediction head, and a super-resolution reconstruction head. The specific implementation methods of each module are as follows: The input encoding module consists of two consecutive convolutional layers and one residual convolutional block. It maps the coupled, degraded, low-resolution input image from pixel space to a high-dimensional latent space, obtaining initial latent features. Specifically, the input image size is H×W×C. The first convolutional layer uses a 3×3 kernel with a stride of 1 and padding of 1, outputting 64 channels, mapping the input image to the initial feature space. The second convolutional layer uses a 3×3 kernel with a stride of 2 and padding of 1, outputting 128 channels, performing feature downsampling and dimensionality enhancement. Subsequently, a residual convolutional block is passed through, containing two 3×3 convolutional layers, one skip connection, and a GELU activation function, maintaining 128 output channels. This results in initial latent features of size (H / 2)×(W / 2)×128, which are then input to the subsequent three-component decoupling branch module.

[0028] The three-component decoupling branch module sets up three completely independent decoupling branches in parallel in the latent space: the medium scattering component decoupling branch, the ambient stray light component decoupling branch, and the scene intrinsic component decoupling branch. The three branches process the initial latent features of the input in parallel and complete the feature encoding and decoupling of the corresponding components respectively.

[0029] Each branch employs a three-concatenated residual convolutional block structure. Each residual convolutional block contains two 3×3 convolutional layers, a normalization layer, a GELU activation function, and skip connections. All three branches have 128 output channels to ensure consistency in feature dimensions. Furthermore, each branch is configured with an independent time-step embedding module corresponding to its time-step interval. This time-step embedding module is implemented using sinusoidal positional encoding and two fully connected layers. Specifically: The time-step embedding module for the decoupling branch of the medium scattering component is only for... Encode the time steps of the interval and output the time step embedding vector that matches the dimension of the branch's latent features, and then incorporate it into each residual convolutional block of the branch. The time-step embedding module for the decoupling branch of ambient stray light components is only for... Encode the time steps of the interval, output the time step embedding vector of the corresponding dimension, and incorporate it into each residual convolutional block of the branch; The time-step embedding module of the scene intrinsic component decoupling branch only applies to... The time steps of the interval are encoded, and the corresponding time step embedding vector is output and incorporated into each residual convolutional block of that branch.

[0030] Through independent branching structures and time step embedding modules, the three branches are activated in their respective time step intervals to complete the independent encoding of the corresponding components. This achieves complete decoupling of the medium scattering component, ambient stray light component, and scene intrinsic component in the latent space, and outputs the latent features of the medium scattering component, ambient stray light component, and scene intrinsic component, respectively. The size of the three latent features is (H / 2)×(W / 2)×128.

[0031] The time-step gated fusion module employs a gated attention mechanism. The input consists of the current time step and the component latent features of the three branches. The output is the weighted fused latent feature. Specifically, the implementation involves: first, globally encoding the current input time step t to obtain the time-step feature vector; then, generating the gate weights corresponding to the three branches through three independent fully connected layers. , , The three weights take values ​​in the range [0, 1], and satisfy the following condition: The gating weights are dynamically adjusted based on the time step interval. hour, Approaching 1, and When the value approaches 0, the fused latent features are dominated by the latent features of the medium scattering component; when... hour, The weights approach 1, and the remaining weights approach 0; when hour, The weights approach 1, and the remaining weights approach 0; finally, the three component latent features are weighted and summed according to the gating weights to obtain the fused latent features, while maintaining the size. The input is sent to the UNet encoding / decoding module.

[0032] The UNet encoder-decoder module adopts a symmetrical encoder-decoder structure, with 4 downsampling encoders and 4 upsampling decoders. Skip connections are set between corresponding layers of the encoder and decoder to pass image detail features and avoid detail loss during downsampling.

[0033] Each layer of the encoder consists of two 3×3 convolutional layers, a downsampling convolutional layer with a stride of 2, a group normalization layer, and a GELU activation function. Each encoder layer downsamples the feature map size to half of its original size and doubles the number of channels. After the four encoder layers complete the encoding, the smallest bottleneck feature is obtained with 1024 channels.

[0034] Each layer of the decoder consists of a single upsampled transposed convolutional layer with a stride of 2, two 3×3 convolutional layers, a group normalization layer, and a GELU activation function. Each layer of the decoder upsamples the feature map size to twice its original size and reduces the number of channels to half its original size. At the same time, the features output by the corresponding layer encoder are spliced ​​and fused with the features of the current layer decoder through skip connections to ensure the transmission of detailed features.

[0035] Meanwhile, in this embodiment, the input low-resolution image is used as a conditional feature. A conditional encoding module generates multi-scale conditional features, which are then injected hierarchically into each layer of the encoder and decoder. This provides conditional guidance from the low-resolution image for noise prediction in the diffusion model, improving the model's reconstruction accuracy. The conditional encoding module consists of multi-scale convolutional layers, generating conditional features that match the feature dimensions of each layer in the encoding and decoding process, with the number of channels consistent with the number of feature channels in the corresponding layer.

[0036] The noise prediction head consists of two consecutive 3×3 convolutional layers. The first convolutional layer has 128 input channels and 64 output channels. The second convolutional layer has the same number of output channels as the input image (C). Both layers have a stride of 1 and padding of 1. The final latent features output by the decoder are mapped to a noise prediction map that is exactly the same as the input image size and number of channels. This map is used to calculate the spread noise prediction loss and guide the training and optimization of the model.

[0037] The super-resolution reconstruction head consists of an upsampling convolutional layer and a pixel reconstruction module. It is used to perform upsampling reconstruction of high-resolution images based on the pure scene intrinsic latent features at the backsampling end. Specifically, the input of the super-resolution reconstruction head is the pure scene intrinsic latent features obtained at t=0, with a size of (H / 2)×(W / 2)×128. First, a 3×3 convolutional layer is used to adjust the number of channels to 256. Then, the pixel reconstruction module is used to upsample by 4 times, increasing the feature map size to 2H×2W×16. Then, a 3×3 convolutional layer is used to output the number of channels C, finally obtaining a high-resolution clear image with a size of 2H×2W×C. The parameters of the pixel reconstruction module can also be adjusted according to the super-resolution requirements to achieve super-resolution reconstruction of different magnifications such as 2x and 8x.

[0038] S4: Construct a multi-constraint joint loss function, integrate the diffusion noise prediction loss, medium physical transmission constraint loss, and super-resolution image fidelity loss through the general formula of the multi-constraint joint loss function, and perform end-to-end training and optimization of the diffusion UNet backbone network based on the multi-constraint joint loss function; The general formula for the joint loss function with multiple constraints is: In the formula, This represents the total loss during model training. , , These are preset weighting coefficients used to balance the optimization priorities of various loss terms. In this embodiment, they are set as follows: , , The weighting coefficients can be dynamically adjusted based on the model's performance during training. For example, in scenarios with strong interference, the weighting coefficients can be increased. The weights are used to strengthen the optimization priority of physical constraints; This is the noise prediction loss, used to constrain the noise prediction accuracy of the model; This is the physical transmission constraint loss of the medium, used to ensure that the decoupling process conforms to the laws of optical physics; This is the super-resolution image fidelity loss, used to constrain the consistency between the super-resolution reconstruction result and the ground truth image.

[0039] The specific implementation methods for each loss item are as follows: Predicted loss of diffused noise The mean squared error (MSE) loss is used for calculation, which is the core loss term in the diffusion model. The specific calculation formula is as follows: In the formula, This is the actual Gaussian noise injected during the forward diffusion process. To diffuse the noise predicted by the UNet backbone network, for Noisy images at time steps, The input is a coupled, degraded, low-resolution image. From 0 to The uniform distribution of the noise sample is used to calculate the average of E over all time steps and all noise samples. This loss term constrains the model to accurately predict the noise component in the corresponding time step interval, providing reliable noise prediction results for the inverse denoising process.

[0040] Medium physical transmission constraint loss To ensure that the three components obtained from model decoupling conform to the physical laws of light transmission in a semi-transparent medium and to avoid artifacts and decoupling errors that do not conform to optical principles, the losses are specifically divided into three categories: component summation constraint loss, non-negativity constraint loss, and atmospheric light smoothing constraint loss. The calculation formula is as follows: Among them, the component summation constraint loss The three components obtained from constraint decoupling are superimposed to maintain consistency with the input image, ensuring pixel conservation during the decoupling process. L1 loss is used for calculation, and the formula is as follows: In the formula, , , , The transmittance map obtained by decoupling the model and the three physical components are used as a loss term to ensure that the decoupled components can completely restore the input image and avoid information loss during the decoupling process.

[0041] Non-negative constraint loss The transmittance and pixel values ​​of each component obtained by constraining decoupling are non-negative, which conforms to the physical characteristics of optical imaging (neither light intensity nor transmittance can be negative). The formula is: This loss term penalizes all cases where pixel values ​​are negative, ensuring that the decoupling result conforms to physical common sense.

[0042] Atmospheric light smoothing constraint loss Used to confine ambient stray light components For globally smooth low-frequency components, which conform to the physical characteristics of ambient stray light, the total variation (TV) loss is used for calculation, and the formula is as follows: In the formula, As a gradient operator, it calculates the sum of the gradients of the ambient stray light component in the horizontal and vertical directions. Through this loss term, the gradient of the ambient stray light component is constrained to be as small as possible, ensuring its globally smooth low-frequency characteristics and avoiding the decoupling of scene details errors to the ambient stray light component.

[0043] Super-resolution image fidelity loss This is used to constrain the consistency between the final super-resolution reconstruction result of the model and the ground truth image, improving the detail fidelity and visual quality of the reconstructed image. Specifically, it integrates pixel-level error loss, deep feature perception loss, and image style loss, and the calculation formula is as follows: ,in, and As the weighting coefficient, it is set in this embodiment. , ; Pixel-level error loss The L1 loss is used to calculate the pixel-level difference between the reconstructed image and the ground truth image. The formula is as follows: In the formula, The super-reconstructed image output by the model. For a clear, high-resolution ground truth image free from interference; Deep feature perception loss The deep features of the network are extracted using a pre-trained visual feature extraction method. The difference in deep semantic features between the reconstructed image and the ground truth image is calculated to ensure that the semantic content of the reconstructed image is consistent with the ground truth image. The formula is as follows: In the formula, This refers to the deep feature extraction function of the pre-trained feature extraction network; Image style loss Based on the multi-layer features of the feature extraction network, the Gram-Matrix difference between the reconstructed image and the ground truth image is calculated to ensure that the style and texture of the reconstructed image are consistent with the ground truth, thus avoiding style shift and artifacts.

[0044] The specific steps for end-to-end training optimization are as follows: Dataset Construction and Preprocessing: A paired image dataset was constructed, containing coupled, degraded low-resolution images with various types of semi-transparent media interference, along with corresponding interference-free, clear, high-resolution ground truth images. Semi-transparent media types include fog, rain, frosted glass, underwater turbidity, and lens condensation. The dataset contains at least 100,000 pairs of images, divided into training and validation sets at a 9:1 ratio. All images in the dataset underwent preprocessing. The image sizes were uniformly scaled to preset dimensions (e.g., 256×256, 512×512), and pixel values ​​were normalized, mapping pixel values ​​from the integer range [0, 255] to the floating-point range [-1, 1] to suit the model's input requirements.

[0045] The training parameters are initialized to include all parameters of the diffusion UNet backbone network. The weights of the convolutional and fully connected layers are initialized using the Kaiming uniform initialization method, and the bias terms are initialized to 0. The weight coefficients of the multi-constraint joint loss function are set, and the total time steps and noise scheduling strategy of the diffusion model are configured. The optimizer is set to the AdamW optimizer, with an initial learning rate of 2e-4, a weight decay coefficient of 1e-5, and a batch size of 16. The learning rate scheduling strategy is cosine annealing, with a minimum learning rate of 1e-6 and a total training epoch of 300 epochs.

[0046] In each training round, batches of samples are randomly selected from the training set, and random data augmentation operations are performed on the coupled and degraded input images. These operations include random horizontal flipping, random cropping, brightness fine-tuning, and contrast fine-tuning. The probability of data augmentation is set to 50% to improve the model's generalization ability. The augmented images are then input into the diffusion UNet backbone network, and time steps t are randomly sampled between 0 and T. Gaussian noise is injected according to the forward noise addition formula of physical constraints to obtain a noisy image. Based on the noisy image, time steps, and input conditional image, the network outputs a noise prediction map, the three decoupled physical components, and a super-reconstructed image. The total loss is calculated based on the multi-constraint joint loss function, and the gradient of the network parameters is calculated through the backpropagation algorithm. The AdamW optimizer updates all trainable parameters of the network based on the gradient. Every 1000 iterations, the current total loss and the values ​​of each sub-loss term are printed to monitor the training process.

[0047] Model validation and parameter saving: Every 10 training rounds, the reconstruction performance of the model is evaluated on the validation set. The evaluation metrics are Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM). PSNR is used to measure the pixel-level fidelity between the reconstructed image and the ground truth image, and SSIM is used to measure the consistency of the structure and texture of the reconstructed image with the ground truth image. All samples are traversed on the validation set, and the average PSNR and average SSIM are calculated. The model parameters with the highest average PSNR on the validation set are retained as the final inference model parameters to avoid model overfitting.

[0048] S5: Input the low-resolution image with semi-transparent medium interference into the trained model. Through the backsampling process, remove the medium scattering component and the ambient stray light component interference in the corresponding time step interval. Then, complete the super-resolution reconstruction based on the pure scene intrinsic features and output a high-resolution clear image end-to-end. A low-resolution image with semi-transparent medium interference is acquired. The image is preprocessed by scaling it to the preset size for model input and normalizing pixel values ​​from [0, 255] to [-1, 1]. This normalized image is then fed into the input encoding module of the trained diffusion UNet backbone network to obtain initial latent features. The backsampling time step is initialized. = The reverse denoising process begins from the maximum time step.

[0049] when When the interval is reached, the initial latent features and the current time step t are input into the three-component decoupling branch module. At this time, the medium scattering component decoupling branch is activated, while the other two branches are suppressed. The noise of the current time step is predicted through the medium scattering component decoupling branch. Combined with the time step gating fusion module and the UNet encoding and decoding module, the noise prediction map of the current time step is output. Based on the DDIM backsampling formula of the diffusion model, the backsampling denoising step is performed to remove the interference of the medium scattering component and obtain the denoised latent features of the current time step. The time step t is successively decremented by 1, and this step is repeated until... Decrease to The lower limit of the interval completes the removal of the medium scattering component.

[0050] when When the interval is reached, the denoised latent features output from the previous interval and the current time step are input into the three-component decoupling branch module. At this time, the ambient stray light component decoupling branch is activated, while the other two branches are suppressed. The noise of the current time step is predicted through the ambient stray light component decoupling branch, and the noise prediction map is output. The DDIM backsampling denoising step is performed to remove the global low-frequency interference of the ambient stray light component, and the denoised latent features of the current time step are obtained. The time step t is decremented by 1 successively, and this step is repeated until... Decrease to The lower limit of the interval is used to completely remove stray ambient light components.

[0051] when When the interval is reached, the denoised latent features output from the previous interval and the current time step t are input into the three-component decoupling branch module. At this time, the scene intrinsic component decoupling branch is activated, while the other two branches are suppressed. The noise at the current time step is predicted through the scene intrinsic component decoupling branch, and a noise prediction map is output. The DDIM backsampling denoising step is executed to gradually recover the high-frequency detail features such as edges and textures of the scene intrinsic components, thus obtaining the denoised latent features at the current time step. The time step t is decremented by 1 successively, and the iteration is repeated until... Decrease to 0 to complete the entire inverse denoising process and obtain pure scene intrinsic latent features.

[0052] Super-resolution reconstruction and image output will When the intrinsic latent features of the pure scene are obtained at =0, the super-resolution reconstruction head is used to complete the upsampling reconstruction of the high-resolution image through the upsampling convolutional layer and the pixel recombination module, and the super-resolution reconstructed image is obtained. The output image is subjected to inverse normalization processing to map the pixel values ​​from [-1, 1] back to the integer range of [0, 255], and finally the end-to-end output is a high-resolution clear image without interference from semi-transparent media.

[0053] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A real-scene image super-resolution reconstruction method based on a diffusion model, characterized in that, The method includes: S1: Construct a physical transmission model for coupling degradation in a semi-transparent medium, decompose the coupled degradation image captured in the real scene into three independent physical components: medium scattering component, ambient stray light component, and scene intrinsic component, and establish a precise mathematical expression for the coupled degradation image. S2: Construct a forward process of the diffusion model with physical process constraints. Through the forward noise addition formula of the diffusion model with physical constraints, the physical process of light transmission in the semi-transparent medium is deeply bound to the forward noise injection process of the diffusion model. Independent noise mapping time step intervals are matched for the three components respectively, and a forward noise addition process with physical constraints is established. S3: Construct a diffusion UNet backbone network with three components decoupled in the latent space. In the latent space of the diffusion model, construct in parallel a decoupling branch for the medium scattering component, a decoupling branch for the ambient stray light component, and a decoupling branch for the scene intrinsic component. Configure an independent time step embedding module for each branch with a corresponding time step interval to achieve independent encoding and decoupling of the three components in the latent space. S4: Construct a multi-constraint joint loss function, integrate the diffusion noise prediction loss, medium physical transmission constraint loss, and super-resolution image fidelity loss through the general formula of the multi-constraint joint loss function, and perform end-to-end training and optimization of the diffusion UNet backbone network based on the multi-constraint joint loss function; S5: Input the low-resolution image with semi-transparent medium interference into the trained model. Through the backsampling process, remove the medium scattering component and environmental stray light component interference in the corresponding time step interval. Then, complete the super-resolution reconstruction based on the pure scene intrinsic features and output a high-resolution clear image end-to-end.

2. The real-scene image super-resolution reconstruction method based on a diffusion model according to claim 1, characterized in that, In S1, a physical transport model for degradation caused by coupling in a semi-transparent medium is constructed, and its model formula is as follows: ,in, Image pixel coordinates, The resulting low-resolution image is a product of coupling degradation. The intrinsic reflection components of the scene, i.e., a clear scene image without any media interference; The transmittance diagram of the medium has a value range of [0, 1], which characterizes the light transmission capability of a semi-transparent medium. The intrinsic transmission component of the scene, that is, the actual reflected light signal of the scene through the semi-transparent medium; The medium scattering component characterizes the interference signal formed by the scattering effect of the semi-transparent medium on incident light; The ambient stray light component represents the low-frequency interference signal formed by global atmospheric light and stray light reflected from the surface of the medium in the shooting environment.

3. The real-scene image super-resolution reconstruction method based on a diffusion model according to claim 1, characterized in that, In S2, the formula for positive noise addition due to physical constraints is: ,in, for Noisy images at time steps; Given an input image hour, Conditional probability distribution of noisy images at time steps; It follows a Gaussian distribution; The preset noise scheduling coefficient varies with time step. Increases monotonically and decreases; It is the identity matrix; This represents the large time step interval corresponding to the medium scattering component. This represents the mid-time step interval corresponding to the ambient stray light component. The three intervals are the small time step intervals corresponding to the scene's intrinsic components. They are continuous and do not overlap, covering all time steps of the diffusion model.

4. The real-scene image super-resolution reconstruction method based on a diffusion model according to claim 1, characterized in that, In S2, the three time step intervals are set based on the physical characteristics of degradation in the semi-transparent medium: the medium scattering component and the ambient stray light component belong to low-frequency, large-scale interference, corresponding to the high-noise distribution of the diffusion model in the large time step; the scene intrinsic components belong to high-frequency detail signals, corresponding to the low-noise distribution of the diffusion model in the small time step; the total time step of the diffusion model is set to... ,but .

5. The real-scene image super-resolution reconstruction method based on a diffusion model according to claim 4, characterized in that, In S3, the diffused UNet backbone network is a conditional diffused network structure, specifically including: Input encoding module: used to map the coupled, degraded, low-resolution input image to the latent space through convolutional layers to obtain initial latent features; Three-component decoupling branch module: Three independent decoupling branches are set up in parallel in the latent space, namely the medium scattering component decoupling branch, the ambient stray light component decoupling branch, and the scene intrinsic component decoupling branch. Each branch adopts a residual convolution block structure and is configured with an independent time step embedding module for the corresponding time step interval, which outputs the latent features of the medium scattering component, the latent features of the ambient stray light component, and the latent features of the scene intrinsic component, respectively. Time-step gated fusion module: Based on the current input time step, it generates gate weights for three branches, performs weighted fusion of the latent features of the three components, and obtains fused latent features; UNet encoding / decoding module: It adopts a symmetrical encoder-decoder structure, sets up skip connections, and inputs the conditional features of the low-resolution image into each layer of the encoder and decoder through hierarchical injection. It completes noise encoding and decoding based on the fused latent features. Noise prediction head: Convolutional layers are used to map the decoded latent features into a noise prediction map with the same number of channels as the input image; Super-resolution reconstruction head: Composed of upsampling convolutional layers and pixel reconstruction modules, it is used to perform upsampling reconstruction of high-resolution images based on pure scene intrinsic latent features at the backsampling end.

6. The real-scene image super-resolution reconstruction method based on the diffusion model according to claim 4, characterized in that, The medium physical transmission constraint loss includes component summation constraint loss, non-negative constraint loss, and atmospheric light smoothing constraint loss. The component summation constraint loss is used to ensure that the superposition of the three components obtained by decoupling is consistent with the input image. The non-negative constraint loss is used to ensure that the transmittance obtained by decoupling and the pixel value of each component are non-negative. The atmospheric light smoothing constraint loss is used to constrain the environmental stray light component to be a globally smooth low-frequency component.

7. The real-scene image super-resolution reconstruction method based on a diffusion model according to claim 1, characterized in that, In S4, the general formula for the multi-constraint joint loss function is: ,in, Total loss; , , These are preset weighting coefficients used to balance the optimization priorities of each loss term; This is the noise prediction loss, used to constrain the noise prediction accuracy of the model; This is the physical transmission constraint loss of the medium, used to ensure that the decoupling process conforms to the laws of optical physics; This is the super-resolution image fidelity loss, used to constrain the consistency between the super-resolution reconstruction result and the ground truth image.

8. The real-scene image super-resolution reconstruction method based on a diffusion model according to claim 1, characterized in that, In step S4, the specific steps for end-to-end training and optimization of the diffusion UNet backbone network based on the multi-constraint joint loss function are as follows: The weight ratios of each component of the multi-constraint joint loss function are determined according to the task characteristics of super-resolution with semi-transparent media degradation. The diffusion noise prediction loss, media physical transmission constraint loss, and super-resolution image fidelity loss are weighted and fused. The diffusion noise prediction loss calculates the error between the actual injected noise and the network predicted noise during the forward diffusion process using mean square error. The media physical transmission constraint loss integrates three types of sub-losses: component summation constraint, pixel non-negativity constraint, and environmental stray light smoothing constraint, ensuring that the decoupled scattering, stray light, and intrinsic components conform to the optical transmission characteristics of the semi-transparent medium. The physical laws of image processing are considered. The super-resolution image fidelity loss integrates pixel-level error loss, deep feature perception loss, and image style loss. Subsequently, the AdamW optimizer is used to set the initial learning rate and weight decay coefficient. The learning rate is dynamically adjusted by combining the cosine annealing learning rate scheduling strategy. On a paired image dataset with interference from multiple semi-transparent media, all parameters of the diffusion UNet backbone network are iteratively updated in an end-to-end mode. During training, random horizontal flipping, random cropping, and brightness fine-tuning data augmentation operations are performed on the input image. The model reconstruction performance is evaluated on the validation set by peak signal-to-noise ratio and structural similarity index at fixed rounds, and the network parameters with the best performance on the validation set are retained.

9. The real-scene image super-resolution reconstruction method based on a diffusion model according to claim 1, characterized in that, In S5, the reverse sampling process specifically includes: The input is a coupled, degraded, low-resolution image to be processed. Initial latent features are obtained through the input encoding module, and the time step of backsampling is initialized. ; when At that time, the noise at the current time step is predicted by the decoupling branch of the medium scattering component, and a backsampling denoising step is performed to remove the interference of the medium scattering component. Decrease successively; when At that time, the noise at the current time step is predicted by decoupling the environmental stray light component, and a backsampling denoising step is performed to remove the interference of the environmental stray light component. Decrease successively; when At that time, the noise at the current time step is predicted by decoupling the scene intrinsic components, and a backsampling denoising step is performed to recover the detailed features of the scene intrinsic components. Decrease successively; when When the value decreases to 0, the pure intrinsic latent features of the scene are obtained. The high-resolution image is then upsampled and reconstructed using a super-resolution reconstruction head, and the final super-resolution clear image is output.

Images