Weather image processing method based on double-branch wavelet diffusion, computer system and readable storage medium

Abstract

The application relates to the technical field of image processing, in particular to a weather image processing method based on a double-branch wavelet diffusion, a computer system and a readable storage medium. A current input image is decomposed into low-frequency components and high-frequency components, and the first branch and the second branch with different processing characteristics are used for processing respectively, so that the network can adopt differentiated recovery strategies for low-frequency global information and high-frequency local details, and the optimization difficulty of a single network in simultaneously fitting multiple degradation mechanisms in a pixel space is reduced; on the other hand, the recovery of high-frequency features is guided by low-frequency features, and the low-frequency features are corrected by high-frequency feature feedback, the consistency between the global structure and the local texture of the recovered image is increased, and reliable severe weather image recovery is realized. The application aims to solve the problem of how to realize reliable severe weather image recovery.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a weather image processing method, computer system, and readable storage medium based on bibranch wavelet diffusion. Background Technology

[0002] In practical applications such as autonomous driving, video surveillance, and outdoor target recognition, acquired images are often severely degraded by adverse weather conditions such as fog, rain, and snow, resulting in reduced image visibility, blurred textures, and color shifts, which greatly limits the performance of high-level vision tasks. Therefore, recovering clear, natural, high-quality images from single degraded images is a highly challenging fundamental research topic in the field of computer vision.

[0003] Currently, generative restoration methods based on diffusion models, such as DiffUIR, introduce diffusion models into image restoration, achieving more realistic texture details than traditional regression models. However, existing diffusion-based restoration methods face technical bottlenecks when handling mixed degradation in the RGB space—the network needs to simultaneously learn mappings for two different physical mechanisms: dehazing (i.e., brightness / color adjustment) and desnow / rain removal (i.e., texture reconstruction), which increases the optimization difficulty and makes the restoration results unstable under complex mixed weather conditions.

[0004] In view of this, this application proposes a new weather image processing method based on a diffusion model, which aims to achieve reliable recovery of severe weather images. Summary of the Invention

[0005] The main objective of this application is to provide a weather image processing method based on bibranch wavelet diffusion, which aims to solve the problem of how to reliably restore severe weather images.

[0006] To achieve the above objectives, this application provides a weather image processing method based on bibranch wavelet diffusion, the method comprising:

[0007] S10, Obtain the image to be recovered;

[0008] S20, the image to be restored is decomposed into low-frequency components and at least one high-frequency component by wavelet transform, the low-frequency component is processed by the first branch to obtain the first branch feature, and the high-frequency component is processed by the second branch to obtain the second branch feature;

[0009] S30, the first modulation signal generated by the first branch feature guides the second branch feature to be modulated to obtain the modulated high-frequency feature, and the second modulation signal generated by the second branch feature guides the first branch feature to be modulated to obtain the modulated low-frequency feature;

[0010] S40, Based on the modulated high-frequency features and the modulated low-frequency features, generate and output a weather recovery image.

[0011] Optionally, in S30, the first modulation signal generated from the first branch feature guides the second branch feature to be modulated to obtain modulated high-frequency features, including:

[0012] The first branch feature is subjected to pooling and tensor reshaping operations to obtain the first modulation signal;

[0013] The first modulation signal is multiplied element-wise with the second branch feature to obtain the high-frequency modulation feature;

[0014] Perform pointwise convolution and tensor reshaping operations on the high-frequency modulation features to obtain the high-frequency modulation weights;

[0015] After multiplying the high-frequency modulation weights element-wise with the second branch features, the channels are concatenated with the second branch features to obtain the modulated high-frequency features.

[0016] Optionally, in S30, the first branch feature is modulated using the second modulation signal generated from the second branch feature to obtain modulated low-frequency features, including:

[0017] Perform depthwise convolution processing on the second branch features to obtain the second modulation signal;

[0018] The second modulation signal is multiplied element by element by the first branch feature to obtain the first low-frequency modulation feature;

[0019] After performing a pooling operation on the first low-frequency modulation feature, it is multiplied element-wise with the feature obtained by point-by-point convolution of the second modulation signal to obtain the second low-frequency modulation feature;

[0020] After adding the first low-frequency modulation feature and the second low-frequency modulation feature, the channel is spliced ​​with the first branch feature to obtain the modulated low-frequency feature.

[0021] Optionally, the method further includes:

[0022] Multi-scale feature extraction and gating screening are performed on the low-frequency components to obtain gating features;

[0023] A degradation representation vector is constructed based on the global statistics of the gated features;

[0024] The weights of the convolutional layers in the image restoration model used to perform steps S20-S40 are modulated based on the degradation representation vector.

[0025] Optionally, modulating the convolutional layer weights in the image restoration model used to perform steps S20-S40 based on the degradation representation vector includes:

[0026] Task-specific weights are generated based on the aforementioned degenerate representation vector;

[0027] The task-specific weights are superimposed with the preset static shared basic weights to obtain the recombined convolutional kernel weights;

[0028] The recombined convolutional kernel weights are used as the original weights of the target convolutional layer in the image restoration model.

[0029] Optionally, S20 includes:

[0030] A wavelet transform is performed on the features of the image to be restored and the noisy input of the current time step to obtain a low-frequency sub-band, as well as a horizontal high-frequency sub-band, a vertical high-frequency sub-band, and a diagonal high-frequency sub-band.

[0031] The high-frequency component is determined based on the horizontal high-frequency sub-band, the vertical high-frequency sub-band, and the diagonal high-frequency sub-band, and the low-frequency component is determined based on the low-frequency sub-band;

[0032] Downsampling feature modeling is performed on the low-frequency components through the first branch to extract first branch features containing global structural information;

[0033] The second branch performs detail enhancement processing on the high-frequency components without changing the spatial resolution to extract second branch features containing local texture information.

[0034] Optionally, after performing feature extraction based on the downsampling feature modeling of the low-frequency encoder, upsampling is performed to recover the first branch feature.

[0035] The high-frequency components are processed by multiple residual detail enhancement modules of the same resolution in series to obtain the second branch feature.

[0036] Optionally, after S40, it also includes,

[0037] The weather recovery image is used as the current input image, and the image to be recovered is used as the condition image;

[0038] After stitching the current input image with the conditional image, return to iterative execution steps S20 to S40, and stop iterating when the preset iteration stop condition is met and output the final weather recovery image.

[0039] In addition, to achieve the above objectives, this application also provides a computer system comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the weather image processing method based on bibranch wavelet diffusion as described in any of the preceding claims.

[0040] In addition, to achieve the above objectives, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the weather image processing method based on bi-branch wavelet diffusion as described in any of the preceding claims.

[0041] This application has at least the following beneficial effects:

[0042] 1. By embedding wavelet transform into the iterative process of the diffusion model, the degraded image is decomposed into low-frequency and high-frequency components for separate processing. The low-frequency branch focuses on restoring the illumination and color interfered with by haze, while the high-frequency branch focuses on reconstructing the edges and details covered by rain and snow. This avoids the confusion of the two physical degradation features in the RGB space by a single network, and significantly improves the naturalness of colors during dehazing and the clarity of textures during rain and snow removal.

[0043] 2. Based on the dynamic modulation of network parameters according to the degradation representation, the same model can adaptively adjust the convolutional layer weights according to the degradation state of the input image, which increases the model's adaptability to multiple degradation tasks and reduces feature interference in multi-task learning.

[0044] 3. By using a cross-branch feature modulation mechanism, low-frequency features are used to guide the recovery of high-frequency features and high-frequency features are used to correct low-frequency features, which increases the consistency between the global structure and local texture of the recovered image. Attached Figure Description

[0045] Figure 1 This is a schematic flowchart of the weather image processing method based on bibranch wavelet diffusion involved in the embodiments of this application;

[0046] Figure 2 This is a schematic diagram of the cross-branch interaction module involved in an embodiment of this application;

[0047] Figure 3 This is a schematic diagram of the multi-scale degradation sensing module structure involved in an embodiment of this application;

[0048] Figure 4 This is a diagram illustrating the overall network framework of the image restoration model involved in this application embodiment; wherein, Figure 4 (a) represents the diffusion process of the model. Figure 4 (b) is a diffusion-backward recovery network;

[0049] Figure 5 This is a schematic diagram of a low-frequency branch structure involved in an embodiment of this application, wherein, Figure 5 (a) represents the residual self-attention module. Figure 5 (b) is the residual detail enhancement module. Figure 5 (c) is the task adaptive module;

[0050] Figure 6 This is a schematic diagram of the hardware operating environment of the computer system involved in the embodiments of this application.

[0051] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0052] To better understand the above technical solutions, exemplary embodiments of this disclosure will be described in more detail below with reference to the accompanying drawings. While exemplary embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of this disclosure to those skilled in the art.

[0053] First Embodiment

[0054] Reference Figure 1 This embodiment provides a weather image processing method based on bi-branch wavelet diffusion, the method comprising the following steps:

[0055] S10, Obtain the image to be recovered;

[0056] In this embodiment, the image to be restored refers to image data affected by fog, rain, snow or other types of degradation.

[0057] S20, the image to be restored is decomposed into low-frequency components and at least one high-frequency component by wavelet transform, the low-frequency component is processed by the first branch to obtain the first branch feature, and the high-frequency component is processed by the second branch to obtain the second branch feature;

[0058] In this embodiment, the first branch processes low-frequency components, and its core function is to recover the global structural information, illumination distribution, and color tone of the image. Since low-frequency components usually account for the majority of the image's energy and their changes are relatively smooth, the first branch can employ processing logic suitable for capturing long-range dependencies or global context. In contrast, the second branch specifically processes high-frequency components, and its core function is to recover the image's local texture, edge contours, and subtle details. High-frequency components often contain rich spatial variation information, and the processing logic of the second branch focuses more on maintaining spatial resolution and enhancing local feature responses. By distributing the components decomposed by wavelet transform to functionally specialized branches for parallel processing, the optimization difficulty of a single network simultaneously fitting two different physical mechanisms—global illumination mapping and local texture reconstruction—within the same feature space is reduced.

[0059] S30, the first modulation signal generated by the first branch feature guides the second branch feature to be modulated to obtain the modulated high-frequency feature, and the second modulation signal generated by the second branch feature guides the first branch feature to be modulated to obtain the modulated low-frequency feature;

[0060] In this step, although frequency domain decomposition decouples the processed objects, the low-frequency structure and high-frequency texture of a natural image are physically coupled. If the two branches operate completely independently, it can easily lead to inconsistencies between global tones and local details, or artifacts at edges, resulting in information fragmentation. Therefore, this step introduces a bidirectional cross-branch feature modulation mechanism. Specifically, the first modulation signal is generated using the global illumination and structural prior information contained in the first branch features to guide the second branch features, ensuring that the restoration of high-frequency details conforms to the overall scene's illumination logic and avoiding texture noise that violates physical laws. On the other hand, the edge and texture information extracted from the second branch features is used to correct the first branch features, preventing the loss of key structural boundaries due to excessive smoothing during low-frequency restoration.

[0061] S40, Based on the modulated high-frequency features and the modulated low-frequency features, generate and output a weather recovery image.

[0062] In this embodiment, after cross-branch modulation in S30, the low-frequency and high-frequency features already contain complementary enhancement information. By performing inverse transform on these two parts of the features, the recovered image of the current iteration step can be reconstructed.

[0063] In the technical solution provided in this embodiment, the current input image is decomposed into low-frequency components and high-frequency components, and processed separately using a first branch and a second branch with different processing characteristics. This enables the network to adopt differentiated recovery strategies for low-frequency global information and high-frequency local details, reducing the optimization difficulty of a single network simultaneously fitting multiple degradation mechanisms in pixel space. On the other hand, low-frequency features are used to guide the recovery of high-frequency features, and high-frequency features are used to correct low-frequency features, increasing the consistency between the global structure and local texture of the recovered image, thereby achieving reliable recovery of severe weather images.

[0064] Furthermore, and optionally, following the S40, it also includes,

[0065] The weather recovery image is used as the current input image, and the image to be recovered is used as the condition image;

[0066] After stitching the current input image with the conditional image, return to iterative execution steps S20 to S40, and stop iterating when the preset iteration stop condition is met and output the final weather recovery image.

[0067] Specifically, the weather reconstruction image generated in the previous round is used as the current input image, and the image to be reconstructed is used as the condition image. Specifically, as the iteration progresses, the current input image is dynamically updated to the intermediate reconstruction result output by the model at the previous time step, and this intermediate result gradually becomes clearer with each iteration. Meanwhile, the original image to be reconstructed remains unchanged and is continuously injected into the network as the condition image. This combination of dynamic input and static conditions allows the model to progressively refine based on the latest reconstruction state, while providing constant content constraints through the original image, preventing semantic drift or the generation of illusory textures during multiple iterations, and ensuring that the reconstruction result remains faithful to the original scene content.

[0068] In each iteration, the current input image and the conditional image are concatenated along the channel dimension, followed by a step of decomposing the current input image into low-frequency and high-frequency components. Specifically, this concatenation operation allows the network to simultaneously perceive the current recovery progress and the original degradation cues at each time step. For example, in the six-channel tensor formed after channel concatenation, the first three channels carry the denoising state at the current moment, while the last three channels provide the original degradation reference. Frequency domain decomposition is then performed on this concatenated tensor, allowing the low-frequency and high-frequency branches to adaptively adjust their processing strategies according to the current recovery stage. In the early stages of recovery, the network may rely more on the structural information in the conditional image for global correction; while in the later stages, it relies more on the detail feedback in the current input image for texture refinement. This concatenation and decomposition mechanism, executed in each iteration, ensures tight coupling between frequency domain decoupling processing and the diffusion temporal evolution.

[0069] When the number of iterations reaches a preset threshold, the iteration stops and the final restored image is output. The preset threshold can be set according to the trade-off between restoration quality and inference speed required by the actual application scenario. For example, in scenarios with high real-time requirements such as autonomous driving or video surveillance, the preset threshold can be set to a smaller value such as 10, 15, or 20; while in offline processing scenarios with extremely high image quality requirements and ample computing resources, the threshold can be appropriately increased. In a preferred implementation of this embodiment, the preset threshold is set to 10 steps. Thanks to the aforementioned initialization strategy starting with the image to be restored and the reduced optimization difficulty brought about by the frequency domain decoupling dual-branch processing mechanism, the model only needs 10 iterations to converge to a high-quality restoration result. Compared to traditional pixel spatial diffusion methods that typically require 50 to 100 sampling steps, this strategy, while maintaining restoration quality, significantly reduces the number of iterations, lowers inference overhead, and also reduces the cumulative error that may be caused by long sequence iterations, increasing the stability of the restoration result. It should be understood that the above 10 steps are only a preferred example. In other implementations, as long as an architecture that starts with the image to be restored and combines frequency domain dual-branch processing is adopted, effective image restoration can be achieved in fewer iterations. The specific number of steps can be flexibly adjusted according to the model capacity and task complexity.

[0070] Second Embodiment

[0071] Based on any embodiment, this embodiment provides a detailed description of the fabrication of a cross-branch feature modulator. Specifically, in S30, the second branch feature is modulated using the first modulation signal generated from the first branch feature to obtain modulated high-frequency features, including:

[0072] S31, perform pooling and tensor reshaping operations on the first branch feature to obtain the first modulation signal;

[0073] S32, Multiply the first modulation signal and the second branch feature element by element to obtain the high-frequency modulation feature;

[0074] S33, Perform pointwise convolution and tensor reshaping operations on the high-frequency modulation features to obtain high-frequency modulation weights;

[0075] S34, after multiplying the high-frequency modulation weights element by element with the second branch features, the channels are concatenated with the second branch features to obtain the modulated high-frequency features.

[0076] The first branch feature is modulated using the second modulation signal generated from the second branch feature to obtain modulated low-frequency features, including:

[0077] S35, perform deep convolution processing on the second branch features to obtain the second modulation signal;

[0078] S36, Multiply the second modulation signal element by element with the first branch feature to obtain the first low-frequency modulation feature;

[0079] S37, After performing a pooling operation on the first low-frequency modulation feature, multiply it element-wise with the feature obtained by point-by-point convolution of the second modulation signal to obtain the second low-frequency modulation feature;

[0080] S38, after adding the first low-frequency modulation feature and the second low-frequency modulation feature, the channel is spliced ​​with the first branch feature to obtain the modulated low-frequency feature.

[0081] Specifically, in the low-frequency-guided-high-frequency path, the first branch features are processed to generate a first modulation signal, which contains prior information about the global illumination state and structural layout of the current scene. Using this signal to modulate the second branch features is equivalent to providing global semantic constraints for the recovery of high-frequency details.

[0082] For example, when the first branch identifies a region as sky, the generated modulation signal guides the second branch to suppress high-frequency noise and maintain smoothness in that region. When identifying building edges, it encourages texture enhancement, thus preventing artifacts or anomalous textures that violate global illumination logic during high-frequency recovery. In the path of high-frequency feedback to low-frequency, the second branch features are converted into a second modulation signal carrying precise edge location and texture structure information. Using this signal to modulate the first branch features injects crucial geometric boundary information into the low-frequency recovery process, effectively preventing the first branch from excessively blurring the structural boundaries of objects when performing global smoothing or color correction, ensuring the structural fidelity of the low-frequency recovery result.

[0083] Reference Figure 2 The schematic diagram of the cross-branch interaction module shows that the features of the low-frequency branch and the high-frequency branch are input into the cross-branch interaction module. The low-frequency information (global illumination) guides the diffusion process of the high-frequency features, while the high-frequency information (edge ​​details) guides the refinement of the low-frequency features, thus achieving complementary enhancement.

[0084] Features of the low-frequency and high-frequency branches at the same network depth are obtained separately. The low-frequency features are then processed through pooling layers and tensor reshaping, followed by element-wise multiplication with the high-frequency features for the first cross-feature transfer. Next, the high-frequency features are weighted and modulated through 1×1 convolution, tensor reshaping, and element-wise multiplication. Finally, the modulated enhanced features are... The high-frequency features modulated after being cascaded with the corresponding original feature channels are expressed as follows:

[0085]

[0086]

[0087] In the formula, R and Pool represent the reshaping and averaging pooling operations, respectively. Represents element-wise multiplication. Represents low-frequency characteristics. Represents high-frequency characteristics, This represents the high-frequency characteristics after modulation. Represents channel cascading. This represents standard convolution. High-frequency features are passed through a 7×7 depthwise convolution and element-wise multiplication for the first cross-feature propagation, then input into a pooling layer. The high-frequency features are then passed through another 7×7 depthwise convolution, followed by a 1×1 convolution and tensor reshaping operation to construct feature modulation weights. Low-frequency features are then weighted and modulated to obtain the second cross-feature. The first cross-feature... and the second cross feature Element-wise addition yields the modulated enhanced features, which are then cascaded with the corresponding original feature channels to output the modulated low-frequency features, represented as:

[0088]

[0089]

[0090]

[0091] In the formula, Represents depthwise convolution, where R and Pool represent the reshaping and averaging pooling operations, respectively. Represents element-wise multiplication. This indicates the addition of elements. This represents the low-frequency characteristics after modulation. This represents a cascaded channel.

[0092] Furthermore, the low-frequency features after interaction are input into the low-frequency U-Net branch (including downsampling and self-attention), and the high-frequency features are input into the high-frequency un-downsampling residual branch (including detail enhancement convolution), and feature extraction and reverse denoising are performed respectively.

[0093] The low-frequency branch employs a U-Net structure that includes downsampling operations, comprising a low-frequency encoder, a bottleneck layer, a low-frequency decoder, and a cross-branch interaction module. The low-frequency encoder, bottleneck layer, and low-frequency decoder are primarily composed of stacked residual self-attention modules, which consist of a positional encoding network, residual blocks, and a self-attention mechanism. Specifically, the input feature map is first obtained through the cross-branch interaction module. In a location encoding network, the time step t or the input feature map is used to... After positional encoding, the encoded feature map is obtained. , is represented as:

[0094]

[0095] In the formula, Indicates a time step. Represents a multilayer perceptron model. This indicates a sine wave position encoding operation. The encoded value will be... The input is fed into the residual self-attention module to obtain the corresponding low-frequency attention feature map. , is represented as:

[0096]

[0097]

[0098]

[0099] In the formula, Indicates the task adaptive module to which it belongs. express Activation function This indicates a group normalization operation. This indicates the self-attention mechanism. This represents the low-frequency characteristics of the residual self-attention module output. for With 1×1 convolution processing The intermediate low-frequency features are obtained by element-wise addition. To scale the feature maps, downsampling is performed after the residual self-attention module in the low-frequency encoder, and upsampling is added after the residual self-attention module in the low-frequency decoder. To mitigate gradient vanishing, skip connections are added between feature maps of the same network depth in both the low-frequency encoder and decoder. The output of the low-frequency branch is... The output of the high-frequency branch interacts with the output of the low-frequency branch through the cross-branch interaction module to obtain the output of the low-frequency branch. .

[0100] The high-frequency branch employs a residual network structure without downsampling, primarily consisting of four sequential residual detail enhancement modules and a cross-branch interaction module. The residual detail enhancement module comprises a positional encoding network and residual blocks. Unlike the residual blocks in the low-frequency branch, in the residual detail enhancement module, detail enhancement convolutions replace 1×1 convolutions to extract high-frequency detail information. Specifically, the input feature map is first obtained through the cross-branch interaction module. In a location encoding network, the time step t or the input feature map is used to... After positional encoding, the encoded feature map is obtained. , is represented as:

[0101]

[0102] In the formula, Indicates a time step. Represents a multilayer perceptron model. This indicates a sine wave position encoding operation. The encoded value will be... The input is fed into the residual detail enhancement module to obtain the corresponding high-frequency detail feature map. , is represented as:

[0103]

[0104]

[0105] In the formula, Indicates the task adaptive module to which it belongs. express Activation function This indicates a group normalization operation. Representation of detail augmentation convolution, for The intermediate high-frequency features generated in the residual detail enhancement module are input. The output of the high-frequency branch is then used. The output of the low-frequency branch interacts with the output of the high-frequency branch through the cross-branch interaction module to obtain the output of the high-frequency branch. .

[0106] Reference Figure 5 A schematic diagram of the low-frequency branch structure is shown. (See attached diagram) Figure 5 As shown in (a), the low-frequency branch adopts a U-Net structure that includes downsampling operations, mainly consisting of a low-frequency encoder, a bottleneck layer, a low-frequency decoder, and a cross-branch interaction module. The main body of the low-frequency encoder and decoder is composed of stacked Residual Self-Attention Modules (RSAMs). The specific processing flow is as follows:

[0107] First, the input feature map of the current network depth is obtained through the Cross-Branch Interaction Module (CBIM). In the position coding network, the current time step t is sinusoidally coded (SinPE) using the sinusoidal position coding formula, and then mapped by the MLP and... By adding element by element, the encoded features are obtained. Subsequently, Input the RSAM module. (See attached) Figure 5 As shown in (a), the RSAM consists of two branches: on the main branch, The input passes through a TAM module (including dynamic convolution), group normalization (GN), SiLU activation function, and 1×1 convolution, repeated four times; in the residual branch, After channel adjustment via 1×1 convolution, the result is added to the main output. Finally, the sum is input into the self-attention (SA) mechanism to obtain the low-frequency attention feature map. .

[0108] In the low-frequency encoder, each RSAM module is followed by a convolutional layer with a stride of 2 for downsampling; in the low-frequency decoder, a 2×2 nearest neighbor upsampling layer is added before each RSAM module. To mitigate gradient vanishing, skip connections are established between feature maps of the same network depth between the encoder and decoder. In this embodiment, the low-frequency encoder contains 3 downsampling stages, the bottleneck layer contains 1 RSAM module, and the decoder contains 3 upsampling stages. The self-attention mechanism uses multi-head self-attention with 4 heads.

[0109] As attached Figure 5 As shown in (b), the high-frequency branch adopts a residual network structure without downsampling operation, which mainly consists of four serial residual detail enhancement modules (RDEM) and a cross-branch interaction module.

[0110] The structure of RDEM is similar to that of RSAM, with the key difference being that, in order to fully extract and preserve high-frequency detail information, RDEM uses detail-enhancing convolutions instead of the ordinary 1×1 convolutions in RSAM. The specific processing flow is as follows:

[0111] First, the input feature map is obtained through CBIM. After encoding, we get Then Input the RDEM module. (See attached file.) Figure 5 As shown in (b), The input path sequentially passes through the TAM module, group normalization (GN), SiLU activation function, and detail enhancement convolution, repeated twice. The residual branch is then processed by detail enhancement convolution for channel adjustment and added to the main output to obtain the high-frequency detail feature map. In this embodiment, the high-frequency branch stacks a total of 4 RDEM modules, and a CBIM is inserted before the first RDEM module and after the fourth RDEM module to facilitate cross-branch information interaction.

[0112] As attached Figure 5 As shown in (c), the Task-Adaptive Module (TAM) receives the implicit weather cue vector Z from the DAM and dynamically generates task-specific convolutional kernel weights.

[0113] Specifically, Z is first expanded in dimension through a linear layer, activated by GELU, and then mapped to a weight vector of the target dimension through a second linear layer. After reshaping, task-specific weights are obtained. The weight generation process can be represented as follows: The recombined convolutional kernel weights are ultimately used for feature extraction. Static shared basic weights Task-specific weights Superimposed .in, These are learnable static parameters that are shared across all tasks. For learnable adaptive scaling factors; It is only used to replace the 3×3 convolutional kernel weights within the TAM module. With this design, a single set of network weights can dynamically adjust local convolutional parameters to adaptively adapt to the three degradation tasks with vastly different physical mechanisms: dehazing, deraining, and desnowing, effectively avoiding negative transfer problems between multiple tasks. In this embodiment, the TAM modules are stacked in high-frequency and low-frequency branches, such as... Figure 5 As shown in the dashed boxes in (a) and (b), each layer includes the above-mentioned dynamic weight generation process.

[0114] Third Embodiment

[0115] Based on any embodiment, in this embodiment, an image restoration model is used as the model for performing steps S20-S40, and the method further includes:

[0116] S50, perform multi-scale feature extraction and gating screening on the low-frequency components to obtain gating features;

[0117] S60, construct a degradation representation vector based on the global statistics of the gated features;

[0118] S70, the weights of the convolutional layers in the image restoration model used to perform steps S20-S40 are modulated based on the degradation representation vector.

[0119] Specifically, the decision to extract degradation representations based on low-frequency components rather than high-frequency components is a design choice based on the physical distribution characteristics of different types of image degradation. In common severe weather degradation scenarios, information such as light attenuation, color shift, and global contrast reduction caused by fog is mainly concentrated in low-frequency components. While high-frequency components contain texture details such as rain lines and snowflakes, their spatial distribution is often sparse and susceptible to noise interference, making it impossible to directly provide stable task-level semantic guidance. Therefore, extracting degradation representations from low-frequency components can more accurately capture global contextual information reflecting the type and severity of degradation.

[0120] In the process of multi-scale feature extraction and gating screening, the low-frequency components are first subjected to initial convolution processing to obtain the basic feature map, and then the feature map is input into multiple deep convolution branches with different receptive fields.

[0121] To suppress background responses unrelated to degradation, this embodiment employs an adaptive gating mechanism. Deep convolution and activation processing are performed on the multi-scale fused features to generate gating weights, which are then multiplied element-wise with the multi-scale fused features to obtain the gated features. This operation is equivalent to applying an attention mask in the feature space, enhancing the feature responses of degradation-related regions and reducing the interference of irrelevant information on the subsequent construction of degradation representations.

[0122] When constructing the degradation representation vector, global average pooling and global standard deviation pooling are performed on the gated features respectively. The resulting mean and standard deviation features are concatenated and mapped through a multilayer perceptron to obtain the degradation representation vector. The mean feature obtained from global average pooling reflects the overall brightness level and color tone of the image, which is crucial for determining the presence of fog or low-light conditions. The standard deviation feature obtained from global standard deviation pooling reflects the distribution of image contrast and the dispersion of degradation intensity, helping to distinguish between uniform and non-uniform degradation. By concatenating these two types of statistics and performing nonlinear mapping through a multilayer perceptron, high-dimensional spatial statistical information can be compressed into a compact and semantically clear degradation representation vector. This vector not only encodes the current degradation type, such as defogging, deraining, or desnowing, but also implicitly contains the severity of degradation, providing rich conditional inputs for subsequent dynamic parameter generation.

[0123] For example, refer to Figure 3 The diagram shows the internal structure of the mathematical operations of the multi-scale degradation sensing module. First, the low-frequency subgraph... Input a standard 3×3 convolutional layer to obtain the initial feature map. Subsequently, Three parallel, large-kernel, depthwise separable convolutional branches with different dilation rates are fed into the system for multi-scale degradation feature extraction. The kernel sizes of the three branches are 3×3, 5×5, and 7×7, respectively. After depthwise convolution, each branch undergoes channel adjustment via a 1×1 convolution. The output features of the three branches are concatenated along the channel dimension and then fused using a 1×1 convolution to obtain the multi-scale fused features. Next, an adaptive gating mechanism is used to suppress responses unrelated to degradation: [The gating mechanism is then applied to the response]. Input a 3×3 depthwise convolution, generate gated weights using the Sigmoid activation function, and then... Element-wise multiplication yields the gating feature. Then, respectively for Perform global average pooling and global standard deviation pooling to obtain the global mean. and global standard deviation The two are concatenated along the channel dimension to form a global statistical feature. Finally, A multilayer perceptron (MLP) is used for nonlinear mapping to output a fixed-dimensional implicit weather cue vector Z. In this embodiment, the MLP consists of two linear layers with a GELU activation function inserted in between, and the output dimension is set to 256. This cue vector Z contains information about the degradation type (fog / rain / snow) and its severity of the current input image, providing a basis for the subsequent adaptive parameter generation.

[0124] Furthermore, the modulation of the convolutional layer weights in the image restoration model used to perform steps S20-S40 based on the degradation representation vector includes:

[0125] S71, Generate task-specific weights based on the degraded representation vector;

[0126] S72, the task-specific weights are superimposed with the preset static shared basic weights to obtain the recombined convolutional kernel weights;

[0127] S73, the recombined convolutional kernel weights are used as the original weights of the target convolutional layer in the image restoration model.

[0128] During the dynamic parameter modulation stage, task-adaptive parameters are generated based on the degraded representation vector, and these parameters are used to dynamically modulate the weights of the convolutional layers in the image restoration model. The task-adaptive parameters are used to generate task-specific weights, which are then superimposed with the preset static shared base weights in the image restoration model to obtain reconstructed convolutional kernel weights. These reconstructed kernel weights are then used to replace the original weights of the target convolutional layer to perform the current feature extraction operation.

[0129] Specifically, the static shared base weights are a general feature extractor learned during training, applicable to all types of degradation recovery tasks; while the task-specific weights are incremental adjustments generated in real time based on the degradation representation of the current input. A learnable scaling factor controls the contribution of the task-specific weights, allowing the final convolutional kernel weights to be fine-tuned for specific degradation types while maintaining general feature extraction capabilities. For example, when the degradation representation indicates a foggy scene, the generated task-specific weights will guide the convolutional kernel to enhance its ability to correct low-frequency colors; while when it indicates a rain or snow scene, it will be adjusted to focus on texture preservation.

[0130] For example, the implicit weather cue vector Z is input into the Task Adaptive Module (TAM). A multilayer perceptron dynamically generates task-specific convolutional kernel weights, which are then superimposed on the statically shared base weights. Dynamic modulation is performed only on the convolutional layers within the TAM module to adapt to the three tasks of defogging, deraining, and desnowing. Simultaneously, depthwise convolution is used to improve computational efficiency. The weight generation process is represented as follows:

[0131]

[0132]

[0133] In the formula, This represents a multilayer perceptron. This represents the weights of the recombined convolutional kernels ultimately used for feature extraction. The task-specific weights are generated by mapping the implicit weather cue vector using an MLP and reshaping its dimensions. For learnable adaptive scaling factors, This represents the static shared base weights as learnable parameters.

[0134] Fourth embodiment

[0135] Based on any embodiment, in this embodiment, S20 includes:

[0136] S21, perform wavelet transform on the features of the image to be restored and the noisy input of the current time step to obtain low-frequency sub-band, horizontal high-frequency sub-band, vertical high-frequency sub-band and diagonal high-frequency sub-band;

[0137] S22, determine the high-frequency component based on the horizontal high-frequency sub-band, the vertical high-frequency sub-band, and the diagonal high-frequency sub-band, and determine the low-frequency component based on the low-frequency sub-band;

[0138] S23, perform downsampling feature modeling on the low-frequency components through the first branch to extract the first branch features containing global structural information;

[0139] S24, perform detail enhancement processing on the high-frequency components through the second branch without changing the spatial resolution to extract second branch features containing local texture information.

[0140] Furthermore, based on the downsampling feature modeling performed by the low-frequency encoder, feature extraction is performed followed by upsampling recovery to obtain the first branch feature;

[0141] The high-frequency components are processed by multiple residual detail enhancement modules of the same resolution in series to obtain the second branch feature.

[0142] In this embodiment, low-frequency components mainly carry information about the image's illumination distribution, color tone, and large-scale smooth structure. This type of information has high spatial redundancy and changes relatively slowly. Therefore, the first branch uses downsampling feature modeling. On the one hand, this effectively expands the network's receptive field by reducing spatial resolution, making it easier to capture long-range global contextual dependencies and thus accurately recover overall illumination and color. On the other hand, downsampling reduces the size of the feature map, lowering the computational cost of subsequent processing, allowing the model to focus more computational power on understanding the global structure. In contrast, high-frequency components mainly contain high-frequency signals such as image edge contours, subtle textures, and noise. This information is extremely sensitive to spatial location, and any downsampling operation may lead to irreversible loss of detail or aliasing artifacts. Therefore, the second branch chooses to process while maintaining the original spatial resolution, focusing on mining and enhancing local texture information to ensure that the recovered details are clear, sharp, and accurately positioned. Through this differentiated design that matches structure and function, this embodiment allows the two branches to work in their most suitable scale space, reducing the optimization difficulty of a single network simultaneously fitting two completely different physical mechanisms—global illumination mapping and local texture reconstruction—within the same feature space.

[0143] Specifically, the two-dimensional discrete wavelet transform (DWT) is used to decompose the input after concatenating the image to be restored with the noisy input at the current time step into one low-frequency sub-band and three high-frequency sub-bands (horizontal, vertical, and diagonal). This achieves physical decoupling between global illumination / color information and local edge / texture information, as expressed in:

[0144]

[0145]

[0146] In the formula, K = LH, HL, HH. , , , These represent the low-frequency sub-band, horizontal high-frequency sub-band, vertical high-frequency sub-band, and diagonal high-frequency sub-band, respectively. and Indicates the corresponding wavelet operator, Representing an image In position Pixel value at that location, Represents the scaling function. This represents the corresponding wavelet function. The size is After the transformation, the size of each subband is... .

[0147] Reference Figure 4The diagram shows the overall network framework of the image restoration model. It illustrates the overall framework and data flow of the network in this invention. During inference, the original input severe weather image is used as the condition image and the noisy image at the current time step. (Initial time) (That is, the Condition image itself), set the total number of sampling steps T=10, and perform reverse denoising iteratively round by round.

[0148] As attached Figure 4 As shown in (b), during inference, the noisy image at the current time step is first... With conditional images By splicing along the channel dimension, we obtain a size of The tensor is input to the 2D Discrete Wavelet Transform (DWT) module. After wavelet decomposition, the input is separated into four subbands: a low-frequency sub-graph. and three high-frequency subgraphs (horizontal high-frequency subgraphs) Vertical high-frequency subgraph Diagonal high-frequency subgraph Each high-frequency subgraph has a size of 1. The low-frequency subband mainly contains global smoothing information such as illumination and color, while the high-frequency subband mainly reflects detailed information such as edges and textures. The specific decomposition process of the two-dimensional discrete wavelet transform has been described in the aforementioned invention using the aforementioned formula. In subsequent processing, the three high-frequency subbands are concatenated along the channel dimension into a unified high-frequency feature representation. , and low-frequency subgraph The two branches are input separately and processed in parallel.

[0149] At the initial time (t=0), the original input severe weather image Condition is used simultaneously as the conditional image and the current noisy image. The input image is fed into the network for forward computation to obtain the prediction result for this round. In subsequent iterations (t=1,2,...,9), the predicted image from the previous round is used as the noisy image for the current time step. The image is then reassembled with the Condition image, and the forward computation is repeated. After a total of 10 iterations, the final, cleared image is output.

[0150] It should be noted that this embodiment uses the degraded image as the starting point for diffusion iteration, rather than starting from pure Gaussian noise. By combining wavelet transform, the feature map size is reduced to half of the original size, so that only 10 iterations are needed to converge to a high-quality restoration result. Compared with existing pixel spatial diffusion methods (which require 50 to 100 steps), high-quality restoration results can be obtained in fewer iterations.

[0151] Verification Implementation Examples

[0152] Based on the above embodiments, this embodiment verifies the effectiveness of the method. A multi-task hybrid training strategy is adopted, and images are randomly sampled in a 1:1:1 ratio from the RESIDE and D-HAZY mixed dataset (dehazing), the Rain13K and DeRaindrop mixed dataset (deraining), and the CSD dataset (desnowing) to form each training batch. All images are uniformly cropped to 256×256 pixels.

[0153] The Adam optimizer was used, with β1=0.9 and β2=0.99. The learning rate was set to... The total number of training steps was set to 500,000, and the batch size was set to 3. During the training phase, a standard forward diffusion noise addition strategy was adopted, in which Gaussian noise was added to the high and low frequency subbands in the wavelet domain for perturbation, and the noise variance scheduling followed a cosine scheduling strategy.

[0154] This embodiment adopts The loss function, subject to end-to-end supervision, is expressed as:

[0155]

[0156] In the formula, Represents the L1 norm, constraining pixel-by-pixel precision. A clear image representing the final prediction. A clear image is shown for reference.

[0157] Table 1

[0158] Table 1 shows a performance comparison between the method of this invention and existing methods. PSNR represents Peak Signal-to-Noise Ratio, SSIM represents Structural Similarity, used to evaluate image reconstruction quality; a higher value (↑) indicates less distortion and more complete structural preservation in the reconstructed image. Params (number of model parameters, in M) and FLOPs (number of floating-point operations, in G) are used to evaluate the lightweight nature and computational complexity of the model; lower values ​​(↓) indicate a lighter model and higher computational efficiency.

[0159] To verify the effectiveness of this invention, the proposed method is compared with eight existing mainstream advanced methods: Restormer, AirNet, PromptIR, ProRes, IDR, AutoDIR, DA-CLIP, and DiffUIR. Except for AutoDIR, all methods use the same training and test sets to ensure fairness.

[0160] On the standard test sets for defogging, deraining, and desnowing tasks, Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM) were used as quantitative evaluation metrics, and the computational complexity of the model was evaluated using the number of parameters (Params(M)) and the number of floating-point operations (FLOPs(G)). As shown in Table 1, in terms of quantitative metrics, the present invention achieved the best values ​​for PSNR and SSIM on all three tasks, proving that the image processed by the present invention has the highest fidelity while maintaining a small computational cost.

[0161] As one implementation scheme, Figure 6 This is a schematic diagram of the hardware operating environment of the computer system involved in the embodiments of this application.

[0162] like Figure 6 As shown, the computer system may include: a processor 1001, such as a CPU; a memory 1005; a user interface 1003; a network interface 1004; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0163] Those skilled in the art will understand that Figure 6 The computer system architecture shown does not constitute a limitation on the computer system and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0164] like Figure 6 As shown, the memory 1005, as a storage medium, may include an operating system, a network communication module, a user interface module, and computer programs. The operating system is a program that manages and controls the hardware and software resources of the computer system, as well as the operation of the computer programs and other software or programs.

[0165] exist Figure 6 In the computer system shown, the user interface 1003 is mainly used to connect to the terminal and communicate with the terminal; the network interface 1004 is mainly used to communicate with the backend server; and the processor 1001 can be used to call the computer program stored in the memory 1005.

[0166] In this embodiment, the computer system includes: a memory 1005, a processor 1001, and a computer program stored in the memory and executable on the processor, wherein:

[0167] When the processor 1001 calls the computer program stored in the memory 1005, it implements each step of the weather image processing method based on bi-branch wavelet diffusion as described in the above embodiment.

[0168] Furthermore, those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in a computer system to implement the process steps of the embodiments of the above methods.

[0169] Therefore, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the various steps of the weather image processing method based on bi-branch wavelet diffusion as described in the above embodiments.

[0170] The computer-readable storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0171] It should be noted that, since the storage medium provided in the embodiments of this application is the storage medium used to implement the methods of the embodiments of this application, those skilled in the art can understand the specific structure and variations of the storage medium based on the methods described in the embodiments of this application, and therefore will not be repeated here. All storage media used in the methods of the embodiments of this application fall within the scope of protection of this application.

[0172] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0173] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0174] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0175] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0176] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0177] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A weather image processing method based on bi-branch wavelet diffusion, characterized in that, The method includes the following steps: S10, Obtain the image to be recovered; S20, the image to be restored is decomposed into low-frequency components and at least one high-frequency component by wavelet transform, the low-frequency component is processed by the first branch to obtain the first branch feature, and the high-frequency component is processed by the second branch to obtain the second branch feature; S30, the first modulation signal generated by the first branch feature guides the second branch feature to be modulated to obtain the modulated high-frequency feature, and the second modulation signal generated by the second branch feature guides the first branch feature to be modulated to obtain the modulated low-frequency feature; S40, Based on the modulated high-frequency features and the modulated low-frequency features, generate and output a weather recovery image; The method further includes: Multi-scale feature extraction and gating screening are performed on the low-frequency components to obtain gating features; A degradation representation vector is constructed based on the global statistics of the gated features; The weights of the convolutional layers in the image restoration model used to perform steps S20-S40 are modulated based on the degradation representation vector.

2. The weather image processing method based on bi-branch wavelet diffusion as described in claim 1, characterized in that, In S30, the first modulation signal generated from the first branch feature guides the second branch feature to be modulated, resulting in modulated high-frequency features, including: The first branch feature is subjected to pooling and tensor reshaping operations to obtain the first modulation signal; The first modulation signal is multiplied element-wise with the second branch feature to obtain the high-frequency modulation feature; Perform pointwise convolution and tensor reshaping operations on the high-frequency modulation features to obtain the high-frequency modulation weights; After multiplying the high-frequency modulation weights element-wise with the second branch features, the channels are concatenated with the second branch features to obtain the modulated high-frequency features.

3. The weather image processing method based on bi-branch wavelet diffusion as described in claim 1, characterized in that, In S30, the second modulation signal generated by the second branch feature guides the first branch feature to be modulated, resulting in modulated low-frequency features, including: Perform depthwise convolution processing on the second branch features to obtain the second modulation signal; The second modulation signal is multiplied element by element by the first branch feature to obtain the first low-frequency modulation feature; After performing a pooling operation on the first low-frequency modulation feature, it is multiplied element-wise with the feature obtained by point-by-point convolution of the second modulation signal to obtain the second low-frequency modulation feature; After adding the first low-frequency modulation feature and the second low-frequency modulation feature, the channel is spliced ​​with the first branch feature to obtain the modulated low-frequency feature.

4. The weather image processing method based on bi-branch wavelet diffusion as described in claim 1, characterized in that, The modulation of the convolutional layer weights in the image restoration model used to perform steps S20-S40 based on the degradation representation vector includes: Task-specific weights are generated based on the aforementioned degenerate representation vector; The task-specific weights are superimposed with the preset static shared basic weights to obtain the recombined convolutional kernel weights; The recombined convolutional kernel weights are used as the original weights of the target convolutional layer in the image restoration model.

5. The weather image processing method based on bi-branch wavelet diffusion as described in claim 1, characterized in that, S20 includes: A wavelet transform is performed on the features of the image to be restored and the noisy input of the current time step to obtain a low-frequency sub-band, as well as a horizontal high-frequency sub-band, a vertical high-frequency sub-band, and a diagonal high-frequency sub-band. The high-frequency component is determined based on the horizontal high-frequency sub-band, the vertical high-frequency sub-band, and the diagonal high-frequency sub-band, and the low-frequency component is determined based on the low-frequency sub-band; Downsampling feature modeling is performed on the low-frequency components through the first branch to extract first branch features containing global structural information; The second branch performs detail enhancement processing on the high-frequency components without changing the spatial resolution to extract second branch features containing local texture information.

6. The weather image processing method based on bi-branch wavelet diffusion as described in claim 5, characterized in that, The first branch feature is obtained by performing downsampling feature modeling, feature extraction, and then upsampling recovery using a low-frequency encoder. The high-frequency components are processed by multiple residual detail enhancement modules of the same resolution in series to obtain the second branch feature.

7. The weather image processing method based on bi-branch wavelet diffusion as described in claim 1, characterized in that, Following the S40, it also includes, The weather recovery image is used as the current input image, and the image to be recovered is used as the condition image; After stitching the current input image with the conditional image, return to iterative execution steps S20 to S40, and stop iterating when the preset iteration stop condition is met and output the final weather recovery image.

8. A computer system, characterized in that, The computer system includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the weather image processing method based on bibranch wavelet diffusion as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the weather image processing method based on bibranch wavelet diffusion as described in any one of claims 1 to 7.

Images