A Method and System for Nighttime Severe Weather Image Restoration Based on Residual Allocation
By employing local and global residual allocation paths and controlled semantic feature injection in nighttime severe weather image restoration, the problem of not being able to balance local and global correction in existing technologies is solved, achieving high-quality image restoration results.
Patent Information
- Application Number
- CN202611426947.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-15
- Publication Date
- 2026-11-17
AI Technical Summary
Existing nighttime severe weather image restoration techniques lack explicit classification and division of labor for local and global residual errors, making it difficult to balance local and global corrections. They are prone to damaging weak textures and edges, and lack constraints on update amplitudes, resulting in poor restoration quality.
A residual allocation-based method is adopted to allocate local concentrated residual errors and scene-level wide-area residual errors to local feature space paths and global image space paths, respectively. Then, the semantic features with appropriate depth are selected by the frozen pre-trained visual encoder for controlled residual processing to achieve local and global collaborative correction.
It significantly improves the local sharpness and overall visual quality of nighttime severe weather images, keeps scene content intact, avoids over-enhancement and color drift, and provides more reliable image input.
Smart Images

Figure CN123118404A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, specifically to an image restoration method and system under adverse weather conditions at night, which can be applied to intelligent driving, road monitoring, security cameras, drone and robot vision, nighttime mobile photography, and other adverse weather visual enhancement scenarios under low light or complex artificial lighting conditions. Background Technology
[0002] Rain, fog, snow, raindrops, and rain streaks reduce scene visibility, obscure object textures, and cause contrast degradation. Nighttime environments also suffer from insufficient illumination, uneven distribution of artificial light sources, headlight and streetlight halos, reflections, and localized overexposure. When weather degradation and nighttime lighting are coupled, different spatial extents of degradation often coexist in the same image: raindrops, snow spots, rain streaks, and artifacts near light sources are mostly concentrated in local areas, while fog, overall brightness deviation, color shift, and insufficient visibility affect a wider area.
[0003] With the increasing demand for all-weather visual perception from applications such as intelligent driving, road monitoring, and drone inspection, image sharpening under adverse weather conditions at night has become a highly anticipated research topic in the field of computer vision. Existing research mostly focuses on single degradation types, while real-world scenarios often involve multiple degradation types stacked together, posing a greater challenge to image restoration.
[0004] Currently, the mainstream technical approaches for nighttime severe weather image restoration can be categorized as follows:
[0005] (1) Traditional methods based on physical models. These methods are based on atmospheric scattering models or Retinex theory, and recover images by estimating transmittance, atmospheric light value, or illuminance components. For example, one all-day haze image sharpening and recovery method extracts effective edge contours through non-sky region segmentation, estimates image complexity based on the main edges, and adaptively adjusts the dehazing weights. Simultaneously, to address the problem of complex artificial light source interference in nighttime haze images, it employs multi-scale Retinex theory with color restoration to remove illuminance components. Another weather-adaptive fast image sharpening processing method achieves image sharpening under various weather conditions, including dense fog, rain, and nighttime, through ambient light prediction, atmospheric light intensity estimation, and fog concentration distribution calculation. However, these methods rely on manually designed prior assumptions and are prone to large estimation errors under complex nighttime lighting conditions.
[0006] (2) End-to-end restoration methods based on deep learning. In recent years, convolutional neural networks (CNN), visual transformers (ViT), and multi-scale hybrid networks have been widely used in image restoration tasks. Existing solutions typically employ an encoder-decoder structure to directly map nighttime severe weather images into clear images. For example, the VQCNIR method provides high-quality priors through vector quantization codebooks and introduces an adaptive illumination enhancement module and a deformable bidirectional cross-attention module to restore details and illumination of nighttime images. A low-illumination overlay rain and fog processing method based on PFRD-Net uses a shared encoder to extract multi-scale primary features and processes rain and fog features and illumination features in parallel through a dual-branch module. Some solutions also introduce weather cues, degradation priors, or pre-trained visual semantic features to assist in restoration.
[0007] The existing technical solutions have the following main shortcomings:
[0008] First, there is a lack of explicit classification and division of labor for residual errors. Existing methods typically delegate all errors to the same backbone network or the same scale correction module, lacking explicit modeling of residual errors that still exist in the output of the main recovery network, and failing to distinguish between local concentrated residuals and scene-level wide-area residuals.
[0009] Second, the correction path is singular and struggles to address both local and global issues. When using a single scale or path for subsequent correction, local raindrops, snow spots, and glare residues may not be adequately removed, while large-scale fogging, brightness, and color deviations may also be insufficiently corrected. Although some existing methods combine global and local residual learning, their primary purpose is to address training problems such as gradient vanishing and network parameter growth, rather than to address residual errors in different spatial ranges within the image content through differentiated correction.
[0010] Third, uniform strong correction can easily damage weak textures and edges. Existing methods tend to misjudge weak textures, road edges, vehicle outlines, and low-contrast targets as degradation and weaken them under low illumination conditions.
[0011] Fourth, the use of semantic priors is limited. Some solutions employ pre-trained visual features to assist in recovery, but these typically use fixed levels or the same injection method, failing to select semantic depth based on the different responsibilities of each stage, such as encoding, decoding, high-resolution refinement, and global correction.
[0012] Fifth, there is a lack of constraints on the update amplitude. When existing methods lack explicit constraints on the local update and global residual amplitude, they are prone to over-enhancement, color drift, or new artifacts.
[0013] Therefore, it is necessary to establish a multi-path collaborative method that takes into account both local and global aspects and can select an appropriate semantic depth based on the responsibilities of different stages to improve the overall restoration quality of severe weather images under complex nighttime lighting conditions and provide more reliable image input for subsequent possible visual perception tasks. Summary of the Invention
[0014] In view of this, the present invention provides a method and system for nighttime severe weather image restoration based on residual allocation. After obtaining basic restoration features in the main restoration network, the method allocates local concentrated residual errors and scene-level wide-area residual errors to local feature space paths and global image space paths, respectively. Simultaneously, according to the responsibilities of each restoration stage, semantic features adapted to the depth are selected and fused from the frozen pre-trained visual encoder to anchor scene content in a controlled residual manner. This allows for selective processing of local raindrops, snow spots, and halo residues, while ensuring consistent processing of fogging, brightness, and color deviations across the entire scene, avoiding the inconsistency of a single correction range.
[0015] This invention discloses a method for nighttime severe weather image restoration based on residual allocation, which includes the following steps:
[0016] S1, Obtain the nighttime severe weather image to be restored as the input image;
[0017] S2, the input image is input to the main recovery network to obtain the main recovery features in the high-resolution feature space; the main recovery network includes at least an encoding stage, a decoding stage and a high-resolution thinning stage; wherein, based on the spatial scale occupied by the residual error in the main recovery features, a local residual allocation path for processing local concentrated residual errors is set in the high-resolution feature space, and a global residual allocation path for processing scene-level wide-area residual errors is set in the image space;
[0018] S3, the local residual allocation path of the input local residual is localized by the main recovery feature and the feature representation of the input image to generate local correction residual, and then fused with the main recovery feature to obtain local correction feature;
[0019] S4, extract semantic features from the input image using a frozen pre-trained visual encoder, and inject the semantic features into the stage recovery features of the current stage of the main recovery network in a controlled residual manner; wherein, in the high-resolution thinning stage, the local correction features are used as stage recovery features for semantic anchoring to obtain corresponding semantic enhancement features.
[0020] S5, input the semantic enhancement features obtained based on local correction features into the image reconstruction head, predict the image residual and add it to the input image to obtain a preliminary restored image;
[0021] S6, input the preliminary restored image into the global residual allocation path, and combine the input image, the semantic enhancement features, and the difference information between the preliminary restored image and the input image to predict the global residual, and fuse it with the preliminary restored image to obtain the final restored image;
[0022] S7, output the final restored image.
[0023] The specific implementation process of the method of the present invention is as follows:
[0024] Let the nighttime severe weather image be the input image. In step S1, the nighttime severe weather image to be recovered is obtained as the input for subsequent processing.
[0025] In step S2, the input image is fed into a hierarchical master restoration network. The master restoration network includes at least a shallow feature extraction stage, an encoding stage, a decoding stage, and a high-resolution thinning stage. This network performs multi-scale feature extraction and preliminary restoration on the input image, outputting master restoration features in a high-resolution feature space. Here, "high-resolution feature space" refers to the space where the feature map of the master restoration network performs refined feature processing at the highest spatial resolution.
[0026] Residual errors remaining in the main restored features that were not fully removed are distinguished based on their spatial scale within the image. Specifically, if the spatial range of the residual error is limited to a small local area in the image, it is considered a "locally concentrated residual error"; if the spatial range of the residual error covers the entire image, it is considered a "scene-level wide-area residual error". Locally concentrated residual errors mainly include residual rain streaks, raindrops, snow spots, local halos, and artifacts near edges; scene-level wide-area residual errors mainly include overall fogging residue, insufficient visibility, brightness deviation, and color shift. A local residual allocation path for processing locally concentrated residual errors is set in the high-resolution feature space, and a global residual allocation path for processing scene-level wide-area residual errors is set in the image space, so that locally concentrated residual errors and scene-level wide-area residual errors can be processed based on different residual paths.
[0027] In step S3, the local residual assignment takes the principal recovery feature and the feature representation of the input image as input. Local feature update values and spatial gating are generated through context fusion. After applying amplitude constraints and spatial smoothing to the local feature update values, spatial gating modulation is applied to generate local correction residuals. These local correction residuals are then added to the principal recovery feature to obtain the local correction feature. The amplitude of the local correction residual is limited by at least one constraint mechanism, including but not limited to hyperbolic tangent function pre-constraint, spatial smoothing operation, spatial gating modulation, and scaling factor control.
[0028] In step S4, the frozen pre-trained visual encoder extracts semantic features at multiple abstract levels (including shallow, mid-level, and deep semantics) from the input image, and injects these semantic features into the current stage of the main recovery network's recovery features using a controlled residual approach. Specifically, in the high-resolution thinning stage, local correction features are used as the current stage's recovery features for semantic anchoring to obtain semantically enhanced features. The "controlled residual approach" means that semantic information is not used as a replacement to directly cover image features, but rather as a correction term, injected into the recovery features in a small, spatially adaptive manner through semantic residual anchors and content-related modulation maps.
[0029] In step S5, the semantic enhancement features are input to the image reconstruction head, predicting image-level residuals, which are then added to the input image. After truncation, a preliminary restored image is obtained. In step S6, a global residual allocation path is set in the image space. This path takes the preliminary restored image as input, combines the input image, the semantic enhancement features, and the difference information between the preliminary restored image and the input image to establish a global context, predict amplitude-constrained three-channel global residuals, and add the three-channel global residuals to the preliminary restored image. After truncation, the final restored image is obtained. In step S7, the final restored image is output.
[0030] In the method of this invention, specialized correction of residual errors at different spatial scales is achieved through a "local-global dual-path divide-and-conquer" approach. The local path refines local residuals such as raindrops and halos in the high-resolution feature space, while the global path holistically handles wide-area degradation such as fogging and brightness deviations in the image space. The two paths work collaboratively in stages: first, the local path completes the fine correction of local residuals in the high-resolution feature space, and then the global path completes the overall consistency adjustment in the image space, achieving a combined effect of "local refinement + global consistency" that a single network cannot achieve simultaneously. Simultaneously, semantic guidance for content preservation is achieved through semantic injection using a "controlled residual method." Specifically, semantic information is injected into the restored features as a small correction term through semantic residual anchors and content-related modulation maps, avoiding the destruction of restored content caused by direct replacement. The modulation map adaptively adjusts the injection intensity according to the image content, reducing the intensity in edge and texture regions to protect details.
[0031] According to the method disclosed in the first aspect of the present invention, in step S2, the main recovery network further includes a shallow feature extraction stage, and the high-resolution thinning stage is set after the decoding stage; the main recovery network further includes pre-stage features before the high-resolution thinning stage; in step S3, the main recovery features, pre-stage features, and feature representation of the input image are used together as input objects for the local residual allocation path; the local residual allocation path generates local feature update amounts and spatial gating by performing context fusion on the main recovery features, pre-stage features, and feature representation of the input image, and applies amplitude constraints and spatial smoothing to the local feature update amounts, and then generates the local correction residuals by spatial gating modulation.
[0032] This three-source input design enables local paths to simultaneously utilize structural information before thinning, the restored results after thinning, and appearance cues from the original image, achieving more accurate local residual estimation.
[0033] According to the method disclosed in the first aspect of the present invention, in step S3, the local correction residual is generated in the following manner: ;in, This represents the amount of local feature updates. For spatial gating, For spatial smoothing operations, These are amplitude constraint parameters. Used to limit the magnitude of local feature updates. The overall strength adjustment parameter is defined as follows; the local correction feature is defined as follows: ; The main recovery feature.
[0034] According to the method disclosed in the first aspect of the present invention, in step S6, the input to the global residual allocation path includes the semantic enhancement features, the input image, the preliminary restored image, and difference information; the global residual is generated in the following manner: in, This represents a global context aggregation operation used to fuse semantically enhanced features. Input image Preliminary image restoration and differential information ; This represents the semantic features used in the global correction phase; This indicates a global residual prediction operation; This represents the global residual magnitude constraint parameter.
[0035] According to the method disclosed in the first aspect of the present invention, in step S5, the preliminary restored image is generated in the following manner: ;in, Represents an image reconstruction map used to enhance semantic features. Generate image-level recovery residuals. This is a truncation operation used to limit pixel values to a valid range.
[0036] According to the method disclosed in the first aspect of the present invention, the global residual is a three-channel image residual with the same spatial size as the preliminary restored image; when the global residual is predicted at a spatial resolution lower than that of the input image, the global residual is scaled before being fused with the preliminary restored image.
[0037] According to the method disclosed in the first aspect of the present invention, in step S3, the local concentrated residual error includes at least one of residual rain streaks, raindrops, snow spots, local halos, and artifacts near the edges; the scene-level wide-area residual error includes at least one of overall fogging residue, insufficient visibility, brightness deviation, and color shift.
[0038] According to the method disclosed in the first aspect of the present invention, in step S4, the semantic features extracted by the pre-trained visual encoder from the input image include shallow semantics, mid-level semantics, and deep semantics; the shallow semantics include texture and edge information, the mid-level semantics include region structure information, and the deep semantics include scene-level content information.
[0039] According to the method disclosed in the first aspect of the present invention, in step S4, the controlled residual injection includes: selecting one of the shallow semantics, mid-level semantics and deep semantics as the main semantic level according to the function of the current stage, and fusing the semantics of other levels with learnable weights as auxiliary semantics to form a hybrid semantic feature; the recovery stage includes at least one of the encoding stage, decoding stage, high-resolution refinement stage and global correction stage.
[0040] According to the method disclosed in the first aspect of the present invention, the high-resolution refinement stage preferentially selects the shallow semantics as the main semantic level; the encoding stage and / or decoding stage preferentially selects the mid-level semantics as the main semantic level; and the global correction stage preferentially selects the deep semantics as the main semantic level.
[0041] According to the method disclosed in the first aspect of the present invention, the hybrid semantic features are generated in the following manner: ;in, For the first The primary semantic level index for each recovery phase. The semantic features corresponding to the main semantic level. The main semantic proportion control parameter. To assist semantic weights, and .
[0042] According to the method disclosed in the first aspect of the present invention, step S4, the controlled residual injection, further includes: inputting the semantic features into a semantic fusion unit, wherein the semantic fusion unit simultaneously generates a semantic residual anchor and a content-related modulation map based on the stage recovery features of the current stage and the semantic features; the content-related modulation map has the same spatial size as the recovery features, and each element of the map takes a value between 0 and 1, which is used to adjust the spatial injection intensity of the semantic residual anchor according to the image content; and fusing the semantic residual anchor modulated by the content-related modulation map with the stage recovery features to obtain semantically enhanced features, i.e., semantically enhanced stage recovery features.
[0043] According to the method disclosed in the first aspect of the present invention, the fusion method of semantic residual anchor and stage recovery feature is as follows: in, This represents the recovery characteristics at the current stage. For the semantic residual anchor, This is a content-related modulation map. This is the stage strength coefficient. For modulation intensity parameters, This indicates element-wise multiplication.
[0044] According to the method disclosed in the first aspect of the present invention, the main recovery network, the local residual allocation path, and the global residual allocation path are optimized by end-to-end joint training. During training, paired nighttime severe weather images and clear target images are used as training data, and the pixel domain reconstruction loss and frequency domain loss between the final recovered image and the clear target image are used as the basic loss, supplemented by at least one of global correction auxiliary loss, local correction auxiliary loss, and semantic auxiliary loss as auxiliary loss.
[0045] According to the method disclosed in the first aspect of the present invention, the basic reconstruction loss is: in, To finally restore the image, To obtain a clear target image, It is an L1 norm. For frequency domain loss, This represents the frequency domain loss weighting coefficient.
[0046] According to the method disclosed in the first aspect of the present invention, the total loss during the training process is: in, Based on the loss of reconstruction, To globally correct auxiliary loss, To locally correct auxiliary loss, For semantic auxiliary loss, This is a weight function that gradually increases as the training process progresses.
[0047] According to the method disclosed in the first aspect of the present invention, the spatial smoothing operation is implemented using at least one of Gaussian smoothing, average pooling, or depthwise separable convolution; the spatial gating is generated by the Sigmoid activation function.
[0048] According to the method disclosed in the first aspect of the present invention, the pre-trained visual encoder adopts any one of ResNet, ViT or DINO pre-trained on ImageNet; the shallow semantics, the middle semantics and the deep semantics are respectively taken from different depth levels of the pre-trained visual encoder.
[0049] According to the method disclosed in the first aspect of the present invention, the nighttime severe weather image includes at least one of rain, fog, snow, raindrops, rain streaks, halo, glare and mixed degradation of the nighttime environment.
[0050] A second aspect of the present invention discloses a system for implementing the residual allocation-based nighttime severe weather image restoration method disclosed in the first aspect of the present invention, comprising:
[0051] The image input module is used to receive the nighttime severe weather image to be recovered as the input image;
[0052] The main recovery network module is used to extract features from the input image and output the main recovery features in the high-resolution feature space. The main recovery network module includes at least an encoding stage, a decoding stage, and a high-resolution thinning stage. Based on the spatial scale occupied by the residual error in the main recovery features, a local residual allocation path is set in the high-resolution feature space to handle local concentrated residual errors, and a global residual allocation path is set in the image space to handle scene-level wide-area residual errors.
[0053] The local residual allocation module is used to receive the main recovery feature in the high-resolution feature space, perform local correction on the main recovery feature, generate a local correction residual, and fuse the local correction residual with the main recovery feature to output the local correction feature.
[0054] The semantic guidance module is used to extract semantic features from the input image using a frozen pre-trained visual encoder, and inject the semantic features into the stage recovery features of the current stage of the main recovery network in a controlled residual manner; wherein, in the high-resolution thinning stage, the local correction features are used as stage recovery features for semantic anchoring to obtain corresponding semantic enhancement features.
[0055] The preliminary reconstruction module is used to map the semantic enhancement features obtained based on the local correction features to the image space through the image reconstruction head, reconstruct and output the preliminary restored image;
[0056] A global residual allocation module is used to receive the preliminary restored image in the image space, and combine the input image, the semantic enhancement features, and the difference information between the preliminary restored image and the input image to predict the global residual, fuse the global residual with the preliminary restored image, and output the final restored image;
[0057] An image output module is used to output the final restored image.
[0058] According to the system disclosed in the second aspect of the present invention, the semantic guidance module includes:
[0059] The semantic feature extraction unit includes a frozen pre-trained visual encoder for extracting shallow semantics, mid-level semantics, and deep semantics from the input image;
[0060] The stage-aware semantic routing unit is used to select the main semantic level from the shallow semantic, middle semantic and deep semantic according to the role of the current stage, and integrate the semantics of other levels as auxiliary semantics to output hybrid semantic features;
[0061] The semantic anchoring injection unit is used to generate a semantic residual anchor and a content-related modulation map based on the stage recovery features of the current stage and the hybrid semantic features. After spatially adaptively modulating the semantic residual anchor using the content-related modulation map, the modulated semantic residual anchor is fused with the recovery features to output semantic enhancement features.
[0062] A third aspect of the present invention discloses an electronic device comprising a memory and a processor, the memory for storing a computer program, the processor for running the computer program to cause the electronic device to perform the nighttime severe weather image restoration method based on residual allocation disclosed in the first aspect of the present invention.
[0063] The fourth aspect of the present invention discloses a readable storage medium storing computer program instructions, which, when read and executed by a processor, perform the nighttime severe weather image restoration method based on residual allocation disclosed in the first aspect of the present invention.
[0064] In summary, the present invention has the following beneficial effects:
[0065] (1) By employing a dual-path collaborative correction approach, both local detail restoration and global consistency adjustment are achieved. The local path selectively restores local residual artifacts such as raindrops, snow spots, and halos through spatial gating, while amplitude constraints and spatial smoothing ensure gentle correction and natural transitions. The global path, in conjunction with differential information and semantic guidance, performs consistency correction on brightness deviations, fog residues, and color shifts across the entire scene. The two approaches complement each other, significantly improving the local sharpness and overall visual quality of the restored image.
[0066] (2) Through stage-aware semantic anchoring, scene content is effectively preserved during the restoration process. Based on the different responsibilities of the stages such as encoding, decoding, high-resolution refinement and global correction, the adaptation level is adaptively selected from shallow, medium and deep semantics, and injected in the form of controlled residuals through semantic residual anchors and modulation maps, rather than direct replacement. While removing degradation, key content such as road edges, vehicle outlines and weak textures are preserved, avoiding over-smoothing.
[0067] (3) By using multiple constraint mechanisms, the risks of over-enhancement and color drift are reduced. The local path is controlled by gating selection, amplitude constraint and spatial smoothing, while the global path is controlled by amplitude constraint to limit the adjustment range, so that the correction process is gentle and controllable, and the restoration results are guaranteed in terms of visual quality and color fidelity.
[0068] The following describes in detail the method for nighttime severe weather image restoration based on residual allocation according to the embodiments shown in the accompanying drawings and reference numerals. Attached Figure Description
[0069] Figure 1 A flowchart of the nighttime severe weather image restoration method based on residual allocation of the present invention is shown.
[0070] Figure 2 A schematic diagram of the overall structure of the nighttime severe weather image restoration method is shown.
[0071] Figure 3 A schematic diagram of the dual-path system mechanism based on residual allocation in this invention is shown.
[0072] Figure 4 A schematic diagram of the module framework of the semantic guidance module in this invention is shown. Detailed Implementation
[0073] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.
[0074] It should be noted that all directional indicators (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative positional relationship and movement of each component in a specific posture (as shown in the figure). If the specific posture changes, the directional indicator will also change accordingly.
[0075] Furthermore, the use of terms such as "first" and "second" in this invention is for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature. Additionally, the technical solutions of the various embodiments can be combined with each other, but only on the basis of being achievable by those skilled in the art. If the combination of technical solutions is contradictory or impossible to implement, such a combination of technical solutions should be considered non-existent and not within the scope of protection claimed by this invention.
[0076] In this article, the nighttime severe weather images refer to images affected by rain, fog, snow, raindrops, rain streaks, halos, glare, and their mixed degradation in nighttime or low-light environments.
[0077] The master restoration network is an image restoration network used to extract multi-scale features from the input image and generate basic restoration features.
[0078] Residual allocation refers to the process of assigning local concentrated residual errors and scene-level wide-area residual errors to local feature space paths and global image space paths for correction, based on the spatial influence range of residual errors during the basic restoration process.
[0079] Stage-aware semantic anchoring refers to selecting and fusing pre-trained visual semantic features of different depths according to the responsibilities of different recovery stages such as encoding, decoding, high-resolution refinement or global correction, and constraining scene content in a controlled residual manner.
[0080] , , These represent the main recovery feature, the feature after local residual correction, and the high-resolution feature after semantic enhancement, respectively.
[0081] , These represent the preliminary restored image and the final restored image, respectively.
[0082] according to Figure 1As shown, this invention discloses a method for nighttime severe weather image restoration based on residual allocation, comprising the following steps: S1, acquiring the nighttime severe weather image to be restored as the input image; S2, inputting the input image into a main restoration network to obtain the main restoration features in the high-resolution feature space; the main restoration network includes at least an encoding stage, a decoding stage, and a high-resolution thinning stage; wherein, based on the spatial scale occupied by the residual error in the main restoration features, a local residual allocation path for processing local concentrated residual errors is set in the high-resolution feature space, and a global residual allocation path for processing scene-level wide-area residual errors is set in the image space; S3, at least the main restoration features and the feature representation of the input image are input into the local residual allocation path for local correction to generate local correction residuals, and then fused with the main restoration features. S4. Obtain local correction features; S5. Extract semantic features from the input image using the frozen pre-trained visual encoder, and inject the semantic features into the stage recovery features of the current stage of the main recovery network in a controlled residual manner; wherein, in the high-resolution thinning stage, the local correction features are used as stage recovery features for semantic anchoring to obtain corresponding semantic enhancement features; S6. Input the semantic enhancement features obtained based on the local correction features into the image reconstruction head, predict the image residual and add it to the input image to obtain a preliminary recovery image; S7. Input the preliminary recovery image into the global residual allocation path, and combine the input image, the semantic enhancement features and the difference information between the preliminary recovery image and the input image to predict the global residual, and fuse it with the preliminary recovery image to obtain the final recovery image; S8. Output the final recovery image.
[0083] Step S1 is the image input and preprocessing step. Specifically, the nighttime severe weather image X to be recovered is acquired, and its size is adjusted, its values are normalized, and its color space is transformed (optionally); during the training phase, the corresponding clear target image Y is acquired simultaneously.
[0084] Step S2 is the main feature extraction step, specifically, the input image... Input a hierarchical master recovery network. The master recovery network includes at least a shallow feature extraction stage, an encoding stage, a decoding stage, and a high-resolution thinning stage, and outputs master recovery features. And the features before high-resolution thinning are denoted as .
[0085] Residual errors remaining in the main restored features that were not fully removed are distinguished based on their spatial scale within the image. If the spatial range of the residual error is limited to a small local area in the image, it is considered a "locally concentrated residual error"; if the spatial range of the residual error covers the entire image, it is considered a "scene-level wide-area residual error". Locally concentrated residual errors mainly include residual rain streaks, raindrops, snow spots, local halos, and artifacts near edges; scene-level wide-area residual errors mainly include overall haze residue, insufficient visibility, brightness deviation, and color shift. A local residual allocation path for processing locally concentrated residual errors is set in the high-resolution feature space, and a global residual allocation path for processing scene-level wide-area residual errors is set in the image space, so that locally concentrated residual errors and scene-level wide-area residual errors can be processed based on different residual paths.
[0086] Step S3 is the local residual allocation step, specifically, it involves allocating the previous-level features... Main recovery features Contextual fusion is performed with the feature representation of the input image X to generate local feature updates. and spatial gating Amplitude constraints, spatial smoothing, and gated modulation are applied to the local feature updates to obtain the local correction residuals. And the local correction residuals are compared with the main recovery features. Add them together to obtain the local correction features. When different feature spaces have inconsistent dimensions, scaling transformation can be used for size matching.
[0087] Step S4 involves multi-layer semantic extraction and stage-aware anchoring. Specifically, it involves freezing the pre-trained visual encoder... Perform a forward computation to obtain the shallow semantics. Mid-level semantics and deep semantics For the k-th injection stage, a primary semantic level is selected, and complementary semantics from other levels are introduced with learnable weights, which are then fused with the current recovery features.
[0088] Step S5 is the initial image reconstruction step, which involves applying the semantic enhancement features obtained in the high-resolution stage. Input image reconstruction head, predict image residual and compare with input image Add them together to obtain a preliminary restored image. .
[0089] Step S6 is the global residual assignment step, specifically, the joint semantic enhancement features. Input image X, preliminary image reconstruction and differential information Establish a global context to predict amplitude-constrained three-channel global residuals. When the spatial dimensions of the global residual are inconsistent with those of the initially reconstructed image, size matching is performed, and the size-matched global residual is then compared with... Add them together to get the final restored image. .
[0090] Step S7 is the step of outputting the final restored image.
[0091] The implementation process of the above method can be further summarized as follows: Let the input be a nighttime severe weather image. The method of this invention includes steps such as main recovery feature extraction, local residual allocation, stage-aware semantic anchoring, preliminary image reconstruction, and global residual allocation. In parallel with the main process, the frozen pre-trained visual encoder extracts shallow, mid-level, and deep semantic features from the input image and provides content guidance to the main recovery network, semantic anchoring module, and global residual allocation module.
[0092] Specifically, such as Figure 2 As shown, input module 101 is used to receive nighttime severe weather image X; main recovery network module 102 generates main recovery features through shallow feature extraction, encoding, decoding and high-resolution thinning. The local residual allocation module 103 performs local residual allocation on the main recovery feature to obtain the local correction feature. The semantic anchoring module 104 performs semantic anchoring on the local correction features to obtain semantically enhanced features. The image reconstruction module 105 reconstructs and restores the image based on semantic enhancement features. The global residual allocation module 106 predicts the scene-level global residual and combines the global residual with the preliminary reconstructed image. Perform residual fusion to obtain the final restored image. Output module 107 is used to output the final restored image; semantic feature extraction module 108 utilizes a frozen pre-trained visual encoder from the input image The shallow, medium and deep semantic features are extracted and used in the main recovery process, semantic anchoring process and global residual correction process.
[0093] It should be noted that in step S3, a local residual allocation path is set in the high-resolution feature space where the main restored feature resides. The local residual allocation path takes the main restored feature, the preceding features before the high-resolution thinning stage, and the feature representation (or primary features) of the input image as input objects. It generates local feature update amounts and spatial gating through context fusion of the input objects. After applying amplitude constraints and spatial smoothing to the local feature update amounts, it generates local correction residuals through spatial gating modulation. Then, the local correction residuals are added to the main restored feature to obtain the local correction feature. The amplitude of the local correction residuals is limited by at least one constraint mechanism, including but not limited to hyperbolic tangent function pre-constraint, spatial smoothing operation, spatial gating modulation, and scaling factor control.
[0094] Specifically, the localized concentrated residual error includes at least one of residual rain streaks, raindrops, snow spots, local halos, and artifacts near the edges; the scene-level wide-area residual error includes at least one of overall fogging residue, insufficient visibility, brightness deviation, and color shift.
[0095] In a preferred embodiment, the above spatial smoothing operation employs Gaussian smoothing, but can also be implemented using at least one of average pooling or depthwise separable convolution; spatial gating is generated using the Sigmoid activation function.
[0096] If the spatial dimensions of the local feature update and the main restored feature are different, size matching can be achieved by interpolation, convolution, pixel rearrangement, or other scaling transformations before the residuals are added. Through the above gating, smoothing, and amplitude constraints, the local path can generate selective and controllable feature corrections for locations with local residual artifacts.
[0097] In step S4, the frozen pre-trained visual encoder is used to extract semantic features at multiple abstract levels from the input image, including shallow semantics, mid-level semantics and deep semantics. The shallow semantics contain texture and edge information, the mid-level semantics contain region structure information, and the deep semantics contain scene-level content information.
[0098] In the high-resolution refinement stage, local correction features are used as the restoration features for the current stage to perform semantic anchoring, thereby obtaining semantically enhanced features. The "controlled residual method" refers to the fact that semantic information is not used as a replacement to directly cover image features, but rather as a correction term. It is injected into the restoration features in a small, spatially adaptive manner through semantic residual anchors and content-related modulation maps.
[0099] The pre-trained visual encoder can be any one of ResNet, ViT, or DINO pre-trained on ImageNet; the shallow semantics, mid-level semantics, and deep semantics are taken from different depth levels of the pre-trained visual encoder.
[0100] Based on the role of the current stage, one of the shallow semantics, mid-level semantics, and deep semantics is selected as the main semantic level, and the semantics of other levels are fused with learnable weights as auxiliary semantics to form hybrid semantic features; the recovery stage includes at least one of the encoding stage, decoding stage, high-resolution refinement stage, and global correction stage.
[0101] The high-resolution refinement stage preferentially selects the shallow semantics as the main semantic level; the encoding stage and / or decoding stage preferentially selects the mid-level semantics as the main semantic level; and the global correction stage preferentially selects the deep semantics as the main semantic level.
[0102] Further, the semantic features are input into a semantic fusion unit, which simultaneously generates a semantic residual anchor and a content-related modulation map based on the current stage recovery features and the semantic features. The content-related modulation map has the same spatial size as the recovery features, and each element takes a value between 0 and 1, which is used to adjust the spatial injection intensity of the semantic residual anchor according to the image content. The semantic residual anchor modulated by the content-related modulation map is fused with the stage recovery features to obtain the semantically enhanced features, that is, the semantically enhanced stage recovery features.
[0103] In step S5, the semantic enhancement features are input into the image reconstruction head, the image-level residual is predicted, and added to the input image. After truncation, a preliminary restored image is obtained.
[0104] Specifically, the preliminary restored image is generated as follows: ;in, Represents an image reconstruction map used to enhance semantic features. Generate image-level recovery residuals. This is a truncation operation used to limit pixel values to a valid range.
[0105] Preferably, the method of the present invention further includes a training step, wherein the main recovery network, the local residual allocation path, and the global residual allocation path are jointly optimized in advance through the following training steps: using paired nighttime severe weather images and clear target images as training data, using pixel domain reconstruction loss and frequency domain loss between the final recovered image and the clear target image as the basic loss, and supplemented by at least one of global correction auxiliary loss, local correction auxiliary loss, and semantic auxiliary loss as auxiliary loss, the main recovery network, the local residual allocation path, and the global residual allocation path are jointly optimized end-to-end.
[0106] Specifically, during training, the final recovered image With clear target image The basic reconstruction loss can include pixel-domain absolute value loss and frequency-domain loss. Specifically, the basic reconstruction loss is: in, To finally restore the image, To obtain a clear target image, It is an L1 norm. For frequency domain loss, This represents the frequency domain loss weighting coefficient.
[0107] In addition, one way to implement the total loss during training is as follows: in, Based on the loss of reconstruction, To globally correct auxiliary loss, To locally correct auxiliary loss, For semantic auxiliary loss, The weight function gradually increases as training progresses. The global auxiliary loss is used to supervise scene-level correction, the local auxiliary loss is used to supervise local selective correction, and the semantic auxiliary loss is used for semantic conditions or semantic consistency constraints. The weight function, which gradually increases as training progresses, is used to prevent auxiliary branches from interfering with the main recovery network in the early stages of training.
[0108] In a preferred embodiment, the main recovery network employs a multi-level encoder-decoder structure and preserves the features before refinement during the high-resolution recovery stage. and refined features Local residual assignment path pairs , and input image The feature representations are scale-matched and context-fused to generate local correction residuals; the pre-trained visual encoder keeps the parameters frozen and outputs multi-layer semantic features from different network depths; global residuals are assigned to path pairs. , , and Context aggregation is performed, and a three-channel global residual is generated. The processing resolution of each module can be set according to the network structure and computing resources. Size matching can be performed between features at different scales using interpolation, convolution, pooling, or other scale transformation methods.
[0109] A second aspect of the present invention discloses a system for implementing the residual allocation-based nighttime severe weather image restoration method disclosed in the first aspect of the present invention, comprising:
[0110] The image input module is used to receive the nighttime severe weather image to be recovered as the input image;
[0111] The main recovery network module is used to extract features from the input image and output the main recovery features in the high-resolution feature space; the main recovery network module includes at least an encoding stage, a decoding stage, and a high-resolution thinning stage;
[0112] The local residual allocation module is used to receive the main recovery feature in the high-resolution feature space, perform local correction on the main recovery feature, generate a local correction residual, and fuse the local correction residual with the main recovery feature to output the local correction feature.
[0113] The semantic guidance module is used to extract semantic features from the input image using a frozen pre-trained visual encoder, and inject the semantic features into the stage recovery features of the current stage of the main recovery network in a controlled residual manner; wherein, in the high-resolution thinning stage, the local correction features are used as stage recovery features for semantic anchoring to obtain corresponding semantic enhancement features.
[0114] The preliminary reconstruction module is used to map the semantic enhancement features obtained based on the local correction features to the image space through the image reconstruction head, reconstruct and output the preliminary restored image;
[0115] A global residual allocation module is used to receive the preliminary restored image in the image space, and combine the input image, the semantic enhancement features, and the difference information between the preliminary restored image and the input image to predict the global residual, fuse the global residual with the preliminary restored image, and output the final restored image;
[0116] An image output module is used to output the final restored image.
[0117] The local residual allocation module includes a local context aggregation unit, which uses the local context aggregation unit to process the previous features. Main recovery features and input image The feature representations are fused and local feature updates are generated separately. and spatial gating .
[0118] In one implementation, the local correction residual after amplitude constraint, spatial smoothing, and gated modulation can be expressed as: The local correction features are obtained according to the following formula: .
[0119] in, This indicates a space smoothing operation. Used to limit the magnitude of local feature updates. Used to adjust the overall strength of local residuals For spatial gating, the value can be limited to 0 to 1. If the spatial dimensions of the local feature update and the main recovered feature are different, size matching can be achieved by interpolation, convolution, pixel rearrangement, or other scale transformations before the residuals are added. Through the above gating, smoothing, and amplitude constraints, the local path can generate selective and controllable feature corrections for locations with local residual artifacts.
[0120] Additionally, the global residual allocation module is set after the initial image and is used to handle residual fog, insufficient visibility, overall brightness deviation, and color shift.
[0121] The preliminary and final images can be represented as follows: ;in, Represents an image reconstruction map used to enhance semantic features. Generate image-level recovery residuals.
[0122] The global residual assignment module generates a three-channel global residual that matches the spatial dimensions of the initially reconstructed image. The finally recovered image can be represented as When the global residual prediction process is performed at a spatial resolution lower than that of the input image, interpolation, convolution, pixel rearrangement, or other scaling transformations can be used before the residuals are summed to adjust the generated residuals to a resolution comparable to the input image. Same space dimensions.
[0123] The image residual reconstruction head generates the initial image residual, and the truncation operator limits the pixel values to a valid range. To avoid excessive image updates during the global correction process, in one implementation, the global residual can be expressed as:
[0124] ; in, This represents a global context aggregation operation used to fuse semantically enhanced features. Input image Preliminary image restoration and differential information ; This represents the semantic features used in the global correction phase; This indicates a global residual prediction operation; This represents the global residual magnitude constraint parameter. The context aggregation operation can be implemented using convolution, pooling, multi-scale feature aggregation, attention, or other methods that can obtain scene-level context information.
[0125] That is, in the method and system of the present invention, a dual-path collaborative mechanism 203 based on residual allocation is established. Figure 3 A schematic diagram of the dual-path system mechanism based on residual allocation in this invention is shown, such as... Figure 3 As shown, the dual-path collaborative mechanism 203 constructed by the present invention includes a local residual allocation path 201 and a global residual allocation path 202.
[0126] Combination Figure 3 As shown, the local residual assignment path 201 uses the primary recovery feature. Input image Feature representation and preceding features As input, local context aggregation, feature update, gated modulation, and amplitude constraint are performed by the local context aggregation unit to generate local correction residuals. And the local correction residuals are compared with the main recovery features. The addition yields the local correction features. .
[0127] Further integration Figure 3 As shown, global residual assignment path 202 is used to initially recover the image. Input image Semantic enhancement features Sum and difference information Using the global context modeling unit as the main input, and the semantic modulation / global mapping unit, perform semantic modulation to generate a three-channel global residual with constrained amplitude. And the global residual is compared with the preliminary reconstructed image. The final restored image is obtained by adding them together. .
[0128] The dual-path collaborative mechanism 203 enables the local residual allocation path 201 and the global residual allocation path 202 to undertake residual error correction tasks in different spatial ranges. The local residual allocation path 201 is used to correct local concentrated residual artifacts such as raindrops, snow spots, and local glare, while the global residual allocation path 202 is used to correct large-scale brightness deviations, fogging residues, and overall color shifts. The two work together to improve the image restoration effect. Figure 4 A schematic diagram of the module framework of the semantic guidance module in this invention is shown. (Combined with...) Figure 4As shown, the semantic guidance module includes a semantic feature extraction unit 301, a stage-aware semantic routing unit 302, and a semantic anchoring injection unit 303. The semantic feature extraction unit 301 includes a frozen pre-trained visual encoder, used to extract shallow semantics, mid-level semantics, and deep semantics from the input image. The stage-aware semantic routing unit 302 is used to select the main semantic level from the shallow, mid-level, and deep semantics according to the role of the current stage, and fuse the semantics of other levels as auxiliary semantics to output mixed semantic features. The semantic anchoring injection unit is used to generate semantic residual anchors and content-related modulation maps according to the stage recovery features of the current stage and the mixed semantic features, and then use the content-related modulation maps to perform spatial adaptive modulation on the semantic residual anchors, and fuse the modulated semantic residual anchors with the recovery features to output semantic enhancement features.
[0129] Specifically, in combination Figure 4 As shown, the semantic feature extraction unit 301 provides shallow semantic features. Mid-level semantic features and deep semantic features The stage-aware semantic routing unit 302 determines the main semantic level based on the current recovery stage and uses auxiliary semantic weights to fuse complementary semantic information from other levels to form hybrid semantic features. Semantic anchoring injection unit 303 restores features based on current characteristics and hybrid semantic features Generate semantic residuals and modulation diagram After controlling the modulation of the semantic residual using the modulation map, the generated controlled semantic residual is then compared with the original recovered features. Add them together to obtain the anchored features. The semantic anchoring is used to preserve key scene content such as road edges, vehicle outlines, and weak textures during residual correction. Typical injection location 304 can include encoding, decoding, high-resolution refinement, and global correction stages. The semantic guidance module outputs semantic features at different abstraction levels through a frozen pre-trained visual encoder. Shallow semantics contain more texture and edge information, mid-level semantics are suitable for representing region structure, and deep semantics are suitable for representing scene-level content and category consistency. For the k-th recovery stage, hybrid semantic features are used. It can be represented as:
[0130] ;
[0131] Among them, the auxiliary semantic weighted representation is as follows: , The primary semantic level index indicates the semantic depth that is preferred in the current stage, and the primary semantic level selection is represented as... The auxiliary semantic ratio ranges from 0 to 1, and the sum of the weights of each auxiliary level after normalization is 1. Mid-level semantics are preferred in the encoding or intermediate stages, shallow semantics are preferred in the high-resolution refinement stage, and deep semantics are preferred in the coarse-scale and global correction stages.
[0132] Hybrid semantic features With recovery characteristics The fusion is represented as:
[0133] ;
[0134] ;
[0135] Among them, semantic fusion unit Simultaneously generate semantic residual anchors Content-related modulation diagram Modulation diagram The value of is between 0 and 1. The learnable scaling factor controls the residual injection intensity, while the modulation intensity parameter controls the degree of spatial adaptability. This structure allows semantic information to participate in the recovery in a controlled residual manner, rather than directly replacing the current recovered features.
[0136] In the methods and systems of the present invention, for the above-mentioned schemes, there are preferred alternatives in other embodiments. For example, the main recovery network preferably adopts a convolutional neural network, but may also adopt a U-shaped network, a visual Transformer, a state-space model, or a multi-scale hybrid network. The pre-trained semantic encoder preferably adopts any model capable of outputting multi-layer visual features, or may adopt a lightweight semantic encoder after knowledge distillation. Local context aggregation is preferably implemented by convolution, window attention, deformable convolution, or frequency domain operators; the smoothing operator may be Gaussian smoothing, average pooling, or depthwise separable convolution. Global context is preferably implemented by global average pooling, multi-scale pooling, self-attention, or low-resolution convolution; residual constraints may adopt hyperbolic tangent, sigmoid, hard truncation, or learnable amplitude constraints. Semantic hybrid weights are preferably generated through learnable parameters, gating networks, or attention networks; the semantic injection position can be added or removed according to the network structure. The training loss preferably includes structural similarity, perception, gradient, color constancy, contrast, or unpaired consistency loss.
[0137] In summary, the method of this invention achieves specialized correction of residual errors at different spatial scales through a "local-global dual-path divide-and-conquer" approach. The local path refines local residuals such as raindrops and halos in the high-resolution feature space, while the global path holistically handles wide-area degradation such as fogging and brightness deviations in the image space. The two paths work collaboratively in stages, achieving a combined effect of "local refinement + global consistency" that a single network cannot simultaneously achieve. Simultaneously, semantic guidance that preserves content is achieved through semantic injection using a "controlled residual method." Specifically, semantic information is injected into the restored features with small correction terms through semantic residual anchors and content-related modulation maps, avoiding the destruction of restored content caused by direct replacement. The modulation map adaptively adjusts the injection intensity according to the image content, reducing the intensity in edge and texture regions to protect details.
[0138] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.
Claims
1. A method for residual distribution based night-time severe weather image restoration, characterized in that, Includes the following steps: S1, Obtain the nighttime severe weather image to be restored as the input image; S2, the input image is input to the main recovery network to obtain the main recovery features in the high-resolution feature space; the main recovery network includes at least an encoding stage, a decoding stage and a high-resolution thinning stage; and, based on the spatial scale occupied by the residual error in the main recovery features, a local residual allocation path for processing local concentrated residual errors is set in the high-resolution feature space, and a global residual allocation path for processing scene-level wide-area residual errors is set in the image space. S3, at least the main recovery feature and the feature representation of the input image are used to input the local residual allocation path for local correction, generating a local correction residual, and then fused with the main recovery feature to obtain the local correction feature; S4, extract semantic features from the input image using a frozen pre-trained visual encoder, and inject the semantic features into the stage recovery features of the current stage of the main recovery network in a controlled residual manner; In the high-resolution refinement stage, the local correction features are used as stage recovery features for semantic anchoring to obtain corresponding semantic enhancement features. S5, input the semantic enhancement features obtained based on local correction features into the image reconstruction head, predict the image residual and add it to the input image to obtain a preliminary restored image; S6, input the preliminary restored image into the global residual allocation path, and combine the input image, the semantic enhancement features, and the difference information between the preliminary restored image and the input image to predict the global residual, and fuse it with the preliminary restored image to obtain the final restored image; S7, output the final restored image.
2. The method of claim 1, wherein, In step S2, the main recovery network further includes pre-stage features prior to the high-resolution refinement stage; In step S3, the main recovery feature, the previous feature, and the feature representation of the input image are used together as the input object of the local residual allocation path. The local residual allocation path generates the local feature update amount and spatial gating by performing context fusion on the input object. After applying amplitude constraints and spatial smoothing to the local feature update amount, the local correction residual is generated by the spatial gating modulation.
3. The method of claim 1, wherein, In step S4, the semantic features extracted by the pre-trained visual encoder from the input image include shallow semantics, mid-level semantics, and deep semantics; the shallow semantics include texture and edge information, the mid-level semantics include region structure information, and the deep semantics include scene-level content information.
4. The method of claim 3, wherein, In step S4, the controlled residual injection includes: selecting one of the shallow semantics, mid-level semantics and deep semantics as the main semantic level according to the role of the current stage, and fusing the semantics of other levels with learnable weights as auxiliary semantics to form a hybrid semantic feature; the recovery stage includes at least one of the encoding stage, decoding stage, high-resolution refinement stage and global correction stage.
5. The method of claim 4, wherein, The high-resolution refinement stage preferentially selects the shallow semantics as the main semantic level; the encoding stage and / or decoding stage preferentially selects the mid-level semantics as the main semantic level; and the global correction stage preferentially selects the deep semantics as the main semantic level.
6. The method of claim 3, wherein, In step S4, the controlled residual injection further includes: The semantic features are input into the semantic fusion unit, which generates a semantic residual anchor and a content-related modulation map based on the current stage recovery features and the semantic features. The content-related modulation map has the same spatial size as the recovery features, and each element takes a value between 0 and 1. It is used to adjust the spatial injection intensity of the semantic residual anchor according to the image content. The semantic residual anchor modulated by the content-related modulation graph is fused with the stage recovery feature to obtain the semantic enhancement feature.
7. The method according to claim 1, characterized in that, The main recovery network, the local residual allocation path, and the global residual allocation path are optimized using an end-to-end joint training method. During training, paired nighttime severe weather images and clear target images are used as training data. The pixel domain reconstruction loss and frequency domain loss between the final recovered image and the clear target image are used as the basic loss, supplemented by at least one of global correction auxiliary loss, local correction auxiliary loss, and semantic auxiliary loss as auxiliary losses.
8. The method according to any one of claims 1 to 7, characterized in that, The nighttime severe weather images include at least one of rain, fog, snow, raindrops, rain streaks, halo, glare, and their mixed degradation in a nighttime environment.
9. A system for implementing the nighttime severe weather image restoration method based on residual allocation according to any one of claims 1-8, characterized in that, include: The image input module is used to receive the nighttime severe weather image to be recovered as the input image; The main recovery network module is used to extract features from the input image and output the main recovery features in the high-resolution feature space; The main recovery network module includes at least an encoding stage, a decoding stage, and a high-resolution thinning stage; and based on the spatial scale occupied by the residual error in the main recovery feature, a local residual allocation path for processing local concentrated residual errors is set in the high-resolution feature space, and a global residual allocation path for processing scene-level wide-area residual errors is set in the image space. The local residual allocation module is used to receive the main recovery feature in the high-resolution feature space, perform local correction on the main recovery feature, generate a local correction residual, and fuse the local correction residual with the main recovery feature to output the local correction feature. A semantic guidance module is used to extract semantic features from the input image using a frozen pre-trained visual encoder and inject the semantic features into the stage recovery features of the current stage of the main recovery network in a controlled residual manner. In the high-resolution refinement stage, the local correction features are used as stage recovery features for semantic anchoring to obtain corresponding semantic enhancement features. The preliminary reconstruction module is used to map the semantic enhancement features obtained based on the local correction features to the image space through the image reconstruction head, reconstruct and output the preliminary restored image; A global residual allocation module is used to receive the preliminary restored image in the image space, and combine the input image, the semantic enhancement features, and the difference information between the preliminary restored image and the input image to predict the global residual, fuse the global residual with the preliminary restored image, and output the final restored image; An image output module is used to output the final restored image.
10. The system according to claim 9, characterized in that, The semantic guidance module includes: The semantic feature extraction unit includes a frozen pre-trained visual encoder for extracting shallow semantics, mid-level semantics, and deep semantics from the input image; The stage-aware semantic routing unit is used to select the main semantic level from the shallow semantic, middle semantic and deep semantic according to the role of the current stage, and integrate the semantics of other levels as auxiliary semantics to output hybrid semantic features; The semantic anchoring injection unit is used to generate a semantic residual anchor and a content-related modulation map based on the stage recovery features of the current stage and the hybrid semantic features. After spatially adaptively modulating the semantic residual anchor using the content-related modulation map, the modulated semantic residual anchor is fused with the recovery features to output semantic enhancement features.
