A weather-affected image inpainting method, device, equipment and storage medium

Abstract

This invention discloses a method, apparatus, device, and storage medium for image restoration affected by weather, belonging to the field of information recognition technology. The method includes: acquiring a blurred image affected by weather as input; constructing an image pyramid through interpolation downsampling and spatial interaction operations; downsampling the original image to multi-scale resolutions of 1 / 2, 1 / 4, and 1 / 8 respectively; extracting features from the multi-scale image using a feature extraction unit to enhance high-frequency response; performing contextual association modeling on the features using a feature modeling unit with an information transmission constraint mechanism; performing upsampling operations and skip connections on the features at each scale; and fusing the restored image features at each scale using a feature fusion module; training and inferring the model using a composite loss function to generate a high-resolution restored image output. This invention effectively improves the ability to restore image details and preserve structure under severe weather conditions.

Description

Technical Field

[0001] This invention relates to the field of information recognition technology, and in particular to a method, apparatus, device, and storage medium for image restoration affected by weather. Background Technology

[0002] With the rapid development of remote sensing technology, drone technology, and intelligent sensing devices, image data has been widely used in fields such as geographic information acquisition, environmental monitoring, target recognition, intelligent transportation, and emergency response. However, in practical applications, image acquisition is often inevitably affected by adverse weather conditions such as rain, fog, snow, and sandstorms, resulting in problems such as blurriness, reduced contrast, loss of detail, and structural distortion in the acquired images, which seriously restricts the accuracy and reliability of subsequent image analysis and applications.

[0003] To address image degradation caused by weather conditions, existing technologies primarily employ image enhancement or inpainting methods. For example, traditional methods typically rely on physical imaging models or prior assumptions to restore images through filtering, denoising, and dehazing. However, these methods often depend on idealized modeling of the degradation process, making it difficult to accurately describe the image degradation mechanism under complex and variable real-world weather conditions. This results in limited restoration effectiveness and high sensitivity to parameter settings.

[0004] With the development of deep learning technology, image inpainting methods based on neural networks have gradually become a research hotspot. Related methods construct convolutional neural networks or attention mechanism networks to perform end-to-end modeling of weather-affected images, improving the overall image quality to some extent. However, existing deep learning-based image inpainting methods still have the following shortcomings: On the one hand, some methods only focus on single-scale or local region features, making it difficult to simultaneously consider global semantic information and local structural details, resulting in insufficient detail restoration or structural distortion in the inpainting results; on the other hand, some methods lack effective context constraint mechanisms during feature modeling, easily introducing redundant information or erroneous associations, affecting the stability of the inpainting. Furthermore, some high-performance models have high parameter scale and computational complexity, making them unsuitable for deployment and application on resource-constrained devices.

[0005] Therefore, there is an urgent need for a weather-affected image restoration method and its corresponding device, equipment and storage medium that can effectively improve image clarity and structural integrity under complex weather conditions, while taking into account multi-scale feature modeling and computational efficiency, in order to overcome the shortcomings of existing technologies and improve the availability and application value of image data under severe weather conditions. Summary of the Invention

[0006] The main objective of this invention is to provide a method, apparatus, device, and storage medium for image restoration affected by weather, aiming to solve the current technical problem of insufficient image restoration capabilities affected by weather.

[0007] To achieve the above objectives, the present invention provides a method for image restoration affected by weather, the method comprising the following steps:

[0008] The image is taken as input by a blurred image affected by weather. An image pyramid is constructed by interpolation downsampling and spatial interaction operations. The original image is downsampled to a multi-scale resolution of 1 / 2, 1 / 4 and 1 / 8 respectively.

[0009] Feature extraction is performed on multi-scale images using a feature extraction unit designed to enhance high-frequency response;

[0010] The features are modeled using a feature modeling unit with an information transmission constraint mechanism to perform contextual association modeling.

[0011] Upsampling and skip connections are performed on features at each scale, and the features of the repaired image at each scale are fused through the feature fusion module;

[0012] The model is trained and inferred using a composite loss function to generate high-resolution restored image output.

[0013] Optionally, a weather-affected blurred image is acquired as input, and an image pyramid is constructed through interpolation downsampling and spatial interaction operations. The original image is downsampled to multi-scale resolutions of 1 / 2, 1 / 4, and 1 / 8, respectively, specifically including:

[0014] The overall network architecture of this invention is based on an encoder-decoder structure and introduces skip connections to achieve feature transfer between the encoder and decoder, thereby preserving shallow texture information while restoring image details. Unlike traditional single encoder structures, this invention constructs a dual encoder structure, which includes a feature extraction network based on spatial interaction and a feature modeling network with an information transfer constraint mechanism. The two are fused through residual connections to fully extract discriminative features in weather-affected images.

[0015] Optionally, during downsampling, a spatial interaction network is used to enhance the interaction capability of feature information while performing fine modeling of the spatial structure information in the image, specifically:

[0016] A spatial interaction network is introduced during the downsampling stage to enhance the interaction capabilities of feature information while performing fine-grained modeling of spatial structural information in the image. This spatial interaction network introduces an attention mechanism in two spatial dimensions to characterize pixel-level spatial relationships, rather than being limited to relationships between feature blocks or feature units, thereby improving the adaptability of feature modeling to image inpainting tasks.

[0017] For the In each downsampling stage, the feature modeling network outputs features... Reshaped into characteristics .in, , , The feature s is input into a dilated convolutional layer for processing. This layer has a 3×3 kernel size and a dilation rate of 2, which expands the receptive field without significantly increasing computational cost, thus reconstructing the structural information of the feature map. To further reduce computational overhead, the number of channels in the convolutional output is compressed to half the original number of channels. h, w, and c represent the height, width, and number of channels of the output feature, while H, W, and C represent the height, width, and number of channels of the input feature.

[0018] Subsequently, global average pooling is performed on the processed feature maps in both spatial directions to obtain their statistical information. The aggregation calculation methods in the horizontal and vertical directions are as follows:

[0019] ;

[0020] ;

[0021] in, , and These represent the indices for the vertical direction, horizontal direction, and channel, respectively. , , .feature ,in This represents the dilated convolution process, which includes batch normalization and the GELU activation function. The aggregation tensors along the horizontal and vertical directions are obtained from the above calculations:

[0022]

[0023]

[0024] in, and This is used to model the importance of pixel locations in the feature map. By performing element-wise operations on both, an attention map related to spatial location is generated.

[0025]

[0026] The attention map Features of feature modeling networks with information transmission constraint mechanisms The feature maps of the spatial interaction network are fused together to obtain the output feature maps. To match the feature dimensions, a convolutional processing unit is needed. The channel dimension is increased. The final output feature is represented as:

[0027]

[0028] in, Represents matrix multiplication. This indicates element-wise addition; This represents a 1×1 convolutional processing unit that includes batch normalization and the GELU activation function.

[0029] Optionally, feature extraction is performed on the multi-scale image using a feature extraction unit to enhance the high-frequency response, specifically including:

[0030] This feature extraction unit explicitly extracts and enhances the high-frequency components of the input features by performing a frequency domain transformation. For the input features:

[0031]

[0032] Where H, W, and C are the height, width, and number of channels of the input feature;

[0033] First, a two-dimensional real-number frequency domain transformation operation is performed on it to map the spatial domain features to the frequency domain representation, resulting in:

[0034]

[0035] The frequency domain transformation is used to separate different frequency components in image features, enabling targeted modeling of high-frequency information. Subsequently, the real and imaginary parts of the frequency domain representation are concatenated along the channel dimension to form a joint frequency domain feature.

[0036]

[0037] The joint frequency domain feature y is sequentially input into multiple feature transformation units for processing. In this embodiment, the feature transformation unit includes at least two 1×1 convolutional layers and a nonlinear activation function, used to perform channel recombination and nonlinear mapping on the frequency domain features, thereby enhancing the expressive power of high-frequency responses. After completing the frequency domain feature enhancement, the processed features are mapped back to the spatial domain through a two-dimensional real inverse frequency domain transformation to obtain the enhanced spatial feature representation.

[0038]

[0039] Furthermore, a residual connection structure is introduced in the feature extraction unit for enhancing high-frequency response to fuse the input features with the frequency-domain enhanced features, thereby improving the transfer stability of features across different scales and preventing the high-frequency enhancement process from damaging the original structural information.

[0040] Optionally, the features are modeled using a feature modeling unit with an information transmission constraint mechanism, specifically including:

[0041] This invention focuses on the problem of estimating counterfactual outcomes of time series data. It provides a... Each individual, for each individual In time step The following sequence of variables was observed: covariates Intervention ,result Within the potential outcome framework, for each time step and intervention values There is a potential outcome for all of them. However, the observational data only show factual results consistent with the actual interventions applied:

[0042]

[0043] The outcome corresponding to no intervention is a counterfactual outcome, which cannot be directly observed. The goal of this invention is to estimate the potential outcomes under different intervention strategies, given historical observations. Historical observations specifically include:

[0044]

[0045] To model long-term dependencies in time series data, this embodiment proposes using a Transformer architecture to encode historical information. Unlike a regular Transformer, the Transformer with information propagation constraints introduces causal constraints into its attention mechanism, ensuring that the model's predictions at any time step depend only on historical information prior to the current moment. At each time step... The covariates, interventions, and historical outcomes are concatenated into a single input vector:

[0046]

[0047] in, This represents a vector concatenation operation. These represent the covariates, intervention, and historical outcome at time step t, respectively; this input format enables the model to simultaneously perceive the current state as well as the intervention and outcome information from the previous time step.

[0048] Therefore, the entire time series can be represented as As input to the Transformer, the time series can be specifically represented as:

[0049]

[0050] Where T is the length of the time series;

[0051] The self-attention mechanism in a standard Transformer allows access between arbitrary time steps, which can lead to the leakage of future information in counterfactual inference tasks. Therefore, this embodiment introduces a causal mask in the attention computation. Specifically, in the computation of the... When assigning attention weights at each time step, the model only allows access to time steps. arrive The input is restricted, while access to any future time steps is prohibited. Formally, self-attention is calculated as follows:

[0052]

[0053] in, These represent the query, key, and value of attention, respectively. It is the dimension of the key. For causal mask matrix, The elements in are:

[0054]

[0055] This design ensures that the model meets the temporal causality constraint, meaning that predictions are based solely on past information.

[0056] After Transformer encoding with multiple layers of information transmission constraint mechanisms, the model at each time step Output a hidden representation Based on this representation, the model predicts potential outcomes under different intervention conditions:

[0057]

[0058] in, For the result prediction head, During training, only predictions consistent with the actual intervention are considered. The supervised loss is calculated, while the predictions from the other branch are used to construct counterfactual constraints.

[0059] Optionally, upsampling operations and skip connections are performed on features at each scale, specifically including:

[0060] For the multi-scale features obtained in the downsampling stage, upsampling operations are sequentially performed on the low-resolution scale features to gradually restore their spatial resolution and align them to the target scale. The upsampling operation is used to map low-scale features to a high-scale space, so that features from different scales maintain consistency in spatial size, thereby providing a foundation for subsequent multi-scale feature fusion.

[0061] While performing upsampling, a skip connection mechanism is introduced to fuse the features corresponding to the current scale in the encoding stage with the upsampled features. This skip connection mechanism allows for the recovery of high-level semantic information while incorporating texture and edge structure information from shallow features, effectively avoiding the loss of detail information caused by multiple downsampling and upsampling operations.

[0062] Through the synergistic effect of the above upsampling operation and skip connections, the network can simultaneously take into account global semantic information and local structural information during the process of gradually restoring image resolution, providing a more complete and stable feature representation for subsequent multi-scale feature fusion and the generation of the final image restoration result.

[0063] Optionally, the features of the restored image at each scale can be fused using a feature fusion module, specifically including:

[0064] A multi-scale feature fusion module is proposed to adaptively aggregate information flows between different scales, thereby improving the effectiveness and stability of feature fusion. Taking the feature fusion process of a half-scale inpainted image as an example, the computation process of the multi-scale feature fusion module for input features from three different scales can be represented as follows:

[0065]

[0066] in, express convolution, Feature map representing the 1 / 8 scale restoration map. Feature map representing the 1 / 4 scale restoration map. A feature map representing a 1 / 2 scale repair map. The feature fusion function is specifically represented as follows:

[0067]

[0068] in express convolution, This indicates element-wise multiplication. , This represents two inputs that need to be merged.

[0069] Through the aforementioned multi-scale feature fusion module, collaborative modeling between features of different scales can be achieved without destroying the structural information of the image. This allows the fused features to simultaneously possess multi-scale contextual information and detail expression capabilities, thereby effectively improving the overall restoration quality of weather-affected images.

[0070] Optionally, the model can be trained and inferred using a composite loss function to generate high-resolution restored image output, specifically including:

[0071] A composite loss function was constructed and used as the optimization objective during the model training phase. After the model training is complete, the trained model parameters are used to infer the image to be restored, and the corresponding high-resolution restored image is output.

[0072] The composite loss function consists of three multi-scale loss functions: multi-scale Charbonnier loss, multi-scale edge loss, and multi-scale frequency domain reconstruction loss, in order to achieve multi-angle constraints on the image restoration results.

[0073] Multi-scale Charbonnier loss (MSC) is used to constrain the overall consistency between the reconstructed image and the real image at the pixel level. Its expression is:

[0074]

[0075] in, and They represent the first Reconstructed images at various scales and their corresponding real images. This is a penalty coefficient used to enhance numerical stability; in this embodiment, it is set to... By introducing this loss term at multiple scales, the model's reconstruction accuracy for image content at different resolutions can be effectively improved.

[0076] Multi-scale edge loss (MSED) is used to emphasize structural information and edge features in an image, and its expression is:

[0077]

[0078] in, This represents the Laplacian operator, used to extract edge and structural information from an image. By constraining edge features in a multi-scale space, it helps improve the clarity of the target contour and structural regions in the restored image, avoiding over-smoothing.

[0079] Multi-scale frequency domain reconstruction loss (MSFR) is used to constrain the consistency between the reconstructed image and the real image in the frequency domain space. Its expression is:

[0080]

[0081] in, This represents the Fourier transform, used to map an image from the spatial domain to the frequency domain. By introducing a frequency domain reconstruction loss, the recovery of high-frequency and low-frequency components can be effectively constrained, improving the reconstruction quality of texture details and periodic structures.

[0082] Combining the three loss functions mentioned above, a total composite loss function is constructed, the expression of which is:

[0083]

[0084] in, , and These represent the weighting coefficients of each loss term, which are taken as values ​​in this embodiment. , , .

[0085] By weighting and combining different loss terms, the model can achieve a balance between pixel accuracy, structure preservation, and frequency domain consistency during training, thereby generating high-resolution restored images with high visual quality and structural integrity.

[0086] Furthermore, to achieve the above objectives, the present invention also provides an image restoration device affected by weather conditions, the image restoration device affected by weather conditions comprising:

[0087] The scale image sampling module is used to acquire blurred images affected by weather as input. It constructs an image pyramid through interpolation downsampling and spatial interaction operations, downsampling the original image to multi-scale resolutions of 1 / 2, 1 / 4 and 1 / 8 respectively.

[0088] The multi-scale feature extraction module extracts features from multi-scale images through a feature extraction unit designed to enhance high-frequency response;

[0089] The context association modeling module is used to perform context association modeling on the features through a feature modeling unit with an information transmission constraint mechanism;

[0090] The feature fusion module is used to perform upsampling operations and skip connections on features at various scales, and to fuse the features of the repaired image at various scales through the feature fusion module;

[0091] The image inpainting module is used to train and infer the model using a composite loss function, generating high-resolution inpainted image outputs.

[0092] In addition, to achieve the above objectives, the present invention also provides an image restoration device affected by weather, the device comprising: a memory, a processor, and an image restoration program affected by weather stored in the memory and executable on the processor, wherein the image restoration program affected by weather implements the steps of the above-described image restoration method affected by weather when executed by the processor.

[0093] In addition, to achieve the above objectives, the present invention also provides a storage medium storing a weather-affected image restoration program, which, when executed by a processor, performs the above-described weather-affected image restoration steps.

[0094] The beneficial effects of this invention are as follows: The embodiments of this invention propose an image restoration method, apparatus, device, and storage medium for images affected by weather. The method includes multi-scale downsampling processing of weather-affected images to fully extract degradation features at different scales, and combining upsampling and skip connection mechanisms to obtain multi-scale restoration results. Furthermore, an image restoration model is constructed through feature fusion and regression modeling. This method effectively improves the ability to restore image details and preserve structure under adverse weather conditions, solving the problems of insufficient geographic information acquisition and low imaging clarity in existing technologies under complex weather environments, and has high practical value. Attached Figure Description

[0095] Figure 1 This is a schematic flowchart illustrating an embodiment of the image restoration method for weather-affected images according to the present invention.

[0096] Figure 2 This is an architectural diagram of the image restoration method affected by weather according to an embodiment of the present invention.

[0097] Figure 3 This is a schematic diagram of a spatial interactive network according to an embodiment of the present invention.

[0098] Figure 4 This is a schematic diagram of the feature extraction module for enhancing high-frequency response according to an embodiment of the present invention.

[0099] Figure 5 This is a schematic diagram of multi-scale feature fusion according to an embodiment of the present invention.

[0100] Figure 6 This is an example diagram illustrating the effect of restoring weather-affected images according to an embodiment of the present invention.

[0101] Figure 7 This is a structural block diagram of the image restoration device affected by weather in an embodiment of the present invention.

[0102] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0103] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0104] Currently, adverse visual environments have become a major obstacle in artificial intelligence, especially in computer vision applications. Among these, severe weather and environmental disasters have sparked extensive research in the field of low-level vision in recent years. Most existing methods for restoring weather-affected images still rely on a single-input, single-output architecture, which not only ignores the potential explicit information in multi-scale images but also limits the model's ability to mine complementary implicit information across different scales.

[0105] To address this problem, various embodiments of the image restoration method of this invention are proposed. The image restoration method provided by this invention performs multi-scale downsampling processing on weather-affected images to fully extract degradation features at different scales. It then combines upsampling and skip connection mechanisms to obtain multi-scale restoration results, and further constructs an image restoration model through feature fusion and regression modeling. This method effectively improves the ability to restore image details and preserve structure under adverse weather conditions, solving the problems of insufficient geographic information acquisition and low imaging clarity in existing technologies under complex weather environments, and has high practical value.

[0106] This invention provides a method for image restoration affected by weather conditions, referring to... Figure 1 , Figure 1 This is a schematic flowchart illustrating an embodiment of the image restoration method for weather-affected images according to the present invention.

[0107] The process involves acquiring a weather-affected blurred image as input, constructing an image pyramid through interpolation downsampling and spatial interaction operations, and downsampling the original image to multi-scale resolutions of 1 / 2, 1 / 4, and 1 / 8 respectively. Feature extraction is performed on the multi-scale image using a feature extraction unit to enhance high-frequency response. Contextual association modeling is then performed on the features using a feature modeling unit with information transmission constraint mechanisms. Upsampling operations and skip connections are applied to the features at each scale, and the restored image features at each scale are fused using a feature fusion module. Finally, the model is trained and inferred using a composite loss function to generate a high-resolution restored image output.

[0108] In this embodiment, the image restoration method affected by weather includes the following steps:

[0109] Step S100: Obtain a blurred image affected by weather as input, construct an image pyramid through interpolation downsampling and spatial interaction operations, and downsample the original image to multi-scale resolutions of 1 / 2, 1 / 4 and 1 / 8 respectively.

[0110] Specifically, such as Figure 2 As shown, the overall network architecture of this invention is based on an encoder-decoder structure design and introduces skip connections to achieve feature transfer between the encoder and decoder, thereby preserving shallow texture information while restoring image details. Unlike the traditional single encoder structure, this embodiment constructs a dual encoder structure, which includes a feature extraction network based on spatial interaction and a feature modeling network with an information transfer constraint mechanism. The two are fused through residual connections to fully mine discriminative features in weather-affected images.

[0111] In the dual-encoder structure, the feature modeling network with information transmission constraint mechanism effectively reduces computational resource consumption while ensuring modeling capability by establishing correlations between feature units within a local scope. However, relying solely on feature modeling within a local scope weakens the ability to characterize global contextual information to some extent. Meanwhile, in weather-affected images, due to occlusion factors such as rain, fog, and snow, target boundaries are often blurred or broken, making it difficult to accurately recover spatial structural information using only local feature modeling.

[0112] Based on the above problem analysis, such as Figure 3 As shown, this invention introduces a spatial interaction network in the downsampling stage to enhance the interaction capability of feature information while performing fine modeling of spatial structural information in the image. This spatial interaction network introduces an attention mechanism in two spatial dimensions to characterize pixel-level spatial relationships, rather than being limited to the relationships between feature blocks or feature units, thereby improving the adaptability of feature modeling to image restoration tasks.

[0113] For the In each downsampling stage, the feature modeling network outputs features... Reshaped into .in, , , The feature s is input into a dilated convolutional layer for processing. This layer has a 3×3 kernel size and a dilation rate of 2, which expands the receptive field without significantly increasing computational cost, thus reconstructing the structural information of the feature map. To further reduce computational overhead, the number of channels in the convolutional output is compressed to half the original number of channels. h, w, and c represent the height, width, and number of channels of the output feature, while H, W, and C represent the height, width, and number of channels of the input feature.

[0114] Subsequently, global average pooling is performed on the processed feature maps in both spatial directions to obtain their statistical information. The aggregation calculation methods in the horizontal and vertical directions are as follows:

[0115] ;

[0116] ;

[0117] in, , and These represent the indices for the vertical direction, horizontal direction, and channel, respectively. , , .feature ,in This represents the dilated convolution process, which includes batch normalization and the GELU activation function. The aggregation tensors along the horizontal and vertical directions are obtained from the above calculations:

[0118]

[0119]

[0120] in, and This is used to model the importance of pixel locations in the feature map. By performing element-wise operations on both, an attention map related to spatial location is generated.

[0121]

[0122] The attention map Features of feature modeling networks with information transmission constraint mechanisms The feature maps of the spatial interaction network are fused together to obtain the output feature maps. To match the feature dimensions, a convolutional processing unit is needed. The channel dimension is increased. The final output feature is represented as:

[0123]

[0124] in, Represents matrix multiplication. This indicates element-wise addition; This represents a 1×1 convolutional processing unit that includes batch normalization and the GELU activation function.

[0125] Step S200 involves extracting features from the multi-scale image using a feature extraction unit designed to enhance high-frequency response. Specifically:

[0126] like Figure 4As shown, the feature extraction unit for enhancing high-frequency response is used to improve the network's ability to model local structures and detailed information in the frequency domain, so as to improve the restoration effect of background textures and edge information in weather-affected images. In this embodiment, the feature extraction unit explicitly extracts and enhances the high-frequency components by performing a frequency-domain transformation on the input features. For the input features:

[0127]

[0128] where H, W, and C are the height, width, and number of channels of the input features;

[0129] First, perform a two-dimensional real-valued frequency-domain transformation operation on it to map the spatial-domain features to the frequency-domain representation, obtaining:

[0130]

[0131] The frequency-domain transformation is used to separate different frequency components in the image features for targeted modeling of high-frequency information. Subsequently, the real and imaginary parts in the frequency-domain representation are concatenated in the channel dimension to form a joint frequency-domain feature:

[0132]

[0133] The joint frequency-domain feature y is sequentially input into multiple feature transformation units for processing. In this embodiment, the feature transformation unit includes at least two 1×1 convolutional layers and a non-linear activation function, which are used to perform channel reorganization and non-linear mapping on the frequency-domain features, thereby enhancing the expression ability for high-frequency response. After completing the enhancement of the frequency-domain features, the processed features are mapped back to the spatial domain through a two-dimensional real-valued inverse frequency-domain transformation to obtain an enhanced spatial feature representation:

[0134]

[0135] In addition, a residual connection structure is introduced in the feature extraction unit for enhancing high-frequency response to fuse the input features and the frequency-domain enhanced features, so as to improve the transfer stability of features between different scales and prevent the high-frequency enhancement process from damaging the original structural information.

[0136] Step S300, perform context association modeling on the features through a feature modeling unit with an information transfer constraint mechanism, specifically including:

[0137] In this embodiment, the feature modeling unit with an information transfer constraint mechanism is used to model the sequential relationship of multi-scale features in the feature dimension or the spatial dimension, so as to enhance the context correlation between features and suppress the interference of irrelevant or redundant information on the restoration result. This embodiment focuses on the problem of estimating counterfactual results in time series. Suppose there is Each individual, for each individual In time step The following sequence of variables was observed: covariates Intervention ,result Within the potential outcome framework, for each time step and intervention values There is a potential outcome for all of them. However, the observational data only show factual results consistent with the actual interventions applied:

[0138]

[0139] The outcome corresponding to no intervention is a counterfactual outcome, which cannot be directly observed. The goal of this embodiment is to estimate the potential outcomes under different intervention strategies, given historical observations. Historical observations specifically include:

[0140]

[0141] To model long-term dependencies in time series data, this embodiment proposes using a Transformer architecture to encode historical information. Unlike a regular Transformer, the Transformer with information propagation constraints introduces causal constraints into its attention mechanism, ensuring that the model's predictions at any time step depend only on historical information prior to the current moment. At each time step... The covariates, interventions, and historical outcomes are concatenated into a single input vector:

[0142]

[0143] in, This represents a vector concatenation operation. These represent the covariates, intervention, and historical outcome at time step t, respectively; this input format enables the model to simultaneously perceive the current state as well as the intervention and outcome information from the previous time step.

[0144] Therefore, the entire time series can be represented as As input to the Transformer, the time series can be specifically represented as:

[0145]

[0146] Where T is the length of the time series;

[0147] The self-attention mechanism in a standard Transformer allows access between arbitrary time steps, which can lead to the leakage of future information in counterfactual inference tasks. Therefore, this embodiment introduces a causal mask in the attention computation. Specifically, in the computation of the... When assigning attention weights at each time step, the model only allows access to time steps. arrive The input is restricted, while access to any future time steps is prohibited. Formally, self-attention is calculated as follows:

[0148]

[0149] in, These represent the query, key, and value of attention, respectively. It is the dimension of the key. For causal mask matrix, The elements in are:

[0150]

[0151] This design ensures that the model meets the temporal causality constraint, meaning that predictions are based solely on past information.

[0152] After Transformer encoding with multiple layers of information transmission constraint mechanisms, the model at each time step Output a hidden representation Based on this representation, the model predicts potential outcomes under different intervention conditions:

[0153]

[0154] in, For the result prediction head, During training, only predictions consistent with the actual intervention are considered. The supervised loss is calculated, while the predictions from the other branch are used to construct counterfactual constraints.

[0155] Step S400: Perform upsampling and skip connections on features at each scale, and fuse the restored image features at each scale through the feature fusion module.

[0156] Specifically, in this embodiment, for the multi-scale features obtained in the downsampling stage, upsampling operations are sequentially performed on the low-resolution scale features to gradually restore their spatial resolution and align them to the target scale. The upsampling operation is used to map low-scale features to a high-scale space, so that features from different scales maintain consistency in spatial size, thereby providing a foundation for subsequent multi-scale feature fusion.

[0157] While performing upsampling, a skip connection mechanism is introduced to fuse the features corresponding to the current scale in the encoding stage with the upsampled features. This skip connection mechanism allows for the recovery of high-level semantic information while incorporating texture and edge structure information from shallow features, effectively avoiding the loss of detail information caused by multiple downsampling and upsampling operations.

[0158] Through the synergistic effect of the above upsampling operation and skip connections, the network can simultaneously take into account global semantic information and local structural information during the process of gradually restoring image resolution, providing a more complete and stable feature representation for subsequent multi-scale feature fusion and the generation of the final image restoration result.

[0159] After upsampling and cross-layer feature transfer of features at each scale, the feature fusion module fuses the features of the repaired image at each scale.

[0160] Specifically, such as Figure 5 As shown, in the process of multi-scale feature fusion, simply aligning features at different scales through upsampling or downsampling operations can easily introduce problems such as structural information distortion or semantic inconsistency. Therefore, this embodiment proposes a multi-scale feature fusion module to adaptively aggregate information flows between different scales, thereby improving the effectiveness and stability of feature fusion. Taking the feature fusion process of a 1 / 2 scale restored image as an example, the calculation process of the multi-scale feature fusion module for input features from three different scales can be expressed as follows:

[0161]

[0162] in express convolution, Feature map representing the 1 / 8 scale restoration map. Feature map representing the 1 / 4 scale restoration map. A feature map representing a 1 / 2 scale repair map. The feature fusion function is specifically represented as follows:

[0163]

[0164] in express convolution, This indicates element-wise multiplication. , This represents two inputs that need to be merged.

[0165] Through the aforementioned multi-scale feature fusion module, collaborative modeling between features of different scales can be achieved without destroying the structural information of the image. This allows the fused features to simultaneously possess multi-scale contextual information and detail expression capabilities, thereby effectively improving the overall restoration quality of weather-affected images.

[0166] Step S500: The model is trained and inferred using a composite loss function to generate a high-resolution repaired image output.

[0167] Specifically, to effectively supervise the model's learning process and guide it to simultaneously focus on pixel consistency, structural information, and frequency domain characteristics of the image at different scales, this embodiment constructs a composite loss function and uses it as the optimization objective during the model training phase. After the model completes training, the trained model parameters are used to infer the image to be restored, outputting the corresponding high-resolution restored image.

[0168] In this embodiment, the composite loss function is composed of three multi-scale loss functions: multi-scale Charbonnier loss, multi-scale edge loss, and multi-scale frequency domain reconstruction loss, so as to achieve multi-angle constraints on the image restoration result.

[0169] Multi-scale Charbonnier loss (MSC) is used to constrain the overall consistency between the reconstructed image and the real image at the pixel level. Its expression is:

[0170]

[0171] in, and They represent the first Reconstructed images at various scales and their corresponding real images. This is a penalty coefficient used to enhance numerical stability; in this embodiment, it is set to... By introducing this loss term at multiple scales, the model's reconstruction accuracy for image content at different resolutions can be effectively improved.

[0172] Multi-scale edge loss (MSED) is used to emphasize structural information and edge features in an image, and its expression is:

[0173]

[0174] in, This represents the Laplacian operator, used to extract edge and structural information from an image. By constraining edge features in a multi-scale space, it helps improve the clarity of the target contour and structural regions in the restored image, avoiding over-smoothing.

[0175] Multi-scale frequency domain reconstruction loss (MSFR) is used to constrain the consistency between the reconstructed image and the real image in the frequency domain space. Its expression is:

[0176]

[0177] in, This represents the Fourier transform, used to map an image from the spatial domain to the frequency domain. By introducing a frequency domain reconstruction loss, the recovery of high-frequency and low-frequency components can be effectively constrained, improving the reconstruction quality of texture details and periodic structures.

[0178] Combining the three loss functions mentioned above, a total composite loss function is constructed, the expression of which is:

[0179]

[0180] in, , and These represent the weighting coefficients of each loss term, which are taken as values ​​in this embodiment. , , .

[0181] By weighting and combining different loss terms, the model can achieve a balance between pixel accuracy, structure preservation, and frequency domain consistency during training, thereby generating high-resolution restored images with high visual quality and structural integrity.

[0182] This invention achieves full extraction of degradation features at different scales by performing multi-scale downsampling processing on weather-affected images, and obtains multi-scale restoration results by combining upsampling and skip connection mechanisms. Furthermore, it constructs an image restoration model through feature fusion and regression modeling. Figure 6 As shown, this method effectively improves the ability to restore image details and preserve structure under adverse weather conditions, and solves the problems of insufficient geographic information acquisition capability and low imaging clarity in existing technologies under complex weather conditions, thus having high practical value.

[0183] To more clearly demonstrate the improved image restoration capabilities of this application under weather conditions, comparative experiments are conducted with other methods under the same objective environment to verify the recognition effect of the method designed in this application.

[0184] The evaluation metrics used in this application are peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM).

[0185] Peak Signal-to-Noise Ratio (PSNR) is an objective image quality assessment metric based on pixel error, used to measure the overall similarity between a reconstructed image and a reference image. This metric is calculated by transcribing the mean squared error between the two images and converting it to a logarithmic form. A higher PSNR value indicates a smaller difference between the reconstructed and original images, and thus higher image reconstruction quality. PSNR can be specifically expressed as:

[0186]

[0187] in, This represents the maximum possible value of an image pixel (e.g., for an 8-bit image). ).

[0188] Structural similarity index (SSIM) is an image quality assessment metric based on human visual perception characteristics. It measures the structural similarity between a reconstructed image and a reference image by simultaneously considering brightness, contrast, and structural information. Unlike PSNR, SSIM focuses more on the image's ability to preserve structure, resulting in higher consistency in evaluating visual quality. The structural similarity index can be specifically expressed as:

[0189]

[0190] in: , These represent the mean values ​​of the reference image and the reconstructed image, respectively. , These represent the variances of the corresponding images; This represents the covariance between two images; and It is a stability constant used to avoid the denominator being zero.

[0191] This application compares the proposed method with various image inpainting algorithms on two datasets. The algorithms compared include DSC, RCDNet, SPDNet, Restormer, IDT, and DRSformer. These methods are existing techniques with strong image inpainting capabilities; using them as comparison methods provides a more objective demonstration of the capabilities and effectiveness of this application.

[0192] Table 1 shows the peak signal-to-noise ratio (PSNR) of the restored images generated by different methods on different datasets.

[0193]

[0194] Table 2 shows the structural similarity of the restored images generated by different methods on different datasets.

[0195]

[0196] This embodiment compares various image restoration methods affected by climate based on the UAV-Rain1k and R200H datasets. These datasets cover a variety of typical severe weather scenarios and can be used to verify the applicability and stability of different image restoration methods under complex climatic conditions. Experimental results, as shown in Table 1, demonstrate that the method of this invention outperforms the comparative methods in the objective evaluation dimensions related to pixel-level reconstruction accuracy on both the UAV-Rain1k and R200H datasets. As shown in Table 2, the method of this invention also achieves high numerical results on both datasets in the evaluation metrics related to image structure consistency. These results indicate that the method of this invention has good restoration effects in terms of both pixel reconstruction accuracy and structure preservation. Therefore, the method of this invention can effectively improve the clarity and structural integrity of images under severe weather conditions, enhance the usability of climate-affected images, and thus alleviate the problem of limited geographic information acquisition quality under complex weather conditions to a certain extent, possessing practical application value.

[0197] Reference Figure 7 , Figure 7 This is a structural block diagram of an embodiment of the image restoration device affected by weather according to the present invention.

[0198] like Figure 7 As shown, the image restoration device affected by weather proposed in this embodiment of the invention includes:

[0199] The scale image sampling module S10 is used to acquire a blurred image affected by weather as input, and construct an image pyramid through interpolation downsampling and spatial interaction operations to downsample the original image to multi-scale resolutions of 1 / 2, 1 / 4 and 1 / 8 respectively;

[0200] The multi-scale feature extraction module S20 extracts features from the multi-scale image through a feature extraction unit for enhancing high-frequency response;

[0201] The context association modeling module S30 is used to perform context association modeling on the features through a feature modeling unit with an information transmission constraint mechanism;

[0202] The feature fusion module S40 is used to perform upsampling operations and skip connections on features at each scale, and to fuse the features of the repaired image at each scale through the feature fusion module;

[0203] The image inpainting module S50 is used to train and infer the model using a composite loss function to generate high-resolution inpainted image output.

[0204] Other embodiments or specific implementations of the image restoration device affected by weather of the present invention can be found in the above-described method embodiments, and will not be repeated here.

[0205] Furthermore, this invention also proposes a computer-readable storage medium storing a weather-affected image restoration program. When the image restoration program is executed by a processor, it implements the various steps of the weather-affected image restoration method described above. Given that this storage medium embodiment is consistent with the foregoing method embodiment in terms of technical solution and technical effect, its specific steps will not be described again.

[0206] Similarly, the beneficial effects achievable using the same technical solution will not be elaborated further here. For technical details not explicitly disclosed in the computer-readable storage medium embodiments involved in this application, please refer to the relevant descriptions of the foregoing method embodiments for understanding.

[0207] In a specific implementation, the program instructions can be deployed to execute on a single computing device, or they can be deployed to execute on multiple computing devices located in the same or different locations. The multiple computing devices can be interconnected through a communication network to achieve centralized or distributed image restoration processing.

[0208] Those skilled in the art will understand that the method flows described in the above embodiments can be implemented by a computer program and executed by program instructions in conjunction with relevant hardware resources. The program can be stored in a computer-readable storage medium, such as a disk, optical disk, read-only memory (ROM), random access memory (RAM), or other readable storage media. When the program is loaded and executed, the various steps described in the foregoing method embodiments can be completed.

[0209] It should be noted that the above device embodiments are merely illustrative structural descriptions. The functional modules described in this specification are not necessarily physically separate units and can be implemented independently or integratedly according to actual application requirements; the functional modules can be deployed in the same device or distributed across multiple network devices. The connection relationships between modules indicate their communication capabilities and can be implemented through communication buses, signal lines, or other communication methods. Those skilled in the art can understand and implement the above structure based on the content of this specification without any inventive effort.

[0210] As can be seen from the above description, the technical solution of this invention can be implemented either through software combined with a general-purpose hardware platform or through dedicated hardware, such as application-specific integrated circuits (ASICs), dedicated processors, dedicated memory, or other custom circuit structures. Generally speaking, functions implemented by programs can also be accomplished using various hardware structures, including analog circuits, digital circuits, or dedicated logic circuits. However, in most application scenarios, software implementation offers greater flexibility and scalability.

[0211] Based on this, the technical solution of the present invention can also be provided in the form of a software product. The software product can be stored in a computer-readable storage medium, such as a floppy disk, USB flash drive, portable hard drive, ROM, RAM, disk, or optical disk, and includes program instructions for causing a computer device (such as a personal computer, server, or network device) to execute the weather-affected image restoration method described in the embodiments of the present invention.

Claims

1. A method for image restoration affected by weather, characterized in that, The method includes the following steps: The image is taken as input by a blurred image affected by weather. An image pyramid is constructed by interpolation downsampling and spatial interaction operations. The original image is downsampled to a multi-scale resolution of 1 / 2, 1 / 4 and 1 / 8 respectively. Feature extraction is performed on multi-scale images using a feature extraction unit designed to enhance high-frequency response; The features are modeled using a feature modeling unit with an information transmission constraint mechanism to perform contextual association modeling. Upsampling and skip connections are performed on features at each scale, and the features of the repaired image at each scale are fused through the feature fusion module; The model is trained and inferred using a composite loss function to generate high-resolution restored image output. Contextual association modeling of the features is performed using a feature modeling unit with an information transmission constraint mechanism, specifically including: Focusing on the problem of estimating counterfactual results from time series data; assuming... Each individual, for each individual In time step The following sequence of variables was observed: covariates Intervention ,result Within the potential outcome framework, for each time step and intervention values There is a potential outcome for all of them. : ； Given historical observations, estimate the potential outcomes under different intervention strategies. Historical observations specifically include: ； The Transformer architecture is used to encode historical information; the Transformer, with its information transmission constraints, introduces causal constraints into the attention mechanism; at each time step... The covariates, interventions, and historical outcomes are concatenated into a single input vector: ； in, This represents a vector concatenation operation. These represent the covariates, intervention, and historical outcomes at time step t, respectively. The entire time series representation As input to the Transformer, the time series is specifically represented as follows: ； Where T is the length of the time series; Introducing causal masks into attention computation; specifically, in the computation of the first... When assigning attention weights at each time step, the model only allows access to time steps. arrive The input is restricted, while access to any future time steps is prohibited. Formally, self-attention is calculated as follows: ； in, These represent the query, key, and value of attention, respectively. It is the dimension of the key. For causal mask matrix, The elements in are: ； After Transformer encoding with multiple layers of information transmission constraint mechanisms, the model at each time step Output a hidden representation Based on this representation, the model predicts potential outcomes under different intervention conditions: ； in, For the result prediction head, During training, only predictions consistent with actual interventions are considered. The supervised loss is calculated, while the predictions from the other branch are used to construct counterfactual constraints.

2. The image restoration method affected by weather as described in claim 1, characterized in that, The process involves acquiring a weather-affected blurred image as input, constructing an image pyramid through interpolation downsampling and spatial interaction operations, and downsampling the original image to multi-scale resolutions of 1 / 2, 1 / 4, and 1 / 8, specifically including: For the In each downsampling stage, the feature modeling network outputs features... Reshaped into characteristics ;in, , , The feature s is input into a dilated convolutional layer for processing. The kernel size of the dilated convolutional layer is 3×3, the dilation rate is 2, and the number of channels of the convolution output is compressed to 1 / 2 of the original number of channels. h, w, c are the height, width and number of channels of the output feature, and H, W, C are the height, width and number of channels of the input feature. Subsequently, global average pooling is performed on the processed feature maps in the spatial directions; the aggregation calculation methods in the horizontal and vertical directions are as follows: ； ； in, , and These represent the indices for the vertical direction, horizontal direction, and channel, respectively. , , ;feature ,in This represents the dilated convolution process involving batch normalization and the GELU activation function; the aggregation tensors along the horizontal and vertical directions are obtained from the above calculations: ； ； in, and This is used to model the importance of pixel locations in the feature map. By performing element-wise operations on both, an attention map related to spatial location is generated. ； attention map Features of feature modeling networks with information transmission constraint mechanisms The feature maps of the spatial interaction network are fused together to obtain the output feature maps. ; where a convolution processing unit is used to process The channel dimension is increased, and the final output feature is represented as follows: ； in, Represents matrix multiplication. This indicates element-wise addition; This represents a 1×1 convolutional processing unit that includes batch normalization and the GELU activation function.

3. The image restoration method affected by weather as described in claim 1, characterized in that, Feature extraction of multi-scale images is performed using a feature extraction unit designed to enhance high-frequency response, specifically including: This feature extraction unit explicitly extracts and enhances the high-frequency components of the input features by performing a frequency domain transformation; for the input features: ； Where H, W, and C are the height, width, and number of channels of the input feature; First, a two-dimensional real-number frequency domain transformation operation is performed on it to map the spatial domain features to the frequency domain representation, resulting in: ； Subsequently, the real and imaginary parts of the frequency domain representation are concatenated along the channel dimension to form a joint frequency domain feature: ； The joint frequency domain feature y is sequentially input into multiple feature transformation units for processing. Each feature transformation unit includes at least two 1×1 convolutional layers and a nonlinear activation function, used to perform channel recombination and nonlinear mapping on the frequency domain feature. After frequency domain feature enhancement, the processed feature is mapped back to the spatial domain through a two-dimensional real inverse frequency domain transformation to obtain the enhanced spatial feature representation. ； Furthermore, a residual connection structure is introduced into the feature extraction unit for enhancing high-frequency response to fuse the input features with the frequency-domain enhanced features.

4. The image restoration method affected by weather as described in claim 1, characterized in that, The upsampling operation and skip connections performed on features at each scale specifically include: For the multi-scale features obtained in the downsampling stage, upsampling operations are performed on the low-resolution scale features in sequence, and then aligned to the target scale. While performing the upsampling operation, a skip connection mechanism is introduced to fuse the features corresponding to the current scale in the encoding stage with the upsampled features. Through the skip connection mechanism, texture information and edge structure information contained in the shallow features are introduced while restoring high-level semantic information.

5. The image restoration method affected by weather as described in claim 1, characterized in that, The feature fusion module fuses the features of the restored image at each scale, specifically including: A multi-scale feature fusion module is proposed to adaptively aggregate information flows between different scales. For the feature fusion process of a 1 / 2 scale inpainted image, the calculation process of the multi-scale feature fusion module for input features from three different scales is represented as follows: ； in express convolution, Feature map representing the 1 / 8 scale restoration map. Feature map representing the 1 / 4 scale restoration map. Feature map representing the 1 / 2 scale repair map; The feature fusion function is specifically represented as follows: ； in express convolution, This indicates element-wise multiplication. , This represents two inputs that need to be merged.

6. The image restoration method affected by weather as described in claim 1, characterized in that, The process of training and inferring the model using a composite loss function to generate a high-resolution restored image output specifically includes: A composite loss function was constructed and used as the optimization objective during the model training phase. After the model training was completed, the trained model parameters were used to infer the image to be restored and output the corresponding high-resolution restored image. The composite loss function is composed of three multi-scale loss functions: multi-scale Charbonnier loss, multi-scale edge loss, and multi-scale frequency domain reconstruction loss. The multi-scale Charbonnier loss is used to constrain the overall consistency between the reconstructed image and the real image at the pixel level, and its expression is: ； in, and They represent the first Reconstructed images at various scales and their corresponding real images; This is a penalty coefficient used to enhance numerical stability; Multi-scale edge loss is used to emphasize structural information and edge features in an image, and its expression is: ； in, This represents the Laplacian operator, used to extract edge and structural information from an image; The multi-scale frequency domain reconstruction loss is used to constrain the consistency between the reconstructed image and the real image in the frequency domain space, and its expression is: ； in, This represents the Fourier transform, used to map an image from the spatial domain to the frequency domain; Combining the three loss functions mentioned above, a total composite loss function is constructed, the expression of which is: ； in, , and These represent the weighting coefficients of each loss term.

7. An image restoration device affected by weather, characterized in that, The weather-affected image restoration device includes: The scale image sampling module is used to acquire blurred images affected by weather as input. It constructs an image pyramid through interpolation downsampling and spatial interaction operations, downsampling the original image to multi-scale resolutions of 1 / 2, 1 / 4 and 1 / 8 respectively. The multi-scale feature extraction module extracts features from multi-scale images through a feature extraction unit designed to enhance high-frequency response; The context association modeling module is used to perform context association modeling on the features through a feature modeling unit with an information transmission constraint mechanism; The feature fusion module is used to perform upsampling operations and skip connections on features at various scales, and to fuse the features of the repaired image at various scales through the feature fusion module; The image inpainting module is used to train and infer the model using a composite loss function, generating high-resolution inpainted image outputs. The context association modeling module is used to perform context association modeling on the features through a feature modeling unit with an information transmission constraint mechanism, specifically including: Focusing on the problem of estimating counterfactual results from time series data; assuming... Each individual, for each individual In time step The following sequence of variables was observed: covariates Intervention ,result Within the potential outcome framework, for each time step and intervention values There is a potential outcome for all of them. : ； Given historical observations, estimate the potential outcomes under different intervention strategies. Historical observations specifically include: ； The Transformer architecture is used to encode historical information; the Transformer, with its information transmission constraints, introduces causal constraints into the attention mechanism; at each time step... The covariates, interventions, and historical outcomes are concatenated into a single input vector: ； in, This represents a vector concatenation operation. These represent the covariates, intervention, and historical outcomes at time step t, respectively. The entire time series representation As input to the Transformer, the time series is specifically represented as follows: ； Where T is the length of the time series; Introducing causal masks into attention computation; specifically, in the computation of the first... When assigning attention weights at each time step, the model only allows access to time steps. arrive The input is restricted, while access to any future time steps is prohibited. Formally, self-attention is calculated as follows: ； in, These represent the query, key, and value of attention, respectively. It is the dimension of the key. For causal mask matrix, The elements in are: ； After Transformer encoding with multiple layers of information transmission constraint mechanisms, the model at each time step Output a hidden representation Based on this representation, the model predicts potential outcomes under different intervention conditions: ； in, For the result prediction head, During training, only predictions consistent with actual interventions are considered. The supervised loss is calculated, while the predictions from the other branch are used to construct counterfactual constraints.

8. An image restoration device affected by weather, characterized in that, The weather-affected image restoration device includes: a memory, a processor, and an image restoration program stored in the memory and executable on the processor. When the weather-affected image restoration program is executed by the processor, it implements the steps of the weather-affected image restoration method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium stores a weather-affected image restoration program, which, when executed by a processor, implements the steps of the weather-affected image restoration method as described in any one of claims 1 to 6.

Images