Rain and fog image restoration method and system based on space-frequency cooperative perception, and electronic equipment

Abstract

The invention belongs to the technical field of rain and fog image processing, and particularly relates to a rain and fog image restoration method and system based on space-frequency collaborative awareness and electronic equipment, and the method comprises the steps: obtaining a rain and fog image, and respectively extracting the spatial domain features and frequency domain features of the rain and fog image; fusing the spatial domain features and the frequency domain features based on an adaptive channel attention mechanism to obtain dual-domain fusion features; the dual-domain fusion features with recovered resolution and the spatial domain features with the same size are added through residual connection to obtain a feature map, convolution processing is performed on the feature map to obtain a clear image, jump connection of traditional U-net is replaced with a residual layer structure in residual connection design, the gradient disappearance problem is relieved through residual connection, and the model training efficiency is improved. According to the design, cross-level fusion of shallow-layer edge information and deep-layer semantic features is achieved, and compared with original jump connection, recovery of local details (such as object contours) and removal of rain and fog textures are better facilitated.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of rain and fog image processing, and particularly relates to a rain and fog image restoration method and system based on space-frequency collaborative perception and an electronic device. BACKGROUND

[0002] Rain and fog image restoration is an important research direction in the field of computer vision, aiming to restore clear objects and background scenes from images affected by rain lines or fog. Rain lines will block image details, while fog will reduce contrast and visibility through atmospheric scattering. These phenomena that reduce image quality pose a major challenge to applications that rely on high-quality images, such as surveillance systems, autonomous driving, and outdoor photography.

[0003] Traditional rain and fog removal practices are usually based on physical models and artificial priori to design rain and fog removal schemes, but these methods are usually based on simple assumptions and require manual parameter adjustment, and often perform poorly when dealing with complex real-world scenarios. On the other hand, traditional methods have limited effectiveness in dealing with heavy rain or thick fog, which may leave residual artifacts or fail to fully restore image quality. In addition, the computational complexity of traditional methods is high, making it difficult to meet the requirements of real-time applications such as autonomous driving. In recent years, the rapid development of deep learning has provided a new technical approach to rain and fog image restoration, efficiently implementing end-to-end image restoration through data-driven methods. Existing deep learning-based methods are mostly single-task image restoration techniques, but heavy rain in real-world scenarios is often accompanied by thick fog, making joint rain and fog removal algorithms based on deep learning more suitable for real-world scenarios. Joint rain and fog removal algorithms can more efficiently restore real images and have more widespread applicability, making further research necessary.

[0004] Therefore, in the joint rain and fog removal process, how to ensure the applicability of the widened method, not leave residual artifacts, and improve the efficiency of the method become challenging problems. SUMMARY

[0005] The present application aims to provide a rain and fog image restoration method and system based on space-frequency collaborative perception and an electronic device to solve the technical problem of how to ensure the applicability of the widened method, not leave residual artifacts, and improve the efficiency of the method in the joint rain and fog removal process.

[0006] To achieve the above-mentioned purpose, the present application adopts the following technical solutions: In a first aspect, the present application provides a rain and fog image restoration method based on space-frequency collaborative perception, comprising: Obtaining a rain and fog image, and extracting spatial domain features and frequency domain features of the rain and fog image respectively; Fusing the spatial domain features and the frequency domain features based on an adaptive channel attention mechanism to obtain dual-domain fusion features; The feature map is obtained by adding the dual-domain fusion features that restore the resolution and the spatial domain features of the same size through residual connection. The feature map is then convolved to obtain a clear image.

[0007] Preferably, the acquisition of rain and fog images involves extracting spatial domain features and frequency domain features from the rain and fog images; specifically, extracting the spatial domain features of the rain and fog images includes: The image is subjected to two convolutional operations to decompose and extract the complex spatial feature patterns of the image; Max pooling is performed on the complex spatial feature patterns of the image to extract key features, remove secondary and redundant features, and output deep complex spatial feature patterns. Max pooling is performed on deep and complex spatial feature patterns to extract key features, remove secondary and redundant features, and output the spatial domain features of the image. Extracting frequency domain features from rain and fog images specifically includes: Rain and fog images are transformed from spatial domain representation to frequency domain representation through Fourier transform, that is, the pixel values ​​of the image are decomposed into frequency components to obtain the frequency domain representation image; By moving the low-frequency components of the frequency domain representation image to the center of the spectrum, a centered spectrum is obtained. Extract the central region from the centralized spectrum, and obtain the real and imaginary parts of the central region respectively; The real and imaginary parts of the central region are updated by performing complex multiplication operations on a learnable complex filter to obtain the filtered central region. The filtered central region is returned to its corresponding position in the original spectrum and integrated with the unprocessed edge region to obtain the updated spectrum; The updated spectrum is then subjected to inverse Fourier reconstruction to the spatial domain to obtain the frequency domain features.

[0008] Preferably, a dual-domain fused feature is obtained by fusing spatial domain features and frequency domain features based on an adaptive channel attention mechanism; specifically including: Channel-dimensional features are concatenated from spatial domain features and frequency domain features to obtain preliminary fused features; The initial fused features are compressed in spatial dimension by global average pooling to obtain scalar descriptive features; The scalar descriptive features are sequentially passed through the first fully connected layer for dimensionality reduction, activated by the ReLU activation function, and then increased in dimensionality by the second fully connected layer to obtain the activation features. The activation features are weighted based on the Sigmoid function to obtain the weight matrix; The weight matrix is ​​multiplied channel by channel with the preliminary fusion feature to obtain the dual-domain fusion feature.

[0009] Preferably, a feature map is obtained by adding the dual-domain fusion features of the restored resolution and the spatial domain features of the same size through residual connections, and then convolving the feature map to obtain a clear image; specifically including: The dual-domain fusion features are upsampled using bilinear interpolation to obtain the upsampled results; The feature map is obtained by adding the upsampling result to the spatial domain features of the same scale through residual connection; Convolution is performed on the feature map to obtain upsampled features; channel feature fusion is then performed on the upsampled features through convolution to obtain a clear image.

[0010] In a second aspect, the present invention provides a rain and fog image restoration system based on spatial-frequency cooperative sensing, comprising: An encoder is used to acquire rain and fog images and extract spatial and frequency domain features from the rain and fog images, respectively. The cross-domain feature fusion module is used to fuse spatial domain features and frequency domain features based on an adaptive channel attention mechanism to obtain dual-domain fused features. The decoder, combined with residual connections, is used to add the dual-domain fused features and spatial domain features to obtain a feature map, and then convolution processing is performed on the feature map to obtain a clear image.

[0011] Preferably, the encoder includes: The spatial domain submodule is used to extract spatial domain features from rain and fog images; The frequency domain submodule is used to extract frequency domain features from rain and fog images.

[0012] Preferably, the spatial domain submodule includes an initial convolutional block and several downsampling blocks, and the extraction of spatial domain features from rain and fog images specifically includes: The initial convolutional block performs two convolutional operations on the image to decompose and extract the complex spatial feature patterns of the image; The downsampling block performs max pooling on the complex spatial feature patterns of the image to extract key features, remove secondary and redundant features, and output deep complex spatial feature patterns. The last downsampling block performs max pooling on deep, complex spatial feature patterns, extracting key features, removing secondary and redundant features, and then outputting the spatial domain features of the image.

[0013] Preferably, in the frequency domain submodule, extracting the frequency domain features of the rain and fog image specifically includes: The rain and fog image is transformed from spatial domain representation to frequency domain representation through Fourier transform to obtain a frequency domain representation image, and the pixel values ​​of the image are decomposed into frequency components; By moving the low-frequency components of the frequency domain representation image to the center of the spectrum, a centered spectrum is obtained. Extract the central region from the centralized spectrum, and obtain the real and imaginary parts of the central region respectively; The real and imaginary parts of the central region are updated by performing complex multiplication operations on a learnable complex filter to obtain the filtered central region. The filtered central region is returned to its corresponding position in the original spectrum and integrated with the unprocessed edge region to obtain the updated spectrum; The updated spectrum is then subjected to inverse Fourier reconstruction to the spatial domain to obtain the frequency domain features.

[0014] Preferably, in the cross-domain feature fusion module, spatial domain features and frequency domain features are fused based on an adaptive channel attention mechanism to obtain dual-domain fused features; specifically including: Channel-dimensional features are concatenated from spatial domain features and frequency domain features to obtain preliminary fused features; The initial fused features are compressed in spatial dimension by global average pooling to obtain scalar descriptive features; The scalar descriptive features are sequentially passed through the first fully connected layer for dimensionality reduction, activated by the ReLU activation function, and then increased in dimensionality by the second fully connected layer to obtain the activation features. The activation features are weighted based on the Sigmoid function to obtain the weight matrix; The weight matrix is ​​multiplied channel by channel with the preliminary fusion feature to obtain the dual-domain fusion feature.

[0015] Preferably, in the decoder, a feature map is obtained by adding the dual-domain fusion features of the restored resolution and the spatial domain features of the same size through residual connections, and then convolution processing is performed on the feature map to obtain a clear image; specifically including: The dual-domain fusion features are upsampled using bilinear interpolation to obtain the upsampled results; The feature map is obtained by adding the upsampling result to the spatial domain features of the same scale and performing residual connection. Convolution is performed on the feature map to obtain upsampled features; the upsampled features are then fused with channel features to obtain a clear image.

[0016] In a third aspect, the present invention provides an electronic device, including a processor and a memory, wherein the processor is configured to execute a computer program stored in the memory to implement the rain and fog image restoration method based on spatial frequency cooperative sensing as described in any one of the preceding claims.

[0017] In a fourth aspect, the present invention provides a computer-readable storage medium storing at least one instruction, wherein the at least one instruction, when executed by a processor, implements the rain and fog image restoration method based on spatial-frequency cooperative sensing as described above.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: The three key structures of the rain and fog image restoration model correspond to three main advantages: (1) The residual connection design replaces the skip connections of the traditional U-net with residual layer structures. The residual connection alleviates the gradient vanishing problem and improves the model training efficiency. This design enables the cross-layer fusion of shallow edge information and deep semantic features. Compared with the original skip connections, it is more conducive to the recovery of local details (such as object contours) and the removal of rain and fog textures.

[0019] (2) The spatial and frequency domain dual-branch encoder realizes the extraction and encoding of image features in different domains. The spatial domain branch consists of an initial convolution block and a downsampling block; the frequency domain branch includes a frequency domain transform and frequency domain feature extraction block and a frequency domain feature adjustment block. The spatial domain branch focuses on local structure restoration, while the frequency domain branch is responsible for global degradation component removal. The dual-branch collaborative mechanism makes the two complementary, thereby improving the model's ability to decouple complex degradation patterns.

[0020] (3) At the encoder output position, a channel attention mechanism is introduced during the fusion operation of spatial domain features and frequency domain features. The spliced ​​features of spatial domain features and frequency domain features are dynamically weighted by channels, and the weights of different channels are automatically adjusted during training. The automatic weight adjustment process enables adaptive suppression of rain and fog features and enhancement of image structural features. Attached Figure Description

[0021] The accompanying drawings, which form part of this application, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention; Figure 2 This is a system schematic diagram according to an embodiment of the present invention; Figure 3 This is a detailed diagram of the spatial domain submodule according to an embodiment of the present invention; Figure 4 This is a detailed diagram of the frequency domain submodule according to an embodiment of the present invention; Figure 5 This is a detailed diagram of the cross-domain feature fusion module according to an embodiment of the present invention; Figure 6 This is a detailed diagram of the decoder according to an embodiment of the present invention; Figure 7 This is a structural block diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0022] The present invention will now be described in detail with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other.

[0023] The following detailed description is exemplary and intended to provide further detailed explanation of the invention. Unless otherwise specified, all technical terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this invention is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention.

[0024] See Figure 1 This application discloses a method for rain and fog image restoration based on spatial-frequency collaborative sensing, including: S1: Acquire rain and fog images, and extract the spatial domain features and frequency domain features of the rain and fog images respectively; S2: Based on the adaptive channel attention mechanism, spatial domain features and frequency domain features are fused to obtain dual-domain fused features; S3: A feature map is obtained by adding the dual-domain fusion features of the restored resolution and the spatial domain features of the same size through residual connections. Convolution processing of the feature map yields a clear image. The residual connection design replaces the skip connections of traditional U-net with residual layer structures, mitigating the gradient vanishing problem and improving model training efficiency. This design enables cross-layer fusion of shallow edge information and deep semantic features, which is more conducive to the restoration of local details (such as object contours) and the removal of rain and fog textures compared to the original skip connections.

[0025] In some embodiments, acquiring a rain and fog image involves extracting spatial domain features and frequency domain features from the rain and fog image, respectively; specifically, extracting the spatial domain features of the rain and fog image includes: The image is subjected to two convolutional operations to decompose and extract the complex spatial feature patterns of the image; Max pooling is performed on the complex spatial feature patterns of the image to extract key features, remove secondary and redundant features, and output deep complex spatial feature patterns. Max pooling is performed on deep and complex spatial feature patterns to extract key features, remove secondary and redundant features, and output the spatial domain features of the image. Extracting frequency domain features from rain and fog images specifically includes: The rain and fog image is transformed from spatial domain representation to frequency domain representation through Fourier transform to obtain a frequency domain representation image, and the pixel values ​​of the image are decomposed into frequency components; By moving the low-frequency components of the frequency domain representation image to the center of the spectrum, a centered spectrum is obtained. Extract the central region from the centralized spectrum, and obtain the real and imaginary parts of the central region respectively; The real and imaginary parts of the central region are updated by performing complex multiplication operations on a learnable complex filter to obtain the filtered central region. The filtered central region is returned to its corresponding position in the original spectrum and integrated with the unprocessed edge region to obtain the updated spectrum; The updated spectrum is then subjected to inverse Fourier reconstruction to the spatial domain to obtain the frequency domain features.

[0026] A dual-branch encoder in the spatial and frequency domains extracts and encodes image features in different domains. The spatial domain branch consists of an initial convolutional block and a downsampling block; the frequency domain branch includes a frequency domain transform and frequency domain feature extraction block, and a frequency domain feature adjustment block. The spatial domain branch focuses on local structure restoration, while the frequency domain branch is responsible for global degradation component removal. The dual-branch collaborative mechanism enables the two branches to complement each other, thereby improving the model's ability to decouple complex degradation patterns.

[0027] In some embodiments, spatial domain features and frequency domain features are fused based on an adaptive channel attention mechanism to obtain dual-domain fused features; specifically including: Channel-dimensional features are concatenated from spatial domain features and frequency domain features to obtain preliminary fused features; The initial fused features are compressed in spatial dimension by global average pooling to obtain scalar descriptive features; The scalar descriptive features are sequentially passed through the first fully connected layer for dimensionality reduction, activated by the ReLU activation function, and then increased in dimensionality by the second fully connected layer to obtain the activation features. The activation features are weighted based on the Sigmoid function to obtain the weight matrix; The weight matrix is ​​multiplied channel by channel with the preliminary fusion feature to obtain the dual-domain fusion feature.

[0028] At the encoder output position, a channel attention mechanism is introduced during the fusion of spatial and frequency domain features. This mechanism dynamically weights the concatenated features from the spatial and frequency domains, with the weights of different channels automatically adjusted during training. This automatic weight adjustment enables adaptive suppression of rain and fog features and enhancement of image structural features.

[0029] In some embodiments, a feature map is obtained by adding the dual-domain fusion features that restore resolution and spatial domain features of the same size through residual connections, and then convolution processing is performed on the feature map to obtain a clear image; specifically including: The dual-domain fusion features are upsampled using bilinear interpolation to obtain the upsampled results; The feature map is obtained by adding the upsampling result to the spatial domain features of the same scale through residual connection; Convolution is performed on the feature map to obtain upsampled features; channel feature fusion is then performed on the upsampled features through convolution to obtain a clear image.

[0030] This application also discloses a rain and fog image restoration system based on spatial-frequency cooperative sensing, see [link to relevant documentation]. Figure 2 ,include: An encoder is used to acquire rain and fog images and extract spatial and frequency domain features from the rain and fog images, respectively. The cross-domain feature fusion module is used to fuse spatial domain features and frequency domain features based on an adaptive channel attention mechanism to obtain dual-domain fused features. The feature map is obtained by adding the dual-domain fusion features that restore the resolution and the spatial domain features of the same size through residual connection. The feature map is then convolved to obtain a clear image.

[0031] The rain and fog image restoration algorithm is designed based on U-net, and includes three key structures: an added frequency domain feature extraction module in the encoder; a channel attention mechanism in the fusion stage of the encoder's spatial and frequency domain features; and the replacement of skip connections between the encoder and decoder with residual connections (skip connections refer to channel-dimensional feature concatenation, while residual connections refer to the addition of two feature maps with the same feature map size and number of channels). First, the input rain and fog image undergoes parallel feature extraction by the encoder's spatial and frequency domain sub-modules. Second, at the encoder's output, the extracted spatial and frequency domain features are fused, incorporating an adaptive channel attention mechanism. Subsequently, the decoder receives the fused encoded features and high-resolution spatial domain features from the encoder's spatial domain sub-module, and during decoding, the upsampled feature map is added to the high-resolution features of the same size from the encoder.

[0032] In some embodiments, the encoder includes: The spatial domain submodule is used to extract spatial domain features from rain and fog images; The frequency domain submodule is used to extract frequency domain features from rain and fog images.

[0033] In some embodiments, the spatial domain submodule includes an initial convolutional block and several downsampling blocks, and extracting spatial domain features from rain and fog images specifically includes: The initial convolutional block performs two convolutional operations on the image to decompose and extract the complex spatial feature patterns of the image; The downsampling block performs max pooling on the complex spatial feature patterns of the image to extract key features, remove secondary and redundant features, and output deep complex spatial feature patterns. The last downsampling block performs max pooling on deep, complex spatial feature patterns, extracting key features, removing secondary and redundant features, and then outputting the spatial domain features of the image.

[0034] In some embodiments, the extraction of frequency domain features from rain and fog images in the frequency domain submodule specifically includes: The rain and fog image is transformed from spatial domain representation to frequency domain representation through Fourier transform to obtain a frequency domain representation image, and the pixel values ​​of the image are decomposed into frequency components; By moving the low-frequency components of the frequency domain representation image to the center of the spectrum, a centered spectrum is obtained. Extract the central region from the centralized spectrum, and obtain the real and imaginary parts of the central region respectively; The real and imaginary parts of the central region are updated by performing complex multiplication operations on a learnable complex filter to obtain the filtered central region. The filtered central region is returned to its corresponding position in the original spectrum and integrated with the unprocessed edge region to obtain the updated spectrum; The updated spectrum is then subjected to inverse Fourier reconstruction to the spatial domain to obtain the frequency domain features.

[0035] In some embodiments, the cross-domain feature fusion module fuses spatial domain features and frequency domain features based on an adaptive channel attention mechanism to obtain dual-domain fused features; specifically including: Channel-dimensional features are concatenated from spatial domain features and frequency domain features to obtain preliminary fused features; The initial fused features are compressed in spatial dimension by global average pooling to obtain scalar descriptive features; The scalar descriptive features are sequentially passed through the first fully connected layer for dimensionality reduction, activated by the ReLU activation function, and then increased in dimensionality by the second fully connected layer to obtain the activation features. The activation features are weighted based on the Sigmoid function to obtain the weight matrix; The weight matrix is ​​multiplied channel by channel with the preliminary fusion feature to obtain the dual-domain fusion feature.

[0036] In some embodiments, the decoder obtains a feature map by adding the dual-domain fusion features of the restored resolution and spatial domain features of the same size through residual connections, and then performs convolution processing on the feature map to obtain a clear image; specifically including: The dual-domain fusion features are upsampled using bilinear interpolation to obtain the upsampled results; The feature map is obtained by adding the upsampling result to the spatial domain features of the same scale and performing residual connection. Convolution is performed on the feature map to obtain upsampled features; the upsampled features are then fused with channel features to obtain a clear image.

[0037] In some embodiments, rain and fog images are acquired and preprocessed: The acquired dataset was divided into a training set (7200 pairs) and a test set (1800 pairs) in an 8:2 ratio. Both the training and test sets cover diverse scenarios such as urban traffic and communities.

[0038] Data augmentation was performed on the training dataset, including: randomly cropping the images to a size of 512×512 and flipping the cropped results horizontally or vertically; then applying color dithering (brightness ±0.1, contrast ±0.1, saturation ±0.1) to the flipped images to simulate device differences; finally, the data augmentation results were normalized to a mean of 0.5 and a variance of 0.5, unifying the mean and variance of the pixel values ​​of each channel of the image to 0.5.

[0039] End image preprocessing and output training and test sets.

[0040] In some embodiments, the encoder includes: The spatial domain submodule is used to extract spatial domain features from rain and fog images; The frequency domain submodule is used to extract frequency domain features from rain and fog images; specifically: Details of the internal structure of the spatial domain submodule are as follows Figure 3 As shown, the encoder spatial domain submodule contains two types of functional blocks: the initial convolutional block and the downsampling block. The initial convolutional block performs two convolutional operations on the image while maintaining the size of the feature map and increasing the number of channels. This operation increases the number of features that can be encoded at each spatial location while preserving the size of the feature map and avoiding loss of spatial details, allowing the network to extract more complex patterns. Through the consecutive operation of two convolutional layers, the size of the receptive field is maintained with fewer parameters (two 3×3 convolutional kernels are equivalent to the receptive field of a 5×5 convolutional kernel). The downsampling block adds a max-pooling operation at the input position based on the initial convolutional block. This max-pooling operation is the key operation for the downsampling block to perform its downsampling function. During downsampling, key information of the image is preserved while interference information is removed.

[0041] Details of the internal structure of the frequency domain submodule are as follows Figure 4 As shown. The three-channel dimensions are... The rain and fog image data undergoes a Fourier transform, converting it from a spatial domain representation to a frequency domain representation. The pixel values ​​of the image are decomposed into frequency components, preparing for feature processing in the frequency domain. In the spectrum centering stage, low-frequency components (typically representing the overall structure of the image) are moved to the center of the spectrum to facilitate subsequent filtering operations targeting the low-frequency region. Then, a value of [missing information - likely a specific value] is extracted from the centered spectrum. The real and imaginary parts of the central region are obtained separately. The central region usually contains low-frequency information of the image (such as smooth areas and main textures), and this region is extracted for targeted complex filtering. In frequency domain feature extraction, a learnable complex filter (containing real and imaginary parts) is used to perform complex multiplication on the central region, and the calculation formulas are shown in Equations (3) and (4), and the results are new real and imaginary parts.

[0042] (1) (2) (3) (4) in For the filter transfer function, This indicates the filtering result. Dot product, It is the result of the frequency domain transformation of the image. For frequency variables, Let represent the real part of the filter's transfer function. This represents the imaginary part of the filter's transfer function. Indicates the updated real part. Indicates the real part of the central region. Let represent the real part of the filter. Indicates the imaginary part of the central region. Represents the imaginary part of the filter. Indicates the updated imaginary part. , They represent The real and imaginary parts. This operation is equivalent to a rotation and scaling operation on the complex domain, which, from the perspective of signal processing, includes the following three effects: (1) the real part filter adjusts the in-phase component; (2) the imaginary part filter controls the quadrature component; (3) the combined effect can realize phase correction and amplitude modulation of specific frequency components.

[0043] Regarding the parameterized filter structure, the size of the filter in this paper is... The filter parameter matrix consists of two independent components, namely , These two parameters control the corresponding real and imaginary parts in the frequency domain, respectively. The filter parameters are automatically optimized through a backpropagation mechanism, enabling the network to adaptively suppress low-frequency interference caused by haze, enhance edge features, and maintain phase consistency of important frequency components.

[0044] Updating the frequency domain information involves placing the processed central region back into its corresponding position in the original spectrum to form a complete spectrum. This process integrates the filtered central region with the unprocessed edge regions. To unify the encoder's feature output, the frequency domain features need to be inversely transformed into spatial domain features. Before performing the inverse Fourier transform, the updated spectrum needs to be inversely centered to restore the spectrum to its original distribution, ensuring the correct arrangement of the spectrum. Then, based on the inverse Fourier transform, i.e., equation (5), the updated spectrum can be restored to the spatial domain.

[0045] (5) in It is the spatial domain result of the inverse Fourier transform. These represent the height and width of the output image, respectively. This is the filtered result of the frequency domain features. The spatial domain result of the inverse Fourier transform has the same size as the original image. To unify the sizes of the spatial domain features and frequency domain features in the feature fusion module, a frequency domain feature adjustment layer is designed to perform multi-layer convolution operations on the spatial domain result of the inverse Fourier transform. This enhances the feature map's expressive power while unifying the feature map size.

[0046] Once the encoder is constructed, it can extract features in both the spatial and frequency domains after inputting a rain and fog image.

[0047] In some embodiments, the cross-domain feature fusion module is used to fuse spatial domain features and frequency domain features based on an adaptive channel attention mechanism to obtain dual-domain fused features. (See also...) Figure 5 Specifically: Input spatial-frequency domain features, and then concatenate these dual-domain features using channel-dimensional features to achieve initial fusion of the dual-domain features.

[0048] An adaptive channel attention mechanism is introduced, and its implementation includes three key stages: feature compression, activation, and weight allocation. In the feature compression stage, global average pooling is used to compress the spatial dimension of the concatenated spatial domain features, reducing the dimensionality of each channel's two-dimensional features to scalar descriptive values. This operation not only reduces computational complexity but, more importantly, establishes a channel-level global receptive field. In the activation stage, two fully connected layers (a 1×1 convolutional kernel is equivalent to a fully connected layer) form a bottleneck structure. Through dimensionality reduction and expansion operations, combined with the ReLU activation function, a non-linear interaction relationship between channels is constructed. In the weight allocation stage, the weights are normalized to the (0,1) interval using the Sigmoid function, with specific values ​​greater than 0 and less than 1 representing the attention given to a particular channel feature.

[0049] The final fusion of dual-domain features involves multiplying the weight matrix with the initially fused dual-domain features channel by channel to achieve differentiated enhancement of different feature channels. It's worth noting that the weight allocation process is not a simple feature selection, but rather preserves the complete information flow of the features through a soft attention mechanism with continuous values. Furthermore, the 1×1 convolutional layer performs dimensionality reduction and fusion of the channels while preserving the spatial structure, and batch normalization ensures that extreme gradient values ​​do not occur during gradient propagation.

[0050] Complete the construction of the cross-domain feature fusion block and output the final fusion result of the dual-domain features.

[0051] In some embodiments, a feature map is obtained by adding the dual-domain fusion features of the restored resolution and spatial domain features of the same size through residual connections, and the feature map is then convolved to obtain a clear image, including: The input dual-domain fusion features are upsampled to restore the spatial resolution of the feature map. The upsampling method used in this algorithm is bilinear interpolation.

[0052] The upsampled result is added to high-resolution spatial features of the same scale from the encoder to complete the residual connection operation. This alleviates the vanishing gradient problem and improves model training efficiency. It enables cross-layer fusion of shallow edge information and deep semantic features, which is more conducive to the recovery of local details (such as object contours) and the removal of rain and fog textures.

[0053] The residual connection results are subjected to two convolutional layers, during which the size of the feature map is kept constant while the number of channels is increased. This increases the number of features that can be encoded at each spatial location while maintaining the size of the feature map and avoiding loss of spatial details, allowing the network to recognize more complex patterns. Through the successive operation of two convolutional layers, the size of the receptive field is maintained with fewer parameters (two 3×3 convolutional kernels are equivalent to the receptive field of a 5×5 convolutional kernel).

[0054] Complete the construction of the upsampling block, such as Figure 6 As shown, the output result serves as one of the input features for the next upsampling block.

[0055] The output of the last upsampled block is processed using the output block. During the feature fusion process, the number of channels is adjusted to ultimately restore the image. The output block is a convolutional layer with 64 input channels and 3 output channels, while maintaining the same feature map size.

[0056] Complete the construction of the decoder to output a clear image.

[0057] The structure of a complete rain and fog image reconstruction network is as follows: Figure 2As shown, the rain and fog image is processed by an encoder, which extracts features in both the spatial and frequency domains in parallel. The extracted dual-domain features are then input into a cross-domain feature fusion module. Based on a channel attention mechanism, different channels receive varying degrees of attention, highlighting the contour features of the main structures and the background features, while suppressing rain lines and fog noise. The dual-domain feature fusion result is input into the encoder's upsampling block to restore spatial pixel values. Residual connections are then used to combine the high-resolution spatial domain features from the encoder, achieving more accurate and efficient spatial pixel restoration. After four upsampling passes, the spatial domain pixel restoration result is fused with features from each channel through the output layer, finally outputting the restored image.

[0058] In some embodiments, the test data is input into a rain and fog image restoration system based on spatial-frequency cooperative sensing for training, including: During training, a loss function combining L1 and L2 is used, as shown in equation (8).

[0059] (6) (7) (8) in, For a clear image i Line number j Column pixel values, To repair the image i Line number j Column pixel values, These represent the number of rows and columns of the image, respectively. To balance the weighting factors, they are typically determined through grid search or task characteristics. This study focuses on detail recovery in rain and fog images, therefore the L1 loss requires a larger weight. In actual training, the following settings are configured... The value is 0.7.

[0060] Table 1 Comparison of L1 loss and L2 loss

[0061] Both types of loss have their advantages and disadvantages. Their respective disadvantages have a negative impact on the task of restoring rain and fog images, but their advantages are complementary and can suppress each other's disadvantages. Therefore, a hybrid loss function is used in the training process. The hybrid loss has three mechanisms of action. (1) Multi-scale error perception, that is, L2 loss constrains the low-frequency structure to ensure the rationality of the overall restoration, while L1 loss enhances high-frequency details to improve local clarity. (2) Dynamic balance of training, that is, in the early stage of training, L2 loss dominates to quickly fit the global structure; in the later stage of training, L1 loss dominates to refine local features. (3) Enhanced noise robustness, that is, L1 loss suppresses abnormal pixel interference, L2 loss prevents local overfitting, and the two work together to improve the generalization ability of the model.

[0062] The Adam algorithm was selected for network training.

[0063] A dynamic learning rate decay strategy was used during training. A step-by-step learning rate scheduler was employed, multiplying the learning rate by a decay factor (set to 0.5 in this paper) every 10 training epochs to halve the learning rate. The optimization effect of the network was understood by observing the changes in peak signal-to-noise ratio (PSNR) and structural similarity (SSIM).

[0064] After completing the system training, the effectiveness of this method in restoring rain and fog images can be verified using test set data.

[0065] like Figure 7 As shown, the present invention also provides an electronic device 100 for implementing a rain and fog image restoration method based on spatial frequency cooperative sensing; The electronic device 100 includes a memory 101, at least one processor 102, a computer program 103 stored in the memory 101 and executable on at least one processor 102, and at least one communication bus 104.

[0066] The memory 101 can be used to store the computer program 103. The processor 102 implements the steps of the rain and fog image restoration method based on spatial frequency cooperative sensing by running or executing the computer program stored in the memory 101 and calling the data stored in the memory 101.

[0067] The memory 101 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created based on the use of the electronic device 100 (such as audio data), etc. In addition, the memory 101 may include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other non-volatile solid-state storage device.

[0068] At least one processor 102 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Processor 102 may be a microprocessor or any conventional processor. Processor 102 is the control center of electronic device 100, connecting various parts of electronic device 100 via various interfaces and lines.

[0069] The memory 101 in the electronic device 100 stores multiple instructions to implement a rain and fog image restoration method based on spatial frequency cooperative sensing, and the processor 102 can execute multiple instructions to achieve the following: Acquire rain and fog images, and extract spatial domain features and frequency domain features from the rain and fog images respectively; Dual-domain fused features are obtained by fusing spatial domain features and frequency domain features based on an adaptive channel attention mechanism; The feature map is obtained by adding the dual-domain fusion features that restore the resolution and the spatial domain features of the same size through residual connection. The feature map is then convolved to obtain a clear image.

[0070] If the modules / units integrated in the electronic device 100 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, and read-only memory (ROM).

[0071] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0072] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0073] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0074] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0075] In the description of this specification, references to terms such as "an embodiment," "example," "specific example," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for rain and fog image restoration based on spatial-frequency collaborative sensing, characterized in that, include: Acquire rain and fog images, and extract spatial domain features and frequency domain features from the rain and fog images respectively; Dual-domain fused features are obtained by fusing spatial domain features and frequency domain features based on an adaptive channel attention mechanism; The feature map is obtained by adding the dual-domain fusion features that restore the resolution and the spatial domain features of the same size through residual connection. The feature map is then convolved to obtain a clear image.

2. The rain and fog image restoration method based on spatial-frequency collaborative sensing according to claim 1, characterized in that, The process involves acquiring rain and fog images and extracting their spatial domain and frequency domain features. Extracting spatial domain features from rain and fog images specifically includes: The image is subjected to two convolutional operations to decompose and extract the complex spatial feature patterns of the image; Max pooling is performed on the complex spatial feature patterns of the image to extract key features, remove secondary and redundant features, and output deep complex spatial feature patterns. Max pooling is performed on deep and complex spatial feature patterns to extract key features, remove secondary and redundant features, and output the spatial domain features of the image. Extracting frequency domain features from rain and fog images specifically includes: The rain and fog image is transformed from its spatial domain representation to its frequency domain representation using a Fourier transform, resulting in a frequency domain representation image. By moving the low-frequency components of the frequency domain representation image to the center of the spectrum, a centered spectrum is obtained. Extract the central region from the centralized spectrum, and obtain the real and imaginary parts of the central region respectively; The real and imaginary parts of the central region are updated by performing complex multiplication operations on a learnable complex filter to obtain the filtered central region. The filtered central region is returned to its corresponding position in the original spectrum and integrated with the unprocessed edge region to obtain the updated spectrum; The updated spectrum is then subjected to inverse Fourier reconstruction to the spatial domain to obtain the frequency domain features.

3. The rain and fog image restoration method based on spatial-frequency collaborative sensing according to claim 1, characterized in that, A dual-domain fused feature is obtained by fusing spatial domain features and frequency domain features based on an adaptive channel attention mechanism; specifically including: Channel-dimensional features are concatenated from spatial domain features and frequency domain features to obtain preliminary fused features; The initial fused features are compressed in spatial dimension by global average pooling to obtain scalar descriptive features; The scalar descriptive features are sequentially passed through the first fully connected layer for dimensionality reduction, activated by the ReLU activation function, and then increased in dimensionality by the second fully connected layer to obtain the activation features. The activation features are weighted based on the Sigmoid function to obtain the weight matrix; The weight matrix is ​​multiplied channel by channel with the preliminary fusion feature to obtain the dual-domain fusion feature.

4. The rain and fog image restoration method based on spatial-frequency collaborative sensing according to claim 1, characterized in that, A feature map is obtained by adding the dual-domain fusion features and spatial domain features of the same size through residual connections to restore resolution. The feature map is then convolved to obtain a clear image; specifically, this includes: The dual-domain fusion features are upsampled using bilinear interpolation to obtain the upsampled results; The feature map is obtained by adding the upsampling result to the spatial domain features of the same scale through residual connection; Convolution is performed on the feature map to obtain upsampled features; channel feature fusion is then performed on the upsampled features through convolution to obtain a clear image.

5. A rain and fog image restoration system based on spatial-frequency collaborative sensing, characterized in that, include: An encoder is used to acquire rain and fog images and extract spatial and frequency domain features from the rain and fog images, respectively. The cross-domain feature fusion module is used to fuse spatial domain features and frequency domain features based on an adaptive channel attention mechanism to obtain dual-domain fused features. The decoder, combined with residual connections, is used to add the dual-domain fused features and spatial domain features to obtain a feature map, and then convolution processing is performed on the feature map to obtain a clear image.

6. A rain and fog image restoration system based on spatial-frequency collaborative sensing according to claim 5, characterized in that, The encoder includes: The spatial domain submodule is used to extract spatial domain features from rain and fog images; The frequency domain submodule is used to extract frequency domain features from rain and fog images.

7. A rain and fog image restoration system based on spatial-frequency collaborative sensing according to claim 5, characterized in that, The spatial domain submodule includes an initial convolutional block and several downsampling blocks. Specifically, the extraction of spatial domain features from rain and fog images includes: The initial convolutional block performs two convolutional operations on the image to decompose and extract the complex spatial feature patterns of the image; The downsampling block performs max pooling on the complex spatial feature patterns of the image to extract key features, remove secondary and redundant features, and output deep complex spatial feature patterns. The last downsampling block performs max pooling on deep, complex spatial feature patterns, extracting key features, removing secondary and redundant features, and then outputting the spatial domain features of the image.

8. A rain and fog image restoration system based on spatial-frequency collaborative sensing according to claim 5, characterized in that, In the frequency domain submodule, extracting the frequency domain features of the rain and fog image specifically includes: The rain and fog image is transformed from spatial domain representation to frequency domain representation through Fourier transform to obtain a frequency domain representation image, and the pixel values ​​of the image are decomposed into frequency components; By moving the low-frequency components of the frequency domain representation image to the center of the spectrum, a centered spectrum is obtained. Extract the central region from the centralized spectrum, and obtain the real and imaginary parts of the central region respectively; The real and imaginary parts of the central region are updated by performing complex multiplication operations on a learnable complex filter to obtain the filtered central region. The filtered central region is returned to its corresponding position in the original spectrum and integrated with the unprocessed edge region to obtain the updated spectrum; The updated spectrum is then subjected to inverse Fourier reconstruction to the spatial domain to obtain the frequency domain features.

9. An electronic device, characterized in that, It includes a processor and a memory, the processor being used to execute a computer program stored in the memory to implement the rain and fog image restoration method based on spatial frequency cooperative sensing as described in any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, which, when executed by a processor, implements the rain and fog image restoration method based on spatial-frequency cooperative sensing as described in any one of claims 1 to 4.

Images