A multi-degraded image restoration method based on frequency domain decomposition
By employing frequency domain decomposition and feature fusion, the degradation problem of images caused by environmental factors such as rain, fog, and noise in traffic accident detection systems was solved, enabling multi-task image restoration and efficient detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENYANG INST OF COMPUTING TECH CO LTD THE CHINESE ACAD OF SCI
- Filing Date
- 2025-06-05
- Publication Date
- 2026-08-04
AI Technical Summary
In existing traffic accident detection systems, images are severely degraded by environmental factors such as rain, fog, and noise, leading to a decrease in the accuracy of detection algorithms. Furthermore, existing methods are not effective in multi-task recovery and cannot meet the real-time requirements in complex environments.
A multi-degraded image restoration method based on frequency domain decomposition is adopted. The UNet architecture is combined with the Transformer module and the HFPM and LTEM modules to process high-frequency and low-frequency components respectively. The image is restored through frequency domain decomposition and feature fusion, including frequency domain transformation, feature processing and inverse Fourier transform.
It achieves strong multi-task image restoration capabilities, can uniformly handle various degradation types such as rain, fog, and noise, and has a restoration effect superior to existing methods, improving image quality and the accuracy of detection algorithms.
Smart Images
Figure CN120707418B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and image processing technology, specifically relating to a multi-degraded image restoration method based on frequency domain decomposition, which is particularly suitable for image restoration tasks that simultaneously handle multiple degradation types such as rain, fog, and noise. Background Technology
[0002] In traffic accident detection systems, the image quality captured by surveillance cameras directly affects the accuracy of the detection algorithm. However, in practical applications, images are often severely degraded due to the following environmental factors:
[0003] Rainy conditions: Raindrops and rain streaks cause high-frequency interference, obscuring key details;
[0004] Foggy conditions: Fog and haze reduce contrast and blur the overall image;
[0005] Noise conditions: Noise affects target recognition;
[0006] Mixed degradation: The situation is more complex when multiple factors coexist.
[0007] The main technical challenges currently facing traffic accident detection include:
[0008] Traditional image restoration methods are more suited to single image degradation restoration tasks and are not very effective in multi-task image restoration.
[0009] Single-task recovery models based on deep learning (such as dedicated networks for de-raining and de-fogging) have insufficient generalization ability in complex and ever-changing road conditions.
[0010] High-frequency details (such as license plate numbers) and low-frequency structures (such as vehicle outlines) interfere with each other during the recovery process;
[0011] Due to the high requirements for real-time performance, existing methods struggle to meet processing speed requirements while ensuring quality. Summary of the Invention
[0012] This invention proposes a multi-degraded image restoration method based on frequency domain decomposition. Through an innovative frequency domain processing mechanism and deep learning architecture, it effectively solves the image degradation problem in complex environments during traffic accident detection. The core innovations of this invention are reflected in the following aspects:
[0013] A multi-degraded image restoration method based on frequency domain decomposition is proposed. A network model is built and trained using the UNet architecture, and Transformer modules, as well as constructed HFPM and LTEM modules, are incorporated into the encoding and decoding processes to repair image degradation caused by degradation factors. The method includes the following steps:
[0014] Outdoor degraded images are acquired and preprocessed; the degraded images contain rain, fog, noise, or a combination of degradation factors.
[0015] Based on the UNet architecture, a multi-level Transformer encoder and decoder is used to mine the dependencies between features in degraded images and to decode and restore the images.
[0016] The high semantic information encoding feature map output by the Transformer encoder is subjected to Fast Fourier Transform (FFT) to transform the image from the spatial domain to the frequency domain, and the frequency domain features are decomposed into high-frequency components and low-frequency components; the high-frequency components contain the edge and texture information of the image, and the low-frequency components contain the overall structure and background information of the image.
[0017] A high-frequency feature adaptive processing module (HFPM) is constructed to differentiate different high-frequency features through a dynamic weight parameter matrix and to capture cross-channel high-frequency feature dependencies using a multi-head attention mechanism.
[0018] A low-frequency feature transformation enhancement module (LTEM) is constructed, which uses an LSTM network to capture the temporal context information of low-frequency component features;
[0019] The high-frequency features processed by HFPM and the low-frequency features processed by LTEM are fused by feature concatenation and weighted summation.
[0020] The fused features are subjected to inverse Fourier transform to convert them from the frequency domain back to the spatial domain, resulting in a high-quality, clear image that is finally restored.
[0021] The image preprocessing includes: processing the image using at least one of rotation, segmentation, or image enhancement, and adjusting the image to a consistent resolution.
[0022] The frequency domain decomposition is obtained using the following formula:
[0023]
[0024] in:
[0025] F2 represents the two-dimensional Fourier transform;
[0026] S represents the spectrum centering operation, which moves the zero-frequency component to the center of the spectrum;
[0027] m is used to separate high and low frequencies;
[0028] This represents the inverse Fourier transform.
[0029] The high-frequency feature adaptive processing module HFPM specifically includes:
[0030] The high-frequency feature X is first processed by a 3×3 convolution to extract high-frequency key information;
[0031] By using a multi-head attention mechanism to compute attention in parallel across different subspaces, complex relationships within high-frequency components are captured, enhancing feature representation capabilities.
[0032] The Feature Enhancement Module (DEF) consists of two cascaded convolutional layers. The first convolutional layer compresses the number of feature channels to 1 / 4, and the second convolutional layer restores the original number of channels and generates a feature weight matrix.
[0033] Finally, a dropout is applied to prevent overfitting.
[0034] The feature enhancement module DEF specifically includes:
[0035] The number of feature channels is adjusted by a 1×1 convolution F0 to achieve linear transformation of the features; the first convolutional layer compresses the number of feature channels to 1 / 4.
[0036] By using the ReLU activation function to introduce nonlinearity, the model's ability to express nonlinear features is enhanced.
[0037] The feature information is further integrated by using a 1×1 convolution F1; the second convolutional layer restores the original number of channels.
[0038] The Sigmoid activation function is used to map the feature values to the range of 0-1, generating weight coefficients A. The weight matrix A is then multiplied element-wise with the input x to highlight the important parts of the input features.
[0039] The low-frequency feature conversion enhancement module LTEM specifically includes:
[0040] The low-frequency feature X is first processed by a 3×3 convolution to extract the initial low-frequency information X1, while keeping the spatial resolution of the feature map unchanged.
[0041] The three-dimensional feature map is transformed into a two-dimensional sequence through dimensional transformation; the long-distance dependencies between features are captured by the LSTM network, linking information from different spatial locations.
[0042] The multi-head attention mechanism (MHA) is used to compute attention in parallel across multiple subspaces, enhancing the correlation between features and further mining important features in low-frequency information.
[0043] Use the downsampling MaxPool2d operation to remove some local noise interference;
[0044] By using a dual-branch structure for differential processing, low-frequency structures are effectively captured while details are optimized. Normalization operations are used to stabilize training and suppress noise.
[0045] The dual-branch structure includes:
[0046] The left branch includes multiple cascaded residual blocks for deep feature extraction; low-frequency main structures are captured through 3×3 convolution, ReLU adapts to complex nonlinearity, and the second 3×3 convolution refines the image structure and transitions to make them more natural.
[0047] The right branch serves as a feature fusion unit, which adapts the feature dimension through a Linear layer for information integration; the BatchNorm layer normalizes the data and reduces noise interference.
[0048] The processed features are combined with the initial features through residual connections to output comprehensive low-frequency features.
[0049] During training, peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) were used as evaluation metrics to continuously optimize model parameters, enabling the model to learn the restoration features of images with different degradation types.
[0050] The present invention has the following beneficial effects and advantages:
[0051] Strong multi-task processing capability: The multi-task image restoration method based on frequency domain decomposition proposed in this invention can uniformly handle various unknown types of image degradation problems, such as rain removal, defogging, and noise reduction, avoiding the tedious process of training a separate model for each degradation type and improving the model's adaptability and generalization ability.
[0052] High-quality restoration: By decomposing the image into high-frequency and low-frequency components for separate processing, interference between high-frequency and low-frequency information during the restoration process is effectively avoided. The HFPM module highlights the detailed texture information of the image, while the LTEM module restores the structural integrity and semantic context of the image. The combination of the two significantly improves the image restoration quality. In experiments on several typical image degradation tasks, the model of this invention outperforms existing mainstream methods in evaluation metrics such as PSNR and SSIM.
[0053] Wide range of applications: This method can be applied to a variety of fields, such as traffic accident detection, environmental monitoring, and security monitoring. It can effectively improve image quality and provide better support for subsequent visual tasks, thus having broad application prospects. Attached Figure Description
[0054] Figure 1 This is a diagram of the overall structure of the invention.
[0055] Figure 2 This is a detailed structural diagram of the high-frequency feature adaptive processing module HFPM.
[0056] Figure 3 This is a detailed structural diagram of the low-frequency feature conversion enhancement module (LTEM). Detailed Implementation
[0057] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.
[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.
[0059] The method of this invention can simultaneously handle multiple image degradation problems, including but not limited to: rain streak removal: effectively restoring image details damaged by rain streak through the HFPM module; haze removal: restoring the overall contrast and color fidelity of haze images through the LTEM module; noise removal: separating noise into high-frequency components through frequency domain decomposition and then performing targeted suppression.
[0060] This invention decomposes an image into high-frequency and low-frequency components using Fast Fourier Transform (FFT), and then employs a High-Frequency Feature Adaptive Processing (HFPM) module and a Low-Frequency Feature Transformation and Enhancement (LTEM) module for targeted processing. Specifically, the HFPM module enhances image detail features through a dynamic weight parameter matrix and a multi-head attention mechanism; the LTM module recovers image structural features through an LSTM network; and finally, the processed high-frequency and low-frequency features are fused to output the restored image. The HFPM and LTM modules work collaboratively to achieve the following technical effects: avoiding mutual interference between high-frequency details and low-frequency structures through frequency domain decomposition; achieving adaptive processing for different degradation types through dynamic weights; and maintaining overall image consistency through feature fusion.
[0061] like Figure 1 As shown, a multi-degraded image restoration method based on frequency domain decomposition has the following overall process:
[0062] Step 1 (Data Acquisition):
[0063] Receive input degraded images, which may contain rain, fog, noise, or a mixture of various degradations;
[0064] Step 2 (Preprocessing the input image):
[0065] Image enhancement operations such as rotation and segmentation were performed on various degraded images. All images were uniformly resized to 128x128 for easier subsequent training.
[0066] Step 3 (Transformer encoding):
[0067] 3.1 Before performing the UNet encoding process, the channel dimension of the image is expanded from 3 dimensions to 48 dimensions to facilitate the processing of inter-channel features.
[0068] 3.2 UNet encoding stage:
[0069] The Transformer encoder employs a three-level coding structure, progressively enhancing feature representation capabilities: the first level extracts basic features through multi-head attention and convolutional downsampling; the second level increases the number of attention heads and improves feature dimensions, strengthening multi-scale feature integration; the third level maintains the same resolution and deeply mines the dependencies between features through multiple TransformerBlocks, ultimately outputting a high-semantic-information feature map, providing strong support for subsequent image restoration.
[0070] Step 4 (Frequency Domain Decomposition):
[0071] After the image encoding process is completed and the encoded feature map is obtained, a two-dimensional Fast Fourier Transform (FFT) is immediately performed to transform the image from the spatial domain to the frequency domain, resulting in a frequency domain feature representation. In the frequency domain, the low-frequency components represent the overall contour structure of the image and smooth areas of background information, while the high-frequency components correspond to detailed features such as edges and textures. This transformation is significant because it can clearly separate different types of image degradation factors. For example, Gaussian blur mainly affects the high-frequency components, while fog interference is mostly concentrated in the low-frequency components, allowing for targeted processing.
[0072] Step 5 (High-Frequency Feature Processing):
[0073] The high-frequency components are input into the high-frequency feature adaptive processing module (HFPM), which performs differentiated processing on different high-frequency features through a dynamic weight parameter matrix and uses a multi-head attention mechanism to capture the high-frequency feature dependencies across channels.
[0074] Step 6 (Low-frequency feature processing):
[0075] The low-frequency components are input into the low-frequency feature transformation enhancement module (LTEM), which uses an LSTM network to capture timing context information;
[0076] Step 7 (Feature Fusion):
[0077] The high-frequency features processed by HFPM and the low-frequency features processed by LTEM are fused. The fusion methods include feature concatenation and weighted summation.
[0078] Step 8 (Inverse Fourier Transform):
[0079] The fused features are subjected to an inverse Fourier transform to convert them from the frequency domain back to the spatial domain, resulting in the final reconstructed image.
[0080] Among them, the frequency domain decomposition mechanism in step 4 decomposes the image into high-frequency and low-frequency components through Fast Fourier Transform (FFT) to achieve feature decoupling.
[0081] The frequency domain decomposition is specifically achieved through the following mathematical formula:
[0082]
[0083] in:
[0084] F2 represents the two-dimensional Fourier transform;
[0085] S represents the spectrum centering operation, which moves the zero-frequency component to the center of the spectrum;
[0086] S -1 This indicates that the frequency component will be shifted back to its original position;
[0087] m is used to separate high and low frequencies;
[0088] This represents the inverse Fourier transform.
[0089] Step 5 specifically includes:
[0090] To address the high-frequency components obtained after FFT decomposition, this invention designs a High-Frequency Feature Adaptive Processing (HFPM) module. This module introduces a dynamic weight parameter matrix to differentiate various detailed features, highlighting key information. Its structure is as follows: Figure 2 As shown, it includes the following steps:
[0091] 5.1 Input high-frequency components:
[0092] The high-frequency component information obtained after frequency domain decomposition is input into the HFPM module. The high-frequency feature X is first processed by a 3×3 convolution to extract key high-frequency information.
[0093] 5.2 Multi-head attention mechanism (MHA) processing:
[0094] By using a multi-head attention mechanism to compute attention in parallel across different subspaces, complex relationships within high-frequency components are captured, thereby enhancing feature representation capabilities.
[0095] Multi-head attention (MHA) involves multiple parallel attention heads, each performing independent lookups, key-value transformations, and attention weights on the input features. MHA maps the input features to Q, K, and V spaces (i.e., Q... i =W qi X, K i =W ki X, V i =W vi X, i = 1, ..., n, W qi W ki W vi (where is the linear transformation matrix of the i-th attention head), enabling the model to learn feature representations from different perspectives.
[0096] 5.3 Feature Enhancement Module (DEF):
[0097] The number of feature channels is adjusted by a 1×1 convolution F0 to achieve linear transformation of the features; the first convolutional layer compresses the number of feature channels to 1 / 4.
[0098] By using the ReLU activation function to introduce nonlinearity, the model's ability to express nonlinear features is enhanced.
[0099] The feature information is further integrated by using a 1×1 convolution F1; the second convolutional layer restores the original number of channels.
[0100] The Sigmoid activation function maps feature values to the range of 0-1, generating weight coefficients A. This matrix reflects the importance of different feature elements in the image. The weight matrix A is then multiplied element-wise with the input x to highlight the important parts of the input features.
[0101] 5.4 High-frequency processing output:
[0102] Finally, a dropout is applied to prevent overfitting.
[0103] Step 6 specifically includes:
[0104] The low-frequency components obtained after FFT separation mainly contain background contour information of the image. This invention designs a low-frequency processing module that uses a Low-Frequency Feature Transformation Enhancement (LTEM) module, employing a Long Short-Term Memory (LSTM) recurrent neural network to effectively preserve the overall contour of the image while extracting features. Its structure is as follows: Figure 3 As shown, it includes the following steps:
[0105] 6.1 Input low-frequency information:
[0106] The low-frequency information obtained after frequency domain transformation of the image is input into the LTEM module. The input low-frequency feature X is first processed by a 3×3 convolution to extract the initial low-frequency information X1, while maintaining the spatial resolution of the feature map.
[0107] 6.2 LSTM Processing:
[0108] Low-frequency information is processed using Long Short-Term Memory (LSTM) networks. LSTM, through memory units and gating mechanisms, can effectively capture long-short-term dependencies in low-frequency information, model sequence information, and filter and retain key features in low-frequency information.
[0109] Dimensional deformation transforms three-dimensional feature maps into two-dimensional sequences to adapt to the input requirements of LSTM networks;
[0110] After undergoing certain dimensional changes, LSTM is used to capture long-distance dependencies between features and link information from different spatial locations.
[0111] 6.3 Multi-head attention mechanism (MHA) processing:
[0112] Features processed by LSTM are then fed into a multi-head attention mechanism (MHA). MHA computes attention in parallel across multiple subspaces, enhancing the correlation between features, further uncovering important features in low-frequency information, and improving feature representation capabilities.
[0113] 6.4 Use the downsampling MaxPool2d operation to remove some local noise interference;
[0114] 6.5 Multi-branch feature processing:
[0115] By using a dual-branch structure for differential processing, low-frequency structures are effectively captured while details are optimized. Normalization operations are used to stabilize training and suppress noise.
[0116] The left branch includes multiple cascaded residual blocks for deep feature extraction; low-frequency main structures are captured through 3×3 convolution, ReLU adapts to complex nonlinearities (such as illumination changes), and the second 3×3 convolution refines the image structure and transitions to make them more natural.
[0117] The right branch serves as a feature fusion unit, adapting the feature dimensions through a Linear layer to facilitate information integration; the BatchNorm layer normalizes the data, stabilizes training, and reduces noise interference; the processed features are combined with the initial features through residual connections, integrating the advantages of both branches to provide comprehensive low-frequency features for image restoration and improve the accuracy of the restoration.
[0118] Model architecture and training of this invention: The overall model of this invention is based on the UNet architecture, incorporating Transformer modules, HFPM, and LTEM during encoding and decoding. During training, peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are used as evaluation metrics to continuously optimize model parameters, enabling the model to learn the restoration features of images with different degradation types.
[0119] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should be considered within the scope of protection of the present invention.
Claims
1. A method for restoring multiple degraded images based on frequency domain decomposition, characterized in that, A network model based on the UNet architecture is trained, and a Transformer module, along with constructed HFPM and LTEM modules, are incorporated into the encoding and decoding processes to correct image degradation caused by degradative factors. The method includes the following steps: Outdoor degraded images are acquired and preprocessed; the degraded images contain rain, fog, noise, or a combination of degradation factors. Based on the UNet architecture, a multi-level Transformer encoder and decoder is used to mine the dependencies between features in degraded images and to decode and restore the images. The high semantic information encoding feature map output by the Transformer encoder is subjected to Fast Fourier Transform (FFT) to transform the image from the spatial domain to the frequency domain, and the frequency domain features are decomposed into high-frequency components and low-frequency components; the high-frequency components contain the edge and texture information of the image, and the low-frequency components contain the overall structure and background information of the image. A high-frequency feature adaptive processing module (HFPM) is constructed to differentiate different high-frequency features through a dynamic weight parameter matrix and to capture cross-channel high-frequency feature dependencies using a multi-head attention mechanism. A low-frequency feature transformation enhancement module (LTEM) is constructed, which uses an LSTM network to capture the temporal context information of low-frequency component features; The high-frequency features processed by HFPM and the low-frequency features processed by LTEM are fused by feature concatenation and weighted summation. The fused features are subjected to inverse Fourier transform to convert them from the frequency domain back to the spatial domain, resulting in a high-quality, clear image that is finally restored.
2. The method for restoring multiple degraded images based on frequency domain decomposition according to claim 1, characterized in that, The image preprocessing includes: processing the image using at least one of rotation, segmentation, or image enhancement, and adjusting the image to a consistent resolution.
3. The method for restoring multiple degraded images based on frequency domain decomposition according to claim 1, characterized in that, The frequency domain decomposition is obtained using the following formula: ； ； in: Represents a two-dimensional Fourier transform; S represents the spectrum centering operation, which moves the zero-frequency component to the center of the spectrum; This indicates that the frequency component will be shifted back to its original position; m is used to separate high and low frequencies; This represents the inverse Fourier transform.
4. The method for restoring multiple degraded images based on frequency domain decomposition according to claim 1, characterized in that, The high-frequency feature adaptive processing module HFPM specifically includes: The high-frequency feature X is first processed by a 3×3 convolution to extract high-frequency key information; By using a multi-head attention mechanism to compute attention in parallel across different subspaces, complex relationships within high-frequency components are captured, enhancing feature representation capabilities. The Feature Enhancement Module (DEF) consists of two cascaded convolutional layers. The first convolutional layer compresses the number of feature channels to 1 / 4, and the second convolutional layer restores the original number of channels and generates a feature weight matrix. Finally, a dropout is applied to prevent overfitting.
5. The method for restoring multiple degraded images based on frequency domain decomposition according to claim 4, characterized in that, The feature enhancement module DEF specifically includes: Through 1×1 convolution The number of feature channels is adjusted to achieve linear transformation of the features; the first convolutional layer compresses the number of feature channels to 1 / 4. By using the ReLU activation function to introduce nonlinearity, the model's ability to express nonlinear features is enhanced. Again using 1×1 convolution The second convolutional layer further integrates feature information; the second convolutional layer restores the original number of channels. The Sigmoid activation function is used to map the feature values to the range of 0-1, generating weight coefficients A. The weight matrix A is then multiplied element-wise with the input x to highlight the important parts of the input features.
6. The method for restoring multiple degraded images based on frequency domain decomposition according to claim 1, characterized in that, The low-frequency feature conversion enhancement module LTEM specifically include: Low frequency characteristics First pass through one Convolution to extract initial low-frequency information Maintain the feature map spatial resolution unchanged; The three-dimensional feature map is transformed into a two-dimensional sequence through dimensional transformation; the long-distance dependencies between features are captured by the LSTM network, linking information from different spatial locations. The multi-head attention mechanism (MHA) is used to compute attention in parallel across multiple subspaces, enhancing the correlation between features and further mining important features in low-frequency information. Use the downsampling MaxPool2d operation to remove some local noise interference; By using a dual-branch structure for differential processing, low-frequency structures are effectively captured while details are optimized. Normalization operations are used to stabilize training and suppress noise.
7. The method for restoring multiple degraded images based on frequency domain decomposition according to claim 6, characterized in that, The dual-branch structure include: The left branch includes multiple cascaded residual blocks for deep feature extraction; low-frequency main structures are captured through 3×3 convolution, ReLU adapts to complex nonlinearity, and the second 3×3 convolution refines the image structure and transitions to make them more natural. The right branch serves as a feature fusion unit, which adapts the feature dimension through a Linear layer for information integration; the BatchNorm layer normalizes the data and reduces noise interference. The processed features are combined with the initial features through residual connections to output comprehensive low-frequency features.
8. The method for restoring multiple degraded images based on frequency domain decomposition according to claim 1, characterized in that, During training, peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) were used as evaluation metrics to continuously optimize model parameters, enabling the model to learn the restoration features of images with different degradation types.