Multi-degraded image restoration method based on frequency domain decomposition

Abstract

The invention discloses a multi-degraded image recovery method based on frequency domain decomposition, and aims to solve the problems that a single model is difficult to deal with various image degradation and recovery processes of different frequency domains are mutually coupled in the prior art. According to the method, a degraded image is decomposed into a high-frequency space and a low-frequency space through fast Fourier transform, and a double-branch network architecture is adopted for targeted processing: for the high-frequency part, a high-frequency feature adaptive processing module HFPM is designed, and detail texture features are effectively extracted and interference is suppressed through feature enhancement and cross-layer fusion technologies; and for the low-frequency part, constructing a low-frequency feature conversion enhancement module LTEM, and capturing global context information by using cyclic convolution to improve the integrity of the structure contour. According to the method, decoupling processing of frequency domain features is realized, and the image restoration performance of the model in various degradation scenes such as rain removal, noise removal and defogging is remarkably improved through the synergistic effect of high-frequency detail enhancement and low-frequency structure optimization. Experimental results show that the method has excellent recovery effect and robustness when a plurality of image degradation tasks are processed at the same time, and can be effectively applied to visual tasks such as traffic accidents with high image quality requirements.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and image processing, and specifically relates to a multi-degraded image restoration method based on frequency domain decomposition, which is particularly suitable for simultaneously processing image restoration tasks with multiple degradation types such as rain, fog, and noise. Background Art

[0002] In traffic accident detection systems, the quality of images captured by surveillance cameras directly affects the accuracy of detection algorithms. However, in practical applications, images are often severely degraded due to the following environmental factors:

[0003] Rainy conditions: Raindrops and rain streaks create high-frequency interference, obscuring key details;

[0004] Foggy conditions: Haze causes a decrease in contrast and overall image blur;

[0005] Noise conditions: Noise affects target recognition;

[0006] Mixed degradation: The situation is more complicated when multiple factors exist at the same time.

[0007] The main technical challenges faced by existing traffic accident detection include:

[0008] Traditional image restoration methods are more focused on single image degradation restoration tasks and are not very effective in multi-task image restoration.

[0009] Single-task restoration models based on deep learning (such as rain removal and defogging dedicated networks) have insufficient generalization capabilities in complex and changing road conditions;

[0010] High-frequency details (such as license plate numbers) and low-frequency structures (such as vehicle outlines) interfere with each other during the restoration process;

[0011] The real-time requirement is high, and existing methods are difficult to meet the processing speed requirements while ensuring quality. Summary of the Invention

[0012] This paper proposes a multi-degraded image restoration method based on frequency domain decomposition. Through an innovative frequency domain processing mechanism and deep learning architecture, it effectively solves the image degradation problem in complex environments used in traffic accident detection. The core innovations of this paper are reflected in the following aspects:

[0013] A multi-degraded image restoration method based on frequency domain decomposition is proposed. A network model is established and trained based on the UNet architecture. The Transformer module and the constructed HFPM and LTEM modules are integrated into the encoding and decoding processes to repair image quality caused by degradation factors. The method includes the following steps:

[0014] Collecting outdoor degraded images for image preprocessing; the degraded images contain rain, fog, noise, or multiple mixed degradation factors;

[0015] Based on the UNet architecture, a multi-level Transformer encoder and decoder are used to mine the dependencies between features of degraded images and decode and restore them.

[0016] Perform a fast Fourier transform (FFT) on the high-semantic information encoding feature map output by the Transformer encoder to convert the image from the spatial domain to the frequency domain, and decompose the frequency domain features into high-frequency components and low-frequency components; the high-frequency components contain the edge and texture information of the image, and the low-frequency components contain the overall structure and background information of the image;

[0017] Construct a high-frequency feature adaptive processing module (HFPM) to differentiate different high-frequency features through a dynamic weight parameter matrix and use a multi-head attention mechanism to capture cross-channel high-frequency feature dependencies.

[0018] Construct a low-frequency feature transformation enhancement module (LTEM) and use the LSTM network to capture the temporal context information of low-frequency component features.

[0019] The high-frequency features processed by HFPM and the low-frequency features processed by LTEM are fused through feature concatenation and weighted summation;

[0020] The fused features are inverse Fourier transformed to convert them from the frequency domain back to the spatial domain to obtain the final restored high-quality and clear image.

[0021] The image preprocessing includes: performing processing by at least one of rotation, segmentation, or image enhancement, and adjusting the image to have a consistent resolution.

[0022] The frequency domain decomposition is obtained by the following formula:

[0023]

[0024] in:

[0025] F2 represents two-dimensional Fourier transform;

[0026] S represents the spectrum centering operation, which moves the zero-frequency component to the center of the spectrum;

[0027] m is used to separate high and low frequencies;

[0028] represents the inverse Fourier transform.

[0029] The high frequency feature adaptive processing module HFPM specifically includes:

[0030] The high-frequency feature X is first processed by a 3×3 convolution to extract high-frequency key information;

[0031] The multi-head attention mechanism calculates attention in parallel in different subspaces to capture the complex correlation relationships within high-frequency components and enhance feature representation capabilities;

[0032] The feature enhancement module DEF consists of two cascaded convolutional layers. The first convolutional layer compresses the number of feature channels to 1 / 4, and the second convolutional layer restores the original number of channels and generates a feature weight matrix.

[0033] Finally, a dropout is performed to prevent overfitting.

[0034] The feature enhancement module DEF specifically includes:

[0035] The number of feature channels is adjusted through 1×1 convolution F0 to achieve linear transformation of features; the first convolution layer compresses the number of feature channels to 1 / 4;

[0036] Use the ReLU activation function to introduce nonlinearity and enhance the model's ability to express features nonlinearly;

[0037] The feature information is further integrated through the 1×1 convolution F1 again; the second convolution layer restores the original number of channels;

[0038] Through the Sigmoid activation function, the eigenvalue is mapped to the range of 0-1 to generate the weight coefficient A; the weight matrix A is multiplied element-by-element by the input x to highlight the important parts of the input features.

[0039] The low-frequency feature conversion enhancement module LTEM specifically includes:

[0040] The low-frequency feature X first undergoes a 3×3 convolution to extract the initial low-frequency information X1, keeping the spatial resolution of the feature map unchanged;

[0041] The 3D feature map is transformed into a 2D sequence through dimension transformation. The LSTM network is used to capture the long-distance dependencies between features and link information at different spatial locations.

[0042] The multi-head attention mechanism (MHA) is used to calculate attention in parallel in multiple subspaces to enhance the correlation between features and further explore important features in low-frequency information.

[0043] Use the downsampling MaxPool2d operation to remove some local noise interference;

[0044] Through dual-branch structure differentiation processing, it effectively captures low-frequency structures while optimizing details, and stabilizes training and suppresses noise through normalization operations.

[0045] The double-branch structure comprises:

[0046] The left branch includes multiple cascaded residual blocks for deep feature extraction; 3×3 convolution captures the main low-frequency structure, ReLU adapts to complex nonlinearities, and the second 3×3 convolution refines the image structure and transitions to make them more natural.

[0047] The right branch acts as a feature fusion unit, which adapts the feature dimension through the Linear layer for information integration; the BatchNorm layer performs normalization to reduce noise interference;

[0048] The processed features are combined with the initial features through residual connections to output comprehensive low-frequency features.

[0049] During the training process, the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are used as evaluation indicators to continuously optimize the model parameters so that the model can learn the restoration features of images with different degradation types.

[0050] The present invention has the following beneficial effects and advantages:

[0051] Strong multi-task processing capability: The multi-task image restoration method based on frequency domain decomposition proposed in this invention can uniformly handle multiple unknown types of image degradation problems, such as rain removal, fog removal, and denoising, avoiding the tedious process of training a separate model for each degradation type, and improving the adaptability and generalization ability of the model.

[0052] Excellent restoration: By decomposing the image into high-frequency and low-frequency components and processing them separately, interference between high-frequency and low-frequency information during the restoration process is effectively avoided. The HFPM module highlights the image's detailed texture information, while the LTEM module restores the image's structural integrity and semantic context. The combination of the two significantly improves image restoration quality. In experiments on multiple typical image degradation tasks, the proposed model outperforms existing mainstream methods in evaluation metrics such as PSNR and SSIM.

[0053] Broad application prospects: This method can be applied to a variety of fields, such as traffic accident detection, environmental monitoring, security monitoring, etc. It can effectively improve image quality and provide better support for subsequent visual tasks, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is the overall structural diagram of the invention.

[0055] Figure 2 This is the detailed structure diagram of the high-frequency feature adaptive processing module HFPM.

[0056] Figure 3 Detailed structure diagram of the low-frequency feature transformation enhancement module LTEM. DETAILED DESCRIPTION

[0057] To make the above-mentioned objects, features, and advantages of the present invention more readily apparent, the specific implementation methods of the present invention are described in detail below with reference to the accompanying drawings. The following description sets forth many specific details to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the invention. Therefore, the present invention is not limited to the specific implementation methods disclosed below.

[0058] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art of the art to which the present invention pertains. The terms used in the specification of the invention herein are for the purpose of describing specific embodiments only and are not intended to limit the present invention.

[0059] The method of the present invention can simultaneously handle multiple image degradation problems, including but not limited to: rain streak removal: effectively restoring image details destroyed by rain streaks through the HFPM module; haze removal: restoring the overall contrast and color fidelity of haze images through the LTEM module; noise removal: separating noise into high-frequency components through frequency domain decomposition and then performing targeted suppression.

[0060] The method of the present invention decomposes an image into high-frequency and low-frequency components using a fast Fourier transform (FFT). It then uses a high-frequency feature adaptive processing module (HFPM) and a low-frequency feature transformation enhancement module (LTEM) for targeted processing. The HFPM module enhances image detail features using a dynamic weight parameter matrix and a multi-head attention mechanism; the LTEM module restores image structural features using an LSTM network; and finally, the processed high-frequency and low-frequency features are fused to output a restored image. The HFPM and LTEM modules work together to achieve the following technical effects: frequency domain decomposition avoids interference between high-frequency details and low-frequency structures; dynamic weights enable adaptive processing of different degradation types; and feature fusion maintains overall image consistency.

[0061] like Figure 1 As shown in Figure 1, a multi-degraded image restoration method based on frequency domain decomposition has the following overall process:

[0062] Step 1 (data collection):

[0063] Receive an input degraded image, which may contain rain, fog, noise, or multiple mixed degradations;

[0064] Step 2 (preprocessing the input image):

[0065] Perform image enhancement operations such as rotation and segmentation on various degraded images. Resize the images to a 128x128 format to facilitate subsequent training.

[0066] Step 3 (Transformer encoding):

[0067] 3.1 Before the UNet encoding process, the channel dimension of the image is expanded from 3 dimensions to 48 dimensions to facilitate the processing of inter-channel features.

[0068] 3.2UNet encoding stage:

[0069] The Transformer encoder adopts a three-level encoding structure to enhance feature expression capabilities step by step: the first level extracts basic features through multi-head attention and convolutional downsampling; the second level increases the number of attention heads and improves the feature dimension to strengthen multi-scale feature integration; the third level maintains the resolution unchanged and deeply mines the dependencies between features through multi-layer TransformerBlock, and finally outputs a high-semantic information feature map to provide strong support for subsequent image restoration.

[0070] Step 4 (frequency domain decomposition):

[0071] After the image is encoded and the encoded feature map is obtained, a two-dimensional fast Fourier transform (FFT) is immediately performed to convert the image from the spatial domain to the frequency domain, obtaining a frequency domain feature representation. In the frequency domain, the low-frequency component represents the overall image structure and smooth areas of background information, while the high-frequency component corresponds to detailed features such as edges and textures. This conversion is of great significance as it can clearly separate different types of image degradation factors. Because Gaussian blur mainly affects high-frequency components, fog interference is mostly concentrated in the low-frequency part, so it can be processed in a targeted manner.

[0072] Step 5 (high frequency feature processing):

[0073] The high-frequency components are input into the high-frequency feature adaptive processing module (HFPM), which performs differential processing on different high-frequency features through a dynamic weight parameter matrix and uses a multi-head attention mechanism to capture cross-channel high-frequency feature dependencies;

[0074] Step 6 (low-frequency feature processing):

[0075] The low-frequency component is input into a low-frequency feature transformation enhancement module (LTEM), which uses an LSTM network to capture temporal context information;

[0076] Step 7 (feature fusion):

[0077] The high-frequency features processed by HFPM and the low-frequency features processed by LTEM are fused using feature concatenation and weighted summation.

[0078] Step 8 (Inverse Fourier Transform):

[0079] The fused features are inverse Fourier transformed to convert them from the frequency domain back to the spatial domain to obtain the final restored image.

[0080] Among them, the frequency domain decomposition mechanism of step 4: decompose the image into high-frequency and low-frequency components through fast Fourier transform (FFT) to achieve feature decoupling;

[0081] The frequency domain decomposition is specifically implemented by the following mathematical formula:

[0082]

[0083] in:

[0084] F2 represents two-dimensional Fourier transform;

[0085] S represents the spectrum centering operation, which moves the zero-frequency component to the center of the spectrum;

[0086] S -1 Indicates moving the frequency component back to its original position;

[0087] m is used to separate high and low frequencies;

[0088] represents the inverse Fourier transform.

[0089] Among them, step 5 specifically includes:

[0090] For the high frequency components obtained after FFT decomposition, the present invention designs a high frequency feature adaptive processing module (HFPM), which introduces a dynamic weight parameter matrix to perform differential processing on various detail features and highlight key information. Its structure is as follows: Figure 2 As shown, the following steps are included:

[0091] 5.1 Input high frequency component:

[0092] The high-frequency component information obtained after frequency domain decomposition is input into the HFPM module. The high-frequency feature X first undergoes a 3×3 convolution to extract high-frequency key information.

[0093] 5.2 Multi-head Attention Mechanism (MHA) Processing:

[0094] Through the multi-head attention mechanism, attention is calculated in parallel in different subspaces to capture the complex correlation relationships within high-frequency components and enhance the feature representation capability.

[0095] The Multi-Head Attention Mechanism (MHA) consists of multiple parallel attention heads. Each attention head performs independent query, key and value transformation on the input features and calculates the attention weight. The Multi-Head Attention Mechanism (MHA) maps the input features to the Q, K and V spaces (i.e. Q i =W qi X, K i =W ki X, V i =W vi X, i = 1, ... n, W qi 、W ki 、W vi is the linear transformation matrix of the i-th attention head), enabling the model to learn feature representations from different perspectives.

[0096] 5.3 Feature Enhancement Module (DEF):

[0097] The number of feature channels is adjusted through 1×1 convolution F0 to achieve linear transformation of features; the first convolution layer compresses the number of feature channels to 1 / 4;

[0098] Use the ReLU activation function to introduce nonlinearity and enhance the model's ability to express features nonlinearly;

[0099] The feature information is further integrated through the 1×1 convolution F1 again; the second convolution layer restores the original number of channels;

[0100] The eigenvalues ​​are mapped to a range of 0-1 through the Sigmoid activation function to generate weight coefficients A. This matrix reflects the importance of different feature elements in the image. The weight matrix A is multiplied element-by-element by the input x to highlight the important parts of the input features.

[0101] 5.4 High frequency processing output:

[0102] Finally, a dropout is performed to prevent overfitting.

[0103] Step 6 specifically includes:

[0104] The low-frequency components obtained after FFT separation mainly contain the background contour information of the image. The present invention designs a low-frequency processing module (LTEM) that uses a long short-term memory recurrent neural network (LSTM) to effectively retain the overall contour of the image while extracting features. Its structure is as follows Figure 3 As shown, the following steps are included:

[0105] 6.1 Input low frequency information:

[0106] The low-frequency information obtained after the image is converted to the frequency domain is input into the LTEM module. The input low-frequency feature X first undergoes a 3×3 convolution to extract the initial low-frequency information X1, maintaining the spatial resolution of the feature map unchanged.

[0107] 6.2LSTM processing:

[0108] Long Short-Term Memory (LSTM) networks are used to process low-frequency information. Through memory cells and gating mechanisms, LSTMs can effectively capture long-term and short-term dependencies in low-frequency information, model sequence information, and filter and retain key features in low-frequency information.

[0109] Dimension deformation, converting the three-dimensional feature map into a two-dimensional sequence to meet the input requirements of the LSTM network;

[0110] After a certain dimensionality change, it is processed by LSTM to capture the long-distance dependencies between features and link information at different spatial locations.

[0111] 6.3 Multi-Head Attention Mechanism (MHA) Processing:

[0112] The features processed by LSTM are fed into the multi-head attention mechanism. MHA computes attention in parallel across multiple subspaces, enhancing the correlation between features, further mining important features in low-frequency information, and improving feature representation capabilities.

[0113] 6.4 Use the downsampling MaxPool2d operation to remove some local noise interference;

[0114] 6.5 Multi-branch feature processing:

[0115] Through dual-branch structure differentiation processing, it effectively captures low-frequency structures while optimizing details, and stabilizes training and suppresses noise through normalization operations.

[0116] Among them, the left branch includes multiple cascaded residual blocks for deep feature extraction; the low-frequency main structure is captured by 3×3 convolution, ReLU adapts to complex nonlinearities (such as illumination changes), and the second 3×3 convolution is refined to make the image structure and transition more natural.

[0117] Among them, the right branch serves as a feature fusion unit, adapting the feature dimension through the Linear layer to facilitate information integration; the BatchNorm layer normalizes, stabilizes training, and reduces noise interference; the processed features are combined with the initial features through a residual connection, combining the advantages of the two branches to provide comprehensive low-frequency features for image restoration and improve the accuracy of the restoration effect.

[0118] Model Architecture and Training: The overall model of this invention is based on the UNet architecture, incorporating Transformer modules, as well as HFPM and LTEM, during encoding and decoding. During training, model parameters are continuously optimized using metrics such as Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM), enabling the model to learn the restored features of images with different degradation types.

[0119] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should be regarded as within the scope of protection of the present invention.

Claims

1. A multi-degraded image restoration method based on frequency domain decomposition, characterized in that: A network model is established based on the UNet architecture for training. The Transformer module, as well as the constructed HFPM and LTEM modules, are integrated into the encoding and decoding processes to repair image quality degradation caused by degradation factors. The method includes the following steps: Collecting outdoor degraded images for image preprocessing; the degraded images contain rain, fog, noise, or multiple mixed degradation factors; Based on the UNet architecture, a multi-level Transformer encoder and decoder are used to mine the dependencies between features of degraded images and decode and restore them. Perform a fast Fourier transform (FFT) on the high-semantic information encoding feature map output by the Transformer encoder to convert the image from the spatial domain to the frequency domain, and decompose the frequency domain features into high-frequency components and low-frequency components; the high-frequency components contain the edge and texture information of the image, and the low-frequency components contain the overall structure and background information of the image; Construct a high-frequency feature adaptive processing module (HFPM) to differentiate different high-frequency features through a dynamic weight parameter matrix and use a multi-head attention mechanism to capture cross-channel high-frequency feature dependencies. Construct a low-frequency feature transformation enhancement module (LTEM) and use the LSTM network to capture the temporal context information of low-frequency component features. The high-frequency features processed by HFPM and the low-frequency features processed by LTEM are fused through feature concatenation and weighted summation; The fused features are inverse Fourier transformed to convert them from the frequency domain back to the spatial domain to obtain the final restored high-quality and clear image.

2. The method for restoring multiple degraded images based on frequency domain decomposition according to claim 1, characterized in that: The image preprocessing includes: performing processing by at least one of rotation, segmentation, or image enhancement, and adjusting the image to have a consistent resolution.

3. The method for restoring multiple degraded images based on frequency domain decomposition according to claim 1, characterized in that: The frequency domain decomposition is obtained by the following formula: in: F2 represents two-dimensional Fourier transform; S represents the spectrum centering operation, which moves the zero-frequency component to the center of the spectrum; m is used to separate high and low frequencies; represents the inverse Fourier transform.

4. The method for restoring multiple degraded images based on frequency domain decomposition according to claim 1, characterized in that: The high frequency feature adaptive processing module HFPM specifically includes: The high-frequency feature X is first processed by a 3×3 convolution to extract high-frequency key information; The multi-head attention mechanism calculates attention in parallel in different subspaces to capture the complex correlation relationships within high-frequency components and enhance feature representation capabilities; The feature enhancement module DEF consists of two cascaded convolutional layers. The first convolutional layer compresses the number of feature channels to 1 / 4, and the second convolutional layer restores the original number of channels and generates a feature weight matrix. Finally, a dropout is performed to prevent overfitting.

5. The method for restoring multiple degraded images based on frequency domain decomposition according to claim 4, characterized in that: The feature enhancement module DEF specifically includes: The number of feature channels is adjusted through 1×1 convolution F0 to achieve linear transformation of features; the first convolution layer compresses the number of feature channels to 1 / 4; Use the ReLU activation function to introduce nonlinearity and enhance the model's ability to express features nonlinearly; The feature information is further integrated through the 1×1 convolution F1 again; the second convolution layer restores the original number of channels; Through the Sigmoid activation function, the eigenvalue is mapped to the range of 0-1 to generate the weight coefficient A; the weight matrix A is multiplied element-by-element by the input x to highlight the important parts of the input features.

6. The method for restoring multiple degraded images based on frequency domain decomposition according to claim 1, characterized in that: The low-frequency feature conversion enhancement module LTEM specifically includes: The low-frequency feature X first undergoes a 3×3 convolution to extract the initial low-frequency information X1, keeping the spatial resolution of the feature map unchanged; The 3D feature map is transformed into a 2D sequence through dimension transformation. The LSTM network is used to capture the long-distance dependencies between features and link information at different spatial locations. The multi-head attention mechanism (MHA) is used to calculate attention in parallel in multiple subspaces to enhance the correlation between features and further explore important features in low-frequency information. Use the downsampling MaxPool2d operation to remove some local noise interference; Through dual-branch structure differentiation processing, it effectively captures low-frequency structures while optimizing details, and stabilizes training and suppresses noise through normalization operations.

7. The method for restoring multiple degraded images based on frequency domain decomposition according to claim 6, characterized in that: The double-branch structure include: The left branch includes multiple cascaded residual blocks for deep feature extraction; 3×3 convolution captures the main low-frequency structure, ReLU adapts to complex nonlinearities, and the second 3×3 convolution refines the image structure and transitions to make them more natural. The right branch acts as a feature fusion unit, which adapts the feature dimension through the Linear layer for information integration; the BatchNorm layer performs normalization to reduce noise interference; The processed features are combined with the initial features through residual connections to output comprehensive low-frequency features.

8. The method for restoring multiple degraded images based on frequency domain decomposition according to claim 1, characterized in that: During the training process, the peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM) are used as evaluation indicators to continuously optimize the model parameters so that the model can learn the restoration features of images with different degradation types.

Images