Space-frequency domain collaborative low-light deblurring method based on Fourier transform and Mama architecture

Abstract

The invention relates to a space-frequency domain collaborative low-light deblurring method based on Fourier transform and Mama architecture, and belongs to the technical field of image processing. According to the method, an image is decomposed through Fourier transform, and effective separation of low light degradation and fuzzy degradation is achieved at a frequency domain source according to the physical characteristics of amplitude spectrum dominant illumination and a phase spectrum dominant structure. In order to solve the problem of accurate reconstruction of frequency domain information in a spatial domain, a space-frequency domain double-branch collaborative architecture is constructed, global illumination recovery and local detail enhancement are respectively focused through low-frequency and high-frequency processing branches which are explicitly distinguished in the spatial domain, and frequency domain decoupling advantages and spatial domain sensing capability are adaptively bridged by combining a dynamic feature fusion mechanism. Meanwhile, in order to process different degradation types in the recovery process, a task-oriented decoupling optimization strategy is designed, the strategy activates amplitude adjustment and low-frequency optimization in the encoding stage to deal with low-light enhancement preferentially, and activates phase reconstruction and high-frequency enhancement in the decoding stage to process blurring removal in a targeted mode.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a spatial-frequency domain collaborative low-light deblurring method based on Fourier transform and Mamba architecture. Background Technology

[0002] In nighttime or low-light environments, long-exposure strategies employed to achieve sufficient brightness easily lead to motion blur in images. Simultaneously, the images may still suffer from insufficient brightness, significant noise, and loss of detail. This combined degradation of illumination attenuation and motion blur makes image content difficult to discern, severely limiting its reliability in critical low-light dynamic scenarios such as nighttime traffic monitoring, autonomous driving perception, drone nighttime reconnaissance, mobile night scene photography, and security deployment. Therefore, achieving high-quality joint restoration of low-light blurred images is crucial. Low-light deblurring technology is a technical direction specifically researched for the sharpening and restoration of such compositely degraded images. In recent years, researchers have proposed a series of methods to solve the low-light deblurring problem, which can be broadly divided into two categories: model-cascade-based approaches and end-to-end joint restoration approaches.

[0003] Low-light deblurring methods based on the model cascade approach do not explicitly model the coupling mechanism between low-light and blur degradation. Instead, they sequentially cascade low-light enhancement models and image deblurring models, attempting to solve the compound degradation problem in stages. These methods can be further divided into two strategies: enhancement-then-deblurring cascade and deblurring-then-enhancement cascade. The enhancement-then-deblurring strategy first uses a low-light enhancement model (such as Zero-DCE) to increase the brightness of the input image, and then uses a deblurring model (such as Restormer) to process the enhanced image. The deblurring-then-enhancement strategy first uses a deblurring model (such as Uformer) to restore the image structure, and then uses an enhancement model (such as LLFlow) to improve lighting conditions. The advantage of the cascaded approach is that it can directly utilize mature single degradation processing models, making it relatively simple to implement. However, the cascaded method has limitations: the original image has a low signal-to-noise ratio under low light conditions, and the enhancement stage will significantly amplify noise and introduce artifacts such as color distortion; the deblurring stage processes the enhanced image, which may contain amplified noise and artifacts, making it difficult to accurately recover structural details; the processing errors of the two stages are mutually propagated and accumulated, resulting in problems such as residual noise, obvious artifacts, and detail distortion in the final output.

[0004] End-to-end joint restoration-based low-light deblurring methods aim to directly learn the complex mapping from low-light blurred images to clear, normally lit images through a unified network architecture, avoiding the accumulation of errors from staged processing. These methods focus on establishing a light-blur composite degradation model and designing end-to-end trainable networks in the spatial or cross-domain domains to jointly compensate for both degradations. They can be categorized into spatial joint processing methods and methods incorporating prior guidance. Spatial joint processing methods directly construct encoder-decoder networks in pixel space, implicitly fusing low-light enhancement and deblurring feature flows through convolutional or Transformer modules. For example, LEDNet uses a multi-scale encoder-decoder to uniformly learn the two degradation compensations. Methods incorporating prior guidance additionally utilize data-driven internal or external prior constraints on the restoration process. FourierDiff uses Fourier frequency domain prior guidance to generate cross-domain consistent structural information using a diffusion model, while LIEDNet integrates global and local illumination through a Local Feature-Visual State Space Module (LF-VSS). While such methods can achieve more synergistic restoration, they still face core bottlenecks: the global contrast attenuation caused by low light and the local structural damage caused by motion blur are highly coupled in the spatial domain, and the unified network cannot explicitly decouple the two, forcing the feature extraction process to simultaneously deal with mixed degradation signals, thus limiting the ability to reconstruct high-frequency details; the reliance on specific prior knowledge may pose challenges in generalization scenarios, and artifacts or texture distortion are easily generated when the prior does not match the specific scene. Existing end-to-end methods suffer from problems such as boundary blurring and noise residue in extremely low light or strongly blurred scenes, and it is difficult to effectively decouple the light attenuation degradation mechanism and the motion blur degradation mechanism, resulting in insufficient recovery of high-frequency details (such as edge sharpness and texture information) in the restored image; at the same time, methods that rely on external prior constraints face challenges in the stability and reliability of restoration results when facing specific scenes that do not match the prior knowledge. Summary of the Invention

[0005] The purpose of this invention is to provide a space-frequency domain collaborative low-light deblurring method based on Fourier transform and Mamba architecture. It proposes a low-light deblurring network SFFMNet based on frequency domain physical characteristics and space-frequency collaboration. The method of this invention can accurately achieve low-light deblurring of images.

[0006] To achieve the above objectives, the technical solution of this invention is: a spatial-frequency domain collaborative low-light deblurring method based on Fourier transform and Mamba architecture, comprising:

[0007] Feature extraction stage: The input degraded image is processed through a single 3x3 convolutional layer to extract image patches and embed them into features, which are then input into the encoding stage;

[0008] Encoding stage: Image patch embedded features are enhanced for low light through a three-layer encoder module, and the low light enhanced features are input into the decoding stage;

[0009] Decoding stage: The low-light enhanced features are deblurred through a two-layer decoder module to output a clean image with low-light deblurring.

[0010] Furthermore, the encoding and decoding stages exchange information through skip connections.

[0011] Furthermore, each encoder module includes multiple spatial-frequency hybrid Mamba blocks (SFMMB), and the first two encoder modules also include a downsampling layer; each decoder module includes multiple spatial-frequency hybrid Mamba blocks (SFMMB) and an upsampling layer.

[0012] Furthermore, after the input features are input into SFMMB, they are first normalized by layers, and then enter the local state module LF-VSS and the frequency domain adaptation module respectively. The local state module LF-VSS extracts spatial domain local features from the spatial domain, and the frequency domain adaptation module transforms the input features into the frequency domain, separating the amplitude information of the dominant low light and the phase information of the dominant blur to obtain frequency domain features. Then, the spatial domain local features and frequency domain features are initially fused through a convolutional layer. The fused features enter the spatial domain adaptation module. The output after processing by the spatial domain adaptation module is adjusted by a convolutional projection layer and connected with the input features in the first residual connection. Then, it is processed by layer normalization and feedforward network FFN before output, and connected with the result of the first residual connection in the second residual connection to form the final output.

[0013] Furthermore, in the encoding stage, the SFMMB in the encoder module is guided to focus on low-light enhancement by calculating the low-light enhancement loss; in the decoding stage, the SFMMB in the decoder module is guided to focus on deblurring by calculating the deblurring loss.

[0014] Furthermore, the frequency domain adaptation module uses Fourier transform to convert spatial domain features to the frequency domain. After the frequency domain transformation, the complex spectrum is decomposed into two independent components:

[0015]

[0016]

[0017] in, Input characteristics for the frequency domain adaptation module; The amplitude spectrum represents the intensity information of each frequency component in the frequency domain; This represents the Fourier transform operator; The phase spectrum represents the phase information of each frequency component in the frequency domain.

[0018] For the amplitude component, the amplitude update follows the formula:

[0019]

[0020] in, This indicates the updated amplitude spectrum; This represents the processing result obtained through the amplitude processing network; These are learnable weight parameters used to control the strength of amplitude adjustments;

[0021] For the phase component, the phase adjustment is calculated based on the amplitude information:

[0022]

[0023] in, This represents the updated phase spectrum; This represents the processing result obtained through the phase processing network; These are learnable weight parameters used to control the magnitude of phase adjustment.

[0024] The modified amplitude and phase are recombine into a complex spectrum, which is then returned to the spatial domain via an inverse Fourier transform, and then... After post-processing of the convolution, the output is connected to the input features via residual connections. Add:

[0025]

[0026] in, This indicates the output characteristics of the frequency domain adaptation module; This represents the inverse Fourier transform operator; These are learnable residual weight parameters.

[0027] Furthermore, the amplitude processing network employs multi-layer convolutions to learn and adjust the amplitude information of the spectrum, including two... Convolutional layers and the LeakyReLU activation function between layers.

[0028] Furthermore, the phase processing network uses three consecutive convolutional layers, each followed by a Tanh activation function, to limit the output range to [-1, 1].

[0029] Furthermore, in the spatial domain adaptation module, the input features are segmented into low-frequency and high-frequency features through a segmentation layer. Firstly, using... Average pooling performs initial smoothing while keeping the feature map size constant, and then applies... The mean filter smooths the features to obtain the low-frequency features; finally, the low-frequency features are subtracted from the input features to obtain the high-frequency features.

[0030] Low-frequency processing network, using Large convolutional kernels are used to capture large-scale spatial structure features, followed by two... Convolutional layers refine features, and all convolutional layers use the LeakyReLU activation function; a learnable weight is used to control the intensity of low-frequency enhancement; the overall process is as follows:

[0031]

[0032] in, This indicates the low-frequency characteristics enhanced by the low-frequency processing network. This represents a low-frequency processing network function. These are learnable weight parameters used to control the intensity of low-frequency enhancement;

[0033] High-frequency processing networks first use depthwise separable convolutions to maintain position sensitivity, and then... Pointwise convolutions are used for feature fusion, and finally dilated convolutions are used to enhance edge detection capabilities; a learnable weight is used to control the intensity of high-frequency enhancements; the overall process is as follows:

[0034]

[0035] in, This indicates the high-frequency features enhanced by the high-frequency processing network. This represents a high-frequency processing network function. These are learnable weight parameters used to control the intensity of high-frequency enhancement;

[0036] The processed high- and low-frequency features are combined through channel concatenation, processed by a feature fusion network, and the output is combined with the input features through a residual connection mechanism. Add:

[0037]

[0038] in, This indicates the output of the spatial adaptation module; Indicates the feature fusion network function; Indicates a channel splicing operation; This represents the input characteristics of the spatial domain adaptation module.

[0039] The present invention also provides a computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, wherein when the processor executes the computer program instructions, it can implement the steps of any of the methods described above.

[0040] Compared to existing technologies, this invention offers the following advantages: The present invention provides a low-light deblurring network (SFFMNet) based on frequency domain physical characteristics and spatial-frequency coordination. The core innovation of this invention lies in its use of a frequency domain degradation source decoupling mechanism. This method utilizes Fourier transform to decompose the image and, based on the physical characteristics of amplitude spectrum dominating illumination and phase spectrum dominating structure, effectively separates low-light degradation and blur degradation at the frequency domain source. To address the problem of accurate reconstruction of frequency domain information in the spatial domain, a spatial-frequency domain dual-branch collaborative architecture is constructed. This architecture focuses on global illumination restoration and local detail enhancement through explicitly distinguished low-frequency and high-frequency processing branches in the spatial domain, respectively, and adaptively bridges the advantages of frequency domain decoupling and spatial domain perception capabilities using a dynamic feature fusion mechanism. Simultaneously, to handle different degradation types in the restoration process, a task-oriented decoupling optimization strategy is designed. This strategy activates amplitude adjustment and low-frequency optimization during the encoding stage to prioritize low-light enhancement, and activates phase reconstruction and high-frequency enhancement during the decoding stage to specifically handle blur removal. Attached Figure Description

[0041] Figure 1 This is the network model architecture of the method of the present invention.

[0042] Figure 2 This invention relates to the network architecture of the Spatial Frequency Domain Hybrid Mamba Block (SFMMB).

[0043] Figure 3 This is the network architecture of the frequency domain adaptation module (FA) of the present invention.

[0044] Figure 4 This is the network architecture of the spatial adaptation module (SA) of the present invention.

[0045] Figure 5 Qualitative comparison of low-light deblurring algorithms on the LOL-Blur dataset (synthetic samples from the test set). Detailed Implementation

[0046] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0047] This invention provides a spatial-frequency domain collaborative low-light deblurring method based on Fourier transform and Mamba architecture, comprising:

[0048] Feature extraction stage: The input degraded image is processed through a single 3x3 convolutional layer to extract image patches and embed them into features, which are then input into the encoding stage;

[0049] Encoding stage: Image patch embedded features are enhanced for low light through a three-layer encoder module, and the low light enhanced features are input into the decoding stage;

[0050] Decoding stage: The low-light enhanced features are deblurred through a two-layer decoder module to output a clean image with low-light deblurring.

[0051] The following is a detailed implementation process of the present invention.

[0052] 1. Backbone Network

[0053] The Spatial-Frequency Fusion Network with Fourier and Mamba Integration for Low-Light Deblurring (SFFMNet) proposed in this invention is as follows: Figure 1 As shown, the encoder stage mainly corresponds to the low-light enhancement task, and the decoder stage mainly corresponds to the deblurring task. Skip connections are used to efficiently exchange information between the two. Given a degraded image as input... First, through a core layer Convolution, i.e., image embedding layer, extracts image patch embeddings. ,in This is the scaling factor. After that, The signal is fed into a three-layer encoder module to perform low-light enhancement. Each encoder module contains multiple Spatial-Frequency Domain Mixed Mamba Blocks (SFMBs) and a downsampling layer with a scaling factor of 2 (the last encoder module does not have a downsampling layer).

[0054] like Figure 2As shown, SFMMB utilizes multi-module collaboration to effectively achieve adaptive decoupling of the low-light and blur tasks, improving the model's feature extraction capabilities. After input features enter the SFMMB module, they undergo layer normalization before entering the Local State Module (LF-VSS) and the proposed Frequency Adaptor Module (FA). LF-VSS extracts local features from the spatial domain, capturing fine-grained positional information such as edges and textures in the image. FA transforms the features to the frequency domain, explicitly separating the amplitude information of the dominant low-light and the phase information of the dominant blur, completing the initial decoupling of degradation factors. Subsequently, the spatial local features output by LF-VSS and the frequency domain features processed by FA are initially fused using a convolutional layer. The fused features enter the proposed Spatial Adaptor Module (SA), which further enhances the structural information that is difficult to fully retain in frequency domain analysis, compensating for the limitations of frequency domain processing through the feature perception capabilities of the spatial domain. The output processed by SA is adjusted by a convolutional projection layer, and then undergoes the first residual connection with the original input of the corresponding SFMMB module. The result of this connection is then processed through layer normalization and a feedforward network (FFN), and finally joined with the result of the aforementioned residual connection (i.e., the input of the layer normalization layer) in a second residual connection to form the final output of the module. To introduce low-light enhancement supervision, after the third encoder layer is processed, the model extracts deep features... That is, the output features of the third-layer encoder are used to predict low-resolution normal light images. and utilize clean images The low-resolution image obtained after downsampling processing As an additional input, calculate the low-light enhancement loss, i.e. Figure 1 loss function in The network is guided to focus on low-light enhancement during the encoding stage. Subsequently, a similar SFMMB block is used in the decoder stage to remove blur. The decoder structure is the same as the encoder, except that the encoder focuses on enhancing the amplitude component in the frequency domain adaptation module and the low-frequency sharing in the spatial domain adaptation module, while the decoder focuses on enhancing the phase and high-frequency components. Features are progressively upsampled to the original image size, and the encoder features are passed to the corresponding decoder layers through skip connections to enhance feature representation capabilities. Finally, the image is restored. By inputting a degraded image The result is obtained by adding it to the upsampling result and comparing it with the clean image. Calculate the final loss.

[0055] This frequency-spatial domain co-decoupling and fusion mechanism enables the model to effectively handle both low-light and blur degradation. Even if the supervision signal fails to perfectly decouple the optimization objectives of the two degradations, the designed module still exhibits sufficient robustness, capable of focusing on one degradation while appropriately handling the other.

[0056] 2. Frequency Domain Adaptation Module (FA)

[0057] For frequency domain adaptation modules, such as Figure 3 As shown, inspired by the fact that amplitude primarily represents illumination information and phase primarily represents structural information, this module utilizes Fourier transform to convert spatial domain features to the frequency domain. Then, it independently learns and adjusts the amplitude and phase spectra, finally returning to the spatial domain via inverse transform. This design effectively captures the global frequency characteristics of the image while maintaining sensitivity to local details. Specifically, the input features are first preprocessed through layer normalization, and then a two-dimensional fast Fourier transform is used to convert the layer-normalized spatial domain features to the frequency domain. After the frequency domain transformation, the complex spectrum is decomposed into two independent components:

[0058]

[0059]

[0060] in, For the input features of the module; The amplitude spectrum represents the intensity information of each frequency component in the frequency domain; This represents the Fourier transform operator; Let be the phase spectrum, representing the phase information of each frequency component in the frequency domain. For the amplitude component, since it contains global information, this method introduces learnable weight parameters and optimizes the global information through global updates. The amplitude update follows the formula:

[0061]

[0062] in, This indicates the updated amplitude spectrum; This represents the processing result obtained through the amplitude processing network; These are learnable weight parameters used to control the strength of amplitude adjustment. The amplitude network uses multi-layer convolutions to learn and adjust the amplitude information of the spectrum, containing two... Convolutional layers and inter-layer LeakyReLU activation function (negative slope set to 0.1). This design allows the network to adaptively learn appropriate amplitude adjustment magnitudes, which can enhance important frequency components while avoiding information loss due to over-modification.

[0063] For the phase component, large phase changes can lead to a significant loss of image structural information. To avoid significant image structural distortion, the proposed method employs an explicit constraint strategy to minimize the degree of phase feature change. The phase processing network uses three consecutive convolutional layers, each followed by a Tanh activation function to limit the output range to [-1, 1]. Simultaneously, the method introduces small initial weight parameters to strictly control the magnitude of the phase adjustment. Notably, the calculation of the phase adjustment amount is based on magnitude information:

[0064]

[0065] in, This represents the updated phase spectrum; This represents the processing result obtained through the phase processing network; It is a learnable weight parameter, initialized to 0.1, used to control the magnitude of phase adjustment.

[0066] After processing, the modified amplitude and phase are recombine into a complex spectrum, which is then returned to the spatial domain via an inverse Fourier transform, and then... Convolution is followed by post-processing. The final output is added to the input features via residual connections:

[0067]

[0068] in, This represents the characteristics of the final module output; This represents the inverse Fourier transform operator; These are learnable residual weight parameters, which ensures the stability of network training and the effective propagation of gradients.

[0069] 3. Spatial Adaptation Module (SA)

[0070] like Figure 4 As shown, to compensate for the shortcomings of frequency domain processing, a spatial domain adaptation module was designed to further separate and process different frequency components in the spatial domain. The design concept of this module is to achieve the separation and processing of high and low frequency features by using spatial domain filters, thereby achieving frequency-selective processing while maintaining computational efficiency.

[0071] In the spatial domain adaptation module, the input features need to be separated into high-frequency and low-frequency features for separate processing. This is achieved by using a segmentation layer to divide the input features into low-frequency and high-frequency features. Average pooling performs initial smoothing while keeping the feature map size constant, and then applies... The mean filter smooths the features to obtain low-frequency features; this multi-layer smoothing strategy can effectively extract the overall structural information of the image while suppressing high-frequency noise and details. The low-frequency features are then subtracted from the input features to obtain the high-frequency features.

[0072] Low-frequency processing networks primarily serve the needs of structural information recovery, employing... Large convolutional kernels are used to capture large-scale spatial structure features, followed by two... Convolutional layers refine features, and all convolutional layers use the LeakyReLU activation function to maintain feature continuity. Furthermore, the method utilizes a learnable weight to control the intensity of low-frequency enhancements. The overall process is as follows:

[0073]

[0074] in, This indicates the enhanced low-frequency characteristics. This represents a low-frequency processing network function. These are learnable weight parameters used to control the intensity of low-frequency enhancement.

[0075] High-frequency processing networks are specifically optimized for processing detailed information. The network first uses depthwise separable convolutions to maintain position sensitivity, and then... Pointwise convolutions are used for feature fusion, and finally, dilated convolutions (with a dilation rate of 2) are used to enhance edge detection capabilities. Similarly, an additional learnable weight is used to control the intensity of high-frequency enhancements.

[0076]

[0077] in, This indicates the enhanced high-frequency features. This represents a high-frequency processing network function. These are learnable weight parameters used to control the intensity of high-frequency enhancement.

[0078] Finally, the processed high- and low-frequency features are combined through channel concatenation and then processed by a fusion network. The final output is added to the original input via a residual connection mechanism.

[0079]

[0080] in, Indicates the final output; This represents the feature fusion network function. This indicates a channel splicing operation.

[0081] 4. Task decoupling strategy

[0082] To adjust processing strategies based on the physical characteristics of different tasks, this invention designs a task-oriented decoupling mechanism. This mechanism, based on the different degradation mechanisms of low-light enhancement and deblurring tasks, decomposes frequency domain processing into targeted processing branches, achieving task-specific frequency domain optimization. In the low-light enhancement stage, the frequency domain adaptation module focuses on amplitude spectrum processing, improving image brightness and contrast by learning and adjusting the spectral amplitude, while maintaining phase information to ensure image structural integrity. The spatial domain adaptation module activates the low-frequency processing branch, focusing on enhancing key structural information, brightness distribution, and large-scale texture features. This combined strategy effectively improves low-light image quality and avoids amplifying high-frequency noise. In the deblurring stage, the frequency domain adaptation module focuses on phase processing, reconstructing edge structures and geometric features by restoring phase information, while maintaining the amplitude spectrum to preserve the image's frequency intensity distribution. The spatial domain adaptation module activates the high-frequency processing branch, specifically recovering lost details and edge information. This targeted design effectively removes blurring effects and restores image sharpness. The entire decoupling mechanism can be represented by the following task-oriented mapping relationship:

[0083] In low-light enhancement mode:

[0084]

[0085] In deblurring mode:

[0086]

[0087] in, This indicates the output characteristics of the SFMMB module. This indicates the amplitude processing branch in the frequency domain adaptation module; This indicates the phase processing branch in the frequency domain adaptation module; This indicates the low-frequency processing branch in the spatial adaptation module; This indicates the high-frequency processing branch in the spatial adaptation module; This indicates the input characteristics of the SFMMB module.

[0088] The proposed decoupling mechanism not only improves the model's task specificity but also enhances the network's interpretability. The activation of each processing branch has a clear physical meaning and theoretical basis, making the overall network behavior more predictable and controllable. Furthermore, this design provides a clear framework for future expansion to other image processing tasks; simply select the appropriate frequency domain processing combination based on the physical characteristics of the new task.

[0089] 5. Experiment G

[0090] The proposed method uses the LOL-Blur dataset to evaluate the performance of SFFMNet on the joint low-light enhancement and deblurring task. This dataset contains 170 training video sequences (10,200 image pairs) and 30 test video sequences (1,800 image pairs). To comprehensively evaluate the effectiveness of the proposed method, multiple end-to-end models for joint low-light and blur removal are selected. To ensure fairness, the experimental evaluation includes both quantitative and qualitative analyses.

[0091] Table 1. Quantitative comparison of low-light deblurring algorithms on the LOL-Blur dataset (synthetic samples from the test set).

[0092]

[0093] 5.1 Quantitative comparison

[0094] Table 1 presents the quantitative comparison results on the LOL-Blur dataset. SFFMNet achieves the best overall performance, reaching a PSNR of 28.23 dB, an SSIM of 0.880, and an LPIPS of 0.179. Compared to the second-best methods, SFFMNet improves PSNR by 0.44 dB and SSIM by 0.001, while maintaining similar perceptual quality. Notably, SFFMNet has only 11.4M parameters and a computational cost of 45.78G, with computational complexity lower than most existing methods, fully demonstrating the performance and efficiency advantages of the frequency domain collaborative processing strategy.

[0095] 5.2 Qualitative Comparison

[0096] Figure 5 Qualitative comparison results are presented on synthetic test samples of the LOL-Blur dataset. Existing methods often suffer from problems such as mutual interference of degradation factors, uneven brightness, or blur residue when dealing with compound degradation. In contrast, SFFMNet fully leverages the global modeling advantages of frequency domain processing while supplementing local detail information through spatial domain processing, achieving a complementary advantage of the two processing methods. From magnified local areas, it can be clearly observed that SFFMNet is closer to the ground truth image in terms of text sharpness, edge sharpness, and overall visual quality, verifying the innovation and effectiveness of the frequency domain decoupling and spatial-frequency domain collaborative processing strategy in the joint low-light image enhancement and deblurring task.

[0097] The present invention also provides a computer-readable storage medium having stored thereon computer program instructions executable by a processor, wherein when the processor executes the computer program instructions, it can implement the steps of any of the methods described above.

[0098] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for low-light deblurring in spatial-frequency domain based on Fourier transform and Mamba architecture, characterized in that, The method comprises the following steps: The feature extraction stage: the input degraded image is extracted by a layer of convolution with a 3*3 kernel to obtain image block embedding features, and the image block embedding features are input into the encoding stage; The encoding stage: the image block embedding features are input into three layers of encoder modules for low-light enhancement, and the low-light enhanced features are input into the decoding stage; The decoding stage: the low-light enhanced features are input into two layers of decoder modules for deblurring, and a clean image after low-light deblurring is output.

2. The Fourier transform and Mamba architecture based spatio-temporal collaborative low-light deblurring method according to claim 1, characterized in that, The encoding stage and the decoding stage exchange information through a skip connection.

3. The Fourier transform and Mamba architecture based spatio-temporal collaborative low-light deblurring method according to claim 1, wherein, Each layer of the encoder module comprises a plurality of spatial-frequency domain hybrid Mamba blocks SFMMB, and the first two layers of the encoder module each further comprises a down-sampling layer; each layer of the decoder module comprises a plurality of SFMMB and an up-sampling layer.

4. The Fourier transform and Mamba architecture based spatio-temporal collaborative low-light deblurring method according to claim 3, characterized in that, After the input features are input into the SFMMB, layer normalization is performed, and then the input features are respectively input into a local state module LF-VSS and a frequency domain adaptation module, wherein the local state module LF-VSS extracts spatial domain local features from a spatial domain, the frequency domain adaptation module converts the input features to a frequency domain, separates amplitude information of dominant low light and phase information of dominant blur, and obtains frequency domain features, and then the spatial domain local features and the frequency domain features are preliminarily fused through a layer of convolution, the fused features are input into a spatial domain adaptation module, the output of the spatial domain adaptation module is adjusted by a convolution projection layer, and the output is connected with the input features for the first time, then the output is processed by layer normalization and a feedforward network FFN, and the output is connected with the result of the first time for the second time, and finally the output is obtained.

5. The Fourier transform and Mamba architecture based spatio-temporal collaborative low-light deblurring method according to claim 4, characterized in that, The encoding stage guides the SFMMB in the encoder module to focus on low-light enhancement in the encoding stage by calculating a low-light enhancement loss; and the decoding stage guides the SFMMB in the decoder module to focus on deblurring in the decoding stage by calculating a deblurring loss.

6. The Fourier transform and Mamba architecture based spatio-temporal collaborative low-light deblurring method according to claim 4, characterized in that, The frequency domain adaptation module converts spatial domain features to a frequency domain by using Fourier transform, and after the frequency domain conversion, the complex spectrum is decomposed into two independent components: wherein, is an input feature of the frequency domain adaptation module; is a magnitude spectrum representing intensity information of frequency components in the frequency domain; denotes a Fourier transform operator; is a phase spectrum representing phase information of frequency components in the frequency domain; For the amplitude component, the amplitude update follows the following formula: wherein, represents the updated magnitude spectrum; represents a processing result obtained through the magnitude processing network; is a learnable weight parameter for controlling the strength of the magnitude adjustment; For the phase component, the calculation of the phase adjustment amount is based on the amplitude information: wherein, represents the updated phase spectrum; represents the processing result obtained by the phase processing network; is a learnable weight parameter for controlling the magnitude of the phase adjustment. The modified amplitudes and phases are recombined into a complex spectrum, which is then returned to the spatial domain by an inverse Fourier transform and post-processed by a convolution with the output features being added to the input features through a residual connection addition: wherein, denotes the output characteristic of the frequency domain adaptation module; denotes an inverse Fourier transform operator; are learnable residual weight parameters.

7. The Fourier transform and Mamba architecture based spatio-temporal collaborative low-light deblurring method according to claim 6, characterized in that, The amplitude processing network adopts multiple layers of convolution to learn and adjust the amplitude information of the spectrum, including two convolution layers and inter-layer LeakyReLU activation functions.

8. The Fourier transform and Mamba architecture based spatio-temporal collaborative low-light deblurring method according to claim 6, characterized in that, The phase processing network uses three consecutive convolution layers, each of which is connected with a Tanh activation function to limit the output range to [-1, 1].

9. The Fourier transform and Mamba architecture based spatio-temporal collaborative low-light deblurring method according to claim 4, characterized in that, In the spatial adaptation module, the input features are divided into low-frequency features and high-frequency features by a split layer, and first, the low-frequency features are smoothed by a mean filter The average pooling operation keeps the feature map size unchanged while performing preliminary smoothing, and then applies The mean filter smooths the features to obtain low-frequency features; Finally, the low-frequency features are subtracted from the input features to obtain high-frequency features. Low-frequency processing network, which adopts large convolution kernels to capture large-scale spatial structure features, and then through two convolution layers for feature refinement, all convolution layers use LeakyReLU activation function; use a learnable weight to control the strength of low-frequency enhancement; The overall process is as follows: wherein, represents a low frequency feature enhanced by a low frequency processing network, represents a low frequency processing network function, is a learnable weight parameter for controlling the strength of low frequency enhancement; High-frequency processing networks first use depthwise separable convolutions to maintain position sensitivity, and then... Pointwise convolutions are used for feature fusion, and finally dilated convolutions are used to enhance edge detection capabilities; a learnable weight is used to control the intensity of high-frequency enhancements; the overall process is as follows: wherein, represents a high frequency feature that is processed by a high frequency processing network enhancement, represents a high frequency processing network function, is a learnable weight parameter for controlling the strength of the high frequency enhancement; The processed high-frequency and low-frequency features are combined through channel splicing, processed through a feature fusion network, and output through a residual connection mechanism and input features addition: wherein, denotes the output of the spatial adaptation module; denotes the feature fusion network function; denotes the channel concatenation operation; denotes the input feature of the spatial adaptation module.

10. A computer readable storage medium having stored thereon computer program instructions capable of being executed by a processor, when the processor executes the computer program instructions, steps of the method according to any one of claims 1-9 can be implemented.

Images