Image enhancement method and system based on multi-scale cross attention and frequency domain enhancement

Abstract

The present application relates to the technical field of machine vision and image processing, in particular to an image enhancement method and system based on multi-scale cross attention and frequency domain enhancement, comprising: for low-light images, extracting multi-scale illumination features through the constructed MSLPM module, preliminarily estimating the illumination distribution, obtaining the illumination feature map and the illumination estimation map, and combining the low-light images and the illumination estimation map to generate an input feature map; the input feature map adopts the constructed encoder-decoder architecture to perform feature fusion and image reconstruction to generate an enhanced image; the feature fusion and image reconstruction include: performing frequency-space dual-domain feature enhancement, through a plurality of constructed FSAB blocks, preliminarily extracting the feature map from the input feature map to perform local aggregation, global correlation, nonlinear transformation and frequency domain enhancement, and obtaining deep features; wherein the global correlation adopts the illumination feature map as a guide signal. The present scheme can improve the enhancement effect, noise robustness and model interpretability.

Description

Technical Field

[0001] This invention relates to the field of machine vision and image processing technology, specifically to an image enhancement method and system based on multi-scale cross-attention and frequency domain enhancement. Background Technology

[0002] Low-light image enhancement is a key research branch in image processing and computer vision, with wide-ranging applications in numerous real-world scenarios, such as industrial visual inspection, nighttime security monitoring, autonomous driving perception, medical image analysis, mobile photography, and underwater imaging. In poorly lit environments, especially when image sensor size is limited, the light signals received by the sensor are typically very weak and introduce significant noise interference, leading to a decline in the visual quality of the resulting images. To recover high-quality visual information from these degraded images, low-light image enhancement techniques are needed to improve image visual quality, address common problems such as insufficient visibility and low contrast, and repair various degradation phenomena caused by insufficient lighting or enhancement processing, including noise, artifacts, and color distortion.

[0003] Currently, various algorithms for low-light image enhancement have emerged, which can be mainly classified into two technical categories: one based on traditional image processing theory, and the other based on deep learning technology. Traditional low-light image enhancement methods are usually based on direct mapping and adjustment of pixels. Their process is simple, fast, and resource-efficient, thus they were widely used in early image enhancement practices. For example, histogram equalization improves contrast by adjusting the distribution of image pixels. Common implementations include standard histogram equalization (HE), adaptive histogram equalization (AHE), and contrast-limited adaptive histogram equalization (CLAHE). Gamma correction is an image processing method based on nonlinear mapping, which effectively improves the brightness of dark areas by transforming pixel values. Enhancement algorithms based on Retinex theory, based on the color constancy characteristics of the human visual system, mimic the human eye's perception of object colors, decomposing the image into two independent components: illumination and reflection, thereby effectively improving image quality. Representative methods within this theoretical framework include single-scale Retinex (SSR), multi-scale Retinex (MSR), and multi-scale Retinex (MSRCR) with color restoration capabilities. These methods can adaptively adjust illumination based on image content while preserving detailed features.

[0004] In recent years, deep learning technology has made significant progress in the field of low-light image enhancement. Benefiting from its powerful nonlinear modeling capabilities and flexible end-to-end learning framework, deep learning-based methods have gradually become the mainstream research direction in this field. LLNet pioneered the introduction of deep learning into this task, employing a sparse autoencoder architecture to simultaneously enhance image brightness and suppress noise. Retinex-Net, based on Retinex theory, uses an encoder-decoder network to decompose an image into two components: reflection and illumination, and then uses an enhancement network to improve illumination conditions. Zero-DCE first proposed an enhancement network that does not require paired data, transforming the enhancement task into a process of learning image correlation curves, achieving iterative enhancement through pixel-by-pixel curve adjustments. KinD adopts a modular design, removing noise and color cast through a reflectivity recovery module and learning illumination mapping relationships through an illumination adjustment module to achieve adjustable illumination transformation. EnlightenGAN uses U-Net as the generator and introduces two discriminators, global and local, to collaboratively learn feature representations at different levels. URetinex-Net designs three learnable modules responsible for data initialization, noise suppression and detail optimization, and illumination adjustment supporting user interaction. Retinexformer pioneered a single-stage Transformer architecture based on Retinex theory, proposing a lighting-guided self-attention mechanism that establishes long-range dependencies using lighting information as a clue. Retinexmamba addresses the limitations of Retinexformer in interpretability and computational efficiency by introducing the Mamba structure to replace the original multi-head attention module, improving both interpretability and computational efficiency while maintaining performance.

[0005] While existing low-light image enhancement techniques have made some progress, they still face numerous technical limitations and challenges in practical applications. Traditional image processing-based methods, such as histogram equalization and gamma correction, often only improve overall brightness through simple pixel-level mapping. These methods tend to ignore the image's intrinsic structure and illumination distribution characteristics during enhancement, leading to problems such as over-enhancement in local areas, artifact generation, and color distortion. They struggle to effectively maintain the image's naturalness and detail while improving brightness. Algorithms based on Retinex theory, while showing advantages in simulating human visual characteristics by decomposing images into illumination and reflection components for separate processing, lack strict mathematical uniqueness in their decomposition process, often suffering from halo effects and color deviations in practical applications. Furthermore, most Retinex-based methods rely on manually set prior constraints or complex iterative optimization processes, resulting in significant computational burdens.

[0006] The emergence of deep learning methods in recent years has brought new solutions to low-light image enhancement, but existing deep learning-based techniques still have significant shortcomings. On the one hand, many models, such as Retinex-Net and KinD, have not fully considered the complexity of multi-scale illumination changes in their feature extraction and fusion mechanisms, and have limited ability to collaboratively model local details and global structure, making it difficult to achieve a good balance between dark detail restoration and overall illumination equalization. On the other hand, existing networks, such as Retinexformer and Retinexmamba, focus mainly on spatial domain feature learning, neglecting the important role of frequency domain information in image texture representation and noise separation, thus limiting the robustness and generalization ability of the models under complex degradation conditions.

[0007] In summary, there is an urgent need for an image enhancement method and system based on multi-scale cross-attention and frequency domain enhancement, which can improve enhancement effect, noise robustness and model interpretability, and systematically solve the problems of insufficient fusion of local and global information, insufficient utilization of frequency domain features and lack of multi-scale interaction mechanism in low-light image enhancement technology. Summary of the Invention

[0008] One of the objectives of this invention is to provide an image enhancement method based on multi-scale cross-attention and frequency domain enhancement, which can improve enhancement effect, noise robustness and model interpretability, and systematically solve the problems of insufficient fusion of local and global information, insufficient utilization of frequency domain features and lack of multi-scale interaction mechanism in low-light image enhancement technology.

[0009] The basic solution provided by this invention is an image enhancement method based on multi-scale cross-attention and frequency domain enhancement, comprising: Multi-scale illumination perception and preliminary estimation steps: For low-light images, multi-scale illumination features are extracted through the constructed MSLPM module to perform preliminary estimation of illumination distribution, obtain preliminary illumination feature maps and illumination estimation maps, and combine the low-light image and illumination estimation map to generate input feature maps; Feature fusion and image reconstruction steps: The input feature map is used to construct an encoder-decoder architecture to perform feature fusion and image reconstruction, generating an enhanced image; The feature fusion and image reconstruction process includes: performing frequency-space dual-domain feature enhancement to obtain depth features; the frequency-space dual-domain feature enhancement is performed through several constructed FSAB blocks, which perform local aggregation, global association, nonlinear transformation, and frequency domain enhancement on the preliminary feature map extracted from the input feature map to obtain depth features; the global association uses the illumination feature map as a guiding signal.

[0010] The second objective of this invention is to provide an image enhancement system based on multi-scale cross-attention and frequency domain enhancement, which can improve the enhancement effect, noise robustness and model interpretability, and systematically solve the problems of insufficient fusion of local and global information, insufficient utilization of frequency domain features and lack of multi-scale interaction mechanism in low-light image enhancement technology.

[0011] This invention provides a second basic scheme: an image enhancement system based on multi-scale cross-attention and frequency domain enhancement, which enhances images through a constructed MCFE-Net network, including: The MC-EPA module is used to extract multi-scale illumination features from low-light images through the constructed MSLPM module, perform preliminary estimation of illumination distribution, obtain preliminary illumination feature maps and illumination estimation maps, and combine the low-light image and illumination estimation map to generate the input feature map. Construct an encoder-decoder architecture to perform feature fusion and image reconstruction on the input feature map, generating an enhanced image; The feature fusion and image reconstruction process includes: performing frequency-space dual-domain feature enhancement to obtain depth features; the frequency-space dual-domain feature enhancement is performed through several constructed FSAB blocks, which perform local aggregation, global association, nonlinear transformation, and frequency domain enhancement on the preliminary feature map extracted from the input feature map to obtain depth features; the global association uses the illumination feature map as a guiding signal.

[0012] Beneficial effects: This scheme extracts multi-scale illumination features from low-light images through the constructed MSLPM module (multi-scale illumination sensing module), performs preliminary estimation of illumination distribution, and obtains preliminary illumination feature maps and illumination estimation maps, which serve as inputs for subsequent steps and as guiding signals; and combines the low-light image and illumination estimation map to generate an input feature map, which serves as the basis for subsequent reconstruction. The input feature map is used to construct an encoder-decoder architecture for feature fusion and image reconstruction to generate an enhanced image. The feature fusion and image reconstruction include: performing frequency-space dual-domain feature enhancement to obtain depth features. The frequency-space dual-domain feature enhancement is performed through several constructed FSAB blocks, which perform local aggregation, global association, nonlinear transformation, and frequency domain enhancement on the preliminary feature map extracted from the input feature map to obtain depth features. The global association uses the illumination feature map as a guiding signal. By constructing several FSAB blocks, the feature maps of the input FSAB blocks are subjected to local aggregation, global association, nonlinear transformation, and frequency domain enhancement to obtain deep features. Among them, local aggregation can adaptively enhance local details under the guidance of the global context. Global association, combined with the illumination estimation map, can more accurately identify and enhance pixels belonging to homogeneous illumination regions, thereby improving contrast while effectively suppressing halo artifacts that may be caused by abrupt changes in illumination. Nonlinear transformation and frequency domain enhancement provide a completely different and complementary processing dimension for separating noise and enhancing texture compared to traditional spatial domain methods. After illumination guidance and local-global interactive optimization, the frequency domain processing here is performed on a more accurate and semantically rich intermediate representation. This deep and guided frequency domain processing can more effectively separate low-frequency noise related to illumination and structure from useful high-frequency texture, which is different from universal transformations on the original or shallow features.

[0013] Unlike the traditional Retinex method's two-stage pipeline of explicit decomposition followed by separate enhancements, and unlike some deep learning models that simply stitch together illumination maps at the input, this approach constructs an end-to-end network that deeply couples illumination estimation and image enhancement. Using the illumination estimation from the MSLPM module as a global guiding signal, it is embedded into the subsequent feature refinement stage to drive bidirectional interaction between local detail restoration and global semantic enhancement. In this interaction process, frequency domain and spatial domain information are simultaneously integrated for joint optimization, forming an integrated process from illumination estimation to interactive refinement and then to dual-domain enhancement.

[0014] This solution effectively improves the overall brightness and contrast of low-light images, while significantly suppressing noise and restoring detailed textures; it achieves adaptive modeling and correction of uneven lighting in images, avoiding local overexposure or underexposure; and it fully utilizes the complementary information in the frequency and spatial domains of images to improve the robustness and visual quality of the enhancement process.

[0015] In summary, this scheme can improve the enhancement effect, noise robustness and model interpretability, and systematically solve the problems of insufficient fusion of local and global information, insufficient utilization of frequency domain features and lack of multi-scale interaction mechanism in low-light image enhancement technology. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating an embodiment of the image enhancement method based on multi-scale cross-attention and frequency domain enhancement of the present invention. Figure 2 This is a flowchart illustrating the MSLPM module in an embodiment of the image enhancement method based on multi-scale cross-attention and frequency domain enhancement of the present invention. Figure 3This is a flowchart illustrating the MC-EPA submodule in an embodiment of the image enhancement method based on multi-scale cross-attention and frequency domain enhancement of the present invention. Figure 4 This is a flowchart illustrating the LCA submodule in an embodiment of the image enhancement method based on multi-scale cross-attention and frequency domain enhancement of the present invention. Figure 5 This is a flowchart illustrating the DFE submodule in an embodiment of the image enhancement method based on multi-scale cross-attention and frequency domain enhancement of the present invention. Figure 6 This is a comparison of the visual effects of image enhancement on the LOL-v1 dataset in an embodiment of the image enhancement method based on multi-scale cross-attention and frequency domain enhancement of the present invention; Figure 7 This is a comparison of the visual effects of image enhancement on the LOL-v2-real dataset in an embodiment of the image enhancement method based on multi-scale cross-attention and frequency domain enhancement of the present invention; Figure 8 This is a comparison of the visual effects of image enhancement on the LOL-v2-syn dataset in the embodiment of the image enhancement method based on multi-scale cross-attention and frequency domain enhancement of the present invention. Detailed Implementation

[0017] The following detailed description illustrates the specific implementation method: The markings in the accompanying drawings include: Example 1 This embodiment provides an image enhancement method based on multi-scale cross-attention and frequency domain enhancement, as shown in the attached figure. Figure 1 As shown, it includes the following: Multi-scale illumination sensing and preliminary estimation steps: For low-light images, the constructed MSLPM module (multi-scale illumination sensing module) is used to extract multi-scale illumination features, perform preliminary estimation of illumination distribution, obtain preliminary illumination feature map and illumination estimation map, and combine the low-light image and illumination estimation map to generate input feature map; The MSLPM module includes: a feature reduction layer, an MC-EPA submodule, a depthwise separable convolutional layer, and a convolutional layer; like Figure 2 As shown, the specific process is as follows: S101. The low-light image is input into the constructed MSLPM module. The input feature is compressed by the feature dimensionality reduction layer to extract the input features and reduce the amount of computation. In this embodiment, the feature dimensionality reduction layer adopts a 1×1 convolutional layer. S102. Input the input features into the MC-EPA submodule (multi-scale cross-enhanced attention submodule) in the MSLPM module to perform multi-scale cross-enhanced enhancement and obtain intermediate features; like Figure 3 The MC-EPA module includes: a multi-scale feature extraction layer, a cross-attention fusion layer, and a multilayer perceptron. The specific process is as follows: The multi-scale feature extraction layer in the S10201 and MC-EPA submodules processes the input features in parallel using three sets of parallel dilated convolutions (with dilation rates of 1, 2, and 3, respectively), extracting spatial contextual information features under three different receptive fields (small, medium, and large) as output features. Considering the multi-scale characteristics of illumination changes, this module adopts a parallel multi-branch structure to extract illumination features under different receptive fields. The input feature is X, and the output features at the three scales are calculated as follows: in, These represent the output features at small, medium, and large scales, respectively. Represents convolution. These three scale features represent the hole rate, and correspond to the local detail, intermediate region correlation, and global illumination trend of the image, respectively.

[0018] The cross-attention fusion layer in the S10202 and MC-EPA submodules employs a cross-attention fusion mechanism based on input features. Attention weights are generated for the output features at each scale of the multi-scale feature extraction layer, and the output features at each scale are weighted to generate weighted multi-scale features. Features at different scales are complementary; large-scale features provide illumination distribution trends, while small-scale features preserve detailed information. To promote information exchange between scales and adaptively fuse features at different scales, a cross-attention mechanism is set up to generate attention weights for each scale feature by learning the interdependencies between scales. in, , For global average pooling, For activation functions; The attention weights are divided into three attention maps along the channel dimension, corresponding to three scale features respectively: ; Attention weights are applied to the output features at the corresponding scale (output features of the multi-scale feature extraction layer) through element-wise multiplication to obtain weighted multi-scale features: These are multi-scale features weighted by cross-attention. This design allows the network to adaptively adjust the importance of features at each scale based on the input content, enhancing the feature representation of key illuminated regions.

[0019] The cross-attention fusion layer in the S10203 and MC-EPA submodules employs an adaptive gating fusion mechanism, based on input features. By using global context information, the output features of each scale of the multi-scale feature extraction layer are dynamically adjusted to generate fusion weights, and the weighted multi-scale features are weighted to generate fusion features. In the MC-EPA submodule, the cross-attention fusion layer learns three global scalar weights and calculates the gating weights through an adaptive gating unit based on global average pooling. : Gating weights Cut into , ; Based on gating weights The fused features are output through the multilayer perceptron in the MC-EPA submodule: ; This adaptive fusion strategy based on global context can dynamically adjust the contribution ratio of each scale according to the overall characteristics of the input image. The multilayer perceptron in the S10204 and MC-EPA submodules will fuse features. Compared with the original input features Perform residual join merging (i.e., perform nonlinear transformation and dimensionality reduction) to obtain intermediate features: ; in It is a multilayer perceptron, consisting of two 1×1 convolutions. For connection.

[0020] S103. For intermediate features, obtain illumination feature maps through the depthwise separable convolutional layer in the MSLPM module; Intermediate features enhanced by the MCEPA module The image then enters a depthwise separable convolutional layer to further refine illumination-related information. In this embodiment, a 5×5 depthwise convolution is used to output an illumination feature map. While maintaining parameter efficiency, it provides a sufficiently large receptive field to model the spatial continuity of illumination. The illumination feature map is a high-dimensional feature representation containing rich multi-scale illumination semantic information, which is used for feature guidance in subsequent FSAB blocks, providing illumination-related contextual information.

[0021] S104. Based on the illumination feature map, obtain the illumination estimation map through a 1×1 convolutional layer (for channel compression) in the MSLPM module; Among them, the illumination estimation map The illumination estimation map is directly used to adjust the input image. Its physical meaning is equivalent to the illumination component in the traditional Retinex, and it obtains a more accurate estimate through data-driven learning.

[0022] S105, and low-light image Illumination estimation diagram Combine them to generate the input feature map. : in The input is a low-light image, and the input feature map is used as input for subsequent image reconstruction.

[0023] Feature fusion and image reconstruction steps: The input feature map is used to construct an encoder-decoder architecture to perform feature fusion and image reconstruction, generating an enhanced image; The encoder-decoder architecture includes: an encoder, a decoder, and an image output layer; The encoder downsamples the input feature map to obtain a preliminary feature map, and performs frequency-space dual-domain feature enhancement to obtain depth features. The frequency-space dual-domain feature enhancement is performed through several constructed FSAB blocks (frequency-space dual-domain feature enhancement blocks) to perform local aggregation, global association, nonlinear transformation, and frequency domain enhancement on the preliminary feature map to obtain depth features. The global association uses the illumination feature map as a guiding signal. The decoder upsamples the depth features, performs skip connection fusion, and frequency-space dual-domain feature enhancement to obtain the highest resolution features; The image output layer reconstructs the image based on the highest resolution features and the input feature map, generating an enhanced image.

[0024] The aforementioned FSAB block consists of an LCA submodule (Local Context Aggregation Submodule), an IG-MSA submodule (Improved Illumination-Guided Multi-Head Self-Attention Submodule), an FFN submodule (Feedforward Network Submodule), and a DFE submodule (Frequency Domain Enhancement Submodule). Specifically, the LCA submodule is used to perform local aggregation to obtain local-global features, that is, to perform the local-global collaborative feature refinement sub-step, the specific process of which is as follows: After the initial illumination estimation, the illumination features need to be further refined to integrate local details and global semantics. To this end, this scheme introduces a frequency-space dual-domain feature enhancement block (FSAB) in the decoding path, and the first sub-module is the Local Context Aggregation Module (LCA). like Figure 4As shown, the LCA submodule adopts a dual-path design to process the illumination feature map and fuse local details with global semantics, including: local path and global path; Local paths capture spatial neighborhood information of the illumination features as input through depthwise separable convolutions, obtaining local detail features: in For local detail features, The feature map of the input FSAB block is, in this embodiment, the preliminary feature map extracted by the encoder downsampling layer or the depth feature output by the previous FSAB block; depth-separable convolution significantly reduces the number of parameters and computational complexity while maintaining the local receptive field, and its grouped convolution characteristics also enable the features of each channel to learn spatial patterns independently. In the global path, global average pooling is used to compress the spatial dimension and extract image-level global semantic features from the input illumination features: ; in To extract global semantic features, the feature map of each channel is compressed into a scalar value, forming a highly abstract global context representation. This global semantic information can provide important guidance for local recovery.

[0025] The outputs of the local and global paths (local detail features and global semantic features) are fused through a lightweight attention interaction layer to obtain local-global features; Specifically, the attention interaction layer treats local detail features as queries and global semantic features as both keys and values, and computes local-global features through a multi-head attention mechanism: ; This allows the features of each local location to be enhanced under the guidance of the global context, ensuring that the restoration of details in dark areas conforms to the overall lighting and color distribution rules.

[0026] The LCA submodule employs a dual-path design to process the feature map input from the current FSAB block. The algorithm integrates local details and global semantics. The local path captures spatial neighborhood information of the input features through depthwise separable convolutions, obtaining local detail features. The global path compresses the spatial dimension through global average pooling, extracting image-level global semantic features of the input features. The LCA submodule processes the input features of the FSAB block, i.e., the intermediate feature representation passed layer by layer during feature extraction. This feature undergoes multi-level transformations in the encoder-decoder structure, encompassing image information at different scales.

[0027] The IG-MSA submodule is used to perform global correlation of local and global features using the illumination feature map as a guiding signal to obtain weighted features. This involves executing the feature correlation modeling sub-step guided by spatial illumination, as follows: Based on local-global feature fusion, in order to further establish long-distance inter-pixel dependencies, this scheme continues to use the IG-MSA submodule (improved illumination-guided multi-head self-attention submodule) in the FSAB block. Unlike ordinary self-attention, IG-MSA uses the illumination features generated in the multi-scale illumination sensing and preliminary estimation steps as additional guiding information. When calculating attention weights, the IG-MSA submodule considers not only the similarity of the features themselves but also the illumination estimate at the corresponding location as a modulating factor, giving higher association weights to pixels in similar illumination regions. This mechanism enables the network to more accurately identify and enhance pixels belonging to homogeneous illumination regions, thereby improving contrast while effectively suppressing halo artifacts that may be caused by sudden changes in illumination. The specific process is as follows: Feature mapping sub-step: converting local features to global features Mapped to query matrices respectively Key matrix Value matrix ; ; ; ; Sub-step of the illumination adjustment value matrix: Convert the illumination feature map... AND-value matrix Perform element-wise multiplication, perform illumination-sensing feature adjustment, and obtain the illumination-adjusted value matrix. : in This step represents element-wise multiplication, which enhances the consistency of feature values ​​in regions with similar lighting conditions, providing a base value for lighting perception for subsequent attention aggregation. Attention weight calculation sub-step: Calculate the query matrix AND key matrix Similarity to obtain the attention weight matrix : in For feature dimension, This is a learnable scaling factor used to adjust the magnitude of the attention weights; Illumination-guided feature aggregation sub-step: Based on the attention weight matrix Value matrix after illumination adjustment Perform weighted aggregation to obtain aggregated features: Output fusion sub-step: For aggregated features Perform linear projection and add positional encoding to obtain weighted features, which are then output as enhanced features. in To output the projected weights, The positional encoding function is implemented using a convolutional network. The output of the IG-MSA submodule is the illumination-guided enhanced features, i.e., weighted features, which maintain the same dimensionality as the input.

[0028] The FFN submodule is used to perform nonlinear transformations on the weighted features to obtain the FFN output features. After the IG-MSA submodule performs illumination-guided feature association, the FSAB block further performs nonlinear feature transformation and enhancement through the feedforward network (FFN) submodule. The main functions of the FFN submodule are as follows: to refine the attention output and perform nonlinear transformation on the attention-weighted features; to prepare features for frequency domain enhancement and provide spatially blended features for processing by the DFE submodule.

[0029] The DFE submodule is used to process the output features of FFN. Then, frequency domain enhancement is performed to obtain deep features, that is, the frequency domain feature enhancement and noise separation sub-steps are executed, and the specific process is as follows: All three steps mentioned above are performed in the spatial domain. To overcome the limitations of spatial domain processing, this scheme introduces a frequency domain enhancement submodule (DFE) in the FSAB block, forming a collaborative scheme of spatial-frequency domain enhancement. The DFE submodule is used to transform the weighted features from the spatial domain to the frequency domain for processing, and then transform them back to the spatial domain; this process is achieved through Fast Fourier Transform, such as... Figure 5 As shown: First, perform a two-dimensional Fourier transform on the input weighted features: in This represents the Fast Fourier Transform (FFT) operation. The transformed features contain two components: a real part and an imaginary part, corresponding to amplitude and phase information in the frequency domain, respectively. This transformation converts local correlations in the spatial domain into a global representation in the frequency domain, making the periodic structure and texture patterns of the image more apparent.

[0030] In the frequency domain, low-frequency components mainly contain information about illumination distribution and overall structure, while high-frequency components correspond to edge details and texture features. However, in low-light images, high-frequency components are often more severely affected by noise. Therefore, the DFE submodule performs frequency domain processing on the real and imaginary parts separately: in and These are the processed real and imaginary components. and These represent the operations of extracting the real and imaginary parts, respectively. The 1×1 convolution enables the network to adaptively adjust the weights of different frequency components, thereby enhancing effective texture information while suppressing noise.

[0031] After frequency domain processing, the data is returned to the spatial domain via inverse Fourier transform, and then fused with the original features via residual connections to obtain deep features. in To return the characteristics of the spatial domain, This is the final output of the DFE submodule, i.e., the depth features.

[0032] The DFE submodule introduces the system's frequency domain processing mechanism into the feature learning process for low-light image enhancement. Compared with the purely spatial domain method, the DFE submodule can more effectively separate noise and useful signals, especially showing a significant advantage when processing periodic textures and structured edges.

[0033] The DFE submodule is a collaborative refining step within the FSAB block, working sequentially and sharing information with the LCA and IG-MSA submodules. Furthermore, the features processed by the DFE module are those optimized through illumination guidance and local-global interaction; the frequency domain processing here is performed on a more accurate and semantically rich intermediate representation. This deep, guided frequency domain processing more effectively separates low-frequency noise related to illumination and structure from useful high-frequency textures, unlike generalized transformations of raw or shallow features.

[0034] In summary, the FSAB proposed in this scheme achieves deep frequency-spatial domain fusion at the network architecture level. Specifically, an independent DFE submodule is designed in FSAB, which directly operates on the intermediate feature map, transforming it to the frequency domain through Fast Fourier Transform (FFT), performing feature modulation and enhancement in the frequency domain, and then returning it to the spatial domain through Inverse Fourier Transform (IFFT). This design allows the network to explicitly model and enhance the frequency domain components of features during forward propagation, thereby achieving targeted optimization of high-frequency details (such as texture and edges) and low-frequency structures (such as illumination and contours) of the image.

[0035] Furthermore, FSAB includes an LCA submodule, which works in conjunction with DFE to form a complete feature processing workflow: spatial local modeling, spatial global attention, and frequency global enhancement. The LCA submodule enhances the local consistency and contextual relevance of features in the spatial domain by combining local convolution and global attention; while DFE optimizes the overall spectral distribution of features in the frequency domain, suppressing noise bands and enhancing effective information bands. Together, they achieve multi-level feature enhancement across the spatial and frequency domains, as well as local and global aspects.

[0036] Therefore, the FSAB proposed in this scheme achieves true frequency-spatial collaborative feature enhancement at the network structure level, which can not only more effectively recover details and textures in low-light images, but also improve the robustness and generalization ability of the model in complex degradation scenarios.

[0037] This solution employs an encoder-decoder architecture based on U-net to achieve feature fusion and image reconstruction. The specific process is as follows: Encoder feature acquisition sub-step: Input the input feature map into the encoder layer number... In the encoder, the depth features of each encoder layer are obtained to form a feature set: in Indicates the first The depth features acquired by the layer encoder, i.e., the first layer... The output features of the layer encoder are used as the input features of the next layer encoder.

[0038] Each encoder layer performs downsampling through multiple sequentially connected downsampling layers to extract preliminary feature maps, i.e., the lowest resolution features, thereby reducing the resolution of the input feature map. The preliminary feature map output from the last downsampling layer is input into the bottleneck layer for processing to obtain depth features, which are then used as the output features of the current layer's encoder. The input features of the next layer encoder are also included; thus, deep feature interaction and enhancement are performed at the lowest resolution; the bottleneck layer includes several sequentially connected FSAB blocks. The features output by each encoder level are saved in the list fea_encoder to form a feature set. These features contain spatial and semantic information at different scales. The number of encoder levels is The saved feature set is: in Indicates the first Characteristics of the output of the layer encoder.

[0039] Decoder reconstruction sub-step: Depth features acquired by layer encoder The number of input layers to the decoder is In the decoder, the highest resolution features are obtained. ; Each decoder layer performs upsampling through multiple sequentially connected upsampling layers to obtain upsampled features and gradually recover depth features. The resolution is improved by a factor of 2 in this embodiment using transposed convolution (ConvTranspose2d) while reducing the number of channels. in For upsampling features, The input to the upsampling layer is the output of the previous upsampling layer. The input to the first upsampling layer of the layer decoder is the output of the previous decoder. However, the input to the first upsampling layer of the first-layer decoder is depth features. In this embodiment, the number of upsampling layers and the number of downsampling layers are the same. The upsampled features are fused with the corresponding depth features from the encoder layer using skip connections, including: channel stitching: in Features of splicing; Channel fusion is performed on the spliced ​​features using 1×1 convolution to obtain skip connection fused features: The skip connection fused features are input into the FSAB block for frequency-space dual-domain feature enhancement to obtain deep features: in For the first The depth features acquired by the layer decoder are used as the features of its output. Depth features acquired by the layer decoder , as the highest resolution feature; The corresponding level of illumination feature map serves as a guide for illumination features; The decoder layers correspond one-to-one with the encoder layers. Each decoder layer performs the following operations: upsampling to restore resolution, connecting to encoder feature skip connections, refining FSAB blocks, and passing the data to the next decoder layer. The layer decoder processes the feature map, gradually restoring its resolution to the original input size.

[0040] Reconstruction sub-step: The image output layer maps the highest-resolution features output by the decoder to a three-channel enhanced image through the output convolutional layer, and combines it with the input feature map to generate the enhanced image. in To enhance the image.

[0041] The entire network is trained end-to-end, driving all modules to collaboratively learn the optimal enhancement mapping function by minimizing the pixel-level loss (L1 loss) between the predicted enhanced image and the real normal lighting image.

[0042] This embodiment also provides an image enhancement system based on multi-scale cross-attention and frequency-domain enhancement, which enhances images through a constructed MCFE-Net (Multi-scale Cross-attention and Frequency-domain Enhancement Network), wherein MCFE-Net includes: an MLSPM module and an encoder-decoder architecture; The MLSPM module is used to extract multi-scale illumination features from low-light images, perform preliminary estimation of illumination distribution, obtain preliminary illumination feature maps and illumination estimation maps, and combine the low-light image and illumination estimation map to generate an input feature map. Construct an encoder-decoder architecture to perform feature fusion and image reconstruction on the input feature map, generating an enhanced image; The feature fusion and image reconstruction process includes: performing frequency-space dual-domain feature enhancement to obtain depth features; the frequency-space dual-domain feature enhancement is performed through several constructed FSAB blocks, which perform local aggregation, global association, nonlinear transformation, and frequency domain enhancement on the preliminary feature map extracted from the input feature map to obtain depth features; the global association uses the illumination feature map as a guiding signal.

[0043] In other embodiments, it further includes: a preprocessing module and a postprocessing module; The preprocessing module receives the raw low-light image, performs operations such as size normalization and numerical standardization, and converts it into a tensor format that can be processed by MCFE-Net. The post-processing module is used to perform operations such as denormalization, cropping, and format conversion on the tensors (i.e., enhanced images) output by MCFE-Net to generate the final enhanced image that can be displayed or stored.

[0044] The process of image enhancement using the above system includes: The input preprocessing module receives the raw low-light image, performs operations such as size normalization and numerical standardization, and converts it into a tensor format that can be processed by MCFE-Net (i.e., the network). The MSLPM module is the core analysis module of the system. It contains the core MC-EPA submodule, which receives the preprocessed low-light image, generates a preliminary illumination feature map and illumination estimation map, and combines the low-light image and illumination estimation map to generate the input feature map. The encoder-decoder architecture is used to perform feature fusion and image reconstruction on the input feature map to generate an enhanced image; In the encoder-decoder architecture, each FSAB block contains four sub-modules in sequence: LCA, IG-MSA, FFN, and DFE. The data undergoes a cyclic process of "local aggregation → global correlation → nonlinear transformation → frequency domain enhancement" in this module. Post-processing module: Performs operations such as denormalization, cropping, and format conversion on the tensors output by the network to generate the final enhanced image that can be displayed or stored.

[0045] In summary, the low-light image sequentially flows through the preprocessing module, the multi-scale illumination sensing module (MSLPM), the encoder-decoder architecture, and the post-processing module. A deep learning model is trained in an end-to-end manner, ultimately inferring and outputting an enhanced image. The illumination features generated by the MSLPM module serve as crucial control signals, which are injected into the subsequent FSAB module to achieve illumination-guided enhancement throughout the entire process.

[0046] This solution proposes a deep fusion low-light image enhancement network architecture, MCFE-Net (Multi-scale Cross-attention and Frequency-domain Enhancement Network), which features illumination perception as the guide, dual-path interaction, and frequency-space dual-domain collaboration. It abandons the traditional approach of "decomposition followed by enhancement" or "simple feature aggregation" and innovatively designs a scheme that uses multi-scale illumination estimation as a global guiding signal to drive bidirectional interaction between local detail restoration and global semantic enhancement. In this interaction process, frequency domain and spatial domain information are simultaneously integrated for joint optimization, forming an integrated process from illumination estimation to interactive refinement and then to dual-domain enhancement.

[0047] Specifically, this solution effectively improves the overall brightness and contrast of low-light images, while significantly suppressing noise and restoring detailed textures; it achieves adaptive modeling and correction of uneven lighting in images, avoiding local overexposure or underexposure; and it fully utilizes the complementary information in the frequency and spatial domains of images to improve the robustness and visual quality of the enhancement process.

[0048] The specific implementation process is as follows: To evaluate the performance of the proposed MCFE-Net in low-light image enhancement tasks, this scheme was quantitatively compared and analyzed on the public datasets LOL-v1 and LOL-v2. The experiments used Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index (SSIM) as objective evaluation metrics, and comprehensively compared it with various mainstream techniques in the current supervised and unsupervised learning fields. The comparison methods included RetinexNet, Sparse, EnGAN, RUAS, FIDE, DRBN, KinD, Restormer, MIRNet, Retinexformer, and Retinexmamba.

[0049] The experimental results are shown in Table 1. Compared with other mainstream methods in recent years, the proposed method achieves the highest PSNR value on both the LOL-v1 and LOL-v2-syn datasets, and achieves the best SSIM index in all test datasets. This shows that the algorithm in this paper has low overall distortion and noise interference in low-light image enhancement, and also shows a significant advantage in maintaining the structural information of the original image.

[0050] Table 1: Quantitative comparison on the LOLv1 and LOLv2 datasets Based on the quantitative comparative experiments, in order to further intuitively compare the enhancement effects of different low-light enhancement techniques, we will conduct a visualization analysis on several representative examples on the LOL-v1 and LOL-v2 datasets, compare the enhancement results of various mainstream low-light enhancement algorithms with the MCFE-Net method proposed in this invention, and magnify key local areas to clearly present the differences in performance of different methods in terms of detail restoration, brightness adjustment and noise suppression.

[0051] Figure 6 , Figure 7 and Figure 8 The paper presents a comparison of low-light enhancement results using various methods on the LOL-v1, LOL-v2-real, and LOL-v2-syn datasets. Figure 6 and Figure 7Of the results shown, the Retinex-Net algorithm exhibits significant noise and noticeable color distortion, resulting in a large discrepancy between the enhanced image and the real-world scene. The KinD method performs well in overall brightness enhancement and noise suppression, but falls short in detail and color restoration in dark areas. The RUAS method results in inconsistent brightness across different regions, with some areas appearing too dark or overexposed, indicating unstable overall brightness control and excessively high contrast. The Restormer algorithm is slightly insufficient in brightness enhancement, and magnified image details show a smearing effect. The MIRNet method performs well, but the enhanced image appears slightly yellowish, and its ability to restore local details is insufficient. The RetinexFormer algorithm outperforms the previous methods, but is slightly weaker than the proposed method in terms of noise suppression and detail restoration. Figure 8 In the results shown, KinD, RUAS, Restormer, and MIRNet all exhibited color cast in image color restoration. The RetinexFormer algorithm achieved better enhancement results, but the color restoration was too vibrant. The enhancement results of the method proposed in this invention are closer to the reference image. Overall, compared to other mainstream low-light image enhancement algorithms, the method proposed in this invention provides more accurate illumination estimation for both bright and dark areas of the image, while also possessing good denoising capabilities. It improves the overall brightness and contrast of the image, better restores details in dark areas, and maintains high color fidelity and structural integrity, demonstrating a more comprehensive low-light image enhancement capability.

[0052] Furthermore, to verify the effectiveness of each key module in the proposed method, including the Multi-Scale Cross-Enhanced Attention Module (MC-EPA), the Local Context Aggregation Module (LCA), and the Frequency Domain Enhancement Module (DFE), ablation experiments were conducted by removing each module individually. The rest of the network structure remained unchanged during the experiments, and training and testing were performed on the LOL-v1 and LOL-v2 datasets, with PSNR and SSIM used as evaluation metrics. The experimental results in Table 2 show that removing any module significantly reduced model performance, demonstrating that each of the three substructures—MC-EPA, LCA, and DFE—significantly contributes to the improvement of model performance and plays an irreplaceable role.

[0053] Table 2: Ablation experiments on LOLv1 and LOLv2 datasets In summary, the MCFE-Net network proposed in this scheme outperforms existing mainstream technologies in terms of enhancement effect, noise robustness, and model interpretability. The specific reasons are as follows: MC-EPA module: Unlike methods that use single-scale convolution or simple concatenation of multi-scale features, this scheme extracts multi-scale features through parallel dilated convolution and then innovatively designs a cross-attention fusion layer. This layer learns a unified weight tensor and assigns it to features of different scales, realizing dynamic feature selection and interaction across scales, and can more accurately model complex lighting gradations and boundaries. LCA submodule: Existing methods typically use convolution (local) or self-attention (global) alone. The LCA submodule designed in this scheme creatively integrates the local features extracted by depthwise separable convolution with the semantic features extracted by global average pooling through a lightweight attention mechanism. This interaction mode of "local query" and "global key value" realizes the adaptive enhancement of local details under the guidance of the global context. The DFE submodule integrates Fast Fourier Transform / Inverse Transform into the deep learning forward propagation process and performs end-to-end optimization as a component of the standard network block. This module enables the network to operate directly in the frequency domain, providing a complementary processing dimension for noise separation and texture enhancement that is completely different from traditional spatial domain methods.

[0054] Illumination-guided end-to-end integrated architecture: Unlike the traditional Retinex method's two-stage pipeline of explicit decomposition followed by separate enhancement, and unlike some deep learning models that simply stitch together illumination maps at the input, this scheme constructs an end-to-end network that deeply couples illumination estimation and image enhancement. The features generated by the MSLPM submodule are used as guiding signals and embedded into the subsequent feature refinement stage, achieving consistent optimization throughout the entire process from global illumination estimation to local detail restoration.

[0055] These differentiated technical approaches work together to make the MCFE-Net network in this scheme superior to existing mainstream technologies in terms of enhancement effect, noise robustness, and model interpretability.

[0056] The above descriptions are merely embodiments of the present invention. Commonly known structures and characteristics are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are aware of all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. An image enhancement method based on multi-scale cross-attention and frequency domain enhancement, characterized in that, include: Multi-scale illumination sensing and preliminary estimation steps: For low-light images, multi-scale illumination features are extracted through the constructed MSLPM module to perform preliminary estimation of illumination distribution, obtain illumination feature map and illumination estimation map, and combine low-light image and illumination estimation map to generate input feature map; Feature fusion and image reconstruction steps: The input feature map is used to construct an encoder-decoder architecture to perform feature fusion and image reconstruction, generating an enhanced image; The feature fusion and image reconstruction process includes: performing frequency-space dual-domain feature enhancement to obtain depth features; the frequency-space dual-domain feature enhancement is performed through several constructed FSAB blocks, which perform local aggregation, global association, nonlinear transformation, and frequency domain enhancement on the preliminary feature map extracted from the input feature map to obtain depth features; the global association uses the illumination feature map as a guiding signal.

2. The image enhancement method based on multi-scale cross-attention and frequency domain enhancement according to claim 1, characterized in that, The multi-scale illumination sensing and preliminary estimation steps include: S101. Input the low-light image into the constructed MSLPM module, compress the number of channels of the input features through the feature dimensionality reduction layer, and extract the input features; S102. Input the input features into the MC-EPA submodule of the MSLPM module, perform multi-scale cross-enhancement, and obtain intermediate features; S103. For intermediate features, obtain illumination feature maps through the depthwise separable convolutional layer in the MSLPM module; S104. Based on the illumination feature map, obtain the illumination estimation map through the convolutional layer in the MSLPM module; S105, and then combine the low-light image and the illumination estimation map to generate the input feature map.

3. The image enhancement method based on multi-scale cross-attention and frequency domain enhancement according to claim 2, characterized in that, S102 includes: The multi-scale feature extraction layer in the S10201 and MC-EPA submodules processes the input features in parallel through three sets of parallel dilated convolutions, extracting spatial contextual information features under three different receptive fields (small, medium, and large) as output features. The cross-attention fusion layer in the S10202 and MC-EPA submodules employs a cross-attention fusion mechanism based on input features. Attention weights are generated for the output features at each scale of the multi-scale feature extraction layer, and the output features at each scale are weighted to generate weighted multi-scale features. The cross-attention fusion layer in the S10203 and MC-EPA submodules employs an adaptive gating fusion mechanism, based on input features. By using global context information, the output features of each scale of the multi-scale feature extraction layer are dynamically adjusted to generate fusion weights, and the weighted multi-scale features are weighted to generate fusion features. The multilayer perceptron in the S10204 and MC-EPA submodules will fuse features. Compared with the original input features Perform residual connection merging to obtain intermediate features.

4. The image enhancement method based on multi-scale cross-attention and frequency domain enhancement according to claim 1, characterized in that, The FSAB block consists of an LCA submodule, an IG-MSA submodule, an FFN submodule, and a DFE submodule; The LCA submodule is used to perform local aggregation and obtain local-global features. The IG-MSA submodule is used to perform global correlation on local and global features to obtain weighted features; The FFN submodule is used to perform nonlinear transformations on the weighted features to obtain the FFN output features; The DFE submodule is used to perform nonlinear transformations and frequency domain enhancements on the FFN output features to obtain deep features.

5. The image enhancement method based on multi-scale cross-attention and frequency domain enhancement according to claim 4, characterized in that, The LCA submodule adopts a dual-path design to process illumination feature maps and fuse local details with global semantics, including: local path and global path; Local paths capture spatial neighborhood information of the illumination features as input through depthwise separable convolutions, thereby obtaining local detail features: ； in The feature map for the input FSAB block includes: the preliminary feature map extracted by encoder downsampling or the depth features output from the previous FSAB block; In the global path, global average pooling is used to compress the spatial dimension and extract image-level global semantic features from the input illumination features: ； The outputs of the local and global paths are fused through a lightweight attention interaction layer to obtain local-global features.

6. The image enhancement method based on multi-scale cross-attention and frequency domain enhancement according to claim 5, characterized in that, The attention interaction layer uses local detail features as queries and global semantic features as both keys and values, and calculates local-global features through a multi-head attention mechanism: 。 7. The image enhancement method based on multi-scale cross-attention and frequency domain enhancement according to claim 4, characterized in that, The processing procedure of the IG-MSA submodule includes: Feature mapping sub-step: converting local features to global features Mapped to query matrices respectively Key matrix Value matrix ; ； ； ； Sub-step of the illumination adjustment value matrix: Convert the illumination feature map... AND-value matrix Perform element-wise multiplication, perform illumination-sensing feature adjustment, and obtain the illumination-adjusted value matrix. : in This indicates element-wise multiplication; Attention weight calculation sub-step: Calculate the query matrix AND key matrix Similarity to obtain the attention weight matrix : in For feature dimension, A learnable scaling factor; Illumination-guided feature aggregation sub-step: Based on the attention weight matrix Value matrix after illumination adjustment Perform weighted aggregation to obtain aggregated features: ； Output fusion sub-step: For aggregated features Perform linear projection and add positional encoding to obtain weighted features; in To output the projected weights, For position encoding functions, For input features.

8. The image enhancement method based on multi-scale cross-attention and frequency domain enhancement according to claim 4, characterized in that, The DFE submodule transforms the weighted features from the spatial domain to the frequency domain for processing, and then transforms them back to the spatial domain. Specifically, this includes: Perform a two-dimensional Fourier transform on the weighted features of the input: in This indicates the Fast Fourier Transform operation; Frequency domain processing is performed on the real and imaginary parts separately: ； in and These are the processed real and imaginary components. and These represent the operations of extracting the real part and the imaginary part, respectively. The inverse Fourier transform is used to return to the spatial domain, and then the residual connection is used to fuse with the original features to obtain deep features: ； in To return the characteristics of the spatial domain, It is a depth feature.

9. The image enhancement method based on multi-scale cross-attention and frequency domain enhancement according to claim 1, characterized in that, The encoder-decoder architecture includes: an encoder, a decoder, and an image output layer; The feature fusion and image reconstruction steps include: Encoder feature acquisition sub-step: Input the input feature map into the encoder layer number... In the encoder, the depth features of each encoder layer are obtained to form a feature set: in Indicates the first The depth features acquired by the layer encoder are used as the input features for the next layer encoder. Each encoder layer performs downsampling through multiple sequentially connected downsampling layers to extract preliminary feature maps; The preliminary feature map output from the last downsampling layer is input into the bottleneck layer for processing to obtain depth features, which are then used as the output features of the current layer's encoder. and the input features of the next layer encoder; The features output by each encoder level form a feature set; The number of encoder levels is The saved feature set is: in Indicates the first Characteristics of the layer encoder's output; Decoder reconstruction sub-step: Depth features acquired by layer encoder The number of input layers to the decoder is In the decoder, the highest resolution features are obtained. ; Each decoder layer performs upsampling through multiple sequentially connected upsampling layers to obtain upsampled features: in For upsampling features, The input to the upsampling layer is the output of the previous upsampling layer. The input to the first upsampling layer of the layer decoder is the output of the previous decoder. However, the input to the first upsampling layer of the first-layer decoder is deep features. ; The upsampled features are fused with the corresponding depth features from the encoder layer using skip connections, including: channel stitching: in Features of splicing; Channel fusion is performed on the spliced ​​features using 1×1 convolution to obtain skip connection fused features: The skip connection fused features are input into the FSAB block for frequency-space dual-domain feature enhancement to obtain deep features: in For the first The depth features acquired by the layer decoder are used as the features of its output. Depth features acquired by the layer decoder , as the highest resolution feature; This is the illumination feature map for the corresponding level; Reconstruction sub-step: The image output layer maps the highest-resolution features output by the decoder to a three-channel enhanced image through the output convolutional layer, and combines it with the input feature map to generate the enhanced image. in To enhance the image.

10. An image enhancement system based on multi-scale cross-attention and frequency domain enhancement, characterized in that, Image enhancement is performed using the constructed MCFE-Net network, including: The MLSPM module is used to extract multi-scale illumination features from low-light images, perform preliminary estimation of illumination distribution, obtain illumination feature maps and illumination estimation maps, and combine the low-light image and illumination estimation map to generate an input feature map. Construct an encoder-decoder architecture to perform feature fusion and image reconstruction on the input feature map, generating an enhanced image; The feature fusion and image reconstruction process includes: performing frequency-space dual-domain feature enhancement to obtain depth features; the frequency-space dual-domain feature enhancement is performed through several constructed FSAB blocks, which perform local aggregation, global association, nonlinear transformation, and frequency domain enhancement on the preliminary feature map extracted from the input feature map to obtain depth features; the global association uses the illumination feature map as a guiding signal.

Images