A semantically perceptive image enhancement method based on amplitude-phase dual-path coordination

Abstract

This invention belongs to the field of computer vision and digital image processing technology, specifically relating to a semantically aware image enhancement method based on amplitude-phase dual-path collaboration. The invention includes: acquiring a degraded image to be processed; performing frequency domain decomposition on the degraded image to obtain amplitude and phase components; designing a semantically aware Mamba algorithm to enhance the amplitude component, and enhancing the phase component; using bidirectional cross-attention to collaboratively enhance the enhanced amplitude and phase features, obtaining optimized amplitude and phase features; recombining the optimized amplitude and phase features and transforming them to the spatial domain to obtain the enhanced image. This invention captures global dependencies by introducing a state-space model, decouples and enhances the amplitude and phase information of the image separately in the Fourier domain, and utilizes cross-attention to achieve collaborative optimization of the two, thereby achieving effective enhancement for various types of image degradation.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and digital image processing technology, and specifically relates to a semantically perceptual image enhancement method with amplitude-phase dual-path coordination. Background Technology

[0002] In the fields of computer vision and image processing, image enhancement aims to improve image quality degradation caused by various degrading factors (such as insufficient lighting, haze, raindrops, etc.), restoring image sharpness, contrast, and detail. This type of technology has wide-ranging applications in autonomous driving, video surveillance, remote sensing imaging, and consumer electronics.

[0003] Existing image enhancement methods are typically designed for specific types of degradation. For example, low-light enhancement methods mainly focus on brightness improvement and noise suppression; dehazing methods aim to eliminate atmospheric scattering effects; and deraining methods focus on removing raindrop streaks. These methods can generally be categorized into physically based methods and deep learning-based methods. Physically based methods (such as dark channel priors for dehazing and Retinex theory for low-light enhancement) rely on prior assumptions about specific degradations, resulting in limited generalization ability in complex real-world scenes. Deep learning-based methods, especially convolutional neural networks, while achieving significant progress, are limited by their inherent local receptive field characteristics, restricting their ability to model global context, and most methods operate only in the spatial domain, failing to fully utilize the frequency domain information of the image.

[0004] More importantly, existing methods are often designed for a single type of degradation and lack the ability to handle different degradation types universally. In practical applications, images may suffer from multiple degradations simultaneously (such as nighttime haze, low light in rainy weather, etc.), and single-task enhancement methods are insufficient to cope with such complex situations. Therefore, there is an urgent need for a general image enhancement framework that can handle multiple degradation types. This framework needs to have strong global modeling capabilities, an effective frequency domain information utilization mechanism, and adaptive processing capabilities for different degradation types. Summary of the Invention

[0005] The purpose of this invention is to address the problems existing in the prior art and propose a semantically aware image enhancement method based on amplitude-phase dual-path collaboration. By introducing a semantically aware Mamba-based method to capture global semantic dependencies, the amplitude and phase information of the image are decoupled and enhanced separately in the Fourier domain. Furthermore, a bidirectional cross-attention mechanism for amplitude and phase is utilized to achieve deep information interaction and collaborative optimization between the two, thereby achieving effective enhancement for various types of image degradation.

[0006] To achieve the above objectives, the technical solution provided by this invention is as follows:

[0007] A semantically aware image enhancement method based on amplitude-phase dual-path coordination includes:

[0008] The degraded image to be processed is acquired, and the frequency domain decomposition of the degraded image is performed to obtain the amplitude and phase components of the image.

[0009] A semantically aware Mamba module is constructed; preliminary amplitude features are obtained by feature extraction of amplitude components, and the preliminary amplitude features are input into the Mamba module for semantic recombination and global dependency modeling. Based on the modeling results, a modulation map is generated, and the amplitude components are adaptively adjusted using the modulation map to obtain the modulated amplitude features.

[0010] Phase components are feature extracted to obtain preliminary phase features. The degraded image is processed using a pre-defined edge structure information extraction network to obtain edge structure information. Enhanced phase features are obtained based on the preliminary phase features and edge structure information.

[0011] The modulated amplitude features and enhanced phase features are subjected to bidirectional cross-attention for bidirectional information complementation and semantic alignment, resulting in optimized amplitude features and optimized phase features.

[0012] The optimized amplitude and phase features are recombined and transformed into the spatial domain to obtain the enhanced image.

[0013] Furthermore, the semantically aware Mamba module includes a semantic routing submodule and a selective state space modeling submodule, wherein the semantic routing submodule includes a convolutional layer and a softmax function.

[0014] Furthermore, the process of extracting features from the amplitude components to obtain preliminary amplitude features, inputting these preliminary amplitude features into the Mamba module for semantic recombination and global dependency modeling, and generating a modulation map based on the modeling results includes:

[0015] Preliminary amplitude features are obtained by performing preliminary feature mapping on the amplitude components through convolutional layers;

[0016] The initial amplitude features are input into a semantically aware Mamba module to generate a semantic routing weight graph.

[0017] The channels of the initial amplitude features are reordered based on the values ​​of the semantic routing weight graph to form semantic reorganization features;

[0018] The selective state-space modeling submodule treats semantic recombination features as sequential data, performs long-distance dependency modeling through a selective state-space model, and outputs a semantically enhanced feature sequence. This process is described by the following state-space equation:

[0019]

[0020] in, For time steps The hidden state, , , , For parameterized and input-dependent selectivity matrices, For time steps semantic recombination features For time steps The output sequence;

[0021] The output sequences of all time steps are concatenated in order to obtain the semantically enhanced feature sequence, which is then mapped to a modulation map through a convolutional layer.

[0022] Furthermore, the step of reordering the channels of the preliminary amplitude features based on the values ​​of the semantic routing weight graph to form semantic reorganization features includes:

[0023] The semantic routing weight map and the initial amplitude features are expanded into a one-dimensional vector in the spatial dimension to obtain the weight vector and the feature vector.

[0024] Sort the elements in the weight vector in descending order to obtain the corresponding sort index;

[0025] The expanded feature vectors are rearranged using a sort index to obtain the rearranged features;

[0026] The rearranged features are restored to a two-dimensional spatial form, resulting in semantic recombination features.

[0027] Furthermore, the enhanced phase features obtained based on the preliminary phase features and edge structure information include:

[0028] A cross-attention mechanism is used to fuse preliminary phase features and edge structure features to obtain the phase features after feature fusion;

[0029] The phase features after feature fusion are residually connected with the initial phase features, and the enhanced phase features are output through an encoder-decoder network.

[0030] Furthermore, the modulated amplitude features and enhanced phase features are subjected to bidirectional cross-attention for bidirectional information complementarity and semantic alignment to obtain optimized amplitude features and optimized phase features, including:

[0031] The amplitude query vector is obtained from the modulated amplitude features, and the phase bond vector and phase value vector are obtained from the enhanced phase features. The optimized amplitude features are then calculated, as expressed by the following formula:

[0032]

[0033] in, This represents the optimized amplitude characteristics. This represents the Softmax function. This represents the amplitude query vector. Represents the phase bond vector. Represents the phase value vector. For feature dimensions;

[0034] The phase query vector is obtained through the enhanced phase features, and the amplitude key vector and amplitude value vector are obtained through the modulated amplitude features. The optimized phase features are calculated and expressed by the following formula:

[0035]

[0036] in, This represents the optimized phase characteristics. Represents the phase query vector. Represents the amplitude bond vector. This represents the amplitude value vector.

[0037] Furthermore, the process of recombining the optimized amplitude features and optimized phase features and converting them to the spatial domain to obtain the enhanced image includes:

[0038] The modulated amplitude features are fused with the optimized amplitude features to obtain the amplitude spectrum;

[0039] The optimized phase features are subjected to nonlinear mapping to obtain the final phase representation;

[0040] The amplitude spectrum is combined with the final phase representation to form a complex spectrum, and an inverse fast Fourier transform is performed to obtain the enhanced image.

[0041] Furthermore, the amplitude-phase dual-path collaborative semantic-aware image enhancement method is trained and optimized using a multi-task loss function, which includes frequency domain contrast regularization loss, pixel-level reconstruction loss, and perceptual loss. The frequency domain contrast regularization loss is expressed by the formula:

[0042]

[0043] in, This represents the frequency domain contrast regularization loss. For the first A degraded image, In order to be with the first A normal image corresponding to a degraded image. For other degraded images, It is a constant. This represents a two-dimensional Fast Fourier Transform. In order to target the The set of indices of other degraded images selected from the degraded image. The number of degraded images, The number of additional degraded images selected for all degraded images.

[0044] Compared with the prior art, the significant advantages of this invention are:

[0045] (1) High versatility: Through a unified frequency domain decomposition and fusion framework, it can handle a variety of image degradation types, including low light enhancement, image dehazing, and image deraining, achieving the goal of "one model, multiple tasks".

[0046] (2) Global modeling and semantic perception: By designing semantic perception-based Mamba, the global long-range dependencies of images are efficiently modeled with linear complexity, overcoming the limitation of the receptive field of CNN. Combined with the semantic routing mechanism, it can dynamically identify and differentiate different semantic regions in the image, achieving more content-appropriate adaptive enhancement.

[0047] (3) Frequency domain decoupling and collaborative optimization: The image is decoupled into amplitude and phase components in the Fourier domain, and processing branches are established for each component. The amplitude branch focuses on the recovery of degradation-related features, while the phase branch focuses on structure preservation, thus separating the key sub-problems in the augmentation task from the source. Through the designed bidirectional cross-attention fusion mechanism, deep interaction and mutual correction between degradation features and structural information are achieved during the augmentation process.

[0048] (4) Superior overall performance: The method of this invention achieves leading levels on multiple publicly available image enhancement benchmark datasets. In low-light enhancement tasks, it can effectively improve brightness and suppress noise; in dehazing tasks, it can effectively remove haze and restore clear scenes; in deraining tasks, it can effectively remove raindrops and preserve background details. Both quantitative indicators and visual quality are superior to existing single-task methods.

[0049] (5) Good practicality and generalization: The network structure is efficient, and the number of parameters and computational cost have been optimized, making it easy to deploy. The model shows good robustness and generalization ability for various degradation types and complex hybrid degradation scenarios, and has broad practical value. Attached Figure Description

[0050] Figure 1 This is a flowchart illustrating the semantic-aware image enhancement method based on amplitude-phase dual-path collaboration according to the present invention.

[0051] Figure 2 This is a network structure diagram of a semantically perceptual image enhancement method based on amplitude-phase dual-path collaboration according to the present invention.

[0052] Figure 3 This is a schematic diagram of the amplitude branching process of the present invention;

[0053] Figure 4 This is a schematic diagram of the phase branching process of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0055] like Figure 1 As shown, this invention provides a semantically aware image enhancement method based on amplitude-phase dual-path collaboration. This method operates end-to-end, its core being the decoupling of the image to the frequency domain via Fourier transform, followed by enhancement along both amplitude and phase paths. Collaboration between these paths is achieved through bidirectional cross-attention, ultimately reconstructing the enhanced image. This method is designed as a general framework, which, by configuring specific training data and task loss, can be flexibly applied to tasks such as low-light enhancement, dehazing, and rain removal. The following section uses low-light enhancement as an example for detailed explanation.

[0056] S1. Data preparation (taking low-light enhancement as an example).

[0057] Collect and construct the training dataset. For the low-light enhancement task, download publicly available datasets such as LOL (LOw-Light) and LSRW (Low-light Street Real-world), which contain paired low-light visible light images. and corresponding normal illumination visible light image For other tasks such as rain and fog removal, the corresponding public datasets were downloaded for network training.

[0058] S2, Network forward propagation (taking low-light enhancement as an example).

[0059] A semantic-aware image enhancement network coordinating amplitude-phase dual paths is constructed. Inputting the trained amplitude-phase dual-path collaborative semantic-aware image enhancement network yields an enhanced visible light image. Network structure such as Figure 2 As shown. The detailed sub-steps of network forward propagation are as follows:

[0060] S2.1 Fourier Decomposition and Frequency Domain Representation:

[0061] First, for low-light visible light input Apply the Fast Fourier Transform (FFT) to transform it from the spatial domain to the frequency domain:

[0062]

[0063] in, For complex spectrum, Indicates the row number of the complex number spectrum. The number of columns in the spectrum of complex numbers.

[0064] Subsequently, the complex spectrum is decomposed into amplitude components that carry global illumination and color information. and the phase components that encode image structure and texture. ,in, This represents the argument function. For ease of processing by neural networks, the phase is represented by cosine and sine values, i.e.:

[0065]

[0066] S2.2 Amplitude Branch: Semantic-Aware Global Illumination Recovery.

[0067] The core innovation of this module lies in the introduction of semantically aware state space modeling to address two major challenges faced by traditional global context modeling methods (such as Transformer or standard Mamba) in low-light enhancement tasks: 1) the blind equalization of image content non-uniformity (such as extremely dark areas, textured areas, and flat areas); and 2) the semantic context fragmentation caused by flattening two-dimensional spatial features into a one-dimensional sequence. Figure 3 As shown, this module achieves adaptive and efficient modeling of global illumination relationships through semantically guided sequence recombination and selective state space scanning. The specific steps are as follows:

[0068] S2.2.1 Semantic routing weight generation: The amplitude component A is processed by a 3×3 convolutional layer for preliminary feature mapping to obtain the preliminary amplitude features. .

[0069] Construct a semantically aware Mamba module, which includes a semantic routing submodule and a selective state-space modeling submodule. Input semantic routing submodule, based on Based on the semantic importance of the channels, a semantic routing weight graph is generated. The process can be represented as follows:

[0070]

[0071] in, Represents a semantic routing weight graph. This represents the convolution operation. The function ensures that the weights are normalized. This indicates the initial amplitude characteristics.

[0072] This image can automatically identify which regions in an image are semantically more important or require more attention. The value of depends dynamically on the content of the input image. For example, higher weights are automatically predicted in textured edges or extremely dark areas with insufficient lighting, while lower weights are predicted in well-lit and structurally flat areas, thus providing a perceptual basis for subsequent feature reconstruction.

[0073] S2.2.2 Semantic-guided feature sequence recombination: To address the problem of semantically irrelevant features interfering with model perception in standard sequence modeling, this step uses a semantic routing weight graph. For preliminary amplitude characteristics The feature vectors at all spatial locations are reordered. The ordering is based on the feature vectors corresponding to each location. value.

[0074] The specific sorting steps are as follows: first, the semantic routing weight graph in the two-dimensional space is... and preliminary amplitude characteristics Expanding this into a one-dimensional vector in spatial dimensions yields the weight vector. and eigenvectors ,in, Indicates the number of channels. It represents the total number of spatial locations.

[0075] Then the weight vector Sort the elements in descending order to obtain the corresponding sort index. Its mathematical expression is:

[0076]

[0077] in, It is a function that returns the sorted indices. This indicates that the items are arranged in descending order. correspond The original position of the maximum value. correspond The original position of the minimum value.

[0078] Then use the sorted index Expanded eigenvectors Perform rearrangement to obtain the rearranged features. Then, it is restored to a two-dimensional spatial form to obtain semantic recombination features. :

[0079]

[0080] in, Indicates semantic recombination features, This indicates the index sorted along the spatial dimension (the second dimension). Select features in sequence, This indicates that the rearranged features are restored to a two-dimensional spatial form.

[0081] After sorting, features with similar semantic importance or attributes (such as dark areas that need enhancement or texture areas that need detail preservation) will be included in the recombined semantic features. The clusters are adjacent to each other. This is equivalent to performing a dynamic, semantically driven clustering preprocessing on the feature sequence before inputting it into the state space model, laying the foundation for subsequent modules to focus on processing the relevant context.

[0082] S2.2.3 Selective state-space modeling: semantic recombination features Input a selective state-space model (standard Mamba). The innovative aspect of this step lies in combining the semantically reconstructed features with the selective scanning mechanism of the selective state-space model, achieving efficient linear complexity modeling of semantically relevant contexts. The specific steps are as follows:

[0083] Parameters are generated dynamically: the model generates parameters based on the current input. By dynamically generating key parameters through linear projection, including step size Δ, input matrix B, output matrix C, and state matrix D, a completely input-dependent selective mechanism is realized.

[0084] System Discretization and Solution: The continuous system is discretized using a dynamic Δ to obtain the discrete parameters. Subsequently, a parallelized selective scan algorithm is used to solve the discrete state-space equations. This process allows the model to dynamically decide whether to retain or ignore information from historical states while scanning the sequence, thus enabling more efficient capture of long-range dependencies most relevant to the current semantic location.

[0085] Output: After the above modeling, the final output is the output sequence Y containing global semantic context information.

[0086] The state-space equations are expressed by the following formula:

[0087]

[0088] in, For time steps The hidden state, For time steps The hidden state, For time steps semantic recombination features This represents the hidden state after the current time step update. For time steps The output sequence. All of them. Sequentially concatenating the sequences yields the semantically enhanced feature sequence Y. , , , The parameterized and input-dependent selection matrix is ​​shown below:

[0089]

[0090] Among them, the step size Δ, the input matrix B, the output matrix C, and the state matrix D are based on the input. The generated dynamic parameters The dynamic step size parameter Δ is in discrete time step The specific value.

[0091] S2.2.4 Modulation Map Generation: Using the original position indices (sorted in descending order) saved in step S2.2.2, the output sequence Y of the state-space model is reverse-sorted to restore the original two-dimensional spatial dimension. Then, it is processed through a 1×1 convolutional layer and a sigmoid activation function to generate an adaptive illumination modulation map with values ​​ranging from [0,1]. The modulation map G encodes pixel-level illumination adjustment suggestions after semantically aware global analysis.

[0092] S2.2.5 Amplitude Modulation: The amplitude component A is adaptively adjusted using the modulation diagram G, with the following formula:

[0093]

[0094] in, This represents the amplitude characteristics after modulation, ⊙ represents element-wise multiplication, and γ is a learnable scaling parameter with an initial value of 0.5, used to control the enhancement amplitude.

[0095] S2.3, Phase branch: Phase enhancement guided by edge structure features.

[0096] See Figure 4 The low-light image is input into a pre-trained edge information extraction network to obtain an edge structure map that corresponds to the input low-light image. Then, the phase component P is compared with the edge structure map. A common input phase branch is used. The edge structure map provides stable edge structure information to the image, effectively compensating for detail loss in dark areas of visible light. The deraining and dehazing networks also use this network to obtain edge structure information. The branch structure is as follows: Figure 4 As shown, specifically:

[0097] S2.3.1 Visible Light Phase Flow: The phase component P first passes through a convolutional layer and is mapped to visible light phase features. (Preliminary phase characteristics).

[0098] S2.3.2, Edge Structure Feature Flow: This involves processing low-light images... Edge structure features are obtained by inputting them into a pre-trained edge information extraction network. Subsequently Edge structure features are extracted through another separate convolutional layer. .

[0099] S2.3.3 Cross-modal feature fusion: An attention mechanism is used to fuse two feature streams to guide the enhancement of visible light phase features. The calculation process is as follows:

[0100]

[0101] in, This represents the Softmax function. , , The projection matrix is ​​learnable. For feature dimension, It is the phase feature resulting from the fusion of the low-light image and the corresponding edge structure features. This cross-attention mechanism enables the network to fully utilize edge structure features to supplement the texture information of the visible light phase features.

[0102] S2.3.4 Residual Enhancement: Phase features after feature fusion With visible light characteristics Residual connections are performed, and the enhanced phase features are output through a codec network consisting of four residual blocks. Each residual block contains two 3×3 convolutional layers, a batch normalization layer, and a ReLU activation function.

[0103] S2.4, Further Enhancement of Bidirectional Cross-Attention: Co-optimization of Amplitude and Phase

[0104] Traditional frequency domain methods typically process amplitude and phase independently or perform simple fusion later, neglecting the inherent coupling between illumination (primarily encoded in amplitude) and structure (primarily encoded in phase) information during image degradation. To overcome this limitation, this invention innovatively introduces a bidirectional cross-attention fusion mechanism. The core of this mechanism lies in abandoning the independent processing paradigm and establishing a dynamic, bidirectional information exchange bridge between the amplitude and phase paths, enabling global illumination estimation and local structure restoration to guide each other and achieve collaborative optimization. The specific implementation steps are as follows:

[0105] S2.4.1 Phase-Guided Amplitude: The innovation of this approach lies in using the clear structural information recovered from the phase branch as a spatial prior to guide and constrain the illumination modulation process of the amplitude branch, thereby preventing unnatural halos or color overflow at structural abrupt changes such as object edges. Specifically, this is achieved through... Obtain the amplitude query vector Through enhanced phase features Obtain the phase bond vector and phase value vector The formula is as follows:

[0106] ,

[0107] in, These are the learnable matrix parameters.

[0108] Subsequently, the optimized amplitude characteristics are obtained using the following formula. :

[0109]

[0110] in, For feature dimensions.

[0111] S2.4.2 Amplitude-Guided Phase: Symmetrically, the innovation of this path lies in using the global illumination distribution estimated by the amplitude branch as semantic context to guide the phase branch in structural recovery in difficult regions, enabling the model to imagine texture details that are more physically consistent with illumination. Specifically, this is achieved through... Obtain the phase query vector, through Obtain the amplitude bond vector and amplitude value vector, and calculate the inverse cross-attention. The optimized phase features are obtained using the following formula. :

[0112]

[0113] in, This represents the optimized phase characteristics. Represents the phase query vector. Represents the amplitude bond vector. This represents the amplitude value vector.

[0114] This allows the enhancement of phase features (i.e., structure recovery) to benefit from accurate lighting priors, helping to recover more plausible details in shadowed or overexposed areas. The output is the further optimized phase feature. .

[0115] S2.5 Feature Fusion and Image Reconstruction.

[0116] S2.5.1 Amplitude Reconstruction: Reconstructing the modulated amplitude characteristics With optimized amplitude characteristics After fusion through a convolutional layer, the final amplitude spectrum is obtained. .

[0117] S2.5.2, Phase Reconstruction: The optimized phase features... The final phase representation is obtained through a convolutional layer and a Tanh activation function. .

[0118] S2.5.3, Inverse Fourier Transform: ... and The images are combined into a complex spectrum, and an inverse fast Fourier transform (IFFT) is performed to convert the image from the frequency domain back to the spatial domain, resulting in the final enhanced image. .

[0119] S3. Loss Function and Optimizer.

[0120] S3.1 In this embodiment, a multi-task loss function is designed to comprehensively supervise model training, including pixel-level reconstruction loss, perceptual quality loss, and frequency domain contrast regularization loss (this multi-task loss function is also effective for deraining and defogging).

[0121]

[0122] Among them, =1.0, =1, =0.2 is the weighting coefficient for each loss.

[0123] S3.1.1 Frequency Domain Comparison Regularization Loss:

[0124]

[0125] in, This represents the frequency domain contrast regularization loss. For the first Anchor point samples (input low-light image). In order to be with the first Each anchor point sample corresponds to a positive sample (corresponding to a normally lit image). Negative samples (other low-light images); It is a constant used for numerical stability. Represents a two-dimensional Fast Fourier Transform; For the first A set of negative sample indices for each anchor point sampling; The number of degraded images, The number of negative samples sampled for all anchor points. The core idea of ​​this loss is to bring low-light images closer to their corresponding normal-light images in the frequency domain, while simultaneously widening the distance between different low-light images.

[0126] S3.1.2, Pixel-level reconstruction loss:

[0127]

[0128] in, Indicates pixel-level reconstruction loss. This represents the enhanced image.

[0129] This pixel-level loss ensures that the enhanced image closely approximates a real, normally lit image at the pixel level.

[0130] S3.1.3, Perceived Loss:

[0131]

[0132] in, Indicates perceived loss. This indicates the first step of the pre-trained VGG-19 network. Layer features, ={relu1_2,relu2_2,relu3_3}, where relu1_2 is a shallow feature, corresponding to low-level visual information such as edges, textures, and colors; relu2_2 is a mid-level feature, corresponding to simple texture patterns and more complex edge combinations; and relu3_3 is a deeper feature, corresponding to high-level semantic information such as object parts, contours, and complex patterns. This perceptual loss is used to improve the visual quality of the augmentation result.

[0133] S3.2, Optimizer.

[0134] The model is implemented in PyTorch and trained using four NVIDIA RTX 3090 GPUs. The Adam optimizer (β1=0.9, β2=0.999) was used, with an initial learning rate of 1e-4 and a batch size of 8. The learning rate decayed by 0.5x every 50 epochs. The model converged after approximately 200 epochs. Data augmentation included random flipping, rotation, and color jitter. All images were uniformly scaled to 400×600 resolution during training.

[0135] S4. Experimental results.

[0136] This embodiment evaluates the performance of the method on the LOL-v1 and LSRW datasets. LOL-v1 consists of 789 training image pairs and 15 test image pairs, while LSRW consists of 3150 training image pairs and 20 test image pairs. Peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and learned perceptual patch similarity (LPIPS) are used as evaluation metrics for quantitative comparison. Generally, higher PSNR and SSIM, and lower LPIPS, indicate higher similarity between two images. Specific experimental results are shown in Table 1. Bold text in the table represents the best performance on the current dataset, and underlined text represents the second-best performance on the current dataset. As can be seen from Table 1, compared with state-of-the-art methods, this invention achieves the best performance in PSNR, SSIM, and LPIPS. Furthermore, the number of parameters in the model is also smaller than most methods.

[0137] Table 1 Comparative Experimental Results

[0138]

[0139] S4.1 Introduction to the comparison method.

[0140] RetinexNet is a deep learning method based on the classic Retinex theory. It decomposes an image into a reflection component (the object's intrinsic color and texture) and an illumination component, and estimates and enhances both separately. Comparative data shows that its PSNR is 18.91 and SSIM is 0.427 on the LOL-v1 dataset, indicating that as an early deep learning method, it has significant room for performance improvement in terms of fidelity and structure restoration.

[0141] UHDFour is a network specifically designed for 4K ultra-high resolution low-light images. It typically employs a multi-scale, reversible downsampling and upsampling architecture (such as using wavelet transform) to process both high-frequency and low-frequency information while reducing computational complexity. Its performance is significantly improved compared to RetinexNet, achieving a PSNR of 21.43 and an SSIM of 0.813 on LOL-v1, demonstrating its adaptability to high-resolution scenes.

[0142] FourLLIE is a representative method for low-light video enhancement. It not only processes data in the spatial domain but also utilizes temporal information (such as the consistency of illumination and motion between adjacent frames) to improve the temporal stability and image quality of the enhancement results. The table shows that its PSNR on LOL-v1 is 23.86, and its LPIPS is relatively low (0.112), indicating good performance in the visual quality of single-frame restoration.

[0143] Retinexformer is one of the first methods to combine the Transformer architecture with the Retinex model. It utilizes a self-attention mechanism to model dependencies between image patches in a non-local context, enabling more efficient decomposition of illumination and reflection components, particularly in handling complex lighting and shadows. It achieves a low LPIPS value (0.169) on the LSRW dataset, indicating that its output is perceptually closer to real images.

[0144] WaveletMamba is a cutting-edge approach that explores the application of state-space models (such as Mamba) in vision tasks. This method typically performs multi-resolution decomposition of images in the wavelet domain and leverages the long-sequence modeling capabilities of the Mamba structure to efficiently process frequency domain subbands, aiming to simultaneously optimize global illumination consistency and local detail recovery. It achieved high PSNR values ​​of 24.57 and SSIM values ​​of 0.853 on LOL-v1, demonstrating the potential of this approach.

[0145] S4.2 Comparison and Summary: This invention addresses the inherent shortcomings of existing methods by leveraging global modeling, semantic adaptability, and the depth and synergy of frequency domain information utilization. Experiments have demonstrated the model's superior performance.

[0146] S5. Reasoning and Application.

[0147] For a test image, its degradation type is first determined (low light, rain, or fog), and then it is input into the corresponding trained amplitude-phase dual-path collaborative semantic-aware image enhancement network for enhancement. This method can be directly applied to scenarios such as nighttime surveillance video frame enhancement and mobile phone low-light photography enhancement.

[0148] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A semantically perceptive image enhancement method based on amplitude-phase dual-path coordination, characterized in that, The amplitude-phase dual-path collaborative semantic-aware image enhancement method includes: The degraded image to be processed is acquired, and the frequency domain decomposition of the degraded image is performed to obtain the amplitude and phase components of the image. A semantically aware Mamba module is constructed; preliminary amplitude features are obtained by feature extraction of amplitude components, and the preliminary amplitude features are input into the Mamba module for semantic recombination and global dependency modeling. Based on the modeling results, a modulation map is generated, and the amplitude components are adaptively adjusted using the modulation map to obtain the modulated amplitude features. Phase components are feature extracted to obtain preliminary phase features. The degraded image is processed using a pre-defined edge structure information extraction network to obtain edge structure information. Enhanced phase features are obtained based on the preliminary phase features and edge structure information. The modulated amplitude features and enhanced phase features are subjected to bidirectional cross-attention for bidirectional information complementation and semantic alignment, resulting in optimized amplitude features and optimized phase features. The optimized amplitude and phase features are recombined and transformed into the spatial domain to obtain the enhanced image.

2. The semantically perceptive image enhancement method based on amplitude-phase dual-path coordination according to claim 1, characterized in that, The semantically aware Mamba module includes a semantic routing submodule and a selective state space modeling submodule. The semantic routing submodule includes a convolutional layer and a softmax function.

3. The semantically perceptive image enhancement method based on amplitude-phase dual-path coordination according to claim 2, characterized in that, The process involves extracting features from the amplitude components to obtain preliminary amplitude features, inputting these features into the Mamba module for semantic recombination and global dependency modeling, and generating a modulation map based on the modeling results. This includes: Preliminary amplitude features are obtained by performing preliminary feature mapping on the amplitude components through convolutional layers; The initial amplitude features are input into a semantically aware Mamba module to generate a semantic routing weight graph. The channels of the initial amplitude features are reordered based on the values ​​of the semantic routing weight graph to form semantic reorganization features; The selective state-space modeling submodule treats semantic recombination features as sequential data, performs long-distance dependency modeling through a selective state-space model, and outputs a semantically enhanced feature sequence. This process is described by the following state-space equation: in, For time step The hidden state, , , , For parameterized and input-dependent selectivity matrices, For time step semantic recombination features For time step The output sequence; The output sequences of all time steps are concatenated in order to obtain the semantically enhanced feature sequence, which is then mapped to a modulation map through a convolutional layer.

4. The semantically perceptive image enhancement method based on amplitude-phase dual-path coordination according to claim 3, characterized in that, The process of reordering the channels of the initial amplitude features based on the values ​​of the semantic routing weight map to form semantic reorganization features includes: The semantic routing weight map and the initial amplitude features are expanded into a one-dimensional vector in the spatial dimension to obtain the weight vector and the feature vector. Sort the elements in the weight vector in descending order to obtain the corresponding sort index; The expanded feature vectors are rearranged using a sort index to obtain the rearranged features; The rearranged features are restored to a two-dimensional spatial form, resulting in semantic recombination features.

5. The semantically perceptive image enhancement method based on amplitude-phase dual-path coordination according to claim 1, characterized in that, The enhanced phase features obtained based on preliminary phase features and edge structure information include: A cross-attention mechanism is used to fuse preliminary phase features and edge structure features to obtain the phase features after feature fusion; The phase features after feature fusion are residually connected with the initial phase features, and the enhanced phase features are output through an encoder-decoder network.

6. The semantically perceptive image enhancement method based on amplitude-phase dual-path coordination according to claim 1, characterized in that, The modulated amplitude features and enhanced phase features are subjected to bidirectional cross-attention for bidirectional information complementation and semantic alignment to obtain optimized amplitude features and optimized phase features, including: The amplitude query vector is obtained from the modulated amplitude features, and the phase bond vector and phase value vector are obtained from the enhanced phase features. The optimized amplitude features are then calculated, as expressed by the following formula: in, This represents the optimized amplitude characteristics. This represents the Softmax function. This represents the amplitude query vector. Represents the phase bond vector. Represents the phase value vector. For feature dimensions; The phase query vector is obtained through the enhanced phase features, and the amplitude key vector and amplitude value vector are obtained through the modulated amplitude features. The optimized phase features are calculated and expressed by the following formula: in, This represents the optimized phase characteristics. Represents the phase query vector. Represents the amplitude bond vector. This represents the amplitude value vector.

7. The semantically perceptive image enhancement method based on amplitude-phase dual-path coordination according to claim 1, characterized in that, The process of recombining the optimized amplitude and phase features and transforming them into the spatial domain to obtain the enhanced image includes: The modulated amplitude features are fused with the optimized amplitude features to obtain the amplitude spectrum; The optimized phase features are subjected to nonlinear mapping to obtain the final phase representation; The amplitude spectrum is combined with the final phase representation to form a complex spectrum, and an inverse fast Fourier transform is performed to obtain the enhanced image.

8. The semantically perceptive image enhancement method based on amplitude-phase dual-path coordination according to claim 1, characterized in that, The amplitude-phase dual-path collaborative semantic-aware image enhancement method is trained and optimized using a multi-task loss function, which includes frequency domain contrast regularization loss, pixel-level reconstruction loss, and perceptual loss. The frequency domain contrast regularization loss is expressed by the formula: in, This represents the frequency domain contrast regularization loss. For the first A degraded image, In order to be with the first A normal image corresponding to a degraded image. For other degraded images, It is a constant. This represents a two-dimensional Fast Fourier Transform. In order to target the The set of indices of other degraded images selected from the degraded image. The number of degraded images, The number of additional degraded images selected for all degraded images.

Images