Dark light image enhancement method and device based on multi-scale wavelet state space model

Abstract

The invention discloses a dark-light image enhancement method and device based on a multi-scale wavelet state space model, and relates to the technical field of computer vision and image processing, and the method comprises the steps: obtaining a to-be-processed dark-light image; performing down-sampling on the to-be-processed dark light image to obtain down-sampled images of multiple scales, performing convolution processing on the to-be-processed dark light image, and extracting image shallow layer features; inputting the image shallow layer features and the multi-scale down-sampling images into a multi-scale wavelet state space model to obtain a reconstructed residual image; and obtaining a final enhanced image according to the reconstructed residual image and the dark light image to be processed. According to the image enhancement method and device, the multi-scale wavelet UNet architecture and the state space model are fused in the multi-scale wavelet state space model, cross-scale global features can be efficiently captured, interference of noise on the state space model is avoided, and the calculation efficiency and the enhancement effect of image enhancement are improved.

Description

Technical Field

[0001] This application relates to the fields of computer vision and image processing technology, and in particular to a method and apparatus for enhancing low-light images based on a multi-scale wavelet state-space model. Background Technology

[0002] Low-light image enhancement, a classic task in computer vision, aims to significantly improve the visual quality of images created in low-light environments by using advanced image processing techniques to raise the brightness to normal lighting levels. Images created in low light often suffer from increased noise, loss of detail, and color distortion. These factors not only degrade the visual quality of the image but can also negatively impact subsequent image analysis and processing tasks. Therefore, developing efficient and robust low-light image enhancement algorithms is crucial not only for directly affecting human visual perception but also for high-level vision-based applications.

[0003] Numerous methods have been proposed to address the problem of low-light image enhancement. Early methods based on prior knowledge typically relied on model-driven strategies, utilizing prior information from normally lit images or specific features of low-light images to guide the enhancement process. Due to the ill-posed nature of the low-light image enhancement problem, traditional methods often depend on manually designed priors based on image statistics to reconstruct the image, such as methods based on Retinex theory, histograms, and dehazing. However, these methods often rely on empirical observations, leading to complex optimization problems and introducing artifacts into the enhanced image, such as color distortion or over-enhancement.

[0004] Subsequently, the rise of deep learning greatly propelled the development of low-light image enhancement. Convolutional neural network (CNN)-based methods excelled at capturing the complex mapping between low-light and normal-light images, enabling them to effectively handle images under varying lighting conditions. For example, RetinexNet combined Retinex theory with deep CNNs in a unified end-to-end learning framework. Zero-DCE formulated the low-light image enhancement task as an image-specific curve estimation task, generating high-order curves for pixel-level dynamic range adjustment to obtain enhanced images. Although these CNN-based methods outperformed prior knowledge-based models, the spatial invariance and locality of convolutional operations limited their ability to capture global and spatially varying information.

[0005] To overcome these limitations, a Transformer-based approach has been proposed, bringing a new research paradigm to the low-light image enhancement problem. Unlike convolutional neural networks, the Transformer can model global contextual information through a self-attention mechanism, which is crucial for achieving high-quality image reconstruction. For example, LLFormer uses a Transformer-based model for ultra-high-resolution low-light image enhancement. However, the self-attention mechanism in the Transformer requires computational complexity quadratic with the label size, significantly increasing the computational cost when processing high-resolution images, which is unacceptable. Therefore, it is necessary to explore a method that can effectively capture non-local information without significantly increasing computational cost, thereby achieving high-quality low-light image enhancement.

[0006] With the rise of state-space models, Mamba-based models have been widely adopted in computer vision to balance efficiency and performance. For example, MambaIR combines convolution and channel attention to enhance the Mamba architecture's ability to preserve local information. MambaLLIE improves local dependencies through enhanced local bias in a 2D selective scanning mechanism and an implicit Retinex perceptual selective kernel module. These studies demonstrate that Mamba models can maintain linear computational complexity while modeling long-range dependencies, exhibiting significant advantages. However, despite the existence of Mamba-based methods for low-light image enhancement, certain limitations still exist. Summary of the Invention

[0007] The purpose of this application is to provide a method and apparatus for enhancing low-light images based on a multi-scale wavelet state-space model, which can improve the computational efficiency and enhancement effect of image enhancement.

[0008] To achieve the above objectives, this application provides the following solution:

[0009] In a first aspect, this application provides a method and apparatus for enhancing low-light images based on a multi-scale wavelet state-space model, including:

[0010] Obtain the low-light image to be processed;

[0011] The low-light image to be processed is downsampled to obtain downsampled images at multiple scales, and then convolution is performed on the low-light image to extract shallow features of the image.

[0012] The shallow features of the image and downsampled images at multiple scales are input into the multi-scale wavelet state space model to obtain the reconstructed residual image; the multi-scale wavelet UNet architecture and the state space model are integrated in the multi-scale wavelet state space model.

[0013] The final enhanced image is derived from the reconstructed residual image and the dark-light image to be processed.

[0014] Secondly, this application provides a low-light image enhancement device based on a multi-scale wavelet state-space model, comprising:

[0015] The image acquisition module is used to acquire the low-light image to be processed;

[0016] The downsampling and feature extraction module is used to downsample the low-light image to be processed, obtain downsampled images at multiple scales, and perform convolution processing on the low-light image to extract shallow features of the image.

[0017] The image reconstruction module is used to input shallow features of the image and downsampled images at multiple scales into the multi-scale wavelet state space model to obtain the reconstructed residual image; the multi-scale wavelet UNet architecture and the state space model are integrated in the multi-scale wavelet state space model.

[0018] The image enhancement module is used to derive the final enhanced image from the reconstructed residual image and the dark-light image to be processed.

[0019] Thirdly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described low-light image enhancement method based on a multi-scale wavelet state-space model.

[0020] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned low-light image enhancement method based on a multi-scale wavelet state-space model.

[0021] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned low-light image enhancement method based on a multi-scale wavelet state-space model.

[0022] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0023] This application provides a low-light image enhancement method and apparatus based on a multi-scale wavelet state-space model. The method involves downsampling the low-light image to be processed, obtaining downsampled images at multiple scales, and then performing convolution processing on the low-light image to extract shallow features. These shallow features and the downsampled images at multiple scales are input into the multi-scale wavelet state-space model to obtain a reconstructed residual image. The multi-scale wavelet UNet architecture and the state-space model are integrated within the multi-scale wavelet state-space model. The final enhanced image is obtained based on the reconstructed residual image and the low-light image to be processed. This application uses a deep learning algorithm based on a multi-scale wavelet state-space model as its core. By combining the multi-scale wavelet UNet architecture with the state-space model, it explicitly mines complementary features across scales, efficiently captures global information, and avoids noise interference with the state-space model. This effectively solves the problems of insufficient global and local feature capture and high computational complexity in existing low-light image enhancement methods, improving the computational efficiency of image enhancement, enhancing image restoration quality, and ensuring the image enhancement effect. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is an application environment diagram of a low-light image enhancement method based on a multi-scale wavelet state-space model according to an embodiment of this application;

[0026] Figure 2 A flowchart illustrating a low-light image enhancement method based on a multi-scale wavelet state-space model, provided as an embodiment of this application;

[0027] Figure 3 A schematic diagram illustrating the technical concept of a low-light image enhancement method based on a multi-scale wavelet state-space model, provided in an embodiment of this application;

[0028] Figure 4 This is a schematic diagram of the structure of a multi-scale wavelet state-space model provided in an embodiment of this application;

[0029] Figure 5 This is a schematic diagram of the structure of a selective kernel feature fusion module provided in an embodiment of this application;

[0030] Figure 6 This is a schematic diagram of the structure of a low-frequency state space module provided in an embodiment of this application;

[0031] Figure 7 This is a schematic diagram of the structure of a visual state space module provided in an embodiment of this application;

[0032] Figure 8 This is a schematic diagram of the structure of a neighborhood sensing scanning module provided in an embodiment of this application;

[0033] Figure 9 A schematic diagram of the functional modules of a low-light image enhancement device based on a multi-scale wavelet state-space model provided in an embodiment of this application;

[0034] Figure 10 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0035] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0036] Low-light image enhancement aims to improve the quality of images captured under low-light conditions, thereby enhancing their visual clarity and the accuracy of perceptual systems. While Mamba-based methods have been proposed for low-light image enhancement, they typically rely on multi-directional scanning mechanisms to model global information. This operation inevitably increases computational complexity, especially for high-resolution images. Furthermore, some discontinuous paths in the scanning sequence can disrupt the spatial structure of the image, making it difficult to capture relevant information from adjacent pixels. More importantly, the unidirectional modeling nature of the state-space model makes it susceptible to noise, hindering the application of Mamba in low-light scenes containing complex noise.

[0037] To address the issues of noise interference and high computational complexity of the models, which impose a series of computational burdens on some hardware training or deployment operators, this application provides a low-light image enhancement method and apparatus based on a multi-scale wavelet state-space model. This method efficiently captures cross-scale global features by combining a multi-scale wavelet UNet architecture with a state-space model. A simple yet effective multi-scale wavelet transform strategy is designed to better extract latent explicit information at different scales and avoid noise interference with the state-space model. Furthermore, a dynamic semantic clustering mechanism is developed, which can achieve global perception capabilities across image regions while maintaining the efficiency advantage of a single scan.

[0038] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, this application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] The low-light image enhancement method based on a multi-scale wavelet state-space model provided in this application can be applied to, for example... Figure 1 The application environment shown illustrates this. The terminal communicates with the server via a network. A data storage system stores the data the server needs to process. This system can be set up independently, integrated into the server, or located in the cloud or on another server. The terminal sends the low-light image to be processed to the server. Upon receiving the image, the server downsamples it to obtain downsampled images at multiple scales and performs convolution processing to extract shallow features. These shallow features and the downsampled images at multiple scales are then input into a multi-scale wavelet state space model to obtain a reconstructed residual image. The multi-scale wavelet state space model integrates the multi-scale wavelet UNet architecture with the state space model. Based on the reconstructed residual image and the low-light image to be processed, the final enhanced image is obtained. The server can then feed back the final enhanced image to the terminal. In addition, in some embodiments, the dark light image enhancement method based on the multi-scale wavelet state space model can also be implemented by the server or the terminal alone. For example, the terminal can directly perform dark light image enhancement based on the multi-scale wavelet state space model on the dark light image to be processed, or the server can obtain the dark light image to be processed from the data storage system and perform dark light image enhancement based on the multi-scale wavelet state space model.

[0040] The terminals can be, but are not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, and smart in-vehicle systems. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted devices. Servers can be implemented using independent servers, server clusters composed of multiple servers, or cloud servers.

[0041] In one exemplary embodiment, such as Figure 2 and Figure 3 As shown, a low-light image enhancement method based on a multi-scale wavelet state-space model is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 The following steps, 101 to 104, are used as an example to illustrate the process of using a server in the example.

[0042] Step 101: Obtain the low-light image to be processed.

[0043] Step 102: Downsample the dark light image to be processed to obtain downsampled images at multiple scales, and perform convolution processing on the dark light image to be processed to extract shallow features of the image.

[0044] like Figure 4 As shown, the low-light image to be processed is divided into three scales using a coarse-to-fine technique, generating images at three different scales. Specifically, the input image is downsampled to 1 / 2, 1 / 4, and 1 / 8 scales using an interpolation algorithm, generating a pyramid-shaped multi-scale low-light image, providing a foundation for subsequent image processing tasks. Simultaneously, shallow features of the original image (the low-light image to be processed) are extracted using 3×3 convolution, generating shallow features F0∈R. H×W×C , where H and W represent the height and width of the image, respectively, and C is the number of channels.

[0045] Step 103: Input the shallow features of the image and downsampled images at multiple scales into the multi-scale wavelet state space model to obtain the reconstructed residual image; the multi-scale wavelet UNet architecture and the state space model are integrated in the multi-scale wavelet state space model.

[0046] Step 104: The final enhanced image is obtained based on the reconstructed residual image and the dark light image to be processed.

[0047] Specifically, the reconstructed residual image is added to the original dark-light image to obtain the final enhanced image.

[0048] By implementing steps 101 to 104 above, this application aims to improve task efficiency and image restoration quality. By fusing a multi-scale wavelet UNet architecture with a state-space model, it efficiently captures global features across scales. A concise and effective multi-scale wavelet transform strategy is designed to accurately extract latent explicit information at each scale, while avoiding noise interference with the state-space model. This effectively solves the problems of insufficient global and local feature capture and high computational complexity in existing low-light image enhancement methods, improving the computational efficiency of image enhancement, enhancing image restoration quality, and ensuring the image enhancement effect.

[0049] In another exemplary embodiment of this application, step 103, the multi-scale wavelet state-space model specifically includes: N branches, Figure 4 An example is shown, employing three branches; each branch includes an encoder and a decoder.

[0050] The encoder includes a discrete wavelet transform module, a first low-frequency state space module, a selective kernel feature fusion module, and a first high-frequency Transformer module; the decoder includes a second high-frequency Transformer module, a convolution module, a second low-frequency state space module, and an inverse wavelet transform module.

[0051] The input to the discrete wavelet transform module of the first branch is the shallow features of the image; the output of the discrete wavelet transform module is connected to the input of the first low-frequency state space module and the selective kernel feature fusion module, respectively; the output of the first low-frequency state space module is connected to the input of the first high-frequency Transformer module and the input of the discrete wavelet transform module in the next branch of its own branch, respectively; the input of the first low-frequency state space module of the nth branch also includes the downsampled image at the corresponding scale; n = 1, 2, 3, ..., N; N is the number of scale types of the downsampled image.

[0052] The output of the selective kernel feature fusion module is connected to the input of the first high-frequency Transformer module; the output of the first high-frequency Transformer module is connected to the input of the second high-frequency Transformer module; the output of the second high-frequency Transformer module is connected to the input of the inverse wavelet transform module after passing through a convolution module; the input of the second low-frequency state space module is connected to the output of the inverse wavelet transform module in the next branch of its own branch; the output of the second low-frequency state space module is connected to the inputs of the second high-frequency Transformer module and the inverse wavelet transform module respectively; the input of the second low-frequency state space module of the last branch is connected to the output of the first low-frequency state space module of its own branch.

[0053] The output of the inverse wavelet transform module in the first branch is processed by convolution to obtain the reconstructed residual image.

[0054] The discrete wavelet transform module is used to extract low-frequency and high-frequency features of the image at the corresponding scale based on the input image.

[0055] The first low-frequency state space module is used to enhance the low-frequency features and obtain the enhanced low-frequency features.

[0056] The first high-frequency Transformer module is used to enhance high-frequency features and obtain enhanced high-frequency features.

[0057] Based on wavelet transform, a multi-scale wavelet UNet architecture is designed for deep feature extraction. Low-frequency features are generated through 2D discrete wavelet transform operations, such as... Figure 4As shown, the three discrete wavelet transform modules correspond to downsampling to 1 / 2, 1 / 4, and 1 / 8 scales, respectively; simultaneously, high-frequency features are generated, and channel compression is performed through a selective kernel feature fusion module. That is, wavelet transform is used to downsample and obtain low-frequency and high-frequency features at 1 / 2, 1 / 4, and 1 / 8 scales.

[0058] Each layer of low-frequency features is combined with a downsampled image at the corresponding scale. Global information is refined by stacking low-frequency state space modules to obtain enhanced low-frequency features. High-frequency features are recovered using a high-frequency Transformer module. The decoder then upsamples the enhanced low-frequency and high-frequency features using inverse wavelet transform to obtain the reconstructed residual image. The decoder takes highly condensed, abstract features as input. The second high-frequency Transformer module and the second low-frequency state space module are designed to process the abstract features more smoothly and structurally, better integrating low-frequency and high-frequency information into the residual image.

[0059] This application significantly enhances the ability to recover local details by finely optimizing global information through a low-frequency state-space module and dynamically recovering high-frequency details using a high-frequency Transformer module. Furthermore, by introducing wavelet transform to separate low-frequency and high-frequency features, the application effectively reduces the impact of noise on the state-space model, thereby achieving efficient low-light image enhancement.

[0060] Specifically, the selective kernel feature fusion module dynamically integrates multi-resolution features through self-attention: it extracts global channel information from multi-scale features, generates branch weights by convolution compression-expansion, calculates attention coefficients at each scale through softmax, and finally weights and fuses multi-resolution features to achieve adaptive enhancement of high-frequency features in a lightweight manner.

[0061] The discrete wavelet transform divides a single channel C into four channels 4C, comprising one low-frequency information channel and three high-frequency information channels (horizontal, vertical, and diagonal). The selective kernel feature fusion module then fuses these three high-frequency information channels into a single channel. The execution process of the selective kernel feature fusion module includes two stages. For example... Figure 5 As shown, the first stage is the fusion operation, the core of which is to generate a global information descriptor that can guide feature selection. This process begins by combining the input feature maps from three parallel convolutional streams (three high-frequency information, corresponding to...) Figure 5 In The input feature maps are merged into an aggregated feature map L by element-wise summation. Then, global average pooling (GAP) is applied to the aggregated feature map L to compute channel statistics s representing global information. To obtain a more compact feature representation, this statistics are passed through a channel-reduction convolutional layer to generate a feature vector z. Finally, this vector z is passed through three parallel channel-incrementing convolutional layers to generate dedicated feature descriptors v1, v2, and v3 for each original branch. The second stage is the selection operation, which aims to recalibrate and aggregate the feature maps using the descriptors generated in the previous step. Specifically, this operation first applies the Softmax function to the feature descriptors v1, v2, and v3 to produce a set of adaptive attention activation weights s1, s2, ..., s3. These weights are then used to combine the original input feature maps (...) and (...) respectively. Multiplication is performed to recalibrate the features. Finally, all weighted feature maps are summed to form the final output feature map U. The entire process of the selective kernel feature fusion module realizes dynamic adjustment of the receptive field and is a feature aggregation method based on a self-attention mechanism.

[0062] In another exemplary embodiment of this application, the low-frequency state-space module models global features using a state-space model, specifically as follows: A state-space model is a mathematical framework commonly used in time series analysis and control systems. State equations describe the evolution of the basic system over time, characterizing the relationship between the system's hidden states and its temporal dynamics. The input signal x(t)∈R is represented by the implicit latent state h(t)∈R. N Mapped to the output response y(t)∈R, this can be represented by the following set of first-order linear ordinary differential equations:

[0063] h'(t) = Ah(t) + Bx(t)

[0064] y(t)=Ch(t)+Dx(t)

[0065] Where A, B, C, and D are learnable weight matrices.

[0066] Subsequently, a discretization process is typically used to integrate the above equations into the actual deep learning algorithm. Specifically, let Δ be the time scale parameter, used to convert continuous parameters A and B into discrete parameters. A commonly used discretization method is the zero-order preservation rule, whose expression is as follows:

[0067]

[0068] y t =Ch t +Dx t

[0069]

[0070] in, Figure 8 middle The expression for ΔB corresponds to the one here. and The form of expression.

[0071] However, the above formula only applies to linear time-invariant systems where parameters remain unchanged under different input conditions. To overcome this theoretical limitation, Mamba innovatively improved the traditional state-space module, proposing a state-space model optimization method based on a selective scanning mechanism. This selective scanning mechanism dynamically adjusts the state transition process by introducing task-related recovery prior knowledge, thereby achieving input-dependent parameter optimization while maintaining linear computational complexity. Specifically, the selective scanning mechanism adaptively selects key state paths and dynamically adjusts state transition weights based on the local features of the input data, thus capturing long-range dependencies more efficiently.

[0072] The first and second low-frequency state space modules have the same structure, but process different data. The low-frequency state space module described in this text refers to both the first and second low-frequency state space modules. Figure 6 As shown, the first low-frequency state space module includes: a first normalization layer, a visual state space module, a first additive layer, a second normalization layer, a convolutional layer, a channel attention layer, and a second additive layer, connected in sequence.

[0073] The input of the first adder layer is also connected to the input of the first normalization layer; the input of the second adder layer is also connected to the output of the first adder layer.

[0074] like Figure 6 As shown, the specific implementation of the low-frequency state-space module is as follows: given the input depth feature F in ∈R H×W×C First, layer normalization is used, followed by capturing long-range spatial dependencies through a visual state space module. To improve network performance, a learnable scaling factor α is introduced into the residual connections, with the specific expression as follows:

[0075] F mid =α·VSSB(LN(F in ))+f in

[0076] Among them, F mid LN() represents intermediate features; LN() represents layer normalization; VSSB() represents the visual state space module.

[0077] Next, another layer of normalization is used for F. midNormalization is performed, and a 3×3 convolutional layer is used to model spatial local similarity priors. Finally, the feature representation is further refined through a channel attention mechanism to obtain the final output F of the low-frequency state space module. out , can be represented as:

[0078] F out =CA(Conv(LN(F) mid )))+F mid

[0079] Here, Conv() and CA() represent 3×3 convolution and channel attention mechanisms, respectively.

[0080] In another exemplary embodiment of this application, such as Figure 7 As shown, the visual state space module includes a first linear layer, a depth-separable convolutional layer, a first activation layer, a neighborhood-aware scanning module, a third normalization layer, a first multiplication layer, and a second linear layer connected in sequence.

[0081] The visual state space module also includes a third linear layer and a second activation layer connected in series; the input of the third linear layer is connected to the input of the first linear layer; and the input of the second activation layer is connected to the input of the first multiplication layer.

[0082] like Figure 7 As shown, for the visual state space module, the input features are processed through two parallel branches. The complete operation of the visual state space module can be described as follows:

[0083] First, in the first branch, the input feature X0 undergoes a linear transformation, followed by local feature extraction through depthwise convolution. Then, a non-linear transformation is performed using the SiLU activation function. Next, a neighborhood-aware scanning module further captures neighborhood information, and finally, layer normalization yields the branch output X1. In the second branch, the input feature X0 undergoes a direct linear transformation, followed by a non-linear transformation using the SiLU activation function, resulting in the branch output X2. Finally, the outputs X1 and X2 from both branches are multiplied by a Hadamard product, and then a linear transformation is performed to obtain the final output X3 of the visual state space module. The processing procedure of the visual state space module is represented as follows:

[0084] X1=LN(NESB(SiLU(DWConv(Linear(X0)))))

[0085] X2 = SiLU(Linear(X0))

[0086]

[0087] Where DWConv() represents depthwise convolution, and SiLU represents the SiLU activation function. NESB() represents the Hadamard product, and NESB() represents the neighborhood sensing scanning module. Linear() represents the linear transformation.

[0088] In another exemplary embodiment of this application, a highly efficient neighborhood-aware scanning module is proposed to address the long-distance attenuation problem in multi-directional scanning of existing Mamba-based low-light image enhancement methods. This module aims to fully utilize the information of distant but similar pixels in the scanning sequence, thereby improving image restoration performance. Figure 8 As shown, the specific processing procedure of the neighborhood sensing scanning module is as follows:

[0089] (1) Unfolding and position coding layer, used to unfold the input two-dimensional image features into a one-dimensional sequence and embed position coding.

[0090] The two-dimensional image is unfolded into a one-dimensional flattened sequence (because Mamba processes one-dimensional sequences), and positional encoding is introduced to preserve spatial information. The features after positional encoding become the input of the state-space model.

[0091] (2) Hint pool construction layer, used to construct a hint pool matrix for storing semantic features.

[0092] Inspired by visual cue learning, a cue pool was designed. Where N represents the number of cue components, and d is the dimension of the hidden states in the Mamba model. The cue pool itself is a parameterized matrix, and its weights can be continuously learned and optimized during model training, resulting from continuous optimization after initialization. The correlation between semantic features and low-light images is achieved by dynamically learning the image, classifying the image pixels into different categories, improving the correlation between pixels within each category, and facilitating image processing by the Mamba model.

[0093] (3) A prompt weight generation layer is used to generate a hard-assigned routing matrix from pixels to semantic categories by applying a Gumbel-Softmax-driven differentiable routing mechanism to the location-encoded features. Figure 8 The prompt weight in the middle.

[0094] (4) Dynamic prompt generation layer, which is used to multiply the hard-assigned routing matrix with the prompt pool matrix to obtain dynamic prompt P carrying cross-region association information.

[0095] (5) The state space equation update layer is used to take the location-encoded features as the input of the state space model and inject the dynamic cue P into the output matrix C of the state space equation through residual superposition, thereby updating the state space equation and ensuring more efficient use of pixel information.

[0096] After injecting the dynamic hint P, the modified state-space equations are as follows:

[0097]

[0098] y t =(C+P)h t +Dx t .

[0099] (6) Neighborhood sensing layer, used to derive neighborhood sensing feature map based on the updated state space equation.

[0100] (7) Folding layer, used to fold the neighborhood-aware feature map to obtain the folded neighborhood-aware feature map; the folded neighborhood-aware feature map is the input of the third normalization layer.

[0101] The unfolding process described above records the position encoding, and the folding process uses the position encoding to perform directional compression to restore the image data structure.

[0102] In the neighborhood awareness module, the input x is used to generate an approximate one-hot encoded routing matrix using the gumble-softmax technique. This routing matrix is ​​multiplied with the cue pool to obtain a dynamic cue P. The vector P is then injected into the Mamba output equation using residual addition, providing global information for each element. Furthermore, during the expansion process of the neighborhood awareness module, the routing matrix generated by the expansion and position encoding layers and the cue weight generation layer aggregates elements with the same semantic category. This facilitates the learning and memorization of semantically similar elements within the context of long sequences, addressing the problem that discontinuous paths in existing scanning sequences may disrupt the spatial structure of the image, making it difficult to capture relevant information from adjacent pixels. Additionally, the neighborhood awareness module is essentially a unidirectional scan, avoiding redundant computations associated with multidirectional scans and reducing computational complexity.

[0103] The neighborhood-aware scanning module reconstructs pixel associations through a dynamic semantic clustering mechanism (steps (1) to (5) above). Its core is to dynamically aggregate semantically similar pixels in the image and establish feature associations. This mechanism achieves global perception capabilities across image regions while maintaining the efficiency advantage of a single scan.

[0104] The low-frequency state space module incorporates a dynamic semantic clustering mechanism to capture global feature information with linear complexity. The module models global features using a state space model, achieving linear complexity as follows: the continuous state space equation is discretized, and a neighborhood-aware scanning module dynamically captures long-distance dependencies. Efficient computation is achieved through differentiable convolution kernels and parallel scanning algorithms, thus maintaining linear time complexity while enabling multi-scale modeling of low-frequency temporal features and cross-period information fusion.

[0105] By combining a low-frequency state space module with a neighborhood-aware scanning module to dynamically capture long-distance dependencies, global perception capabilities across image regions are achieved. The neighborhood-aware scanning module reconstructs pixel relationships through a dynamic semantic clustering mechanism. Its core lies in dynamically aggregating semantically similar pixels in the image and establishing feature associations, thereby achieving global perception capabilities across image regions while maintaining the efficiency advantage of a single scan.

[0106] In another exemplary embodiment of this application, the high-frequency Transformer module achieves optimized recovery of high-frequency features and precise enhancement of local information through low-frequency feature matching, cross-frequency attention fusion, and hierarchical feature correction. The first and second high-frequency Transformer modules have the same structure; the high-frequency Transformer module described herein refers to both the first and second high-frequency Transformer modules. The first high-frequency Transformer module includes a low-frequency feature matching module, a cross-frequency attention fusion module, and a hierarchical feature correction module.

[0107] The low-frequency feature matching module is used to calculate the similarity between the input high-frequency and low-frequency features and to filter the low-frequency feature segments that are most relevant to the high-frequency features. Specifically, it measures the correlation between high-frequency and low-frequency features using cosine similarity and uses a top-k filtering mechanism to select the most representative low-frequency feature segments, ensuring that low-frequency information can effectively guide the recovery of high-frequency features.

[0108] The cross-frequency attention fusion module is used to concatenate matched low-frequency segments with high-frequency features, and generates dynamic weights through bi-branch convolution to fuse cross-frequency domain information. Specifically, parallel 1×1 convolutional layers are used to extract channel information of low-frequency and high-frequency features respectively, and dynamic weights are generated through the softmax function to weightedly fuse low-frequency and high-frequency features, thereby achieving adaptive integration of cross-frequency domain information.

[0109] The hierarchical feature correction module is used to re-embed low-frequency guiding information into the attention-weighted high-frequency features, and further adjust the high-frequency details through a feedforward network. Specifically, the fused features are input into a multilayer perceptron, and the high-frequency features are corrected layer by layer by combining the global information of the low-frequency features, dynamically supplementing high-frequency details, solving the problem of easy loss of high-frequency components, and strengthening edge texture expression to improve image clarity and detail recovery capabilities.

[0110] In this application, the proposed multi-scale wavelet state-space model consists of a three-layer UNet structure. It generates multi-scale high-frequency and low-frequency features through wavelet transform, captures global information using a low-frequency state-space module combined with a dynamic semantic clustering mechanism, and recovers local details through a high-frequency Transformer module. Finally, the enhanced features are upsampled using inverse wavelet transform to obtain a reconstructed residual image, which is then fused with the original image to achieve efficient low-light image enhancement.

[0111] This application has the following beneficial effects:

[0112] 1. This application takes a deep learning algorithm based on a multi-scale wavelet state space model as its core. By combining the multi-scale wavelet UNet architecture with the state space model, it explicitly mines cross-scale complementary features, efficiently captures global information, and avoids noise interference to the state space model.

[0113] 2. This application, by combining a domain-aware scanning module with a state-space model, significantly reduces computational complexity compared to traditional Transformer model methods. While maintaining high performance, this application is more suitable for deployment on resource-constrained mobile or embedded devices, exhibiting high practicality and scalability.

[0114] 3. This application achieves collaborative optimization of global and local features by modeling global features using a low-frequency state space module and dynamically recovering high-frequency details using a high-frequency Transformer module. The low-frequency state space module dynamically captures long-distance dependencies through a state space model and a neighborhood-aware scanning module, while the high-frequency Transformer module dynamically supplements high-frequency details through low-frequency feature matching, cross-frequency attention fusion, and hierarchical feature correction, significantly improving the image's edge texture representation and detail recovery capabilities.

[0115] 4. This application introduces a dynamic semantic clustering mechanism. Through a Gumbel-Softmax-driven differentiable routing mechanism, it dynamically aggregates semantically similar pixels in the image and establishes feature associations. While maintaining the efficiency advantage of a single scan, it achieves global perception capabilities across image regions. This mechanism effectively improves the long-range dependency modeling capability of the state-space model and further enhances the global consistency of image restoration.

[0116] To further verify the effectiveness of the image enhancement method proposed in this application, the low-light enhancement effect of the multi-scale wavelet state-space model was evaluated on low-light image enhancement datasets such as UHD-LL. On the aforementioned datasets, peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and perceptual loss (LPIPS) were used as core evaluation metrics to achieve quantitative analysis and comparison of image quality.

[0117] In this application, peak signal-to-noise ratio (PSNR) is used to measure the difference between two images, such as a compressed image and the original image, to evaluate the quality of the compressed image; or a restored image and the original image, to evaluate the performance of the restoration algorithm, etc. Its calculation formula is as follows:

[0118]

[0119] Where MSE is the mean square error per pixel of the two images. n represents the bit depth of the image, which is the number of bits per pixel.

[0120] In this application, the structural similarity index is a metric based on the perceptual model of the human visual system, used to measure the similarity between two images in terms of brightness, contrast, and structure. Unlike peak signal-to-noise ratio (PSNR), the structural similarity index more closely approximates human visual perception and can more accurately reflect image quality. Its formula can be expressed as:

[0121]

[0122] Where, μ x and μ y These are the average values ​​of image x and y within a local window, representing the brightness level of the image; and σ and y are the variances of the image x and y within the local window, respectively, representing the image contrast; xy C1 represents the covariance of images x and y within a local window, indicating the structural similarity of the images; C1 = (K1L) 2 C2 = (K2L) 2 Typically, K1 = 0.01, K2 = 0.03, and L is the dynamic range of pixel values.

[0123] In this application, perceptual loss is used to further evaluate image quality, particularly in measuring perceptual similarity. Perceptual loss is a deep learning-based metric that extracts image features through a pre-trained convolutional neural network and calculates distances in the feature space, thus more closely approximating human visual perception. Unlike peak signal-to-noise ratio and structural similarity index, perceptual loss better captures perceptual differences in images, especially in terms of detail and texture. Its calculation formula is as follows:

[0124]

[0125] Where, φ l (x) and φ l (y) represent the feature representations of images x and y in the l-th layer of the neural network, respectively. l These are the weight coefficients of the l-th layer.

[0126] Table 1 shows the performance comparison results of different image enhancement algorithms. By combining peak signal-to-noise ratio, structural similarity index, and perceptual loss, this application can comprehensively evaluate image quality from three dimensions: pixel level, structural level, and perceptual level, providing a more comprehensive performance analysis for tasks such as image compression and restoration.

[0127] Table 1. Performance comparison results of different image enhancement algorithms

[0128]

[0129] Among them, the higher the PSNR and SSIM indexes, the better, and the lower the LPIPS index, the better. Finally, it was concluded that the image enhancement algorithm of this application outperforms the mainstream low-light image enhancement algorithms with 1M parameters.

[0130] This application also provides an application scenario in which the aforementioned low-light image enhancement method based on a multi-scale wavelet state-space model is applied. Specifically, the low-light image enhancement method based on a multi-scale wavelet state-space model provided in this embodiment can be applied in autonomous driving scenarios. This scenario includes an image acquisition stage, a low-light image enhancement stage, and a target recognition stage. The image acquisition stage is used to acquire images of autonomous vehicles in traffic scenarios (e.g., low-light images at night). The low-light image enhancement stage is used to perform image enhancement processing on the acquired low-light images to obtain enhanced images. The target recognition stage is used to identify targets in traffic scenarios based on the enhanced images for autonomous driving control. The low-light image enhancement method based on a multi-scale wavelet state-space model provided in this embodiment belongs to the low-light image enhancement stage.

[0131] Based on the same inventive concept, this application also provides a device for enhancing low-light images based on a multi-scale wavelet state-space model to implement the aforementioned method for enhancing low-light images based on a multi-scale wavelet state-space model. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the device for enhancing low-light images based on a multi-scale wavelet state-space model provided below can be found in the limitations of the method for enhancing low-light images based on a multi-scale wavelet state-space model described above, and will not be repeated here.

[0132] In one exemplary embodiment, such as Figure 9 As shown, a low-light image enhancement device based on a multi-scale wavelet state-space model is provided, comprising:

[0133] The image acquisition module M1 is used to acquire the low-light image to be processed.

[0134] The downsampling and feature extraction module M2 is used to downsample the low-light image to be processed, obtain downsampled images at multiple scales, and perform convolution processing on the low-light image to extract shallow features of the image.

[0135] The image reconstruction module M3 is used to input shallow features of the image and downsampled images at multiple scales into the multi-scale wavelet state space model to obtain the reconstructed residual image. The multi-scale wavelet state space model integrates the multi-scale wavelet UNet architecture and the state space model.

[0136] The image enhancement module M4 is used to derive the final enhanced image based on the reconstructed residual image and the dark light image to be processed.

[0137] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 10 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data based on a multi-scale wavelet state-space model and the final enhanced image. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a low-light image enhancement method based on a multi-scale wavelet state-space model.

[0138] Those skilled in the art will understand that Figure 10 The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0139] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0140] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0141] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0142] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0143] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0144] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0145] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method and apparatus for enhancing low-light images based on a multi-scale wavelet state-space model, characterized in that, include: Obtain the low-light image to be processed; The low-light image to be processed is downsampled to obtain downsampled images at multiple scales, and then convolution is performed on the low-light image to extract shallow features of the image. The shallow features of the image and downsampled images at multiple scales are input into the multi-scale wavelet state space model to obtain the reconstructed residual image; the multi-scale wavelet UNet architecture and the state space model are integrated in the multi-scale wavelet state space model. The final enhanced image is derived from the reconstructed residual image and the dark-light image to be processed.

2. The low-light image enhancement method and apparatus based on a multi-scale wavelet state-space model according to claim 1, characterized in that, The multi-scale wavelet state-space model specifically includes: N branches; each branch includes an encoder and a decoder; The encoder includes a discrete wavelet transform module, a first low-frequency state space module, a selective kernel feature fusion module, and a first high-frequency Transformer module; the decoder includes a second high-frequency Transformer module, a convolution module, a second low-frequency state space module, and an inverse wavelet transform module. The input to the discrete wavelet transform module of the first branch is the shallow features of the image; the output of the discrete wavelet transform module is connected to the input of the first low-frequency state space module and the selective kernel feature fusion module, respectively; the output of the first low-frequency state space module is connected to the input of the first high-frequency Transformer module and the input of the discrete wavelet transform module in the next branch of its own branch, respectively; the input of the first low-frequency state space module of the nth branch also includes the downsampled image at the corresponding scale; n = 1, 2, 3, ..., N; N is the number of scale types of the downsampled image; The output of the selective kernel feature fusion module is connected to the input of the first high-frequency Transformer module; the output of the first high-frequency Transformer module is connected to the input of the second high-frequency Transformer module; the output of the second high-frequency Transformer module is connected to the input of the inverse wavelet transform module after passing through a convolution module; the input of the second low-frequency state space module is connected to the output of the inverse wavelet transform module in the next branch of its own branch; the output of the second low-frequency state space module is connected to the inputs of the second high-frequency Transformer module and the inverse wavelet transform module respectively; the input of the second low-frequency state space module of the last branch is connected to the output of the first low-frequency state space module of its own branch; The output of the inverse wavelet transform module in the first branch is processed by convolution to obtain the reconstructed residual image; The discrete wavelet transform module is used to extract low-frequency and high-frequency features of the image at the corresponding scale based on the input image; The first low-frequency state space module is used to enhance the low-frequency features and obtain the enhanced low-frequency features. The first high-frequency Transformer module is used to enhance high-frequency features and obtain enhanced high-frequency features.

3. The low-light image enhancement method and apparatus based on a multi-scale wavelet state-space model according to claim 2, characterized in that, The first low-frequency state space module includes: a first normalization layer, a visual state space module, a first addition layer, a second normalization layer, a convolutional layer, a channel attention layer, and a second addition layer, connected in sequence. The input of the first adder layer is also connected to the input of the first normalization layer; the input of the second adder layer is also connected to the output of the first adder layer.

4. The low-light image enhancement method and apparatus based on a multi-scale wavelet state-space model according to claim 3, characterized in that, The visual state space module includes a first linear layer, a depthwise separable convolutional layer, a first activation layer, a neighborhood-aware scanning module, a third normalization layer, a first multiplication layer, and a second linear layer connected in sequence. The visual state space module also includes a third linear layer and a second activation layer connected in series; the input of the third linear layer is connected to the input of the first linear layer; and the input of the second activation layer is connected to the input of the first multiplication layer.

5. The low-light image enhancement method and apparatus based on a multi-scale wavelet state-space model according to claim 4, characterized in that, The neighborhood-aware scanning module includes: The unfolding and positional encoding layer is used to unfold the input two-dimensional image features into a one-dimensional sequence and embed positional codes; The cue pool construction layer is used to construct a cue pool matrix for storing semantic features; The hint weight generation layer is used to generate a hard-assigned routing matrix from pixels to semantic categories by applying a Gumbel-Softmax driven differentiable routing mechanism to the location-encoded features. The dynamic prompt generation layer is used to multiply the hard-assigned routing matrix with the prompt pool matrix to obtain dynamic prompts carrying cross-region association information; The state space equation update layer is used to take the location-encoded features as input to the state space model and inject dynamic prompts into the output matrix C of the state space equation through residual superposition to obtain the updated state space equation. The neighborhood sensing layer is used to derive the neighborhood sensing feature map based on the updated state space equation; The folding layer is used to fold the neighborhood-aware feature map to obtain the folded neighborhood-aware feature map; the folded neighborhood-aware feature map is the input to the third normalization layer.

6. The low-light image enhancement method and apparatus based on a multi-scale wavelet state-space model according to claim 2, characterized in that, The first high-frequency Transformer module includes a low-frequency feature matching module, a cross-frequency attention fusion module, and a hierarchical feature correction module; The low-frequency feature matching module is used to calculate the similarity between the input high-frequency features and low-frequency features, and to filter the low-frequency feature segments that are most relevant to the high-frequency features. The cross-frequency attention fusion module is used to extract the channel information of the selected low-frequency feature fragments and the input high-frequency features respectively. Based on the channel information, the softmax function is used to generate dynamic weights. The input low-frequency features and high-frequency features are fused using dynamic weights to obtain the attention-weighted high-frequency features. The hierarchical feature correction module is used to input the attention-weighted high-frequency features into the multilayer perceptron, and combine the global information of the low-frequency features to correct the high-frequency features layer by layer, so as to obtain the corrected high-frequency features.

7. A low-light image enhancement device based on a multi-scale wavelet state-space model, characterized in that, include: The image acquisition module is used to acquire the low-light image to be processed; The downsampling and feature extraction module is used to downsample the low-light image to be processed, obtain downsampled images at multiple scales, and perform convolution processing on the low-light image to extract shallow features of the image. The image reconstruction module is used to input shallow features of the image and downsampled images at multiple scales into the multi-scale wavelet state space model to obtain the reconstructed residual image; the multi-scale wavelet UNet architecture and the state space model are integrated in the multi-scale wavelet state space model. The image enhancement module is used to derive the final enhanced image from the reconstructed residual image and the dark-light image to be processed.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the low-light image enhancement method based on a multi-scale wavelet state-space model as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the low-light image enhancement method based on a multi-scale wavelet state-space model as described in any one of claims 1-6.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the low-light image enhancement method based on a multi-scale wavelet state-space model as described in any one of claims 1-6.

Images