Multi-scale three-channel state space algorithm based on Retinex theory

Abstract

The invention discloses a multi-scale three-channel state space algorithm based on a Retinex theory, and relates to a multi-scale three-channel state space algorithm. The invention aims to solve the problems of poor detail, brightness and texture information of a shot image in a dark environment and unnatural color transition of a local area of the image. The method comprises the following steps: step 1, inputting a dark light image I, and averaging in a channel dimension to construct illumination prior information Lp; 2, estimating an illumination feature map Flu and an illumination map L through an illumination estimator, and constructing an illumination image Ilu; 3, processing the illumination image by using an illumination restorer to generate an illumination recovery image Iir; and step 4, generating an enhanced image Ien by using the illumination recovery image Iir and the illumination image Ilu. The invention belongs to the technical field of computer vision.

Description

Technical Field

[0001] The present invention relates to a multi-scale three-channel state space model, belonging to the technical field of computer vision. Background Art

[0002] Computer vision has undergone significant transformations driven by the rise of deep learning techniques, which have played a central role in addressing various visual challenges, including low-light image enhancement. Low-light image enhancement is a key aspect of computer vision and image processing research, as it not only optimizes visual perception but also improves the performance of downstream applications such as autonomous driving systems, object recognition, and image semantic segmentation. Images captured in low-light environments often suffer from various degradation issues such as low visibility, weak contrast, and noise. Therefore, enhancing such images has become an essential preprocessing step.

[0003] With advances in deep learning technology, convolutional neural networks (CNNs) have been widely used to improve image quality in low-light conditions. Currently, these techniques fall into two main categories. The first category directly constructs a mapping relationship between low-light images and standard-light images; however, these methods lack interpretability and theoretical support. The second category, based on Retinex theory and convolutional neural networks, divides the training process into three separate stages: image decomposition, reflection noise removal, and illumination adjustment. Each stage requires separate fine-tuning, resulting in a complex and time-consuming process. Convolutional neural networks have significant advantages in extracting local features and spatial information, but they are limited in capturing long-range similarities, making them inadequate for fully capturing holistic image information. This manifests as challenges in noise amplification and detail preservation in low-light image processing. In contrast, the Transformer model, through its self-attention mechanism, achieves global perception and long-range dependency modeling, while the Mamba model achieves this through a spatial state model, helping to more accurately restore detail and structure in low-light images. However, they still have limitations in multi-scale feature interactions, channel importance, and feature matching between encoders and decoders, resulting in poor results in recovering image details, color transitions in local areas, and texture information in local areas. Summary of the Invention

[0004] In order to solve the problems of poor detail, brightness and texture information of images taken in low-light environments and unnatural color transitions in local areas of the images, the present invention proposes a multi-scale three-channel state space algorithm based on Retinex theory.

[0005] The technical solution adopted by the present invention to solve the above problems is: the steps of the present invention include: Step 1: Input dark light image , constructing illumination prior information by averaging in the channel dimension ; Step 2: Estimate the illumination feature map through the illumination estimator and light mapping And build the lighting image ; Step 3: Use the illumination restorer to process the illumination image to generate the illumination restoration map ; Step 4: Use illumination to restore the image and lighting images Generate enhanced images .

[0006] Furthermore, the method for constructing the illumination prior information in step 1 is: The illumination prior information of an image is obtained by taking the average of the three channel dimensions: (1), In formula (1), 、 and Represent the values ​​of the red, green, and blue channels of the image respectively. Formula (1) sums the three channel values ​​of each pixel and then divides it by 3 to get the average value of the pixel.

[0007] Furthermore, step 2 specifically includes: Step 201: Low-light image With illumination prior Concatenate in the channel dimension as input ; Step 202: Extract features through three convolution operations. The first one The layer combines the previously combined input Merge, which is used to fuse the illumination prior information into the dark light image, and the second convolution is a depth-separable Convolution, which performs upsampling and further integrates the lighting prior to obtain a lighting feature map with 40 dimensions , the last one The layer downsamples the output to produce a three-channel light map ; Step 203: By mapping the light With low-light images Perform element-by-element multiplication to obtain the illumination image . , , and They can be obtained by the following calculation methods: (2), (3), (4), (5), In formulas (2) to (5), represents the splicing operation of the channel dimension, Represents an element-wise multiplication operation.

[0008] Furthermore, step 3 specifically includes: Step 301: Lighting image from lighting estimator pass , step size = 2, downsample, and its dimension is the same as the illumination feature map Alignment , for subsequent processing; Step 302: After extracting image features, upsampling is performed to restore the image; Step 303: Input The application size is , and Depth-wise separable convolution, resulting in input feature maps of three scales , and ; Step 304: By changing the dimension, and Adjusted to a form suitable for multi-head attention mechanism: (6), (7), In formulas (6) and (7), , is the number of heads, represents the dimension of each head, represents the scale of the depthwise separable convolution (DWConv), ,in Indicates the dimension of the input; Will Considered as ,enter Considered as , to fuse the two input features so that Able to guide Self-attention calculation; (8), In formula (8), , and is a learnable parameter matrix constructed by the convolutional layer; the multi-scale self-attention of each head is expressed as: (9), In formula (9), is a learnable parameter that acts as a scaling factor to adjust the attention score at each scale, thereby controlling the sharpness of the attention weight; Step 305 The image is reshaped back to the standard image format. , and aggregated through convolutional layers to obtain an output that matches the original size.

[0009] Furthermore, the downsampling operation in step 301 is divided into two stages, each of which includes a multi-scale three-channel state space algorithm, followed by a convolution kernel with a stride of 2 and a size of After each convolution layer, the width and height of the image are halved, while the feature dimension is doubled. After two levels of downsampling, the feature map dimension will be .

[0010] Furthermore, the upsampling process in step 302 is divided into two stages, each of which includes a , stride = 2, and a multi-scale three-channel state space algorithm; each time through a deconvolution layer, the width and height of the image will increase, while the feature dimension will decrease, the output of the deconvolution layer and the output of the MTSSM of the corresponding level are processed by a spatial channel weighted feature fusion module; finally, a , stride = 2 to reduce the dimension, restore the image to RGB format, and generate , by restoring the illumination map and lighting images Pixel-by-pixel addition generates the final enhanced image .

[0011] Furthermore, step 4 specifically includes: Restoring the light map and lighting images Perform pixel-by-pixel addition to generate an enhanced image : (15).

[0012] The present invention provides the following beneficial effects: It proposes a multi-scale, three-channel state-space algorithm based on Retinex theory. First, to address the problem of insufficient multi-scale feature interaction in low-light image enhancement, the present invention proposes a multi-scale illumination fusion attention module. This module enhances the overall brightness of the image while preserving and enhancing local details, making the enhanced image more natural and delicate in visual effect. Second, to address the problems of insufficient multi-dimensional feature interaction and ineffective noise suppression in low-light image enhancement, the present invention proposes a three-channel state-space model. This model utilizes a multi-dimensional attention mechanism to automatically identify and suppress noise in the image, effectively controlling the noise level and ensuring image naturalness and clarity. Finally, to address the problems of inconsistent feature alignment, insufficient information utilization, and insufficient feature fusion robustness during encoder and decoder feature fusion in low-light image enhancement, the present invention proposes a spatial channel weighted feature fusion module. By generating channel-specific spatial interaction maps, the model enhances its adaptability and stability under different lighting conditions and noise levels. The present invention significantly improves the detail preservation and brightness enhancement of low-light images, effectively reduces noise and artifacts while maintaining natural color transitions, and improves the overall visual quality and naturalness of the image.

[0013] The significance of this invention lies in significantly improving the overall visual quality and naturalness of low-light images, so that images taken under complex lighting conditions can present detailed information more clearly and realistically. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a schematic diagram of the system model. Figure 1 (a) is the lighting estimator, Figure 1 (b) is the light restorer; Figure 2 This is a schematic diagram of the multi-scale three-channel state space algorithm; Figure 3 It is a schematic diagram of the three-channel state space model; Figure 4 This is a schematic diagram of the spatial channel weighted feature fusion module. Figure 4 (a) is the content-guided attention module, Figure 4 (b) is the fusion strategy.

[0015] Figure 5 It is the result of dark light enhancement. Figure 5 (a) is the input dark light image, Figure 5 (b) is the true value image, Figure 5 (c) is the dark light enhancement result diagram proposed by the present invention. DETAILED DESCRIPTION

[0016] Specific implementation method 1: Figures 1 to 5As shown in Figure 1, a multi-scale three-channel state space algorithm based on Retinex theory includes the following steps: Step 1: Input dark light image , constructing illumination prior information by averaging in the channel dimension ; Step 2: Estimate the lighting feature map through the lighting estimator and light mapping And build the lighting image ; Step 3: Use the illumination restorer to process the illumination image to generate the illumination restoration map ; Step 4: Use lighting to restore the image and lighting images Generate enhanced images .

[0017] Specific implementation method 2: Figures 1 to 5 As shown in Figure 1, the specific process of constructing the illumination prior information in step 1 is as follows: the illumination prior information of an image is obtained by taking the average value of the three channel dimensions: (1) in, 、 and Representing the values ​​of the red, green, and blue channels of the image respectively. This formula sums the three channel values ​​for each pixel and then divides it by 3 to get the average value of the pixel.

[0018] Specific implementation method three: Figures 1 to 5 As shown, in step 2, the illumination feature map is estimated by the illumination estimator and light mapping And build the lighting image The specific process is: The structure of the lighting estimator is as follows Figure 1 As shown in (a), the low-light image With illumination prior Concatenate in the channel dimension as input Next, three convolution operations are performed to extract features. The first The layer combines the previously combined input Merge, the main function is to integrate the illumination prior information into the dark light image. The second convolution is a depth-separable Convolution, whose main function is to perform upsampling and further integrate the lighting prior to obtain a lighting feature map with 40 dimensions The last one The layer downsamples the output to produce a three-channel light map . Then by adding the light map With low-light images Perform element-by-element multiplication to obtain the illumination image . , , and They can be obtained by the following calculation methods: (2) (3) (4) (5) in, represents the splicing operation of the channel dimension, Represents an element-wise multiplication operation.

[0019] Specific implementation method four: Figures 1 to 5 As shown, in step 3, the illumination image is processed using the illumination restorer to generate the illumination restoration map The specific process is: Light restorer such as Figure 1 As shown in (b), the restorer is based on the multi-scale spatial channel fusion Mamba framework. The framework consists of an encoder and a decoder, where the encoder is responsible for downsampling and the decoder is responsible for upsampling. The two processes are symmetrical and each is divided into two stages. First, the illumination image from the illumination estimator is pass (step = 2) for downsampling, and its dimension is consistent with the illumination feature map Alignment , for subsequent processing. The downsampling operation is divided into two stages, each of which includes a multi-scale three-channel state space algorithm (MTSSM), followed by a convolution kernel with a stride of 2 and a size of After each convolution layer, the width and height of the image are halved, while the feature dimension is doubled. After two levels of downsampling, the feature map dimension will be After extracting the image features, upsampling is performed to restore the image. The upsampling process is similar to the downsampling process and is also divided into two stages. Each stage includes a (stride = 2), and a multi-scale three-channel state space algorithm (MTSSM). After each deconvolution layer, the width and height of the image will increase, while the feature dimension will decrease. In order to optimize the fusion of shallow and deep features and solve the feature matching problem between the encoder and decoder, the output of the deconvolution layer and the output of the MTSSM at the corresponding level will be processed by a spatial channel weighted feature fusion (SCWFusion) module. Finally, a (stride=2) to reduce the dimensionality and restore the image to RGB format, generating By restoring the illumination map and lighting images Pixel-by-pixel addition generates the final enhanced image .

[0020] The core component of the illumination restorer is a multi-scale three-channel state-space algorithm, such as Figure 2 As shown in the figure, it has layer normalization (LN), multi-scale illumination fusion attention module (MIFA), triplet-channel State Space Model and feed-forward network (FFN).

[0021] Dimensionally aligned lighting image Apply separately 、 and To extract local features of different scales, we use the depthwise separable convolution of The calculation results and the lighting feature map obtained by the lighting estimator Perform cross attention fusion and finally sum the feature maps of the three scales pixel by pixel.

[0022] First, the input The application size is , and Depth-wise separable convolution, resulting in input feature maps of three scales , and .

[0023] Second, by changing the dimension, and Adjusted to a form suitable for multi-head attention mechanism: (6) (7) in , is the number of heads, represents the dimension of each head, represents the scale of the depthwise separable convolution (DWConv), ,in Represents the dimension of the input. Note that using the illumination feature map as the query vector allows the model to target dark areas in the image more specifically. This invention uses feature maps at three scales to capture image details more comprehensively, retaining the original texture and structural information, especially those details that are often overlooked in low-light environments, thereby better guiding the processing of dark areas. Therefore, we will Considered as ,enter Considered as , to fuse the two input features so that Able to guide Self-attention calculation.

[0024] (8) in , and is a learnable parameter matrix constructed by the convolutional layer. Therefore, the multi-scale self-attention of each head can be expressed by the following formula: (9) in is a learnable parameter that acts as a scaling factor to adjust the attention score at each scale, thereby controlling the sharpness of the attention weight. The image is reshaped back to the standard image format. , and aggregated through convolutional layers to obtain an output that matches the original size.

[0025] like Figure 3 As shown in Figure 2, the Tri-Channel State Space Model (TriSSM) leverages multi-dimensional, cross-channel, and spatial dynamic features to improve the quality of low-light images. The TriSSM design deeply considers the information interaction of low-light images in all dimensions (height, width, and channels) and adopts an innovative multi-branch architecture to significantly improve image detail recovery and brightness enhancement.

[0026]

[0027] Algorithm 1 shows The overall architecture of the model. Initially, the SS2D module promotes feature interaction and fusion in the spatial dimension. Subsequently, the model interacts in three different dimensions through three branches. The first branch is responsible for calculating the spatial dimension and The second branch calculates the channel and spatial dimensions The attention weights on , while the third branch handles the channel dimension and spatial dimensions The three branches finally integrate their attention weights by taking the average. In addition, The module consists of two parts: global average pooling and global maximum pooling. Assume there is an input tensor , which is processed and rotated by the SS2D module to produce , which is then fed into a three-channel architecture for processing.

[0028] like Figure 4 As shown in the figure, the core idea of ​​the SCWFusion module is to guide the fusion of feature maps in the spatial and channel dimensions by generating channel-specific spatial interaction maps (SIMs). Specifically, the input feature First, after a series of processing, SIMs with the same dimension are generated ( ), thus providing guidance for subsequent weighted fusion.

[0029] First, the channel attention and spatial attention maps are calculated using global average pooling (GAP) and global maximum pooling (GMP) operations. and .

[0030] (10) (11) in express activation function, Indicates a Convolutional layers with kernel size, Represents a channel splicing operation. 、 and Represent the spatial dimension features processed by the global average pooling operation, the channel dimension features processed by the global average pooling operation, and the channel dimension features processed by the global maximum pooling operation. In order to reduce the number of parameters and limit the complexity of the model, the first Convolution changes the channel dimension from Reduce to (in refers to the reduction ratio), the second Convolution expands it back In the implementation of the present invention, we choose to set for to reduce the channel dimension to a fixed value.

[0031] Then we use a simple addition operation to follow the broadcasting rules to and Fuse together to obtain a coarse spatial attention map .

[0032] (12) After getting the initial SIMs ( ), the SCWFusion module further fuses the features by adaptive weighting. Specifically, the SIMs of each channel are alternately rearranged with the input features to generate the final channel-specific SIMs ( This process ensures that each channel feature can be effectively combined with the corresponding SIMs through the channel shuffle operation, thereby enhancing the key area features in the image.

[0033] (13) in, Represents the sigmoid operation, Represents a channel shuffle operation, Indicates a Grouped convolutional layers with different kernel sizes.

[0034] To further improve feature fusion and ensure training stability, the SCWFusion module employs a weighted fusion of encoder and decoder features rather than a simple skip connection. This strategy avoids the vanishing gradient problem, ensuring better transfer of gradient information during training, promoting more efficient learning, and making low-light image enhancement more efficient and robust.

[0035] (14) The SCWFusion module combines feature representations at different levels through a weighted fusion of encoder and decoder features. This feature fusion enables the network to learn image detail and brightness information at multiple scales while avoiding the feature information loss problem encountered in traditional methods. After feature fusion, concatenating the encoder features with the fused features in the channel dimension effectively balances the influence of local and global information. Dimensionality reduction through 1×1 convolution allows the network to focus on the detailed processing of local features without losing global information. This balance between global and local information is crucial for low-light image enhancement, particularly in terms of detail recovery and brightness improvement.

[0036] Specific embodiment 5: This embodiment differs from the specific embodiment 1 in that the light recovery image is used in step 4. and lighting images Generate enhanced images ; The specific process is: Restoring the light map and lighting images Perform pixel-by-pixel addition to generate an enhanced image .

[0037] (15) Table 1 Symbol Description

[0038] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any technician familiar with the present profession can make some changes or modifications to equivalent embodiments of equivalent changes using the technical content disclosed above without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent replacement and improvement of the above embodiments made according to the technical essence of the present invention, within the spirit and principles of the present invention, without departing from the content of the technical solution of the present invention, shall still fall within the scope of protection of the technical solution of the present invention.

Claims

1. A multi-scale three-channel state space algorithm based on Retinex theory, characterized by: The specific steps include: Step 1: Input dark light image , constructing illumination prior information by averaging in the channel dimension ; Step 2: Estimate the illumination feature map through the illumination estimator and light mapping And build the lighting image ; Step 3: Use the illumination restorer to process the illumination image to generate the illumination restoration map ; Step 4: Use illumination to restore the image and lighting images Generate enhanced images .

2. A multi-scale three-channel state space algorithm based on Retinex theory according to claim 1, characterized in that: The method for constructing illumination prior information in step 1 is: The illumination prior information of an image is obtained by taking the average of the three channel dimensions: （1）， In formula (1), 、 and Represent the values ​​of the red, green, and blue channels of the image respectively. Formula (1) sums the three channel values ​​of each pixel and then divides it by 3 to get the average value of the pixel.

3. The multi-scale three-channel state space algorithm based on Retinex theory according to claim 1, characterized in that: Step 2 specifically includes: Step 201: Low-light image With illumination prior Concatenate in the channel dimension as input ; Step 202: Extract features through three convolution operations. The first one The layer combines the previously combined input Merge, which is used to fuse the illumination prior information into the dark light image. The second convolution is a depth-separable Convolution, which performs upsampling and further integrates the lighting prior to obtain a lighting feature map with 40 dimensions , the last one The layer downsamples the output to produce a three-channel light map ; Step 203: By mapping the light With low-light images Perform element-by-element multiplication to obtain the illumination image . , , and They can be obtained by the following calculation methods: （2）， （3）， （4）， （5）， In formulas (2) to (5), represents the splicing operation of the channel dimension, Represents an element-wise multiplication operation.

4. The multi-scale three-channel state space algorithm based on Retinex theory according to claim 1, characterized in that: Step 3 specifically includes: Step 301: Lighting image from lighting estimator pass , step size = 2, downsample, and its dimension is the same as the illumination feature map Alignment , for subsequent processing; Step 302: After extracting image features, upsampling is performed to restore the image; Step 303: Input The application size is , and Depth-wise separable convolution, resulting in input feature maps of three scales , and ; Step 304: By changing the dimension, and Adjusted to a form suitable for multi-head attention mechanism: （6）， （7）， In formulas (6) and (7), , is the number of heads, represents the dimension of each head, represents the scale of the depthwise separable convolution (DWConv), ,in Indicates the dimension of the input; Will Considered as ,enter Considered as , to fuse the two input features so that Able to guide Self-attention calculation; （8）， In formula (8), , and is a learnable parameter matrix constructed by the convolutional layer; the multi-scale self-attention of each head is expressed as: （9）， In formula (9), is a learnable parameter that acts as a scaling factor to adjust the attention score at each scale, thereby controlling the sharpness of the attention weight; Step 305 The image is reshaped back to the standard image format. , and aggregated through convolutional layers to obtain an output that matches the original size.

5. The multi-scale three-channel state space algorithm based on Retinex theory according to claim 4, characterized in that: The downsampling operation in step 301 is divided into two stages, each of which includes a multi-scale three-channel state space algorithm, followed by a convolution kernel with a stride of 2 and a size of After each convolution layer, the width and height of the image are halved, while the feature dimension is doubled. After two levels of downsampling, the feature map dimension will be .

6. The multi-scale three-channel state space algorithm based on Retinex theory according to claim 4, characterized in that: The upsampling process in step 302 is divided into two stages, each of which includes a , stride = 2, and a multi-scale three-channel state space algorithm; each time through a deconvolution layer, the width and height of the image will increase, while the feature dimension will decrease, the output of the deconvolution layer and the output of the MTSSM of the corresponding level are processed by a spatial channel weighted feature fusion module; finally, a , stride = 2 to reduce the dimension, restore the image to RGB format, and generate , by restoring the illumination map and lighting images Pixel-by-pixel addition generates the final enhanced image .

7. The multi-scale three-channel state space algorithm based on Retinex theory according to claim 1, characterized in that: Step 4 specifically includes: Restoring the light map and lighting images Perform pixel-by-pixel addition to generate an enhanced image : （15）。

Images