A method, system and device for enhancing dark light images

Abstract

This invention provides a method, system, and apparatus for enhancing low-light images, relating to the field of image processing technology. The method mainly includes: preprocessing the original low-light image to obtain a prior information map; concatenating and fusing the prior information map with the original image to extract a brightness feature map and a brightness map; multiplying the brightness map element-wise with the original image to obtain a brightened image; extracting shallow features from the brightened image through a convolutional layer to obtain a fourth feature map; fusing the brightened image with the fourth feature map through a cross-attention mechanism to obtain a feature map AF; performing color correction, multi-scale feature extraction, and sampling operations on the feature map AF and the brightened image to obtain a twelfth feature map; reconstructing the image through a convolutional layer to obtain a thirteenth feature map; and adding the thirteenth feature map element-wise with the brightened image to obtain the output image. This solution has a simple and reasonable architecture and can effectively enhance low-light images.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a dark-light image enhancement method, system and device. Background Art

[0002] Dark-light image enhancement technology aims to improve problems such as insufficient image brightness, high noise, and poor contrast caused by environmental factors, so as to restore the image to normal visual effects. This technology has been widely used in fields such as autonomous driving, security monitoring, and mobile device photography. Existing image enhancement methods are mainly divided into traditional methods and deep learning-based methods.

[0003] Traditional methods, such as histogram equalization and gamma correction, are fast but relatively poor in image enhancement effects.

[0004] Although deep learning-based methods, including convolutional neural networks (CNN) and Transformer, have significantly improved image enhancement effects, they also have their own shortcomings: the CNN method is unable to obtain global information due to the limitation of the convolution kernel size, resulting in deficiencies in detail recovery; the Transformer architecture can effectively capture global information, but its computational complexity is high, which limits its practical application.

[0005] In general, the specific shortcomings of deep learning-based methods are summarized as follows: in the pursuit of fast reasoning speed, the structural information and detail information of the image may be severely lost; while improving the image quality, the processing speed drops rapidly as the number of parameters and calculations increase significantly; it is difficult to effectively integrate the structural information and detail information of the image, resulting in an imbalance between reasoning speed and imaging quality. Summary of the invention

[0006] The object of the present invention is to provide a dark-light image enhancement method, system and device to solve at least one of the above-mentioned technical problems existing in the prior art.

[0007] In a first aspect, in order to solve the above technical problems, the present invention provides a dark light image enhancement method, comprising the following steps:

[0008] Step 1: Preprocess the original image of the dark light image to obtain a prior information map (Prior).

[0009] Preferably, the step 1 specifically includes: calculating the average value of each pixel point of the original image on all color channels to obtain a priori information map.

[0010] Step 2: The prior information map is spliced ​​and fused with the original map to extract the brightness feature map LF (Light Feature) and the brightness map LM (Light Map).

[0011] Preferably, the step 2 specifically includes: splicing the original image and the prior information map by channel to obtain a first feature map; passing the first feature map through a convolution layer to perform image feature fusion to obtain a second feature map; extracting and fusing features of the second feature map through a deep separable convolution (DWConv) to obtain a brightness feature map LF and a third feature map respectively; passing the third feature map through a convolution layer to extract a brightness map LM.

[0012] Step 3: Multiply the brightness map LM by the original image element by element to obtain a brightened image IF, so that the brightness information can be integrated with the original image information to improve the image brightness; extract shallow features from the brightened image IF through a convolutional layer to obtain a fourth feature map.

[0013] Step 4: The highlighted image is fused with the fourth feature map through a cross attention mechanism to obtain a feature map AF. The specific formula can be:

[0014] ;

[0015] in, Display key Dimensions; Indicates a query; Represents a value.

[0016] Step 5: The feature map AF and the highlight map IF are processed multiple times through the color correction and multi-scale feature extraction module (CDUM), and a sampling operation is performed to obtain the twelfth feature map, which specifically includes:

[0017] The feature map AF and the highlight map IF are respectively subjected to layer normalization processing to obtain the corresponding fifth feature map; the two fifth feature maps are spliced ​​and fused in the channel dimension through a color correction module (CSSM) to output a sixth feature map; the sixth feature map is superimposed with the normalized feature map AF and the highlight map IF to obtain a seventh feature map to prevent detail loss;

[0018] The color correction module is a parallel state space model (SSM);

[0019] In this way, through color correction, the defect of ignoring the dark light image itself in the Retinex theory is solved, which leads to a large difference between the color of the image and the real image.

[0020] The seventh feature map is processed again by layer normalization to obtain an eighth feature map; the noise is eliminated by a noise removal module (UnoisyM) to obtain a ninth feature map; in this way, by processing the noise separately, the influence of the noise can be reduced, thereby improving the imaging quality of the image;

[0021] The multi-scale features are extracted by a multi-scale feature extraction module (DFM) to obtain a tenth feature map; the seventh feature map, the ninth feature map and the tenth feature map are added element by element to obtain an eleventh feature map, thereby ensuring the integrity of the image details; the eleventh feature map is downsampled or deconvolved upsampled to obtain a twelfth feature map;

[0022] In this way, through multi-scale feature extraction, the structural features of the image can be maintained to the maximum extent, and the detail information of the image can be better extracted; through multi-scale feature fusion, the image structure information and image information can be better fused, thereby improving the image enhancement quality and detail recovery capabilities.

[0023] Step 6: Reconstruct the twelfth feature map through a convolution layer to obtain the thirteenth feature map; add the thirteenth feature map and the brightened image IF element by element to obtain an output image, thereby enhancing the detail features again, thereby ensuring that the dark light image is enhanced.

[0024] In a feasible implementation, the noise removal module (UnoisyM) removes noise from the image by performing a depthwise separable convolution on the feature map.

[0025] In a feasible implementation, the extraction method of the multi-scale feature extraction module (DFM) includes: processing the second normalized image through two parallel convolution branches, and then splicing it in the channel dimension; convolving the splicing result to perform information fusion in the channel dimension; and activating the information fusion result through a channel attention mechanism;

[0026] The convolution branch includes a first convolution branch and a second convolution branch;

[0027] The middle layer of the first convolution branch uses a convolution kernel of a certain size to extract detailed information of the image;

[0028] The middle layer of the second convolution branch uses a convolution kernel of another size to extract more detailed information of the image and enhance the connection between contexts in the image.

[0029] In a second aspect, based on the same inventive concept, the present application provides a dark light image enhancement system, which specifically includes a brightening module, a U-Net network and a first adding unit;

[0030] The brightening module is used to receive the original image (Input) and the prior information image (Prior) of the dark light image, perform splicing and fusion processing, and output a brightened image;

[0031] The U-Net network specifically includes a first convolutional layer (Conv), a first enhancement module (AtSSM), a second convolutional layer, a second enhancement module, a third convolutional layer, a third enhancement module, a first deconvolutional layer (DeConv), a fourth convolutional layer, a fourth enhancement module, a second deconvolutional layer, a fifth convolutional layer, a fifth enhancement module and a sixth convolutional layer;

[0032] The input end of the first convolutional layer is connected to the output end of the highlighting module, and the output end of the first convolutional layer is connected to the first input end of the first enhancement module;

[0033] The second input end of the first enhancement module is connected to the output end of the highlighting module, and the first output end of the first enhancement module is connected to the input end of the second convolutional layer;

[0034] The output end of the second convolutional layer is connected to the input end of the second enhancement module; the first output end of the second enhancement module is connected to the input end of the third convolutional layer; the output end of the third convolutional layer is connected to the input end of the third enhancement module; the output end of the third enhancement module is connected to the input end of the first deconvolutional layer; the first output end of the first deconvolutional layer and the second output end of the second enhancement module are spliced ​​according to channels, and then connected to the first input end of the fourth convolutional layer; the second output end of the first deconvolutional layer is connected to the second input end of the fourth convolutional layer; the output end of the fourth convolutional layer is connected to the input end of the fourth enhancement module; the output end of the fourth enhancement module is connected to the input end of the second deconvolutional layer; the first output end of the second deconvolutional layer and the second output end of the first enhancement module are spliced ​​according to channels, and then connected to the first input end of the fifth convolutional layer; the second output end of the second deconvolutional layer is connected to the second input end of the fifth convolutional layer; the output end of the fifth convolutional layer is connected to the input end of the fifth enhancement module; the output end of the fifth enhancement module is connected to the input end of the sixth convolutional layer; the output end of the sixth convolutional layer serves as the output end of the U-Net network;

[0035] The first adding unit is used to add the feature map output by the U-Net network and the highlighting map output by the highlighting module element by element to obtain an output image (Output).

[0036] In a feasible implementation manner, the highlighting module specifically includes a seventh convolutional layer, a first depth-separable convolutional layer, an eighth convolutional layer and a first multiplication unit which are arranged in sequence;

[0037] The input end of the seventh convolutional layer is used to receive the first feature map after the original image and the prior information image are spliced ​​according to the channel, perform image feature fusion, and output the second feature map;

[0038] The first depth separable convolution layer is used to receive the second feature map, perform feature extraction and fusion, and output a brightness feature map LF and a third feature map respectively;

[0039] The eighth convolutional layer is used to receive the third feature map and extract the brightness map LM;

[0040] The first multiplication unit is used to perform element-by-element multiplication of the original image and the brightness image LM to obtain a brightened image IF.

[0041] In a feasible implementation, the enhancement module specifically includes a cross-attention mechanism (IFA) module and a color correction and multi-scale feature extraction module (CDUM) which are arranged in sequence.

[0042] In a feasible implementation, the cross-attention mechanism (IFA) module specifically includes a key branch and a value branch; the key branch includes a second multiplication unit for matrix multiplying the key of the fourth feature map with the highlight map IF to obtain a thirteenth feature map; the value branch includes a third multiplication unit for matrix multiplying the value of the fourth feature map with the thirteenth feature map to obtain a feature map AF.

[0043] In a feasible implementation, the cross-attention mechanism module can also use the brightness feature map LF as additional information for comprehensive calculations, thereby realizing additional information input and further improving image quality.

[0044] In a feasible implementation, the color correction and multi-scale feature extraction module (CDUM) includes a first normalization layer, a second normalization layer, a state space model (CSSM), a second addition unit, a third normalization layer, a multi-scale feature extraction module (DFM), a noise removal module (UnoisyM) and a third addition unit;

[0045] An input end of the first normalization layer is connected to an output end of the highlighting module, and a first output end of the first normalization layer is connected to a first input end of the state space model;

[0046] An input end of the second normalization layer is connected to an output end of the cross attention mechanism module, and a first output end of the second normalization layer is connected to a second input end of the state space model;

[0047] The output end of the state space model is connected to the first input end of the second adding unit; the second input end of the second adding unit is connected to the second output end of the first normalization layer; the third input end of the second adding unit is connected to the second output end of the second normalization layer; the output end of the second adding unit is connected to the input end of the third normalization layer; the output end of the third normalization layer is respectively connected to the input end of the multi-scale feature extraction module and the input end of the noise removal module;

[0048] The first input end of the third adding unit is connected to the output end of the multi-scale feature extraction module; the second input end of the third adding unit is connected to the output end of the second adding unit; the third input end of the third adding unit is connected to the output end of the noise removal module; the output end of the third adding unit serves as the output end of the color correction and multi-scale feature extraction module;

[0049] Through the above architecture, the output features that have not been fused with the Retinex theory information can be fused with the output features that have been fused with the Retinex theory information, so that the original color information can be retained.

[0050] In a feasible implementation, the state space model (CSSM) includes an original feature state space model (OVSSM), a retinal feature state space model (RVSSM) and a ninth convolutional layer; the input end of the original feature state space model serves as the first input end of the state space model; the input end of the retinal feature state space model serves as the second input end of the state space model; the output end of the original feature state space model and the output end of the retinal feature state space model are spliced ​​by channel and then connected to the input end of the ninth convolutional layer; the output end of the ninth convolutional layer serves as the output end of the state space model.

[0051] Preferably, the original feature state space model and the retinal feature state space model both include a first linear layer, a second linear layer, a second depthwise separable convolutional layer, a first SiLu activation layer, a second SiLu activation layer, a two-dimensional state space model layer (2D-SSM), a fourth normalization layer, a fourth multiplication unit and a third linear layer;

[0052] The input end of the first linear layer is connected to the input end of the second linear layer for synchronously receiving preceding input data;

[0053] The output end of the first linear layer is connected to the input end of the second depthwise separable convolutional layer; the output end of the second depthwise separable convolutional layer is connected to the input end of the first SiLu activation layer; the output end of the first SiLu activation layer is connected to the input end of the two-dimensional state space model layer; the output end of the two-dimensional state space model layer is connected to the input end of the fourth normalization layer; the output end of the fourth normalization layer is connected to the first input end of the fourth multiplication unit, thus forming a first processing branch;

[0054] The output end of the second linear layer is connected to the input end of the second SiLu activation layer; the output end of the second SiLu activation layer is connected to the second input end of the fourth multiplication unit, thus forming a second processing branch;

[0055] The output end of the fourth multiplication unit is connected to the input end of the third linear layer for matrix multiplication; the output end of the third linear layer is used to output data to a subsequent order.

[0056] In a feasible implementation, the multi-scale feature extraction module (DFM) includes a tenth convolutional layer, an eleventh convolutional layer, a twelfth convolutional layer, a thirteenth convolutional layer, a fourteenth convolutional layer, a fifteenth convolutional layer, a sixteenth convolutional layer and a channel attention layer;

[0057] The input end of the tenth convolutional layer is connected to the input end of the thirteenth convolutional layer, and is used for synchronously receiving the preceding input data;

[0058] The output end of the tenth convolutional layer is connected to the input end of the eleventh convolutional layer; the output end of the eleventh convolutional layer is connected to the input end of the twelfth convolutional layer, thus forming a detail extraction branch; the eleventh convolutional layer is used to extract detail features of the image;

[0059] The output end of the thirteenth convolutional layer is connected to the input end of the fourteenth convolutional layer; the output end of the fourteenth convolutional layer is connected to the input end of the fifteenth convolutional layer, thus forming a semantic extraction branch; the fourteenth convolutional layer is used to extract image context semantic information, so as to better restore local features of the image;

[0060] The output end of the twelfth convolutional layer and the output end of the fifteenth convolutional layer are connected to the input end of the sixteenth convolutional layer after channel splicing processing; the output end of the sixteenth convolutional layer is connected to the input end of the channel attention layer; the output end of the channel attention layer is used to output data to the subsequent sequence.

[0061] In a feasible implementation, the noise removal module (UnoisyM) includes a third depth-separable convolution layer, a third SiLu activation layer, a fourth depth-separable convolution layer, a fourth SiLu activation layer, a fifth depth-separable convolution layer, a fifth SiLu activation layer and a sixth depth-separable convolution layer, which are arranged in sequence; in this way, the noise in the image can be processed by depth-separable convolution, the influence of image noise on the enhancement effect can be reduced, and the imaging clarity can be improved. At the same time, thanks to the volume advantage of depth-separable convolution, the number of parameters and the amount of calculation will not increase significantly.

[0062] In the third aspect, based on the same inventive concept, the present application also provides a low-light image enhancement device, including a processor, a memory and a bus, wherein the memory stores instructions and data read by the processor, the processor is used to call the instructions and data in the memory to execute the low-light image enhancement method as described above, and the bus connects the functional components for transmitting information.

[0063] By adopting the above technical solution, the present invention has the following beneficial effects:

[0064] The present invention provides a method, system and device for dark light image enhancement, which solve the color difference between the imaged image and the real image by performing color correction, and can avoid the color deviation problem caused by the traditional Retinex theoretical enhancement method; through multi-scale feature extraction, while maximally maintaining the image structural features, the image detail information is better extracted and effectively integrated, thereby improving the image enhancement quality and detail recovery ability; by separately processing the noise, the noise impact is reduced, and the image quality of the image is further improved; through the state space model, while maintaining the global information acquisition ability, the computational complexity is significantly reduced; compared with the traditional Transformer architecture, this solution not only has a simple structure and a small number of parameters, but also can reduce the amount of calculation to a linear level. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0066] Figure 1 A flow chart of a dark light image enhancement method provided by an embodiment of the present invention;

[0067] Figure 2 A low-light image enhancement system architecture diagram provided by an embodiment of the present invention;

[0068] Figure 3 A diagram of the cross-attention mechanism module architecture provided for an embodiment of the present invention;

[0069] Figure 4 A diagram showing the architecture of a color correction and multi-scale feature extraction module provided in an embodiment of the present invention;

[0070] Figure 5 A state space model architecture diagram provided for an embodiment of the present invention;

[0071] Figure 6 The architecture diagram of the original feature state space model and the retinal feature state space model provided in the embodiment of the present invention;

[0072] Figure 7 A diagram of the architecture of a multi-scale feature extraction module provided in an embodiment of the present invention;

[0073] Figure 8 A diagram of the noise removal module architecture provided by an embodiment of the present invention;

[0074] Fig. 9 A comparison diagram of the dark light image enhancement effect provided by an embodiment of the present invention: Figure a is the original dark light image; Figure b is the enhanced output image. DETAILED DESCRIPTION

[0075] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0076] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0077] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0078] The present invention is further explained below in conjunction with specific implementation modes.

[0079] It should also be noted that the following specific embodiments or specific implementations are a series of optimized settings listed in the present invention to further explain the specific content of the invention, and these settings can be used in combination or in association with each other.

[0080] Embodiment 1:

[0081] like Figure 1 As shown, a dark light image enhancement method provided in this embodiment includes the following steps:

[0082] Step 1: Preprocess the original image of the dark light image (size is H*W*3; where H represents the height of the image, W represents the width of the image, and 3 represents the number of channels of the image) to obtain a prior information map (Prior).

[0083] Preferably, the step 1 specifically includes: calculating the average value of each pixel of the original image on all color channels to obtain a priori information map (size is H*W*1).

[0084] Step 2: The prior information map is spliced ​​and fused with the original map to extract the brightness feature map LF (Light Feature) and the brightness map LM (Light Map).

[0085] Preferably, the step 2 specifically includes: splicing the original image and the prior information map by channel to obtain a first feature map (size is H*W*4); passing the first feature map through a convolution layer to perform image feature fusion to obtain a second feature map; extracting and fusing features of the second feature map through a depthwise separable convolution (DWConv) to obtain a brightness feature map LF and a third feature map respectively; passing the third feature map through a convolution layer to extract a brightness map LM (size is H*W*3).

[0086] Step 3: Multiply the brightness map LM by the original image element by element to obtain a brightened image IF, so that the brightness information can be integrated with the original image information to improve the image brightness; extract shallow features from the brightened image IF through a convolutional layer to obtain a fourth feature map.

[0087] Step 4: The highlighted image is fused with the fourth feature map through a cross attention mechanism to obtain a feature map AF (size is H*W*C). The specific formula can be:

[0088] ;

[0089] in, Display key Dimensions; Indicates a query; Represents a value.

[0090] Step 5: The feature map AF and the highlight map IF are processed multiple times through the color correction and multi-scale feature extraction module (CDUM), and a sampling operation is performed to obtain the twelfth feature map, which specifically includes:

[0091] The feature map AF and the highlight map IF are respectively subjected to layer normalization processing to obtain the corresponding fifth feature map; the two fifth feature maps are spliced ​​and fused in the channel dimension through a color correction module (CSSM) to output a sixth feature map; the sixth feature map is superimposed with the normalized feature map AF and the highlight map IF to obtain a seventh feature map to prevent detail loss;

[0092] The color correction module is a parallel state space model (SSM);

[0093] In this way, color correction is used to solve the defect of Retinex theory that ignores the imaging of dark light images, which leads to a large difference between the color of the image and the real image.

[0094] The seventh feature map is processed again by layer normalization to obtain an eighth feature map; the noise is eliminated by a noise removal module (UnoisyM) to obtain a ninth feature map; in this way, by processing the noise separately, the influence of the noise can be reduced, thereby improving the imaging quality of the image;

[0095] The multi-scale features are extracted by a multi-scale feature extraction module (DFM) to obtain a tenth feature map; the seventh feature map, the ninth feature map and the tenth feature map are added element by element to obtain an eleventh feature map, thereby ensuring the integrity of the image details; the eleventh feature map is downsampled or deconvolved upsampled to obtain a twelfth feature map;

[0096] In this way, through multi-scale feature extraction, the structural features of the image can be maintained to the maximum extent, and the detail information of the image can be better extracted; through multi-scale feature fusion, the image structure information and image information can be better fused, thereby improving the image enhancement quality and detail recovery capabilities.

[0097] Step 6: Reconstruct the twelfth feature map through the convolution layer to obtain the thirteenth feature map (size is H*W*3); add the thirteenth feature map and the brightened image IF element by element to obtain the output image, so as to enhance the detail features again, thereby ensuring that the dark light image is enhanced.

[0098] Furthermore, the noise removal module (UnoisyM) removes noise from the image by performing a depthwise separable convolution on the feature map.

[0099] Furthermore, the extraction method of the multi-scale feature extraction module (DFM) includes: processing the second normalized image through two parallel convolution branches, and then splicing it in the channel dimension; convolving the splicing result to perform information fusion in the channel dimension; and activating the information fusion result through the channel attention mechanism;

[0100] The convolution branch includes a first convolution branch and a second convolution branch;

[0101] The middle layer of the first convolution branch uses a convolution kernel of size 3 to extract detailed information of the image;

[0102] The middle layer of the second convolution branch uses a convolution kernel of size 5 to extract more detailed information of the image and enhance the connection between contexts in the image.

[0103] Embodiment 2:

[0104] like Figure 2 As shown, this embodiment provides a dark light image enhancement system, which specifically includes a brightening module, a U-Net network and a first adding unit;

[0105] The brightening module is used to receive the original image (Input) and the prior information image (Prior) of the dark light image, perform splicing and fusion processing, and output a brightened image;

[0106] The U-Net network specifically includes a first convolutional layer (Conv), a first enhancement module (AtSSM), a second convolutional layer, a second enhancement module, a third convolutional layer, a third enhancement module, a first deconvolutional layer (DeConv), a fourth convolutional layer, a fourth enhancement module, a second deconvolutional layer, a fifth convolutional layer, a fifth enhancement module and a sixth convolutional layer;

[0107] The input end of the first convolutional layer is connected to the output end of the highlighting module, and the output end of the first convolutional layer is connected to the first input end of the first enhancement module;

[0108] The second input end of the first enhancement module is connected to the output end of the highlighting module, and the first output end of the first enhancement module is connected to the input end of the second convolutional layer;

[0109] The output end of the second convolutional layer is connected to the input end of the second enhancement module; the first output end of the second enhancement module is connected to the input end of the third convolutional layer; the output end of the third convolutional layer is connected to the input end of the third enhancement module; the output end of the third enhancement module is connected to the input end of the first deconvolutional layer; the first output end of the first deconvolutional layer and the second output end of the second enhancement module are spliced ​​according to channels, and then connected to the first input end of the fourth convolutional layer; the second output end of the first deconvolutional layer is connected to the second input end of the fourth convolutional layer; the output end of the fourth convolutional layer is connected to the input end of the fourth enhancement module; the output end of the fourth enhancement module is connected to the input end of the second deconvolutional layer; the first output end of the second deconvolutional layer and the second output end of the first enhancement module are spliced ​​according to channels, and then connected to the first input end of the fifth convolutional layer; the second output end of the second deconvolutional layer is connected to the second input end of the fifth convolutional layer; the output end of the fifth convolutional layer is connected to the input end of the fifth enhancement module; the output end of the fifth enhancement module is connected to the input end of the sixth convolutional layer; the output end of the sixth convolutional layer serves as the output end of the U-Net network;

[0110] The first adding unit is used to add the feature map output by the U-Net network and the highlighting map output by the highlighting module element by element to obtain an output image (Output).

[0111] Furthermore, the highlighting module specifically includes a seventh convolutional layer, a first depth-separable convolutional layer, an eighth convolutional layer and a first multiplication unit which are arranged in sequence;

[0112] The input end of the seventh convolutional layer is used to receive the first feature map after the original image and the prior information image are spliced ​​according to the channel, perform image feature fusion, and output the second feature map;

[0113] The first depth separable convolution layer is used to receive the second feature map, perform feature extraction and fusion, and output a brightness feature map LF and a third feature map respectively;

[0114] The eighth convolutional layer is used to receive the third feature map and extract the brightness map LM;

[0115] The first multiplication unit is used to perform element-by-element multiplication of the original image and the brightness image LM to obtain a brightened image IF.

[0116] Furthermore, the enhancement module (AtSSM) specifically includes a cross-attention mechanism (IFA) module and a color correction and multi-scale feature extraction module (CDUM) which are arranged in sequence.

[0117] Furthermore, the cross attention mechanism (IFA) module calculates the global information of the highlight image IF according to the Retinex theoretical formula to obtain the feature map AF; the Retinex theoretical formula is specifically:

[0118] ;

[0119] in, Indicates the image seen; represents a reflected image; Represents the incident image.

[0120] Furthermore, if Figure 3 As shown, the cross-attention mechanism (IFA) module specifically includes a key branch and a value branch; the key branch includes a second multiplication unit for matrix multiplying the key of the fourth feature map with the highlight map IF to obtain the thirteenth feature map; the value branch includes a third multiplication unit for matrix multiplying the value of the fourth feature map with the thirteenth feature map to obtain the feature map AF.

[0121] Furthermore, if Figure 4 As shown, the color correction and multi-scale feature extraction module (CDUM) includes a first normalization layer, a second normalization layer, a state space model (CSSM), a second addition unit, a third normalization layer, a multi-scale feature extraction module (DFM), a noise removal module (UnoisyM) and a third addition unit;

[0122] An input end of the first normalization layer is connected to an output end of the highlighting module, and a first output end of the first normalization layer is connected to a first input end of the state space model;

[0123] An input end of the second normalization layer is connected to an output end of the cross attention mechanism module, and a first output end of the second normalization layer is connected to a second input end of the state space model;

[0124] The output end of the state space model is connected to the first input end of the second adding unit; the second input end of the second adding unit is connected to the second output end of the first normalization layer; the third input end of the second adding unit is connected to the second output end of the second normalization layer; the output end of the second adding unit is connected to the input end of the third normalization layer; the output end of the third normalization layer is respectively connected to the input end of the multi-scale feature extraction module and the input end of the noise removal module;

[0125] The first input end of the third adding unit is connected to the output end of the multi-scale feature extraction module; the second input end of the third adding unit is connected to the output end of the second adding unit; the third input end of the third adding unit is connected to the output end of the noise removal module; the output end of the third adding unit serves as the output end of the color correction and multi-scale feature extraction module;

[0126] Through the above architecture, the output features that have not been fused with the Retinex theory information can be fused with the output features that have been fused with the Retinex theory information, so that the original color information can be retained.

[0127] Furthermore, the parameter matrix calculation formula of the state space model includes:

[0128] ;

[0129] ;

[0130] in, represents the evolution parameter matrix, and , which specifically describes how the hidden state is updated as the time step changes and how the internal state of the system evolves without input, and can be regarded as a weight; Represents the discretization ; represents the projection parameter matrix, and , specifically describes the contribution of input to state update and the impact of external input on system state, which can be regarded as weight; Represents the discretization ; represents the same matrix; represents the time scale parameter, which belongs to the real number domain ; Indicates the size of the state space; in this way, the continuous parameter matrix can be discretized by the zero-order hold method to obtain a discrete parameter matrix. Compared with the traditional state space model (SMM), this scheme is more in line with the characteristics of non-continuous images input in visual tasks;

[0131] The calculation formula of the state space model can be expressed in a manner similar to RNN as follows:

[0132] ;

[0133] ;

[0134] in, express The sequence of input moments, , Indicates the length of the input sequence; represents the hidden state, and ; Represents the output matrix, which is used to transform the hidden state Mapping is performed, that is, linear transformation of the hidden state; Represents a constant term, which is used to represent the direct impact of input on output. It can be used when input has an impact on output. It is mainly determined by system characteristics and is randomly initialized and continuously adjusted in machine learning. It can be used as a weight matrix.

[0135] The calculation formula of the state space model can also be expressed in the form of CNN, specifically including:

[0136] ;

[0137] ;

[0138] in, represents the convolution kernel, and ; Represents the approximate convolution kernel.

[0139] Furthermore, if Figure 5 As shown, the state space model (CSSM) includes an original feature state space model (OVSSM), a retinal feature state space model (RVSSM) and a ninth convolutional layer; the input end of the original feature state space model serves as the first input end of the state space model; the input end of the retinal feature state space model serves as the second input end of the state space model; the output end of the original feature state space model and the output end of the retinal feature state space model are spliced ​​according to channels and then connected to the input end of the ninth convolutional layer; the output end of the ninth convolutional layer serves as the output end of the state space model;

[0140] like Figure 6 As shown, the original feature state space model (OVSSM) and the retinal feature state space model (RVSSM) both include a first linear layer, a second linear layer, a second depth-separable convolutional layer, a first SiLu activation layer, a second SiLu activation layer, a two-dimensional state space model layer (2D-SSM), a fourth normalization layer, a fourth multiplication unit and a third linear layer;

[0141] The input end of the first linear layer is connected to the input end of the second linear layer for synchronously receiving preceding input data;

[0142] The output end of the first linear layer is connected to the input end of the second depthwise separable convolutional layer; the output end of the second depthwise separable convolutional layer is connected to the input end of the first SiLu activation layer; the output end of the first SiLu activation layer is connected to the input end of the two-dimensional state space model layer; the output end of the two-dimensional state space model layer is connected to the input end of the fourth normalization layer; the output end of the fourth normalization layer is connected to the first input end of the fourth multiplication unit, thus forming a first processing branch;

[0143] The output end of the second linear layer is connected to the input end of the second SiLu activation layer; the output end of the second SiLu activation layer is connected to the second input end of the fourth multiplication unit, thus forming a second processing branch;

[0144] The output end of the fourth multiplication unit is connected to the input end of the third linear layer for matrix multiplication; the output end of the third linear layer is used to output data to a subsequent order.

[0145] Furthermore, the formula of the state space model (CSSM) processing process can be expressed as:

[0146] ;

[0147] ;

[0148] ;

[0149] ;

[0150] ;

[0151] ;

[0152] ;

[0153] in, Represents the input image, the result of information fusion through Retinex theory; express The result of the first processing branch is passed; express The result of the second processing branch is passed; express The results after processing by the retinal feature state space model (RVSSM); Indicates splicing by channel;

[0154] Represents the input image that has not been fused through the Retinex theory information; express The result of the first processing branch is passed; express The result of the second processing branch is passed; express The results after processing by the original feature state space model (OVSSM);

[0155] Represents the processing results of the state space model (CSSM).

[0156] Furthermore, if Figure 7 As shown, the multi-scale feature extraction module (DFM) includes a tenth convolutional layer, an eleventh convolutional layer, a twelfth convolutional layer, a thirteenth convolutional layer, a fourteenth convolutional layer, a fifteenth convolutional layer, a sixteenth convolutional layer and a channel attention layer;

[0157] The input end of the tenth convolutional layer is connected to the input end of the thirteenth convolutional layer, and is used for synchronously receiving the preceding input data;

[0158] The output end of the tenth convolutional layer is connected to the input end of the eleventh convolutional layer; the output end of the eleventh convolutional layer is connected to the input end of the twelfth convolutional layer, thereby forming a detail extraction branch; the convolution kernel size of the eleventh convolutional layer is 3*3, so as to extract detail features of the image;

[0159] The output end of the thirteenth convolutional layer is connected to the input end of the fourteenth convolutional layer; the output end of the fourteenth convolutional layer is connected to the input end of the fifteenth convolutional layer, thus forming a semantic extraction branch; the convolution kernel size of the fourteenth convolutional layer is 5*5, so as to obtain more image context semantic information, so as to better restore the local features of the image;

[0160] The output end of the twelfth convolutional layer and the output end of the fifteenth convolutional layer are connected to the input end of the sixteenth convolutional layer after channel splicing processing; the output end of the sixteenth convolutional layer is connected to the input end of the channel attention layer; the output end of the channel attention layer is used to output data to the subsequent sequence.

[0161] Furthermore, the formula for the processing process of the multi-scale feature extraction module can be expressed as:

[0162] ;

[0163] ;

[0164] ;

[0165] in, Represents the output result obtained above in the detail extraction branch; Represents the output result obtained above in the semantic extraction branch; Represents a feedforward network with a convolution kernel size of 3*3; Represents a feedforward network with a convolution kernel size of 5*5; Represents the output result after the channel attention mechanism is activated.

[0166] Furthermore, if Figure 8 As shown, the noise removal module (UnoisyM) includes a third depth-separable convolution layer, a third SiLu activation layer, a fourth depth-separable convolution layer, a fourth SiLu activation layer, a fifth depth-separable convolution layer, a fifth SiLu activation layer and a sixth depth-separable convolution layer, which are sequentially arranged; in this way, the noise in the image can be processed by the depth-separable convolution, the influence of the image noise on the enhancement effect can be reduced, and the imaging clarity can be improved. At the same time, thanks to the volume advantage of the depth-separable convolution, the amount of parameters and the amount of calculation will not increase significantly;

[0167] Furthermore, the noise removal module processing process may be expressed as follows:

[0168] ;

[0169] in, Represents the processing result of the noise removal module.

[0170] The scheme of this embodiment is used to enhance the dark light image of an archery hall, and the original image is compared with the output image. Fig. 9 As shown in the figure, Figure a is the original image in dark light, and Figure b is the enhanced output image. It can be seen from the lamp tube, text, target and other parts in Figure b that both image details and colors have been effectively improved, and the imaging quality of the output image is very close to the real image under normal light.

[0171] Embodiment three:

[0172] The present embodiment provides a low-light image enhancement device, including a processor, a memory and a bus, wherein the memory stores instructions and data read by the processor, the processor is used to call the instructions and data in the memory to execute the low-light image enhancement method as described above, and the bus connects the functional components to transmit information.

[0173] In another embodiment, the present solution can be implemented by an integrated device, which may include corresponding modules for performing each or several steps in the above-mentioned embodiments. The module may be one or more hardware modules specially configured to perform the corresponding steps, or implemented by a processor configured to perform the corresponding steps, or stored in a computer-readable medium for implementation by a processor, or implemented by some combination.

[0174] The processor performs the various methods and processes described above. For example, the method implementation in the present solution can be implemented as a software program, which is tangibly contained in a machine-readable medium, such as a memory. In some embodiments, part or all of the software program can be loaded and / or installed via a memory and / or a communication interface. When the software program is loaded into the memory and executed by the processor, one or more steps in the method described above can be executed. Alternatively, in other embodiments, the processor can be configured to perform one of the above methods by any other appropriate means (for example, by means of firmware).

[0175] The device can be implemented using a bus architecture. The bus architecture can include any number of interconnecting buses and bridges, depending on the specific application and overall design constraints of the hardware. The bus connects various circuits including one or more processors, memories, and / or hardware modules together. The bus can also connect various other circuits such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0176] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0177] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A dark light image enhancement method, characterized in that: include: Step 1: preprocess the original image of the dark light image to obtain a prior information image; Step 2, splicing and fusing the prior information map with the original map to extract a brightness feature map and a brightness map, specifically comprising: splicing the original map with the prior information map by channel to obtain a first feature map; passing the first feature map through a convolution layer to perform image feature fusion to obtain a second feature map; performing feature extraction and fusion on the second feature map through a depthwise separable convolution to obtain a brightness feature map and a third feature map respectively; passing the third feature map through a convolution layer to extract a brightness map; Step 3: multiply the brightness image by the original image element by element to obtain a brightened image; extract shallow features from the brightened image through a convolutional layer to obtain a fourth feature map; Step 4: The highlighted image is fused with the fourth feature map through a cross attention mechanism to obtain a feature map AF. The specific formula is: ； in, Display key Dimensions; Indicates a query; Represents a value; Step 5: Process the feature map AF and the highlight map multiple times through the color correction and multi-scale feature extraction modules, and perform sampling operations to obtain the twelfth feature map, which specifically includes: The feature map AF and the highlight map are respectively subjected to layer normalization processing to obtain the corresponding fifth feature map; the two fifth feature maps are spliced ​​and fused in the channel dimension through the color correction module to output the sixth feature map; the sixth feature map is superimposed with the normalized feature map AF and the highlight map to obtain the seventh feature map; The color correction module is a parallel state space model; The seventh feature map is subjected to layer normalization again to obtain an eighth feature map; noise is eliminated by a noise removal module to obtain a ninth feature map; Extracting multi-scale features through a multi-scale feature extraction module to obtain a tenth feature map; adding the seventh feature map, the ninth feature map, and the tenth feature map element by element to obtain an eleventh feature map; performing a downsampling or a deconvolution upsampling operation on the eleventh feature map to obtain a twelfth feature map; Step 6: reconstruct the twelfth feature map through a convolution layer to obtain a thirteenth feature map; add the thirteenth feature map and the highlight map element by element to obtain an output image; The state space model includes an original feature state space model, a retinal feature state space model and a ninth convolutional layer; the input end of the original feature state space model serves as the first input end of the state space model; the input end of the retinal feature state space model serves as the second input end of the state space model; the output end of the original feature state space model and the output end of the retinal feature state space model are spliced ​​according to channels and then connected to the input end of the ninth convolutional layer; the output end of the ninth convolutional layer serves as the output end of the state space model; The original feature state space model and the retinal feature state space model both include a first linear layer, a second linear layer, a second depth-separable convolutional layer, a first SiLu activation layer, a second SiLu activation layer, a two-dimensional state space model layer, a fourth normalization layer, a fourth multiplication unit and a third linear layer; The input end of the first linear layer is connected to the input end of the second linear layer for synchronously receiving preceding input data; The output end of the first linear layer is connected to the input end of the second depthwise separable convolutional layer; the output end of the second depthwise separable convolutional layer is connected to the input end of the first SiLu activation layer; the output end of the first SiLu activation layer is connected to the input end of the two-dimensional state space model layer; the output end of the two-dimensional state space model layer is connected to the input end of the fourth normalization layer; the output end of the fourth normalization layer is connected to the first input end of the fourth multiplication unit, thus forming a first processing branch; The output end of the second linear layer is connected to the input end of the second SiLu activation layer; the output end of the second SiLu activation layer is connected to the second input end of the fourth multiplication unit, thus forming a second processing branch; The output end of the fourth multiplication unit is connected to the input end of the third linear layer for matrix multiplication; the output end of the third linear layer is used to output data to the subsequent sequence; The parameter matrix calculation formula of the state space model includes: ； ； in, represents the evolution parameter matrix, and ; Represents the discretization ; represents the projection parameter matrix, and ; Represents the discretization ; represents the same matrix; represents the time scale parameter, which belongs to the real number domain ; Represents the size of the state space.

2. The method according to claim 1, characterized in that The step 1 specifically includes: calculating the average value of each pixel point of the original image on all color channels to obtain a priori information map.

3. The method according to claim 1, characterized in that The extraction method of the multi-scale feature extraction module includes: processing the second normalized image through two parallel convolution branches, and then splicing the images in the channel dimension; convolving the splicing result to perform information fusion in the channel dimension; and activating the information fusion result through a channel attention mechanism; The convolution branch includes a first convolution branch and a second convolution branch; The middle layer of the first convolution branch uses a convolution kernel of a certain size to extract detailed information of the image; The middle layer of the second convolution branch uses a convolution kernel of another size to extract more detailed information of the image and enhance the connection between contexts in the image.

4. A low-light image enhancement system using the method as claimed in any one of claims 1 to 3, characterized in that: include: Highlighting module, U-Net network and first adding unit; The brightening module is used to receive the original image and the prior information image of the dark light image, perform splicing and fusion processing, and output a brightened image; The U-Net network specifically includes a first convolutional layer, a first enhancement module, a second convolutional layer, a second enhancement module, a third convolutional layer, a third enhancement module, a first deconvolutional layer, a fourth convolutional layer, a fourth enhancement module, a second deconvolutional layer, a fifth convolutional layer, a fifth enhancement module and a sixth convolutional layer; The input end of the first convolutional layer is connected to the output end of the highlighting module, and the output end of the first convolutional layer is connected to the first input end of the first enhancement module; The second input end of the first enhancement module is connected to the output end of the highlighting module, and the first output end of the first enhancement module is connected to the input end of the second convolutional layer; The output end of the second convolutional layer is connected to the input end of the second enhancement module; the first output end of the second enhancement module is connected to the input end of the third convolutional layer; the output end of the third convolutional layer is connected to the input end of the third enhancement module; the output end of the third enhancement module is connected to the input end of the first deconvolutional layer; the first output end of the first deconvolutional layer and the second output end of the second enhancement module are spliced ​​according to channels, and then connected to the first input end of the fourth convolutional layer; the second output end of the first deconvolutional layer is connected to the second input end of the fourth convolutional layer; the output end of the fourth convolutional layer is connected to the input end of the fourth enhancement module; the output end of the fourth enhancement module is connected to the input end of the second deconvolutional layer; the first output end of the second deconvolutional layer and the second output end of the first enhancement module are spliced ​​according to channels, and then connected to the first input end of the fifth convolutional layer; the second output end of the second deconvolutional layer is connected to the second input end of the fifth convolutional layer; the output end of the fifth convolutional layer is connected to the input end of the fifth enhancement module; the output end of the fifth enhancement module is connected to the input end of the sixth convolutional layer; the output end of the sixth convolutional layer serves as the output end of the U-Net network; The first adding unit is used to add the feature map output by the U-Net network and the highlighting map output by the highlighting module element by element to obtain an output image.

5. The system according to claim 4, characterized in that The highlighting module specifically includes a seventh convolutional layer, a first depth-separable convolutional layer, an eighth convolutional layer and a first multiplication unit which are arranged in sequence; The input end of the seventh convolutional layer is used to receive the first feature map after the original image and the prior information image are spliced ​​according to the channel, perform image feature fusion, and output the second feature map; The first depth-separable convolutional layer is used to receive the second feature map, perform feature extraction and fusion, and output a brightness feature map and a third feature map respectively; The eighth convolutional layer is used to receive the third feature map and extract the brightness map; The first multiplication unit is used to perform element-by-element multiplication of the original image and the brightness image to obtain a brightened image.

6. The system according to claim 5, characterized in that The enhancement module specifically includes a cross attention mechanism module and a color correction and multi-scale feature extraction module which are arranged in sequence; The cross attention mechanism module specifically includes a key branch and a value branch; the key branch includes a second multiplication unit, which is used to perform matrix multiplication of the key of the fourth feature map and the highlight map to obtain a thirteenth feature map; the value branch includes a third multiplication unit, which is used to perform matrix multiplication of the value of the fourth feature map and the thirteenth feature map to obtain a feature map AF; The color correction and multi-scale feature extraction module includes a first normalization layer, a second normalization layer, a state space model, a second addition unit, a third normalization layer, a multi-scale feature extraction module, a noise removal module and a third addition unit; An input end of the first normalization layer is connected to an output end of the highlighting module, and a first output end of the first normalization layer is connected to a first input end of the state space model; An input end of the second normalization layer is connected to an output end of the cross attention mechanism module, and a first output end of the second normalization layer is connected to a second input end of the state space model; The output end of the state space model is connected to the first input end of the second adding unit; The second input end of the second adding unit is connected to the second output end of the first normalization layer; the third input end of the second adding unit is connected to the second output end of the second normalization layer; the output end of the second adding unit is connected to the input end of the third normalization layer; the output end of the third normalization layer is respectively connected to the input end of the multi-scale feature extraction module and the input end of the noise removal module; The first input end of the third adding unit is connected to the output end of the multi-scale feature extraction module; the second input end of the third adding unit is connected to the output end of the second adding unit; the third input end of the third adding unit is connected to the output end of the noise removal module; and the output end of the third adding unit serves as the output end of the color correction and multi-scale feature extraction module.

7. The system according to claim 6, characterized in that The multi-scale feature extraction module includes a tenth convolutional layer, an eleventh convolutional layer, a twelfth convolutional layer, a thirteenth convolutional layer, a fourteenth convolutional layer, a fifteenth convolutional layer, a sixteenth convolutional layer and a channel attention layer; The input end of the tenth convolutional layer is connected to the input end of the thirteenth convolutional layer, and is used for synchronously receiving the preceding input data; The output end of the tenth convolutional layer is connected to the input end of the eleventh convolutional layer; the output end of the eleventh convolutional layer is connected to the input end of the twelfth convolutional layer, thus forming a detail extraction branch; the eleventh convolutional layer is used to extract detail features of the image; The output end of the thirteenth convolutional layer is connected to the input end of the fourteenth convolutional layer; the output end of the fourteenth convolutional layer is connected to the input end of the fifteenth convolutional layer, thus forming a semantic extraction branch; The fourteenth convolutional layer is used to extract image context semantic information; The output end of the twelfth convolutional layer and the output end of the fifteenth convolutional layer are connected to the input end of the sixteenth convolutional layer after channel splicing processing; the output end of the sixteenth convolutional layer is connected to the input end of the channel attention layer; the output end of the channel attention layer is used to output data to the subsequent sequence.

8. The system according to claim 7, characterized in that The noise removal module includes a third depth-separable convolution layer, a third SiLu activation layer, a fourth depth-separable convolution layer, a fourth SiLu activation layer, a fifth depth-separable convolution layer, a fifth SiLu activation layer and a sixth depth-separable convolution layer, which are arranged in sequence.

9. A low-light image enhancement device, characterized in that: It includes a processor, a memory and a bus, wherein the memory stores instructions and data read by the processor, the processor is used to call the instructions and data in the memory to implement the system as described in claim 4, and the bus connects the functional components to transmit information.

Images