A low-light image enhancement method based on multi-scale Transformer-Unet wavelet transform

Abstract

The present invention discloses a method for enhancing low-light images based on a multi-scale Transformer-Unet wavelet transform, comprising the following steps: Step 1: Given a low-light image, a convolutional neural network is used to enhance the image and extract the illumination feature F of the low-light image. light , shallow feature F0, and gradient feature F gradient , Step 2, the shallow feature F0 and the gradient feature F gradient As the input of the Transformer-Unet network, the intermediate enhanced image I is obtained PE , Step 3, use the illumination feature F lighe As a guide for fine-tuning the illumination, the mid-level enhanced image I PE The illumination is fine-tuned to obtain the target exposure image. The present invention overcomes the limitations of convolutional neural network calculations and makes up for the Transformer block's neglect of local information in low-light images, ultimately obtaining a target exposure image with normal exposure and perfect details. This not only ensures the detail information of the low-light image, but also achieves illumination enhancement of the low-light image, and can adapt to the execution of high-level visual tasks such as target tracking.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a multi-scale Transformer-Unet low-light image enhancement method based on wavelet transform. Background Art

[0002] In recent years, science and technology have developed rapidly, and artificial intelligence technology has gradually become popular in all walks of life. It has broad prospects in the fields of autonomous driving and security. However, at night or in other conditions of insufficient light, the images captured are often damaged, with problems such as low brightness and poor contrast, which greatly affects the performance of high-level visual tasks such as target detection and target tracking. Therefore, it is necessary to correct the illumination of low-light images.

[0003] Traditional low-light image enhancement usually uses histogram equalization to adjust the overall contrast of the image, but this method can only achieve a small improvement, and because the grayscale after equalization is reduced, some information will disappear, which will also cause unnatural enhancement of the image. Low-light image enhancement methods based on learning often use convolutional neural networks, but convolutional neural networks focus on extracting local features and are not good at extracting global features.

[0004] In view of this, whether it is possible to achieve enhancement of low-light images while maintaining image details to adapt to high-level visual tasks is an urgent problem to be solved;

[0005] Therefore, it is necessary to invent a multi-scale Transformer-Unet low-light image enhancement method based on wavelet transform to solve the above problems. Summary of the invention

[0006] The purpose of the present invention is to provide a multi-scale Transformer-Unet low-light image enhancement method based on wavelet transform. By making full use of the global modeling capability of Transformer and the multi-scale interactive framework constructed by the Unet network, and using wavelet transform to realize information interaction across feature domains, the limitations of convolutional neural network calculation are overcome, and the Transformer block makes up for the neglect of local information in the low-light image. Finally, a target exposure image with normal exposure and perfect details is obtained, which not only ensures the detail information of the low-light image, but also realizes the illumination enhancement of the low-light image, and can adapt to the execution of high-level visual tasks such as target tracking, so as to solve the above-mentioned shortcomings in the technology.

[0007] In order to achieve the above object, the present invention provides the following technical solution: a multi-scale Transformer-Unet low-light image enhancement method based on wavelet transform, comprising the following steps:

[0008] Step 1: Given a low-light image, use a convolutional neural network to enhance it and extract the illumination feature F of the low-light image. light , shallow feature F0, and gradient feature F gradient ;

[0009] Step 2: Combine shallow feature F0 and gradient feature F gradient As the input of the Transformer-Unet network, the intermediate enhanced image I is obtained PE ;

[0010] Step 3: Use the illumination feature F light As a guide for fine-tuning the illumination, the mid-level enhanced image I PE Fine-tune the lighting to obtain the target exposure image.

[0011] In the aforementioned multi-scale Transformer-Unet low-light image enhancement method based on wavelet transform, in step 1, a low-light image is given, and a convolutional neural network is used to enhance it to extract the illumination feature F of the low-light image. light , shallow feature F0, and gradient feature F gradient The specific steps are as follows:

[0012] 1.1. Assume I L Represents a low-light image, and uses edge detection operators to extract the low-light image I L The gradient feature F gradient ;

[0013] 1.2. Low-light image I L and mean image I mean Cascade, use convolutional neural network for enhancement processing, and get the primary enhanced image I PL ;

[0014] 1.3. Extracting primary enhanced image I from the middle layer of convolutional neural network PL The illumination feature F light ;

[0015] Among them, I L ∈I H×W×3 , H is the height of image I, W is the width of image I, and C is the number of channels of image I;

[0016] And I mean ∈I H×W×1 , for I L The average of three channels is obtained;

[0017] 1.4. Use 3×3 convolutional layer to extract primary enhanced image I PL The shallow feature F0 of .

[0018] In the aforementioned wavelet-based multi-scale Transformer-Unet low-light image enhancement method, in step 2, the shallow feature F0 and the gradient feature F gradient As the input of the Transformer-Unet network, the intermediate enhanced image I is obtained PE The specific steps are as follows:

[0019] 2.1. Use the Transformer block to learn the features in the RGB domain for the shallow feature F0 to obtain the intermediate feature F1;

[0020] 2.2. Decompose the intermediate layer feature F1 using Haar wavelet and convert it into the wavelet domain, using the gradient feature F gradient As a guide and combined with the gradient feature F gradient Fusion, using stacked multiple Transformer blocks for feature learning to obtain the intermediate layer feature F2;

[0021] 2.3. Use convolution to downsample by 2 times and use gradient feature F gradient As a guide, it is fused with the downsampling result, and multiple stacked Transformer blocks are used for feature learning to obtain the intermediate layer feature F3;

[0022] 2.4. Use convolution to downsample by 2 times and use gradient feature F gradient As a guide, it is fused with the down-sampling result, and multiple cascaded Transformer blocks are used for feature learning to obtain the intermediate layer feature F4, which is used as the bottleneck layer;

[0023] 2.5. In the jump connection part, the intermediate layer features F1, F2, F3, and F4 are uniformly input into the CDSF module to obtain the feature weights of different scales and domains and fuse them, and add the weights with the output of the decoder stage to obtain the intermediate enhanced image I PE .

[0024] In the aforementioned wavelet transform-based multi-scale Transformer-Unet low-light image enhancement method, in step 2.1, the shallow feature F0 is learned in the RGB domain using the Transformer block to obtain the intermediate feature F1. The specific formula is:

[0025] F′=LayerNorm(F0)

[0026] F A =WSA(F′)+AbSA(F′)

[0027] F0′=F0+F A

[0028] F″=LayerNorm(F′0)

[0029] F1=DGFN(F″)+F0′

[0030] Among them, LayerNorm(·) is the layer normalization processing,

[0031] WSA(·) is the window attention,

[0032] AbSA(·) is axial attention,

[0033] DGFN(·) is a feed-forward network with a double gating mechanism.

[0034] In the aforementioned wavelet-based multi-scale Transformer-Unet low-light image enhancement method, in step 2.2, the intermediate layer feature F1 is decomposed into the wavelet domain using Haar wavelet, and the gradient feature F gradient As a guide and combined with the gradient feature F gradient Fusion, using stacked multiple Transformer blocks for feature learning, to obtain the intermediate layer feature F2, the specific formula is:

[0035] F LL ,F LH ,F HL ,F HH =DWT(F1)

[0036] F1′=[F LL ,F LH ,F HL ,F HH ]

[0037] F1″=DOF(F1′,F gradient )

[0038] F2=WAT(F1″)

[0039] Among them, DWT(·) is wavelet transform,

[0040] DOF(·) is the feature fusion using deep over-parameter convolution.

[0041] WAT(·) is the calculation process in step 2.1;

[0042] F LL is the low frequency component,

[0043] F LH is the high frequency component in the horizontal direction,

[0044] F HL is the high frequency component in the vertical direction,

[0045] F HH is the high frequency component in the diagonal direction.

[0046] The aforementioned wavelet-based multi-scale Transformer-Unet low-light image enhancement method uses convolution for 2x downsampling in step 2.3, and uses the gradient feature F gradient As a guide, it is fused with the downsampling result, and multiple stacked Transformer blocks are used for feature learning to obtain the intermediate layer feature F3. The specific formula is:

[0047] F2′=Downsampling(F2)

[0048] F2″=DOF(F2′,F gradient )

[0049] F3=WAT(F2″)

[0050] Among them, Downsampling(·) is the convolution downsampling by 2 times.

[0051] The aforementioned wavelet-based multi-scale Transformer-Unet low-light image enhancement method uses convolution for 2x downsampling in step 2.4, and uses the gradient feature F gradient As a guide, it is fused with the downsampling result, and multiple cascaded Transformer blocks are used for feature learning to obtain the intermediate layer feature F4. The specific formula is:

[0052] F3′=Downsampling(F3)

[0053] F3″=DOF(F3′,F gradient )

[0054] F4 = WAT(F3″).

[0055] In the aforementioned wavelet-based multi-scale Transformer-Unet low-light image enhancement method, in step 3, the illumination feature F light As a guide for fine-tuning the illumination, the mid-level enhanced image I PE Fine-tune the lighting to obtain the target exposure image. The specific process is as follows:

[0056] The illumination feature F light and intermediate enhanced image I PE The target exposure image can be obtained by inputting them into the illumination adjustment module LAM to perform illumination adjustment.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] The present invention uses a convolutional neural network to extract shallow features and gradient features of low-light images, which are then input into the Transformer-Unet network, and uses illumination features as a guide to obtain a target exposure image. The global modeling capability of the Transformer and the multi-scale interaction framework constructed by the Unet network are fully utilized, and wavelet transform is used to realize information interaction across feature domains, thereby overcoming the limitations of convolutional neural network calculations and making up for the Transformer block's neglect of local information in low-light images. Finally, a target exposure image with normal exposure and perfect details is obtained, which not only ensures the detail information of the low-light image, but also realizes illumination enhancement of the low-light image, and can adapt to the execution of high-level visual tasks such as target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0060] Figure 1 is a flow chart of the present invention;

[0061] Figure 2 is a given low-light image in an embodiment of the present invention;

[0062] Figure 3 is the target exposure image obtained by the present invention;

[0063] Figure 4 is the actual exposure image for a given low-light image. DETAILED DESCRIPTION

[0064] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0065] The present invention provides Figure 1-4 The multi-scale Transformer-Unet low-light image enhancement method shown in the figure includes the following steps:

[0066] Step 1: Given a low-light image, extract the illumination features and gradient features of the low-light image, and use a convolutional neural network to enhance it to obtain a primary enhanced image, and extract the shallow features of the primary enhanced image. The specific steps are as follows:

[0067] 1.1. Assume I L Represents a low-light image, and uses edge detection operators to extract the low-light image I LThe gradient feature F gradient ;

[0068] 1.2. Low-light image I L and mean image I mean Cascade, use convolutional neural network for enhancement processing, and get the primary enhanced image I PL ;

[0069] 1.3. Extracting primary enhanced image I from the middle layer of convolutional neural network PL The illumination feature F light ;

[0070] Among them, I L ∈I H×W×3 , H is the height of image I, W is the width of image I, and C is the number of channels of image I;

[0071] I mean ∈I H×W×1 , for I L The average of three channels is obtained;

[0072] 1.4. Use 3×3 convolutional layer to extract primary enhanced image I PL The shallow feature F0;

[0073] In this step, by introducing the mean image I mean , the low-light image can be initially enhanced to achieve a certain degree of lighting, which is convenient for subsequent operations.

[0074] Step 2: Combine shallow feature F0 and gradient feature F gradient As the input of the Transformer-Unet network, the intermediate enhanced image I is obtained PE , the specific steps are as follows:

[0075] 2.1. Use the Transformer block to learn the features in the RGB domain for the shallow feature F0 to obtain the intermediate feature F1. The specific formula is:

[0076] F′=LayerNorm(F0)

[0077] F A =WSA(F′)+AbSA(F′)

[0078] F0′=F0+F A

[0079] F″=LayerNorm(F′0)

[0080] F1=DGFN(F″)+F0′

[0081] Among them, LayerNorm(·) is the layer normalization processing,

[0082] WSA(·) is the window attention,

[0083] AbSA(·) is axial attention,

[0084] DGFN(·) is a feed-forward network with a double gating mechanism;

[0085] 2.2. Decompose the intermediate layer feature F1 using Haar wavelet and convert it into the wavelet domain, using the gradient feature F gradient As a guide and combined with the gradient feature F gradient Fusion, using stacked multiple Transformer blocks for feature learning, to obtain the intermediate layer feature F2, the specific formula is:

[0086] F LL ,F LH ,F HL ,F HH =DWT(F1)

[0087] F1′=[F LL ,F LH ,F HL ,F HH ]

[0088] F1″=DOF(F1′,F gradient )

[0089] F2=WAT(F1″)

[0090] Among them, DWT(·) is wavelet transform,

[0091] DOF(·) is the feature fusion using deep over-parameter convolution.

[0092] WAT(·) is the calculation process in step 2.1;

[0093] F LL is the low frequency component,

[0094] F LH is the high frequency component in the horizontal direction,

[0095] F HL is the high frequency component in the vertical direction,

[0096] F HH is the high frequency component in the diagonal direction;

[0097] 2.3. Use convolution to downsample by 2 times and use gradient feature F gradient As a guide, it is fused with the downsampling result, and multiple stacked Transformer blocks are used for feature learning to obtain the intermediate layer feature F3. The specific formula is:

[0098] F2′=Downsampling(F2)

[0099] F2″=DOF(F2′,F gradient )

[0100] F3=WAT(F2″)

[0101] Among them, Downsampling (·) is the convolution downsampling by 2 times;

[0102] 2.4. Use convolution to downsample by 2 times and use gradient feature F gradient As a guide, it is fused with the downsampling result, and multiple cascaded Transformer blocks are used for feature learning to obtain the intermediate layer feature F4, which is used as the bottleneck layer. The specific formula for obtaining the intermediate layer feature F4 is:

[0103] F3′=Downsampling(F3)

[0104] F3″=DOF(F3′,F gradient )

[0105] F4=WAT(F3″)

[0106] 2.5. In the jump connection part, the intermediate layer features F1, F2, F3, and F4 are uniformly input into the CDSF module to obtain the feature weights of different scales and domains and fuse them. The weights are added to the output of the decoder stage to obtain the intermediate enhanced image I. PE ;

[0107] In this step, the DOF module helps to enhance the decomposition effect in the encoder stage. Combined with the window attention and axial attention in the WAT module, local features and global correlations can be better learned. The feedforward network with a dual-gating mechanism can help the model extract more beneficial information. At the same time, the traditional jump connection uses simple addition, while the CDSF module in this step is more helpful in integrating cross-domain multi-scale features, which can achieve mutual compensation for insufficient semantic information of high-resolution images and insufficient details of low-resolution images, and can better extract useful features. DSF has good cross-domain multi-scale fusion effect, good Transformer feature learning effect, and through the introduction of gradient features, the shallow feature F0 details are restored in the Transformer-Unet network part, and the amplified noise for preliminary enhancement of low-light images can be suppressed.

[0108] Step 3: Use the illumination feature F light As a guide for fine-tuning the illumination, the mid-level enhanced image I PEFine-tune the lighting to obtain the target exposure image. The specific process is as follows:

[0109] The illumination feature F light and intermediate enhanced image I PE The target exposure image can be obtained by inputting them into the illumination adjustment module LAM to perform illumination adjustment.

[0110] In this step, by taking the extracted illumination features as a guide, the local illumination of the intermediate enhanced image can be fine-tuned to avoid the problem of local overexposure in the final enhanced image, thereby obtaining a target exposure image with good exposure and perfect details.

[0111] Verification experiment

[0112] like Figure 2 As shown, it is a low-light image. After being processed by the present invention, a target exposure image is obtained, that is, Figure 3 As shown, at this point, it can be seen visually that Figure 3 A good lighting enhancement effect is achieved, which not only restores the dark information, but also enhances the texture and other details. Figure 4 , and there is no obvious difference, the present invention can be used to obtain reliable target exposure images.

[0113] In summary, the present invention uses a convolutional neural network to extract shallow features and gradient features of low-light images, and then inputs them into the Transformer-Unet network, and uses illumination features as a guide to obtain a target exposure image, making full use of the global modeling capability of the Transformer and the multi-scale interaction framework constructed by the Unet network, and using wavelet transform to realize information interaction across feature domains, overcoming the limitations of convolutional neural network calculations, and making up for the Transformer block's neglect of local information in low-light images. Finally, a target exposure image with normal exposure and perfect details is obtained, which not only ensures the detail information of the low-light image, but also realizes illumination enhancement of the low-light image, and can adapt to the execution of high-level visual tasks such as target tracking.

[0114] The above description is only by way of illustration of certain exemplary embodiments of the present invention. It is undoubted that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A multi-scale Transformer-Unet low-light image enhancement method based on wavelet transform, characterized by: The following steps are involved: Step 1: Given a low-light image, use a convolutional neural network to enhance it and extract the illumination feature F of the low-light image. light , shallow feature F0, and gradient feature F gradient ; Step 2: Combine shallow feature F0 and gradient feature F gradient As the input of the Transformer-Unet network, the intermediate enhanced image I is obtained PE , the specific steps are as follows: 2.

1. Use the Transformer block to learn the features in the RGB domain for the shallow feature F0 to obtain the intermediate feature F1; 2.

2. Decompose the intermediate layer feature F1 using Haar wavelet and convert it into the wavelet domain, using the gradient feature F gradient As a guide and combined with the gradient feature F gradient Fusion, using stacked multiple Transformer blocks for feature learning to obtain the intermediate layer feature F2; 2.

3. Use convolution to downsample by 2 times and use gradient feature F gradient As a guide, it is fused with the downsampling result, and multiple stacked Transformer blocks are used for feature learning to obtain the intermediate layer feature F3; 2.

4. Use convolution to downsample by 2 times and use gradient feature F gradient As a guide, it is fused with the down-sampling result, and multiple cascaded Transformer blocks are used for feature learning to obtain the intermediate layer feature F4, which is used as the bottleneck layer; 2.

5. In the jump connection part, the intermediate layer features F1, F2, F3, and F4 are uniformly input into the CDSF module to obtain the feature weights of different scales and domains and fuse them, and add the weights with the output of the decoder stage to obtain the intermediate enhanced image I PE ; Step 3: Use the illumination feature F light As a guide for fine-tuning the illumination, the mid-level enhanced image I PE Fine-tune the lighting to obtain the target exposure image.

2. The method for enhancing low-light images using a multi-scale Transformer-Unet wavelet transform according to claim 1, characterized in that: In step 1, given a low-light image, a convolutional neural network is used to enhance it and extract the illumination feature F of the low-light image. light , shallow feature F0, and gradient feature F gradient The specific steps are as follows: 1.

1. Assume I L Represents a low-light image, and uses edge detection operators to extract the low-light image I L The gradient feature F gradient ; 1.

2. Low-light image I L and mean image I mean Cascade, use convolutional neural network for enhancement processing, and get the primary enhanced image I PL ; 1.

3. Extracting primary enhanced image I from the middle layer of convolutional neural network PL The illumination feature F light ; Among them, I L ∈I H×W×3 , H is the height of image I, W is the width of image I, and C is the number of channels of image I; And I mean ∈I H×W×1 , for I L The average of three channels is obtained; 1.

4. Use 3×3 convolutional layer to extract primary enhanced image I PL The shallow feature F0 of .

3. The method for enhancing low-light images using a multi-scale Transformer-Unet wavelet transform according to claim 1, characterized in that: In step 2.1, the Transformer block is used to learn the features in the RGB domain for the shallow feature F0 to obtain the intermediate feature F1. The specific formula is: F′=LayerNorm(F0) F A =WSA(F′)+AbSA(F′) F′0=F0+F A F″=LayerNorm(F′0) F1=DGFN(F″)+F0′ Among them, LayerNorm(·) is the layer normalization processing, WSA(·) is the window attention, AbSA(·) is axial attention, DGFN(·) is a feed-forward network with a double gating mechanism.

4. The method for enhancing low-light images using a multi-scale Transformer-Unet wavelet transform according to claim 1, characterized in that: In step 2.2, the intermediate layer feature F1 is decomposed using Haar wavelet and converted to the wavelet domain, and the gradient feature F gradient As a guide and combined with the gradient feature F gradient Fusion, using stacked multiple Transformer blocks for feature learning, to obtain the intermediate layer feature F2, the specific formula is: F LL ,F LH ,F HL ,F HH =DWT(F1) F1′=[F LL ,F LH ,F HL ,F HH ] F1″=DOF(F1′,F gradient ) F2=WAT(F1″) Among them, DWT(·) is wavelet transform, DOF(·) is the feature fusion using deep over-parameter convolution. WAT(·) is the calculation process in step 2.1; F LL is the low frequency component, F LH is the high frequency component in the horizontal direction, F HL is the high frequency component in the vertical direction, F HH is the high frequency component in the diagonal direction.

5. The method for enhancing low-light images using multi-scale Transformer-Unet based on wavelet transform according to claim 1, characterized in that: In step 2.3, convolution is used to perform 2-fold downsampling and the gradient feature F is used gradient As a guide, it is fused with the downsampling result, and multiple stacked Transformer blocks are used for feature learning to obtain the intermediate layer feature F3. The specific formula is: F2′=Downsampling(F2) F2″=DOF(F2′,F gradient ) F3=WAT(F2″) Among them, Downsampling(·) is the convolution downsampling by 2 times.

6. The method for enhancing low-light images using multi-scale Transformer-Unet based on wavelet transform according to claim 1, characterized in that: In step 2.4, convolution is used to perform 2-fold downsampling and the gradient feature F is used gradient As a guide, it is fused with the downsampling result, and multiple cascaded Transformer blocks are used for feature learning to obtain the intermediate layer feature F4. The specific formula is: F3′=Downsampling(F3) <h2 style=";text-align:left;direction:ltr">F3″=DOF(F3′,F<h2 style=";text-align:left;direction:ltr"> gradient <h2 style=";text-align:left;direction:ltr"> ) F4 = WAT(F3″).

7. The method for enhancing low-light images using multi-scale Transformer-Unet based on wavelet transform according to claim 1, characterized in that: In step 3, the illumination feature F is used light As a guide for fine-tuning the illumination, the mid-level enhanced image I PE Fine-tune the lighting to obtain the target exposure image. The specific process is as follows: The illumination feature F light and intermediate enhanced image I PE The target exposure image can be obtained by inputting them into the illumination adjustment module LAM to perform illumination adjustment.

Images