Low-light image enhancement method based on multi-scale information fusion

Abstract

The invention belongs to the field of low-light image enhancement, and provides a low-light image enhancement method based on multi-scale information fusion. In the prior art, data characteristics of a low-light image are not fully considered so that a recovered image is distorted, or an image quality enhancement task in a more complex scene is not coped, and a lightweight and efficient multi-scale information extraction method is lacked. A brand-new multi-scale residual module is creatively provided, features are extracted and fused on different scales, feature expression is enriched, the perception and generalization ability of the model is enhanced, a recursive residual module is constructed based on the multi-scale residual module, and finally a low-light image enhancement network is constructed. Compared with an existing low light enhancement network, the method has the advantages that the parameter quantity and the calculation quantity are greatly reduced, and meanwhile, excellent performance is kept; in conclusion, according to the low-light image enhancement method based on multi-scale information fusion, the low-light image enhancement effect can be remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the field of low-light image enhancement, and specifically provides a low-light image enhancement method based on multi-scale information fusion. Background Art

[0002] Night photography usually inevitably introduces feature image distortion due to insufficient lighting such as backlight and weak light due to environmental reasons and technical limitations. This will lead to a decrease in the quality of the feature image, thereby affecting the visual effect of the feature image and even having a negative impact on some higher-level feature image processing tasks. Therefore, low-light image enhancement processing is extremely important.

[0003] At present, the low-light image enhancement network has great limitations because it fails to fully consider the characteristics of low-light images such as non-uniform illumination, small pixel values, noise suppression and color constancy; therefore, it is a general trend to fully explore the characteristics of low-light data and consider designing a reasonable end-to-end low-light image enhancement network. Reasonable multi-scale information fusion helps to suppress feature image noise and smooth feature images. The field of low-light image enhancement still lacks effective multi-scale information extraction and aggregation methods, and it is difficult to maintain efficient performance while keeping the model lightweight. Therefore, the present invention designs a multi-scale information fusion module and proposes a low-light image enhancement method based on multi-scale information fusion, which is of great significance. Summary of the invention

[0004] The purpose of the present invention is to provide a low-light image enhancement method based on multi-scale information fusion, so as to effectively improve the low-light enhancement quality of feature images.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] A low-light image enhancement method based on multi-scale information fusion, characterized by comprising the following steps:

[0007] Construct a low-light image enhancement network, including: 4 recursive residual modules connected in sequence; the input of the low-light image enhancement network is the low-light image U low , the output is the enhanced image U en , low light image U low After passing through four recursive residual modules in sequence, the feature image U is obtained fe , feature image U fe With low light images low Add to get the enhanced image U en ;

[0008] The loss function is set, and a progressive training strategy is adopted to complete the training of the low-light image enhancement network to obtain a low-light image enhancement model;

[0009] The low-light image to be processed is input into the trained low-light image enhancement model, and the low-light image enhancement model outputs an enhanced image.

[0010] Furthermore, the recursive residual module includes: two multi-scale residual modules connected in sequence; the input of the recursive residual module is recorded as the feature image U in1 , the output is recorded as feature image U out1 , feature image U in1 After passing through two multi-scale residual modules in sequence, the feature image U is obtained mid1 , feature image U mid1 That is the output of the second multi-scale residual module, the feature image U mid1 With the feature image U in1 Add to get the feature image U out1 .

[0011] Furthermore, the multi-scale residual module includes: 3 information extraction modules and a multi-scale aggregation module; the input of the multi-scale residual module is recorded as the feature image U in10 , the output is recorded as feature image U out10 , feature image U in2 After two downsamplings, the feature image U is obtained after the first downsampling. in11 , after the second downsampling, we get the feature image U in12 , feature image U in10 , feature image U in11 With the feature image U in12 Constitute three-scale input; feature image U in10 , feature image U in11 With the feature image U in12 They are input into three information extraction modules respectively, and the three information extraction modules output feature images U mid10 , feature image U mid11 With the feature image U mid12 , and are jointly input into the multi-scale aggregation module, which outputs the feature image U out10 .

[0012] Furthermore, the information extraction module includes: a shallow information extraction unit, a context unit, and a conversion unit, wherein the structure of the shallow information extraction unit is CONV 3×3, and the structure of the context unit is: The structure of the conversion unit is: CONV 1×1+CONV 1×1; the input of the information extraction module is recorded as the feature image U in100 , the output is recorded as feature image U out100 , feature image U in100 Input to the shallow information extraction unit, and the shallow information extraction unit outputs the feature image Umid100 , feature image U mid100 After the context unit, the feature image U is obtained mid101 , feature image U mid101 After the transformation unit, the feature image U is obtained mid102 , feature image U in100 , feature image U mid100 With the feature image U mid102 Add to get the feature image U out100 .

[0013] Furthermore, the multi-scale aggregation module includes: 3 global pooling units and MLP units; the structure of the global pooling unit is: ADAPTIVEAVGPOOL 1×1, and the structure of the MLP unit is CONV 1×1+CONV 1×1+CONV 1×1; the feature image U mid10 , feature image U mid11 With the feature image U mid12 Input three global pooling units respectively to obtain the feature images U mid103 , feature image U mid104 With the feature image U mid105 , feature image U mid103 , feature image U mid104 With the feature image U mid105 After splicing, it is input into the MLP unit, and the MLP unit outputs the feature image U mid106 , feature image U mid106 After SPLIT decomposition, we get coefficients m1, m2, and m3. According to the coefficients m1, m2, and m3, we can calculate the characteristic image U in10 , U in11 , U in12 Aggregate and get the feature image U out10 :U out10 =U in10 ×m1+U in11 ×m2+U in12 ×m3.

[0014] Furthermore, the loss function adopts L1 loss, which is specifically expressed as:

[0015]

[0016] in, represents the enhanced image output by the low-light image enhancement network, and y is the label image.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] The present invention proposes a low-light image enhancement method based on multi-scale information fusion, and realizes efficient low-light image enhancement by constructing a low-light image enhancement network; in view of the problem that the prior art fails to fully consider the characteristics of low-light image data, resulting in a certain degree of distortion in the restored image, or fails to cope with the image quality enhancement task in more complex scenes, and lacks a lightweight and efficient multi-scale information extraction method for low-light image enhancement tasks, the present invention creatively proposes a new multi-scale residual module, which can extract and fuse features at different scales, enrich feature expression, enhance the perception and generalization ability of the model, and further constructs a recursive residual module based on the multi-scale residual module, and finally constructs a low-light image enhancement network; compared with other existing low-light enhancement networks, the present invention greatly reduces the number of parameters and calculations while maintaining excellent performance; in summary, the low-light image enhancement method based on multi-scale information fusion proposed in the present invention can significantly improve the low-light image enhancement effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Schematic diagram of the structure of the low-light image enhancement network in the present invention.

[0020] Figure 2 Schematic diagram of the structure of the recursive residual module in the present invention.

[0021] Figure 3 Schematic diagram of the structure of the multi-scale residual module in the present invention.

[0022] Figure 4 It is a structural schematic diagram of the information extraction module in the present invention.

[0023] Figure 5 Schematic diagram of the structure of the multi-scale aggregation module in the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and beneficial effects of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Obviously, the embodiments described below are only part of the embodiments of the present invention, rather than all of the embodiments.

[0025] This embodiment provides a low-light image enhancement method based on multi-scale information fusion, and the specific steps are as follows:

[0026] Step 1. Build a low-light image enhancement network, such as Figure 1 As shown, it specifically includes: 4 recursive residual modules connected in sequence; the input of the low-light image enhancement network is the low-light image U low , the output is the enhanced image U en , low light image U low After passing through four recursive residual modules in sequence, the feature image U is obtainedfe , feature image U fe That is the output of the fourth recursive residual module, the feature image U fe With low light images low Add to get the enhanced image U en ;

[0027] The recursive residual module is as follows Figure 2 As shown, it specifically includes: two multi-scale residual modules connected in sequence; the input of the recursive residual module is recorded as the feature image U in1 , the output is recorded as feature image U out1 , feature image U in1 After passing through two multi-scale residual modules in sequence, the feature image U is obtained mid1 , feature image U mid1 That is the output of the second multi-scale residual module, the feature image U mid1 With the feature image U in1 Add to get the feature image U out1 ;

[0028] The multi-scale residual module is Figure 3 As shown in Fig. 1, it specifically includes: 3 information extraction modules and a multi-scale aggregation module; the input of the multi-scale residual module is recorded as the feature image U in10 , the output is recorded as feature image U out10 , feature image U in2 After two downsamplings, the feature image U is obtained after the first downsampling. in11 , after the second downsampling, we get the feature image U in12 , feature image U in10 , feature image U in11 With the feature image U in12 Constitute three-scale input; feature image U in10 , feature image U in11 With the feature image U in12 They are input into three information extraction modules respectively, and the three information extraction modules output feature images U mid10 , feature image U mid11 With the feature image U mid12 , and are jointly input into the multi-scale aggregation module, which outputs the feature image U out10 ;

[0029] The information extraction module is as follows Figure 4 As shown, it specifically includes: a shallow information extraction unit, a context unit, and a conversion unit, wherein the structure of the shallow information extraction unit is CONV 3×3, and the structure of the context unit is: The structure of the conversion unit is: CONV 1×1+CONV 1×1; the input of the information extraction module is recorded as the feature image U in100 , the output is recorded as feature image U out100 , feature image U in100 Input to the shallow information extraction unit, and the shallow information extraction unit outputs the feature image U mid100 , feature image U mid100 After the context unit, the feature image U is obtained mid101 , feature image U mid101 After the transformation unit, the feature image U is obtained mid102 , feature image U in100 , feature image U mid100 With the feature image U mid102 Add to get the feature image U out100 ;

[0030] The multi-scale aggregation module is as follows Figure 5 As shown in FIG. 1 , it specifically includes: 3 global pooling units and MLP units; wherein, the structure of the global pooling unit is: ADAPTIVEAVGPOOL 1×1, and the structure of the MLP unit is CONV 1×1+CONV 1×1+CONV 1×1; the feature image U mid10 , feature image U mid11 With the feature image U mid12 Input three global pooling units respectively to obtain the feature images U mid103 , feature image U mid104 With the feature image U mid105 , feature image U mid103 , feature image U mid104 With the feature image U mid105 After splicing, it is input into the MLP unit, and the MLP unit outputs the feature image U mid106 , feature image U mid106 After SPLIT decomposition, we get coefficients m1, m2, and m3. According to the coefficients m1, m2, and m3, we can calculate the characteristic image U in10 , U in11 , U in12 Aggregate and get the feature image U out10 :U out10 =U in10 ×m1+U in11 ×m2+U in12 ×m3;

[0031] It should be noted that: in the present invention, CONV A×A represents a convolution layer with a convolution kernel of A×A, RESHAPE represents an image reshaping operation, SOFTMAX represents a Softmax activation operation, ADAPTIVEAVGPOOL B×B represents a global adaptive pooling operation with an output feature map size of B×B, represents the multiplication operation, Indicates addition operation, ⊙ indicates weighted operation; in the description of network structure, + indicates sequential connection;

[0032] Step 2. Set the loss function and use a progressive training strategy to complete the training of the low-light image enhancement network to obtain a low-light image enhancement model;

[0033] For the LOLv1 dataset, 485 pairs of low-light-normal-light images in the dataset are used as training images, and low-light images are used as inputs and normal-light images are used as labels to form training samples, thereby constructing a training set; in addition, for the LOLv2_real and LOLv2_syn datasets, the training sample pairs are 689 and 900 respectively, and the rest of the settings are the same as LOLv1; in addition, a progressive training strategy is adopted, that is, according to the increase in the number of training iterations, the batch size is gradually adjusted from 8 to 1, and the random cropping size is gradually adjusted from 128×128 to 384×384; in this embodiment, the Adam optimizer is used to train the network;

[0034] The loss function adopts L1 loss, which is specifically expressed as:

[0035]

[0036] in, represents the enhanced image output by the low-light image enhancement network, and y is the label image;

[0037] Step 3. Input the low-light image to be processed into the trained low-light image enhancement model, and the low-light image enhancement model outputs an enhanced image.

[0038] Based on the above technical solution, this embodiment is tested on the remaining 15, 100 and 100 pairs of feature image sets of LOLv1, LOLv2_real and LOLv2_syn, respectively, and compared with the existing representative low-light enhancement methods. The comparison results are shown in Tables 1, 2 and 3.

[0039] Table 1

[0040]

[0041] Table 2

[0042]

[0043] Table 3

[0044]

[0045] As can be seen from the table, the low-light enhancement model adopted in this embodiment shows excellent performance on various data sets while maintaining a small number of parameters and calculations, and has achieved the best performance on the LOLv2_syn test set, thereby proving the effectiveness and superiority of the present invention.

[0046] The above description is only a specific implementation mode of the present invention. Any feature disclosed in this specification, unless otherwise stated, can be replaced by other alternative features that are equivalent or have similar purposes; all the disclosed features, or all the steps in the methods or processes, except for mutually exclusive features and / or steps, can be combined in any way.

Claims

1. A low-light image enhancement method based on multi-scale information fusion, characterized in that: The following steps are involved: Construct a low-light image enhancement network, including: 4 recursive residual modules connected in sequence; The input of the low-light image enhancement network is the low-light image U low , the output is the enhanced image U en , low light image U low After passing through four recursive residual modules in sequence, the feature image U is obtained fe , feature image U fe With low light images low Add to get the enhanced image U en ; The loss function is set, and a progressive training strategy is adopted to complete the training of the low-light image enhancement network to obtain a low-light image enhancement model; The low-light image to be processed is input into the trained low-light image enhancement model, and the low-light image enhancement model outputs an enhanced image.

2. The low-light image enhancement method based on multi-scale information fusion according to claim 1, characterized in that: The recursive residual module includes: two multi-scale residual modules connected in sequence; the input of the recursive residual module is recorded as the feature image U in1 , the output is recorded as feature image U out1 , feature image U in1 After passing through two multi-scale residual modules in sequence, the feature image U is obtained mid1 , feature image U mid1 That is the output of the second multi-scale residual module, the feature image U mid1 With the feature image U in1 Add to get the feature image U out1 .

3. The low-light image enhancement method based on multi-scale information fusion according to claim 2, characterized in that: The multi-scale residual module includes: 3 information extraction modules and a multi-scale aggregation module; the input of the multi-scale residual module is recorded as the feature image U in10 , the output is recorded as feature image U out10 , feature image U in2 After two downsamplings, the feature image U is obtained after the first downsampling. in11 , after the second downsampling, we get the feature image U in12 , feature image U in10 , feature image U in11 With the feature image U in12 Constitute three-scale input; feature image U in10 , feature image U in11 With the feature image U in12 They are input into three information extraction modules respectively, and the three information extraction modules output feature images U mid10 , feature image U mid11 With the feature image U mid12 , and are jointly input into the multi-scale aggregation module, which outputs the feature image U out10 .

4. The low-light image enhancement method based on multi-scale information fusion according to claim 3, characterized in that: The information extraction module includes: a shallow information extraction unit, a context unit, and a conversion unit. The structure of the shallow information extraction unit is CONV 3×3, and the structure of the context unit is: The structure of the conversion unit is: CONV 1×1+CONV 1×1; the input of the information extraction module is recorded as the feature image U in100 , the output is recorded as feature image U out100 , feature image U in100 Input to the shallow information extraction unit, and the shallow information extraction unit outputs the feature image U mid100 , feature image U mid100 After the context unit, the feature image U is obtained mid101 , feature image U mid101 After the transformation unit, the feature image U is obtained mid102 , feature image U in100 , feature image U mid100 With the feature image U mid102 Add to get the feature image U out100 .

5. The low-light image enhancement method based on multi-scale information fusion according to claim 3, characterized in that: The multi-scale aggregation module includes: 3 global pooling units and MLP units; the structure of the global pooling unit is: ADAPTIVEAVGPOOL 1×1, and the structure of the MLP unit is CONV 1×1+CONV 1×1+CONV 1×1; the feature image U mid10 , feature image U mid11 With the feature image U mid12 Input three global pooling units respectively to obtain the feature images U mid103 , feature image U mid104 With the feature image U mid105 , feature image U mid103 , feature image U mid104 With the feature image U mid105 After splicing, it is input into the MLP unit, and the MLP unit outputs the feature image U mid106 , feature image U mid106 After SPLIT decomposition, we get coefficients m1, m2, and m3. According to the coefficients m1, m2, and m3, we can calculate the characteristic image U in10 , U in11 , U in12 Aggregate and get the feature image U out10 :U out10 =U in10 ×m1+U in11 ×m2+U in12 ×m3.

6. The low-light image enhancement method based on multi-scale information fusion according to claim 1, characterized in that: The loss function uses L1 loss, which is specifically expressed as: in, represents the enhanced image output by the low-light image enhancement network, and y is the label image.

Images