A multi-scale low-light image enhancement method based on full-resolution semantic guidance

Abstract

The present invention discloses a multi-scale low-light image enhancement method based on full-resolution semantic guidance. The present invention adopts RB module, LFB module and SFB module to form MSRM module, and connects multiple MSRM modules in series to design MSRB module, which can extract shallow feature F0 and semantic feature F from low-light image. seg Processing is performed to enhance the restoration effect of low-light images, retain the necessary detail information, and introduce semantic features F seg As a guide, the loss of semantic information in the enhancement process is avoided, which is more conducive to the subsequent high-level visual tasks. Then, the favorable information of different layers is fused through the MF module, so that the exposure effect of the low-light image is further enhanced. Finally, the features are mapped to the image through 3×3 convolution to obtain the enhanced normal exposure image with good recovery effect.

Description

Technical Field

[0001] The present invention specifically relates to a multi-scale low-light image enhancement method based on full-resolution semantic guidance. Background Art

[0002] In recent years, the rapid development of visual sensors has also led to the popularization of applications in fields such as intelligent driving and security. A common problem in night driving and night monitoring is that due to dark light or uneven lighting, visual judgment errors or detection errors occur, which leads to frequent accidents in night driving and greatly increased security difficulties. Therefore, it is necessary to enhance the exposure of low-light images. In recent years, some scholars have introduced deep learning algorithms and constructed neural networks to enhance low-light images.

[0003] However, when traditional neural networks are used to enhance the exposure of low-light images, the detail information and semantic information of the low-light images are difficult to retain, resulting in poor restoration of the low-light images. In addition, the lack of semantic information is not conducive to the subsequent high-level visual tasks and affects the exposure effect.

[0004] Therefore, it is necessary to invent a multi-scale low-light image enhancement method based on full-resolution semantic guidance to solve the above problems. Summary of the invention

[0005] The purpose of the present invention is to provide a multi-scale low-light image enhancement method based on full-resolution semantic guidance. The present invention adopts RB module, LFB module and SFB module to form MSRM module, and connects multiple MSRM modules in series to design MSRB module, which can enhance the shallow feature F0 and semantic feature F extracted from low-light image. seg Processing is performed to enhance the restoration effect of low-light images, retain the necessary detail information, and introduce semantic features F seg As a guide, the loss of semantic information in the enhancement process is avoided, which is more conducive to the subsequent high-level visual tasks and makes the exposure recovery effect of low-light images good, so as to solve the above-mentioned shortcomings in the technology.

[0006] In order to achieve the above object, the present invention provides the following technical solution: a multi-scale low-light image enhancement method based on full-resolution semantic guidance, comprising the following steps:

[0007] Step 1: Given a low-light image, use 3×3 convolution to extract the shallow feature F0 of the low-light image, and input the low-light image into the semantic segmentation model to obtain the semantic feature F of the low-light image. seg As prior information;

[0008] Step 2: Combine the shallow feature F0 and the semantic feature F segInput to the stacked multi-scale residual MSRB module for processing to obtain F layer1 、F layer2 、F layer3 、F layer4 ;

[0009] Step 3: F layer1 , F layer2 , F layer3 , F layer4 Input it into the multi-layer fusion MF module to extract the beneficial information of different layers and fuse them to get F PE ;

[0010] Step 4: F PE The features are mapped to an image through a 3×3 convolution, that is, an enhanced normal exposure image is obtained.

[0011] In the aforementioned multi-scale low-light image enhancement method based on full-resolution semantic guidance, in step 2, the MSRM module is assumed to be the basic unit of the multi-scale residual MSRB module, and the MSRB module is composed of multiple stacked MSRM modules connected in series, and the MSRM module includes an RB module, an LFB module, and an SFB module.

[0012] In the aforementioned multi-scale low-light image enhancement method based on full-resolution semantic guidance, in step 2, the first MSRM module is used as the benchmark to obtain F layer1 The specific steps are as follows:

[0013] 2.1. Input the shallow feature F0 into the RB module for global feature learning to obtain the intermediate feature F 0，1 , downsample the shallow feature F0 by two times to get F1, input F1 into the RB module for global feature learning, and get the intermediate feature F 1，1 , downsample feature F1 by two times to get F2, input F2 into the RB module for global feature learning, and get the intermediate layer feature F 2，1 ;

[0014] 2.2. F 1，1 and F 2，1 Input into the optional local feature fusion LFB module to fuse the local features to obtain F 1，2 , F 1，2 and semantic features F seg Input into the semantic fusion SFB module to fuse the semantic features and obtain F seg1 ;

[0015] 2.3、F 0，1 and F 1，2 Input into the optional local feature fusion LFB module to fuse the local features to obtain F0，2 , F 0，2 and F seg Input into the semantic fusion SFB module to fuse the semantic features and obtain F seg0 ;

[0016] 2.4、F 2，1 Input into the RB module to get F 2，2 , F seg1 Input into the RB module to get F 1，3 , F seg0 Input into the RB module to get F 0，3 ;

[0017] 2.5、F 2，2 and F 1，3 Input to the optional local feature fusion LFB module to fuse local features to obtain F 2，3 , F 2，3 and F 0，3 The input is sent to the optional local feature fusion LFB module for local feature fusion, and features are extracted through 3×3 convolution to obtain F layer1 .

[0018] In the aforementioned multi-scale low-light image enhancement method based on full-resolution semantic guidance, in step 2, F layer2 、F layer3 、F layer4 The specific steps are as follows:

[0019] 2.6. Taking the second MSRM module as the benchmark, layer1 and semantic features F seg Input into the MSRB module and repeat steps 2.1-2.5 to obtain F layer2 ;

[0020] 2.7. Taking the third MSRM module as the benchmark, layer2 and semantic features F seg Input into the MSRB module and repeat steps 2.1-2.5 to obtain F layer3 ;

[0021] 2.8. Taking the fourth MSRM module as the benchmark, layer3 and semantic features F seg Input into the MSRB module and repeat steps 2.1-2.5 to obtain F layer4 .

[0022] In the aforementioned multi-scale low-light image enhancement method based on full-resolution semantic guidance, in step 2.1, the intermediate layer feature F is obtained 0，1 , intermediate layer features F 1，1, and the intermediate layer features F 2，1 The specific formula is as follows:

[0023] F1=Downsampling(F0)

[0024] F2=Downsampling(F1)

[0025] F 0，1 =RB(F0)

[0026] F 1，1 =RB(F1)

[0027] F 2，1 =RB(F2)

[0028] Among them, Downsampling(·) is double downsampling,

[0029] RB(·) is the calculation process of the RB module.

[0030] In the aforementioned multi-scale low-light image enhancement method based on full-resolution semantic guidance, in step 2.2, we obtain F seg1 The specific formula is as follows:

[0031] F 1，2 =LFB(F 1，1 ,Upsampling(F 2，1 ))

[0032] F seg1 =SFB(F 1，2 , F seg )

[0033] Among them, Upsampling(·) means double upsampling,

[0034] LFB(·) represents the optional local feature fusion module calculation process,

[0035] SFB(·) represents the computation process of the semantic fusion module.

[0036] In the aforementioned multi-scale low-light image enhancement method based on full-resolution semantic guidance, in step 2.3, we obtain F seg0 The specific formula is as follows:

[0037] F 0，2 =LFB(F 0，1 ,Upsampling(F 1，2 ))

[0038] F seg0 =SFB(F 0，2 , F seg ).

[0039] In the aforementioned multi-scale low-light image enhancement method based on full-resolution semantic guidance, in step 2.4, we obtain F 0，3 The specific formula is as follows:

[0040] F 2，2 =RB(F 2，1 )

[0041] F 1，3 =RB(F seg1 )

[0042] F 0，3 =RB(F seg0 ).

[0043] In the aforementioned multi-scale low-light image enhancement method based on full-resolution semantic guidance, in step 3, F layer1 , F layer2 , F layer3 , F layer4 Input it into the multi-layer fusion module, extract the beneficial information of different layers and fuse them to get F PE The specific formula is as follows:

[0044] F PE =MF(F layer1 , F layer2 , F layer3 , F layer4 )

[0045] Among them, MF(·) represents the multi-layer fusion module calculation process.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] The present invention adopts RB module, LFB module and SFB module to form MSRM module, and connects multiple MSRM modules in series to design MSRB module, which can extract shallow feature F0 and semantic feature F from low-light image. seg Processing is performed to enhance the restoration effect of low-light images, retain the necessary detail information, and introduce semantic features F segAs a guide, the loss of semantic information in the enhancement process is avoided, which is more conducive to the subsequent high-level visual tasks. Subsequently, the favorable information of different layers is fused through the MF module, and more favorable local features can be selected from the multi-layer features, so that the exposure effect of the low-light image is further enhanced. In this process, the MSRB module can obtain local weights to improve local details. Finally, the features are mapped to images through 3×3 convolution to obtain the enhanced normal exposure image. When processing low-light images, this method effectively retains the detail information and semantic information, which is conducive to the subsequent high-level visual tasks, and the exposure recovery effect of low-light images is good. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0049] Figure 1 is a flow chart of the present invention;

[0050] Figure 2 is a low-light image given in an embodiment of the present invention;

[0051] Figure 3 is the actual exposure image for a given low-light image;

[0052] Figure 4 This is the target exposure image obtained by the present invention. DETAILED DESCRIPTION

[0053] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings.

[0054] The present invention provides Figure 1-4 A multi-scale low-light image enhancement method based on full-resolution semantic guidance is shown, comprising the following steps:

[0055] Step 1: Given a low-light image, use 3×3 convolution to extract the shallow feature F0 of the low-light image, and input the low-light image into the semantic segmentation model to obtain the semantic feature F of the low-light image. seg As prior information;

[0056] In this step 1, the semantic features F of the low-light image are obtained seg As prior information, it avoids the loss of semantic information in the enhancement process, facilitates the learning of features in subsequent steps, and is beneficial to the subsequent high-level visual tasks.

[0057] Step 2: Combine the shallow feature F0 and the semantic feature F seg Input to the stacked multi-scale residual MSRB module for processing to obtain F layer1 、F layer2 、F layer3 、F layer4 :

[0058] The MSRM module is assumed to be the basic unit of the multi-scale residual MSRB module. The MSRB module is composed of multiple stacked MSRM modules connected in series. The MSRM module includes an RB module, an LFB module, and an SFB module.

[0059] Among them, taking the first MSRM module as the benchmark, we get F layer1 The specific steps are as follows:

[0060] 2.1. Input the shallow feature F0 into the RB module for global feature learning to obtain the intermediate feature F 0，1 ,

[0061] Among them, the RB module is a module for global feature learning;

[0062] The shallow feature F0 is downsampled twice to obtain F1, and F1 is input into the RB module for global feature learning to obtain the intermediate feature F 1，1 ,

[0063] The feature F1 is downsampled twice to obtain F2, and F2 is input into the RB module for global feature learning to obtain the intermediate layer feature F 2，1 , the specific formula is as follows:

[0064] F1=Downsampling(F0)

[0065] F2=Downsampling(F1)

[0066] F 0，1 =RB(F0)

[0067] F 1，1 =RB(F1)

[0068] F 2，1 =RB(F2)

[0069] Among them, Downsampling(·) is double downsampling,

[0070] RB(·) is the calculation process of the RB module;

[0071] 2.2. F 1，1 and F 2，1Input into the optional local feature fusion LFB module to fuse the local features to obtain F 1，2 , F 1，2 and semantic features F seg Input into the semantic fusion SFB module to fuse the semantic features and obtain F seg1 , the specific formula is as follows:

[0072] F 1，2 =LFB(F 1，1 ,Upsampling(F 2，1 ))

[0073] F seg1 =SFB(F 1，2 , F seg )

[0074] Among them, Upsampling(·) means double upsampling,

[0075] LFB(·) represents the optional local feature fusion module calculation process,

[0076] SFB(·) represents the computation process of the semantic fusion module.

[0077] Among them, since the features of low-light images may present different effects at different scales, through LFB module learning, more favorable features can be selected, which is helpful for selecting local features of low-light images;

[0078] Since the features are downsampled and then upsampled, some semantic information will inevitably be lost. The SFB module can combine the source features and the acquired semantic features F seg Fusion, to preserve semantic information;

[0079] 2.3、F 0，1 and F 1，2 Input into the optional local feature fusion LFB module to fuse the local features to obtain F 0，2 , F 0，2 and F seg Input into the semantic fusion SFB module to fuse the semantic features and obtain F seg0 , the specific formula is as follows:

[0080] F 0，2 =LFB(F 0，1 ,Upsampling(F 1，2 ))

[0081] F seg0 =SFB(F 0，2 , F seg )

[0082] 2.4、F 2，1 Input into the RB module to get F 2，2 , F seg1 Input into the RB module to get F 1，3 , F seg0 Input into the RB module to get F 0，3 , the specific formula is as follows:

[0083] F 2，2 =RB(F 2，1 )

[0084] F 1，3 =RB(F seg1 )

[0085] F 0，3 =RB(F seg0 );

[0086] 2.5、F 2，2 and F 1，3 Input to the optional local feature fusion module LFB to fuse local features to obtain F 2，3 , F 2，3 and F 0，3 The input is sent to the optional local feature fusion module LFB for local feature fusion, and features are extracted through 3×3 convolution to obtain F layer1 , the specific formula is as follows:

[0087] F 2，3 =LFB(F 1，3 ,Upsampling(F 2，2 ))

[0088] F pre =LFB(F 0，3 ,Upsampling(F 2，3 ))

[0089] F layer1 =Conv 33 (F pre )

[0090] Among them, Conv 33 (·) indicates 3×3 convolution calculation;

[0091] 2.6. Taking the second MSRM module as the benchmark, layer1 and semantic features F seg Input into the MSRB module and repeat steps 2.1-2.5 to obtain F layer2 ;

[0092] 2.7. Taking the third MSRM module as the benchmark,layer2 and semantic features F seg Input into the MSRB module and repeat steps 2.1-2.5 to obtain F layer3 ;

[0093] 2.8. Taking the fourth MSRM module as the benchmark, layer3 and semantic features F seg Input into the MSRB module and repeat steps 2.1-2.5 to obtain F layer4 ;

[0094] In this step 2, by repeating the calculation three times, we finally get F layer1 、F layer2 、F layer3 、F layer4 In this way, the best balance can be achieved between enhancing the exposure effect of low-light images and the computational overhead. Multi-scale feature learning is performed internally to enhance the restoration effect of low-light images while retaining semantic information to the maximum extent and restoring the details of low-light images, thereby enhancing the exposure effect of low-light images.

[0095] Step 3: F layer1 , F layer2 , F layer3 , F layer4 Input it into the multi-layer fusion MF module to extract the beneficial information of different layers and fuse them to get F PE , the specific formula is as follows:

[0096] F PE =MF(F layer1 , F layer2 , F layer3 , F layer4 )

[0097] Among them, MF(·) represents the multi-layer fusion module calculation process;

[0098] In this step 3, since the effect of feature learning is different at different network depths, through the reference of the MF module, each layer can learn information that is beneficial to the exposure effect of the low-light image, that is, it can select more locally favorable features from multiple layers of features, so that the exposure effect of the low-light image is further enhanced.

[0099] Step 4: F PE The features are mapped to an image through a 3×3 convolution, that is, an enhanced normal exposure image is obtained.

[0100] Verification experiment

[0101] like Figure 2 As shown, it is a low-light image. After being processed by the present invention, a target exposure image is obtained, that is, Figure 4 As shown, at this point, it can be seen visually that Figure 4 A good lighting enhancement effect is achieved, which not only restores the dark information, but also enhances the texture and other details. Figure 3 , and there is no obvious difference, the present invention can be used to obtain reliable target exposure images;

[0102] In summary, the present invention adopts RB module, LFB module and SFB module to form MSRM module, and connects multiple MSRM modules in series to design MSRB module, which can extract shallow feature F0 and semantic feature F from low-light image. seg Processing is performed to enhance the restoration effect of low-light images, retain the necessary detail information, and introduce semantic features F seg As a guide, the loss of semantic information in the enhancement process is avoided, which is more conducive to the subsequent high-level visual tasks. Subsequently, the favorable information of different layers is fused through the MF module, and more favorable local features can be selected from the multi-layer features, so that the exposure effect of the low-light image is further enhanced. In this process, the MSRB module can obtain local weights to improve local details. Finally, the features are mapped to images through 3×3 convolution to obtain the enhanced normal exposure image. When processing low-light images, this method effectively retains the detail information and semantic information, which is conducive to the subsequent high-level visual tasks, and the exposure recovery effect of low-light images is good.

[0103] The above description is only by way of illustration of certain exemplary embodiments of the present invention. It is undoubted that those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the above drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

Claims

1. A multi-scale low-light image enhancement method based on full-resolution semantic guidance, characterized by: The following steps are involved: Step 1: Given a low-light image, use 3×3 convolution to extract the shallow feature F0 of the low-light image, and input the low-light image into the semantic segmentation model to obtain the semantic feature F of the low-light image. seg As prior information; Step 2: Combine the shallow feature F0 and the semantic feature F seg Input to the stacked multi-scale residual MSRB module for processing to obtain F layer1 、F layer2 、F layer3 、F layer4 ; Step 3: F layer1 , F layer2 , F layer3 , F layer4 Input it into the multi-layer fusion MF module to extract the beneficial information of different layers and fuse them to get F PE ; Step 4: F PE The feature is mapped to an image through a 3×3 convolution, that is, the enhanced normal exposure image is obtained Wherein, in step 2, the MSRM module is assumed to be a basic unit of the multi-scale residual MSRB module, the MSRB module is composed of a plurality of stacked MSRM modules connected in series, and the MSRM module includes an RB module, an LFB module, and an SFB module; Taking the first MSRM module as the benchmark, we get F layer1 The specific steps are as follows: 2.

1. Input the shallow feature F0 into the RB module for global feature learning to obtain the intermediate feature F 0,1 , downsample the shallow feature F0 by two times to get F1, input F1 into the RB module for global feature learning, and get the intermediate feature F 1,1 , downsample feature F1 by two times to get F2, input F2 into the RB module for global feature learning, and get the intermediate layer feature F 2,1 ; 2.

2. F 1,1 and F 2,1 Input into the optional local feature fusion LFB module to fuse the local features to obtain F 1,2 , F 1,2 and semantic features F seg Input into the semantic fusion SFB module to fuse the semantic features and obtain F seg1 ; 2.3、F 0,1 and F 1,2 Input into the optional local feature fusion LFB module to fuse the local features to obtain F 0,2 , F 0,2 and F seg Input into the semantic fusion SFB module to fuse the semantic features and obtain F seg0 ; 2.4、F 2,1 Input into the RB module to get F 2,2 , F seg1 Input into the RB module to get F 1,3 , F seg0 Input into the RB module to get F 0,3 ; 2.5、F 2,2 and F 1,3 Input to the optional local feature fusion LFB module to fuse local features to obtain F 2,3 , F 2,3 and F 0,3 The input is sent to the optional local feature fusion LFB module for local feature fusion, and features are extracted through 3×3 convolution to obtain F layer1 ; Get F layer2 、F layer3 、F layer4 The specific steps are as follows: 2.

6. Taking the second MSRM module as the benchmark, layer1 and semantic features F seg Input into the MSRB module and repeat steps 2.1-2.5 to obtain F layer2 ; 2.

7. Taking the third MSRM module as the benchmark, layer2 and semantic features F seg Input into the MSRB module and repeat steps 2.1-2.5 to obtain F layer3 ; 2.

8. Taking the fourth MSRM module as the benchmark, layex3 and semantic features F seg Input into the MSRB module and repeat steps 2.1-2.5 to obtain F layer4 .

2. The multi-scale low-light image enhancement method based on full-resolution semantic guidance according to claim 1, characterized in that: In step 2.1, the intermediate layer feature F is obtained 0,1 , intermediate layer features F 1,1 , and the intermediate layer features F 2,1 The specific formula is as follows: F1=Downsampling(F0) F2=Downsampling(F1) F 0,1 =RB(F0) F 1,1 =RB(F1) F 2,1 =RB(F2) Among them, Downsampling(·) is double downsampling, RB(·) is the calculation process of the RB module.

3. The multi-scale low-light image enhancement method based on full-resolution semantic guidance according to claim 1, characterized in that: In step 2.2, we get F seg1 The specific formula is as follows: F 1,2 =LFB(F 1,1 ,Upsampling(F 2,1 )) F seg1 =SFB(F 1,2 ,F seg ) Among them, Upsampling(·) means double upsampling, LFB(·) represents the optional local feature fusion module calculation process, SFB(·) represents the computation process of the semantic fusion module.

4. The multi-scale low-light image enhancement method based on full-resolution semantic guidance according to claim 1, characterized in that: In step 2.3, we get F seg0 The specific formula is as follows: F 0,2 =LFB(F 0,1 ,Upsampling(F 1,2 )) F seg0 =SFB(F 0,2 ,F seg )。 5. The multi-scale low-light image enhancement method based on full-resolution semantic guidance according to claim 1, characterized in that: In step 2.4, we get F 0,3 The specific formula is as follows: F 2,2 =RB(F 2,1 ) F 1,3 =RB(F seg1 ) F 0,3 =RB(F seg0 )。 6. The multi-scale low-light image enhancement method based on full-resolution semantic guidance according to claim 1, characterized in that: In step 2.5, F 2,2 and F 1,3 Input to the optional local feature fusion LFB module to fuse local features to obtain F 2,3 , F 2,3 and F 0,3 The input is sent to the optional local feature fusion LFB module for local feature fusion, and features are extracted through 3×3 convolution to obtain F layer1 The specific formula is as follows: F 2,3 =LFB(F 1,3 ,Upsampling(F 2,2 )) F pre =LFB(F 0,3 ,Upsampling(F 2,3 )) F layer1 =Conv 33 (F pre ) Among them, Conv 33 (·) indicates 3×3 convolution calculation.

7. The multi-scale low-light image enhancement method based on full-resolution semantic guidance according to claim 1, characterized in that: In step 3, F layer1 , F layer2 , F layer3 , F layer4 Input it into the multi-layer fusion MF module to extract the beneficial information of different layers and fuse them to get F PE The specific formula is as follows: FAVORITE PE =MF(F layer1 ,FAVORITE layer2 ,FAVORITE layer3 ,FAVORITE layer4 ) Among them, MF(·) represents the multi-layer fusion module calculation process.

Images