Night low-illumination image enhancement method and system

Abstract

The invention relates to the field of image processing, in particular to a night low-illumination image enhancement method and system, and the method comprises the steps: obtaining a dark channel image of a low-illumination image shot at night, obtaining the estimated light intensity of each channel according to the dark channel image, generating a minimum channel constraint graph, and calculating the pixel attention of the minimum channel constraint graph, obtaining a light intensity feature map of each channel based on pixel attention and the estimated light intensity of each channel; inputting a low-illumination image into the feature pyramid network, performing convolution operation on the light intensity feature map to obtain a feature map with the same size as the downsampling output of each layer of the feature pyramid network, and fusing the downsampling output of the feature pyramid network with the same size and the convolution output of the light intensity feature map according to channels, and further fusing the fusion result into the input of up-sampling of the feature pyramid network with the same size as the fusion result according to channels, and taking the last layer output of the up-sampling of the feature pyramid network as a night low-illumination image enhancement result.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a method and system for enhancing nighttime low-illuminance images. Background Art

[0002] At night or in low-light environments, due to insufficient light, imaging devices often find it difficult to capture enough photons, resulting in a significant decrease in image quality, increased noise, reduced contrast, and loss of details. In addition, the impact of random noise in the image becomes more significant, forming a large number of noise points. These noise points will cover various parts of the image, seriously interfering with the overall visual effect and detail performance of the image, making it difficult to distinguish the edges and details of objects, and the image appears blurry and dull. In terms of security monitoring, the decline in the quality of low-light images at night may lead to untimely identification of potential threats, affecting public safety; in terms of unmanned driving, low-light images at night may make it impossible for the autonomous driving system to accurately perceive the environment, increasing the risk of traffic accidents; in medical imaging, the lack of details in low-light images may affect the early diagnosis and treatment of diseases.

[0003] Traditional methods are prone to problems such as over-enhancement, loss of details and artifacts when processing low-light image enhancement, and the results are often unsatisfactory. Therefore, a new method that can achieve high-quality image enhancement in various low-light environments is urgently needed to meet the needs of practical applications. Summary of the invention

[0004] In order to solve the above problems, the present invention provides a method for enhancing low-light images at night, which comprises the following steps: Acquire a dark channel image of a low-light image taken at night, obtain an estimated light intensity of each channel according to the dark channel image, generate a minimum channel constraint map, calculate a pixel attention of the minimum channel constraint map, and obtain a light intensity feature map of each channel based on the pixel attention and the estimated light intensity of each channel; The low-illumination image is input into a feature pyramid network, a convolution operation is performed on the light intensity feature map to obtain a feature map of the same size as the downsampled output of each layer of the feature pyramid network, the downsampled output of the feature pyramid network of the same size and the convolution output of the light intensity feature map are fused by channel, the fusion result is further fused by channel to the input sampled on the feature pyramid network of the same size as the fusion result, and the upsampled last layer output of the feature pyramid network is used as the nighttime low-illumination image enhancement result.

[0005] Preferably, the pixel attention of the minimum channel constraint graph is calculated as follows: A bright channel image of the low-light image is obtained, the bright channel image and the minimum channel constraint map are fused into an image of one channel, and pixel attention of the fused image is calculated.

[0006] Preferably, the light intensity feature map of each channel is obtained based on the pixel attention and the estimated light intensity of each channel, specifically: Multiply the estimated light intensity of the channel by each element in the normalized pixel attention, and use the multiplied result as a light intensity feature map.

[0007] Preferably, the light intensity feature map of each channel is obtained based on the pixel attention and the estimated light intensity of each channel, specifically: Calculating the local contrast of each pixel in each channel of the low-light image, and fusing the normalized local contrast map of the channel with the normalized pixel attention to generate a light intensity weight map; The estimated light intensity of the channel is multiplied by each element in the light intensity weight map, and the multiplied result is used as the light intensity feature map.

[0008] Preferably, the normalized local contrast map of the channel and the normalized pixel attention are fused to generate a light intensity weight map, specifically: A weighted sum is performed on the normalized local contrast map of the channel and the normalized value of the same position of the pixel attention, and the weighted sum result is used as the value of the same position in the light intensity weight map.

[0009] In addition, the present invention also provides a nighttime low-illumination image enhancement system, the system comprising the following modules: A light intensity feature generation module, used to obtain a dark channel image of a low-light image taken at night, obtain an estimated light intensity of each channel according to the dark channel image, generate a minimum channel constraint map, calculate a pixel attention of the minimum channel constraint map, and obtain a light intensity feature map of each channel based on the pixel attention and the estimated light intensity of each channel; The image enhancement module is used to input the low-light image into a feature pyramid network, perform a convolution operation on the light intensity feature map to obtain a feature map of the same size as the downsampled output of each layer of the feature pyramid network, fuse the downsampled output of the feature pyramid network of the same size with the convolution output of the light intensity feature map by channel, further fuse the fusion result by channel into the input sampled on the feature pyramid network of the same size as the fusion result, and use the upsampled last layer output of the feature pyramid network as the nighttime low-light image enhancement result.

[0010] Preferably, the pixel attention of the minimum channel constraint graph is calculated as follows: A bright channel image of the low-light image is obtained, the bright channel image and the minimum channel constraint map are fused into an image of one channel, and pixel attention of the fused image is calculated.

[0011] Preferably, the light intensity feature map of each channel is obtained based on the pixel attention and the estimated light intensity of each channel, specifically: Multiply the estimated light intensity of the channel by each element in the normalized pixel attention, and use the multiplied result as a light intensity feature map.

[0012] Preferably, the light intensity feature map of each channel is obtained based on the pixel attention and the estimated light intensity of each channel, specifically: Calculating the local contrast of each pixel in each channel of the low-light image, and fusing the normalized local contrast map of the channel with the normalized pixel attention to generate a light intensity weight map; The estimated light intensity of the channel is multiplied by each element in the light intensity weight map, and the multiplied result is used as the light intensity feature map.

[0013] Preferably, the normalized local contrast map of the channel and the normalized pixel attention are fused to generate a light intensity weight map, specifically: A weighted sum is performed on the normalized local contrast map of the channel and the normalized value of the same position of the pixel attention, and the weighted sum result is used as the value of the same position in the light intensity weight map.

[0014] Finally, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described above is implemented.

[0015] The existing low-light image enhancement easily leads to the problems of loss of image details and uncoordinated overall enhancement. The present invention uses pixel-level features to enhance the low-light image. Specifically, the pixel attention of the minimum channel constraint map is calculated, and the light intensity feature map of each channel is obtained based on the pixel attention and the estimated light intensity of each channel. The light intensity feature map and the feature pyramid are combined to achieve enhancement of the low-light image. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of Embodiment 1; Figure 2 Comparison chart of low illumination image and dark channel image; Figure 3 Comparison diagram of low illumination image and minimum channel constraint map; Figure 4 The figure is an effect diagram of enhancing a low-illuminance image using the present invention. DETAILED DESCRIPTION

[0017] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0018] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0019] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0020] In the first embodiment, if Figure 1 As shown, the present invention provides a method for enhancing low-light images at night, the method comprising the following steps: Step 1, obtaining a dark channel image of a low-light image taken at night, obtaining an estimated light intensity of each channel according to the dark channel image, generating a minimum channel constraint map, calculating a pixel attention of the minimum channel constraint map, and obtaining a light intensity feature map of each channel based on the pixel attention and the estimated light intensity of each channel; The dark channel image is obtained by calculating the low illumination image. The dark channel image can be used to estimate the light intensity. After obtaining the dark channel image, the present invention obtains the estimated light intensity of each channel. Specifically, the calculation process of the dark channel image is to calculate the minimum value of the three RGB channels within a preset range of each pixel point, and use the minimum value as the value of the dark channel image of this pixel point. Figure 2The dark channel image is shown. After the dark channel image is obtained, the light intensity value is estimated. Specifically, the pixels with the highest brightness in the dark channel image are selected, and the estimated light intensity is obtained according to the values ​​of these pixels. In one embodiment, the average value of the RGB values ​​of these pixels is calculated. For example, there are 3 pixels with the highest brightness in the preset ratio, and the average value of the RGB values ​​of each pixel is calculated, and then the average of the average values ​​of the 3 pixels is calculated.

[0021] After obtaining the estimated light intensity, the minimum channel constraint map of the low-light image is calculated. Specifically, the maximum value of each pixel in the low-light image in all channels is calculated, and 1-I max / A is the value of the same position of the pixel in the minimum channel constraint map; where I max Indicates the maximum value of the pixel in all channels, and A indicates the estimated light intensity. For example, in a low-light image, the RGB channel values ​​of the pixel at (2,5) are 10, 20, and 5, and the maximum value of this pixel in all channels is 20. If the estimated light intensity of the G channel is 40, the value of (2,5) in the minimum channel constraint map is 0.5.

[0022] The minimum channel constraint map contains the detailed information of the low-light image. Figure 3 The figure shows a low-light image and the minimum channel constraint map obtained through the above process. Since each part of the low-light image needs to be enhanced differently during low-light image enhancement, and the minimum channel constraint map contains object information in the image, the present invention adopts a pixel attention mechanism to obtain the enhancement strength of different parts of different low-light images. In a specific embodiment, the pixel attention of the minimum channel constraint map is calculated. Specifically, I⊙σ(Conv 1×1 (I)) to obtain pixel attention; where ⊙ represents element-by-element multiplication, σ represents the Sigmoid function, and I is the minimum channel constraint graph.

[0023] Low-light images are different from normal images in that there are fewer bright pixels, and the bright pixels generally need to be enhanced more. In another embodiment, the information in the bright channel image is further used to calculate the pixel attention of the minimum channel constraint map, specifically: obtain the bright channel image of the low-light image, fuse the bright channel image and the minimum channel constraint map into an image of one channel, and calculate the pixel attention of the fused image. The method of fusing into an image of one channel includes but is not limited to pixel-by-pixel addition or multiplication.

[0024] The estimated light intensity of each channel is different, and the light intensity feature map of each channel is obtained by the minimum channel constraint map and the estimated light intensity of each channel. In one embodiment, the light intensity feature map of each channel is obtained based on the pixel attention and the estimated light intensity of each channel, specifically: The estimated light intensity of the channel is multiplied by each element in the normalized pixel attention, and the result of the multiplication is used as the light intensity feature map. For example, the estimated light intensity of channel R is A, and A is multiplied by each element in the normalized pixel attention to obtain a light intensity feature map.

[0025] Local contrast can improve the details of an image. In another embodiment, the light intensity feature map of each channel is obtained based on the pixel attention and the estimated light intensity of each channel, specifically: Calculate the local contrast of each pixel in each channel in the low-light image, and fuse the normalized local contrast map of the channel and the normalized pixel attention to generate a light intensity weight map; in a more specific embodiment, a weighted summation method is adopted for fusion, that is, a weighted summation is performed on the local contrast map of the normalized channel and the value of the same position of the normalized pixel attention, and the weighted summation result is used as the value of the same position in the light intensity weight map.

[0026] The estimated light intensity of the channel is multiplied by each element in the light intensity weight map, and the multiplied result is used as the light intensity feature map.

[0027] Step 2: Input the low-light image into a feature pyramid network, perform a convolution operation on the intensity feature map to obtain a feature map of the same size as the downsampled output of each layer of the feature pyramid network, fuse the downsampled output of the feature pyramid network of the same size with the convolution output of the intensity feature map by channel, further fuse the fusion result by channel into the input sampled on the feature pyramid network of the same size as the fusion result, and use the upsampled last layer output of the feature pyramid network as the nighttime low-light image enhancement result.

[0028] The feature pyramid network includes downsampling and upsampling processes, such as Figure 3As shown, during the downsampling process, the feature map keeps getting smaller, and during the upsampling process, the feature map keeps getting larger until it is the same size as the input image. In order to be able to fuse the intensity feature map into the feature pyramid network, the intensity feature map is convolved to obtain a feature map of the same size as the output of each downsampling layer. The same size refers to W×H×C, where W represents the width of the feature map, H represents the height of the feature map, and C represents the number of channels of the feature map. Then, the downsampling output of the same size and the output of the convolution of the intensity feature map are fused according to the channel. The fusion method includes but is not limited to pixel-by-pixel addition or multiplication. The intensity feature map is convolved to obtain a feature map of the same size as the output of each downsampling layer of the feature pyramid network. Different convolution kernels can be used each time to obtain a feature map of the same size as the output of each downsampling layer, or a cascaded convolution method can be used to obtain a feature map of the same size as the output of each downsampling layer. Among them, channel fusion refers to the fusion of feature maps of the same channel, that is, the feature map output after the convolution of the intensity feature map of each channel is fused with the feature map of the same channel in the feature pyramid in the low-light image. For example, the light intensity feature map of the G channel and the feature map corresponding to the G channel in the feature map of the down-sampling output are fused. After obtaining the fusion result, it is further fused into the input of each layer of the upsampling according to the channel. The same is true for the upsampling process. In order to facilitate channel fusion, in another embodiment, the input and output of each layer in the feature pyramid network are 3 channels, corresponding to the RGB channels respectively. When fusion is performed by channel, the feature maps of the same channel are fused, wherein the feature maps of the same channel are the feature maps of the same channel of the low-light image and the light intensity feature map. Figure 4 A schematic diagram showing the enhancement of a low-illuminance image using the present invention is shown.

[0029] In a second embodiment, the present invention further provides a nighttime low-light image enhancement system, the system comprising the following modules: A light intensity feature generation module, used to obtain a dark channel image of a low-light image taken at night, obtain an estimated light intensity of each channel according to the dark channel image, generate a minimum channel constraint map, calculate a pixel attention of the minimum channel constraint map, and obtain a light intensity feature map of each channel based on the pixel attention and the estimated light intensity of each channel; The image enhancement module is used to input the low-light image into a feature pyramid network, perform a convolution operation on the light intensity feature map to obtain a feature map of the same size as the downsampled output of each layer of the feature pyramid network, fuse the downsampled output of the feature pyramid network of the same size with the convolution output of the light intensity feature map by channel, further fuse the fusion result by channel into the input sampled on the feature pyramid network of the same size as the fusion result, and use the upsampled last layer output of the feature pyramid network as the nighttime low-light image enhancement result.

[0030] Preferably, the pixel attention of the minimum channel constraint graph is calculated as follows: A bright channel image of the low-light image is obtained, the bright channel image and the minimum channel constraint map are fused into an image of one channel, and pixel attention of the fused image is calculated.

[0031] Preferably, the light intensity feature map of each channel is obtained based on the pixel attention and the estimated light intensity of each channel, specifically: Multiply the estimated light intensity of the channel by each element in the normalized pixel attention, and use the multiplied result as a light intensity feature map.

[0032] Preferably, the light intensity feature map of each channel is obtained based on the pixel attention and the estimated light intensity of each channel, specifically: Calculating the local contrast of each pixel in each channel of the low-light image, and fusing the normalized local contrast map of the channel with the normalized pixel attention to generate a light intensity weight map; The estimated light intensity of the channel is multiplied by each element in the light intensity weight map, and the multiplied result is used as the light intensity feature map.

[0033] Preferably, the normalized local contrast map of the channel and the normalized pixel attention are fused to generate a light intensity weight map, specifically: A weighted sum is performed on the normalized local contrast map of the channel and the normalized value of the same position of the pixel attention, and the weighted sum result is used as the value of the same position in the light intensity weight map.

[0034] In another embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the computer-executable instructions are used to implement the above method.

[0035] In yet another embodiment of the present invention, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the above method can be implemented.

[0036] All or part of the steps of the above-mentioned method embodiments can be completed by hardware related to program instructions. The above-mentioned program can be stored in a readable memory. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the above-mentioned memory (storage medium) includes: read-only memory (ROM), random access memory (RAM), flash memory, hard disk, solid state drive, magnetic tape, floppy disk, optical disc and any combination thereof.

[0037] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

Claims

1. A nighttime low-illumination image enhancement method, characterized in that: The method comprises the following steps: Acquire a dark channel image of a low-light image taken at night, obtain an estimated light intensity of each channel according to the dark channel image, generate a minimum channel constraint map, calculate a pixel attention of the minimum channel constraint map, and obtain a light intensity feature map of each channel based on the pixel attention and the estimated light intensity of each channel; The low-illumination image is input into a feature pyramid network, a convolution operation is performed on the light intensity feature map to obtain a feature map of the same size as the downsampled output of each layer of the feature pyramid network, the downsampled output of the feature pyramid network of the same size and the convolution output of the light intensity feature map are fused by channel, the fusion result is further fused by channel to the input sampled on the feature pyramid network of the same size as the fusion result, and the upsampled last layer output of the feature pyramid network is used as the nighttime low-illumination image enhancement result.

2. The method according to claim 1, characterized in that The pixel attention of the minimum channel constraint graph is calculated as follows: A bright channel image of the low-light image is obtained, the bright channel image and the minimum channel constraint map are fused into an image of one channel, and pixel attention of the fused image is calculated.

3. The method according to claim 1, characterized in that The light intensity feature map of each channel is obtained based on the pixel attention and the estimated light intensity of each channel, specifically: Multiply the estimated light intensity of the channel by each element in the normalized pixel attention, and use the multiplied result as a light intensity feature map.

4. The method according to claim 1, characterized in that The light intensity feature map of each channel is obtained based on the pixel attention and the estimated light intensity of each channel, specifically: Calculating the local contrast of each pixel in each channel of the low-light image, and fusing the normalized local contrast map of the channel with the normalized pixel attention to generate a light intensity weight map; The estimated light intensity of the channel is multiplied by each element in the light intensity weight map, and the multiplied result is used as the light intensity feature map.

5. The method according to claim 3, characterized in that The normalized local contrast map of the channel and the normalized pixel attention are fused to generate a light intensity weight map, specifically: A weighted sum is performed on the normalized local contrast map of the channel and the normalized value of the same position of the pixel attention, and the weighted sum result is used as the value of the same position in the light intensity weight map.

6. A nighttime low-light image enhancement system, characterized in that: The system includes the following modules: A light intensity feature generation module, used to obtain a dark channel image of a low-light image taken at night, obtain an estimated light intensity of each channel according to the dark channel image, generate a minimum channel constraint map, calculate a pixel attention of the minimum channel constraint map, and obtain a light intensity feature map of each channel based on the pixel attention and the estimated light intensity of each channel; The image enhancement module is used to input the low-light image into a feature pyramid network, perform a convolution operation on the light intensity feature map to obtain a feature map of the same size as the downsampled output of each layer of the feature pyramid network, fuse the downsampled output of the feature pyramid network of the same size with the convolution output of the light intensity feature map by channel, further fuse the fusion result by channel into the input sampled on the feature pyramid network of the same size as the fusion result, and use the upsampled last layer output of the feature pyramid network as the nighttime low-light image enhancement result.

7. The system according to claim 6, characterized in that The pixel attention of the minimum channel constraint graph is calculated as follows: A bright channel image of the low-light image is obtained, the bright channel image and the minimum channel constraint map are fused into an image of one channel, and pixel attention of the fused image is calculated.

8. The system according to claim 6, characterized in that The light intensity feature map of each channel is obtained based on the pixel attention and the estimated light intensity of each channel, specifically: Multiply the estimated light intensity of the channel by each element in the normalized pixel attention, and use the multiplied result as a light intensity feature map.

9. The system according to claim 6, characterized in that The light intensity feature map of each channel is obtained based on the pixel attention and the estimated light intensity of each channel, specifically: Calculating the local contrast of each pixel in each channel of the low-light image, and fusing the normalized local contrast map of the channel with the normalized pixel attention to generate a light intensity weight map; The estimated light intensity of the channel is multiplied by each element in the light intensity weight map, and the multiplied result is used as the light intensity feature map.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 5.

Images