Image enhancement method suitable for underground coal mine low-illumination image

Abstract

The invention relates to an image enhancement method suitable for an underground coal mine low-illumination image. The method comprises the steps that a low-illumination target image is loaded into a low-illumination image enhancement model, image enhancement processing is carried out on the low-illumination target image through the low-illumination image enhancement model, and the low-illumination target image is decomposed into a corresponding target image illumination image and a target image reflection image; reconstructing and generating a target image enhanced illumination image; performing denoising processing on the target image reflection image generated by decomposition so as to generate a denoised reflection image of the target image after denoising processing; and carrying out merging processing on the target image enhanced illumination image and the target image denoised reflection image to generate a target enhanced image after merging processing. According to the method, the low-illumination image of the underground coal mine can be effectively enhanced, the requirement for computing resources is reduced, the reasoning process of image enhancement is accelerated, and the visual task requirement of the underground coal mine is met.

Description

Technical Field

[0001] The invention relates to an image enhancement method, in particular to an image enhancement method suitable for low-illuminance images in coal mines. Background Art

[0002] In underground coal mines, due to the low light intensity and the presence of a lot of dust, most images captured by recorders or mobile cameras have low contrast and low visibility, which poses a challenge to human visualization and a large number of high-level downstream visual tasks such as image segmentation and object detection. In the past few decades, researchers have been committed to solving the problem of low-light image enhancement and have made a lot of contributions in texture restoration, contrast enhancement, noise removal, etc.

[0003] In recent years, low-light image enhancement methods based on Retinex theory have gained widespread attention. With the rapid development of deep neural networks, DeepCNN has been widely used in low-level image processing. Retinex-net combines Retinex theory with DeepCNN to estimate and adjust low-light images to achieve image contrast enhancement. Specifically, Retinex-net includes a decomposition network for image decomposition and an enhancement network for illumination adjustment.

[0004] Based on the characteristics of Retinex-net, we can see that when using Retinex-net for image enhancement, the following situations mainly exist:

[0005] 1) When performing image enhancement based on Retinex-net, it has to go through multiple forward and backward propagations, which consumes a lot of computing resources and reasoning time. The efficiency and robustness of the network are low, which cannot meet the design and development requirements of the automated detection system for underground coal mine environments.

[0006] 2) The key to image enhancement based on Retinex is the decomposition estimation of the illumination map and the reflectance map. Due to the limited decomposition capability, the traditional decomposition network will cause the loss of detail information of the decomposed reflectance map and cause blur and noise amplification.

[0007] The environment in coal mines is inherently harsh, with low light intensity and a lot of dust. Therefore, when using Retinex-net for image enhancement, it is still difficult to adapt to different downstream visual tasks in coal mines.

[0008] 3) When enhancing the image based on Retinex-net, the denoising of the reflectance map is mainly based on the BM3D algorithm. However, it is sensitive to the type of noise during denoising and has a poor effect on noise other than salt and pepper noise. The dust environment in coal mines is quite different from salt and pepper noise. Therefore, it is impossible to denoise the images obtained in the specific environment of coal mines.

[0009] It can be seen from the above description that for images generated in the underground coal mine environment, the image enhancement methods in the prior art cannot effectively achieve image enhancement and cannot meet the requirements of visual tasks in the underground coal mine. Summary of the invention

[0010] The purpose of the present invention is to overcome the shortcomings of the prior art and provide an image enhancement method suitable for low-illuminance images in coal mines, which can effectively enhance the low-illuminance images in coal mines, reduce the demand for computing resources, accelerate the reasoning process of image enhancement, and meet the visual task requirements in coal mines.

[0011] According to the technical solution provided by the present invention, an image enhancement method suitable for low-light images in coal mines, the image enhancement method comprises:

[0012] A low-illuminance target image generated by shooting in a coal mine is obtained, and the low-illuminance target image is loaded into a low-illuminance image enhancement model, so as to perform image enhancement processing on the low-illuminance target image using the low-illuminance image enhancement model, wherein:

[0013] When performing image enhancement processing on a low-illuminance target image, the low-illuminance target image is first decomposed into a corresponding target image illumination map and a target image reflection map;

[0014] Extracting target illumination map global information and high-frequency feature information of the target illumination map of the target image from the decomposed target illumination map;

[0015] Performing diffusion enhancement on the target illumination map global information to generate diffusion enhanced global information of the target illumination map after the diffusion enhancement;

[0016] Based on the diffuse enhancement global information of the target illumination map and the high-frequency feature information of the target illumination map, the enhanced illumination map of the target image is reconstructed and generated;

[0017] Performing denoising processing on the target image reflectance map generated by the decomposition to generate a denoised target image reflectance map after the denoising processing;

[0018] The target image enhanced illumination map and the target image denoised reflection map are merged to generate a target enhanced image.

[0019] The low-light image enhancement model includes an image decomposition network, an illumination map processing network, a reflectance map denoising network, and a merging output module for performing merging processing, wherein:

[0020] The image decomposition network is adaptively connected with the illumination map processing network and the reflectance map denoising network;

[0021] The illumination map processing network, the reflectance map denoising network and the merge output module are connected;

[0022] When performing image enhancement processing on a low-illuminance target image, the low-illuminance target image is decomposed into a target image illumination map and a target image reflection map through an image decomposition network, and the target image illumination generated by the decomposition is loaded into the illumination image processing network, and at the same time, the target image reflection map generated by the decomposition is loaded into the reflection map denoising network;

[0023] For the received target image illumination map, a corresponding target image enhanced illumination map is generated by using an illumination map processing network, wherein when generating the target image illumination map, the illumination map processing network first extracts target illumination map global information and target illumination map high-frequency feature information of the target illumination map, and then diffusely enhances the target illumination map global information, and then reconstructs and generates the target image enhanced illumination map based on the target illumination map high-frequency feature information and the target illumination map diffusely enhanced global information;

[0024] For the received target image reflection map, denoising the target image reflection map by using a reflection map denoising network, and generating a denoised target image reflection map after the denoising process;

[0025] The merging output module is used to merge the target image denoising reflection map and the target image enhanced illumination map to generate a target enhanced image after merging, wherein the merging process at least includes convolution operation processing of the target image denoising reflection map and the target image enhanced illumination map.

[0026] The image decomposition network includes an image decomposition main network and an image decomposition output unit connected to the image decomposition main network, wherein:

[0027] The image decomposition main network includes at least one feature extraction decomposition unit, wherein the feature extraction decomposition unit includes a decomposition feature extraction module and an image decomposition multiple residual module for extracting spatial features of different sizes;

[0028] The decomposition feature extraction module includes an image decomposition ReLU activation function and an image decomposition convolution layer adaptively connected to the image decomposition ReLU activation function;

[0029] In the same feature extraction decomposition unit, the image decomposition convolution layer is connected to the image decomposition multiple residual module through the image decomposition ReLU activation function.

[0030] The image decomposition multiple residual module includes a module head convolution layer, an outer residual convolution layer, and an inner residual unit adaptively connected to the module head convolution layer, wherein:

[0031] The inner residual unit at least includes a first inner convolutional layer, a second inner convolutional layer, a third inner convolutional layer and a fourth inner convolutional layer connected in sequence;

[0032] The inner residual unit is connected to the module head convolution layer through the inner first convolution layer;

[0033] The fourth inner convolutional layer is connected to the inner connection accumulator;

[0034] The inner first convolution layer is connected to the inner connection accumulator through the inner residual first convolution layer, and the inner second convolution layer is connected to the inner connection accumulator through the inner residual second convolution layer;

[0035] The inner layer connection accumulator is connected to the outer layer connection accumulator through the inter-accumulator convolution layer.

[0036] The outer residual convolution layer forms an outer residual connection with the module head convolution layer, wherein when the outer residual connection is formed, the outer residual convolution layer is connected to the outer connection accumulator;

[0037] The output end of the image decomposition multiple residual module is formed through the outer layer connection accumulator.

[0038] The illumination map processing network includes an illumination map feature extraction unit, an enhanced diffusion model, and a reconstruction generation unit, wherein:

[0039] Extracting features of the illumination map of the target image using an illumination map feature extraction unit, so as to generate target illumination map global information of the illumination map of the target image and target illumination map high-frequency feature information after feature extraction;

[0040] The enhanced diffusion model is used to perform diffusion enhancement on the global information of the target illumination map, and the diffusion enhanced global information of the target illumination map is generated;

[0041] The reconstruction generation unit is used to reconstruct the high-frequency feature information of the target illumination map and the diffusion enhancement global information of the target illumination map, so as to generate a target image enhanced illumination map after the reconstruction processing.

[0042] When extracting features from the target illumination map, the feature extraction method used by the illumination map feature extraction unit includes a two-dimensional discrete wavelet transform method;

[0043] The enhanced diffusion model is generated based on the U-net network;

[0044] During the reconstruction process, the reconstruction process method adopted by the reconstruction generation unit includes a two-dimensional discrete wavelet inverse transform method.

[0045] The reflectance image denoising network includes a denoising feature extraction unit and a denoising size restoration unit adaptively connected to the denoising feature extraction unit, wherein:

[0046] The denoising feature extraction unit is connected to the denoising size restoration unit through an intermediate transition multiple residual module, so that the reflection image denoising network forms a U-shaped network;

[0047] The denoising feature extraction unit includes a denoising feature extraction first convolution layer and a number of denoising feature extraction sub-units connected in sequence, wherein:

[0048] Any denoising feature extraction subunit includes a denoising feature extraction multiple residual module and a denoising feature extraction subconvolution layer adaptively connected to the denoising feature extraction multiple residual module;

[0049] For two adjacent denoising feature extraction subunits, the denoising feature extraction subconvolution layer in the previous denoising feature extraction subunit is connected to the denoising feature extraction multiple residual module in the next denoising feature extraction subunit;

[0050] The denoising and size restoration unit includes a denoising feature extraction tail convolution layer and a number of denoising and size restoration sub-units connected in sequence, wherein:

[0051] For any denoising and size restoration subunit, it includes a denoising and size restoration multiple residual module and a denoising and size restoration subconvolutional layer adaptively connected to the denoising and size restoration multiple residual module;

[0052] For adjacent denoising and size restoration subunits, the denoising and size restoration multiple residual modules in the previous denoising and size restoration subunit are connected to the denoising and size restoration subconvolutional layers in the next denoising and size restoration subunit;

[0053] In the reflection image denoising network, the number of denoising feature extraction subunits is consistent with the number of denoising size restoration subunits, and a denoising feature extraction subunit and a corresponding denoising size restoration subunit are configured to be located in the same denoising network layer;

[0054] In any denoising network layer in the reflectance image denoising network, a jump connection is used between a denoising feature extraction subunit in the denoising network layer and a denoising size restoration subunit in the denoising network layer;

[0055] When the reflectance map denoising network is used to denoise the reflectance map of the target image, the reflectance map of the target image is loaded into the first convolutional layer of denoising feature extraction, and the denoising feature extraction tail convolutional layer outputs the denoised reflectance map of the target image.

[0056] When building a low-light image enhancement model, the construction method includes:

[0057] First build an image decomposition network and / or a reflectance denoising network.

[0058] After the image decomposition network is constructed, the illumination map processing network is constructed using the image decomposition network, wherein:

[0059] When constructing the illumination map processing network, first construct the illumination map processing basic network and the illumination map processing network training data set, and adapt the illumination map processing basic network to the image decomposition network, and train the illumination map processing basic network using the illumination map processing network training data set, so that when the training reaches the target state, the illumination map processing network is generated based on the corresponding illumination map processing basic network, wherein,

[0060] The illumination map processing network training data set includes a plurality of illumination map training samples, each illumination map training sample includes a normal illumination basic image and a corresponding low illumination basic image;

[0061] During training, each illumination map training sample is first loaded into the image decomposition network, so that the normal illumination basic image is decomposed by the image decomposition network to generate a high-brightness sample illumination map and a high-brightness sample reflection map, and the low-light basic image is decomposed by the image decomposition network to generate a low-brightness sample illumination map and a low-brightness sample reflection map;

[0062] Loading the high-brightness sample illumination map and the low-brightness sample illumination map into the illumination map processing basic network, after which the illumination map processing basic network extracts the high-brightness sample illumination map global information of the high-brightness sample illumination map, and extracts the low-brightness sample illumination map global information and the low-brightness sample high-frequency feature information of the low-brightness sample illumination map;

[0063] The global information of the high-brightness sample illumination map is used to enhance the global information of the low-brightness sample illumination map to generate enhanced global information of the low-brightness sample illumination map;

[0064] The enhanced global information of the low-brightness sample illumination map and the high-frequency feature information of the low-brightness sample are reconstructed to generate an enhanced illumination map of the low-brightness sample.

[0065] When acquiring a low-illuminance target image, first acquire a low-illuminance source image generated by shooting, and generate a low-illuminance target image based on the acquired low-illuminance source image, wherein generating a low-illuminance target image based on the low-illuminance source image includes:

[0066] When shooting and generating the low-illuminance source image, collecting the mine environment parameters of the coal mine where the low-illuminance source image is shot and generated, wherein the mine environment parameters include the mine environment brightness and / or the mine dust concentration;

[0067] Bilateral filtering parameters are configured based on the mine environment parameters, and bilateral filtering processing is performed on the low-illuminance source image based on the configured bilateral filtering parameters to generate a low-illuminance target image after the bilateral filtering processing.

[0068] When bilateral filtering is performed on low-illumination source images, we have:

[0069]

[0070] Among them, I filered (x, y) is the pixel value after filtering, I(x, y) is the pixel value in the original image, Ω is the neighborhood window during bilateral filtering, and W p (x, y) is the normalized weight of the pixel value in the neighborhood window Ω, F spatial (i, j) is the Gaussian kernel function in the spatial domain, F rang (I(x,y),I(x+i,y+j)) is the Gaussian kernel function in pixel domain;

[0071] When configuring bilateral filtering parameters based on mine environment parameters, it at least includes configuring the size of the neighborhood window Ω, the spatial domain Gaussian kernel function F based on the mine environment parameters. spatial (i, j), pixel domain Gaussian kernel function F rang (I(x,y),I(x+i,y+j)) is the width of the corresponding kernel function.

[0072] The advantages of the present invention are as follows: an image decomposition network is constructed based on Retinex, and a low-illuminance target image can be decomposed into a target image illumination map and a target image reflection map by using the image decomposition network; the visual quality of the target image illumination map can be improved by using the illumination map processing network; and when the illumination map of the target image is processed by using the illumination map processing network, two-dimensional discrete wavelet transform and two-dimensional discrete wavelet inverse transform can be used to accelerate reasoning and reduce computing resources, thereby improving the efficiency and robustness of the low-illuminance image enhancement model and ensuring the design and development of an automatic detection system for the underground environment of a coal mine.

[0073] The image decomposition network includes image decomposition multiple residual modules, and the reflection image denoising network includes denoising feature extraction multiple residual modules and denoising size recovery multiple residual modules. The image decomposition multiple residual modules, denoising feature extraction multiple residual modules and denoising size recovery multiple residual modules are all based on multiple residual modules, using residual connections to avoid gradient vanishing or explosion, and using 1*1 convolution kernels to help understand additional spatial information.

[0074] The image decomposition network built based on multiple residual modules can improve the detail information extraction ability of the image decomposition network, solve the problem of low light intensity and large amount of dust in images obtained underground in coal mines, and provide a basis for downstream advanced visual tasks. In addition, it can extract spatial features of different sizes and reduce memory requirements.

[0075] The reflectance image denoising network constructed based on multiple residual modules can improve the denoising ability of the denoising network for dust environment images in underground coal mines. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 The present invention is a flowchart of an embodiment of image enhancement.

[0077] Figure 2 A schematic diagram of an embodiment of the low-illumination image enhancement model of the present invention.

[0078] Figure 3 A schematic diagram of an embodiment of the present invention for training and generating a low-light image enhancement model.

[0079] Figure 4 A schematic diagram of an embodiment of the image decomposition network of the present invention.

[0080] Figure 5 A schematic diagram of an embodiment of the image decomposition multiple residual module of the present invention.

[0081] Figure 6 A schematic diagram of an embodiment of a reflectance image denoising network of the present invention. DETAILED DESCRIPTION

[0082] The present invention will be further described below in conjunction with specific drawings and embodiments.

[0083] In order to effectively enhance the low-light image in a coal mine, the present invention provides an image enhancement method suitable for low-light images in a coal mine. Specifically, the image enhancement method includes:

[0084] A low-illuminance target image generated by shooting in a coal mine is obtained, and the low-illuminance target image is loaded into a low-illuminance image enhancement model, so as to perform image enhancement processing on the low-illuminance target image using the low-illuminance image enhancement model, wherein:

[0085] When performing image enhancement processing on a low-illuminance target image, the low-illuminance target image is first decomposed into a corresponding target image illumination map and a target image reflection map;

[0086] Extracting target illumination map global information and high-frequency feature information of the target illumination map of the target image from the decomposed target illumination map;

[0087] Performing diffusion enhancement on the target illumination map global information to generate diffusion enhanced global information of the target illumination map after the diffusion enhancement;

[0088] Based on the diffuse enhancement global information of the target illumination map and the high-frequency feature information of the target illumination map, the enhanced illumination map of the target image is reconstructed and generated;

[0089] Performing denoising processing on the target image reflectance map generated by the decomposition to generate a denoised target image reflectance map after the denoising processing;

[0090] The target image enhanced illumination map and the target image denoised reflection map are merged to generate a target enhanced image.

[0091] It should be understood that the purpose of image enhancement for low-light images is consistent with the prior art. Figure 1 An embodiment of image enhancement for low-illuminance images in a coal mine is shown in the figure. As can be seen from the figure, when enhancing the image, it is necessary to obtain a low-illuminance target image, wherein the low-illuminance target image is the image to be enhanced. In one embodiment of the present invention, the low-illuminance target image is at least an image captured and generated in a coal mine. Due to the poor illumination conditions in a coal mine, the images captured and generated in a coal mine are mostly low-illuminance images.

[0092] In order to perform image enhancement on low-illuminance target images, it is necessary to construct a low-illuminance image enhancement model. After that, the low-illuminance target image can be loaded into the low-illuminance image enhancement model so that the low-illuminance target image can be enhanced using the low-illuminance image enhancement model. It can be understood that after the image enhancement processing, an enhanced target image can be generated.

[0093] Figure 1 An embodiment of image enhancement processing of a low-illuminance target image using a low-illuminance image enhancement model is shown in the figure. As can be seen from the figure, when enhancing a low-illuminance target image, the low-illuminance target image is first decomposed, and after decomposition, a target image illumination map and a target image reflection map can be generated. The corresponding state between the target image illumination map, the target image reflection map and the low-illuminance target image can be consistent with the prior art. The method of decomposing the low-illuminance target image using the low-illuminance image enhancement model can refer to the corresponding description below. After decomposing and generating the target image illumination map and the target image reflection map, the target image illumination map and the target image reflection map should be enhanced respectively. The method and process of enhancing the target image illumination map and the target image reflection map are specifically described below.

[0094] Specifically, when enhancing the illumination map of the target image, the global information of the illumination map of the target image and the high-frequency feature information of the illumination map of the target image should be extracted, wherein the global information of the illumination map of the target image can generally be the low-frequency feature information of the illumination map of the target image, and the high-frequency feature information of the illumination map of the target image can generally be the vertical feature information, horizontal feature information and / or diagonal feature information of the illumination map of the target image. In specific implementation, the low-illumination image enhancement model can diffusely enhance the global information of the illumination map of the target image, while retaining the high-frequency feature information of the illumination map of the target image, wherein after the global information of the illumination map of the target image is diffusely enhanced, the diffusely enhanced global information of the illumination map of the target image can be generated, and the manner and process of diffusely enhancing the global information of the illumination map of the target image can refer to the corresponding description below. After the diffusely enhanced global information of the illumination map of the target image is generated, the enhanced illumination map of the target image can be reconstructed based on the diffusely enhanced global information of the illumination map of the target image and the corresponding high-frequency feature information of the illumination map of the target image; it can be understood that the enhanced illumination map of the target image is the illumination map after the illumination map of the target image is enhanced.

[0095] When the target image reflectance map is enhanced, the target image reflectance map should at least be denoised, so that a denoised target image reflectance map can be generated after the denoising process. That is, the enhancement process of the target image reflectance map is the denoising process of the target image reflectance map, and the denoised target image reflectance map is the reflection map after the target image reflectance map is enhanced.

[0096] From the above description, it can be seen that the target image enhanced illumination map and the target image denoising reflection map both correspond to the same low-illuminance target image. Therefore, the target image enhanced illumination map and the target image denoising reflection map can be merged to generate a target enhanced image after merging. It can be understood that the target enhanced image is the image after the low-illuminance target image is enhanced.

[0097] In one embodiment of the present invention, when acquiring a low-illuminance target image, a low-illuminance source image generated by shooting is first acquired, and the low-illuminance target image is generated based on the acquired low-illuminance source image, wherein the low-illuminance target image is generated based on the low-illuminance source image, including:

[0098] When shooting and generating the low-illuminance source image, collecting the mine environment parameters of the coal mine where the low-illuminance source image is shot and generated, wherein the mine environment parameters include the mine environment brightness and / or the mine dust concentration;

[0099] Bilateral filtering parameters are configured based on the mine environment parameters, and bilateral filtering processing is performed on the low-illuminance source image based on the configured bilateral filtering parameters to generate a low-illuminance target image after the bilateral filtering processing.

[0100] It is known to those skilled in the art that, due to the low illumination and high dust content in the coal mine environment, it will be difficult to achieve the desired enhancement effect if the captured image is directly enhanced. In order to achieve effective image enhancement, the image captured in the coal mine should be filtered first. When filtering the image, the current mine environment parameters in the coal mine should be considered. The current mine environment parameters are also the environment parameters when the image was captured. Since the ambient brightness and dust concentration in the coal mine are the main factors affecting image imaging, the mine environment parameters may include the mine environment brightness and / or the mine dust concentration. Preferably, the mine environment brightness and the mine dust concentration should be considered at the same time.

[0101] In order to obtain the mine environment parameters, the current mine environment brightness and mine dust concentration should be obtained at the same time when taking images. For example, the current environment brightness (cd / m 2 ), the GP2Y1014AU dust sensor can be used to measure the current ambient dust concentration in the coal mine (mg / m 3 ), of course, other methods can also be used to obtain the coal mine environment brightness and the corresponding mine dust concentration.

[0102] In specific implementation, a low-illuminance source image is first captured using an imaging device such as a camera. At the same time, the mine environment parameters corresponding to the low-illuminance source image can be obtained. After that, the bilateral filtering parameters can be configured based on the mine environment parameters. After that, the low-illuminance image can be subjected to bilateral filtering based on the bilateral filtering parameters. When performing the bilateral filtering, the spatial proximity of the low-illuminance source image and the similarity of the pixel values ​​can be considered to smooth the low-illuminance source image while preserving the edge information of the low-illuminance source image. It can be understood that a low-illuminance target image can be generated after the bilateral filtering process. When the low-illuminance source image does not need to be subjected to bilateral filtering, the low-illuminance source image can be configured as the low-illuminance target image.

[0103] In one embodiment of the present invention, when bilateral filtering is performed on a low illumination source image, the following is achieved:

[0104]

[0105] Among them, I filered (x, y) is the pixel value after filtering, I(x, y) is the pixel value in the original image, Ω is the neighborhood window during bilateral filtering, and W p (x, y) is the normalized weight of the pixel value in the neighborhood window Ω, F spatial (i, j) is the Gaussian kernel function in the spatial domain, F rang (I(x,y),I(x+i,y+j)) is the Gaussian kernel function in pixel domain;

[0106] When configuring bilateral filtering parameters based on mine environment parameters, it at least includes configuring the size of the neighborhood window Ω, the spatial domain Gaussian kernel function F based on the mine environment parameters. spatial (i, j), pixel domain Gaussian kernel function F rang (I(x,y),I(x+i,y+j)) is the width of the corresponding kernel function.

[0107] Specifically, an example is given to configure bilateral filtering parameters based on mine environment parameters. For example, if the mine environment brightness is greater than >100 (cd / m 2 ) and the mine dust concentration is less than 1000 (mg / m 3 ), the size of the neighborhood window Ω can be 5, and the spatial domain Gaussian kernel function F spatial The width of the (i, j) kernel function can be 3, and the pixel domain Gaussian kernel function F rang The width of the kernel function corresponding to (I(x,y),I(x+i,y+j)) can be 3; in other cases, the size of the neighborhood window Ω can be 5, and the spatial domain Gaussian kernel function F spatial The width of the (i, j) kernel function can be 9, and the pixel domain Gaussian kernel function F rang The width of the kernel function corresponding to (I(x,y),I(x+i,y+j)) can be 9. It can be understood that for the spatial domain Gaussian kernel function, once the width of the kernel function is determined, those skilled in the art can determine to use the spatial domain Gaussian kernel function for corresponding filtering processing.

[0108] From the above description, it can be seen that the size of the neighborhood window Ω can be configured based on the mine environment parameters. After that, during filtering, the normalized weight W can be calculated according to the pixel value of the current neighborhood window Ω. p (x, y). After that, the low-light source image can be subjected to bilateral filtering based on the above bilateral filtering processing formula. Of course, other methods can also be used to perform bilateral filtering on the low-light source image. The specific bilateral filtering processing method can be selected according to actual needs, and will not be listed here one by one.

[0109] In one embodiment of the present invention, the low-light image enhancement model includes an image decomposition network, an illumination map processing network, a reflectance map denoising network, and a merging output module for performing merging processing, wherein:

[0110] The image decomposition network is adaptively connected with the illumination map processing network and the reflectance map denoising network;

[0111] The illumination map processing network, the reflectance map denoising network and the merge output module are connected;

[0112] When performing image enhancement processing on a low-illuminance target image, the low-illuminance target image is decomposed into a target image illumination map and a target image reflection map through an image decomposition network, and the target image illumination generated by the decomposition is loaded into the illumination image processing network, and at the same time, the target image reflection map generated by the decomposition is loaded into the reflection map denoising network;

[0113] For the received target image illumination map, a corresponding target image enhanced illumination map is generated by using an illumination map processing network, wherein when generating the target image illumination map, the illumination map processing network first extracts target illumination map global information and target illumination map high-frequency feature information of the target illumination map, and then diffusely enhances the target illumination map global information, and then reconstructs and generates the target image enhanced illumination map based on the target illumination map high-frequency feature information and the target illumination map diffusely enhanced global information;

[0114] For the received target image reflection map, denoising the target image reflection map by using a reflection map denoising network, and generating a denoised target image reflection map after the denoising process;

[0115] The merging output module is used to merge the target image denoising reflection map and the target image enhanced illumination map to generate a target enhanced image after merging, wherein the merging process at least includes convolution operation processing of the target image denoising reflection map and the target image enhanced illumination map.

[0116] Figure 2 An embodiment of a low-illumination image enhancement model is shown. In the figure, the low-illumination image is the low-illumination target image mentioned above. It can be seen from the above description that the low-illumination image can be a low-illumination source image or a low-illumination image generated after bilateral filtering of the low-illumination source image. Figure 2 In the figure, the MR-Decompose decomposition network is the image decomposition network. After the low-illuminance image passes through the image decomposition network, it can be decomposed into a low-brightness image illumination map and a low-brightness image reflection map. It can be seen that the low-brightness image illumination map is the above-mentioned target image illumination map, and the low-brightness image reflection map is the above-mentioned target image reflection map. The MRU-Denoise denoising network is the reflection map denoising network. The reflection map denoising network can be used to denoise the target image reflection map and generate a reflection map denoising network.

[0117] Specifically, the illumination map processing network is used to complete the above-mentioned enhancement processing of the illumination map of the target image, and the merging output module is used to perform the merging processing, wherein the merging processing specifically refers to the convolution operation of the target image denoised reflection map and the target image enhanced illumination map, that is, the target image denoised reflection map and the target image enhanced illumination map are convolved to generate the pixel value of the corresponding position of the target enhanced image after the convolution. It should be noted that the method and process of convolving the target image denoised reflection map and the target image enhanced illumination map can be consistent with the prior art and will not be repeated here.

[0118] The following is a detailed description of the image decomposition network, illumination map processing network, and reflection map denoising network.

[0119] In one embodiment of the present invention, the image decomposition network includes an image decomposition main network and an image decomposition output unit connected to the image decomposition main network, wherein:

[0120] The image decomposition main network includes at least one feature extraction decomposition unit, wherein the feature extraction decomposition unit includes a decomposition feature extraction module and an image decomposition multiple residual module for extracting spatial features of different sizes;

[0121] The decomposition feature extraction module includes an image decomposition ReLU activation function and an image decomposition convolution layer adaptively connected to the image decomposition ReLU activation function;

[0122] In the same feature extraction decomposition unit, the image decomposition convolution layer is connected to the image decomposition multiple residual module through the image decomposition ReLU activation function.

[0123] It should be noted that the image decomposition network is constructed based on the Retinex theory. The image decomposition network generally includes an image decomposition main network and an image decomposition output unit. The image decomposition main network should include at least one feature extraction decomposition unit. Figure 4 An embodiment of the image decomposition network of the present invention is shown in FIG. Figure 4 , an embodiment of an image decomposition main network including three feature extraction decomposition units is shown, wherein the three feature extraction decomposition units are connected in series in sequence, and the last feature extraction decomposition unit is connected to the image decomposition output unit.

[0124] Figure 4In , a feature extraction decomposition unit includes a decomposition feature extraction module and an image decomposition multiple residual module. The decomposition feature extraction module is connected to the image decomposition multiple residual module. For two feature extraction decomposition units, along the direction of the image decomposition main network pointing to the image decomposition output unit, the image decomposition multiple residual module in the previous feature extraction decomposition unit is connected to the decomposition feature extraction module in the next feature extraction decomposition unit, such as Figure 4 In the figure, the feature extraction decomposition unit and the image decomposition multiple residual module within the dotted line form a feature extraction decomposition unit, and the decomposition feature extraction module and the image decomposition multiple residual module adjacent to the dotted line form another feature extraction decomposition unit. At this time, the feature extraction decomposition unit corresponding to the dotted box is the previous feature extraction decomposition unit, and the other feature extraction decomposition unit formed is the next feature extraction decomposition unit. Other situations can be referred to the description here, and they will not be listed here one by one.

[0125] In a specific implementation, the decomposition feature extraction module may include an image decomposition ReLU activation function and an image decomposition convolutional layer. Figure 4 In the figure, the image decomposition convolutional layers in the decomposition feature extraction module all use a 3*3 convolution kernel, the image decomposition ReLU activation function in the decomposition feature extraction module and the role of the image decomposition convolutional layer in image decomposition are consistent with the prior art and will not be repeated here. Figure 4 In the image decomposition network shown in FIG, CONVR1, CONVR2, and CONVR3 are corresponding decomposition feature extraction modules, and CONVR3 is an image decomposition output unit. Figure 4 In the figure, the image decomposition output unit adopts the same form as the decomposition feature extraction module, that is, it includes the image decomposition ReLU activation function and a 3*3 convolution kernel.

[0126] Figure 4 In the figure, MRM1-1 to MRM1-3 are three image decomposition multiple residual modules in the image decomposition network, wherein the image decomposition multiple residual module MRM1-1 and the decomposition feature extraction module CONVR1 constitute a feature extraction decomposition unit, the image decomposition multiple residual module MRM1-2 and the decomposition feature extraction module CONVR2 constitute a feature extraction decomposition unit, and the image decomposition multiple residual module MRM1-3 and the decomposition feature extraction module CONVR3 constitute a feature extraction decomposition unit. When the image decomposition main network includes other numbers of feature extraction decomposition units, please refer to Figure 4 And the explanation is here, no more examples are given.

[0127] right Figure 4As for the image decomposition network described above, when decomposing a low-illuminance target image, the low-illuminance target image is loaded into the image decomposition main network, and feature extraction is performed by the feature extraction decomposition unit in the image decomposition main network, and the target image illumination map and the target image reflection map are simultaneously outputted by the image decomposition output unit.

[0128] In order to extract spatial features of different sizes and reduce memory requirements, in one embodiment of the present invention, the image decomposition multiple residual module includes a module head convolution layer, an outer residual convolution layer, and an inner residual unit adaptively connected to the module head convolution layer, wherein:

[0129] The inner residual unit at least includes a first inner convolutional layer, a second inner convolutional layer, a third inner convolutional layer and a fourth inner convolutional layer connected in sequence;

[0130] The inner residual unit is connected to the module head convolution layer through the inner first convolution layer;

[0131] The fourth inner convolutional layer is connected to the inner connection accumulator;

[0132] The inner first convolution layer is connected to the inner connection accumulator through the inner residual first convolution layer, and the inner second convolution layer is connected to the inner connection accumulator through the inner residual second convolution layer;

[0133] The inner layer connection accumulator is connected to the outer layer connection accumulator through the inter-accumulator convolution layer.

[0134] The outer residual convolution layer forms an outer residual connection with the module head convolution layer, wherein when the outer residual connection is formed, the outer residual convolution layer is connected to the outer connection accumulator;

[0135] The output end of the image decomposition multiple residual module is formed through the outer layer connection accumulator.

[0136] Figure 5 An embodiment of an image decomposition multiple residual module is shown in the figure, in which CLRU1 is a module head convolution layer, CLRU6 is an outer residual convolution layer, CLRU2 is an inner first convolution layer, CLRU3 is an inner second convolution layer, CLRU4 is an inner third convolution layer, CLRU5 is an inner fourth convolution layer, CLRU6 is an outer residual convolution layer, CLRU7 is an inter-accumulator convolution layer, CLRU8 is an inner first residual convolution layer, CLRU9 is an inner second residual convolution layer, AD1 is an inner connection accumulator, and AD2 is an outer connection accumulator.

[0137] In specific implementation, the module head convolution layer, outer residual convolution layer, inter-accumulator convolution layer, inner fourth convolution layer, inner residual first convolution layer, and inner residual second convolution layer should use the same convolution layer, such as the same size of convolution kernel and the activation function with LeakyReLU. Figure 5 In the module, the head convolution layer, the outer residual convolution layer, the inter-accumulator convolution layer, the inner residual first convolution layer, and the inner residual second convolution layer are all selected as 1×1 convolution kernels with LeakyReLU activation function.

[0138] The first inner convolutional layer, the second inner convolutional layer, and the third inner convolutional layer should use the same convolutional layer, such as the convolution kernel of the same size and the activation function with LeakyReLU. Figure 5 In the figure, the first inner convolutional layer, the second inner convolutional layer, and the third inner convolutional layer are selected as 3×3 convolution kernels with LeakyReLU activation function.

[0139] Specifically, the inner connection accumulator can be used to perform positional addition on the feature map output by the inner fourth convolutional layer, the feature map output by the inner residual second convolutional layer, and the feature map output by the inner residual first convolutional layer. The outer connection accumulator can be used to perform positional addition on the feature map output by the convolutional layer between the accumulators and the feature map output by the outer residual convolutional layer.

[0140] The following is an example of the working process of the image decomposition multiple residual module, specifically:

[0141] For each input feature map A, it is first convolved through the 1*1 convolution in the convolution layer of the module head, and activated by the LeakyRuLU function after the convolution operation. Then it is convolved through the 3*3 convolution kernel in the first convolution layer of the inner layer, and activated by the LeakyRuLU function after the convolution operation. At this time, the feature map B can be obtained.

[0142] The feature map B is divided into two paths. One path is convolved by the 1*1 convolution in the first convolution layer of the inner residual, and activated by the LeakyRuLU function after the convolution operation to obtain the feature map B_1. The other path is convolved by the 3*3 convolution kernel in the second convolution layer of the inner residual, and activated by the LeakyRuLU function after the convolution operation to obtain the feature map C.

[0143] The feature map C is also divided into two paths. One path is convolved through the 1*1 convolution in the second inner residual convolution layer, and activated by the LeakyRuLU function after the convolution operation to obtain the feature map C_1. The other path is convolved through the 3*3 convolution kernel in the third inner residual convolution layer, and activated by the LeakyRuLU function after the convolution operation, and then convolved through the 1*1 convolution kernel in the fourth inner convolution layer, and activated by the LeakyRuLU function after the convolution to obtain the feature map D.

[0144] The feature maps B_1, C_1, and D are added through the inner connection accumulator, and the added feature maps are convolved and activated by the LeakyRuLU function in the 1*1 convolution in the inter-accumulator convolution layer to obtain the output feature map E. At the same time, the input feature map A is convolved with the 1*1 convolution kernel in the outer residual convolution layer, and directly activated by the LeakyRuLU function after the convolution operation to obtain the feature map A_2. After that, the outer connection accumulator is used to add the feature map E and the feature map A_2 to form a residual connection.

[0145] From the above description of the working process of the image decomposition multiple residual module, it can be seen that the image decomposition multiple residual module is equivalent to using convolution kernels of different sizes to check the interest points of feature maps of different sizes, expand the feature extraction receptive field, fuse feature information of different sizes, and pass it to a deeper network to understand the additional spatial information. At the same time, through the outer residual convolution layer, the inner residual first convolution layer, and the inner residual second convolution layer, the network gradient disappearance can be avoided and the network depth can be improved.

[0146] In one embodiment of the present invention, the illumination map processing network includes an illumination map feature extraction unit, an enhanced diffusion model, and a reconstruction generation unit, wherein:

[0147] Extracting features of the illumination map of the target image using an illumination map feature extraction unit, so as to generate target illumination map global information of the illumination map of the target image and target illumination map high-frequency feature information after feature extraction;

[0148] The enhanced diffusion model is used to perform diffusion enhancement on the global information of the target illumination map, and the diffusion enhanced global information of the target illumination map is generated;

[0149] The reconstruction generation unit is used to reconstruct the high-frequency feature information of the target illumination map and the diffusion enhancement global information of the target illumination map, so as to generate a target image enhanced illumination map after the reconstruction processing.

[0150] In a specific implementation, the illumination map processing network may include an illumination map feature extraction unit, an enhanced diffusion model, and a reconstruction generation unit, wherein the illumination map feature extraction unit can be used to extract features of the illumination map of the target image, and after feature extraction, the global information of the target illumination map and the high-frequency feature information of the target illumination map can be obtained.

[0151] In order to extract features from the illumination map of the target image, in one embodiment of the present invention, when extracting features from the illumination map of the target, the feature extraction method adopted by the illumination map feature extraction unit includes a two-dimensional discrete wavelet transform method, that is, a two-dimensional discrete wavelet transform is performed on the target illumination map, so that after the two-dimensional offline wavelet transform processing, the global information of the target illumination map and the high-frequency feature information of the target illumination map can be obtained, such as Figure 2 As shown, Figure 2 In , the global information generated after two-dimensional discrete wavelet transform is the global information of the target illumination map, and Figure 2 An embodiment in which the high-frequency characteristic information of the target illumination map is vertical information, horizontal information and diagonal information is also shown. In specific implementation, the method and process of performing two-dimensional discrete wavelet transform on the illumination map of the target image can be consistent with the prior art and will not be repeated here.

[0152] It should be noted that the horizontal information, vertical information and diagonal information represent the features of the image in different directions respectively, among which the horizontal information mainly reflects the edge and texture features of the target image illumination map in the horizontal direction; the vertical information reflects the edge and texture features of the target image illumination map in the vertical direction; the diagonal information reflects the edge and texture changes of the target image illumination map in the diagonal direction. The enhanced diffusion model is used to perform diffusion enhancement on the extracted global information of the target illumination map, and the diffusion enhanced global information of the target illumination map is generated. In specific implementation, the enhanced diffusion model can be generated based on the U-net network. Figure 2 The diffusion model in is the enhanced diffusion model. The global information of the target illumination map mainly refers to the overall characteristics and change trends of the illumination map of the target image at different scales, which corresponds to the smooth part or low-frequency component of the global information of the target illumination map.

[0153] In one embodiment of the present invention, during the reconstruction process, the reconstruction process method adopted by the reconstruction generation unit includes a two-dimensional discrete wavelet inverse transform method. Figure 2 An embodiment of reconstruction processing by two-dimensional discrete wavelet inverse transform is shown in FIG. Figure 2 In the above, the output of the diffusion model is the above-mentioned global information of the diffusion enhancement of the target illumination map. The method and process of reconstruction by the two-dimensional discrete wavelet inverse transform method are consistent with the prior art and will not be repeated here.

[0154] In one embodiment of the present invention, the reflectance image denoising network includes a denoising feature extraction unit and a denoising size restoration unit adaptively connected to the denoising feature extraction unit, wherein:

[0155] The denoising feature extraction unit is connected to the denoising size restoration unit through an intermediate transition multiple residual module, so that the reflection image denoising network forms a U-shaped network;

[0156] The denoising feature extraction unit includes a denoising feature extraction first convolution layer and a number of denoising feature extraction sub-units connected in sequence, wherein:

[0157] Any denoising feature extraction subunit includes a denoising feature extraction multiple residual module and a denoising feature extraction subconvolution layer adaptively connected to the denoising feature extraction multiple residual module;

[0158] For two adjacent denoising feature extraction subunits, the denoising feature extraction subconvolution layer in the previous denoising feature extraction subunit is connected to the denoising feature extraction multiple residual module in the next denoising feature extraction subunit;

[0159] The denoising and size restoration unit includes a denoising feature extraction tail convolution layer and a number of denoising and size restoration sub-units connected in sequence, wherein:

[0160] For any denoising and size restoration subunit, it includes a denoising and size restoration multiple residual module and a denoising and size restoration subconvolutional layer adaptively connected to the denoising and size restoration multiple residual module;

[0161] For adjacent denoising and size restoration subunits, the denoising and size restoration multiple residual modules in the previous denoising and size restoration subunit are connected to the denoising and size restoration subconvolutional layers in the next denoising and size restoration subunit;

[0162] In the reflection image denoising network, the number of denoising feature extraction subunits is consistent with the number of denoising size restoration subunits, and a denoising feature extraction subunit and a corresponding denoising size restoration subunit are configured to be located in the same denoising network layer;

[0163] In any denoising network layer in the reflectance image denoising network, a jump connection is used between a denoising feature extraction subunit in the denoising network layer and a denoising size restoration subunit in the denoising network layer;

[0164] When the reflectance map denoising network is used to denoise the reflectance map of the target image, the reflectance map of the target image is loaded into the first convolutional layer of denoising feature extraction, and the denoising feature extraction tail convolutional layer outputs the denoised reflectance map of the target image.

[0165] Specifically, the reflection image denoising network may adopt a U-shaped network. Specifically, the reflection image denoising network may include a denoising feature extraction unit, an intermediate transition multiple residual module, and a denoising size recovery unit. At this time, the denoising feature extraction unit is connected to the denoising size recovery unit through the intermediate transition multiple residual module to form a U-shaped network. The intermediate transition multiple residual module is located at the U-shaped bottom of the U-shaped network. Figure 6 In the example, MRM2-4 is an intermediate transition multiple residual module. In specific implementation, the intermediate transition multiple residual module can adopt the same architecture as the above-mentioned image decomposition multiple residual module. The intermediate transition multiple residual module can be referred to Figure 5 And the corresponding description of the above-mentioned image decomposition multiple residual module.

[0166] Figure 6 An embodiment of the reflectance image denoising network is shown in FIG. In the figure, the number of denoising feature extraction subunits and the number of denoising size recovery subunits are both 3. Figure 6 The denoising feature extraction subunit and the denoising size restoration subunit are described in detail.

[0167] Figure 6 In the figure, CLRU10 is the first convolutional layer for denoising feature extraction, which is a convolutional kernel with LeakyReLu activation function and size 3×3. MRM2-1 is the first denoising feature extraction multiple residual module in the reflectance image denoising network, CLRU11 is the first denoising feature extraction sub-convolutional layer in the reflectance image denoising network, and the denoising feature extraction multiple residual module MRM2-1 and the denoising feature extraction sub-convolutional layer CLRU11 constitute a denoising feature extraction sub-unit; similarly, the denoising feature extraction multiple residual module MRM2-2 and the denoising feature extraction sub-convolutional layer CLRU12 constitute another denoising feature extraction sub-unit, and the denoising feature extraction multiple residual module MRM2-3 and the denoising feature extraction sub-convolutional layer CLRU13 constitute the third denoising feature extraction sub-unit.

[0168] Specifically, the denoising feature extraction multiple residual module MRM2-1, the denoising feature extraction multiple residual module MRM2-2 and the denoising feature extraction multiple residual module MRM2-3 also adopt the same structure, such as the structure of the above-mentioned image decomposition multiple residual module. The denoising feature extraction sub-convolution layer CLRU11, the denoising feature extraction sub-convolution layer CLRU12 and the denoising feature extraction sub-convolution layer CLRU13 all have LeakyReLu activation functions and use convolution kernels of size 2×2.

[0169] Figure 6In the figure, CLRU17 is the denoising feature extraction tail convolution layer, the denoising feature extraction tail convolution layer is a convolution kernel with a LeakyReLu activation function and a size of 3×3, MRM2-5 is the first denoising size recovery multiple residual module in the reflection image denoising network, CLRU14 is the first denoising size recovery subconvolution layer in the reflection image denoising network, the denoising size recovery multiple residual module MRM2-5 and the denoising size recovery subconvolution layer CLRU14 constitute a denoising size recovery subunit; similarly, the denoising size recovery multiple residual module MRM2-6 and the denoising size recovery subconvolution layer CLRU15 constitute another denoising size recovery subunit, and the denoising size recovery multiple residual module MRM2-7 and the denoising size recovery subconvolution layer CLRU16 constitute the third denoising size recovery subunit.

[0170] Specifically, the denoising size restoration multiple residual module MRM2-5, the size restoration multiple residual module MRM2-6 and the denoising size restoration multiple residual module MRM2-7 also adopt the same structure, such as the structure of the above-mentioned image decomposition multiple residual module. The denoising size restoration subconvolution layer CLRU14, the denoising size restoration subconvolution layer CLRU15 and the denoising size restoration subconvolution layer CLRU16 are all convolution kernels with LeakyReLu activation function and size of 2×2.

[0171] Figure 6 In the figure, the input end of the intermediate transition multiple residual module MRM2-4 is connected to the output end of the denoising feature extraction sub-convolutional layer CLRU13, and the output end of the intermediate transition multiple residual module MRM2-4 is connected to the input end of the denoising size recovery sub-convolutional layer CLRU14.

[0172] The denoising feature extraction multiple residual module MRM2-1 and the denoising size restoration multiple residual module MRM2-7 are located in the same denoising network layer, so the output end of the denoising feature extraction multiple residual module MRM2-1 is also jump-connected to the denoising size restoration multiple residual module MRM2-7. Similarly, the denoising feature extraction multiple residual module MRM2-2 and the denoising size restoration multiple residual module MRM2-6 are located in the same denoising network layer, so the output end of the denoising feature extraction multiple residual module MRM2-2 is also jump-connected to the denoising size restoration multiple residual module MRM2-6; the denoising feature extraction multiple residual module MRM2-3 and the denoising size restoration multiple residual module MRM2-5 are located in the same denoising network layer, so the output end of the denoising feature extraction multiple residual module MRM2-3 is also jump-connected to the denoising size restoration multiple residual module MRM2-5.

[0173] Below Figure 6 The denoising process of the reflection image denoising network is shown in FIG. 1 to illustrate the denoising process, specifically:

[0174] For the denoising target image reflection map, a convolution operation is first performed through the 3*3 dilated convolution in the first convolution layer of the denoising feature extraction, and after the convolution operation, a large amount of feature information is obtained by activating the LeakyRuLU function, and then input into the denoising feature extraction multiple residual module MRM2-1. The denoising feature extraction multiple residual module MRM2-1 can output a feature map A100. The way in which the denoising feature extraction multiple residual module MRM2-1 processes the feature information can refer to the description of the above-mentioned image decomposition multiple residual module.

[0175] The feature map A100 is convolved through a 2*2 stride convolution in the denoising feature extraction subconvolution layer CLRU11, and then activated by a LeakyRuLU function and input into the denoising feature extraction multiple residual module MRM2-2, so that the feature map A110 can be obtained.

[0176] The feature map A110 is convolved by a 2*2 stride convolution in the denoising feature extraction subconvolution layer CLRU12, and then activated by the LeakyRuLU function and input into the denoising feature extraction multiple residual module MRM2-3 to obtain the feature map A120.

[0177] The feature map A120 is convolved with a 2*2 stride convolution in the denoising feature extraction sub-convolution layer CLRU13, and then activated by the LeakyRuLU function and input into the intermediate transition multiple residual module MRM2-4, and then deconvolved with a 2*2 stride deconvolution in the denoising size recovery sub-convolution layer CLRU14, and then activated by the LeakyRuLU function to obtain the feature map A130.

[0178] The feature map A120 and the feature map A130 are concatenated, and then processed by the denoising and size recovery multiple residual module MRM2-5, and deconvolved by the 2*2 stride deconvolution in the denoising and size recovery subconvolution layer CLRU15, and after the deconvolution operation, the feature map A140 is obtained by activation by the LeakyRuLU function.

[0179] The feature map A110 and the feature map A140 are concatenated, and then passed through the denoising and size recovery multiple residual module MRM2-6 and input into the denoising and size recovery sub-convolution layer CLRU15. The deconvolution operation is performed through the 2*2 stride deconvolution in the denoising and size recovery sub-convolution layer CLRU15. After that, the feature map A150 is obtained by activation through the LeakyRuLU function.

[0180] The feature map A100 and the feature map A150 are spliced, and then passed through the denoising size recovery multiple residual module MRM2-7 and then input into the denoising feature extraction tail convolution layer CLRU17. The convolution operation is performed through the 3*3 dilated convolution in the denoising feature extraction tail convolution layer. After that, the final output is obtained by activation with the LeakyRuLU function, and the denoising reflection map of the target image can be obtained.

[0181] In one embodiment of the present invention, when constructing a low-illumination image enhancement model, the construction method includes:

[0182] First build an image decomposition network and / or a reflectance denoising network.

[0183] After the image decomposition network is constructed, the illumination map processing network is constructed using the image decomposition network, wherein:

[0184] When constructing the illumination map processing network, first construct the illumination map processing basic network and the illumination map processing network training data set, and adapt the illumination map processing basic network to the image decomposition network, and train the illumination map processing basic network using the illumination map processing network training data set, so that when the training reaches the target state, the illumination map processing network is generated based on the corresponding illumination map processing basic network, wherein,

[0185] The illumination map processing network training data set includes a plurality of illumination map training samples, each illumination map training sample includes a normal illumination basic image and a corresponding low illumination basic image;

[0186] During training, each illumination map training sample is first loaded into the image decomposition network, so that the normal illumination basic image is decomposed by the image decomposition network to generate a high-brightness sample illumination map and a high-brightness sample reflection map, and the low-light basic image is decomposed by the image decomposition network to generate a low-brightness sample illumination map and a low-brightness sample reflection map;

[0187] Loading the high-brightness sample illumination map and the low-brightness sample illumination map into the illumination map processing basic network, after which the illumination map processing basic network extracts the high-brightness sample illumination map global information of the high-brightness sample illumination map, and extracts the low-brightness sample illumination map global information and the low-brightness sample high-frequency feature information of the low-brightness sample illumination map;

[0188] The global information of the high-brightness sample illumination map is used to enhance the global information of the low-brightness sample illumination map to generate enhanced global information of the low-brightness sample illumination map;

[0189] The enhanced global information of the low-brightness sample illumination map and the high-frequency feature information of the low-brightness sample are reconstructed to generate an enhanced illumination map of the low-brightness sample.

[0190] The above-mentioned low-illumination image enhancement model generally needs to be constructed in advance. The following is an example of a method for constructing a low-illumination image enhancement model.

[0191] From the above description, it can be seen that the low-light image enhancement model includes an image decomposition network, a reflectance map denoising network, and an illumination map processing network. In specific implementation, the image decomposition network and the reflectance map denoising network can be independently constructed first, and then the constructed image decomposition network can be used to assist in the construction of the illumination map processing network. That is, when constructing the illumination map processing network, it is necessary to rely on the constructed image decomposition network. After the image decomposition network, the reflectance map denoising network, and the illumination map processing network are constructed, the low-light image enhancement model can be formed by referring to the connection and coordination described above.

[0192] The following is a detailed description of the process of constructing the image decomposition network, the reflectance map denoising network, and the illumination map processing network.

[0193] When constructing an image decomposition network, the image decomposition basic network should be constructed first. It can be understood that the image decomposition basic network should have the same network architecture as the image decomposition network. The situation of the image decomposition basic network can refer to the description of the above image decomposition network.

[0194] When constructing the image decomposition basic network, an image decomposition network training data set should also be constructed, wherein the image decomposition network training data set includes several image decomposition network training samples, and each image decomposition network training sample should include a normal light training image and a corresponding low light training image. In specific implementation, an image decomposition network training data set can be constructed based on the public data set LOLv1 and the public data set LOLv2, wherein at least 2189 pairs of images can be obtained, 220 pairs of images can be used for evaluation, and the remaining pairs of images can be used to construct an image decomposition network training data set. At this time, a pair of images can form an image decomposition network training sample.

[0195] When training the image decomposition basic network, the loss function used can be:

[0196]

[0197] Among them, L DC is the training loss of the image decomposition base network, L DC-com To decompose the network training content loss, L DC-per To decompose the perceptual loss of network training, λ1, λ2, λ4, and λ5 are all balanced reflection consistency coefficients, λ3 is the smoothness coefficient, and R low The low-light training image reflectance map is generated by decomposing the low-light training image through the image decomposition basic network, R norI is the normal lighting training image reflectance map generated by decomposing the normal lighting training image through the image decomposition basic network. low I is the illumination map of the low-light training image generated by decomposing the low-light training image through the image decomposition basic network. nor is the illumination map of the normal illumination training image generated by decomposing the normal illumination training image through the image decomposition basic network, j represents the number of output feature layers in the image decomposition basic network, and C j H j W j represents the size of the output feature map of the j-th output feature layer, φ j is the j-th output feature layer in the image decomposition basic network, S low is the low-light training image, S nor is a normal lighting training image, M is the number of image decomposition network training samples loaded in each batch when the image decomposition basic network is trained once, * is the convolution operation, is the L2 norm operation, and FJ is the total number of feature layers in the image decomposition base network.

[0198] It should be noted that when using Figure 4 When the image decomposition network in is constructed and the corresponding image decomposition basic network is constructed, the total number of feature layers FJ in the image decomposition basic network should be 7, such as Figure 4 The decomposition feature extraction module CONVR1, solution feature extraction module CONVR1, solution feature extraction module CONVR2, solution feature extraction module CONVR3, and image decomposition output unit CONVR4 are all output feature layers.

[0199] is the reflection feature map output by the j-th output feature layer of the low-light training image, is the illumination feature map output by the j-th output feature layer for the low-light training image, is the reflection feature map output by the j-th output feature layer of the normal illumination training image, is the illumination feature map output by the j-th output feature layer of the normal illumination training image. j (S low ) is the jth reference feature map generated from the low-light training image, φ j (S nor ) is the jth reference feature map generated from the normal illumination training image.

[0200] For the j-th reference feature map φ generated from the low-light training image j (S low), generally, the feature extraction of the low-light training image can be performed by decomposing the reference feature extraction network. Specifically, the decomposing reference feature extraction network can be an existing commonly used network, such as the decomposing reference feature extraction network can select the feature extraction network VGG-16, wherein the number of output feature layers in the feature extraction network VGG-16 is not less than the number of output feature layers in the image decomposition basic network. For example, when the total number of feature layers FJ in the image decomposition basic network is 7, the 1, 3, 5, 7, 9, 13, and 16 output feature layers in the feature extraction network VGG-16 can be respectively corresponded to the corresponding output feature layers in the image decomposition basic network, so as to obtain and generate the corresponding reference feature map. For example, when j=1, the low-light training image is loaded into the feature extraction network VGG-16, and the feature map of the first output feature layer in the feature extraction network VGG-16 is used as the reference feature map φ1(S low ), when j = 2, for the same low-light training image loaded into the feature extraction network VGG-16, the feature map of the third output feature layer in the feature extraction network VGG-16 is used as the reference feature map φ2 (S low ), when j takes other values, the corresponding reference feature graph can be referred to here.

[0201] For the jth reference feature map φ generated from the normal lighting training image j (S nor ), please refer to the j-th reference feature map φ j (S low ), the difference is that the normal lighting training image is loaded into the feature extraction network VGG-16, and the corresponding j-th reference feature map φ is obtained j (S nor ).

[0202] In a specific implementation, the corresponding values ​​of the balanced reflection consistency coefficient λ1, the balanced reflection consistency coefficient λ2, the balanced reflection consistency coefficient λ4, the balanced reflection consistency coefficient λ5 and the smoothness coefficient λ3 can all be 1. When training the image decomposition basic network, when the value of the loss function tends to be stable, it can be determined that the image decomposition basic network training has reached the target state. At this time, the current image decomposition basic network configuration can be used as the image decomposition network.

[0203] In specific implementation, when constructing the reflection image denoising network, the reflection image denoising basic network should be constructed first. It can be understood that the reflection image denoising basic network should have the same architecture as the reflection image denoising network. Therefore, the reflection image denoising basic network can be constructed by referring to the description of the above reflection image denoising network.

[0204] When constructing a reflectance map denoising basic network, a reflectance map denoising network training data set should also be constructed, wherein the reflectance map denoising network training data set includes a number of denoising network training samples, and each denoising network training sample may include a training noise image and a training true value image corresponding to the training noise image. In specific implementation, the reflectance map denoising network training data set can be constructed using the three data sets of the public data set SIDD, the public data set RENOIR, and the public data set NIND, wherein at least 1246 pairs of noise / true value images can be obtained, 120 pairs of noise / true value images can be used for evaluation, and the remaining pairs of images are used to construct the reflectance map denoising network training data set. At this time, a denoising network training sample is formed with the pairs of images.

[0205] When training the basic network for reflectance image denoising, the loss function used can be:

[0206]

[0207] Among them, L DN is the training loss of the reflectance image denoising base network, L DN-com is the denoising network training content loss, L DN-per Train the perceptual loss for the denoising network, is the denoised reflection image of the training noise image corresponding to the training noise image and generated after the reflection image denoising basic network, is the training truth image reflection map generated after the training truth image passes through the image decomposition network, m represents the number of feature layers in the image decomposition basic network, and A m B m D m Indicates the size of the output feature map of the m-th output feature layer, It is the denoising feature map output by the mth output feature layer in the reflectance map denoising basic network. is the mth reference feature map generated by the training truth image, MN is the number of denoising network training samples used in each batch when the reflectance map denoising basic network is trained once, QZ is the number of output feature layers in the reflectance map denoising basic network, To calculate the L2 norm operation.

[0208] From the above description, it can be seen that when training the reflection image denoising basic network, it is necessary to use the above-mentioned image decomposition network, that is, to load the training noise image into the image decomposition network, and obtain the corresponding training noise image reflection map. After that, the training noise image reflection map is loaded into the reflection image denoising basic network to obtain the reflection map after denoising the training noise image. Similarly, loading the training true value image into the image decomposition network also obtains the corresponding training true value image reflection map

[0209] Specifically, when using Figure 6 When the reflection image denoising network is constructed, the number of feature layers QZ in the reflection image denoising basic network is 15.

[0210] For the mth reference feature map generated by the training true value image Generally, the denoising reference feature extraction network can be used to generate features for low-light training images. Specifically, the denoising reference feature extraction network can be an existing commonly used network. For example, the denoising reference feature extraction network can select the feature extraction network VGG-16, wherein the number of output feature layers in the feature extraction network VGG-16 is not less than the number of output feature layers in the reflectance map denoising basic network. For example, when the total number of feature layers QZ in the reflectance map denoising basic network is 7, the output feature layers 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, and 16 in the feature extraction network VGG-16 can be respectively corresponded to the corresponding output feature layers in the reflectance map denoising basic network, so as to obtain and generate the corresponding reference feature map. For example, when m=1, the training true value image is loaded into the feature extraction network VGG-16, and the feature map of the first output feature layer in the feature extraction network VGG-16 is used as the reference feature map When m=2, for the same low-light training image loaded into the feature extraction network VGG-16, the feature map of the second output feature layer in the feature extraction network VGG-16 is used as the reference feature map When m takes other values, the corresponding reference feature graph can be found in the description here

[0211] When training the reflection image denoising basic network, when the value of the corresponding loss function tends to be stable, it can be determined that the reflection image denoising basic network training has reached a target state. At this time, the current reflection image denoising basic network configuration can be used as the reflection image denoising network.

[0212] It is understandable that when constructing the illumination map processing network, the image decomposition network should be used, that is, the image decomposition network should be connected to the constructed illumination map processing basic network, and the illumination map processing network training data set should be constructed. In specific implementation, when constructing the illumination map processing network training data set, the public data set LSRW can be used. For example, the public data set LSRW can be used to obtain 5650 pairs of low-light images and normal-light images, 550 pairs of images of which are used for evaluation, and the remaining image data can be used to construct the illumination map processing network training data set. At this time, a pair of images is used as an illumination map training sample. It can be seen that each illumination map training sample includes a normal-light basic image and a corresponding low-light basic image.

[0213] It should be understood that the illumination map processing basic network has the same architecture as the illumination map processing network described above. For details, please refer to the above description to build the corresponding illumination map processing basic network. When training the illumination map processing basic network, the loss function used can refer to the DDPM loss function used in the literature: "Ho J, Jain A, Abbeel P. Denoising diffusion probabilistic models [J]. Advances in neural information processing systems, 2020, 33: 6840-6851." During training, when the loss function region is stable, the illumination map processing basic training reaches the target state, and thereafter, the illumination map processing basic network configuration can be used as the illumination map processing network.

[0214] From the above description, we can see that when building a low-light image enhancement model, we should first build an image decomposition network. After that, we use the image decomposition network to build a reflection map denoising network and an illumination map processing network. Figure 3 An embodiment of the present invention for constructing a low-illumination image enhancement model is shown in FIG.

[0215] From the above description, it can be seen that the present invention constructs an image decomposition network based on Retinex. The image decomposition network can be used to decompose the low-illuminance target image into a target image illumination map and a target image reflection map. The illumination map processing network can improve the visual quality of the target image illumination map. When the illumination map processing network is used to process the target image illumination map, two-dimensional discrete wavelet transform and two-dimensional discrete wavelet inverse transform can be used to accelerate reasoning and reduce computing resources, thereby improving the efficiency and robustness of the low-illuminance image enhancement model and ensuring the design and development of an automated detection system for the underground coal mine environment.

[0216] The image decomposition network includes an image decomposition multiple residual module, and the reflectance image denoising network includes a denoising feature extraction multiple residual module and a denoising size restoration multiple residual module. The image decomposition multiple residual module, the denoising feature extraction multiple residual module and the denoising size restoration multiple residual module are all based on the multiple residual module (MultiRM), using residual connections to avoid gradient vanishing or explosion, and using 1*1 convolution kernels to help understand additional spatial information.

[0217] The image decomposition network built based on multiple residual modules can improve the detail information extraction ability of the image decomposition network, solve the problem of low light intensity and large amount of dust in images obtained underground in coal mines, and provide a basis for downstream advanced visual tasks. In addition, it can extract spatial features of different sizes and reduce memory requirements.

[0218] The reflectance image denoising network constructed based on multiple residual modules (MultiRM) can improve the denoising ability of the denoising network for dust environment images in underground coal mines.

Claims

1. An image enhancement method suitable for low-light images in coal mines, characterized in that: The image enhancement method comprises: A low-illuminance target image generated by shooting in a coal mine is obtained, and the low-illuminance target image is loaded into a low-illuminance image enhancement model, so as to perform image enhancement processing on the low-illuminance target image using the low-illuminance image enhancement model, wherein: When performing image enhancement processing on a low-illuminance target image, the low-illuminance target image is first decomposed into a corresponding target image illumination map and a target image reflection map; Extracting target illumination map global information and high-frequency feature information of the target illumination map of the target image from the decomposed target illumination map; Performing diffusion enhancement on the target illumination map global information to generate diffusion enhanced global information of the target illumination map after the diffusion enhancement; Based on the diffuse enhancement global information of the target illumination map and the high-frequency feature information of the target illumination map, the enhanced illumination map of the target image is reconstructed and generated; Performing denoising processing on the target image reflectance map generated by the decomposition to generate a denoised target image reflectance map after the denoising processing; The target image enhanced illumination map and the target image denoised reflection map are merged to generate a target enhanced image.

2. The image enhancement method suitable for low-light images in coal mines according to claim 1, characterized in that: The low-light image enhancement model includes an image decomposition network, an illumination map processing network, a reflectance map denoising network, and a merging output module for performing merging processing, wherein: The image decomposition network is adaptively connected with the illumination map processing network and the reflectance map denoising network; The illumination map processing network, the reflectance map denoising network and the merge output module are connected; When performing image enhancement processing on a low-illuminance target image, the low-illuminance target image is decomposed into a target image illumination map and a target image reflection map through an image decomposition network, and the target image illumination generated by the decomposition is loaded into the illumination image processing network, and at the same time, the target image reflection map generated by the decomposition is loaded into the reflection map denoising network; For the received target image illumination map, a corresponding target image enhanced illumination map is generated by using an illumination map processing network, wherein when generating the target image illumination map, the illumination map processing network first extracts target illumination map global information and target illumination map high-frequency feature information of the target illumination map, and then diffusely enhances the target illumination map global information, and then reconstructs and generates the target image enhanced illumination map based on the target illumination map high-frequency feature information and the target illumination map diffusely enhanced global information; For the received target image reflection map, denoising the target image reflection map by using a reflection map denoising network, and generating a denoised target image reflection map after the denoising process; The merging output module is used to merge the target image denoising reflection map and the target image enhanced illumination map to generate a target enhanced image after merging, wherein the merging process at least includes convolution operation processing of the target image denoising reflection map and the target image enhanced illumination map.

3. The image enhancement method suitable for low-light images in coal mines according to claim 2, characterized in that: The image decomposition network includes an image decomposition main network and an image decomposition output unit connected to the image decomposition main network, wherein: The image decomposition main network includes at least one feature extraction decomposition unit, wherein the feature extraction decomposition unit includes a decomposition feature extraction module and an image decomposition multiple residual module for extracting spatial features of different sizes; The decomposition feature extraction module includes an image decomposition ReLU activation function and an image decomposition convolution layer adaptively connected to the image decomposition ReLU activation function; In the same feature extraction decomposition unit, the image decomposition convolution layer is connected to the image decomposition multiple residual module through the image decomposition ReLU activation function.

4. The image enhancement method suitable for low-light images in coal mines according to claim 3 is characterized by: The image decomposition multiple residual module includes a module head convolution layer, an outer residual convolution layer, and an inner residual unit adaptively connected to the module head convolution layer, wherein: The inner residual unit at least includes a first inner convolutional layer, a second inner convolutional layer, a third inner convolutional layer and a fourth inner convolutional layer connected in sequence; The inner residual unit is connected to the module head convolution layer through the inner first convolution layer; The fourth inner convolutional layer is connected to the inner connection accumulator; The inner first convolution layer is connected to the inner connection accumulator through the inner residual first convolution layer, and the inner second convolution layer is connected to the inner connection accumulator through the inner residual second convolution layer; The inner layer connection accumulator is connected to the outer layer connection accumulator through the inter-accumulator convolution layer. The outer residual convolution layer forms an outer residual connection with the module head convolution layer, wherein when the outer residual connection is formed, the outer residual convolution layer is connected to the outer connection accumulator; The output end of the image decomposition multiple residual module is formed through the outer layer connection accumulator.

5. The image enhancement method suitable for low-light images in coal mines according to claim 2, characterized in that: The illumination map processing network includes an illumination map feature extraction unit, an enhanced diffusion model, and a reconstruction generation unit, wherein: Extracting features of the illumination map of the target image using an illumination map feature extraction unit, so as to generate target illumination map global information of the illumination map of the target image and target illumination map high-frequency feature information after feature extraction; The enhanced diffusion model is used to perform diffusion enhancement on the global information of the target illumination map, and the diffusion enhanced global information of the target illumination map is generated; The reconstruction generation unit is used to reconstruct the high-frequency feature information of the target illumination map and the diffusion enhancement global information of the target illumination map, so as to generate a target image enhanced illumination map after the reconstruction processing.

6. The image enhancement method suitable for low-light images in coal mines according to claim 5, characterized in that: When extracting features from the target illumination map, the feature extraction method used by the illumination map feature extraction unit includes a two-dimensional discrete wavelet transform method; The enhanced diffusion model is generated based on the U-net network; During the reconstruction process, the reconstruction process method adopted by the reconstruction generation unit includes a two-dimensional discrete wavelet inverse transform method.

7. The image enhancement method suitable for low-light images in coal mines according to claim 2, characterized in that: The reflectance image denoising network includes a denoising feature extraction unit and a denoising size restoration unit adaptively connected to the denoising feature extraction unit. in, The denoising feature extraction unit is connected to the denoising size restoration unit through an intermediate transition multiple residual module, so that the reflection image denoising network forms a U-shaped network; The denoising feature extraction unit includes a denoising feature extraction first convolution layer and a number of denoising feature extraction sub-units connected in sequence, wherein: Any denoising feature extraction subunit includes a denoising feature extraction multiple residual module and a denoising feature extraction subconvolution layer adaptively connected to the denoising feature extraction multiple residual module; For two adjacent denoising feature extraction subunits, the denoising feature extraction subconvolution layer in the previous denoising feature extraction subunit is connected to the denoising feature extraction multiple residual module in the next denoising feature extraction subunit; The denoising and size restoration unit includes a denoising feature extraction tail convolution layer and a number of denoising and size restoration sub-units connected in sequence, wherein: For any denoising and size restoration subunit, it includes a denoising and size restoration multiple residual module and a denoising and size restoration subconvolutional layer adaptively connected to the denoising and size restoration multiple residual module; For adjacent denoising and size restoration subunits, the denoising and size restoration multiple residual modules in the previous denoising and size restoration subunit are connected to the denoising and size restoration subconvolutional layers in the next denoising and size restoration subunit; In the reflection image denoising network, the number of denoising feature extraction subunits is consistent with the number of denoising size restoration subunits, and a denoising feature extraction subunit and a corresponding denoising size restoration subunit are configured to be located in the same denoising network layer; In any denoising network layer in the reflectance image denoising network, a jump connection is used between a denoising feature extraction subunit in the denoising network layer and a denoising size restoration subunit in the denoising network layer; When the reflectance map denoising network is used to denoise the reflectance map of the target image, the reflectance map of the target image is loaded into the first convolutional layer of denoising feature extraction, and the denoising feature extraction tail convolutional layer outputs the denoised reflectance map of the target image.

8. The image enhancement method suitable for low-light images in coal mines according to claim 2, characterized in that: When building a low-light image enhancement model, the construction method includes: First build an image decomposition network and / or a reflectance denoising network. After the image decomposition network is constructed, the illumination map processing network is constructed using the image decomposition network, wherein: When constructing the illumination map processing network, first construct the illumination map processing basic network and the illumination map processing network training data set, and adapt the illumination map processing basic network to the image decomposition network, and train the illumination map processing basic network using the illumination map processing network training data set, so that when the training reaches the target state, the illumination map processing network is generated based on the corresponding illumination map processing basic network, wherein, The illumination map processing network training data set includes a plurality of illumination map training samples, each illumination map training sample includes a normal illumination basic image and a corresponding low illumination basic image; During training, each illumination map training sample is first loaded into the image decomposition network, so that the normal illumination basic image is decomposed by the image decomposition network to generate a high-brightness sample illumination map and a high-brightness sample reflection map, and the low-light basic image is decomposed by the image decomposition network to generate a low-brightness sample illumination map and a low-brightness sample reflection map; Loading the high-brightness sample illumination map and the low-brightness sample illumination map into the illumination map processing basic network, after which the illumination map processing basic network extracts the high-brightness sample illumination map global information of the high-brightness sample illumination map, and extracts the low-brightness sample illumination map global information and the low-brightness sample high-frequency feature information of the low-brightness sample illumination map; The global information of the high-brightness sample illumination map is used to enhance the global information of the low-brightness sample illumination map to generate enhanced global information of the low-brightness sample illumination map; The enhanced global information of the low-brightness sample illumination map and the high-frequency feature information of the low-brightness sample are reconstructed to generate an enhanced illumination map of the low-brightness sample.

9. The image enhancement method suitable for low-light images in coal mines according to any one of claims 1 to 8, characterized in that: When acquiring a low-illuminance target image, first acquire a low-illuminance source image generated by shooting, and generate a low-illuminance target image based on the acquired low-illuminance source image, wherein generating a low-illuminance target image based on the low-illuminance source image includes: When shooting and generating the low-illuminance source image, collecting the mine environment parameters of the coal mine where the low-illuminance source image is shot and generated, wherein the mine environment parameters include the mine environment brightness and / or the mine dust concentration; Bilateral filtering parameters are configured based on the mine environment parameters, and bilateral filtering processing is performed on the low-illuminance source image based on the configured bilateral filtering parameters to generate a low-illuminance target image after the bilateral filtering processing.

10. The image enhancement method suitable for low-light images in coal mines according to claim 9, characterized in that: When bilateral filtering is performed on low-illumination source images, we have: Among them, I filered (x, y) is the pixel value after filtering, I(x, y) is the pixel value in the original image, Ω is the neighborhood window during bilateral filtering, and W p (x, y) is the normalized weight of the pixel value in the neighborhood window Ω, F spatial (i, j) is the Gaussian kernel function in the spatial domain, F rang (I(x,y),I(x+i,y+j)) is the Gaussian kernel function in pixel domain; When configuring bilateral filtering parameters based on mine environment parameters, it at least includes configuring the size of the neighborhood window Ω, the spatial domain Gaussian kernel function F based on the mine environment parameters. spatial (i, j), pixel domain Gaussian kernel function F rang (I(x,y),I(x+i,y+j)) is the width of the corresponding kernel function.

Images