Low-illumination water-floating garbage image enhancement method based on illumination intensity diffusion model

Abstract

The invention provides a low-illumination water-floating garbage image enhancement method based on an illumination intensity diffusion model, and the method comprises the steps: inputting an obtained water-floating garbage image into a light source illumination intensity diffusion model, and obtaining a light source illumination compensation image; extracting texture and features of the light source illumination compensation image by using an edge feature fusion depth curve estimation network to obtain a feature image; and when the light enhancement curve is used for estimating a low-illumination water floating garbage enhancement curve of the feature image, atmospheric light estimation is introduced to correct the curve, so that an enhanced image is obtained. The purpose of improving the recognition accuracy of the floating garbage is achieved.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and in particular relates to a low-light floating garbage image enhancement method based on a light intensity diffusion model. Background Art

[0002] The pollution caused by floating garbage seriously threatens the ecological environment and poses severe challenges to the water quality monitoring and cleaning work in the basin. Due to the diverse types and complex shapes of floating garbage, accurate identification and monitoring of these garbage has become a key task in management. However, in complex scenes such as cloudy days, haze, and dusk, the images of floating garbage obtained have problems such as low light, noise interference, loss of details, and low contrast, which seriously affect the accuracy of subsequent processing such as identification, tracking, and early warning of floating garbage. Therefore, how to effectively enhance the images of floating garbage under low light conditions and improve the efficiency and accuracy of water environment management is of great significance.

[0003] In early studies, researchers mostly used traditional image enhancement methods to enhance low-light images from the spatial and frequency domains of the image, such as histogram equalization, gamma transform, Retinex theory, and wavelet transform. Although these methods improve the brightness and contrast of the image to a certain extent, they are accompanied by noise amplification and detail loss. In addition, due to the diversity and complexity of images of floating garbage in the Yellow River, traditional enhancement methods are difficult to meet the needs of its diverse scenarios.

[0004] With the widespread application of deep learning technology in the field of computer vision, low-light image enhancement algorithms based on deep learning have gradually emerged. By constructing a deep neural network model and using large-scale data sets for training, the algorithm can learn the complex mapping relationship between low-light and normal-light images, capture the illumination change law and detail features in the image, and effectively suppress noise, retain details and restore natural colors while enhancing image brightness and contrast. Low-light image enhancement algorithms based on deep learning are divided into supervised and unsupervised low-light image enhancement algorithms based on implementation methods. The supervised method requires the collection of paired low-light and normal-light images in advance to establish a mapping relationship between normal images and low-light images. LLNet (low-light image enhancement network) uses a variant of the stacked sparse denoising autoencoder to simultaneously brighten and denoise low-light images. Lv et al. proposed an end-to-end multi-branch enhancement network (MBLLEN), which suppresses noise interference in low-light images while improving brightness and contrast and maintaining natural colors through multi-layer feature extraction and multi-branch feature fusion network design. RetinexNet is based on the Retinex theory. It estimates illumination and reflectivity through a deep convolutional network and compensates for illumination components, thereby improving the details and brightness of the image. However, the decomposition illumination model of RetinexNet is too simple, and the enhanced image is prone to noise amplification and color aberration problems. Zhu et al. proposed an edge-enhanced multi-exposure fusion network (EEMEFN), which uses a two-stage low-light enhancement strategy of multi-exposure fusion network (MEF) and edge enhancement (EE) to make the target edges in the image clearer and sharper, thereby improving the overall clarity of the image. In addition, low-light images can be enhanced by introducing Gaussian pyramid networks, residual networks, and Laplace transforms. These methods learn integrated feature representations through end-to-end network structures and achieve good performance, but they show poor generalization in real low-light environments. In most scenarios, paired data cannot be obtained, which leads to the limitations of supervised methods in these scenarios.

[0005] Compared with supervised methods, unsupervised methods do not require paired data sets, which greatly reduces the requirements for data production and improves the computational efficiency of the model. Unsupervised methods are divided into methods based on generative adversarial networks (GANs) and deep convolutions from the perspective of network structure. Generative adversarial networks consist of two parts: a generator and a discriminator. The goal of the generator is to convert low-light images into normal-light images, while the goal of the discriminator is to distinguish whether the generated image is a real normal-light image or an image generated by the generator. During the training process, the generator and the discriminator compete with each other and continuously optimize, so that the generator can finally generate high-quality enhanced images. The literature proposes CycleGAN (Cycle-Consistent Generative Adversarial Networks), which performs well in image conversion, but in terms of low-light enhancement, the details and textures of the generated images become blurred, color deviation and distortion occur, and the enhanced image effect is not natural enough. In order to solve the above problems, the literature introduced a layer decomposition network and a light effect suppression network, and used GAN to train the encoder, but the enhancement effect of the GAN-based method was unstable and the generalization performance was poor. The EnlightenGAN (Enhancing Low-Light GenerativeAdversarial Network) proposed in the literature creatively introduced a global-local discriminator, which greatly improved the generalization performance compared with the previous adversarial network method. However, when processing images with complex textures and structures, especially when there are overexposed or underexposed images such as flash and multiple light sources in the picture, EnlightenGAN cannot accurately restore the natural image effect.

[0006] In general, generative adversarial networks require the design of complex generators and discriminators, require a lot of computing resources, and are sensitive to hyperparameters, which leads to certain limitations and challenges in generative adversarial networks. Compared with GAN, methods based on deep convolution are easier to train in some specific scenarios, are less sensitive to hyperparameters, and have higher generalization capabilities, and have gradually become the mainstream of low-light image enhancement. Li et al. proposed a progressive recursive image enhancement network (PRIEN) for improving the quality of low-light images. The network designed a recursive unit consisting of a recursive layer and a residual block to process the input image, and combined with a dual attention module for global feature extraction and local feature extraction, but the image enhanced by this method is prone to overexposure. Zhou et al. proposed a lightweight low-light image enhancement network SurroundNet, which contains multiple adaptive retinex blocks (ARBlock) and a low-exposure denoiser (LED) to solve the contradiction between model complexity and image enhancement performance. This method can achieve high-performance low-light image enhancement with only a few parameters, but there is a deviation in the calculation of color information, resulting in color distortion in the enhanced image. Li et al. proposed a lightweight luminance-aware pyramid network (LPNet), which combines multi-scale contrast feature blocks (MSCFB) and brightness perception mechanism to reconstruct high-quality images in a coarse-to-fine manner, solving the problem of overexposure, but this method will cause the enhanced image to lose edge detail information. Guo et al. proposed a new zero-reference deep curve estimation (Zero-DCE), which uses a lightweight deep network DCENet to adjust the dynamic range of a given image, and does not require the preparation of paired datasets in advance. However, this method does not consider the influence of noise, and the restored image lacks contour information. Li et al. proposed an accelerated and lightweight version of ZeroDCE, Zero DCE++, which has a faster processing speed while maintaining the enhancement performance of Zero DCE. SCI (Self-Calibrated Illumination) uses a cascade fusion network to search for a network structure suitable for low-light enhancement, but because it uses Retinex theory as prior knowledge, although relatively good results are achieved, it is easy to introduce noise in the enhanced image. Summary of the invention

[0007] In view of the problems existing in the prior art, the present invention provides a low-light floating garbage image enhancement method based on a light intensity diffusion model, which at least partially solves the problem of inaccurate floating garbage identification in the prior art.

[0008] The embodiment of the present invention provides a method for enhancing a low-light floating garbage image based on a light intensity diffusion model, comprising:

[0009] The acquired floating garbage image is input into the diffusion model of light source illumination intensity to obtain a light source illumination compensation image;

[0010] The texture and features of the light source illumination compensation image are extracted using the deep curve estimation network with edge feature fusion to obtain a feature image;

[0011] When using the light enhancement curve to estimate the low-light floating garbage enhancement curve of the feature image, atmospheric light estimation is introduced to correct the curve to obtain the enhanced image.

[0012] Optionally, the construction of the diffusion model of the light source illumination intensity includes:

[0013] The atmospheric light intensity of the floating garbage images at different low light levels was calculated using atmospheric light estimation.

[0014] A light source illumination intensity model is constructed based on the calculated atmospheric illumination intensity of different low-light floating garbage images;

[0015] Construct a diffusion model of light source intensity based on the light source intensity model;

[0016] The diffusion model of light intensity is trained using training samples generated with different light intensities and light source positions.

[0017] Optionally, the edge feature fusion deep curve estimation network realizes feature extraction and fusion through a deep convolutional network.

[0018] Optionally, the edge feature fused deep curve estimation network includes an image texture contour feature extraction network DCENet and an edge detail feature extraction network WTCNet.

[0019] Optionally, the image texture contour feature extraction network DCENet includes symmetrically connected CNN convolutions, and uses a ReLU activation function and a jump connection to splice and fuse features extracted from different convolutional layers.

[0020] Optionally, the edge detail feature extraction network WTCNet first uses Conv convolution to integrate texture and detail features, and then uses WTConv to implement texture and contour feature extraction from shallow to deep.

[0021] Optionally, WTConv first transforms the input features through the first layer of db1 wavelet transform, divides the input features into a low-frequency component and three high-frequency components, and then transforms the low-frequency components using the second layer of db1 wavelet transform to complete the feature extraction of the low-frequency components. After the three high-frequency components are extracted through convolution, they are rescaled and the high-frequency components and low-frequency components after feature extraction are inversely transformed by wavelet transform; finally, the features after wavelet transform are fused with the input features through convolution, and then jump-connected to achieve feature fusion of different scales.

[0022] Optionally, the iAFF attention mechanism is introduced to realize the feature fusion of different levels of the image texture contour feature extraction network DCENet and the edge detail feature extraction network WTCNet.

[0023] Optionally, the iAFF attention mechanism includes:

[0024] Calculate the weights of the two input features, and use the obtained weights to perform weighted fusion on the two input features;

[0025] Then the attention weight of the second layer feature map of the fused features is calculated, and the attention weight is used to realize the feature fusion of different levels of the image texture contour feature extraction network DCENet and the edge detail feature extraction network WTCNet.

[0026] Optionally, when using the light enhancement curve to estimate the low-light floating garbage enhancement curve of the feature image, atmospheric light estimation is introduced to correct the curve to obtain an enhanced image, and the atmospheric light estimation is an atmospheric light estimation value calculated by a dark channel defogging algorithm.

[0027] The low-light floating garbage image enhancement method based on the light intensity diffusion model provided by the present invention increases the generalization ability of the model for low-light floating garbage image enhancement in different river scenes through the diffusion model of the light source light intensity, avoids the overexposure phenomenon caused by the large light intensity of the enhanced image through the deep curve estimation network of edge feature fusion, fine-tunes the exposure intensity by fusing the light enhancement curve of atmospheric light estimation, and restores the enhanced image that is closer to normal light by calculating the light intensity at different positions, thereby achieving the purpose of improving the accuracy of floating garbage recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The above and other objects, features and advantages of the present invention will become more apparent through a more detailed description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present invention.

[0029] Figure 1 A diagram of the LightDiffu-DCE network structure diagram disclosed in an embodiment of the present invention;

[0030] Figure 2 The WTCNet network structure diagram disclosed in the embodiment of the present invention;

[0031] Figure 3 The iAFF attention and MS-CAM structure diagrams disclosed in the embodiments of the present invention, (a) is iAFF, (b) is MS-CAM;

[0032] Figure 4 The visualized ablation experiment diagram on the floating garbage dataset disclosed in the embodiment of the present invention, (a) is the original image, (b) is Zero DCE+M1, (c) is Zero DCE+M2, and (d) is LightDiffu-DCE;

[0033] Figure 5 This is a diagram of the enhanced effect of the LightDiffu-DCE method disclosed in an embodiment of the present invention, (a) is the original image, and (b) is the image processed by the LightDiffu-DCE method. DETAILED DESCRIPTION

[0034] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0035] It should be clear that the following embodiments of the present invention are described by specific specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present invention.

[0036] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. It should be understood by those skilled in the art based on the invention that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0037] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention. The illustrations only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0038] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the aspects described may be practiced without these specific details.

[0039] This embodiment discloses a method for enhancing a low-light floating garbage image based on a light intensity diffusion model, including:

[0040] The acquired floating garbage image is input into the diffusion model of light source illumination intensity to obtain a light source illumination compensation image;

[0041] The texture and features of the light source illumination compensation image are extracted using the deep curve estimation network with edge feature fusion to obtain a feature image;

[0042] When using the light enhancement curve to estimate the low-light floating garbage enhancement curve of the feature image, atmospheric light estimation is introduced to correct the curve to obtain the enhanced image.

[0043] Optionally, the construction of the diffusion model of the light source illumination intensity includes:

[0044] The atmospheric light intensity of the floating garbage images at different low light levels was calculated using atmospheric light estimation.

[0045] A light source illumination intensity model is constructed based on the calculated atmospheric illumination intensity of different low-light floating garbage images;

[0046] Construct a diffusion model of light source intensity based on the light source intensity model;

[0047] The diffusion model of light intensity is trained using training samples generated with different light intensities and light source positions.

[0048] Optionally, the edge feature fusion deep curve estimation network realizes feature extraction and fusion through a deep convolutional network.

[0049] Optionally, the edge feature fused deep curve estimation network includes an image texture contour feature extraction network DCENet and an edge detail feature extraction network WTCNet.

[0050] Optionally, the image texture contour feature extraction network DCENet includes symmetrically connected CNN convolutions, and uses a ReLU activation function and a jump connection to splice and fuse features extracted from different convolutional layers.

[0051] Optionally, the edge detail feature extraction network WTCNet first uses Conv convolution to integrate texture and detail features, and then uses WTConv to implement texture and contour feature extraction from shallow to deep.

[0052] Optionally, WTConv first transforms the input features through the first layer of db1 wavelet transform, divides the input features into a low-frequency component and three high-frequency components, and then transforms the low-frequency components using the second layer of db1 wavelet transform to complete the feature extraction of the low-frequency components. After the three high-frequency components are extracted through convolution, they are rescaled and the high-frequency components and low-frequency components after feature extraction are inversely transformed by wavelet transform; finally, the features after wavelet transform are fused with the input features through convolution, and then jump-connected to achieve feature fusion of different scales.

[0053] Optionally, the iAFF attention mechanism is introduced to realize the feature fusion of different levels of the image texture contour feature extraction network DCENet and the edge detail feature extraction network WTCNet.

[0054] Optionally, the iAFF attention mechanism includes:

[0055] Calculate the weights of the two input features, and use the obtained weights to perform weighted fusion on the two input features;

[0056] Then the attention weight of the second layer feature map of the fused features is calculated, and the attention weight is used to realize the feature fusion of different levels of the image texture contour feature extraction network DCENet and the edge detail feature extraction network WTCNet.

[0057] Optionally, when using the light enhancement curve to estimate the low-light floating garbage enhancement curve of the feature image, atmospheric light estimation is introduced to correct the curve to obtain an enhanced image, and the atmospheric light estimation is an atmospheric light estimation value calculated by a dark channel defogging algorithm.

[0058] This embodiment aims at how to improve the generalization ability of the model in different low-light scenes; how to enhance the detail contour information that is almost invisible under low-light conditions while enhancing the overall brightness of the image; how to control the brightness intensity so that the enhanced image does not lose color, etc. A low-light floating garbage image enhancement method LightDiffu-DCE (Light Diffusion based Zero-DCE Image Enhancement Algorithm) based on the illumination intensity diffusion model is proposed to improve the visual quality and recognition performance of the Yellow River floating garbage image. Starting from the actual needs of low-light enhancement of floating garbage images, in the training stage, in order to increase the generalization ability of the model, according to the characteristics of different illumination intensities caused by the position of the light source in the low-light image, a PM model based on illumination intensity modeling LIPM (Probabilistic models based on illumination intensity and light source modeling) is designed to generate training data sets with different illumination intensities, so that the model can obtain more mapping relationships between low-light and normal-light images. Secondly, taking advantage of the fact that discrete wavelet transform can extract contour features from images, a parallel edge feature fusion deep curve estimation network is designed to extract details and contour features of low-light floating garbage images, and an attention mechanism is used to weightedly fuse the extracted details and contour features. Finally, in order to avoid overexposure due to high light intensity in the enhanced image, a light enhancement curve fused with atmospheric light estimation is proposed to fine-tune the exposure intensity. By calculating the light intensity at different locations, an enhanced image that is closer to normal light is restored. LightDiffu-DCE enhances the overall brightness and quality of low-light floating garbage images, and also refines the overall contour information of the image. The model trained by LIPM diffusion increases the generalization ability of the model for enhancing low-light floating garbage images in different river scenes.

[0059] The overall framework of the low-light floating garbage image enhancement algorithm LightDiffu-DCE based on contour enhancement constructed in this embodiment is as follows Figure 1As shown in the figure, the atmospheric light intensity of different low-light water-washed garbage images is calculated by using atmospheric light estimation and a light source light intensity model is constructed. A diffusion model based on light source light intensity is designed, and training samples are generated according to different light intensities and light source positions to improve the generalization ability of the model in different low-light water-washed garbage image enhancement scenarios. Then, an edge detail feature extraction network WTCNet (Wavelet Transform Convolution Network) is designed to extract richer texture and detail features to improve the overall image quality. The iAFF attention mechanism (Iterative Attention Feature Fusion) is used to fuse the different scale features extracted by the WTCNet and DCENet parallel networks to improve the estimation accuracy of the subsequent light enhancement curve and restore the low-light water-washed garbage enhanced image that is closer to the real scene. Finally, when the light enhancement curve (LE-curve) estimates the low-light water-washed garbage enhancement curve, the atmospheric light estimation is introduced to correct and fine-tune the curve to avoid overexposure, so that the enhanced image details and texture features are closer to the image in the real scene.

[0060] Diffusion image generation with different light intensities based on LIPM:

[0061] Due to the limitation of specific scenes, it is impossible to continuously collect the same image under different exposure states. Therefore, in order to better realize the enhancement of low-light floating garbage images, this embodiment uses the PM model LIPM (Probabilistic models based on illumination intensity and light source modeling) based on light source illumination intensity modeling to generate a training data set to improve the generalization ability of the model. This section will start with the diffusion model, then analyze the advantages and disadvantages of the nonlinear diffusion PM algorithm, and finally introduce the implementation method and calculation details of the PM model based on light source illumination intensity modeling proposed in this embodiment. The specific introduction is as follows:

[0062] Diffusion Model:

[0063] The purpose of the diffusion model is to learn a Markov chain that gradually transforms the distribution of Gaussian noise into training data. It is a generative model based on probability theory, and its inspiration originally came from the diffusion process in physics, such as the diffusion of ink in water. In the field of machine learning, this concept has been creatively applied to data generation tasks, especially the synthesis of images, videos, and audio. The diffusion model reconstructs high-quality data samples from noise by simulating a gradual "diffusion" process from data distribution to simple noise distribution and learning the inverse process.

[0064] The core idea of ​​the diffusion model includes two processes: forward diffusion and reverse diffusion. Forward diffusion defines a process of gradually transforming from data distribution to Gaussian noise distribution. By learning to move forward from pure data to noise distribution, the model can deeply understand and capture the inherent structure of the data. Reverse diffusion is a dual of forward diffusion. It is based on the output adjustment of the previous step to gradually reduce the noise while retaining and enhancing the main features. The process of adding noise forward is expressed as:

[0065] (1)

[0066] in, for Input image data at step time, Input image data for the previous time step, It is a coefficient that decreases over time and controls the amount of noise added. It means the mean , the covariance matrix is Gaussian distribution, is the scaling of the input image data at the previous time step, is the noise variance, is the unit matrix, indicating that the noise is independent in each dimension.

[0067] Formula (1) shows that in the time step Under the control of It is through according to Scaling and adding a variance of Gaussian noise calculation, with time step As the time step increases, the noise interference becomes larger and larger. When it is almost completely turned into noise.

[0068] Reverse diffusion is the opposite of the forward diffusion process. It gradually recovers the original data samples from pure noise or highly noisy data. In the image generation task, the reverse diffusion process gradually removes noise and finally generates a clear image. The reverse diffusion process can be regarded as the inverse operation of the forward diffusion process. In the forward diffusion process, the data samples are gradually covered by noise until they eventually become pure noise. The reverse diffusion process starts from such pure noise or highly noisy data, and gradually removes noise through a series of operations to recover the original data samples. This process is expressed as:

[0069] (2)

[0070] in, and represents the mean and variance parameters of the neural network prediction, which depend on the current noise image and time step The inverse denoising process is usually performed in an iterative manner, where each step updates the image based on the current noise image and the predicted noise. The reverse diffusion process can learn and simulate the complex nonlinear mapping of reverse diffusion. Therefore, in the diffusion model, the goal of training the diffusion model is to minimize the loss function to minimize the reverse error in the diffusion model.

[0071] Nonlinear diffusion PM algorithm:

[0072] The PM model is an image enhancement technology based on partial differential equations. Its basic idea is to achieve denoising by simulating the diffusion process in the image while maintaining the edge information of the image as much as possible. In a two-dimensional image, the model performs nonlinear diffusion on the pixel values ​​according to the gradient information of the image to achieve the purpose of denoising. It has the advantages of retaining the foreground of the image and smoothing the background content. PM integrates the modulus information of the gradient into the transmission coefficient, which is expressed as:

[0073] (3)

[0074] in: is the image pixel value at the spatial position and time The function of represents the rate of change of image pixel value over time, For images The gradient of Represents the modulus value at the spatial position: , is the divergence operator, which is used to calculate the divergence of the vector field. is the diffusion coefficient, which determines the diffusion rate at different gradient values. is the gradient threshold, which can be controlled The diffusion rate of is the common diffusion coefficient expression:

[0075] (4)

[0076] In the field of image processing, especially low-light image enhancement, the Denoising Diffusion Probabilistic Models (DDPM) can gradually remove noise and restore high-quality images by simulating the reverse diffusion process of data generation. This method can not only generate diverse training samples and enhance the generalization ability of the model, but also flexibly adjust to images of different exposures in the reverse process, effectively retaining image details and improving their quality. Through precise noise modeling and gradual image detail recovery, the DDPM algorithm can generate low-light training data sets of different exposures.

[0077] LIPM model based on light source intensity modeling:

[0078] In order to make the diffusion model PM more suitable for low-light floating garbage image enhancement tasks, the influence of different light intensities on the diffusion model is established based on the different atmospheric light estimates of low-light images. Secondly, low-light images are affected by uneven lighting. Therefore, according to the uneven lighting intensity caused by different positions of the light source, the light source intensity is designed to compensate for the lighting changes caused by changes in external conditions (such as time or weather changes).

[0079] According to the classical atmospheric scattering model, the image pixel intensity It can be represented by the real light intensity in the low-light floating garbage image scene. With atmospheric light The combination is expressed as:

[0080] (5)

[0081] in, For image pixels The strength of the position, is the actual scene brightness (without the influence of atmospheric scattering), is the intensity of atmospheric light, usually taken as the color of the brightest area in the image (i.e. the light source). is the transmittance, which indicates the proportion of light in the scene that is not scattered before reaching the observer. and transmittance Calculated by the dark channel dehazing algorithm.

[0082] In low-light floating garbage scenes, the image brightness is easily affected by the light source. Therefore, if there are one or more light sources in the scene, each light source is considered to produce a certain intensity of light at its position, and the position of the light source in the image is , the impact on the surrounding area can be simulated by Gaussian distribution:

[0083] (6)

[0084] in, Control the range of light source influence. Therefore, formula (3) is modified as

[0085] (7)

[0086] in, is the light intensity-dependent diffusion coefficient:

[0087] (8)

[0088] is the transmittance adjustment coefficient, and its value is [0-1].

[0089] Deep Curve Estimation Network with Edge Feature Fusion:

[0090] The deep curve estimation network of edge feature fusion is as follows Figure 2 As shown in the figure, a deep convolutional network is designed to achieve feature extraction and fusion. The network consists of DCENet and WTCNet. DCENet uses 6 ordinary CNN convolutions with symmetrical connections, a convolution kernel size of 3×3, a step size of 1, and uses ReLU activation function and jump connection to achieve splicing and fusion of features extracted from different convolutional layers, extracting richer features.

[0091] In order to highlight the contour features of the enhanced image and refine the texture details of the low-light floating garbage image, WTCNet was designed inspired by WTConv. The network first uses a layer of 3×3Conv convolution to integrate texture and detail features, and then uses three layers of WTConv to realize texture and contour feature extraction from shallow to deep. In order to better estimate the low-light floating garbage features and then calculate a more accurate light enhancement curve, this embodiment introduces iAFF (Iterative Attentional Feature Fusion) attention to deeply fuse different features, realize feature complementarity, and extract more contour features and texture features, so that the texture of the enhanced low-light floating garbage image is clearer and the details are finer.

[0092] like Figure 2As shown in the figure, WTConv first transforms the input features through the first layer of db1 wavelet transform, and divides the input features into a low-frequency component LL and three high-frequency components (HL HH LH). Then, the low-frequency components are transformed by the second layer of db1 wavelet transform to complete the feature extraction of the low-frequency components. After the three high-frequency components are extracted by convolution, the scales are readjusted, and the high-frequency components and low-frequency components after feature extraction are subjected to inverse wavelet transform. Finally, the features after wavelet transform are fused with the input features through convolution, and then jump connected to achieve feature fusion of different scales. Such a network design not only makes full use of the decomposition and fusion of multi-scale feature information of wavelet transform and highlights the edge contour features of the entire image, but also maximizes the network's receptive field and feature extraction capabilities while consuming very little computation.

[0093] In order to better integrate the features of different levels of DCENet and WTCNet networks, the idea of ​​"weighted" and "gated" fusion is adopted, and the iAFF attention mechanism is introduced to realize the feature fusion of different levels of the two networks. The structure of the iAFF attention mechanism is as follows: Figure 3 As shown in the figure. iAFF attention requires that the dimensions of the two input features are consistent. First, the two input features are subjected to MS-CAM (Multi-scale Channel Attention Module) to calculate the weights of the two inputs, and then the two input features are weighted fused. The fused features are then passed through the second-layer MS-CAM module to calculate the attention weights of the second-layer feature map. After multiple iterative fusions, the final fused feature maps of different scales are obtained. This attention can effectively integrate features of different scales and semantics, interact with multiple layers of features, and have richer features.

[0094] In the iAFF attention mechanism, the output feature is calculated as:

[0095] (9)

[0096] in, is the output feature after the iAFF module, and are two input features of the same dimension, In order to calculate the weights of each input feature through the MS-CAM module, it can be obtained from the formula that iAFF attention achieves better weighted fusion by calculating the fusion weights of two input features.

[0097] Light enhancement curve integrating atmospheric light estimation:

[0098] Zero-DCE uses the light enhancement curve (LE-curve) to dynamically adjust the intensity of low-light water floating garbage enhancement. The entire low-light enhancement task is regarded as adaptive adjustment through a quadratic curve, and the whole process is described as:

[0099] (10)

[0100] in is the enhanced pixel point, is a trainable curve parameter that adjusts the size of the light enhancement curve and controls the exposure level. In addition, LE-curve has designed a three-channel light enhancement curve solution and a high-order iterative curve solution method, so that the Zero-DCE algorithm can better retain the inherent color and reduce the risk of oversaturation. The high-order enhancement curve solution is:

[0101] (11)

[0102] in, is the number of iterative training of the model, For a given input An enhanced version of , with trainable curve parameters to adjust the size of the LE curve and control its exposure level.

[0103] However, in low-light floating garbage scenes, low light is also affected by the intensity of different light sources. The uneven intensity of the light makes the enhanced low-light image unnatural and causes overexposure in some areas. Therefore, when solving the LE curve, atmospheric light is added. The curve is corrected to reduce the risk of overexposure and achieve more realistic low-light image enhancement. The light enhancement curve fused with atmospheric light estimation is described as:

[0104] (12)

[0105] in, is the intensity parameter of atmospheric light estimation, is the atmospheric light estimate calculated by the dark channel dehazing algorithm.

[0106] Experiment and result analysis:

[0107] Dataset Introduction:

[0108] In order to verify the enhancement effect of the method of this embodiment on the low-light floating garbage image, data collection was carried out in the Lanzhou section of the Yellow River. The Fable Story Park on Beibinhe Road, Anning District, Lanzhou City (latitude: 36.086220117901036, longitude 103.74410033226013) was selected to collect video data using fixed-angle cameras and drones. In order to better simulate low-light images, data was collected continuously starting in the afternoon, and the collection time lasted for about a month.

[0109] The collected data is extracted according to a certain frame interval. The entire monitoring data is cleaned, and the blurred and unclear images caused by camera shake and dark light are deleted to screen out clear and targeted images. Then, the screened images are adjusted by filling and cropping to ensure that the size of each image is 640×640 pixels, and a low-light floating garbage image dataset is obtained. This dataset has a total of 3,350 images, including low-light images of different light sources, different environments, different time periods, and different targets, with significant characteristics and certain representativeness.

[0110] Experimental environment and evaluation indicators:

[0111] The experiment uses the deep learning framework Pytorch 1.12.1, Python 3.7.6, CUDA 10.0, the hardware is Intel (R) Xeon (R) CPU E5-2678 v3, and the GPU is NVIDIA GeForce RTX 3060 for single-card training. During the training process, the learning rate is set to 0.0001, the epoch is 100, and the batch size is 8. In the training process of the two data sets, all reference graphs are used, and pre-trained weights are not introduced. The network is retrained to achieve the best effect.

[0112] In order to verify the generalization ability of the algorithm proposed in this embodiment, since it is impossible to collect floating garbage image pairs under normal lighting and low lighting conditions, no-reference evaluation indicators are used on this dataset to evaluate the enhanced image from three aspects: the overall quality (naturalness) of the enhanced image, the degree of noise interference, and the brightness and contrast of the image. NIQE (Naturalness Image Quality Evaluator), PIQE (Perception based Image Quality Evaluator) and RISQ (Referenceless Image Spatial Quality Evaluator) are selected as the evaluation indicators of this dataset. The functions and introductions of the specific evaluation indicators are as follows:

[0113] NIQE: reflects the natural preservation ability of the image. The smaller the value, the better the brightness order of the enhanced image and the more natural the effect.

[0114] PIQE: PIQE uses the block structure and noise characteristics of the image to calculate the image quality score, and comprehensively evaluates it from the aspects of contrast sensitivity, brightness adaptation, color vision, etc. The lower the value, the higher the image quality.

[0115] RISQ: RISQ uses support vector machines to learn the mapping relationship between image quality and image features. It can reflect the contrast changes in local areas of the image and calculate whether the enhanced image is distorted. The lower the value, the higher the image quality.

[0116] Comparative ablation experiment analysis:

[0117] Ablation experiment analysis:

[0118] Ablation experiments were conducted on two public datasets to verify the effectiveness and rationality of the different improvement strategies proposed in this embodiment. In this embodiment, the LIPM diffusion data generation method is denoted as M1, the edge feature fusion depth curve estimation network is denoted as M2, and the light enhancement curve method fused with atmospheric light estimation is denoted as M3. The ablation experiments on the low-light floating garbage dataset are shown in Tables 1 and Figure 4 shown.

[0119] Table 1 Ablation experiments on the floating garbage dataset

[0120] .

[0122] From Table 1 and Figure 4 It can be seen that the use of various optimization and improvement strategies has certain rationality and superiority. The training data under different light intensities generated by the LIPM diffusion data generation method increases the generalization and low-light enhancement capabilities of the model; the deep curve estimation network using edge feature fusion extracts more texture detail features, and the enhanced image contour texture is more delicate; the light enhancement curve method integrating atmospheric light estimation fine-tunes the high-exposure image, making the enhanced image closer to the image under normal light. In summary, compared with the baseline ZERO-DCE method, the LightDiffu-DCE method proposed in this embodiment has certain advantages on the floating garbage dataset, and its NIQE, PIQE and RISQ indicators are increased by 0.88, 0.78 and 2.6, respectively, and increased by about 8.35%, 6.20% and 21.83%, respectively, with good enhancement effects.

[0123] Enhanced result graph display:

[0124] The enhanced results of the LightDiffu-DCE method are shown in Figure 5 As shown in the figure, it can be seen that the method in this paper can achieve low-light floating garbage image enhancement. The enhanced image has more obvious details, higher overall brightness and contrast than the original image, and the enhancement effect is obvious.

[0125] In summary, in order to better improve the brightness of low-light images, while finely preserving the details and contour information of natural colors and light distribution, and avoiding overexposure, this embodiment proposes a low-light image enhancement LightDiffu-DCE model based on the light source illumination intensity diffusion model. According to the illumination intensity and characteristics of different light sources in low light, a diffusion model LIPM based on illumination intensity modeling is designed to generate training data sets with different illumination intensities, thereby improving the generalization ability of the model; secondly, in order to restore the overall contour and detail information of the image, a deep curve estimation network with edge feature fusion is designed to extract richer multi-scale contours and detail features of low-light images, avoid noise amplification, and improve the accuracy of subsequent illumination intensity estimation; finally, the atmospheric light estimation is integrated to fine-tune the image to avoid overexposure in the restored image, and by calculating the illumination intensity of different areas, an image with more natural illumination is restored. Three no-reference evaluation indicators are used on the low-light floating garbage dataset. The experiment shows the effectiveness of this method in various improvement optimization strategies. Compared with the baseline algorithm, the LightDiffu-DCE algorithm proposed in this embodiment improves NIQE, PIQE and RISQ indicators by 8.35%, 6.20% and 21.83% respectively. This shows that the algorithm in this embodiment has a better enhancement effect, the enhanced image quality is higher, the contour texture information is richer, and the enhanced effect is more natural. In the future, we will explore the fusion of more complex features when enhancing in low light to obtain enhanced images with higher quality.

[0126] The basic principle of the present invention is described above in conjunction with specific embodiments. However, it should be pointed out that the advantages, strengths, effects, etc. mentioned in the present invention are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present invention. In addition, the specific details disclosed above are only for the purpose of illustration and facilitation of understanding, rather than limitation, and the above details do not limit the present invention to being implemented by adopting the above specific details.

[0127] In the present invention, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Words such as "including", "comprising", "having", etc. are open words, meaning "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to", and can be used interchangeably with it.

[0128] Additionally, as used herein, "or" used in a list of items beginning with "at least one" indicates a separate list, so that, for example, a list of "at least one of A, B, or C" means A or B or C, or AB or AC or BC, or ABC (i.e., A and B and C). Furthermore, the word "exemplary" does not mean that the example described is preferred or better than other examples.

[0129] It should also be noted that in the system and method of the present invention, each component or each step can be decomposed and / or recombined. Such decomposition and / or recombination should be regarded as equivalent solutions of the present invention.

[0130] Various changes, substitutions, and modifications of the techniques described herein may be made without departing from the teachings defined by the appended claims. Furthermore, the scope of the claims of the present invention is not limited to the specific aspects of the processes, machines, manufactures, compositions of events, means, methods, and actions described above. Currently existing or later to be developed processes, machines, manufactures, compositions of events, means, methods, or actions that perform substantially the same functions or achieve substantially the same results as the corresponding aspects described herein may be utilized. Thus, the appended claims include such processes, machines, manufactures, compositions of events, means, methods, or actions within their scope.

[0131] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of the present invention. Therefore, the present invention is not intended to be limited to the aspects shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0132] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present invention to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A low-light floating garbage image enhancement method based on a light intensity diffusion model, characterized in that: include: The acquired floating garbage image is input into the diffusion model of light source illumination intensity to obtain a light source illumination compensation image; The texture and features of the light source illumination compensation image are extracted using the deep curve estimation network with edge feature fusion to obtain a feature image; When using the light enhancement curve to estimate the low-light floating garbage enhancement curve of the feature image, atmospheric light estimation is introduced to correct the curve to obtain the enhanced image.

2. The low-light floating garbage image enhancement method based on the light intensity diffusion model according to claim 1 is characterized in that: The construction of the diffusion model of the light source illumination intensity includes: The atmospheric light intensity of the floating garbage images at different low light levels was calculated using atmospheric light estimation. A light source illumination intensity model is constructed based on the calculated atmospheric illumination intensity of different low-light floating garbage images; Construct a diffusion model of light source intensity based on the light source intensity model; The diffusion model of light intensity is trained using training samples generated with different light intensities and light source positions.

3. The low-light floating garbage image enhancement method based on the light intensity diffusion model according to claim 1 is characterized in that: The edge feature fusion deep curve estimation network realizes feature extraction and fusion through a deep convolutional network.

4. The low-light floating garbage image enhancement method based on the light intensity diffusion model according to claim 3 is characterized in that: The edge feature fused depth curve estimation network includes an image texture contour feature extraction network DCENet and an edge detail feature extraction network WTCNet.

5. The low-light floating garbage image enhancement method based on the light intensity diffusion model according to claim 4 is characterized in that: The image texture contour feature extraction network DCENet includes symmetrically connected CNN convolutions, and uses ReLU activation function and jump connection to splice and fuse the features extracted by different convolutional layers.

6. The low-light floating garbage image enhancement method based on the light intensity diffusion model according to claim 5 is characterized in that: The edge detail feature extraction network WTCNet first uses Conv convolution to integrate texture and detail features, and then uses WTConv to realize texture and contour feature extraction from shallow to deep.

7. The low-light floating garbage image enhancement method based on the light intensity diffusion model according to claim 6 is characterized in that: WTConv first transforms the input features through the first layer of db1 wavelet transform, divides the input features into a low-frequency component and three high-frequency components, then transforms the low-frequency components through the second layer of db1 wavelet transform to complete the feature extraction of the low-frequency components. After the three high-frequency components are extracted through convolution, they are rescaled and the high-frequency components and low-frequency components after feature extraction are inversely transformed by wavelet transform; finally, the features after wavelet transform are fused with the input features through convolution, and then jump-connected to achieve feature fusion of different scales.

8. The low-light floating garbage image enhancement method based on the light intensity diffusion model according to claim 5 is characterized in that: The iAFF attention mechanism is introduced to realize the feature fusion of different levels of the image texture contour feature extraction network DCENet and the edge detail feature extraction network WTCNet.

9. The low-light floating garbage image enhancement method based on the light intensity diffusion model according to claim 8 is characterized in that: The iAFF attention mechanism includes: Calculate the weights of the two input features, and use the obtained weights to perform weighted fusion on the two input features; Then the attention weight of the second layer feature map of the fused features is calculated, and the attention weight is used to realize the feature fusion of different levels of the image texture contour feature extraction network DCENet and the edge detail feature extraction network WTCNet.

10. The low-light floating garbage image enhancement method based on the light intensity diffusion model according to claim 1 is characterized in that: When the light enhancement curve is used to estimate the low-light floating garbage enhancement curve of the feature image, atmospheric light estimation is introduced to correct the curve to obtain an enhanced image, where the atmospheric light estimation is an atmospheric light estimation value calculated by a dark channel defogging algorithm.

Images