Low-illumination image enhancement method and system based on superpixel analysis

Abstract

The present invention discloses a low-light image enhancement method and system based on superpixel analysis. The method classifies images in an existing data set according to their average grayscale values and constructs multiple training sets. A network model for low-light image enhancement is constructed using zero-reference image enhancement technology, and multiple network models are trained based on multiple training sets. The corresponding network model is used to enhance the image for the first time. The enhanced image is segmented and pixel-level labeled, and low-light image enhancement is performed on all superpixel regions in the segmented image. The IoU value between the enhanced re-segmentation result and the pre-enhancement segmentation result is calculated to achieve iterative processing. The RGB value of the iteratively enhanced image is thresholded. If the value exceeds the threshold, histogram equalization is performed, and the segmentation and enhancement iterative processing is repeated until the value meets the threshold. The present invention performs image enhancement on different brightness areas of the same image, effectively solving the problem of overall color deviation in the image.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a low-illumination image enhancement method and system based on superpixel analysis. Background Art

[0002] The statements in this section merely mention background art related to the present invention and do not necessarily constitute prior art.

[0003] Image enhancement and segmentation are two important tasks in image processing and computer vision. Image enhancement aims to improve image quality and detail, making it easier to observe and analyze. Image segmentation, on the other hand, aims to divide an image into several distinct regions to enable applications such as target detection, object recognition, and scene understanding.

[0004] Traditional image enhancement and segmentation methods typically require the use of multiple techniques and algorithms, which are often not efficient or accurate. With the development of deep learning technology, more and more researchers have begun applying deep learning to image enhancement and segmentation tasks. Among them, zero-reference image enhancement technology has attracted widespread attention due to its ability to adaptively learn and extract information from the image itself.

[0005] However, existing zero-reference image enhancement methods still suffer from several issues, such as over-enhancement, distortion, and blurring. Furthermore, many current low-light image enhancement applications suffer from the presence of small amounts of monochromatic light sources, which can lead to overall color deviations in the enhanced image. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the present invention provides a low-light image enhancement method and system based on superpixel analysis. The present invention also proposes a method that combines image segmentation technology with low-light image enhancement technology to further improve the quality and accuracy of image enhancement. In low-light images, the present invention can segment the low-light image to obtain multiple superpixels. These superpixels of different brightness levels are then enhanced separately, forming an iterative system. Ultimately, this effectively addresses the issue of overall color deviation in the enhanced image.

[0007] In a first aspect, the present invention provides a low-light image enhancement method based on superpixel analysis;

[0008] Low-light image enhancement methods based on superpixel analysis include:

[0009] S101: Acquire a low-light image to be processed, select a pre-trained network model of a corresponding level according to the average grayscale value of the low-light image to be processed, and perform a first enhancement process on the image to be processed according to the selected pre-trained network model;

[0010] S102: performing a first image segmentation on the image after the first enhancement process to segment a number of independent objects, each of which corresponds to a superpixel; performing a second enhancement process on the corresponding superpixel using a pre-trained network model of a corresponding level based on the average grayscale value of each superpixel; merging all images after the second enhancement process to obtain a low-light image after the second enhancement process;

[0011] S103: performing a second image segmentation on the low-light image after the second enhancement; calculating the overlap between the second image segmentation result and the first image segmentation result; if the overlap is less than a first set threshold, returning to S101; if the overlap is greater than the first set threshold, proceeding to S104;

[0012] S104: Calculate the overall RGB values of the low-light image after the second enhancement, and compare the RGB values with the second set threshold value respectively. If the values exceed the second set threshold value, perform histogram equalization on the low-light image after the second enhancement, and send the processed image back to S101; if the values are less than the second set threshold value, the process ends and the low-light image after the second enhancement is output as the enhancement result.

[0013] In a second aspect, the present invention provides a low-light image enhancement system based on superpixel analysis;

[0014] The low-light image enhancement system based on superpixel analysis includes:

[0015] an acquisition module configured to: acquire a low-light image to be processed, select a pre-trained network model of a corresponding level according to an average grayscale value of the low-light image to be processed, and perform a first enhancement process on the image to be processed according to the selected pre-trained network model;

[0016] The first processing module is configured to: perform a first image segmentation on the image after the first enhancement process to segment a plurality of independent objects, each independent object corresponding to a superpixel; perform a second enhancement process on the corresponding superpixel using a pre-trained network model of a corresponding level according to the average grayscale value of each superpixel; and merge all the images after the second enhancement process to obtain a low-light image after the second enhancement process;

[0017] The second processing module is configured to: perform a second image segmentation on the low-light image after the second enhancement; calculate the overlap between the second image segmentation result and the first image segmentation result, and return to the acquisition module if the overlap is less than a first set threshold; if the overlap is greater than the first set threshold, enter the third processing module;

[0018] The third processing module is configured to: calculate the RGB value of the entire low-light image after the second enhancement, compare the RGB value with the second set threshold value respectively, if it exceeds the second set threshold value, perform histogram equalization processing on the low-light image after the second enhancement, and send the processed image back to the acquisition module; if it is less than the second set threshold value, end, and output the low-light image after the second enhancement as the enhancement result.

[0019] In a third aspect, the present invention further provides an electronic device, comprising:

[0020] a memory for non-transitory storage of computer-readable instructions; and

[0021] a processor for executing said computer-readable instructions,

[0022] When the computer-readable instructions are executed by the processor, the method described in the first aspect is executed.

[0023] In a fourth aspect, the present invention further provides a storage medium that non-temporarily stores computer-readable instructions, wherein when the non-temporary computer-readable instructions are executed by a computer, the instructions of the method described in the first aspect are executed.

[0024] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, wherein the computer program is used to implement the method described in the first aspect when running on one or more processors.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] This paper proposes an analysis method based on superpixels derived from image segmentation. This segmentation technique automatically divides an image into several superpixels and performs independent zero-reference image enhancement on each region. This method not only improves the quality and detail of image enhancement, but also better preserves the image's structure and features.

[0027] By performing low-light image enhancement after superpixel analysis, this invention provides a new and efficient image enhancement method. This method can improve the quality and efficiency of image enhancement and segmentation in various application scenarios, and can solve the problems of localized overexposure or poor enhancement in enhanced images. Furthermore, the invention optimizes and solves the problem of overall color aberration in enhanced images. This technology can be applied to various image processing and analysis fields, including medical images, remote sensing images, and video images.

[0028] The present invention can reasonably distinguish objects of different brightness in low-light images and use multiple models to perform enhancement processing separately, thereby avoiding the problem of overexposure of areas or objects with excessively high brightness in low-light images after enhancement and avoiding the problem of poor effect of areas or objects with relatively low brightness in low-light images after enhancement.

[0029] The present invention uses image segmentation technology to segment each complete object in the image, and each object corresponds to a superpixel, so as to ensure the rationality of the color of any object in the image after enhancement while enhancing the same low-light image, and avoid excessive differences in enhancement of different areas of the same object.

[0030] The present invention uses a self-built training set production module to specifically classify the existing training set, so that multiple low-light image enhancement models can be trained subsequently to deal with objects of different brightness in the image.

[0031] The image enhancement model based on superpixel analysis proposed in the present invention can be adjusted and optimized according to different scenarios and needs. It has certain universality and adaptability, can meet the needs of different fields and applications, and makes its application more extensive and flexible.

[0032] The present invention adopts a nested approach of two iterative systems, and while performing low-light image enhancement, continuously iterates the process until the best effect is achieved.

[0033] The present invention uses the IoU indicator to judge the degree of overlap between the segmented image and the segmentation and enhancement results, thereby avoiding the problem of being unable to effectively segment different superpixels under low illumination conditions and improving the distinction between different objects in the final enhanced image.

[0034] The present invention avoids color deviation of the entire processed image by limiting the RGB threshold value of the processed image, thereby achieving color rationality of the final enhanced image. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0036] Figure 1 Schematic diagram of the overall process of an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of training set construction and network model training according to an embodiment of the present invention;

[0038] Figure 3 Schematic diagram of an image enhancement network model system according to an embodiment of the present invention;

[0039] Figure 4 Schematic diagram of a neural network in an image enhancement network model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0041] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0042] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0043] Explanation of terms:

[0044] 1. Superpixel: This refers to an image segmentation technique that groups adjacent pixels in an image into larger, more meaningful, compact regions. By clustering pixels into superpixels, it is possible to reduce image detail and extract image regions with greater semantic and structural significance.

[0045] 2. Grayscale: Grayscale refers to the brightness intensity of each pixel in an image, describing the degree of brightness or darkness. In digital images, grayscale values are typically represented as integers between 0 and 255, or floating-point numbers between 0 and 1. Larger grayscale values indicate brighter objects in the image, closer to white; smaller grayscale values indicate darker objects, closer to black.

[0046] 3. Tanh: The hyperbolic tangent function is a type of hyperbolic function. It is often written as tanh in mathematical language. It solves the problem of the sigmoid function's output not being centered around zero.

[0047] 4. ReLU: The full name is Rectified Linear Unit, which is an activation function commonly used in artificial neural networks. In general, it refers to the ramp function in mathematics, that is,

[0048] f(x)=max(0,x).

[0049] 5. U-Net: U-Net is a classic deep learning model used for image segmentation tasks, particularly semantic segmentation. It was proposed by Olaf Ronneberger, Philipp Fischer, and Thomas Brox in 2015. U-Net's network structure features a symmetrical encoder-decoder architecture, resembling the shape of the letter "U," hence the name.

[0050] 6. IoU: Intersection over Union is a commonly used evaluation metric used to measure the degree of overlap between two regions (such as the predicted region and the true region in object detection or image segmentation). The IoU calculation formula is as follows:

[0051] IoU = (Intersection over Union) / (Area of Union)

[0052] The intersection area refers to the area of the intersection of two regions, while the union area refers to the area of the union of the two regions. The IoU value ranges from 0 to 1. Values closer to 1 indicate a higher degree of overlap between the two regions, while values closer to 0 indicate a lower or no overlap between the two regions.

[0053] Example 1

[0054] This embodiment provides a low-light image enhancement method based on superpixel analysis;

[0055] like Figure 1 As shown in FIG, the low-light image enhancement method based on superpixel analysis includes:

[0056] S101: Acquire a low-light image to be processed, select a pre-trained network model of a corresponding level according to the average grayscale value of the low-light image to be processed, and perform a first enhancement process on the image to be processed according to the selected pre-trained network model;

[0057] S102: performing a first image segmentation on the image after the first enhancement process to segment a number of independent objects, each of which corresponds to a superpixel; performing a second enhancement process on the corresponding superpixel using a pre-trained network model of a corresponding level based on the average grayscale value of each superpixel; merging all images after the second enhancement process to obtain a low-light image after the second enhancement process;

[0058] S103: performing a second image segmentation on the low-light image after the second enhancement; calculating the overlap between the second image segmentation result and the first image segmentation result; if the overlap is less than a first set threshold, returning to S101; if the overlap is greater than the first set threshold, proceeding to S104;

[0059] S104: Calculate the overall RGB values of the low-light image after the second enhancement, and compare the RGB values with the second set threshold value respectively. If the values exceed the second set threshold value, perform histogram equalization on the low-light image after the second enhancement, and send the processed image back to S101; if the values are less than the second set threshold value, the process ends and the low-light image after the second enhancement is output as the enhancement result.

[0060] This embodiment is used to enhance superpixels of different brightness in a low-light image separately and prevent overall chromatic aberration in the final image.

[0061] Furthermore, the low-illumination image to be processed, wherein low illumination means that the overall average grayscale value of the image is lower than 85.

[0062] Furthermore, the average grayscale value of the low-light image to be processed, wherein the calculation process of the average grayscale value includes:

[0063] First calculate the RGB value of each pixel of the low-light image to be processed;

[0064] Convert the RGB value of each pixel into the grayscale value of each pixel;

[0065] Calculate the sum of the grayscale values of all pixels;

[0066] The average grayscale value is calculated based on the sum of the grayscale values of all pixels and the number of all pixels.

[0067] It should be understood that the calculation of the average grayscale value of the image is based on multiple existing low-light image datasets, including the LOL dataset, the SICE dataset, and so on. Since the resolution of the images in these datasets is different, it is possible to read the RGB image and obtain the pixel value of the entire image, decode the image into an array, and each element of the array represents a pixel. The corresponding position of the pixel in the array, that is, the row and column, can be accessed to obtain the RGB value of each channel at this position (red, green, and blue channels). After obtaining the RGB value of each pixel, the RGB value of each pixel is converted to a grayscale value according to the following formula:

[0068] Gray=0.299*R+0.587*G+0.114*B

[0069] After calculating the grayscale value of each pixel, calculate the sum of the grayscale values and divide the sum by the number of pixels to get the average grayscale value. This operation is repeated for each image in the dataset to obtain the average grayscale value of each image.

[0070] Furthermore, if Figure 4 As shown, the pre-trained network model has a network structure including:

[0071] The first convolution layer, the first activation function layer, the second convolution layer, the second activation function layer, the third convolution layer, the third activation function layer, the fourth convolution layer, the fourth activation function layer, the fifth convolution layer, the fifth activation function layer, the sixth convolution layer, the sixth activation function layer, the seventh convolution layer, the seventh activation function layer, the eighth convolution layer, the eighth activation function layer, the ninth convolution layer and the ninth activation function layer, connected sequentially;

[0072] The feature map output by the first activation function layer is concatenated and fused with the feature map output by the eighth activation function layer, and then input into the ninth convolutional layer;

[0073] The feature map output by the second activation function layer is concatenated and fused with the feature map output by the seventh activation function layer, and then input into the eighth convolutional layer;

[0074] The feature map output by the third activation function layer and the feature map output by the sixth activation function layer are concatenated and fused in series and then input into the seventh convolutional layer;

[0075] The feature map output by the fourth activation function layer and the feature map output by the fifth activation function layer are concatenated and fused in series and then input into the sixth convolutional layer.

[0076] Furthermore, if Figure 3 As shown, the pre-trained network model, the specific training process includes:

[0077] Constructing a first training set, wherein the first training set includes several sub-training sets, each of which is obtained by classifying all images in the data set according to brightness into several levels, and each level of images constitutes a sub-training set;

[0078] Input a sub-training set of a level into the corresponding network model and train the network model. When the total loss function value of the network model no longer decreases, the pre-trained network model of the current level is obtained.

[0079] Different levels of sub-training sets are used to train pre-trained network models of different levels.

[0080] For example, all images in the existing training set are classified according to brightness into 7 levels ag, and images of each level constitute a new training set, ultimately forming 7 training sets ag;

[0081] For the 7 training sets, low-illumination image enhancement technology was used to construct and train the network model, and 7 different pre-trained network models ag were obtained.

[0082] The existing training set is implemented using the LOL dataset and the SICE dataset.

[0083] Exemplarily, the construction of the training set specifically includes: since most of the images processed by the present invention are low-brightness images, in order to ensure that the multiple models trained subsequently can still be distinguished at low brightness, each image is divided into 7 categories according to the following standards. The average grayscale values of these 7 categories are 0-7, 8-15, 16-31, 32-63, 64-127, and 128-256, respectively. These 7 levels are defined as levels a, b, c, d, e, f, and g, respectively. Finally, the data set is classified into 7 training sets with different brightness - defined as training sets ag in turn.

[0084] Furthermore, all images in the data set are classified according to brightness into several levels, and each level of images constitutes a sub-training set. The specific process includes:

[0085] Convert the RGB value of each pixel in the image to a grayscale value:

[0086] Gray=0.299*R+0.587*G+0.114*B

[0087] After calculating the grayscale value of each pixel, calculate the average grayscale value of the entire image and then calculate the average grayscale value of all images in the existing dataset in turn;

[0088] Multiple levels are constructed according to the average grayscale value, and the images of each level constitute a new sub-training set, ultimately forming multiple sub-training sets.

[0089] Furthermore, performing a first enhancement process on the image to be processed according to the selected pre-trained network model specifically includes:

[0090] The image to be processed is input into the selected pre-trained network model to generate an enhanced image.

[0091] Furthermore, the image to be processed is input into the selected pre-trained network model to generate an enhanced image, and the formula is expressed as:

[0092]

[0093] Among them, E k (X) is the enhanced image, A k (X) is the learnable parameter of the current pixel, E k-1 (X) is the input image, which is the output enhanced image of the previous level in the nested structure.

[0094] Exemplarily, the pre-trained network model uses zero-reference image enhancement technology to fit a brightness mapping curve with a neural network, and then generates an enhanced image based on the brightness mapping curve and the original image. First, the image to be processed is input. Since the parameters of the brightness mapping curve are determined according to the brightness of each pixel, the pixels of different brightness in the input original image are respectively input into the corresponding brightness mapping curve, and the curve maps the RGB value of the brightened pixel. All pixels in the image undergo this operation to obtain an enhanced overall image. The entire calculation process uses the gradient descent method to optimize the neural network. While brightening the image, the optimization process is constrained by using some set error functions, so that the algorithm can ensure that the brightness of the generated image is reasonable and that the image is relatively realistic and close to the original image.

[0095] In the present invention, when designing the brightness mapping curve, it is necessary to ensure that the curve is designed according to the following two standards:

[0096] 1. To avoid the brightness decreasing instead of increasing after image enhancement, the mapping curve must be a monotonically increasing curve.

[0097] 2. Since the brightness value falls in the interval [0,1], in order to ensure that the range of the brightness value remains unchanged, the curve value at 0 should be 0 and the value at 1 should be 1.

[0098] The following formula is used to describe the curve:

[0099]

[0100] Where X is the pixel coordinate, α is a learnable parameter, and E(X) is the enhanced image of the input I(X). In order to make the curve fit higher order, the curve can be nested:

[0101]

[0102] Therefore, the general form of this formula is:

[0103]

[0104] In this embodiment, n can be set to 2 and k to 6, that is, the curve is set to a 3rd order curve, nested 6 times. At the same time, in order to avoid using the same mapping curve for each pixel.

[0105] The final mapping curve function is:

[0106]

[0107] The neural network for fitting the enhancement curve has 9 layers, see Figure 2Because operations are performed on RGB separately, outputs for three channels are obtained. Each layer is a 3x3 convolutional layer of equal length. To maintain connections between adjacent pixels, batch normalization is not used after the convolutional layer. The activation function for layer 8 is ReLU. Since the output falls in the range [-1, 1], the activation function for the output layer is the tanh function. The outputs of layers 4 and 5 are concatenated and fed into layer 6. The outputs of layers 3 and 6 are concatenated and fed into layer 7. The outputs of layers 2 and 7 are concatenated and fed into layer 8. The outputs of layers 1 and 8 are concatenated and fed into the output layer 9.

[0108] Furthermore, the total loss function of the network model is specifically:

[0109] The total error function is the weighted sum of four error functions:

[0110] Ltotal=W1Lspa+W2Lexp+W3Lcol+W4Ltva

[0111] According to actual tests, the four weighted values are modified and set according to the specific training results of the model, and the weights are set to: W1=1, W2=5, W3=5, W5=200.

[0112] Spatial consistency error Lspa: This is used to ensure that the image content does not change after enhancement. More precisely, it prevents the difference between the value of a pixel and its adjacent pixels from changing too much.

[0113]

[0114] Where K is the number of pixels, i is the pixel traversal, Ω(i) is the n-neighborhood of the i-th pixel, and the value of n depends on the resolution of the input image. Y and I are the enhanced image and the input image, respectively.

[0115] Exposure control error Lexp: Avoid overexposure in some areas, underexposure in others, reduce extreme brightness, and make the brightness of each pixel closer to a certain intermediate value. This constraint can be expressed as the following error function:

[0116]

[0117] Here, the constant A describes the median brightness, Y is the average brightness in an n-neighborhood, where n depends on the resolution of the input image, and M is the total number of such neighborhoods.

[0118] Color constancy error Lcol: The value of any one of the RGB channels of the image should not significantly exceed that of other channels. Therefore, there are the following errors:

[0119]

[0120] Among them, (p,q) traverses all pairwise combinations of the three color channels, and Jp represents the grayscale average of color channel p.

[0121] Lighting smoothing error LtvA: In order to maintain the monotonic relationship between adjacent pixels and make the brightness changes between adjacent pixels insignificant, the parameters α∈A between adjacent pixels should be made closer.

[0122]

[0123] Where N is the number of iterations, They are the horizontal and vertical gradient operators respectively. For an image, the horizontal gradient and vertical gradient are the numerical differences between the pixels to the left and above.

[0124] It should be understood that in order to avoid using reference data, four error functions are designed to constrain the enhanced image from different angles. According to the created training set ag, the network model is trained 7 times respectively, and a brightness mapping curve is fitted using the loss function. Finally, 7 model parameters are trained to obtain the network model ag. In order to avoid the overall brightness of the low-light image being low and the brightness difference between different areas being too small, which leads to the inability to effectively segment the image, before segmenting and enhancing the low-light image, the pixel value and RGB value of the image are first read, the average grayscale value of the image is calculated, the level of the average grayscale value ag is determined, and a network model of the same level is selected for image enhancement processing.

[0125] Furthermore, the step S102 of performing a first image segmentation on the image after the first enhancement process to segment a plurality of independent objects, each of which corresponds to a superpixel, includes:

[0126] The trained image segmentation model U-Net is used to perform the first image segmentation on the image after the first enhancement processing, and several independent objects are segmented. Each independent object corresponds to a superpixel.

[0127] It should be understood that the multiple independent objects refer to multiple regions after the image is segmented.

[0128] Furthermore, the training process of the trained image segmentation model U-Net includes:

[0129] Constructing a second training set, wherein the second training set includes original images with known image segmentation results;

[0130] The second training set is input into the image segmentation model U-Net, and the model is trained to obtain a trained image segmentation model U-Net.

[0131] Furthermore, the trained image segmentation model U-Net is used to perform a first image segmentation on the image after the first enhancement process, specifically including:

[0132] When the image is input, it first passes through the encoder, which consists of two 3x3 convolutional layers (ReLU) and a 2x2 maximum pooling layer to form a downsampling module.

[0133] The image after the encoder is then input into the decoder, which consists of an upsampling convolution layer (deconvolution layer), a feature splicing layer, and two 3x3 convolution layers (ReLU). After passing through the decoder, the segmented image is finally output.

[0134] Each independent object segmented from the image corresponds to a superpixel. All superpixels of the segmented image are labeled at the pixel level to obtain the segmentation result.

[0135] Read the segmented superpixels one by one and get the pixel value of each superpixel. Then decode the superpixels into arrays and get the RGB value (red, green, and blue channels) of each pixel. After getting the RGB value of each pixel, convert the RGB value of each pixel to grayscale value according to the following formula:

[0136] G=0.299*R+0.587*G+0.114*B

[0137] After calculating the grayscale value of each pixel, the sum of the grayscale values is calculated, and the sum of the grayscale values is divided by the number of pixels in the superpixel to obtain the average grayscale value of each superpixel.

[0138] The low-light image after preliminary enhancement is input into the image segmentation model, including but not limited to U-Net. After passing through the model, each independent object in the image will be segmented out, corresponding to a superpixel area, and the information of each superpixel in the image will be obtained, and the average grayscale value of each superpixel will be calculated.

[0139] Exemplarily, the step S102 of performing a second enhancement process on the corresponding superpixel using a pre-trained network model of a corresponding level according to the average grayscale value of each superpixel; and merging the images after the second enhancement process to obtain a low-light image after the second enhancement, includes:

[0140] According to the calculated average grayscale value of the object image, according to the 7 grayscale levels, namely 0-7, 8-15, 16-31, 32-63, 64-127, and 128-256, it is determined which category the average grayscale value of the current object image belongs to. According to the category, the corresponding network model in the trained network model ag is used to enhance the current superpixel.

[0141] After all superpixels in the segmented image have been enhanced, the enhanced superpixels are merged to obtain a low-light image after the second enhancement.

[0142] It should be understood that the step S103 of performing a second image segmentation on the low-light image after the second enhancement includes the step of segmenting the enhanced image again to prevent incomplete segmentation from resulting in poor enhancement. The second image segmentation also uses the image segmentation model U-Net.

[0143] Furthermore, the S103: calculating the overlap between the second image segmentation result and the first image segmentation result, if the overlap is less than the first set threshold, then returning to S101, if the overlap is greater than the first set threshold, then entering S104, including:

[0144] Calculate the overlap of the two segmentation results and set the IoU threshold. The IoU threshold is set to 0.98. If the IoU value is less than the threshold, return to S101 until the IoU indicator meets the threshold limit.

[0145] The IoU indicator is used to calculate the overlap between the results of the two image segmentation before and after enhancement, thereby avoiding the problem of poor image segmentation effect under low illumination conditions.

[0146] Furthermore, the step S104: calculating the overall RGB values of the low-light image after the second enhancement, comparing the RGB values with a second set threshold value, and if the values exceed the second set threshold value, performing histogram equalization on the low-light image after the second enhancement, and sending the processed image back to step S101; if the values are less than the second set threshold value, then ending the step and outputting the low-light image after the second enhancement as the enhancement result, specifically includes:

[0147] Add up the R, G, and B values of each pixel in the image and divide the sum by the number of pixels in the image to get the average R, G, and B values of the image. Set the R, G, and B thresholds (0-255) respectively and compare the average R, G, and B values with the set thresholds.

[0148] If it is less than the threshold, no adjustment is required;

[0149] If it is greater than the threshold, the image is processed using the histogram equalization method. In order to prevent the histogram equalization from changing the overall brightness of the image, the image is sent back to S101 for iterative processing until the average R, G, and B values of the final image are less than or equal to the set threshold.

[0150] The present invention provides a low-light image enhancement method based on superpixel analysis. First, images in an existing data set are classified according to their average grayscale values, and multiple training sets are constructed. A network model for low-light image enhancement is constructed using zero-reference image enhancement technology, and multiple network models are trained based on the multiple training sets. The corresponding network model is used to perform a first enhancement on the image. The enhanced image is segmented and pixel-level labeled, and low-light image enhancement is performed on all superpixel regions in the segmented image. The IoU value between the segmentation result after enhancement and the segmentation result before enhancement is calculated to achieve iterative processing. The RGB value of the iteratively enhanced image is thresholded. If it exceeds the threshold, histogram equalization is performed, and the segmentation and enhancement iterative processing is repeated until it meets the threshold. The present invention can perform image enhancement on different brightness areas of the same image separately, and effectively solve the problem of overall color deviation in the image based on the enhanced image.

[0151] Example 2

[0152] This embodiment provides a low-light image enhancement system based on superpixel analysis;

[0153] The low-light image enhancement system based on superpixel analysis includes:

[0154] an acquisition module configured to: acquire a low-light image to be processed, select a pre-trained network model of a corresponding level according to an average grayscale value of the low-light image to be processed, and perform a first enhancement process on the image to be processed according to the selected pre-trained network model;

[0155] The first processing module is configured to: perform a first image segmentation on the image after the first enhancement process to segment a plurality of independent objects, each independent object corresponding to a superpixel; perform a second enhancement process on the corresponding superpixel using a pre-trained network model of a corresponding level according to the average grayscale value of each superpixel; and merge all the images after the second enhancement process to obtain a low-light image after the second enhancement process;

[0156] The second processing module is configured to: perform a second image segmentation on the low-light image after the second enhancement; calculate the overlap between the second image segmentation result and the first image segmentation result, and return to the acquisition module if the overlap is less than a first set threshold; if the overlap is greater than the first set threshold, enter the third processing module;

[0157] The third processing module is configured to: calculate the RGB value of the entire low-light image after the second enhancement, compare the RGB value with the second set threshold value respectively, if it exceeds the second set threshold value, perform histogram equalization processing on the low-light image after the second enhancement, and send the processed image back to the acquisition module; if it is less than the second set threshold value, end, and output the low-light image after the second enhancement as the enhancement result.

[0158] It should be noted that the acquisition module, first processing module, second processing module, and third processing module described above correspond to steps S101 to S104 in Example 1. The examples and application scenarios implemented by the modules and corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the modules described above, as part of a system, can be executed in a computer system, such as a set of computer-executable instructions.

[0159] The descriptions of the various embodiments in the above embodiments have different focuses. For parts not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0160] The proposed system can be implemented in other ways. For example, the system embodiment described above is merely illustrative. For example, the above module division is only a logical function division. In actual implementation, other division methods may be used. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not implemented.

[0161] Example 3

[0162] This embodiment also provides an electronic device, comprising: one or more processors, one or more memories, and one or more computer programs; wherein the processor is connected to the memory, and the one or more computer programs are stored in the memory. When the electronic device is running, the processor executes the one or more computer programs stored in the memory, so that the electronic device executes the method described in the above embodiment one.

[0163] It should be understood that in this embodiment, the processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), off-the-shelf field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0164] The memory may include a read-only memory and a random access memory, and provides instructions and data to the processor. A portion of the memory may also include a non-volatile random access memory. For example, the memory may also store information about the device type.

[0165] During implementation, each step of the above method may be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software.

[0166] The method in Example 1 can be directly implemented as being executed by a hardware processor, or by a combination of hardware and software modules within the processor. The software module can be located in a storage medium well-established in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not given here.

[0167] Those skilled in the art will appreciate that the units and algorithm steps of the various examples described in conjunction with this embodiment can be implemented using electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0168] Example 4

[0169] This embodiment further provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the method described in the first embodiment is performed.

[0170] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A low-light image enhancement method based on superpixel analysis, characterized by: include: S101: Acquire a low-light image to be processed, select a pre-trained network model of a corresponding level according to the average grayscale value of the low-light image to be processed, and perform a first enhancement process on the image to be processed according to the selected pre-trained network model; S102: performing a first image segmentation on the image after the first enhancement process to segment a number of independent objects, each of which corresponds to a superpixel; performing a second enhancement process on the corresponding superpixel using a pre-trained network model of a corresponding level according to the average grayscale value of each superpixel; Merging all images after the second enhancement process to obtain a low-light image after the second enhancement process; S103: performing a second image segmentation on the low-light image after the second enhancement; Calculate the overlap between the second image segmentation result and the first image segmentation result. If the overlap is less than the first set threshold, return to S101. If the overlap is greater than the first set threshold, proceed to S104. S104: Calculate the overall RGB values of the low-light image after the second enhancement, and compare the RGB values with the second set threshold value respectively. If the values exceed the second set threshold value, perform histogram equalization on the low-light image after the second enhancement, and send the processed image back to S101; if the values are less than the second set threshold value, the process ends and the low-light image after the second enhancement is output as the enhancement result.

2. The low-light image enhancement method based on superpixel analysis according to claim 1, wherein: The average grayscale value of the low-light image to be processed, wherein the calculation process of the average grayscale value includes: First calculate the RGB value of each pixel of the low-light image to be processed; Convert the RGB value of each pixel into the grayscale value of each pixel; Calculate the sum of the grayscale values of all pixels; The average grayscale value is calculated based on the sum of the grayscale values of all pixels and the number of all pixels.

3. The low-light image enhancement method based on superpixel analysis according to claim 1, wherein: The pre-trained network model has a network structure comprising: The first convolution layer, the first activation function layer, the second convolution layer, the second activation function layer, the third convolution layer, the third activation function layer, the fourth convolution layer, the fourth activation function layer, the fifth convolution layer, the fifth activation function layer, the sixth convolution layer, the sixth activation function layer, the seventh convolution layer, the seventh activation function layer, the eighth convolution layer, the eighth activation function layer, the ninth convolution layer and the ninth activation function layer, connected sequentially; The feature map output by the first activation function layer is concatenated and fused with the feature map output by the eighth activation function layer, and then input into the ninth convolutional layer; The feature map output by the second activation function layer is concatenated and fused with the feature map output by the seventh activation function layer, and then input into the eighth convolutional layer; The feature map output by the third activation function layer and the feature map output by the sixth activation function layer are concatenated and fused in series and then input into the seventh convolutional layer; The feature map output by the fourth activation function layer and the feature map output by the fifth activation function layer are concatenated and fused in series and then input into the sixth convolutional layer.

4. The low-light image enhancement method based on superpixel analysis according to claim 1, wherein: The specific training process of the pre-trained network model includes: Constructing a first training set, wherein the first training set includes several sub-training sets, each of which is obtained by classifying all images in the data set according to brightness into several levels, and each level of images constitutes a sub-training set; Input a sub-training set of a level into the corresponding network model and train the network model. When the total loss function value of the network model no longer decreases, the pre-trained network model of the current level is obtained. Different levels of sub-training sets are used to train pre-trained network models of different levels.

5. The low-light image enhancement method based on superpixel analysis according to claim 4, characterized in that: The total loss function of the network model is specifically: The total error function is the weighted sum of four error functions: According to actual tests, the four weighted values are modified and set according to the specific training results of the model, and the weights are set to: ; Spatial consistency error : in, is the number of pixels, It is pixels n neighborhood, n The value of depends on the resolution of the input image; Y, I are the enhanced image and the input image respectively; Exposure control error : Among them, the constant A describes the median value of brightness, Y is a n The average brightness in the neighborhood, n The value of depends on the resolution of the input image. M is the total number of such neighborhoods; Color Constant Error : in, (p,q) Traversing all the two combinations of the three color channels, Represents color channels p The grayscale average value of Lighting smoothing error : in, N is the number of iterations, They are the horizontal and vertical gradient operators respectively; for images, the horizontal gradient and vertical gradient are the numerical differences between the adjacent pixels to the left and above.

6. The low-light image enhancement method based on superpixel analysis according to claim 1, wherein: The first image segmentation is performed on the image after the first enhancement process to segment several independent objects. Each independent object corresponds to a superpixel, including: The trained image segmentation model U-Net is used to perform the first image segmentation on the image after the first enhancement processing, and several independent objects are segmented. Each independent object corresponds to a superpixel.

7. A low-light image enhancement system based on superpixel analysis, characterized by: include: an acquisition module configured to: acquire a low-light image to be processed, select a pre-trained network model of a corresponding level according to an average grayscale value of the low-light image to be processed, and perform a first enhancement process on the image to be processed according to the selected pre-trained network model; The first processing module is configured to: perform a first image segmentation on the image after the first enhancement processing to segment a plurality of independent objects, each independent object corresponding to a superpixel; and perform a second enhancement processing on the corresponding superpixel using a pre-trained network model of a corresponding level according to the average grayscale value of each superpixel; Merging all images after the second enhancement process to obtain a low-light image after the second enhancement process; A second processing module is configured to: perform a second image segmentation on the low-light image after the second enhancement; Calculate the overlap between the second image segmentation result and the first image segmentation result. If the overlap is less than the first set threshold, return to the acquisition module. If the overlap is greater than the first set threshold, enter the third processing module. a third processing module configured to calculate the RGB values of the entire low-light image after the second enhancement, compare the RGB values with a second set threshold value, and if the values exceed the second set threshold value, perform histogram equalization on the low-light image after the second enhancement, and return the processed image to the acquisition module; If it is less than the second set threshold, the process ends and the low-light image after the second enhancement is output as the enhancement result.

8. An electronic device, comprising: a memory for non-transitory storage of computer-readable instructions; as well as a processor for executing said computer-readable instructions, When the computer-readable instructions are executed by the processor, the method according to any one of claims 1 to 6 is executed.

9. A storage medium, characterized in that: Computer-readable instructions are non-transitory stored, wherein when the non-transitory computer-readable instructions are executed by a computer, the instructions of the method according to any one of claims 1 to 6 are executed.

Images