Low-illumination image enhancement method and device based on trainable HSL color gamut distribution

By adopting U-Net that can trainable HSL color gamut distribution and fused cross attention mechanism in low-illumination image processing, the problems of noise increase and detail loss in the prior art are solved, and high-quality low-illumination image enhancement is achieved, and It exhibits excellent results in different scenarios, reducing the complexity of model calculations.

CN119090729AActive Publication Date: 2024-12-06ZHEJIANG UNIV
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411166205.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-23
Publication Date
2024-12-06
Estimated Expiration
2044-08-23

AI Technical Summary

Technical Problem

The existing low-illumination image enhancement technology can easily lead to noise increase and detail loss when processing images, and the application of deep learning models on resource-constrained devices is limited, the computing complexity is high, making it difficult to perform well in different lighting conditions and scenarios. Effect.

Method used

Using a method based on trainable HSL color gamut distribution, low-illumination RGB images are converted into HSL color gamut, brightness improvement and color noise reduction are performed through U-Net with fusion cross attention mechanism, and target color gamut loss and RGB color are passed. Domain loss trains deep learning models, optimizing data distribution parameters for high-quality low-illumination image enhancement.

Benefits of technology

High-quality low-illumination image enhancement is achieved under low computing complexity, which is suitable for different application scenarios, improves the performance and stability of the model and reduces the computational complexity.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090729A_ABST
    Figure CN119090729A_ABST
Patent Text Reader

Abstract

The invention discloses a low-illumination image enhancement method and device based on trainable HSL color gamut distribution. The method comprises the steps of converting a low-illumination RGB image into an HSL color gamut, and then performing data distribution adjustment by using a conversion function to obtain target color gamut data distribution; inputting the target color gamut data distribution into a brightness improvement module and a color noise reduction module of a deep learning model, fusing and outputting an enhanced target color gamut image, and performing color gamut inverse transformation on the enhanced target color gamut image to obtain an enhanced RGB image; constructing a target color gamut loss and RGB color gamut loss training deep learning model and optimizing a transfer function; and performing color gamut conversion on a new low-illumination RGB image based on the optimized conversion function to obtain a target color gamut, inputting the target color gamut into the trained deep learning model, and performing color gamut inverse transformation to obtain an enhanced RGB image. Based on the trainable HSL color gamut data distribution, the low-illumination image enhancement effect can be greatly improved, and the model calculation complexity can be reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a low-illumination image enhancement method and device based on trainable HSL color gamut distribution. Background Art

[0002] In the field of digital image processing, low-light image enhancement is an important technology that aims to improve the visibility and quality of images taken under low-light conditions. Low-light conditions often lead to increased noise, reduced contrast, and loss of details in images, which are particularly prominent in applications such as security monitoring, nighttime driving assistance systems, and smartphone photography.

[0003] Traditional image enhancement techniques, such as histogram equalization and gamma correction, can improve the brightness and contrast of images, but they are often accompanied by side effects such as over-enhancement and noise amplification. In addition, these methods usually deal with the global properties of the image, ignoring the complexity of local details and scene content, resulting in insufficient preservation of image details.

[0004] With the development of deep learning technology, low-light image enhancement methods based on deep learning have become a hot topic of research. These methods can automatically extract and utilize high-level features of images by learning a large amount of image data, thereby achieving more refined and natural image enhancement effects. Deep learning models can not only generate enhanced images directly from original low-light images through end-to-end training, but also integrate multiple loss functions during model training to optimize the pixel-level and semantic-level information of the image, further improving the quality of the enhanced image.

[0005] Deep learning has a variety of color gamut processing solutions to enhance the model's capabilities in the direction of low-light enhancement. In addition to the basic RGB color gamut, there are many other solutions, such as: changing the RGB color gamut distribution model; decomposing the observed image into a clear image and a point product of light intensity based on the Retinex theory; extracting pixel brightness signals based on the YUV color gamut; extracting hue saturation and brightness based on the HSV color gamut. Different solutions have shown that for low-light enhancement tasks, converting to a suitable color gamut to improve data distribution quality plays a very important role in improving model performance.

[0006] In addition, in order to deal with the problems of overfitting and insufficient generalization that deep learning models may encounter in practical applications, researchers have also explored a variety of regularization and optimization strategies, such as data enhancement, network structure innovation, and complexity control. These strategies help improve the stability and adaptability of the model, enabling it to show good enhancement effects in different lighting conditions and scenes. At the same time, deep learning models usually require a lot of computing resources to support their complex network structure and training process, which to some extent limits their application on resource-constrained devices.

[0007] Therefore, for low-light image enhancement tasks, how to select the appropriate color gamut conversion method according to specific task requirements and effectively fuse image information in different color gamuts, how to design more efficient and stable regularization methods and optimization algorithms to further improve the performance and stability of the model, and how to reduce the computational complexity of the model while ensuring the enhancement effect have become important directions of current research. Summary of the invention

[0008] In view of the above, the purpose of the present invention is to provide a low-light image enhancement method and device based on a trainable HSL color gamut distribution, and to adjust data distribution based on the HSL color gamut. While separating brightness information, the continuously distributed hue and saturation information can provide excellent gradient information. The proposed trainable HSL color gamut data distribution can compact the dark and bright parts of the HSL color gamut for low-light enhancement tasks to better meet the requirements of low-light enhancement tasks. By inputting the trainable HSL color gamut data distribution into a deep learning model, and using U-Net with a cross-attention mechanism to process data for the color channel and brightness channel after the adjusted distribution, a high-quality low-light image enhancement solution with low computational complexity in the current field is provided, which can well meet the data distribution requirements of low-light enhancement tasks. At the same time, since a trainable parameter, i.e., a compactness index, is provided in the trainable HSL color gamut data distribution, it can be adaptively adjusted for different models and different application scenarios to achieve the best effect.

[0009] In order to achieve the above-mentioned invention object, the technical solution provided by the present invention is as follows:

[0010] In a first aspect, an embodiment of the present invention provides a low-light image enhancement method based on a trainable HSL color gamut distribution, comprising the following steps:

[0011] The RGB color gamut data distribution of the low-light RGB image is converted into the HSL color gamut data distribution, and the HSL color gamut data distribution is adjusted based on the conversion function to obtain the target color gamut data distribution;

[0012] The target color gamut data distribution is input into the brightness enhancement module and color noise reduction module of the deep learning model respectively, and the results of the two modules are fused to output the enhanced target color gamut image, and the enhanced target color gamut image is subjected to color gamut inverse transformation to finally obtain the enhanced RGB image;

[0013] The bright RGB image corresponding to the low-light RGB image is used as the second label, and the target color gamut image after the data distribution of the bright RGB image is adjusted is used as the first label. The target color gamut loss is constructed based on the first label and the enhanced target color gamut image, and the RGB color gamut loss is constructed based on the second label and the enhanced RGB image. The deep learning model is trained based on the target color gamut loss and the RGB color gamut loss, and the conversion function is optimized at the same time, that is, the HSL color gamut data distribution is adjusted through training;

[0014] Based on the optimized conversion function, the RGB color gamut of the new low-light RGB image is converted to the target color gamut and then input into the trained deep learning model for inference. The inference result is inversely transformed to achieve low-light image enhancement.

[0015] Preferably, the step of adjusting the HSL color gamut data distribution based on a conversion function to obtain a target color gamut data distribution includes:

[0016] The vector HSL=(H, S, L) of the HSL color gamut data distribution is adjusted using a conversion function to obtain the target color gamut data distribution, where H is hue, S is saturation, and L is brightness. The conversion function formula is as follows:

[0017] XYZ=f(H,S,L)

[0018] Wherein, XYZ is the new three-dimensional coordinates that maps the HSL color gamut data distribution to the target color gamut data distribution, and f(·) is the conversion function;

[0019] The specific calculation steps of the conversion function f(·) are as follows:

[0020] θ=clamp(L-0.5,-0.5,0.5)

[0021] Among them, θ is the adjusted hue value, clamp(·) is the value limiting function;

[0022] color_sensitive=(cos(θ·π)+ε) k

[0023] Among them, color_sensitive is the color sensitivity, ε is a small constant, k is the compactness index, and k is the parameter to be optimized in the conversion function;

[0024] c x =cos(2·π·H)

[0025] c y = sin(2·π·H)

[0026] Among them, c x is the cosine value of the hue, c y is the sine value of the hue, cos(·) is the cosine function, and sin(·) is the sine function;

[0027] X=color_sensitive·S·c x

[0028] Y=color_sensitive·S·c y

[0029] Z=L

[0030] Wherein, X, Y and Z are three coordinate values ​​in the new three-dimensional coordinates of the target color gamut data distribution, X and Y represent color, and Z represents brightness.

[0031] Preferably, the step of inputting the target color gamut data distribution into the brightness enhancement module and the color noise reduction module of the deep learning model respectively and fusing the results of the two modules to output an enhanced target color gamut image comprises:

[0032] The brightness enhancement module and color denoising module of the deep learning model both use U-Net integrated with the cross-attention mechanism for data processing. The brightness enhancement module is used to enhance the target color gamut data distribution, and the color denoising module is used to reduce the color of the target color gamut data distribution. Finally, the output results of the brightness enhancement module and the color denoising module are fused to obtain the enhanced target color gamut image.

[0033] Preferably, the brightness enhancement module and the color noise reduction module of the deep learning model both use U-Net fused with a cross-attention mechanism for data processing, including:

[0034] In the brightness enhancement route of the deep learning model for data processing, the brightness enhancement module includes a first encoder, a first integrated cross attention module, a second encoder, a second integrated cross attention module, a third integrated cross attention module, a first decoder, a fourth integrated cross attention module and a second decoder connected in sequence, and a first jump connection is established between the first integrated cross attention module and the fourth integrated cross attention module; in the color denoising route of the deep learning model for data processing, the color denoising module includes a third encoder, a first integrated cross attention module, a fourth encoder, a second integrated cross attention module, a third integrated cross attention module, a third decoder, a fourth integrated cross attention module and a fourth decoder connected in sequence, and a second jump connection is established between the first integrated cross attention module and the fourth integrated cross attention module; the target color gamut data distribution is input into the first encoder and the third encoder respectively, and the output of the second decoder and the output of the fourth decoder are obtained respectively after the brightness enhancement route and the color denoising route are respectively passed, and the output of the second decoder and the output of the fourth decoder are fused to obtain an enhanced target color gamut image;

[0035] Among them, each comprehensive cross-attention module includes a brightness cross-attention module and a color cross-attention module. The brightness cross-attention module obtains the query Q based on the intermediate input of the brightness improvement route and obtains the key K and value V based on the input of the color noise reduction route, and obtains the intermediate output of the brightness improvement route through cross-attention calculation. The color cross-attention module obtains the key K and value V based on the intermediate input of the brightness improvement route and obtains the query Q based on the input of the color noise reduction route, and obtains the intermediate output of the color noise reduction route through cross-attention calculation. The intermediate input here corresponds to the previous encoder, comprehensive cross-attention module or decoder adjacent to the current comprehensive cross-attention module, and the intermediate output here corresponds to the next encoder, comprehensive cross-attention module or decoder adjacent to the current comprehensive cross-attention module.

[0036] Preferably, the RGB color gamut loss and the target color gamut loss both include: Charbonnier loss, structural similarity loss, edge loss and perceptual loss.

[0037] Preferably, by dividing the color gamut conversion, model reasoning and inverse color gamut conversion into three threads, the color gamut conversion, model reasoning and inverse color gamut conversion of multiple images are respectively calculated in parallel through the three threads.

[0038] In a second aspect, to achieve the above-mentioned purpose of the invention, an embodiment of the present invention further provides a low-light image enhancement device based on a trainable HSL color gamut distribution, comprising: a color gamut data distribution adjustment module, a deep learning model construction module, a model training and parameter optimization module, and a low-light image enhancement module;

[0039] The color gamut data distribution adjustment module is used to convert the RGB color gamut data distribution of the low-illuminance RGB image into the HSL color gamut data distribution, and perform data distribution adjustment on the HSL color gamut data distribution based on the conversion function to obtain the target color gamut data distribution;

[0040] The deep learning model construction module is used to input the target color gamut data distribution into the brightness enhancement module and the color noise reduction module of the deep learning model respectively, and output an enhanced target color gamut image after fusing the results of the two modules, and perform color gamut inverse transformation on the enhanced target color gamut image to finally obtain an enhanced RGB image;

[0041] The model training and parameter optimization module is used to use the bright RGB image corresponding to the low-light RGB image as the second label, and the target color gamut image after the data distribution of the bright RGB image is adjusted as the first label, construct the target color gamut loss based on the first label and the enhanced target color gamut image, construct the RGB color gamut loss based on the second label and the enhanced RGB image, train the deep learning model based on the target color gamut loss and the RGB color gamut loss, and optimize the conversion function at the same time, that is, adjust the HSL color gamut data distribution through training;

[0042] The low-light image enhancement module is used to convert the RGB color gamut of the new low-light RGB image into the target color gamut based on the optimized conversion function, and then input it into the trained deep learning model for reasoning, and perform inverse color gamut transformation on the reasoning result to finally achieve low-light image enhancement.

[0043] In the third aspect, in order to achieve the above-mentioned purpose of the invention, an embodiment of the present invention also provides a low-light image enhancement device based on a trainable HSL color gamut distribution, comprising a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-mentioned low-light image enhancement method based on a trainable HSL color gamut distribution when executing the computer program.

[0044] In a fourth aspect, in order to achieve the above-mentioned purpose of the invention, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computer, the above-mentioned low-light image enhancement method based on trainable HSL color gamut distribution is implemented.

[0045] In a fifth aspect, in order to achieve the above-mentioned purpose of the invention, an embodiment of the present invention further provides a computer product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned low-light image enhancement method based on trainable HSL color gamut distribution is implemented.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] (1) Aiming at the low-light image enhancement task, the present invention designs a trainable HSL color gamut data distribution and provides a trainable compactness index, so that the data of the dark and bright parts can be more compact and the gradient information is more in line with the requirements of the low-light image enhancement task. Since the low-light enhancement task faces a variety of application scenarios, the data distribution parameters of the trainable HSL color gamut data distribution are optimized through training, which provides stronger applicability for different scenarios and can well demonstrate excellent effects in different application scenarios.

[0048] (2) The present invention constructs a deep learning model including a brightness enhancement module and a color noise reduction module, and processes image information through two routes respectively. It can learn multi-level semantic information well and achieve pixel-level and semantic-level optimization, thereby greatly improving the low-illumination image enhancement effect of the deep learning model in the trainable HSL color space, and achieves high-quality image enhancement effect in low computational complexity scenarios based on U-Net fused with the cross-attention mechanism.

[0049] (3) The present invention constructs a deep learning model training and data distribution parameter optimization including target color gamut loss and RGB color gamut loss, and performs fusion operations based on multiple losses as a guide, which can further improve the performance and stability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0051] Figure 1 is a flow chart of a low-illumination image enhancement method based on a trainable HSL color gamut distribution provided by an embodiment of the present invention;

[0052] Figure 2 is a visualized schematic diagram of a mathematical curve of target color gamut data distribution (compact index k=0.2) provided by an embodiment of the present invention;

[0053] Figure 3 is a schematic diagram of a comprehensive cross-attention module in a deep learning model provided by an embodiment of the present invention;

[0054] Figure 4 A comparison diagram of the effects of the low-light RGB image in the input LOL_v1 dataset, the enhanced RGB image output by model inference, and GroundTruth in the RGB space and the target color gamut space respectively provided by an embodiment of the present invention;

[0055] Figure 5 It is a structural schematic diagram of a low-illumination image enhancement device based on a trainable HSL color gamut distribution provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0057] The inventive concept of the present invention is: in view of the problems of insufficient applicability of color gamut conversion, insufficient model generalization ability and high computational complexity of low-illuminance image enhancement methods based on deep learning in the prior art, the embodiments of the present invention provide a low-illuminance image enhancement method and device based on trainable HSL color gamut distribution, propose to optimize the data distribution parameters of the trainable HSL color gamut distribution, and extract multi-level features including brightness and color by constructing a deep learning model based on U-Net and cross-attention mechanism, train the deep learning model by constructing a fused loss function and optimize the data distribution parameters at the same time, thereby improving the applicability of the color gamut conversion method for low-illuminance image enhancement tasks, further improving the performance and stability of the model, and being able to reduce the model calculation complexity while ensuring the enhancement effect.

[0058] Figure 1 FIG. 1 is a flow chart of a low-light image enhancement method based on a trainable HSL color gamut distribution provided by an embodiment of the present invention. Figure 1 As shown, the embodiment provides a low-light image enhancement method based on a trainable HSL color gamut distribution, comprising the following steps:

[0059] S1, converting the RGB color gamut data distribution of the low-illuminance RGB image into the HSL color gamut data distribution, and adjusting the HSL color gamut data distribution based on the conversion function to obtain the target color gamut data distribution.

[0060] In the embodiment, the RGB color gamut data distribution of the low-light RGB image is first converted into the HSL color gamut data distribution, and then the vector HSL=(H, S, L) of the HSL color gamut data distribution is adjusted using a conversion function to obtain a target color gamut data distribution, where H is hue, S is saturation, and L is brightness. The conversion function formula is as follows:

[0061] XYZ=f(H,S,L)

[0062] Wherein, XYZ is a new three-dimensional coordinate corresponding to HSL that maps the HSL color gamut data distribution to the target color gamut data distribution, and f(·) is a conversion function.

[0063] The specific calculation steps of the conversion function f(·) are as follows:

[0064] (1) Adjust the hue value:

[0065] θ=clamp(L-0.5,-0.5,0.5)

[0066] Among them, θ is the adjusted hue value, and clamp(·) is the value limiting function.

[0067] (2) Calculate color sensitivity:

[0068] color_sensitive=(cos(θ·π)+ε) k

[0069] Among them, color_sensitive is the color sensitivity, ε is a small constant and is used to prevent numerical stability problems caused by the denominator being 0 or the value within the square root being too small, k is the compactness index, and k is the parameter to be optimized in the conversion function.

[0070] (3) Calculate the cosine and sine of the hue:

[0071] c x =cos(2·π·H)

[0072] c y = sin(2·π·H)

[0073] Among them, c x is the cosine value of the hue, c y is the sine value of the hue, cos(·) is the cosine function, and sin(·) is the sine function.

[0074] (4) Convert to the coordinates of the target color space:

[0075] X=color_sensitive·S·c x

[0076] Y=color_sensitive·S·c y

[0077] Z=L

[0078] Wherein, X, Y and Z are three coordinate values ​​in the new three-dimensional coordinates of the target color gamut data distribution, X and Y represent color, and Z represents brightness.

[0079] A mathematical curve visualization of a specific type of target color gamut data distribution (compactness index k = 0.2) is shown below: Figure 2 As shown, it can be vividly described as the target color gamut data for low-light image enhancement is distributed in the three-dimensional space in the shape of a rugby ball. The positions where the brightness is close to 1 and close to 0 are more compact, while the part with brightness close to 0.5 has the richest color and is the part where the saturation effect is most obvious.

[0080] The trainable feature is reflected in the compactness index k in the color sensitivity calculation in the calculation step (2). By combining the training of the deep learning model, the most suitable compactness index k in a specific application scenario and a specific model network structure can be calculated, thereby improving the applicability of data distribution.

[0081] S2, input the target color gamut data distribution into the brightness enhancement module and color noise reduction module of the deep learning model respectively, and fuse the results of the two modules to output the enhanced target color gamut image, perform color gamut inverse transformation on the enhanced target color gamut image, and finally obtain the enhanced RGB image.

[0082] In the embodiment, after the RGB color gamut data distribution of the low-light RGB image is converted into the HSL color gamut data distribution, it is input into the deep learning model and data enhancement is performed through the brightness enhancement route and the color noise reduction route. In the deep learning model, a brightness enhancement module and a color noise reduction module are designed for the two routes respectively. The brightness enhancement module and the color noise reduction module both use U-Net fused with a cross-attention mechanism for data processing. The target color gamut data distribution is subjected to brightness enhancement processing through the brightness enhancement module and the target color gamut data distribution is subjected to color noise reduction processing through the color noise reduction module. Finally, the output results of the brightness enhancement module and the color noise reduction module are fused to obtain an enhanced target color gamut image. Both routes have the ability of two image processing methods, namely brightness enhancement and color noise reduction, and can well learn multi-level semantic information. With the appropriate target color gamut data distribution model, they can well handle the task of low-light image enhancement and improve the low-light image enhancement effect.

[0083] like Figure 1As shown, in the brightness enhancement route, the brightness enhancement module includes a first encoder, a first integrated cross-attention module, a second encoder, a second integrated cross-attention module, a third integrated cross-attention module, a first decoder, a fourth integrated cross-attention module and a second decoder connected in sequence, and a first jump connection is established between the first integrated cross-attention module and the fourth integrated cross-attention module; in the color denoising route, the color denoising module includes a third encoder, a first integrated cross-attention module, a fourth encoder, a second integrated cross-attention module, a third integrated cross-attention module, a third decoder, a fourth integrated cross-attention module and a fourth decoder connected in sequence, and a second jump connection is established between the first integrated cross-attention module and the fourth integrated cross-attention module; by inputting the target color gamut data distribution into the first encoder and the third encoder respectively and respectively passing through the brightness enhancement route and the color denoising route to obtain the output of the second decoder and the output of the fourth decoder, the output of the second decoder and the output of the fourth decoder are fused to obtain an enhanced target color gamut image.

[0084] like Figure 3 As shown, each integrated cross-attention module includes a brightness cross-attention module and a color cross-attention module. The brightness cross-attention module obtains the query Q based on the intermediate input of the brightness enhancement route and obtains the key K and the value V based on the input of the color denoising route, and obtains the intermediate output of the brightness enhancement route through cross-attention calculation. The color cross-attention module obtains the key K and the value V based on the intermediate input of the brightness enhancement route and obtains the query Q based on the input of the color denoising route, and obtains the intermediate output of the color denoising route through cross-attention calculation. The intermediate input here corresponds to the previous encoder, integrated cross-attention module or decoder adjacent to the current integrated cross-attention module, and the intermediate output here corresponds to the next encoder, integrated cross-attention module or decoder adjacent to the current integrated cross-attention module.

[0085] S3, taking the bright RGB image corresponding to the low-illumination RGB image as the second label, and taking the target color gamut image after data distribution adjustment of the bright RGB image as the first label, constructing the target color gamut loss based on the first label and the enhanced target color gamut image, constructing the RGB color gamut loss based on the second label and the enhanced RGB image, training the deep learning model based on the target color gamut loss and the RGB color gamut loss and optimizing the conversion function at the same time, that is, adjusting the HSL color gamut data distribution through training.

[0086] In the embodiment, a composite loss is used to guide the training. The bright RGB image corresponding to the low-light RGB image is used as the second label, the target color gamut image after the data distribution adjustment of the bright RGB image is used as the first label, the target color gamut loss is constructed based on the first label and the enhanced target color gamut image, and the RGB color gamut loss is constructed based on the second label and the enhanced RGB image.

[0087] Among them, the target color gamut loss and RGB color gamut loss each include four loss functions: Charbonnier loss, structural similarity loss, edge loss and perceptual loss for composite guidance.

[0088] (1) Charbonnier loss:

[0089] Charbonnier loss is a smoothed version of L1 loss, which provides pixel-level features with better robustness and is used for regression tasks in image processing. Its formula is:

[0090]

[0091] Among them, L charb (x,y) is Charbonnier loss, x i and i are the model prediction value and true label of the i-th pixel respectively, V is the pixel set, |V| is the number of pixels, ∈ is a small constant and is used to prevent numerical stability problems caused by the denominator being 0 or the value within the square root being too small.

[0092] (2) Structural similarity loss:

[0093] The structural similarity index is an index to measure the similarity between two images. It is used to evaluate image quality. It can measure the similarity of the overall structure of two images and show the structural characteristics of the images. The structural similarity loss is a loss function designed based on the structural similarity index. It is used to optimize the model during the training process so that the generated image is more similar in structure to the target image. The formula is:

[0094]

[0095] L SSIM (x,y)=1-SSIM(x,y)

[0096] Among them, SSIM(x,y) is the structural similarity index, L SSIM (x,y) is the structural similarity loss, μ x and μ y are the means of images x and y, respectively. and are the variances of images x and y, σ xyis the covariance of x and y, c1 and c2 are small constants used to maintain stability to avoid the denominator being zero.

[0097] (3) Edge Loss:

[0098] Edge loss is used to enhance the edge information of the image. It is based on Gaussian filtering and Laplace operators. It can provide additional attention to high-frequency information such as edges while maintaining the similarity of the overall image. The formula is:

[0099] L edge (x,y)=MSE(Laplacian(x),Laplacian(y))

[0100] Among them, L edge (x, y) is the edge loss, MSE(·) is the mean square error function, and Laplacian(x) and Laplacian(y) are the results obtained by applying the Laplacian operator after Gaussian filtering.

[0101] (4) Perceptual Loss:

[0102] Perceptual loss is used to compare the differences between two images in high-level features. The features extracted based on pre-trained deep networks (such as VGG networks) are more in line with the human eye's perception of images, and can extract the semantic information of images for multi-level comparison, which can well evaluate the similarity of images. Its formula is:

[0103]

[0104] Among them, L perceptual (x,y) is the perceptual loss, F j (x) and F j (y) is the feature extracted from the jth layer of the pre-trained deep network, w j is the weight of the j-th layer. Criterion(·) is the criterion used to calculate the loss, such as L1, L2, or MSE. These loss functions are widely used in deep learning for tasks such as image restoration, super-resolution, and style migration. By combining these losses, the model can be better optimized to achieve specific visual effects.

[0105] During the deep learning model training process, the learning rate is adjusted by combining periodic cosine annealing restart and progressive warm-up strategy. At the same time, random cropping and reversal strategies of training data are introduced to improve data strength and avoid continuous oscillation or overfitting.

[0106] S4, based on the optimized conversion function, converts the RGB color gamut of the new low-light RGB image to the target color gamut and then inputs it into the trained deep learning model for inference. The inference result is inversely transformed in color gamut to finally achieve low-light image enhancement.

[0107] In an embodiment, for a new low-light RGB image to be enhanced, it is first converted from the RGB color gamut to the target color gamut using an optimized conversion function, and then input into a trained deep learning model for model inference, and the image in the target color gamut output by the model is inversely transformed to obtain an enhanced image in the RGB color gamut, thereby achieving low-light image enhancement.

[0108] In an embodiment, by dividing color gamut conversion, model reasoning and color gamut inverse transformation into three threads, the color gamut conversion, model reasoning and color gamut inverse transformation processes of multiple images can be implemented through task queues or other methods to achieve multi-threaded parallel calculation, which can further improve the operating efficiency.

[0109] like Figure 4 Shown is the input low-light RGB image in RGB space ( Figure 4 (a) in the figure), model inference output enhanced RGB image ( Figure 4 (b)) and GroundTruth( Figure 4 The effect comparison of (c) in the target color space and the input low-light RGB image ( Figure 4 (d) in the figure), model inference output enhanced RGB image ( Figure 4 (e)) and GroundTruth( Figure 4 The comparison diagram of the effects of (f)) in FIG. 1 shows that the embodiment of the present invention can successfully implement multi-scale semantic analysis in the constructed color gamut space and thus achieve the effect of high-quality low-light image enhancement.

[0110] In summary, the embodiment of the present invention provides a low-light image enhancement method based on a trainable HSL color gamut distribution, and proposes a trainable HSL color gamut data distribution to optimize the distribution of input data, so that the data distribution information of the low-light image can obtain a better gradient when the model is trained, thereby improving the low-light image enhancement performance of the deep learning model. At the same time, the various low-light image enhancement algorithms currently available have proved that a good color gamut distribution has a great impact on the work. The trainable HSL color gamut data distribution in the present invention provides trainable data distribution parameters, so that it can automatically adapt to a variety of application scenarios and different reasoning network structures. After obtaining the optimal data distribution parameters, it can be extracted and used in conjunction with the model reasoning process. The trainable HSL color gamut distribution can be transplanted into different network models, and the distribution model is extracted separately after training as a preprocessing, so it can be simply and easily combined with the existing low-light image enhancement method. The present invention also uses U-Net, which integrates the cross-attention mechanism, to design a deep learning model, which greatly improves the low-light image enhancement effect while reducing the model calculation complexity.

[0111] Based on the same inventive concept, Figure 5 As shown, an embodiment of the present invention also provides a low-light image enhancement device 500 based on a trainable HSL color gamut distribution, comprising: a color gamut data distribution adjustment module 510, a deep learning model construction module 520, a model training and parameter optimization module 530, and a low-light image enhancement module 540.

[0112] The color gamut data distribution adjustment module 510 is used to convert the RGB color gamut data distribution of the low-illuminance RGB image into the HSL color gamut data distribution, and perform data distribution adjustment on the HSL color gamut data distribution based on the conversion function to obtain the target color gamut data distribution.

[0113] The deep learning model construction module 520 is used to input the target color gamut data distribution into the brightness enhancement module and color noise reduction module of the deep learning model respectively, and fuse the results of the two modules to output an enhanced target color gamut image, perform color gamut inverse transformation on the enhanced target color gamut image, and finally obtain an enhanced RGB image.

[0114] The model training and parameter optimization module 530 is used to take the bright RGB image corresponding to the low-illuminance RGB image as the second label, and take the target color gamut image after data distribution adjustment of the bright RGB image as the first label, construct the target color gamut loss based on the first label and the enhanced target color gamut image, construct the RGB color gamut loss based on the second label and the enhanced RGB image, train the deep learning model based on the target color gamut loss and the RGB color gamut loss and optimize the conversion function at the same time, that is, adjust the HSL color gamut data distribution through training.

[0115] The low-light image enhancement module 540 is used to convert the RGB color gamut of the new low-light RGB image to the target color gamut based on the optimized conversion function, and then input it into the trained deep learning model for inference, and perform inverse color gamut transformation on the inference result to finally achieve low-light image enhancement.

[0116] Based on the same inventive concept, an embodiment of the present invention also provides a low-light image enhancement device based on a trainable HSL color gamut distribution, comprising a memory and one or more processors, the memory being used to store a computer program, and the processor being used to implement the above-mentioned low-light image enhancement method based on a trainable HSL color gamut distribution when executing the computer program.

[0117] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computer, the above-mentioned low-illumination image enhancement method based on trainable HSL color gamut distribution is implemented.

[0118] Based on the same inventive concept, an embodiment of the present invention further provides a computer product, which includes a computer program. When the computer program is executed by a processor, the above-mentioned low-illumination image enhancement method based on trainable HSL color gamut distribution is implemented.

[0119] It should be noted that the low-light image enhancement apparatus based on trainable HSL color gamut distribution, the low-light image enhancement device based on trainable HSL color gamut distribution, the computer-readable storage medium and the computer product provided in the above-mentioned embodiments all belong to the same inventive concept as the low-light image enhancement method based on trainable HSL color gamut distribution. The specific implementation process is detailed in the embodiment of the low-light image enhancement method based on trainable HSL color gamut distribution, which will not be repeated here.

[0120] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A low-light image enhancement method based on trainable HSL color gamut distribution, It is characterized in that The following steps are involved: The RGB color gamut data distribution of the low-light RGB image is converted into the HSL color gamut data distribution, and the HSL color gamut data distribution is adjusted based on the conversion function to obtain the target color gamut data distribution; The target color gamut data distribution is input into the brightness enhancement module and color noise reduction module of the deep learning model respectively, and the results of the two modules are fused to output the enhanced target color gamut image, and the enhanced target color gamut image is subjected to color gamut inverse transformation to finally obtain the enhanced RGB image; The bright RGB image corresponding to the low-light RGB image is used as the second label, and the target color gamut image after the data distribution of the bright RGB image is adjusted is used as the first label. The target color gamut loss is constructed based on the first label and the enhanced target color gamut image, and the RGB color gamut loss is constructed based on the second label and the enhanced RGB image. The deep learning model is trained based on the target color gamut loss and the RGB color gamut loss, and the conversion function is optimized at the same time, that is, the HSL color gamut data distribution is adjusted through training; Based on the optimized conversion function, the RGB color gamut of the new low-light RGB image is converted to the target color gamut and then input into the trained deep learning model for inference. The inference result is inversely transformed to achieve low-light image enhancement.

2. The low-light image enhancement method based on trainable HSL color gamut distribution according to claim 1, It is characterized in that The step of adjusting the HSL color gamut data distribution based on the conversion function to obtain the target color gamut data distribution includes: The vector HSL=(H, S, L) of the HSL color gamut data distribution is adjusted using a conversion function to obtain the target color gamut data distribution, where H is hue, S is saturation, and L is brightness. The conversion function formula is as follows: XYZ=f(H,S,L) Wherein, XYZ is the new three-dimensional coordinates that maps the HSL color gamut data distribution to the target color gamut data distribution, and f(·) is the conversion function; The specific calculation steps of the conversion function f(·) are as follows: θ=clamp(L-0.5,-0.5,0.5) Among them, θ is the adjusted hue value, clamp(·) is the value limiting function; color_sensitive=(cos(θ·π)+ε) k Among them, color_sensitive is the color sensitivity, ε is a small constant, k is the compactness index, and k is the parameter to be optimized in the conversion function; c x =cos(2·π·H) c y =sin(2·π·H) Among them, c x is the cosine value of the hue, c y is the sine value of the hue, cos(·) is the cosine function, and sin(·) is the sine function; X=color_sensitive·S·c x Y=color_sensitive·S·c y Z=L Wherein, X, Y and Z are three coordinate values ​​in the new three-dimensional coordinates of the target color gamut data distribution, X and Y represent color, and Z represents brightness.

3. The low-light image enhancement method based on trainable HSL color gamut distribution according to claim 1, It is characterized in that The target color gamut data distribution is input into the brightness enhancement module and the color noise reduction module of the deep learning model respectively, and the results of the two modules are fused to output an enhanced target color gamut image, including: The brightness enhancement module and color denoising module of the deep learning model both use U-Net integrated with the cross-attention mechanism for data processing. The brightness enhancement module is used to enhance the target color gamut data distribution, and the color denoising module is used to reduce the color of the target color gamut data distribution. Finally, the output results of the brightness enhancement module and the color denoising module are fused to obtain the enhanced target color gamut image.

4. The low-light image enhancement method based on trainable HSL color gamut distribution according to claim 3, It is characterized in that The brightness enhancement module and color noise reduction module of the deep learning model both use U-Net integrated with the cross-attention mechanism for data processing, including: In the brightness enhancement route of the deep learning model for data processing, the brightness enhancement module includes a first encoder, a first integrated cross attention module, a second encoder, a second integrated cross attention module, a third integrated cross attention module, a first decoder, a fourth integrated cross attention module and a second decoder connected in sequence, and a first jump connection is established between the first integrated cross attention module and the fourth integrated cross attention module; in the color denoising route of the deep learning model for data processing, the color denoising module includes a third encoder, a first integrated cross attention module, a fourth encoder, a second integrated cross attention module, a third integrated cross attention module, a third decoder, a fourth integrated cross attention module and a fourth decoder connected in sequence, and a second jump connection is established between the first integrated cross attention module and the fourth integrated cross attention module; the target color gamut data distribution is input into the first encoder and the third encoder respectively, and the output of the second decoder and the output of the fourth decoder are obtained respectively after the brightness enhancement route and the color denoising route are respectively passed, and the output of the second decoder and the output of the fourth decoder are fused to obtain an enhanced target color gamut image; Among them, each comprehensive cross-attention module includes a brightness cross-attention module and a color cross-attention module. The brightness cross-attention module obtains the query Q based on the intermediate input of the brightness improvement route and obtains the key K and value V based on the input of the color noise reduction route, and obtains the intermediate output of the brightness improvement route through cross-attention calculation. The color cross-attention module obtains the key K and value V based on the intermediate input of the brightness improvement route and obtains the query Q based on the input of the color noise reduction route, and obtains the intermediate output of the color noise reduction route through cross-attention calculation. The intermediate input here corresponds to the previous encoder, comprehensive cross-attention module or decoder adjacent to the current comprehensive cross-attention module, and the intermediate output here corresponds to the next encoder, comprehensive cross-attention module or decoder adjacent to the current comprehensive cross-attention module.

5. The low-light image enhancement method based on trainable HSL color gamut distribution according to claim 1, It is characterized in that Both RGB color gamut loss and target color gamut loss include: Charbonnier loss, structural similarity loss, edge loss and perceptual loss.

6. The low-light image enhancement method based on trainable HSL color gamut distribution according to claim 1, It is characterized in that By dividing the color gamut conversion, model reasoning and inverse color gamut transformation into three threads, the color gamut conversion, model reasoning and inverse color gamut transformation of multiple images are calculated in parallel through three threads respectively.

7. A low-light image enhancement device based on trainable HSL color gamut distribution, It is characterized in that include: Color gamut data distribution adjustment module, deep learning model construction module, model training and parameter optimization module, and low-light image enhancement module; The color gamut data distribution adjustment module is used to convert the RGB color gamut data distribution of the low-illuminance RGB image into the HSL color gamut data distribution, and perform data distribution adjustment on the HSL color gamut data distribution based on the conversion function to obtain the target color gamut data distribution; The deep learning model construction module is used to input the target color gamut data distribution into the brightness enhancement module and the color noise reduction module of the deep learning model respectively, and output an enhanced target color gamut image after fusing the results of the two modules, and perform color gamut inverse transformation on the enhanced target color gamut image to finally obtain an enhanced RGB image; The model training and parameter optimization module is used to use the bright RGB image corresponding to the low-light RGB image as the second label, and the target color gamut image after the data distribution of the bright RGB image is adjusted as the first label, construct the target color gamut loss based on the first label and the enhanced target color gamut image, construct the RGB color gamut loss based on the second label and the enhanced RGB image, train the deep learning model based on the target color gamut loss and the RGB color gamut loss, and optimize the conversion function at the same time, that is, adjust the HSL color gamut data distribution through training; The low-light image enhancement module is used to convert the RGB color gamut of the new low-light RGB image into the target color gamut based on the optimized conversion function, and then input it into the trained deep learning model for reasoning, and perform inverse color gamut transformation on the reasoning result to finally achieve low-light image enhancement.

8. A low-light image enhancement device based on a trainable HSL color gamut distribution, comprising a memory and one or more processors, wherein the memory is used to store a computer program, It is characterized in that The processor is used to implement the low-light image enhancement method based on trainable HSL color gamut distribution as described in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a computer, the low-illumination image enhancement method based on trainable HSL color gamut distribution described in any one of claims 1 to 6 is implemented.

10. A computer product comprising a computer program, It is characterized in that When the computer program is executed by a processor, the low-light image enhancement method based on trainable HSL color gamut distribution described in any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • A low-illumination image adaptive brightness enhancement method based on noise suppression

    CN109816608A

  • Low-light image enhancement method based on U-shaped network and attention mechanism

    CN116596793A

  • Illumination image enhancement method based on HSV space

    CN117291818A