HSV color domain-based ltmnet contrast enhancement method

Abstract

The invention provides an ltmnet contrast enhancement method based on an HSV color domain, and the method comprises the steps: S1, collecting and sorting low-contrast-high-contrast image sample pairs, and constructing a training set and a test set; s2, mapping the RGB image data of the sample pair to an HSV color space, and converting the image size into (512, 512, 3) as the input and output size of network training; s3, constructing an ltmnet adjustable network model with enhanced HSV domain contrast ratio; a low-contrast-high-contrast data set is cut and processed, and a model structure is adjusted, so that HSV domain contrast enhancement is adapted; performing ltmnet curve mapping on the low-contrast image; and S4, performing loss calculation on a result obtained after mapping each grid with different sizes by using L1loss and ssim and the corresponding high-quality images with different enhancement degrees, and summing the result as the loss of parameter updating. Deep learning contrast enhancement is carried out on low contrast in an HSV domain, good adaptive contrast enhancement is achieved for images of different scenes, bright region overexposure can be effectively inhibited, and dark region details are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and particularly relates to an ltmnet contrast enhancement method based on the HSV color domain. Background Art

[0002] In the prior art, in real scenarios, due to problems such as low light and inappropriate field of view, images often exhibit various degradations, such as low contrast, color distortion, and noise. These degradations not only affect the visual effect but also affect computer vision tasks. Image enhancement can enhance the effective information in the picture, improve the visual effect of the image, magnify the details of low-quality pictures, such as portrait enhancement, blurry picture sharpening, old photo sharpening, etc., and enrich information and strengthen the image interpretation and recognition effect by improving the image quality. Among them, contrast enhancement plays an important role in improving the visual quality of image processing. Traditional contrast enhancement methods include histogram equalization, CLAHE, Gamma transformation, etc., and appropriate method selection and parameter adjustment need to be carried out for specific scenarios. In the field of deep learning, end-to-end network learning is carried out in the RGB color space, and good results have been achieved without special parameter selection.

[0003] However, the defects of the prior art are as follows:

[0004] Currently, in the research of deep learning, image enhancement mainly focuses on the RGB color space. The three components of the RGB color space are all closely related to brightness, that is, as long as the brightness changes, the three components will change accordingly, and it cannot well solve the problems of overexposure in bright areas and disappearance of details in dark areas that occur during enhancement.

[0005] In addition, the commonly used terms in the prior art include:

[0006] Contrast: The measurement of different brightness levels between the brightest white and the darkest black in the light and dark areas of an image.

[0007] RGB: The three primary colors of light, where R, G, and B represent red, green, and blue respectively.

[0008] HSV: The HSV color space describes colors through three dimensions: color, shade, and lightness. In the standard HSV space, the value range of the H channel is 0 - 360 (degrees), the value range of the S channel is 0 - 1 (0% - 100%), and the value range of the V channel is 0 - 1 (0% - 100%). Summary of the Invention

[0009] To solve the above problems, the purpose of this application is: for images with low contrast, low permeability, and low saturation, the present invention proposes a method of end-to-end training of a deep learning network in the HSV color space to learn the mapping curves of each channel, and obtaining a high-contrast image through interpolation post-processing to obtain an efficient and generalizable contrast enhancement technical method.

[0010] Specifically, the present invention provides an ltmnet contrast enhancement method based on the HSV color domain, and the method includes the following steps:

[0011] S1: Collect and organize low-contrast - high-contrast image sample pairs, and construct a training set and a test set;

[0012] S2: Map the RGB image data of the sample pairs to the HSV color space, and at the same time change the image size to (512, 512, 3), which serves as the input and output size for network training;

[0013] S3: Construct an adjustable network model of ltmnet for contrast enhancement in the HSV domain; adjust the model structure to adapt to contrast enhancement in the HSV domain; perform ltmnet curve mapping on the low-contrast image;

[0014] S4: Use Llloss and ssim to calculate the loss between the results of mapping each grid of different sizes and the high-quality images with corresponding different enhancement degrees, and sum them as the loss for parameter update;

[0015] Loss(x, y) = sum(loss gs ) where gs = 1, 2, 4, 8, 16 (1)

[0016] Loss gs (x, y) = 1 - SSIM gs (x, y) + L1 gs (x, y) (2)

[0017] SSIM ( x, y) = c(x, y) + l(x, y) + s(x, y) (3)

[0018]

[0019]

[0020]

[0021] L1(x, y) = |x - y| (7)

[0022] where x represents the predicted output of the network, and y represents the high-contrast image corresponding to the low-contrast image; μx and μ y are the average values of x and y respectively, and σ x and σ y are the standard deviations of x and y respectively.

[0023] In the step S3, the ltmnet curve mapping is specifically as follows:

[0024] First, the HSV color space is separated into the H - S channel and the V channel, and a dual - branch network module is designed; the dual - branch network module extracts features respectively, and the results after the fifth - layer curve mapping of the two are downsampled multiple times to obtain mapping curves of different sizes, and the mapping curves of different sizes are used as the output of the network;

[0025] The curves of corresponding sizes obtained by the dual - branch network module are spliced to restore the three - channel curve form;

[0026] The learned three - channel curve is transferred to the post - processing part to enhance the mapping contrast of curves of different sizes.

[0027] The structure of the dual - branch network module is as follows:

[0028] The HSV - domain adjustable contrast enhancement ltmnet model includes a dual - branch feature extraction module for deep feature extraction and gradually learning the mapping curve; one branch is the H - S channel branch, and the other branch is the V channel branch. Each branch has the same structure, including a layer of convolution processing to map the input image channels to a four - channel space, including four feature learning sub - modules for deep feature extraction and gradually learning the mapping curve, including a layer of curve - transformation pooling convolution layer to map to a mapping curve space of 256, and four layers of maximum downsampling layers to obtain mapping curves of different sizes; the corresponding gs results (i.e., the results after mapping with different - sized grids) obtained by the dual - branch module are spliced to restore the three - channel mapping curve form;

[0029] The input is interpolated and post - processed, and the processed predicted image and the input are used for loss calculation to update the network; the low - contrast input image is interpolated and post - processed.

[0030] The network structure further includes:

[0031] ① The dual - branch feature extraction module of the HSV - domain adjustable contrast enhancement ltmnet model for deep feature extraction and gradually learning the mapping curve;

[0032] The input size of the H - S branch is (512, 512, 2), and the input of the V branch is (512, 512, 1);

[0033] Each branch has a similar structural composition, including a layer of convolutional processing. After mapping the input image channels to a four-channel space, it is sent to four feature learning sub-modules for deep feature extraction; gradually learning the mapping curve, including a layer of curve transformation pooling convolutional layer to map to a mapping curve space of 256, and four layers of max-pooling layers to obtain mapping curves of different sizes;

[0034] ② The dual-branch feature extraction sub-module includes a layer of max-pooling and a layer of convolutional processing;

[0035] Each branch has four sub-modules, and the output sizes of each layer are the same for both branches, which are (256, 256, 8), (128, 128, 16), (64, 64, 32), and (32, 32, 64);

[0036] ③ The HSV-domain adjustable contrast enhancement ltmnet model curve mapping module; maps the features learned above to the channel curve mapping space. That is, for the H-S branch, the result of (32, 32, 64) obtained by the feature learning module is downsampled to (16, 16, 64) and then mapped to (16, 16, 512) through a layer of convolution. For the V branch, it is mapped to (16, 16, 256);

[0037] ④ The mapping curve form transformation processing of the HSV-domain adjustable contrast enhancement ltmnet model; splices the curve mapping results of the H-S branch and the V branch to obtain (16, 16, 768);

[0038] ⑤ The four layers of max-pooling layers of the HSV-domain adjustable contrast enhancement ltmnet model to obtain mapping curves of different gs sizes. The output results of each layer are (8, 8, 768), (4, 4, 768), (2, 2, 768), and (1, 1, 768) respectively;

[0039] ⑥ The post-processing module of the HSV-domain adjustable contrast enhancement ltmnet model; changes the distribution of the mapping curve training results obtained from the initially spliced (16, 16, 768) and the four layers of max-pooling layers to (256, 3, 256), (64, 3, 256), (16, 3, 256), (4, 3, 256), and (1, 3, 256) respectively; at the same time, the input low-contrast image is sliced into 16, 8, 4, and 2 respectively, and interpolation mapping calculations are performed with the corresponding curves to obtain the predicted image.

[0040] In step S4, calculate the loss between the processed predicted image and the input, and update the network.

[0041] The method is for images with low contrast, low permeability, and low saturation.

[0042] Therefore, the advantages of this application are as follows: Deep learning contrast enhancement for low contrast in the HSV domain has good adaptive contrast enhancement for images in different scenarios, can effectively suppress overexposure in bright areas, and enhance details in dark areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings described herein are used to provide a further understanding of the present invention, form a part of this application, and do not limit the present invention.

[0044] Figure 1 It is a schematic diagram of the method flow of this application.

[0045] Figure 2 It is a schematic diagram of specific modifications in the ltmnet curve mapping method. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0046] In order to more clearly understand the technical content and advantages of the present invention, the present invention will be further described in detail below with reference to the drawings.

[0047] This application provides an ltmnet contrast enhancement method based on the HSV color domain. The process is as Figure 1 shown, and the specific steps are as follows:

[0048] Step S1: Collect and organize low-contrast - high-contrast image sample pairs, and construct a training set and a test set.

[0049] Step S2: Map the RGB image data of the sample pairs to the HSV color space, and at the same time, change the image size to (512, 512, 3), which is used as the input and output size for network training.

[0050] Step S3: Construct an adjustable network model for contrast enhancement in the HSV domain; by post-processing the cropping of the low-contrast - high-contrast data set, adjust the model structure to adapt to contrast enhancement in the HSV domain.

[0051] As Figure 2 shown, the specific modification in the ltmnet curve mapping algorithm is as follows: First, separate the HSV color space into the h-s channel and the v channel, and design a dual-branch network module; the dual-branch module performs feature extraction respectively, and the results after the fifth-layer curve mapping of the two are downsampled multiple times to obtain mapping curves of different sizes, and the mapping curves of different sizes are used as the output of the network; splice the corresponding-size curves obtained by the dual-branch module to restore the three-channel curve form; transfer the learned curve to the post-processing part to achieve contrast enhancement of curve mapping of different sizes.

[0052] The structure of the network is:

[0053] The HSV-domain adjustable contrast enhancement ltmnet model mainly includes a dual-branch feature extraction module for deep feature extraction and gradually learning the mapping curve. One branch is the H-S channel branch, and the other branch is the V channel branch. Each branch has the same structure, including a layer of convolution processing to map the input image channels to a four-channel space, four feature learning sub-modules for deep feature extraction and gradually learning the mapping curve, and a curve transformation pooling convolution layer to map to a mapping curve space of 256, and four max-pooling layers to obtain mapping curves of different sizes. The corresponding gs results obtained from the dual-branch module are concatenated to restore the three-channel mapping curve form. The predicted image obtained by interpolating and post-processing the input low-contrast image is used to calculate the loss with the high-contrast image to update the network.

[0054] Further, the network structure includes:

[0055] ① The dual-branch feature extraction module of the HSV-domain adjustable contrast enhancement ltmnet model for deep feature extraction and gradually learning the mapping curve;

[0056] The input size of the H-S branch is (512, 512, 2), and the input of the V branch is (512, 512, 1);

[0057] Each branch has a similar structural composition, including a layer of convolution processing to map the input image channels to a four-channel space and then send them into four feature learning sub-modules for deep feature extraction;

[0058] Gradually learn the mapping curve, including a curve transformation pooling convolution layer to map to a mapping curve space of 256, and four max-pooling layers to obtain mapping curves of different sizes;

[0059] ② The dual-branch feature extraction sub-module, including a layer of max-pooling and a layer of convolution processing;

[0060] Each branch has four sub-modules, and the output size of each layer of the two branches is the same, which are (256, 256, 8), (128, 128, 16), (64, 64, 32), and (32, 32, 64) respectively;

[0061] ③ The HSV-domain adjustable contrast enhancement ltmnet model curve mapping module; map the learned features to the channel curve mapping space, that is, for the H-S branch, the result of (32, 32, 64) obtained from the feature learning module is downsampled to (16, 16, 64) and then mapped to (16, 16, 512) through a layer of convolution. For the V branch, it is mapped to (16, 16, 256);

[0062] ④Morphological transformation processing of the mapping curve of the HSV-domain adjustable contrast enhancement ltmnet model; splicing the mapping results of the H-S branch and the V-branch curves to obtain (16, 16, 768);

[0063] ⑤Four-layer maximum downsampling layer of the HSV-domain adjustable contrast enhancement ltmnet model to obtain mapping curves of different gs sizes, and the output results of each layer are (8, 8, 768), (4, 4, 768), (2, 2, 768), (1, 1, 768) respectively;

[0064] ⑥Post-processing module of the HSV-domain adjustable contrast enhancement ltmnet model; the distribution of the training results of the mapping curves obtained from the initial splicing of (16, 16, 768) and the four-layer maximum downsampling layer are respectively changed to (256, 3, 256), (64, 3, 256), (16, 3, 256), (4, 3, 256), (1, 3, 256); at the same time, the input low-contrast image is sliced into 16, 8, 4, and 2 respectively, and interpolation mapping calculations are performed with the corresponding curves to obtain the predicted image.

[0065] Step S4: Use L1 loss and ssim to calculate the loss between the results obtained for each gs and the corresponding gt with different enhancement degrees, and sum them as the loss for parameter update.

[0066] Loss(x, y) = sum(loss gs ) for gs = 1, 2, 4, 8, 16 (1)

[0067] Loss gs (x, y) = 1 - SSIM gs (x, y) + L1 gs (x, y) (2)

[0068] SSIM(x, y) = c(x, y) + l(x, y) + s(x, y) (3)

[0069]

[0070]

[0071]

[0072] L1(x, y) = |x - y| (7)

[0073] where x represents the predicted output of the network, and y represents the high-contrast image corresponding to the low-contrast image. μ x and μ y are the average values of x and y respectively, and σ x and σ yThey are the standard deviations of x and y respectively.

[0074] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. An ltmnet contrast enhancement method based on the HSV color domain, characterized in that, The method includes the following steps: S1: Collect and organize low-contrast / high-contrast image sample pairs, and construct a training set and a test set; S2: Map the RGB image data of the sample pairs to the HSV color space, and at the same time change the image size to (512, 512, 3), which serves as the input and output size for network training; S3: Construct an adjustable ltmnet network model for contrast enhancement in the HSV domain; adjust the model structure to adapt to contrast enhancement in the HSV domain; perform ltmnet curve mapping on the low-contrast image; S4: Use L1 loss and ssim to calculate the loss between the results after mapping each grid of different sizes and the high-quality images with corresponding different enhancement degrees, and sum them as the loss for parameter update; Loss(x, y) = sum(loss gs ) where gs = 1, 2, 4, 8, 16 (1) Loss gs (x, y) = 1 - SSIM gs (x, y) + L1 gs (x, y) (2) SSIM(x, y) = c(x, y) + l(x, y) + s(x, y) (3) L1(x, y) = |x - y| (7) where x represents the predicted output of the network, y represents the high-contrast image corresponding to the low-contrast image; μ x and μ y are the average values of x and y respectively, and σ x and σ y are the standard deviations of x and y respectively.

2. The ltmnet contrast enhancement method based on the HSV color domain according to claim 1, wherein In the step S3, the ltmnet curve mapping is specifically as follows: First, separate the HSV color space into the H-S channel and the V channel, and design a dual-branch network module; The dual-branch network module respectively performs feature extraction, and the results after the fifth-layer curve mapping of the two are downsampled multiple times to obtain mapping curves of different sizes, and the mapping curves of different sizes are used as the output of the network; Stitch the curves of corresponding sizes obtained by the dual-branch network module to restore the three-channel curve form; Transfer the learned three-channel curve to the post-processing part to achieve contrast enhancement of curve mapping of different sizes.

3. The ltmnet contrast enhancement method based on the HSV color domain according to claim 2, wherein The structure of the dual-branch ltmnet network module is as follows: The HSV-domain adjustable contrast enhancement ltmnet model includes a dual-branch feature extraction module for deep feature extraction and gradually learning the mapping curve; one branch is the H-S channel branch, and the other branch is the V channel branch. Each branch has the same structure, including one layer of convolution processing to map the input image channels to a four-channel space, including four layers of feature learning sub-modules for deep feature extraction and gradually learning the mapping curve, including one layer of curve transformation pooling convolution layer to map to a mapping curve space of 256, and four layers of maximum downsampling layers to obtain mapping curves of different sizes; stitch the corresponding results of different grids obtained by the dual-branch module to restore the three-channel mapping curve form; perform interpolation post-processing on the input.

4. A method for enhancing the contrast of ltmnet based on the HSV color domain according to claim 3, characterized in that, The network structure further includes: ① The dual-branch feature extraction module of the HSV-domain adjustable contrast enhancement ltmnet model for deep feature extraction and gradually learning the mapping curve; The input size of the H-S branch is (512, 512, 2), and the input of the V branch is (512, 512, 1); Each branch has a similar structural composition, including one layer of convolution processing to map the input image channels to a four-channel space and then send them into four layers of feature learning sub-modules for deep feature extraction; Gradually learn the mapping curve, including one layer of curve transformation pooling convolution layer to map to a mapping curve space of 256, and four layers of maximum downsampling layers to obtain mapping curves of different sizes; ② The dual-branch feature extraction sub-module includes one layer of max pooling and one layer of convolution processing; Each branch has four sub-modules. The output sizes of the two branches are the same for each layer, which are (256, 256, 8), (128, 128, 16), (64, 64, 32), and (32, 32, 64) respectively; ③HSV domain adjustable contrast enhancement ltmnet model curve mapping module; Map the learned features above to the channel curve mapping space. That is, for the H-S branch, the result of (32, 32, 64) obtained by the feature learning module is downsampled to (16, 16, 64) and then mapped to (16, 16, 512) through one layer of convolution. For the V branch, it is mapped to (16, 16, 256); ④HSV domain adjustable contrast enhancement ltmnet model mapping curve morphology transformation processing; Concatenate the curve mapping results of the H-S branch and the V branch to obtain (16, 16, 768); ⑤HSV domain adjustable contrast enhancement ltmnet model four-layer maximum downsampling layer to obtain mapping curves of different gs sizes. The output results of each layer are (8, 8, 768), (4, 4, 768), (2, 2, 768), and (1, 1, 768) respectively; ⑥HSV domain adjustable contrast enhancement ltmnet model post-processing module; The distributions of the initial concatenated result of (16, 16, 768) and the mapping curve training results obtained by the four-layer maximum downsampling layer are changed to (256, 3, 256), (64, 3, 256), (16, 3, 256), (4, 3, 256), and (1, 3, 256) respectively. At the same time, the input low-contrast image is sliced into 16, 8, 4, and 2 respectively, and interpolation mapping calculations are performed with the corresponding curves to obtain the predicted image.

5. The ltmnet contrast enhancement method based on the HSV color domain according to claim 3, characterized in that, In step S4, calculate the loss between the processed predicted image and the input, and update the network.

6. The ltmnet contrast enhancement method based on the HSV color domain according to claim 1, wherein, The method is for images with low contrast, low permeability, and low saturation.

Images