Optimized LTMNET-based network weight fusion contrast enhancement method
Through deep learning network and weight fusion technology, adaptive contrast enhancement is performed on low-contrast images, which solves the problem that contrast enhancement methods in the prior art cannot adapt and adjust the enhancement intensity, achieves the effect of visible details in dark areas and unexposed bright areas, and improves AIISP performance.
Patent Information
- Application Number
- CN202311066474.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-23
- Publication Date
- 2025-02-28
AI Technical Summary
In the prior art, the contrast enhancement method is difficult to effectively adjust the enhancement intensity, and cannot adapt to it in different scenarios, resulting in invisible details in dark areas or overexposure in bright areas.
By inputting low-contrast images into deep learning networks for learning, using interpolation post-processing and weight fusion technology, the balance between dark and bright areas is adjusted, thereby achieving contrast enhancement. Specific steps include collecting low-contrast images, enhancing them using the CLAHE method, building an LTMNET network model, performing feature learning and curve mapping, using fusion weights for fusion, and calculating the loss update parameters.
Adaptive contrast enhancement of low-contrast images in the RGB domain is achieved, ensuring that dark area details are visible and bright area is not exposed, improving AIISP performance and adjusting the intensity of contrast enhancement.
Smart Images
Figure CN119540105A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image processing, and particularly relates to a method for enhancing contrast based on weight fusion of an optimized LTMNET network. Background Art
[0002] In the prior art, with the wide application of computer vision technology, high-quality images are not only used for people's entertainment, but also many visual tasks require high-quality images as support to be realized. And contrast enhancement plays an important role in improving the visual quality of computer vision, pattern recognition and digital image processing. In actual scenarios, due to various conditions such as harsh shooting environments and difficult camera parameter settings, the images obtained by imaging devices may have problems such as low contrast and lack of details. At present, low-contrast image enhancement mainly uses traditional algorithms or is learned as a subtask of deep learning image enhancement, which is relatively cumbersome and has an obvious brightness improvement effect, and cannot effectively adjust the enhancement intensity.
[0003] The disadvantages of the prior art are as follows:
[0004] There are many classic enhancement methods in the current contrast enhancement research, such as gamma transformation for global gray-scale adjustment, histogram equalization, and local gray-scale adjustment, such as contrast-limited adaptive histogram equalization and adaptive contrast enhancement. These methods can only be used for single-image processing, and the same parameters cannot adapt to different scenarios. In the research of deep learning image enhancement, contrast enhancement is mostly an additional direction, and the enhancement effect is not good for underexposed and overexposed areas, that is, the details in the dark areas are invisible, the bright areas are overexposed, and there is a lack of targeted contrast improvement performance.
[0005] In addition, the common terms in the prior art include:
[0006] Contrast: The measurement of different brightness levels between the brightest white and the darkest black in the light and dark areas of an image.
[0007] RGB: The three primary colors of light, where R, G, and B represent red, green, and blue respectively.
[0008] CLAHE: Contrast Limited Adaptive Histogram Equalization algorithm. Summary of the Invention
[0009] In order to solve the above problems, the purpose of this application is to propose a method of sending a low-contrast image into a deep learning network to learn the mapping curve, obtaining a high-contrast image through interpolation post-processing, and then balancing the dark and bright areas through a weight fusion network to obtain an RGB-domain contrast enhancement technical method with visible details in the dark areas and no overexposure in the bright areas.
[0010] Specifically, the present invention provides a method for optimizing the weight fusion contrast enhancement based on the LTMNET network, and the method includes the following steps:
[0011] S1, low-contrast image: collect and organize low-contrast images; use the CLAHE contrast enhancement method, and set the enhancement parameters (block, threshold) to (1, 0.6), (1, 0.8), (1, 1.2), (2, 1.2), (2, 1.6) respectively to perform five different degrees of enhancement on the low-contrast images to form sample pairs;
[0012] S2, crop (512, 512, 3): convert the image data of the sample pairs into a size of (512, 512, 3) as the input and output dimensions for network training;
[0013] S3, construct an LTMNET network model with adjustable contrast enhancement in the RGB domain:
[0014] S3.1, LTMNET network;
[0015] S3.2, interpolation post-processing;
[0016] S3.3, weight fusion;
[0017] S3.4, calculate the loss.
[0018] Among them, the step S3 further includes:
[0019] S3: By post-processing the cropped low-contrast - high-contrast data set, adjust the model structure to adapt to the weight fusion adjustable contrast enhancement; specifically modified in the LTMNET network: the LTMNET weight fusion network structure includes: one layer of convolution processing, mapping the input three-channel image processing to a four-channel space; four layers of feature learning modules for deep feature extraction and gradually learning the mapping curve; one layer of curve transformation pooling convolution layer to map to the mapping curve space of 3 * 256; four layers of maximum downsampling layers to obtain mapping curves of different sizes; directly perform four downsamplings on the result of the fifth layer of curve mapping to obtain mapping curves of different sizes, and the mapping curves of different sizes are used as the outputs of the network at the same time;
[0020] Transfer the curve shape transformation operation from the inside of the network to the post-processing part to achieve contrast enhancement of mapping curves of different sizes;
[0021] Fuse the mapping results of different curves using the fusion weights to obtain an enhanced effect that combines global and local information; at the same time, calculate the loss with the image with the optimal enhancement intensity and update the network parameters.
[0022] The step S3.1 further includes:
[0023] The adjustable contrast enhancement LTMNET weight fusion model includes
[0024] One layer of convolution processing, mapping the input three-channel image processing to a four-channel space;
[0025] Four layers of feature learning modules for deep feature extraction and gradually learning the mapping curve;
[0026] One layer of curve transformation pooling convolution layer to map to the mapping curve space of 3*256;
[0027] Four layers of max-pooling layers to obtain mapping curves of different sizes.
[0028] The step S3.1 further includes:
[0029] The adjustable contrast enhancement LTMNET model feature learning module, each layer contains one layer of max-pooling and one layer of convolution processing; the output sizes of each layer are (256, 256, 8), (128, 128, 16), (64, 64, 32), (32, 32, 64);
[0030] The fifth layer curve mapping module of the adjustable contrast enhancement LTMNET model maps the learned features to the RGB three-channel curve mapping space, that is, the result of (32, 32, 64) obtained by the feature learning module is downsampled to (16, 16, 64) and then mapped to (16, 16, 768) through one layer of convolution;
[0031] The four layers of max-pooling layers of the adjustable contrast enhancement LTMNET model respectively obtain mapping curves of (8, 8, 768), (4, 4, 768), (2, 2, 768), (1, 1, 768), and the network finally returns the mapping curves of (16, 16, 768), (8, 8, 768), (4, 4, 768), (2, 2, 768), (1, 1, 768);
[0032] The post-processing module of the adjustable contrast enhancement LTMNET model transforms the shapes of the five curves, and the distributions become (256, 3, 256), (64, 3, 256), (16, 3, 256), (4, 3, 256), (1, 3, 256). At the same time, the input low-contrast image is respectively sliced with block sizes of 16, 8, 4, and 2, and interpolation mapping calculations are performed with the corresponding curves to obtain the predicted image, where the original image is directly globally mapped with (1, 3, 256).
[0033] The step S3.3 further includes fusing the mapped predicted images using the fusion weights, and using 0.2 to fuse the five images during the training process, as shown in formula (1),
[0034] Pred_im = 0.2 * Interp(x 16 ) + 0.2 * Interp(x 8 ) + 0.2 * Interp(x 4 ) + 0.2 * Interp(x 2 ) + 0.2 * Interp(x 1 ) (1)
[0035] Among them, Pred_im represents the final fused prediction, and gt_im represents the optimal high-contrast image corresponding to the low-contrast image.
[0036] The step S3.4 further includes finally calculating the loss using L1loss and vggloss as the loss for the network to calculate the gradient, as shown in formula (2):
[0037] Loss(pred_im, gt_im) = 1 - Vgg(pred_im, gt_im) + L1(pred_im, gt_im) (2); where Vgg() represents the vgg loss calculation function.
[0038] Therefore, the advantages of this application are as follows:
[0039] Deep learning contrast enhancement is performed on low-contrast in the RGB domain, which has good adaptive contrast enhancement for images in different scenarios, achieving the effect that details in dark areas are visible and bright areas are not overexposed, and finally improving the AIISP performance; at the same time, the intensity of contrast enhancement can be adjusted. Brief Description of the Drawings
[0040] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation to the present invention.
[0041] Figure 1 It is a schematic flowchart of the method embodiment of this application.
[0042] Figure 2 It is a schematic diagram of the specific modification in the LTMNET network in the method embodiment of this application. Detailed Embodiment
[0043] In order to be able to more clearly understand the technical content and advantages of the present invention, the present invention will now be further described in detail with reference to the drawings.
[0044] The embodiment of this application proposes a method for contrast enhancement based on the fusion of optimized LTMNET network weights, as Figure 1 shown, the method includes the following steps:
[0045] S1, Low-contrast images: Collect and organize low-contrast images; use the CLAHE contrast enhancement method and set the enhancement parameters (block, threshold) to (1, 0.6), (1, 0.8), (1, 1.2), (2, 1.2), and (2, 1.6) respectively to enhance the low-contrast images to five different degrees and form sample pairs.
[0046] S2, Crop to (512, 512, 3): Convert the image data of the sample pairs to a size of (512, 512, 3) as the input and output dimensions for network training.
[0047] S3, Build the LTMNET network model with adjustable contrast enhancement in the RGB domain:
[0048] S3.1, LTMNET network;
[0049] S3.2, Interpolation post-processing;
[0050] S3.3, Weight fusion;
[0051] S3.4, Calculate the loss.
[0052] Specifically, the steps further include the following:
[0053] Step S1: Collect and organize low-contrast images; use the CLAHE contrast enhancement method and set different thresholds to enhance the low-contrast images to five different degrees and form sample pairs. Step S2: Uniformly convert the image data of the sample pairs to a size of (512, 512, 3) as the input and output dimensions for network training.
[0054] Step S3: Build the LTMNET network model with adjustable contrast enhancement in the RGB domain. Through post-processing the low-contrast - high-contrast data set cropping, adjust the model structure to adapt to weight fusion adjustable contrast enhancement. As Figure 2As shown in the figure, the specific modification in the LTMNET network is as follows: The LTMNET network structure includes: one layer of convolution processing, which maps the input three-channel image processing to a four-channel space; four layers of feature learning modules for deep feature extraction and gradually learning the mapping curve; one layer of curve transformation pooling convolution layer to map to a 3*256 mapping curve space; four layers of max-pooling layers to obtain mapping curves of different sizes. The result after the fifth layer of curve mapping is directly downsampled multiple times to obtain mapping curves of different sizes, and the mapping curves of different sizes are used as the output of the network at the same time; the curve shape transformation operation is transferred from the inside of the network to the post-processing part to enhance the mapping contrast of curves of different sizes. The mapping results of different curves are fused using the fusion weight to obtain an enhanced effect that combines global and local information; at the same time, loss calculation is performed with the image with the optimal enhancement intensity to update the network parameters.
[0055] The specific structure of the network is as follows:
[0056] Step S3.1, the adjustable contrast enhancement LTMNET model mainly includes one layer of convolution processing, which maps the input three-channel image processing to a four-channel space; four layers of feature learning modules for deep feature extraction and gradually learning the mapping curve; one layer of curve transformation pooling convolution layer to map to a 3*256 mapping curve space; four layers of max-pooling layers to obtain mapping curves of different sizes; perform interpolation post-processing on the input, fuse the processed predicted image, perform loss calculation with the optimal enhanced image, and update the network.
[0057] The feature learning module of the adjustable contrast enhancement LTMNET model includes one layer of max-pooling and one layer of convolution processing in each layer; the output size of each layer is (256, 256, 8), (128, 128, 16), (64, 64, 32), (32, 32, 64) respectively.
[0058] The fifth layer curve mapping module of the adjustable contrast enhancement LTMNET model maps the learned features to the RGB three-channel curve mapping space, that is, the result of (32, 32, 64) obtained by the feature learning module is downsampled to (16, 16, 64) and then mapped to (16, 16, 768) through one layer of convolution.
[0059] The four layers of max-pooling layers of the adjustable contrast enhancement LTMNET model obtain mapping curves of (8, 8, 768), (4, 4, 768), (2, 2, 768), (1, 1, 768) respectively, and the network finally returns the mapping curves of (16, 16, 768), (8, 8, 768), (4, 4, 768), (2, 2, 768), (1, 1, 768).
[0060] Step S3.2: The post - processing module of the adjustable contrast enhancement LTMNET model transforms the shapes of the five curves, and the distributions become (256, 3, 256), (64, 3, 256), (16, 3, 256), (4, 3, 256), (1, 3, 256). At the same time, the input low - contrast image is segmented into blocks with sizes of 16, 8, 4, and 2 respectively, and interpolation mapping calculations are performed with the corresponding curves to obtain the predicted images. Among them, the original image is directly subjected to global mapping with (1, 3, 256).
[0061] Steps S3.3 and S3.4: The predicted images after mapping are fused using the fusion weights. During the training process, 0.2 is used to fuse the five images, as shown in formula (1). Finally, L1 loss and vgg loss are used for loss calculation, which serves as the loss for the network to calculate the gradient, as shown in formula (2).
[0062] Pred_im = 0.2 * Interp(x 16 ) + 0.2 * Interp(x 8 ) + 0.2 * Interp(x 4 ) + 0.2 * Interp(x 2 ) + 0.2 * Interp(x 1 ) (1)
[0063] Loss(pred_im, gt_im) = 1 - Vgg(pred_im, gt_im) + L1(pred_im, gt_im) (2)
[0065] Among them, Pred_im represents the final fused prediction, gt_im represents the optimal high - contrast image corresponding to the low - contrast image, and Vgg() represents the vgg loss calculation function. That is, in S3.3 and S3.4, the curves predicted by the network are used to perform interpolation post - processing on the input, the processed predicted images are fused, and loss calculation is performed with the optimal enhanced image to update the network.
[0066] The above is only the preferred embodiment of the present invention and is not used to limit the present invention. For those skilled in the art, various changes and modifications can be made to the embodiments of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.
Claims
1. A method for contrast enhancement based on optimized LTMNET network weight fusion, characterized in that: The method comprises the following steps: S1, low contrast image: collect and organize low contrast images; use the CLAHE contrast enhancement method, set the enhancement parameters (block,threshold) to (1,0.6), (1,0.8), (1,1.2), (2,1.2), (2,1.6) to enhance the low contrast images to five different degrees, forming sample pairs; S2, cropping (512, 512, 3): convert the image data of the sample pair to the size of (512, 512, 3) as the input and output size of network training; S3, build an LTMNET weight fusion network model with adjustable RGB domain contrast enhancement: S3.1, LTMNET weight fusion network; S3.2, interpolation post-processing; S3.3, weight fusion; S3.4, calculate the loss.
2. The method for contrast enhancement based on optimized LTMNET network weight fusion according to claim 1, characterized in that: The step S3 further comprises: S3: By post-processing the low-contrast-high-contrast data set, the model structure is adjusted to adapt to the weight fusion adjustable contrast enhancement; in the LTMNET network, it is specifically modified as follows: the LTMNET weight fusion network structure includes: a layer of convolution processing, which maps the input three-channel image processing to a four-channel space; a four-layer feature learning module, which performs deep feature extraction and gradually learns the mapping curve; a layer of curve conversion pooling convolution layer to map to a 3*256 mapping curve space; four layers of maximum downsampling layers to obtain mapping curves of different sizes; the result after the fifth layer of curve mapping is directly downsampled four times to obtain mapping curves of different sizes, and mapping curves of different sizes are simultaneously used as the output of the network; Move the curve morphology transformation operation from the network to the post-processing part to achieve contrast enhancement of curve mapping of different sizes; The different curve mapping results are fused using fusion weights to obtain the enhancement effect of integrating global and local information. At the same time, the loss is calculated with the image with the best enhancement strength to update the network parameters.
3. A method for contrast enhancement based on optimized LTMNET network weight fusion according to claim 2, characterized in that: The step S3.1 further comprises: The adjustable contrast enhancement LTMNET model contains One layer of convolution processing maps the input three-channel image to a four-channel space; Four-layer feature learning module, performs deep feature extraction and gradually learns the mapping curve; A layer of curve transformation pooling convolution layer to map to a 3*256 mapping curve space; Four maximum downsampling layers are used to obtain mapping curves of different sizes.
4. The method for contrast enhancement based on optimized LTMNET network weight fusion according to claim 3 is characterized in that: The step S3.1 also includes: Adjustable contrast enhancement LTMNET model feature learning module, each layer contains a layer of maximum pooling and a layer of convolution processing; the output size of each layer is (256,256,8), (128,128,16), (64,64,32), (32,32,64) respectively; The fifth layer curve mapping module of the adjustable contrast enhancement LTMNET model maps the above learned features to the RGB three-channel curve mapping space, that is, the (32,32,64) result obtained by the feature learning module is downsampled to (16,16,64) and then mapped to (16,16,768) through a layer of convolution; The four maximum downsampling layers of the adjustable contrast enhancement LTMNET model obtain the mapping curves of (8, 8, 768), (4, 4, 768), (2, 2, 768), and (1, 1, 768), respectively. The network finally returns the mapping curves of (16, 16, 768), (8, 8, 768), (4, 4, 768), (2, 2, 768), and (1, 1, 768); The post-processing module of the adjustable contrast enhancement LTMNET model transforms the shapes of the five curves into (256, 3, 256), (64, 3, 256), (16, 3, 256), (4, 3, 256), and (1, 3, 256). At the same time, the input low-contrast image is divided into blocks of size 16, 8, 4, and 2, respectively, and the predicted image is obtained by interpolation mapping with the corresponding curves, in which the original image is directly globally mapped with (1, 3, 256).
5. A method for contrast enhancement based on optimized LTMNET network weight fusion according to claim 4, characterized in that: The step S3.3 further includes fusing the mapped predicted images using a fusion weight, and fusing the five images using 0.2 during the training process, as shown in formula (1). Pred_im=0.2*Interp(x 16 )+0.2*Interp(x8)+0.2*Interp(x4)+0.2*Interp(x2)+0.2*Interp(x1) (1) Among them, Pred_im represents the final fusion prediction, and gt_im represents the optimal high-contrast image corresponding to the low-contrast image.
6. A method for contrast enhancement based on optimized LTMNET network weight fusion according to claim 5, characterized in that: The step S3.4 further includes, finally, using L11oss and vggloss to perform loss calculation as the loss of the network calculation gradient, such as formula (2): Loss(pred_im, gt_im)=1-Vgg(pred_im, gt_im)+L1(pred_im, gt_im) (2); Among them, Vgg() represents the Vgg loss calculation function.