A tone mapping method and system based on a lookup table and laplacian filtering

Abstract

The application discloses a tone mapping method and system based on a lookup table and a Laplace filter, and belongs to the technical field of image processing.The input image is first decomposed into an adaptive Laplace pyramid;then a low-frequency image at the bottom layer of the adaptive Laplace pyramid is input into a weight predictor to obtain a pixel-level weight map, and a three-dimensional lookup table is used to perform three-linear interpolation on the low-frequency image to obtain a preliminary mapping image; then the preliminary mapping image and the weight map are fused pixel by pixel to generate an image at the bottom layer of the fine-tuned Laplace pyramid; then a filter parameter prediction module is used to learn a parameter value map of the remaining layer images of the adaptive Laplace pyramid, and the parameter value map is applied to a local Laplace filter to obtain the remaining layer images of the fine-tuned Laplace pyramid; finally, the fine-tuned Laplace pyramid is reconstructed to obtain a tone-mapped image.The application can retain local edge details of an image while performing global tone mapping.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and more specifically, relates to a tone mapping method and system based on lookup tables and Laplacian filtering. Background Technology

[0002] Tone mapping technology renders high dynamic range (HDR) images to the displayable range of a display device while preserving important image details and color appearance for appreciating the original scene content. Traditional tone mapping algorithms can be divided into global and local algorithms based on their processing method. Global algorithms map each pixel based on global features, regardless of its spatial location. The advantage of global methods is that they effectively avoid overexposure and underexposure of mapped pixels and have high computational efficiency. However, global methods use non-local operations, meaning they apply the same compression processing to all regions of the image. Furthermore, their color range shrinkage property means that their output cannot maintain sufficient contrast in the image, resulting in a somewhat faded scene. In contrast, local methods adaptively adjust the tone mapping operation based on the local content of the image. Local algorithms consider the spatial location of each pixel in the image, calculate brightness adaptability based on the specific location of each pixel, and process accordingly. Therefore, images generated by local methods are closer to human visual perception and have richer details, but compared to global algorithms, local methods take longer to process and are more prone to artifacts in the image. However, these traditional algorithms typically require manual tuning by experienced engineers, a tedious task for situations where evaluation results need to be tested across various scenarios. While some system contributions aim to simplify the implementation of high-performance executables, they still require programming expertise, incur runtime costs as pipeline complexity increases, and are only applicable when filter source code is available. Therefore, seeking automatic high dynamic range image tone mapping strategies is of great significance.

[0003] In recent years, with the rapid development of deep learning technology, significant progress has been made in deep learning-based automatic tone mapping methods. Many of these methods focus on learning pixel-level dense mappings between high dynamic range (HDR) and low dynamic range (LVR) image pairs, or learning pixel-level transformations to map input HDR images. However, these studies face considerable challenges in terms of computational burden, which increases linearly with the size of the input image. To address this, industry has proposed a novel method based on image adaptive 3D lookup tables (3DLUTs), which offers good image quality, efficient computation, and minimal memory usage. However, utilizing a global (spatially uniform) tone mapping operator may produce unsatisfactory results in local regions. Furthermore, this method requires an initial downsampling step to reduce network computation. For high-resolution (4K) images, this downsampling process requires a significant reduction in image resolution, up to 16 times (typically downsampling to 256×256 resolution). This leads to a significant loss of image detail in the generated image and a decrease in enhancement performance. Summary of the Invention

[0004] In view of the above-mentioned defects or improvement needs of the existing technology, the present invention provides a tone mapping method and system based on lookup table and Laplacian filtering, which aims to solve the technical problems of excessive computation and easy loss of image details in the existing high dynamic range image tone mapping methods.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a tone mapping method based on lookup tables and Laplace filtering, the method comprising:

[0006] (1) Decompose the input image into an adaptive Laplacian pyramid;

[0007] (2) The low-frequency image of the bottom layer of the adaptive Laplacian pyramid is passed through a Transformer-based weight predictor to obtain a pixel-level weight map. At the same time, a three-dimensional lookup table is used to perform trilinear interpolation on the low-frequency image to obtain a preliminary mapping map. The preliminary mapping map is then fused with the weight map pixel by pixel to generate the fine-tuned bottom layer image of the Laplacian pyramid.

[0008] (3) The parameter value map of the adaptive Laplacian pyramid residual layer image is learned by the CNN-based filter parameter prediction module, and the parameter value map is applied to the local Laplacian filter to obtain the fine-tuned Laplacian pyramid residual layer image.

[0009] (4) The image after tone mapping is obtained by reconstructing the finely adjusted Laplacian pyramid.

[0010] Preferably, step (2) specifically involves feeding the low-frequency image of the bottom layer of the adaptive Laplacian pyramid into a Transformer-based weight predictor to obtain three pixel-level weight maps. Simultaneously, the low-frequency image is trilinearly interpolated with three basic three-dimensional lookup tables to obtain three preliminary mapping maps. The three preliminary mapping maps are then multiplied by the three pixel-level weight maps respectively. Finally, the three products are added together to obtain the fine-tuned bottom layer image of the Laplacian pyramid.

[0011] Preferably, in step (3), the fine-tuned Laplacian pyramid sub-bottom image is obtained through the following method:

[0012] The edge image obtained by passing the low-frequency image of the bottom layer of the fine-tuned Laplacian pyramid through the Canny edge detection operator is then upsampled along with the low-frequency image of the bottom layer of the adaptive Laplacian pyramid, so that the size of the upsampled image is consistent with the size of the second-to-bottom layer image of the Laplacian pyramid. The two upsampled images and the second-to-bottom layer image of the adaptive Laplacian pyramid are then fed into a CNN-based filter parameter prediction module to obtain a parameter value map. The parameter value map is then applied to a local Laplacian filter to obtain the fine-tuned second-to-bottom layer image of the Laplacian pyramid.

[0013] Preferably, in step (3), the images of the layers other than the bottom and second-to-bottom layers in the fine-tuned Laplacian pyramid are obtained through the following method:

[0014] The image of the previous layer of the fine-tuned Laplacian pyramid is upsampled so that the size of the upsampled image is the same as the size of the current layer image. The upsampled image and the image of the same layer of the adaptive Laplacian pyramid are then fed into the filter parameter prediction module based on CNN to obtain the parameter value map. The parameter value map is then applied to the local Laplacian filter to obtain the image of the current layer of the fine-tuned Laplacian pyramid.

[0015] Preferably, the CNN network includes two 3×3 convolutional layers and one 1×1 convolutional layer.

[0016] Preferably, the adaptive Laplacian pyramid uses a dynamically adjusted number of decomposition layers to match the resolution of the input image.

[0017] Secondly, the present invention provides a tone mapping system based on lookup tables and Laplace filtering, the system comprising:

[0018] The image decomposition module is used to decompose the input image into an adaptive Laplacian pyramid;

[0019] The bottom-level image generation module is used to obtain a pixel-level weight map by passing the low-frequency image of the bottom layer of the adaptive Laplacian pyramid through a Transformer-based weight predictor. At the same time, a three-dimensional lookup table is used to perform trilinear interpolation on the low-frequency image to obtain a preliminary mapping map. Then, the preliminary mapping map and the weight map are fused pixel by pixel to generate the fine-tuned bottom-level image of the Laplacian pyramid.

[0020] The remaining layer image generation module is used to learn the parameter value map of the adaptive Laplacian pyramid remaining layer image through the CNN-based filter parameter prediction module, and apply the parameter value map to the local Laplacian filter to obtain the fine-tuned Laplacian pyramid remaining layer image.

[0021] The image reconstruction module is used to reconstruct the tone-mapped image from the finely adjusted Laplacian pyramid.

[0022] Preferably, the bottom-level image generation module is specifically used to feed the low-frequency image of the bottom layer of the adaptive Laplacian pyramid into a Transformer-based weight predictor to obtain three pixel-level weight maps. At the same time, the low-frequency image is trilinearly interpolated with three basic three-dimensional lookup tables to obtain three preliminary mapping maps. The three preliminary mapping maps are then multiplied with the three pixel-level weight maps respectively. Finally, the three products are added together to obtain the fine-tuned bottom-level image of the Laplacian pyramid.

[0023] Preferably, the remaining layer image generation module includes a sub-bottom layer image generation unit. The sub-bottom layer image generation unit is used to obtain an edge image from the low-frequency image of the bottom layer of the fine-tuned Laplacian pyramid through the Canny edge detection operator, upsample the edge image and the low-frequency image of the bottom layer of the adaptive Laplacian pyramid so that the size of the upsampled image is consistent with the size of the sub-bottom layer image of the Laplacian pyramid, and then send the two upsampled images and the adaptive Laplacian pyramid sub-bottom layer image together into a CNN-based filter parameter prediction module to obtain a parameter value map. The parameter value map is then applied to a local Laplacian filter to obtain the fine-tuned sub-bottom layer image of the Laplacian pyramid.

[0024] Preferably, the remaining layer image generation module further includes other layer image generation units. These other layer image generation units are used to generate images of other layers in the fine-tuned Laplacian pyramid except for the bottom and second-bottom layers. Specifically, they are used to upsample the previous layer image of the fine-tuned Laplacian pyramid so that the size of the upsampled image is consistent with the size of the current layer image. Then, the upsampled image and the same layer image of the adaptive Laplacian pyramid are sent together into the CNN-based filter parameter prediction module to obtain a parameter value map. The parameter value map is then applied to the local Laplacian filter to obtain the current layer image of the fine-tuned Laplacian pyramid.

[0025] Thirdly, the present invention realizes an electronic device, comprising:

[0026] Memory, used to store programs;

[0027] A processor for executing a program stored in the memory, wherein when the program stored in the memory is executed, the processor is configured to perform any of the methods in the first aspect.

[0028] Fourthly, the present invention provides a computer-readable storage medium storing a computer program, characterized in that, when the computer program is run on a processor, the processor performs any of the methods in the first aspect.

[0029] In summary, the technical solutions conceived by this invention have the following beneficial effects compared with the prior art:

[0030] (1) The input image and the reference image present a multi-pixel mapping relationship, that is, the input image has the same pixel value at some different locations, while these locations contain different pixel values ​​in the reference image. Therefore, the traditional three-dimensional lookup table framework cannot adapt to this complex multi-pixel mapping relationship, thus affecting its performance in accurately representing this pixel transformation; this invention uses a pixel-level weight map to replace the image-level weight and the mapping map obtained by trilinear interpolation for pixel-by-pixel fusion, accurately expressing the multi-pixel mapping relationship;

[0031] (2) In order to better extract information from low-resolution pyramid images and perform global tone mapping, a Transformer network is introduced to generate weight predictions for the bottom Laplacian pyramid. Since the weight predictor aims to understand the global content, such as the overall brightness, color and hue of the image, the Transformer network has certain advantages in extracting global information compared with traditional convolutional neural networks. It can capture the long-range dependencies of different locations in the image, thereby better understanding the overall content of the image. This is very important for the global tone mapping task. Therefore, introducing the Transformer network as the backbone network can improve the performance and effectiveness of the algorithm.

[0032] (3) Local Laplacian filter is a commonly used image enhancement method that can highlight edge information in an image. However, in traditional methods, the parameter values ​​of the filter are usually fixed and lack adaptability. This invention proposes an image-adaptive learnable local Laplacian filter for thinning high-frequency components. For other layers of the Laplacian pyramid, a lightweight convolutional neural network (CNN) is used to learn the parameter value map of the remapping function and apply it to the local Laplacian filter to achieve adaptive enhancement of edge details.

[0033] (4) In order to improve the improvement effect of the remaining layers of the Laplacian pyramid with higher resolution, this invention proposes an efficient and progressive upsampling strategy. This strategy improves the high-resolution components by progressively upsampling the improved components of the previous levels, connecting them with the corresponding Laplacian layers, and then using a lightweight convolutional block to perform another local Laplacian filter. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the overall method architecture provided in the embodiments of the present invention;

[0035] Figure 2 This is a schematic diagram of the structure of the three-dimensional lookup table fusion strategy provided in the embodiments of the present invention;

[0036] Figure 3 This is a schematic diagram of the image adaptive local Laplacian filter structure provided in an embodiment of the present invention;

[0037] Figure 4 This is a high dynamic range image tone mapping effect diagram provided in an embodiment of the present invention; wherein, Figure 4 The left side of the image is a high dynamic range image. Figure 4 The right-hand side of the image shows a low dynamic range image generated using the method of this invention. Detailed Implementation

[0038] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0039] The overall architecture of the method of the present invention is further illustrated through embodiments, such as... Figure 1As shown, the entire architecture of this embodiment consists of three parts: 1) adaptive Laplacian pyramid decomposition; 2) low-frequency information mapping based on the three-dimensional lookup table method; 3) high-frequency information enhancement based on the image adaptive Laplacian filter; and 4) Laplacian pyramid reconstruction. These four parts will be described one by one below.

[0040] The first part decomposes the input high dynamic range image into an adaptively leveled Laplacian pyramid, which uses dynamically adjusted decomposition layers to match the resolution of the input image. This adaptive process brings the lowest-level Laplacian filtered low-frequency image to a resolution of approximately 64×64, thus providing a coarser but globally rich image representation.

[0041] The second part is to perform global tone mapping on the low-frequency image. The low-frequency image of the bottom Laplacian pyramid is passed through the Transformer module (weight predictor) to obtain a pixel-level weight map. Then, a basic 3D lookup table is used to perform trilinear interpolation on the low-frequency image and multiply it with the weight map.

[0042] Part Three Figure 3 This method enhances the details of high-frequency information by generating a parameter map of the remaining Laplacian pyramid through a convolutional neural network (filter parameter prediction module) and using it to enhance the details of the remaining Laplacian pyramid through a local Laplacian filter.

[0043] The fourth part is the result generated by reconstructing the mapped and enhanced Laplace pyramid.

[0044] The following section details the low-frequency information mapping based on the three-dimensional lookup table method. The network structure is as follows: Figure 2 As shown, the network first feeds the bottom-level Laplacian pyramid into a Transformer-based weight predictor to obtain three pixel-level weight maps. Then, the bottom-level Laplacian pyramid is trilinearly interpolated with three basic 3D lookup tables to obtain three preliminary mapping maps. Next, the three preliminary mapping maps are multiplied by the three pixel-level weight maps and then summed to obtain the globally mapped bottom-level Laplacian pyramid.

[0045] Next, we will explain in detail the high-frequency information enhancement based on the image adaptive Laplacian filter. The network structure is as follows: Figure 3As shown, the edge maps obtained by the Canny edge operator are upsampled from the bottom-level Laplacian pyramid and the globally mapped bottom-level Laplacian pyramid to make their size consistent with the next-lower-level Laplacian pyramid. Then, the two upsampled maps and the next-lower-level Laplacian pyramid are fed into two 3×3 convolutional networks and one 1×1 convolutional network to obtain the parameter value map. The parameter value map is then applied to the fast Laplacian filter to enhance the details of the next-lower-level Laplacian pyramid.

[0046] For subsequent enhancement steps: upsample the detail enhancement components of the previous Laplacian layer step by step, concatenate them with the corresponding original Laplacian layer, and then perform another local Laplacian filter.

[0047] Figure 4 This is an image showing the effect of tone mapping for a high dynamic range image using the method of this invention; the left part is the high dynamic range image, and the right part is the low dynamic range image generated using the method of this invention. It can be seen that the tone mapping method of this invention preserves local edge details while performing global tone mapping.

[0048] The above content is readily understood by those skilled in the art. The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method of tone mapping based on a lookup table and a Laplacian filter, characterized in that, The method comprises: (1) decomposing an input image into an adaptive Laplacian pyramid; (2) passing a low-frequency image at the bottom layer of the adaptive Laplacian pyramid through a weight predictor based on a Transformer to obtain a pixel-level weight map, simultaneously performing trilinear interpolation on the low-frequency image using a three-dimensional lookup table to obtain a preliminary mapping image, and then pixel-by-pixel fusing the preliminary mapping image and the weight map to generate a fine-tuned low-frequency image at the bottom layer of the Laplacian pyramid; (3) learning a parameter value map of an image at a remaining layer of the adaptive Laplacian pyramid through a CNN-based filter parameter prediction module, and applying the parameter value map to a local Laplacian filter to obtain a fine-tuned image at the remaining layer of the Laplacian pyramid; In step (3), the fine-tuned image at the second bottom layer of the Laplacian pyramid is obtained by the following method: passing the low-frequency image at the bottom layer of the fine-tuned Laplacian pyramid through a Canny edge detection operator to obtain an edge image, upsampling the edge image and the low-frequency image at the bottom layer of the adaptive Laplacian pyramid to make the sizes of the upsampled images consistent with the size of the image at the second bottom layer of the Laplacian pyramid, and then inputting the two upsampled images and the image at the second bottom layer of the adaptive Laplacian pyramid into the CNN-based filter parameter prediction module to obtain a parameter value map, and applying the parameter value map to the local Laplacian filter to obtain the fine-tuned image at the second bottom layer of the Laplacian pyramid; In step (3), the fine-tuned image at a layer other than the bottom layer and the second bottom layer of the Laplacian pyramid is obtained by the following method: upsampling the fine-tuned image at the previous layer of the Laplacian pyramid to make the size of the upsampled image consistent with the size of the image at the current layer, and then inputting the upsampled image and the image at the same layer of the adaptive Laplacian pyramid into the CNN-based filter parameter prediction module to obtain a parameter value map, and applying the parameter value map to the local Laplacian filter to obtain the fine-tuned image at the current layer of the Laplacian pyramid; (4) reconstructing the fine-tuned Laplacian pyramid to obtain a tone-mapped image.

2. The method of claim 1, wherein, In step (2), the low-frequency image at the bottom layer of the adaptive Laplacian pyramid is input into the weight predictor based on the Transformer to obtain three pixel-level weight maps, the low-frequency image is trilinearly interpolated with three basic three-dimensional lookup tables to obtain three preliminary mapping images, the three preliminary mapping images are multiplied by the three pixel-level weight maps respectively, and finally the three products are added to obtain the fine-tuned image at the bottom layer of the Laplacian pyramid.

3. A look-up table and Laplacian filter based tone mapping system, characterized in that, The system comprises: an image decomposition module configured to decompose an input image into an adaptive Laplacian pyramid; a bottom layer image generation module configured to pass a low-frequency image at the bottom layer of the adaptive Laplacian pyramid through a weight predictor based on a Transformer to obtain a pixel-level weight map, simultaneously perform trilinear interpolation on the low-frequency image using a three-dimensional lookup table to obtain a preliminary mapping image, and then pixel-by-pixel fuse the preliminary mapping image and the weight map to generate a fine-tuned low-frequency image at the bottom layer of the Laplacian pyramid; a residual layer image generation module configured to learn a parameter value map of an adaptive Laplacian pyramid residual layer image by a CNN-based filter parameter prediction module and apply the parameter value map to a local Laplacian filter to obtain a fine-tuned Laplacian pyramid residual layer image; the residual layer image generation module includes a sub-bottom layer image generation unit configured to obtain an edge image of a low-frequency image at the bottom layer of the fine-tuned Laplacian pyramid by a Canny edge detection operator, up-sample the edge image and the low-frequency image at the bottom layer of the adaptive Laplacian pyramid, make the size of the up-sampled image consistent with the size of a sub-bottom layer image of the Laplacian pyramid, and then input the two up-sampled images and the sub-bottom layer image of the adaptive Laplacian pyramid into the CNN-based filter parameter prediction module to obtain a parameter value map, and apply the parameter value map to the local Laplacian filter to obtain a fine-tuned sub-bottom layer image of the Laplacian pyramid; the residual layer image generation module further includes other layer image generation units configured to generate other layer images of the fine-tuned Laplacian pyramid except the bottom layer and the sub-bottom layer, and specifically configured to up-sample a previous layer image of the fine-tuned Laplacian pyramid, make the size of the up-sampled image consistent with the size of a current layer image, and then input the up-sampled image and the current layer image of the adaptive Laplacian pyramid into the CNN-based filter parameter prediction module to obtain a parameter value map, and apply the parameter value map to the local Laplacian filter to obtain the current layer image of the fine-tuned Laplacian pyramid; an image reconstruction module configured to reconstruct the fine-tuned Laplacian pyramid to obtain a tone-mapped image.

4. The system of claim 3, wherein, The bottom layer image generation module is specifically configured to input a low-frequency image at the bottom layer of the adaptive Laplacian pyramid into a Transformer-based weight predictor to obtain three pixel-level weight maps, perform trilinear interpolation on the low-frequency image and three basic three-dimensional lookup tables respectively to obtain three preliminary mapping images, multiply the three preliminary mapping images with the three pixel-level weight maps respectively, and finally add the three products to obtain the bottom layer image of the fine-tuned Laplacian pyramid.

5. An electronic device, comprising: comprise: a memory configured to store a program; a processor configured to execute the program stored in the memory, and when the program stored in the memory is executed, the processor is configured to execute the method of any one of claims 1-2.

6. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 5. The computer program, when executed on the processor, causes the processor to execute the method of any one of claims 1-2.

Images