Image dynamic range compression method based on deep learning and joint bilateral filtering

Abstract

The invention discloses an image dynamic range compression method based on deep learning and joint bilateral filtering, which comprises the following steps: collecting original high-bit image data, constructing a data set, and carrying out primary processing on the data set to obtain a linear RGB image and a Y channel image; constructing a deep learning network in combination with a jump connection structure of the UNet and an efficient lightweight unit of the ShuffleNet, and processing the linear RGB image and the Y channel image by using the deep learning network to obtain a brightness gain image; processing the brightness gain graph and the original high-bit image by means of united bilateral filtering up-sampling operation to obtain an original resolution brightness gain graph; and performing reverse gamma operation on the original high-bit image, dividing the original high-bit image by the original resolution brightness gain graph, and performing gamma operation to obtain a compressed 8-bit image. According to the method, a dynamic range adjustment algorithm is realized, and the process of mapping from a high bit to a low bit is favorable for recovering highlight details and brightening and darkness details and keeping a good image contrast.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to an image dynamic range compression method based on deep learning and joint bilateral filtering. Background Art

[0002] In the field of image style imaging technology, dynamic range adjustment is a crucial step in achieving high-quality image display. Traditional tonemapping algorithms, by adjusting the image's brightness curve, compress high dynamic range images to a range that the display device can render, and are widely used in image and video processing. However, this algorithm has limitations in practical applications. It relies on a fixed mathematical transformation model, making it difficult to adaptively handle the complex lighting distributions of images in different scenes. This can easily lead to the loss of highlight details or blurring of details in dark areas, failing to fully meet users' requirements for image detail restoration and visual effects. With the development of mobile devices and photography technology, smartphones, DSLRs, and other similar devices have become mainstream tools for image acquisition. Users frequently encounter high dynamic range (HDR) scenes during shooting, such as backlit environments and scenes with both strong light and shadow, which places higher demands on the dynamic range processing capabilities of devices. Existing imaging systems struggle to simultaneously preserve detail in highlight areas, enhance detail in shadow areas, and maintain good image contrast in such scenarios. Therefore, a more efficient and intelligent image dynamic range compression technology is urgently needed to improve the visual quality and display effect of images, meeting the needs of users in diverse shooting scenarios. Summary of the Invention

[0003] To address the aforementioned technical problems and shortcomings in the background technology, the present invention proposes an image dynamic range compression method based on deep learning and joint bilateral filtering. This method is applied to the photography paths of mobile phones and SLR cameras. Given an image with high dynamic range and high resolution, the image is first downsampled. Then, deep learning is performed on the downsampled high dynamic range image. After obtaining the result, joint bilateral filtering is used to upsample the original input image, the downsampled input image, and the output image obtained through the network after downsampling. Finally, a well-compressed image with good dynamic range is obtained, which can restore highlight details, brighten shadow details, and maintain good image contrast, ultimately resulting in a displayable high dynamic range image. This solves the current problems of lost highlight details or blurred shadow details.

[0004] The technical solution adopted in this invention: S1) Collect raw high-bit image data, construct a dataset, and perform preliminary processing on the dataset to obtain a linear RGB image and a Y channel image; S2) Combine the skip connection structure of UNet with the efficient and lightweight units of ShuffleNet to build a deep learning network. Use the deep learning network to process the linear RGB image and Y channel image to obtain the brightness gain image. S3) The brightness gain map and the original high-bit image are processed by a joint bilateral filtering upsampling operation to obtain the original resolution brightness gain map; S4) Perform an inverse gamma operation on the original high-bit image, then divide it by the original resolution brightness gain map and obtain the compressed 8-bit image through the gamma operation.

[0005] Preferably, in step S1), the dataset undergoes preliminary processing to obtain a linear RGB image and a Y channel image. The specific process is as follows: Sa1) Downsampling operation: The original high-bit image in the input dataset is downsampled using bilinear interpolation. The four nearest neighbor pixels around the target pixel are selected, and the new pixel value of the target pixel is calculated by linear weighted averaging of these four nearest neighbor pixels. This makes the resolution of the image after downsampling 512×512, thereby reducing the resolution and computational load of subsequent processing. Sa2) Inverse gamma operation and color gamut conversion operation: For the downsampled image, perform an inverse gamma operation pixel by pixel. The inverse gamma operation uses gamma operation... The inverse function of the formula transforms the nonlinear domain image into a linear domain image, resulting in a linear RGB image. ,in It is 2.2. This is the image after the inverse gamma operation. The image is shown before the inverse gamma operation. Simultaneously, an RGB-to-Y color gamut conversion operation is performed on the downsampled image, converting the image containing RGB color space into a Y-channel image containing brightness information. The formula is Y = 0.2126R + 0.0722B + 0.7125G, where Y is the brightness value obtained after conversion, R is the red channel value of the pixel in the RGB color space, B is the green channel value of the pixel in the RGB color space, and G is the blue channel value of the pixel in the RGB color space.

[0006] Preferably, in step S2), a deep learning network is constructed by combining the skip connection structure of UNet with the efficient and lightweight units of ShuffleNet. The deep learning network is then used to process the linear RGB image and the Y channel image to obtain the brightness gain image. The specific process is as follows: Sb1) Constructing a Deep Learning Network: A deep learning network structure is constructed using an Encoder, Decoder, and Bottleneck part, combining the skip connection structure of UNet with the efficient and lightweight units of ShuffleNet. The Encoder part employs the efficient and lightweight units of ShuffleNetV2 based on ShuffleNet, introducing channel shuffling operations to gradually reduce the image resolution through a series of convolutional operations while increasing the number of feature channels, thus achieving multi-level feature extraction. The Decoder part uses the skip connection structure of UNet to gradually restore the spatial resolution of the image. Simultaneously, skip connections fuse the multi-level features of the Encoder part with the features processed by the Decoder part, restoring the spatial detail information of the image. The Bottleneck part utilizes the depthwise separable convolutional structure of ShuffleNet to obtain a compressed and color-corrected non-linear image through depthwise separable convolution processing, reducing the number of parameters and computational cost of convolution operations. Sb2) Input to the deep learning network: The linear RGB image obtained through inverse gamma and the Y channel Figure 1 Both are used as inputs to deep learning networks; Sb3) Output and processing of deep learning network: After processing by deep learning network, output y nonlinear domain map with dynamic range adjustment is output, and then perform inverse gamma operation on it to obtain y linear domain map. At the same time, perform inverse gamma operation on linear RGB map to obtain Y linear domain map. By dividing Y linear domain map by y linear domain map, luminance gain map with resolution of 512×512 is obtained.

[0007] Preferably, in step S3), the luminance gain map and the original high-bit image are processed by a joint bilateral filtering upsampling operation to obtain the original resolution luminance gain map. The specific operation steps include: By using low-resolution brightness gain graphs High-resolution original high-bit image with resolution size before downsampling Together as input, in the brightness gain graph Find the original high-bit image medium pixel The corresponding 5x5 local window area , brightness gain diagram The original high-bit image is interpolated onto a high-resolution grid using bilinear interpolation. Each pixel in Its value is obtained through the brightness gain graph. Middle Neighborhood Pixels The weighted average yields the original resolution luminance gain map. , the calculation formula is:

[0008] In the formula As the normalization factor, For joint weights; Among them, the joint weight is calculated. The calculation formula is:

[0009] In the formula To control the standard deviation of spatial weights; To control for the standard deviation of the similarity to the reference images; For pixels and pixels The square of the spatial distance; For pixels and pixels Differences between reference images; It is an exponential function.

[0010] Preferably, in step S4), the original high-bit image undergoes an inverse gamma operation, and after being divided by the original resolution luminance gain map, the compressed 8-bit image is obtained through another gamma operation. Specific steps include: The original high-bit image is subjected to an inverse gamma operation to obtain a linear domain image. This image is then divided by the original resolution brightness gain map to recover the image, resulting in the original resolution map after dynamic range adjustment in the linear domain. Finally, a gamma operation is performed to obtain the compressed 8-bit image.

[0011] Compared with existing technologies, the image dynamic range compression method proposed in this invention based on deep learning and joint bilateral filtering has the following advantages: This invention implements a dynamic range adjustment algorithm, which maps from high-bit to low-bit values. Compared to traditional algorithms, it provides better visualization of dynamic range, effectively improving dark areas and restoring highlight areas. In terms of efficiency, by reducing the image to a smaller size and passing it through a simple deep learning network, followed by upsampling using a joint bilateral filter, computational performance is significantly improved. Furthermore, the upsampling process eliminates light spot defects. In image processing, this is a highly efficient algorithm that produces high-quality images. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating the image dynamic range compression method based on deep learning and joint bilateral filtering proposed in this invention. Figure 2 This is a diagram of the deep learning network framework proposed in this invention; Figure 3 This is a comparison image of the original high-bit indoor image and the 8-bit image of the present invention, where a is the original high-bit indoor image and b is a comparison image of the indoor 8-bit image. Figure 4 This is a comparison image of the original outdoor high-bit image and the 8-bit image of the present invention, where c is the original outdoor high-bit image and d is a comparison image of the outdoor 8-bit image. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be further described clearly and completely below with reference to the accompanying drawings. It should be noted that the described embodiments are only some embodiments of this application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0014] To make the inventive objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings: In order to better understand the above-mentioned objectives, features, and advantages of this invention, the advantages of this invention will be further illustrated below by comparing the embodiments with the accompanying drawings and specific implementation methods.

[0015] The technical solution of the present invention will be described in more detail and completely below with reference to the accompanying drawings and specific embodiments.

[0016] The proposed image dynamic range compression method based on deep learning and joint bilateral filtering is illustrated in the flowchart below. Figure 1 As shown, it includes the following steps: S1) Collect raw high-bit image data, construct a dataset, and perform preliminary processing on the dataset to obtain a linear RGB image and a Y channel image; Specifically, in step S1), the dataset undergoes preliminary processing to obtain a linear RGB image and a Y channel image. The specific process is as follows: Sa1) Downsampling operation: The original high-bit image in the input dataset has a high resolution. The image is downsampled using bilinear interpolation. The four nearest neighbor pixels around the target pixel are selected. The new pixel value of the target pixel is calculated by linearly weighted averaging these four nearest neighbor pixels. This makes the resolution of the image after downsampling 512×512, thereby reducing the resolution and the amount of computation in subsequent processing. Sa2) Inverse gamma operation and color gamut conversion operation: For the downsampled image, perform an inverse gamma operation pixel-by-pixel. The gamma operation (using...) Formula, where It is 2.2. The image after gamma operation. The gamma operation (image before gamma) is a process of performing a non-linear brightness transformation on an image to match the non-linear response of the human eye to light intensity. The inverse gamma operation, the inverse of the gamma operation, transforms the non-linear domain image into a linear domain image, preparing it for subsequent processing. The inverse gamma operation uses the gamma operation... The inverse function of the formula yields a linear RGB image, i.e. ,in It is 2.2. This is the image after the inverse gamma operation. The image is shown before the inverse gamma operation. Simultaneously, the image after downsampling undergoes an RGB-to-Y color gamut conversion, converting the image containing RGB color space into a Y-channel image containing brightness information. The formula is Y = 0.2126R + 0.0722B + 0.7125G, where Y is the converted brightness value, R is the red channel value of a pixel in the RGB color space, B is the green channel value of a pixel in the RGB color space, and G is the blue channel value of a pixel in the RGB color space (the coefficients 0.2126, 0.0722, and 0.7125 in the formula are derived from experimental data on the sensitivity of the human eye to different colors of light).

[0017] S2) Combine the skip connection structure of UNet with the efficient and lightweight units of ShuffleNet to build a deep learning network. Use the deep learning network to process the linear RGB image and Y channel image to obtain the brightness gain image. Specifically, in step S2), a deep learning network is constructed by combining the skip connection structure of UNet with the efficient and lightweight units of ShuffleNet. The deep learning network is then used to process the linear RGB image and the Y channel image to obtain the brightness gain image. The deep learning network framework diagram is shown below. Figure 2 The process is as follows: Sb1) Deep Learning Network Structure: The deep learning network structure is constructed by combining the skip connection structure of UNet and the efficient and lightweight units of ShuffleNet, consisting of an Encoder, a Decoder, and a Bottleneck. The Encoder uses efficient and lightweight units based on ShuffleNetV2, introducing channel shuffling operations to gradually reduce image resolution through a series of convolutional operations while increasing the number of feature channels, thus achieving multi-level feature extraction. The Decoder uses the skip connection structure of UNet to gradually restore the resolution of the linear image. Simultaneously, skip connections fuse the multi-level features from the Encoder with the features processed by the Decoder, restoring spatial detail information and enabling the network to reconstruct a richly detailed image. The Bottleneck utilizes the depthwise separable convolutional structure of ShuffleNet to obtain the Y-image through depthwise separable convolution processing, reducing the number of parameters and computational cost of convolution operations. The advantages of deep learning network architecture include: ensuring speed and performance, which is beneficial for use on mobile devices; ① Progressive downsampling and feature abstraction (Encoder part) Layered compression ratio optimization: A 16× spatial compression is achieved through four stages of downsampling (512→16 resolution). Each stage uses a ShuffleNet Unit with a stride of 2 instead of a traditional pooling layer for downsampling, avoiding information loss in traditional pooling and enabling multi-level feature extraction, as detailed below: Initially, Conv3×3 convolution and MaxPool operations are used to quickly reduce the resolution (512→128) while retaining 24 channels of basic features; In Encoder3, the deep ShuffleNet Unit enhances 32×32 scale feature extraction through 3 repetitions (Repeat=3). Its 232 channels are similar to the 256 channels of VGG16 at the same stage, but the number of parameters is reduced by 37%. In the Encoder2 (64×64×116) stage, through grouped convolution and channel rearrangement, the feature reuse rate was improved by about 2.1 times according to the actual test comparison. ②Bottleneck channel expansion design (Bottleneck section) The efficiency of using depthwise separable convolution: In the process of generating a 16×16×928 feature map, the computational cost of standard convolution is 928×464×3×3=3.9MFLOPs, while the actual computational cost of depthwise separable convolution is 464×3×3 + 928×464×1×1=0.43M FLOPs. By splitting the standard convolution into depthwise separable convolution, the computational cost is reduced by 89%; and no downsampling operation (Stride=1) is used to preserve the integrity of spatial information. ③ Symmetric upsampling and skip connections (Decoder part) Layer-by-layer feature fusion mechanism: Size matching: Each UpConv2×2 is concatenated with the corresponding Encoder feature (e.g., the 32×32×928 of Decoder4 is concatenated with the 32×32×464 of Encoder4), and the number of channels is unified (2^32) through 1×1 convolution to avoid information overload; Application of ShuffleNet Unit Decoder: At the 32×32 scale, the ShuffleNet Unit's 1×1→3×3→1×1 structure (with only 1 / 5 the number of parameters of the standard residual block) is applied to achieve efficient feature fusion, and the inference speed is 2.3 times faster than the traditional UNet. ④ Advantages of end-to-end design Input-output consistency: The 512×512×1 output is directly obtained by Conv1×1 convolution, which eliminates redundant upsampling (such as the 32x upsampling of FCN). Mobile compatibility: The entire network has only 3.8M parameters, which can be quantized to INT8 (accuracy loss <0.5%), and the measured power consumption on the Snapdragon 865 chip is <1.2W, meeting real-time requirements; Sb2) Input to the deep learning network: The linear RGB image and Y channel image (4 channels in total) obtained by inverse gamma are used as the input to the deep learning network. Sb3) Output processing of deep learning network: After processing by deep learning network, the output is a y nonlinear domain map with dynamic range adjustment. Then, an inverse gamma operation is performed on it to obtain a y linear domain map. At the same time, an inverse gamma operation is performed on the linear RGB map to obtain a Y linear domain map. By dividing the Y linear domain map by the y linear domain map, a 512×512 brightness gain map is obtained.

[0018] S3) The brightness gain map and the original high-bit image are processed by a joint bilateral filtering upsampling operation to obtain the original resolution brightness gain map; Specifically, in step S3), the luminance gain map and the original high-bit image are processed using a joint bilateral filtering upsampling operation to obtain the original resolution luminance gain map. The specific operation steps include: Joint bilateral filtering upsampling is an edge-preserving image super-resolution technique that combines the geometric structure of high-resolution images (such as color information from RGB or depth maps) to perform high-quality upsampling of low-resolution images (such as depth maps or semantic labels). Its core idea is to use the weighting mechanism of bilateral filtering to preserve edge details while magnifying the image. By using low-resolution brightness gain graphs High-resolution original high-bit image with resolution size before downsampling Together as input, in the brightness gain graph Find the original high-bit image medium pixel The corresponding 5x5 local window area , brightness gain diagram The original high-bit image is interpolated onto a high-resolution grid using bilinear interpolation. Each pixel in Its value is obtained through the brightness gain graph. Middle Neighborhood Pixels The weighted average yields the luminance gain map at the original resolution. , the calculation formula is:

[0019] In the formula As the normalization factor, For joint weights; Among them, joint weight The brightness gain map is determined by both spatial distance and intensity differences in the reference image. pixels For the original high-bit image pixel The magnitude of the contribution depends on the spatial distance as a factor in the brightness gain diagram. pixels and the original high-bit image pixels In terms of spatial location, distance; on the other hand, the difference in intensity of the reference image is in pixels. and pixels The differences in color and brightness intensity make the original high-bit image... Edge and brightness gain graph Edge alignment preserves edge details while magnifying the image, avoiding blurring, and calculates joint weights. The calculation formula is:

[0020] In the formula To control the standard deviation of spatial weights (when Larger values ​​allow for the participation of more distant pixels. To control the standard deviation of the similarity of reference images (when Larger values ​​allow for greater intensity differences. For pixels and pixels The square of the spatial distance; For pixels and pixels Differences between reference images; It is an exponential function.

[0021] S4) Perform an inverse gamma operation on the original high-bit image, divide it by the original resolution brightness gain map, and then perform a gamma operation to obtain the compressed 8-bit image; Specifically, in step S4), two sets of original high-bit images (the first set being indoor images and the second set being outdoor images) are used. The original high-bit images undergo an inverse gamma operation, and after being divided by the original resolution brightness gain image, a compressed 8-bit image is obtained through another gamma operation. A comparison of the resulting original high-bit images and 8-bit images is shown below. Figure 3 , Figure 4 As shown, the specific operating steps include: The original high-bit image is subjected to an inverse gamma operation to obtain a linear domain image. This image is then divided by the original resolution brightness gain map to recover the image, resulting in the original resolution map after dynamic range adjustment in the linear domain. Finally, a gamma operation is performed to obtain the compressed 8-bit image.

[0022] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

[0023] Finally, it should be noted that the above specific embodiments are only used to illustrate the technical solution of the present invention and are not intended to limit its scope of protection. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that any modifications, substitutions, or equivalent variations to the technical content without departing from the core idea of ​​the present invention should be considered to fall within the scope of protection of the present invention, and the scope of protection of the present invention should be determined by the claims.

Claims

1. An image dynamic range compression method based on deep learning and joint bilateral filtering, characterized in that, The method includes the following steps: S1) Collect raw high-bit image data, construct a dataset, and perform preliminary processing on the dataset to obtain a linear RGB image and a Y channel image; S2) Combine the skip connection structure of UNet with the efficient and lightweight units of ShuffleNet to build a deep learning network. Use the deep learning network to process the linear RGB image and Y channel image to obtain the brightness gain image. Sb1) Constructing a Deep Learning Network: A deep learning network structure is constructed using an Encoder, Bottleneck, and Decoder part, combining the skip connection structure of UNet with the efficient and lightweight units of ShuffleNet. The Encoder part uses the efficient and lightweight units of ShuffleNetV2 based on ShuffleNet, introducing channel shuffling operations to gradually reduce the image resolution through a series of convolutional operations while increasing the number of feature channels, thus achieving multi-level feature extraction. The Bottleneck part utilizes the depthwise separable convolutional structure of ShuffleNet to obtain a compressed and color-corrected non-linear image through depthwise separable convolution processing, reducing the number of parameters and computational cost of convolution operations. The Decoder part uses the skip connection structure of UNet to gradually restore the spatial resolution of the image. Simultaneously, skip connections fuse the multi-level features of the Encoder part with the features processed by the Decoder part, restoring the spatial detail information of the image. Sb2) Input to the deep learning network: The linear RGB image and Y channel image obtained by inverse gamma are used as the input to the deep learning network. Sb3) Output and processing of deep learning network: After processing by deep learning network, output y nonlinear domain map with dynamic range adjustment is output, and then perform inverse gamma operation on it to obtain y linear domain map. At the same time, perform inverse gamma operation on linear RGB map to obtain Y linear domain map. By dividing Y linear domain map by y linear domain map, luminance gain map with resolution of 512×512 is obtained. S3) The brightness gain map and the original high-bit image are processed by a joint bilateral filtering upsampling operation to obtain the original resolution brightness gain map; S4) Perform an inverse gamma operation on the original high-bit image, then divide it by the original resolution brightness gain map and obtain the compressed 8-bit image through the gamma operation.

2. The image dynamic range compression method based on deep learning and joint bilateral filtering according to claim 1, characterized in that, In step S1), the dataset undergoes preliminary processing to obtain a linear RGB image and a Y channel image. The specific process is as follows: Sa1) Downsampling operation: The original high-bit image in the input dataset is downsampled using bilinear interpolation. The four nearest neighbor pixels around the target pixel are selected, and the new pixel value of the target pixel is calculated by linear weighted averaging of these four nearest neighbor pixels. This makes the resolution of the image after downsampling 512×512, thereby reducing the resolution and computational load of subsequent processing. Sa2) Inverse gamma operation and color gamut conversion operation: For the downsampled image, perform an inverse gamma operation pixel by pixel. The inverse gamma operation uses gamma operation... The inverse function of the formula transforms the nonlinear domain image into a linear domain image, resulting in a linear RGB image. ,in It is 2.

2. This is the image after the inverse gamma operation. The image is shown before the inverse gamma operation. Simultaneously, an RGB-to-Y color gamut conversion operation is performed on the downsampled image, converting the image containing RGB color space into a Y-channel image containing brightness information. The formula is Y = 0.2126R + 0.0722B + 0.7125G, where Y is the converted brightness value, R is the red channel value of a pixel in the RGB color space, B is the green channel value of a pixel in the RGB color space, and G is the blue channel value of a pixel in the RGB color space.

3. The image dynamic range compression method based on deep learning and joint bilateral filtering according to claim 1, characterized in that, In step S3), the luminance gain map and the original high-bit image are processed using a joint bilateral filtering upsampling operation to obtain the original resolution luminance gain map. The specific operation steps include: By using low-resolution brightness gain graphs High-resolution original high-bit image with resolution size before downsampling Together as input, in the brightness gain graph Find the original high-bit image medium pixel The corresponding 5x5 local window area , brightness gain diagram The original high-bit image is interpolated onto a high-resolution grid using bilinear interpolation. Each pixel in Its value is obtained through the brightness gain graph. Middle Neighborhood Pixels The weighted average yields the original resolution luminance gain map. The calculation formula is: ； In the formula As the normalization factor, For joint weights; Among them, the joint weight is calculated. The calculation formula is: ； In the formula To control the standard deviation of spatial weights; To control for the standard deviation of the similarity to the reference images; For pixels and pixels The square of the spatial distance; For pixels and pixels Differences between reference images; It is an exponential function.

4. The image dynamic range compression method based on deep learning and joint bilateral filtering according to claim 1, characterized in that, In step S4), the original high-bit image undergoes an inverse gamma operation, and after being divided by the original resolution luminance gain map, the compressed 8-bit image is obtained through another gamma operation. The specific operation steps include: The original high-bit image is subjected to an inverse gamma operation to obtain a linear domain image. This image is then divided by the original resolution brightness gain map to recover the original resolution map after dynamic range adjustment in the linear domain. Finally, a gamma operation is performed to obtain a compressed 8-bit image.

Images