A high dynamic range image reconstruction method based on spatial adaptive modulation

Abstract

The present application relates to a kind of high dynamic range image reconstruction methods based on spatial adaptive modulation, belong to image processing and identification method technical field;First, image data set is acquired and preprocessed, input network;The overall architecture of network includes the weighted network of upper branch and the main stem U-Net network of lower branch, weighted network carries out feature extraction and obtains weighted image;The main stem network introduces spatial adaptive feature modulation and convolution channel mixing, and both are integrated into residual block, and the feature on the different scale branch of U-Net is realized feature mixing;Spatial adaptive modulation generates attention map for spatial adaptive feature modulation, and dynamically modulates input feature to learn multi-scale feature representation;Convolution channel mixing simultaneously extracts local context information and realizes channel mixing, significantly enhances the correlation between long-distance features from multi-scale feature aggregation, solves the problem that the details of the reconstructed image saturated area recovery are not rich enough.

Description

Technical Field

[0001] This invention relates to a high dynamic range image reconstruction method based on spatial adaptive modulation, belonging to the technical field of image processing and recognition methods. Background Technology

[0002] High dynamic range (HDR) imaging is an important field in image processing, computer graphics / vision, and photography, representing a technique that allows for a greater dynamic range than traditional imaging methods. HDR images are now routinely used in many applications, including image compositing and a range of post-processing operations. High dynamic range image reconstruction methods based on convolutional networks have limited receptive fields in their convolutional kernels, capturing only local pixel relationships. Their ability to represent long-distance dependencies is insufficient, leading to incomplete structural recovery of saturated regions in reconstructed images with large exposure areas. This results in artificial artifacts in local areas, loss of high-frequency detail information, and ultimately, low fidelity and insufficient texture detail in the reconstructed image. While the self-attention mechanism in Transformers addresses long-distance dependencies, it lacks the inductive bias of convolution and the computational constraints imposed by the network. Given the continuous development of HDR display devices and the insufficient amount of native high dynamic range content to meet the demands of these devices, the problem of converting a large amount of existing low dynamic range content into corresponding high dynamic range content remains to be solved. Summary of the Invention

[0003] The purpose of this invention is to address the shortcomings of existing technologies and provide a high dynamic range image reconstruction method based on spatial adaptive modulation, which is a method proposed to address this problem by using SDR images for HDR image reconstruction tasks.

[0004] To achieve the above objectives, the present invention is implemented through the following technical solution: a high dynamic range image reconstruction method based on spatial adaptive modulation, the method being as follows:

[0005] Step 1: Obtain the image dataset for training;

[0006] Step 2: Preprocess the dataset from Step 1 to generate input images and input them into the network;

[0007] Step 3: The overall architecture of the network includes a weighted network in the upper branch and a backbone U-Net network in the lower branch. The weighted network extracts features from the input image in Step 2 to obtain a weighted image, so that the information of the unsaturated region in the input image is preserved. The weighted image is used as one of the output branches.

[0008] Step 4: The backbone U-Net network of the lower branch first converts the input image in Step 2 into a feature map through a convolutional layer;

[0009] Step 5: The feature map generated in Step 4 is subjected to deep extraction through feature mixing blocks and convolutional layers, so that the feature map generated in Step 4 can learn multi-scale feature representations, while capturing local contextual relationships to enhance the global information and local details of the feature map;

[0010] Step 6: At the encoding end, the feature map extracted in step 5 is downsampled twice. Between the two downsampling operations, serial residual blocks are used to extract features at different scales. Then, at the decoding end, the feature map is upsampled twice. Between the two upsampling operations, the same number of residual blocks as at the encoding end are used for image reconstruction. At the same time, the feature map before downsampling is added to the feature map after upsampling using skip connections to make up for the information lost due to downsampling and feature extraction.

[0011] Step 7: The feature map output from the decoding end in Step 6 is transformed back into the output image through the feature mixing block and convolutional layer. Then, it is added to the weighted image obtained in Step 3 to obtain the reconstructed image. Next, the pixel-level loss function with hyperbolic tangent is used to calculate the loss value of the ground truth of the input image and the reconstructed image. Backpropagation is performed on the network, and the network hyperparameters are iteratively optimized to finally obtain the trained model.

[0012] Step 8: Reconstruct the real SDR image based on the trained model. Feed the SDR image into the model trained in Step 7 for repair to obtain a high-quality and highly faithful reconstructed image.

[0013] The weighted network uses four convolutional layers to predict a weighted feature map, preserving information from unsaturated regions of the input image and allowing the network to focus on reconstructing details in saturated regions. The backbone network takes the SDR input image as input and reconstructs a high dynamic range image with more details in the saturated regions. Downsampling compresses the feature map, reducing redundant information and retaining useful information; upsampling increases the resolution of the feature map.

[0014] Preferably, the formula for preprocessing the dataset in step 2 is shown in (1):

[0015] (1)

[0016] in: For the initial image dataset, For normalization function, This is a normalized image.

[0017] Preferably, in step 3, the weighted network first generates a weight feature map from the input image in step 2 through four convolutional layers, and then performs an element-wise multiplication operation between the weight feature map and the input image in step 2 to obtain the weighted image; the weighted network is shown in formula (2):

[0018] (2)

[0019] in: This is an element-wise multiplication operation. It is a 3×3 convolutional layer. This is the weighted image.

[0020] Preferably, the size of the convolutional layers in the backbone U-Net network is 3×3, wherein the last convolutional layer of the backbone U-Net network has 3 output channels, and the rest have 64.

[0021] Preferably, the feature mixing block in step 5 includes a spatial adaptive modulation layer and a convolutional channel mixing layer; both layers are residual connection structures; the spatial adaptive modulation layer learns image spatial structure information from multi-scale feature representations; the convolutional channel mixing layer utilizes channel mixing to enhance local contextual feature relationships and enhance local detail information. The residual connection structure is a structure that adds the enhanced feature map obtained after processing the input feature map to the input feature map using a skip connection.

[0022] Preferably, in order to reduce model complexity and obtain pyramid feature representation, the spatial adaptive modulation layer first performs layer normalization on the feature map, and then performs channel splitting on the layer normalized input features to generate four parts;

[0023] The first part is processed by a 3×3 depthwise convolution, and the remaining parts are fed into feature generation units of multiple scales. The process of extracting pyramid features is shown in formulas (3), (4), and (5):

[0024] (3)

[0025] (4)

[0026] (5)

[0027] in: This indicates the different characteristics after channel separation; This indicates a channel separation operation; X represents the input feature. Represents features extracted at different scales; This represents a depthwise convolution of size 3×3; This indicates that the feature map at initial resolution p is downsampled to... ;and For nearest neighbor upsampling, the feature map resolution is restored to the initial resolution p;

[0028] Then, the multi-scale feature maps are merged along the channels, and 1×1 convolutions are used to aggregate local and global relationships. Next, the GELU activation function is used to predict the attention map, and the input feature maps are adaptively modulated in the form of element-wise multiplication. This process is represented by formulas (6) and (7):

[0029] (6)

[0030] (7)

[0031] in: This indicates the characteristics after channel merging; This indicates a merge operation along the channel dimension; It is a 1×1 convolution; For GELU activation function, For element-wise multiplication, This represents the characteristics after spatial adaptive modulation.

[0032] Preferably, the convolutional channel mixing layer first performs layer normalization on the feature map, and then applies 3×3 convolution, GELU activation function and 1×1 convolution to the layer normalized feature map to enhance the local spatial modeling ability and achieve channel mixing.

[0033] Preferably, the residual block includes a spatial adaptive modulation layer and a convolutional channel mixing layer in the feature mixing block. It employs a spatial adaptive modulation layer, 3×3 convolution, a convolutional channel mixing layer, and 3×3 convolution, and applies spatial adaptive modulation and channel mixing to the input feature map in an element-wise addition manner to features at different scales. The residual block is a module that adds the enhanced feature map obtained after passing the input feature map through the feature mixing block to the input feature map.

[0034] Preferably, in step 7, a loss function is used to calculate the loss of the ground truth (GT) of the input image and the reconstructed image. Step 7 uses a pixel-level loss function with hyperbolic tangent to calculate the loss value of the GT and the reconstructed image. This loss function is an L1 loss function with Tanh transform, balancing the learning weights between high and low brightness values. The loss function is defined as follows:

[0035] (8)

[0036] in: To reconstruct the image; For reference GT image; H represents the image height; W represents the image width; C represents the RGB three channels; It is the hyperbolic tangent function; To perform L1 norm operations on the results.

[0037] Preferably, in formula (8) .

[0038] The present invention has the following beneficial effects: (1) The present invention introduces spatial adaptive feature modulation and convolutional channel mixing on the basis of Vision Transformer blocks, and integrates the spatial adaptive feature modulation layer and the convolutional channel mixing layer into the traditional residual block, thereby realizing feature mixing operation on features on different scale branches of U-Net; spatial adaptive modulation generates attention maps for spatial adaptive feature modulation by applying a feature pyramid structure, and dynamically modulates the input features to learn long-distance dependencies from multi-scale feature representations. Convolutional channel mixing extracts local context information and realizes channel mixing at the same time, significantly enhancing the correlation between long-distance features from multi-scale feature aggregation, and solving the problem of insufficient detail recovery in saturated regions of reconstructed images.

[0039] (2) This invention introduces the hyperbolic tangent L1 loss function to enhance the model’s simultaneous attention to high and low brightness values, making the texture and local details of the saturated region of the reconstructed image richer. The traditional L1 loss function directly calculates the generated HDR value and the true HDR value, which makes the network pay more attention to the high brightness value and ignores the impact of the low brightness value on the network learning, resulting in poor visual quality. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of the overall framework of the high dynamic range image reconstruction method based on spatial adaptive modulation of the present invention;

[0041] Figure 2 This is a schematic diagram of the feature mixing block in this invention. Detailed Implementation

[0042] The present invention will be further described below with reference to embodiments, but these embodiments are not intended to limit the scope of the invention.

[0043] A high dynamic range image reconstruction method based on spatial adaptive modulation is described below:

[0044] Step 1: Obtain the image dataset for training.

[0045] Step 2: Preprocess the dataset from Step 1 to generate input images and input them into the network;

[0046] The formula for preprocessing the dataset is shown in (1):

[0047] (1)

[0048] in: For the initial image dataset, For normalization function, This is a normalized image.

[0049] Step 3: The overall architecture of the network includes a weighted network in the upper branch and a backbone U-Net network in the lower branch. The weighted network extracts features from the input image in Step 2 to obtain a weighted image, so that the information of the unsaturated region in the input image is preserved. The weighted image is used as one of the output branches.

[0050] The weighted network first generates a weight feature map from the input image in step 2 through four convolutional layers, and then performs an element-wise multiplication operation between the weight feature map and the input image in step 2 to obtain the weighted image; the weighted network is shown in formula (2):

[0051] (2)

[0052] in: This is an element-wise multiplication operation. It is a 3×3 convolutional layer. This is the weighted image.

[0053] Step 4: The backbone U-Net network of the lower branch first converts the input image in Step 2 into a feature map through a convolutional layer;

[0054] The convolutional layers in the backbone U-Net network are all 3×3 in size. The last convolutional layer in the backbone U-Net network has 3 output channels, while the others have 64.

[0055] Step 5: The feature map generated in Step 4 is subjected to deep extraction through feature mixing blocks and convolutional layers, so that the feature map generated in Step 4 can learn multi-scale feature representations, while capturing local contextual relationships to enhance the global information and local details of the feature map;

[0056] The feature mixing block comprises a spatial adaptive modulation layer and a convolutional channel mixing layer; both layers are residual connection structures; the spatial adaptive modulation layer learns image spatial structure information from multi-scale feature representations; the convolutional channel mixing layer utilizes channel mixing to enhance local contextual feature relationships and enhance local detail information.

[0057] To reduce model complexity and obtain pyramid feature representation, the spatial adaptive modulation layer first performs layer normalization on the feature map, and then performs channel splitting on the layer normalized input features to generate four parts.

[0058] The first part is processed by a 3×3 depthwise convolution, and the remaining parts are fed into feature generation units of multiple scales. The process of extracting pyramid features is shown in formulas (3), (4), and (5):

[0059] (3)

[0060] (4)

[0061] (5)

[0062] Where: Xi (i=0,1,2,3) represents different features after channel separation; This indicates a channel separation operation; X represents the input feature. (i=0,1,2,3) represent features extracted at different scales; This represents a depthwise convolution of size 3×3; This indicates that the feature map at initial resolution p is downsampled to... ,and For nearest neighbor upsampling, the feature map resolution is restored to the initial resolution p;

[0063] Then, the multi-scale feature maps are merged along the channels, and 1×1 convolutions are used to aggregate local and global relationships. Next, the GELU activation function is used to predict the attention map, and the input feature maps are adaptively modulated in the form of element-wise multiplication. This process is represented by formulas (6) and (7):

[0064] (6)

[0065] (7)

[0066] in: This indicates the characteristics after channel merging; This indicates a merge operation along the channel dimension; It is a 1×1 convolution; For GELU activation function, For element-wise multiplication, This represents the characteristics after spatial adaptive modulation;

[0067] The convolutional channel blending layer first performs layer normalization on the feature map, and then applies 3×3 convolution, GELU activation function and 1×1 convolution to the normalized feature map to enhance the local spatial modeling ability and achieve channel blending.

[0068] Step 6: At the encoding end, the feature map extracted in step 5 is downsampled twice. Between the two downsampling operations, serial residual blocks are used to extract features at different scales. Then, at the decoding end, the feature map is upsampled twice. Between the two upsampling operations, the same number of residual blocks as at the encoding end are used for image reconstruction. At the same time, the feature map before downsampling is added to the feature map after upsampling using skip connections to make up for the information lost due to downsampling and feature extraction.

[0069] The residual block contains a spatial adaptive modulation layer and a convolutional channel mixing layer in the feature mixing block. It employs a spatial adaptive modulation layer, 3×3 convolution, convolutional channel mixing layer and 3×3 convolution, and applies spatial adaptive modulation and channel mixing to the input feature map in the form of element-wise addition to the features at different scales.

[0070] Step 7: The feature map output from the decoding end in Step 6 is transformed back into the output image through the feature mixing block and convolutional layer. Then, it is added to the weighted image obtained in Step 3 to obtain the reconstructed image. Next, the pixel-level loss function with hyperbolic tangent is used to calculate the loss value of the ground truth of the input image and the reconstructed image. Backpropagation is performed on the network, and the network hyperparameters are iteratively optimized to finally obtain the trained model.

[0071] The ground truth (GT) of the input image and the reconstructed image are used to calculate the loss using a loss function. In step 7, a pixel-level loss function with hyperbolic tangent is used to calculate the loss value of the GT of the input image and the reconstructed image. This loss function is an L1 loss function with Tanh transform, which balances the learning weights between high brightness values ​​and low brightness values. The loss function is defined as follows:

[0072] (8)

[0073] in: To reconstruct the image; For reference GT image; H represents the image height; W represents the image width; C represents the RGB three channels; It is the hyperbolic tangent function. To perform L1 norm operations on the results.

[0074] In this invention .

[0075] Step 8: Reconstruct the real SDR image based on the trained model. Feed the SDR image into the model trained in Step 7 for repair to obtain a high-quality and highly faithful reconstructed image.

[0076] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. However, the above description is merely a specific embodiment of the present invention, and the technical features of the present invention are not limited thereto. Any other embodiments derived by those skilled in the art without departing from the technical solution of the present invention should be covered within the patent scope of the present invention.

[0077] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A high dynamic range image reconstruction method based on spatial adaptive modulation, characterized in that, The method is as follows: Step 1: Obtain the image dataset for training; Step 2: Preprocess the dataset from Step 1 to generate input images and input them into the network; Step 3: The overall architecture of the network consists of a weighted network in the upper branch and a backbone U-Net network in the lower branch. The weighted network extracts features from the input image in Step 2 to obtain a weighted image, which preserves the information of the unsaturated regions in the input image. The weighted image is used as one of the output branches. Step 4: The backbone U-Net network in the lower branch first converts the input image in Step 2 into a feature map through a convolutional layer. Step 5: The feature map generated in Step 4 is subjected to deep extraction through feature mixing blocks and convolutional layers, so that the feature map generated in Step 4 can learn multi-scale feature representations, while capturing local contextual relationships to enhance the global information and local details of the feature map; Step 6: At the encoding end, the feature map extracted in step 5 is downsampled twice. Between the two downsampling operations, serial residual blocks are used to extract features at different scales. Then, at the decoding end, the feature map is upsampled twice. Between the two upsampling operations, the same number of residual blocks as at the encoding end are used for image reconstruction. At the same time, the feature map before downsampling is added to the feature map after upsampling using skip connections to make up for the information lost due to downsampling and feature extraction. Step 7: The feature map output from the decoding end in Step 6 is transformed back into the output image through the feature mixing block and convolutional layer. Then, it is added to the weighted image obtained in Step 3 to obtain the reconstructed image. Next, the pixel-level loss function with hyperbolic tangent is used to calculate the loss value of the ground truth of the input image and the reconstructed image. Backpropagation is performed on the network, and the network hyperparameters are iteratively optimized to finally obtain the trained model. Step 8: Reconstruct the real SDR image based on the trained model. Feed the SDR image into the model trained in Step 7 for repair to obtain a high-quality and highly faithful reconstructed image.

2. The high dynamic range image reconstruction method based on spatial adaptive modulation according to claim 1, characterized in that, The formula for preprocessing the dataset in step 2 is shown in (1): ； (1) in: For the initial image dataset, For normalization function, This is a normalized image.

3. The high dynamic range image reconstruction method based on spatial adaptive modulation according to claim 2, characterized in that, In step 3, the weighted network first generates a weight feature map from the input image in step 2 through four convolutional layers, and then performs an element-wise multiplication operation between the weight feature map and the input image in step 2 to obtain the weighted image; the weighted network is shown in formula (2): ； (2) in: This is an element-wise multiplication operation. It is a 3×3 convolutional layer. This is the weighted image.

4. The high dynamic range image reconstruction method based on spatial adaptive modulation according to claim 1, characterized in that, The convolutional layers in the backbone U-Net network are all 3×3 in size, with the last convolutional layer of the backbone U-Net network having 3 output channels and the rest having 64.

5. The high dynamic range image reconstruction method based on spatial adaptive modulation according to claim 1, characterized in that, The feature mixing block in step 5 includes a spatial adaptive modulation layer and a convolutional channel mixing layer; both layers are residual connection structures; the spatial adaptive modulation layer learns image spatial structure information from multi-scale feature representations; the convolutional channel mixing layer uses channel mixing to enhance local contextual feature relationships and enhance local detail information.

6. The high dynamic range image reconstruction method based on spatial adaptive modulation according to claim 5, characterized in that, To reduce model complexity and obtain pyramid feature representation, the spatial adaptive modulation layer first performs layer normalization on the feature map, and then performs channel splitting on the layer normalized input features to generate four parts. The first part is processed by a 3×3 depthwise convolution, and the remaining parts are fed into feature generation units of multiple scales. The process of extracting pyramid features is shown in formulas (3), (4), and (5): ； (3) ； (4) ； (5) in: (i=0,1,2,3) represent different characteristics after channel separation; This indicates a channel separation operation; X represents the input feature. (i=0,1,2,3) represent features extracted at different scales; This represents a depthwise convolution of size 3×3; This indicates that the feature map at initial resolution p is downsampled to... ;and For nearest neighbor upsampling, the feature map resolution is restored to the initial resolution p; Then, the multi-scale feature maps are merged along the channels, and 1×1 convolutions are used to aggregate local and global relationships. Next, the GELU activation function is used to predict the attention map, and the input feature maps are adaptively modulated in the form of element-wise multiplication. This process is represented by formulas (6) and (7): ； (6) ； (7) in: This indicates the characteristics after channel merging; This indicates a merge operation along the channel dimension; It is a 1×1 convolution; For GELU activation function, For element-wise multiplication, This represents the characteristics after spatial adaptive modulation.

7. The high dynamic range image reconstruction method based on spatial adaptive modulation according to claim 5, characterized in that, The convolutional channel blending layer first performs layer normalization on the feature map, and then applies 3×3 convolution, GELU activation function and 1×1 convolution to the layer normalized feature map to enhance the local spatial modeling ability and achieve channel blending.

8. The high dynamic range image reconstruction method based on spatial adaptive modulation according to claim 1, characterized in that, The residual block includes a spatial adaptive modulation layer and a convolutional channel mixing layer in the feature mixing block. It employs a spatial adaptive modulation layer, 3×3 convolution, a convolutional channel mixing layer, and 3×3 convolution, and applies spatial adaptive modulation and channel mixing to the input feature map in the form of element-wise addition to the features at different scales.

9. The high dynamic range image reconstruction method based on spatial adaptive modulation according to claim 1, characterized in that, In step 7, a loss function is used to calculate the loss of the ground truth (GT) of the input image and the reconstructed image. Step 7 uses a pixel-level loss function with hyperbolic tangent to calculate the loss value of the GT and the reconstructed image. This loss function is an L1 loss function with Tanh transform, balancing the learning weights between high and low brightness values. The loss function is defined as follows: ； (8) in: To reconstruct the image; For reference GT image; H represents the image height; W represents the image width; C represents the RGB three channels; It is the hyperbolic tangent function; To perform L1 norm operations on the results.

10. The high dynamic range image reconstruction method based on spatial adaptive modulation according to claim 9, characterized in that, Official (8) .

Images