High dynamic range image tone mapping method and device based on attention mechanism

Abstract

The invention relates to a high dynamic range image tone mapping method and device based on an attention mechanism. The method comprises the following steps: S1, determining a first brightness channel; the first brightness channel is a brightness channel of the high dynamic range image. And S2, adaptively adjusting the global brightness of the first brightness channel based on a channel attention mechanism to obtain a first brightness feature map subjected to global brightness mapping. And S3, enhancing local details of the first brightness feature map based on a self-attention mechanism to obtain a second brightness feature map. And S4, obtaining a second brightness channel according to the first brightness channel and the second brightness feature map. The second brightness channel is a brightness channel of the low dynamic range image. According to the method, adaptive brightness modification is realized based on channel attention, the flexibility of brightness mapping can be improved, detail retention in tone mapping is enhanced based on a self-attention mechanism, global brightness mapping and local detail enhancement can be effectively integrated, and a more excellent tone mapping result is obtained.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a high dynamic range image tone mapping method and device based on an attention mechanism. Background Art

[0002] Existing tone mapping methods can be divided into traditional tone mapping methods and deep learning-based methods. Traditional methods implement tone mapping by manually designing tone mapping functions, which are usually parameter-sensitive and have poor adaptability to scenes. Manual adjustment of parameters is required to obtain satisfactory tone mapping results. Deep learning-based methods solve this problem by using a large amount of training data to enable the tone mapping model to learn complex mapping functions, thereby adapting to a variety of scenes.

[0003] About traditional tone mapping methods:

[0004] In the past few decades, researchers have proposed many tone mapping algorithms for high dynamic range (HDR) images to compress the dynamic range while preserving details. Traditional tone mapping methods can be divided into two types: global tone mapping algorithms and local tone mapping algorithms.

[0005] The global tone mapping algorithm uses a tone mapping function to map all pixels in the image, which is relatively simple, fast and efficient. The global tone mapping algorithm can maintain the overall effect of the original image. However, a single tone mapping operator is not suitable for all areas of HDR images. For darker and brighter areas in the scene, a single tone mapping function cannot distinguish the details in these areas and will result in the loss of local detail information in the low dynamic range image.

[0006] The mapping of pixels by the local tone mapping algorithm depends on the values ​​of its neighboring pixels. Different mapping methods are used in different areas of the image according to the local spatial distribution. Therefore, the local method can handle contrast well while retaining local details. However, the local tone mapping algorithm may introduce some obvious halo artifacts.

[0007] The limitation of traditional tone mapping algorithms lies in their parameter sensitivity. The performance of tone mapping for HDR images of different scenes varies greatly. In order to obtain satisfactory tone mapping results, extensive manual parameter adjustments are usually required.

[0008] About the deep learning tone mapping method:

[0009] In recent years, with the development of convolutional neural networks, some tone mapping algorithms based on deep learning technology have been proposed and achieved good results. Deep learning models have strong fitting and generalization capabilities and can learn the tone mapping function of HDR images from a large amount of training data. Existing tone mapping algorithms based on deep learning have better performance than traditional methods, and to a large extent solve the parameter sensitivity of traditional methods and have a wider range of adaptability.

[0010] However, since the current methods are all based on convolution, which is local and has limited modeling capabilities for global dependencies, the model's performance in modeling image details is limited, resulting in poor processing capabilities of CNN-based tone mapping algorithms for original image details, and problems such as over-enhancement, detail loss, and the introduction of halo artifacts. In addition, the current methods combine the two important operations of HDR images in tone mapping, namely brightness mapping and detail enhancement, without explicit distinction, so it is difficult to balance the two operations, resulting in the problem of unbalanced brightness mapping of tone mapped images.

[0011] For example, in CSRNet (Crowded Scene Recognition Network), the pixel modification operation is approximated as a multi-layer perceptron (MLP), and a conditional network is used to generate global features for modulation. Since this method generates fixed global features, it limits the flexibility of the network to perform adaptive modifications at different layers, which ultimately leads to blurred tone mapping results. Summary of the invention

[0012] The technical problem to be solved by the present invention is to provide a high dynamic range image tone mapping method and device based on an attention mechanism in view of the above-mentioned defects.

[0013] The technical solution adopted by the present invention to solve the technical problem is: a high dynamic range image tone mapping method based on an attention mechanism, comprising the following steps:

[0014] S1. Determine a first brightness channel; the first brightness channel is a brightness channel of a high dynamic range image;

[0015] S2. Adaptively adjusting the global brightness of the first brightness channel based on a channel attention mechanism to obtain a first brightness feature map after global brightness mapping;

[0016] S3. Enhancing local details of the first brightness feature map based on a self-attention mechanism to obtain a second brightness feature map;

[0017] S4. Obtain a second brightness channel according to the first brightness channel and the second brightness feature map; the second brightness channel is a brightness channel of a low dynamic range image.

[0018] In one embodiment, after step S1 and before step S2, the method further includes:

[0019] S11, decomposing the first brightness channel into multi-dimensional brightness components;

[0020] Step S2 includes:

[0021] Each brightness component is adaptively modified based on the channel attention allocation weight to obtain the first brightness feature map.

[0022] In one embodiment, step S11 includes:

[0023] The first brightness channel is decomposed through a 1×1 convolution layer and projected into a high-dimensional space to obtain a multi-dimensional brightness component.

[0024] In one embodiment, step S2 comprises:

[0025] Generate a weight vector based on the difference of the brightness component itself by stacking multiple adaptive brightness modulation modules, and determine whether to compress the brightness component or enhance the brightness component according to the weight vector, so as to realize adaptive brightness mapping and obtain the first brightness feature map;

[0026] The adaptive brightness modulation module is formed by combining a gated feedforward network and a channel attention modulation module.

[0027] In one embodiment, the working principle of the adaptive brightness adjustment module is expressed as follows:

[0028] X N =LN(X)

[0029] X M =CAM(X N )+X

[0030] Y = GDFN(LN(X M ))+X M

[0031] Among them, X is the input brightness image, H and W represent the height and width of the input image, C represents the number of feature channels; X N and X Mis the intermediate feature; Y is the output of the adaptive brightness modulation module, representing the modified first brightness feature map; LN(·) represents the layer normalization operation; GDFM(·) represents the gated feedforward network; CAM(·) represents the channel attention modulation module, which is composed of a convolutional layer and a GELU activation function.

[0032] In one embodiment, step S3 comprises:

[0033] Extracting a global self-attention map and local features of the first brightness feature map through a multi-head convolution transposed self-attention module and a convolutional neural network path;

[0034] A modulation matrix is ​​generated based on the global self-attention map, and the local features are modulated using the modulation matrix to obtain the second brightness feature map.

[0035] In one embodiment, step S4 comprises:

[0036] fusing the first brightness channel and the second brightness feature map by a global residual connection, and reconstructing the fused result by a reconstruction module to obtain the second brightness channel; or

[0037] Step S1 includes:

[0038] A high dynamic range image is acquired, and a brightness channel of the color space thereof is normalized to the maximum and minimum values ​​to obtain the first brightness channel.

[0039] In one embodiment, the method further comprises:

[0040] Get the hue channel and saturation channel of a high dynamic range image;

[0041] The second brightness channel is combined with the hue channel and the saturation channel to obtain a tone mapping output result.

[0042] In addition, the present invention also provides a high dynamic range image tone mapping device based on an attention mechanism, the device comprising a tone mapping network model; the tone mapping network model is constructed using the tone mapping method as described above.

[0043] In one embodiment, the device further comprises a training module for training the tone mapping network model according to training data and an L1 loss function; wherein the training data is determined as follows:

[0044] A preset number of tone mapping algorithms are selected to generate tone mapped LDR images, and the generated LDR images are evaluated and ranked by a tone mapping quality index to determine a target LDR image, and the target LDR image is used as training data.

[0045] The implementation of the high dynamic range image tone mapping method and device based on the attention mechanism of the present invention has the following beneficial effects: the two-stage tone mapping method proposed in the present invention can improve the flexibility of brightness mapping by realizing adaptive brightness modification based on channel attention, enhance the detail retention in tone mapping based on the self-attention mechanism, and can effectively integrate global brightness mapping and local detail enhancement to obtain better tone mapping results. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0047] Figure 1 is a flow chart of a high dynamic range image tone mapping method based on an attention mechanism provided by an embodiment of the present invention;

[0048] Figure 2 is a flow chart of a high dynamic range image tone mapping method based on an attention mechanism provided by an embodiment of the present invention;

[0049] Figure 3 It is a schematic diagram of the architecture of a high dynamic range image tone mapping network model based on an attention mechanism provided by an embodiment of the present invention;

[0050] Figure 4 is a schematic diagram of a global brightness mapping process provided by an embodiment of the present invention;

[0051] Figure 5 is a logical diagram of channel attention modulation provided by an embodiment of the present invention;

[0052] Figure 6 is a logic diagram of local enhanced self-attention processing provided by an embodiment of the present invention;

[0053] Figures 7 to 10 A subjective comparison of different tone mapping results is shown;

[0054] Fig.11 It is a subjective visual comparison diagram of the architecture ablation experiment of the method of the present invention. DETAILED DESCRIPTION

[0055] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described in detail with reference to the accompanying drawings.

[0056] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0057] In order to facilitate the understanding of the present invention, the present invention will be described more fully below, and the present invention can be implemented in many different forms and is not limited to the embodiments described herein. On the contrary, the purpose of providing these embodiments is to make the disclosure of the present invention more thorough and comprehensive.

[0058] It should be understood that although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present invention, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. It is not necessarily required or implied that there is any such actual relationship or order between these entities or operations.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the present invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention.

[0060] In the present invention, HDR image refers to a high dynamic range image, and LDR image refers to a low dynamic range image.

[0061] In one of the embodiments provided by the present invention, reference Figure 1 The high dynamic range image tone mapping method based on the attention mechanism of this embodiment includes the following steps:

[0062] S1. Determine a first brightness channel. The first brightness channel is a brightness channel of a high dynamic range image. Alternatively, in this step, the embodiment obtains a high dynamic range image and performs maximum and minimum normalization processing on the brightness channel of its color space to obtain a first brightness channel.

[0063] After step S1 and before step S2, the method further includes: S11, decomposing the first brightness channel into multi-dimensional brightness components In some embodiments, the first brightness channel may be decomposed by a 1×1 convolution layer and projected into a high-dimensional space to obtain a multi-dimensional brightness component.

[0064] Of course, as an option, the first brightness channel can also be directly the brightness channel of the high-dimensional space

[0065] like Figure 3 As shown, following the classic tone mapping algorithm design, this embodiment preferably processes the brightness channel of a 32-bit HDR image and performs color restoration after output. Specifically, before the brightness channel is input into the network, the brightness channel (V) of the HSV color space is normalized using the maximum and minimum values ​​and then input into the network. For the input of the network, given a brightness channel of a high dynamic range image First, a 1×1 convolutional layer is used to process the input single brightness channel to a high-dimensional space representation. Among them, H and W represent the height and width of the input image, and C represents the number of feature channels. In other words, the process can be regarded as an operation of brightness decomposition, decomposing it into different brightness components. Represents a real tensor of dimension H×W×1, which represents a three-dimensional tensor with height H, width W and a single channel. This symbol is used here mainly to indicate the shape of I_HDR_v, that is, the height and width are H×W, and there is only one channel. m is multi, which means multi-dimensional, and is an optional hyperparameter. In the actual implementation of this embodiment, it is 64.

[0066] S2. Adaptively adjust the global brightness of the first brightness channel based on the channel attention mechanism to obtain a first brightness feature map after global brightness mapping.

[0067] It should be noted that the Channel Attention Mechanism is a weighted strategy based on the channel dimension of the input feature map. It focuses on the most important channels in the input feature map and gives them greater weight, so that the network can extract and utilize key information more effectively. The design of the Channel Attention Mechanism is inspired by the selective attention to different visual features in the human visual system.

[0068] The channel attention mechanism usually maps each channel of the input feature map to a weight value by learning a mapping function. Common implementation methods include using global average pooling or global maximum pooling to generate global features that describe the importance of the channel, and then calculating the attention weight of each channel through a fully connected layer and an activation function. A typical channel attention mechanism model is SENet (Squeeze-and-Excitation Networks), which performs global average pooling on each channel of the feature map to generate a one-dimensional vector representation. Then, after a series of fully connected layers and nonlinear activation functions, the attention weight of each channel is finally obtained. This weight is then applied to the original feature map to weight each channel, thereby enhancing or suppressing the importance of different channels.

[0069] The main advantage of the channel attention mechanism is that it is simple and effective, can significantly improve the representation ability of the network, and has low computational complexity. However, the channel attention mechanism also has certain limitations. For example, it only focuses on the information of the channel dimension and may ignore important features in the spatial dimension.

[0070] In some embodiments, the step includes: adaptively modifying each brightness component based on the channel attention allocation weight to obtain a first brightness feature map. Alternatively, a weight vector based on the difference of its own brightness components is generated by stacking multiple adaptive brightness modulation modules, and judging whether to compress the brightness component or enhance the brightness component according to the weight vector, thereby realizing adaptive brightness mapping and obtaining the first brightness feature map. The adaptive brightness modulation module is formed by combining a gated feedforward network and a channel attention modulation module.

[0071] S3. Enhance the local details of the first brightness feature map based on the self-attention mechanism to obtain a second brightness feature map.

[0072] It should be noted that the overall brightness and local detail quality of the image are the key factors in evaluating the tone mapping results of HDR images. This paper divides the tone mapping process into two stages: global brightness mapping and local detail enhancement.

[0073] The Self-Attention Mechanism is a method for feature extraction by calculating the correlation between each element in the input sequence and other elements. Unlike channel attention, the self-attention mechanism is no longer limited to the channel dimension, but models the relationship between elements of the entire input sequence (such as an image or a text). The self-attention mechanism was initially widely used in the field of natural language processing, and was successfully promoted in the Transformer model.

[0074] The core of the self-attention mechanism is to calculate the "query", "key" and "value" of each element in the input sequence, and obtain the attention weight through the dot product operation of these vectors. Specifically, each element of the input sequence is mapped to a query vector and a key vector, and then the similarity between the query vector and the key vector is calculated to obtain the attention score. Then, these attention scores are used to perform weighted summation on the corresponding value vectors to generate a new output representation. In the multi-head self-attention mechanism, the input is divided into multiple heads, each head calculates attention independently, and then the outputs of each head are spliced ​​together to form the final output. This multi-head mechanism allows the model to capture different features of the input sequence from multiple angles.

[0075] The self-attention mechanism is widely used in natural language processing and computer vision. For example, in machine translation tasks, the Transformer model uses the self-attention mechanism to effectively model long-distance dependencies between sequences, thereby outperforming traditional recurrent neural networks (RNNs) and convolutional neural networks (CNNs). In the field of computer vision, the self-attention mechanism is used for tasks such as image generation, image classification, and object detection. The Visual Transformer (ViT) model is a typical example, which achieves excellent classification performance by dividing the image into several small blocks and performing self-attention calculations on these small blocks.

[0076] The advantage of the self-attention mechanism is its powerful representation ability, which can capture the dependencies between distant elements in the input sequence, which makes it perform well when processing long sequence data. In addition, the self-attention mechanism is also highly interpretable, because the attention weights can directly show which input elements contribute more to the decision-making process. However, the self-attention mechanism also has some limitations. First, the computational complexity is high, especially when processing long sequence data, calculating the correlation between each element and other elements consumes a lot of computing resources. Second, the self-attention mechanism has a strong dependence on large-scale data and may not perform well on small data sets. In addition, in some tasks, the self-attention mechanism may ignore local information, which may affect model performance in tasks with strong local features such as processing images.

[0077] In some embodiments, this step includes: extracting the global self-attention map and local features of the first brightness feature map through a multi-head convolution transposed self-attention module and a convolutional neural network path. Generate a modulation matrix based on the global self-attention map, and modulate the local features using the modulation matrix to obtain a second brightness feature map. This embodiment performs high dynamic range image tone mapping by building a two-stage network model based on an attention mechanism; adaptively assigns brightness component weights based on a channel attention mechanism to modulate the global brightness of the HDR image. Based on Tranformer self-attention, the advantages of Transformer global features and convolution local features are combined to enhance detail retention in tone mapping.

[0078] S4, obtaining a second brightness channel according to the first brightness channel and the second brightness feature map. The second brightness channel is a brightness channel of the low dynamic range image.

[0079] In some embodiments, this step includes: fusing the first brightness channel and the second brightness feature map by means of a global residual connection, and reconstructing the fused result by means of a reconstruction module to obtain a second brightness channel.

[0080] It is understandable that Figure 3As shown, in order to ensure the stability of training, this embodiment adds a global residual connection to fuse shallow features and deep features, and then reconstructs high-quality results through a reconstruction module (Also the second brightness channel):

[0081]

[0082] Among them, H Rec represents the reconstruction module, which contains a 1×1 convolution layer to project the high-dimensional brightness feature map to 1 dimension to obtain a high-quality LDR image brightness channel. Combined with the original unprocessed H and S channels to obtain the tone-mapped output. The network is trained using L1 loss:

[0083]

[0084] in, for The luminance channel of the target LDR image.

[0085] In some embodiments, Figure 3 As shown, the method further includes: obtaining a hue channel (H channel) and a saturation channel (S channel) of the high dynamic range image, and merging the second brightness channel with the hue channel and the saturation channel to obtain a tone mapping output result. It can be understood that this embodiment preferably processes the brightness channel of a 32-bit HDR image and performs color restoration after output.

[0086] In a specific embodiment, Figure 2 and Figure 3 As shown, step S2 corresponds to the first stage of the network, namely the global brightness mapping stage. This stage is composed of N1 adaptive brightness modification modules (ALMBs) stacked together to adaptively modify each brightness component based on the channel attention weight allocation, and finally obtain the mapped first brightness feature map. Step S3 corresponds to the detail enhancement stage of the network. After global brightness mapping, some details in the original HDR image may be smoothed. This stage is composed of N2 local detail enhancement Transformer modules (LETB) stacked together to reproduce and enhance the local details in the original HDR image. It receives the output of the brightness mapping stage. Get the output after enhancing the details That is the second brightness characteristic map.

[0087] About global brightness mapping: Figure 4As shown, the network of this embodiment takes the brightness channel of the HDR image as input, and the single-channel input is first projected into a high-dimensional space through a convolutional layer. This process can be regarded as a decomposition operation on the brightness. Therefore, the intermediate features can be regarded as brightness maps, and each channel represents a different brightness component. Performing an affine transformation on each brightness component can achieve a brightness mapping effect. This approximate mathematical representation is:

[0088] I′ c (x,y)=a c I c (x,y)+b c

[0089] Among them, I c (x, y) represents the original pixel value of channel c at position (x, y), I′ c (x, y) represents the pixel value after affine transformation, a c is the scaling factor, b c is the offset of channel c.

[0090] In addition, we note that in CSRNet, the modification operation on the intermediate feature map uses a conditional network to generate global features to modulate the intermediate features. This results in using fixed global features to perform affine transformations on each channel of the intermediate feature map, limiting the network's ability to adaptively perform different modification operations on different layers, resulting in over-smoothing of the image.

[0091] To address this problem, we propose a simple yet effective adaptive luminance modulation block (ALMB) at this stage and stack multiple ALMB modules to flexibly perform brightness modification operations. The proposed ALMB first generates a weight vector based on the difference of its own brightness component through channel attention modulation (CAM), thereby determining whether to compress or enhance the brightness component to achieve adaptive brightness mapping. Figure 5As shown in the figure, CAM performs an affine transformation on each channel of the feature map through channel attention. In order to perform more precise transformations on each channel, we use global average pooling and maximum pooling to generate scaling and shifting parameters, respectively. Average pooling is used to generate scaling parameters that compress brightness to a low dynamic range, while maximum pooling generates shifting parameters that achieve adaptive balancing of brightness components. Specifically, CAM first processes the input features through deep convolution (DWConv). The features are then divided into two parallel paths: one path undergoes maximum pooling and the other path undergoes average pooling. Both paths undergo 1×1 convolution, GELU activation, and another 1×1 convolution. The outputs of these paths are combined through element-wise addition and multiplication operations to generate the final modulated features. Through CAM, the proposed ALMB can perform brightness modification operations more flexibly.

[0092] At the same time, ALMB follows the network design of Transformer. Similar to using a feedforward network in Transformer to improve feature representation, we use a gated-DWConv feed-forward network (GDFN) to enhance the effect of brightness modification for better performance. This feedforward network filters out features with less information through a gating mechanism, thereby retaining more useful information and using DWConv to avoid local information loss.

[0093] Specifically, given an input brightness map The working principle of the adaptive brightness adjustment module is expressed as:

[0094] X N =LN(X)

[0095] X M =CAM(X N )+X

[0096] Y = GDFN(LN(X M ))+X M

[0097] Where H and W represent the height and width of the input image, and C represents the number of feature channels. N and X M is the intermediate feature. Y is the output of the adaptive brightness modulation module, which represents the modified first brightness feature map. LN(·) represents the layer normalization operation. GDFN(·) represents the gated feedforward network. CAM(·) represents the channel attention modulation module, which consists of a convolutional layer and a GELU activation function. Figure 5 As shown, given input Generate scale parameters through average pooling and maximum pooling respectively and shift parameters Then CAM can be formulated as: CAM(X N )=γ*X N +β, performs channel-by-channel affine transformation on the input features.

[0098] Regarding local detail enhancement: Although ALMB can modify the global brightness by compressing the dynamic range of HDR images, it has poor modeling effect on local details because it relies on global features to guide channel-level brightness modification without considering local details. Some CNN-based tone mapping algorithms perform tone mapping, but due to the limitation of the convolution receptive field, they cannot model long-range dependencies, resulting in poor detail preservation. Transformer-based methods can provide long-range feature dependencies through cascaded self-attention mechanisms, but their computational complexity is usually high. In order to reduce the computational complexity, researchers have recently proposed a transposed self-attention mechanism, which implicitly generates an attention map that encodes global context by calculating the cross-covariance across channels. However, this method lacks the ability to preserve local feature details. To address the above problems, we propose LETB (Local Enhanced Self-Attention Block), which combines the advantages of CNN and Transformer to achieve better detail preservation while reducing computational complexity.

[0099] Specifically, Figure 6 As shown in the figure, LETB contains a local enhancement self-attention module (LESA). This module combines the multi-head DWConv transposed self-attention (MDTA) module and the CNN path. Among them, MDTA extracts the global self-attention map, while CNN extracts local features. Then, MDTA modulates the CNN features using a modulation matrix based on the global self-attention map, thereby enhancing the global modeling ability of local features. Finally, the output of LETB naturally inherits the global context modeling ability of Transformer and the local representation ability of CNN, achieving better detail enhancement effect.

[0100] Specifically, given a layer normalized feature For the self-attention path, we first generate the query vector query using a 1×1 convolution and a 3×3 depth-wise separable convolution key vector key and the value vector value Then reshape them into At the same time, they are divided into multiple heads according to the channels, and self-attention is calculated in parallel. For simplicity, one head is used as an example below. The calculation of self-attention can be expressed as:

[0101]

[0102] Among them, α is a learnable temperature parameter used to adjust the softmax function before and The inner product calculation of . We also need to reshape it into Get the same shape as the input. For the parallel CNN path, we use depthwise separable convolution to obtain the local features Y C =DWConv(X), Finally, we generate a modulation matrix with the global self-attention map to modulate the local features obtained by CNN:

[0103]

[0104] Among them, W (·) represents the convolutional layer, f(·) represents the sigmoid function, σ(·) represents the GELU activation function, and ⊙ represents element-wise multiplication. Y is the output of the local detail enhanced self-attention. Finally, the output of LETB is obtained through the gated feedforward network.

[0105] It should be noted that HxWxC is a common three-dimensional representation of an image, which retains spatial and channel information. HWxC is an input form that flattens the image into a sequence, which is suitable for Transformer processing. The main difference is the different representations of image data. In this embodiment, the Transformer can process the image data after two reshapings, that is, it is flattened into a sequence (HWxC) before input and restored to the original image form (HxWxC) after output. Dividing into multiple heads by channel means that every several channels form a head. In Transformer, the typical choice of the number of heads is 4-16 heads. Self-attention is calculated in units of heads. The concept of head division is in the internal calculation process of MDTA's self-attention, which is not reflected in the figure. In the present invention, the Transformer and L1 damage function can refer to the prior art and will not be repeated here.

[0106] It can be understood that each brightness component will be processed by all ALMB modules. For an ALMB module, it generates adaptive weights for each brightness component based on all current brightness components based on channel attention, and then each brightness component is modified at the same time. The LETB module is similar, except that it performs enhancement of local details. The ALMB module processes each channel (brightness component) as a whole, while the LETB module processes some pixels in each channel (brightness component). The principle of brightness modification operation can be flexibly performed by stacking multiple ALMB modules: similar to a multi-layer perceptron (MLP), if it is single, it is similar to making a simple linear transformation of each brightness component, and the multi-layer can be regarded as a combination of functions, which can approximate complex functions. The multi-layer structure can enhance the nonlinear representation ability of the network, thereby effectively learning the characteristics of complex data and improving the adaptability to various complex tasks.

[0107] In another embodiment provided by the present invention, a high dynamic range image tone mapping device based on an attention mechanism in this embodiment includes a tone mapping network model. The tone mapping network model is constructed using the tone mapping method of the above embodiment. It can be understood that the overall brightness and local detail quality of the image are key factors in evaluating the tone mapping results of the HDR image. The two-stage tone mapping network proposed in this embodiment divides the tone mapping process into two stages: global brightness mapping and local detail enhancement. The overall architecture of the tone mapping network model is as follows: Figure 3 shown.

[0108] In some embodiments, the apparatus of this embodiment further comprises a training module for training the tone mapping network model according to the training data and the L1 loss function. The training data is determined by selecting a preset number of tone mapping algorithms to generate a tone mapped LDR image, and evaluating and ranking the generated LDR images by a tone mapping quality index to determine a target LDR image, and using the target LDR image as training data.

[0109] The network model is trained using L1 loss:

[0110]

[0111] in, for The luminance channel of the target LDR image.

[0112] It should be noted that L1 loss (L1 Loss), also known as Mean Absolute Error (MAE) or L1 norm loss, is a commonly used loss function, especially in regression tasks. It measures the performance of the model by calculating the average of the absolute difference between the model's predicted value and the true value.

[0113] During the training process, the L1 loss function has the following key features and application scenarios:

[0114] Robustness to outliers: L1 loss is not as sensitive to outliers as L2 loss (mean squared error) because it does not square the error. This makes L1 loss more robust in the presence of noise or outliers.

[0115] Sparsity: In some cases, L1 loss can lead to sparsity of model parameters, that is, the coefficients of some features can be compressed to zero, thereby achieving feature selection. This is particularly evident in Lasso regression.

[0116] Optimization properties: The gradient of the L1 loss is discontinuous when the predicted value is equal to the true value, which can cause some challenges during optimization, but can usually be handled efficiently by modern optimization algorithms (such as Adam).

[0117] Application scenarios: L1 loss is suitable for regression problems, especially when the distribution of the target variable has outliers. L1 loss is more suitable because of its robustness to outliers.

[0118] 1. Experimental Details

[0119] In order to test the tone mapping algorithm proposed in this paper, we need a large-scale HDR image dataset with diverse scenes. To this end, we collected a total of 659 HDR images from LVZ-HDR and HDRI Haven, covering a variety of scenes including indoor, outdoor, sunset, night, etc.

[0120] LVZ-HDR contains 456 32-bit HDR images synthesized from LDR images at five different exposure levels. We selected 288 high-quality HDR images from this dataset and eliminated low-quality images. HDRI Haven contains 736 high-quality panoramic HDR images taken and produced by different creators. Since there are many similar and low-diversity images in HDRI Haven, we carefully selected 371 unique HDR images covering a variety of scenes. To standardize the dataset, all selected HDR images were resized to 512×512. In particular, for the panoramic images of HDRI Haven, due to the obvious distortion in its edge areas, we first cropped the center part of the image to focus on the main subject, and then resized it to 512×512.

[0121] Since we use a supervised training method, we also need to obtain tone-mapped LDR images. Following previous methods, we selected 10 high-quality tone mapping algorithms: Gamma mapping, FattalTMO, ReinhardTMO, AshikhminTMO, LischinskiTMO, DragoTMO, RamanTMO, LiangTMO, DurandTMO, KimKautzTMO. These algorithms are used to generate target LDR images. Specifically, for each HDR image, we apply these 10 tone mapping algorithms to generate the corresponding LDR output, and then evaluate and rank these results using the Tone Mapping Quality Index (TMQI). The output image with the highest score is selected as the target LDR image for that HDR image. TMQI is a widely used objective indicator specifically designed to evaluate the quality of tone-mapped images. In this way, we ensure that the selected target LDR images have the highest visual quality, thereby providing high-quality training data for our model.

[0122] To evaluate the performance of our model, we used 300 HDR images as well as LDR images manually adjusted by photographers. These images serve as a benchmark for evaluating the performance of the model in real applications. This evaluation method allows us to fully understand the tone mapping effect of the model in different scenes, thereby verifying its effectiveness and robustness.

[0123] 2. Experimental results

[0124] Our proposed method (TSTMNet) is compared with the current state-of-the-art tone mapping methods, including DurandTMO, DeepTMO, CSRNet, DragoTMO, AshikhminTMO, and RamanTMO. CSRNet and DeepTMO are deep learning based methods implemented using the source code provided by the authors, while the remaining methods are traditional methods implemented using HDRToolBox. All methods are executed using the default parameters provided in their original papers.

[0125] 2.1 Subjective evaluation

[0126] Figures 7 to 10 A subjective comparison of different tone mapping results is shown. HDR Linear refers to the HDR image displayed without tone mapping, showing severe underexposure. From the visual results, it can be observed that our method achieves a good balance between global brightness mapping and detail preservation, while other methods have various problems. Figures 6 to 8 In the figure, the enlarged local area of ​​interest (marked with red or green boxes) shows specific details.

[0127] Figure 7 An outdoor scene is shown, where our method effectively balances the global brightness. Whether in the highly bright area in the upper left corner or the dark areas in the center and bottom of the image, our method preserves the details of the original image without overexposure or underexposure. In contrast, other methods exhibit underexposure and saturation anomalies.

[0128] exist Figure 8 In the indoor scene analysis, the first four traditional methods still exhibit the above problems. The deep learning methods CSRNet and DeepTMO as well as our TSTMNet all achieve good overall visual effects. However, in terms of details, our method performs better: CSRNet causes the image to become blurred and loses local details, while DeepTMO over-enhances and turns blue text into black. The present invention strikes a balance between the two.

[0129] like Fig. 9 As shown in Figure 3, wide-angle scenes with extremely high dynamic range pose challenges to most traditional methods, making them almost ineffective. CSRNet exhibits overexposure, while DeepTMO generates artifacts around the window. In contrast, our method achieves superior visual results, preserving details without overexposure or artifacts. AshikhminTMO, DragoTMO, and RamanTMO lose details in dark areas, while DurandTMO leads to overenhancement.

[0130] Fig.10The performance of different tone mapping operations on mixed indoor and outdoor scenes is demonstrated. Our method once again strikes a good balance, effectively preserving details at different brightness levels. Traditional methods such as AshikhminTMO and DragoTMO lose details in dark areas, while DurandTMO over-enhances. RamanTMO suffers from color cast issues, and CSRNet blurs the image, losing key details. DeepTMO also generates artifacts in the sky. In contrast, our method consistently provides high-quality results, demonstrating its robustness across a variety of scenarios.

[0131] 2.2 Objective evaluation

[0132] To further demonstrate the performance of our method, we use TMQI as an objective evaluation metric for tone mapping results. TMQI evaluates the structural fidelity and naturalness of tone mapped images and provides a final score from 0 to 1, with higher scores indicating better fidelity and naturalness. Table 1 shows the average TMQI score for each tone mapping method on a test set of 300 HDR images (based on the average of 5 training runs for deep learning methods). Our method achieves the highest overall TMQI (0.9014) and naturalness (0.6699) scores, indicating excellent visual quality.

[0133] Table 1 Comparison of objective indicators

[0134] method TMQI Fidelity Naturalness AshikhminTMO

[57] 0.8266 0.8377 0.2678 DragoTMO

[33] 0.8519 0.8965 0.2955 RamanTMO

[59] 0.8523 0.8469 0.3870 Durand TMO

[10] 0.8526 0.8592 0.3782 CSRNet

[21] 0.8781 0.8052 0.5619 DeepTMO

[17] 0.8876 0.8198 0.5986 Methods 0.9014 0.8219 0.6699

[0135] Although our method ranks fifth in terms of structural fidelity, we believe this represents a trade-off. The TMQI score is composed of structural fidelity and naturalness, and there is a balance between the two. Excessive preservation of details in the original image can lead to over-enhancement, negatively affecting the naturalness of the tone-mapped image and thus the overall visual quality, as seen in DurandTMO. Our method balances these factors, avoiding over-enhancement and maintaining a high level of naturalness.

[0136] Overall, our tone mapping results are closest to the real visual perception. Our method performs well in color, detail, and brightness, further verifying the effectiveness of our method. By dividing the tone mapping process into two stages: global brightness mapping and local detail enhancement, and balancing these two operations, our method not only achieves excellent brightness mapping, but also effectively preserves details.

[0137] 2.3 Ablation Experiment

[0138] We further evaluate the impact of each component in the proposed method and compare it with baseline models. To ensure a fair comparison, all baseline models are trained using the same settings as our proposed TSTMNet and tested using the same test set as above.

[0139] 2.3.1 Effectiveness of network architecture design

[0140] The proposed model consists of two stages: global brightness mapping and local detail enhancement, which are implemented by ALMB and LETB modules respectively. To demonstrate the rationality of the network architecture design, we compare the proposed model with three baseline models. One model only contains the global brightness mapping stage, i.e., it only consists of the ALMB module (called TSTMNetALMB). Another model only contains the local detail enhancement stage, i.e., the LETB module (called TSTMNetLETB). The last model swaps the order of the two stages, i.e., detail enhancement is performed first and then brightness mapping (called TSTMNetL-A). To ensure fairness, the total number of modules in all models remains unchanged, and we train the three baseline models using the same settings as the proposed method. The comparison results are summarized in Table 2.

[0141] Table 2 Ablation experiments on model architecture design

[0142] Model ALMB LETB TMQI <![CDATA[TSTMNet ALMB ]]> √ 0.8901 <![CDATA[TSTMNet LETB ]]> √ 0.8939 <![CDATA[TSTMNet L-A ]]> √ √ 0.8985 TSTMNet √ √ 0.9028

[0143] The results show that using ALMB or LETB alone leads to lower performance than using both modules simultaneously, while swapping the order of stages also performs worse than the proposed model. Using ALMB alone achieves a good TMQI score (0.8901), but shows limitations in other aspects of tone mapping. Although using LETB alone enhances local details, the improvement in TMQI score is not significant. Combining the two modules significantly improves the TMQI, reaching a score of 0.9028. Fig.11 The differences in visual effects of different models are intuitively demonstrated, further proving the effectiveness of the proposed architecture design.

[0144] 2.3.2 Effectiveness of ALMB

[0145] The proposed ALMB module adaptively adjusts the global brightness of the HDR image based on channel attention. To demonstrate the effectiveness of the proposed ALMB, we modified the channel attention modulation module in the ALMB module in the proposed network, that is, removed the channel attention modulation and only kept the convolution (DWConv). We trained this baseline model with the same parameter settings as the proposed model for fair comparison. As shown in Table 3, the use of the channel attention-based modulation module improves the TMQI score by 0.0034, demonstrating the effectiveness of ALMB.

[0146] Table 3 Ablation experiments on the effectiveness of ALMB

[0147] Model CAM TMQI w / o CAM 0.8994 TSTMNet √ 0.9028

[0148] 2.3.3 Effectiveness of LETB

[0149] The proposed LETB utilizes global self-attention to guide detail enhancement. To demonstrate the effectiveness of LETB, we gradually add components to LETB. Initially, LETB only includes the DWConv path, then we add the MDTA path, and finally, we add the modulation of the convolutional features by global self-attention. All baseline models are trained under the same setting. As shown in Table 4, adding the MDTA path and global self-attention modulation significantly improves the performance, demonstrating the effectiveness of LETB.

[0150] Table 4 Ablation experiment of LETB effectiveness

[0151] Model DWConv path SA path SA Modulation TMQI <![CDATA[TSTMNet1]]> √ 0.8707 <![CDATA[TSTMNet2]]> √ √ 0.8876 TSTMNet √ √ √ 0.9028

[0152] The key technical points of the present invention are:

[0153] (1) A two-stage network model is built based on the attention mechanism to perform high dynamic range image tone mapping.

[0154] (2) Based on the channel attention mechanism, the brightness component weights are adaptively allocated to modulate the global brightness of the HDR image.

[0155] (3) Based on Transformer self-attention, it combines the advantages of Transformer global features and convolutional local features to enhance detail preservation in tone mapping.

[0156] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0157] Any modification, replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention. In addition, the design of a two-stage network model based on the attention mechanism to perform tone mapping, the use of a global brightness modulation module to map global brightness, and the use of a local detail enhancement Transformer module to enhance the details of tone mapping all belong to the protection scope of the present invention.

[0158] It can be understood that the above embodiments only express the preferred implementation modes of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the patent scope of the present invention. It should be pointed out that, for ordinary technicians in this field, the above technical features can be freely combined without departing from the concept of the present invention, and several deformations and improvements can be made, which all belong to the protection scope of the present invention. Therefore, all equivalent changes and modifications made to the scope of the claims of the present invention should belong to the coverage of the claims of the present invention.

Claims

1. A high dynamic range image tone mapping method based on attention mechanism, characterized in that: The following steps are involved: S1. Determine a first brightness channel; the first brightness channel is a brightness channel of a high dynamic range image; S2. Adaptively adjusting the global brightness of the first brightness channel based on a channel attention mechanism to obtain a first brightness feature map after global brightness mapping; S3. Enhancing local details of the first brightness feature map based on a self-attention mechanism to obtain a second brightness feature map; S4. Obtain a second brightness channel according to the first brightness channel and the second brightness feature map; The second brightness channel is a brightness channel of a low dynamic range image.

2. The tone mapping method according to claim 1, characterized in that: After step S1 and before step S2, the method further includes: S11, decomposing the first brightness channel into multi-dimensional brightness components; Step S2 includes: Each brightness component is adaptively modified based on the channel attention allocation weight to obtain the first brightness feature map.

3. The tone mapping method according to claim 2, characterized in that: Step S11 includes: The first brightness channel is decomposed through a 1×1 convolution layer and projected into a high-dimensional space to obtain a multi-dimensional brightness component.

4. The tone mapping method according to claim 2, characterized in that: Step S2 includes: Generate a weight vector based on the difference of the brightness component itself by stacking multiple adaptive brightness modulation modules, and determine whether to compress the brightness component or enhance the brightness component according to the weight vector, so as to realize adaptive brightness mapping and obtain the first brightness feature map; The adaptive brightness modulation module is formed by combining a gated feedforward network and a channel attention modulation module.

5. The tone mapping method according to claim 4, characterized in that: The working principle of the adaptive brightness adjustment module is expressed as follows: X N =LN(X) X M =CAM(X N )+X Y=GDFN(LN(X M ))+X M Among them, X is the input brightness image, H and W represent the height and width of the input image, C represents the number of feature channels; X N and X M is the intermediate feature; Y is the output of the adaptive brightness modulation module, representing the modified first brightness feature map; LN(·) represents the layer normalization operation; GDFN(·) represents the gated feedforward network; CAM(·) represents the channel attention modulation module, which is composed of a convolutional layer and a GELU activation function.

6. The tone mapping method according to claim 1, characterized in that: Step S3 includes: Extracting a global self-attention map and local features of the first brightness feature map through a multi-head convolution transposed self-attention module and a convolutional neural network path; A modulation matrix is ​​generated based on the global self-attention map, and the local features are modulated using the modulation matrix to obtain the second brightness feature map.

7. The tone mapping method according to claim 1, characterized in that: Step S4 includes: fusing the first brightness channel and the second brightness feature map by a global residual connection, and reconstructing the fused result by a reconstruction module to obtain the second brightness channel; or Step S1 includes: A high dynamic range image is acquired, and a brightness channel of the color space thereof is normalized to the maximum and minimum values ​​to obtain the first brightness channel.

8. The tone mapping method according to claim 1, characterized in that: The method further includes: Get the hue channel and saturation channel of a high dynamic range image; The second brightness channel is combined with the hue channel and the saturation channel to obtain a tone mapping output result.

9. A high dynamic range image tone mapping device based on attention mechanism, characterized in that: The device comprises a tone mapping network model; the tone mapping network model is constructed by using the tone mapping method according to any one of claims 1 to 8.

10. The tone mapping device according to claim 9, characterized in that: The device also includes a training module for training the tone mapping network model according to training data and an L1 loss function; wherein the training data is determined in the following manner: A preset number of tone mapping algorithms are selected to generate tone mapped LDR images, and the generated LDR images are evaluated and ranked by a tone mapping quality index to determine a target LDR image, and the target LDR image is used as training data.

Images