A method, apparatus, device, and storage medium for ultra-high-definition dynamic multi-exposure image fusion.

Abstract

This application relates to the technical field of image processing, and more particularly to an ultra-high-definition dynamic multi-exposure image fusion method, apparatus, device, and storage medium. The method includes: extracting dynamic scene samples from the video to be processed, and constructing a dynamic multi-exposure image dataset based on the dynamic scene samples; stitching the images along the channel dimensions of three exposure levels to obtain a stitched image; performing downsampling on the stitched image to obtain low-level features; performing dimensional attention enhancement based on the low-level features to obtain enhanced local features, and performing dimensional rolling transformation based on the enhanced local features to obtain global features; performing upsampling on the global features to obtain a high-resolution feature map; and combining the high-resolution feature map with a medium-exposure image to obtain a fused image result. This application has the beneficial effect of efficiently processing multiple ultra-high-definition images with different exposure levels on a single consumer-grade GPU, ensuring a clear and natural fusion effect in complex dynamic scenes.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an ultra-high-definition dynamic multi-exposure image fusion method, apparatus, device and storage medium. Background Technology

[0002] With the advent of sophisticated imaging sensors and displays, Ultra High Definition (UHD) imaging technology has made significant progress, leading to a rapid increase in the adoption of UHD images. Ultra High Definition dynamic multi-exposure image fusion is an advanced image processing technique that combines multiple images of the same dynamic scene captured at different exposure times to generate a single high-quality image. The combination of these two technologies leverages their respective advantages to provide users with a more detailed and realistic visual experience.

[0003] However, most existing ultra-high-definition dynamic multi-exposure image fusion methods are still limited to processing low-resolution images, failing to fully leverage the advantages of UHD imaging technology. These methods are inefficient when processing multiple ultra-high-definition images with different exposures on computationally limited devices, and the quality of the fusion result deteriorates significantly when the exposure differences of the source images are large or the number of images is small. Especially in dynamic scenes with camera or object movement, traditional methods are prone to exhibiting noticeable ghosting artifacts.

[0004] Furthermore, existing deep learning-based multi-exposure image fusion methods consistently focus on low-resolution images. When faced with ultra-high-resolution images, these methods frequently encounter issues such as memory overflow, making full-resolution inference impossible on a single consumer-grade GPU. Even though a few methods can achieve real-time processing of ultra-high-resolution images on consumer-grade GPUs (Graphics Processing Units), the ultra-high-resolution images generated by these algorithms typically suffer from insufficient detail preservation or artifacts, resulting in low image quality. These limitations make it difficult for current deep learning methods to balance efficiency and image quality in multi-exposure fusion scenarios for ultra-high-resolution images.

[0005] In summary, in order to adapt to the development of high-resolution imaging technology, overcome the limitations of existing methods, and make full use of the performance advantages of consumer-grade GPUs, how to efficiently process multiple ultra-high-definition images with different exposures on a single consumer-grade GPU and ensure clear and natural fusion effects in complex dynamic scenes is a technical problem that urgently needs to be solved. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, this application provides an ultra-high-definition dynamic multi-exposure image fusion method, apparatus, device, and storage medium, which enables efficient processing of multiple ultra-high-definition images with different exposures on a single consumer-grade GPU, ensuring a clear and natural fusion effect in complex dynamic scenes.

[0007] The technical solution adopted by this application to solve its technical problem is:

[0008] In a first aspect, this application provides an ultra-high-definition dynamic multi-exposure image fusion method, the method comprising:

[0009] Extract dynamic scene samples from the video to be processed, and construct a dynamic multi-exposure image dataset based on the dynamic scene samples. The dynamic multi-exposure image dataset consists of low-exposure images, medium-exposure images, and high-exposure images under the same scene.

[0010] The images are stitched together along the image channel dimensions of the low-exposure image, medium-exposure image, and high-exposure image to obtain a stitched image.

[0011] The stitched image is downsampled to obtain low-level features;

[0012] Dimensional attention enhancement is performed based on the low-level features to obtain enhanced local features, and dimensional rolling transformation is performed based on the enhanced local features to obtain global features;

[0013] Upsampling is performed on the global features to obtain a high-resolution feature map;

[0014] The fused image result is obtained by combining the high-resolution feature map with the medium-exposure image.

[0015] Optionally, the step of downsampling the stitched image to obtain low-level features includes:

[0016] The stitched image is mapped to the feature space using a preset downsampling method to obtain the low-level features. The downsampling includes at least one subpixel convolutional layer and at least one 3×3 convolutional layer;

[0017] in, Indicates channel dimension, Indicates the width dimension. Indicates the height dimension.

[0018] Optionally, the step of performing dimensionality attention enhancement based on the low-level features to obtain enhanced local features includes:

[0019] The low-level features are divided into blocks along the channel dimension, width dimension, and height dimension, resulting in dimension blocks corresponding to each dimension.

[0020] A preset local feature extractor is used to extract deep features of each dimension block, and the extracted deep features are processed based on the Sigmoid function to generate attention weights corresponding to each dimension in the low-level features.

[0021] Based on the attention weights corresponding to each dimension, attention enhancement is performed on the low-level features to obtain the enhanced local features.

[0022] Optionally, the expression for the local feature extractor is:

[0023] ;

[0024] ;

[0025] ;

[0026] in, These are input features. It is a pooling feature. It is the channel weight. This represents the output features of the local feature extractor. Indicates global average pooling. For the Sigmoid function, For ReLU function, This represents a 1×1 convolution. This represents element-wise multiplication.

[0027] Optionally, the step of extracting deep features of each dimension block using a preset local feature extractor, and processing each extracted deep feature based on the Sigmoid function to generate the attention weights corresponding to each dimension in the low-level features includes:

[0028]

[0029] in, Representing a dimension block The final local features, This refers to the attention cache. This represents a local feature extractor. This indicates a read operation in the attention cache. This indicates a write operation in the attention cache.

[0030] Optionally, the process of obtaining the corresponding quantization tensor, and the expression for the quantization compression process, is as follows:

[0031]

[0032]

[0033]

[0034] in, It is a scaling factor. It's midnight. It is a quantization tensor. It is a tensor The maximum value in, It is a tensor The minimum value in, It is the maximum value of the quantization range. It is the minimum value of the quantization range. Used to retrieve the maximum value of the comparison. Used to find the minimum value of the comparison values Used to convert floating-point numbers to the nearest integer.

[0035] Optionally, the method includes:

[0036] The quantized tensor is decompressed using a pre-built decoder, and the decompression process is expressed as follows:

[0037]

[0038] in, This represents the inverse quantization tensor.

[0039] Optionally, the step of performing dimensionality attention enhancement based on the low-level features to obtain enhanced local features can be written as follows:

[0040]

[0041] in, This indicates a quantization compression operation. This indicates a decompression operation.

[0042] Optionally, the step of performing a dimensionality rolling transformation based on the enhanced local features to obtain global features includes:

[0043] Interpolation is used to adjust the resolution of the enhanced local features, and a preset dimensionality transformation step is performed on the adjusted resolution to obtain intermediate features. The intermediate features obtained The process expression is:

[0044]

[0045] ;

[0046] in, and It is an intermediate result. This indicates an interpolation operation. This represents a 1×1 convolution. It is the GELU function. This indicates a dimensional transformation operation; ×3 indicates that the operation is performed three times sequentially. It enhances local features.

[0047] Optionally, the resolution of the enhanced local features is adjusted using interpolation, and a preset dimensionality transformation step is performed on the adjusted resolution to obtain intermediate features. Following the steps, the method includes:

[0048] Interpolation is used to transform intermediate features that have undergone dimensionality transformation. The resolution is restored to the original resolution before adjustment, based on the restored intermediate features. The attention map is estimated;

[0049] Adjust and enhance local features based on the attention map. The global features are obtained, and the process of obtaining the global features is expressed as follows:

[0050]

[0051] in, It is the output feature. This represents element-wise multiplication.

[0052] Secondly, this application provides an ultra-high-definition dynamic multi-exposure image fusion device, comprising:

[0053] The multi-exposure image set construction module is used to extract dynamic scene samples from the video to be processed and construct a dynamic multi-exposure image dataset based on the dynamic scene samples. The dynamic multi-exposure image dataset consists of low-exposure images, medium-exposure images and high-exposure images under the same scene.

[0054] The multi-exposure image stitching module is used to stitch together the low-exposure image, medium-exposure image and high-exposure image along the image channel dimensions to obtain a stitched image;

[0055] The downsampling module is used to perform downsampling on the stitched image to obtain low-level features;

[0056] The feature integration processing module is used to perform dimensionality attention enhancement based on the low-level features to obtain enhanced local features, and to perform dimensionality rolling transformation based on the enhanced local features to obtain global features;

[0057] The upsampling module is used to perform upsampling on the global features to obtain a high-resolution feature map;

[0058] The image fusion output module is used to combine the high-resolution feature map with the medium-exposure image to obtain the fused image result.

[0059] Thirdly, this application provides an electronic device, including: at least one processor and at least one memory, wherein computer-readable instructions are stored on the memory;

[0060] The computer-readable instructions are executed by one or more of the processors, causing the electronic device to implement the above-described ultra-high-definition dynamic multi-exposure image fusion method.

[0061] Fourthly, this application provides a storage medium in which the computer-readable instructions are executed by a processor to implement the above-described ultra-high-definition dynamic multi-exposure image fusion method.

[0062] The beneficial effects of this application are as follows: First, a dynamic multi-exposure dataset that can be used for model training and testing is constructed. After mapping the dataset to the feature space, the image features are processed using a block-caching-quantization machine learning paradigm. This block-caching-quantization method includes: first, slicing the input stitched image into blocks to reduce the pressure on the model to process the data stream; second, using attention caching technology to process the infinitely large data stream; and third, using quantization compression to reduce the pressure on the device's cache. After feature processing, a high-resolution image is output and fused with gold-standard samples, thereby achieving efficient processing of multiple ultra-high-definition images with different exposures on a single consumer-grade GPU, ensuring a clear and natural fusion effect in complex dynamic scenes. Attached Figure Description

[0063] Figure 1 This is a schematic diagram of the framework for implementing the ultra-high-definition dynamic multi-exposure image fusion method provided in the embodiments of this application;

[0064] Figure 2 This is a flowchart illustrating the ultra-high-definition dynamic multi-exposure image fusion method provided in the embodiments of this application;

[0065] Figure 3 This is a list of quantitative comparison results of the ultra-high-definition dynamic multi-exposure dataset provided in the embodiments of this application;

[0066] Figure 4 This is a table illustrating the quantitative comparison results of the comparative experiments in this application on the Kalantari dataset and the quantitative comparison results on the Mobile-HDR dataset;

[0067] Figure 5This is a schematic diagram of the ablation experiment results of the key components DAEM and DRTM provided in the embodiments of this application;

[0068] Figure 6 This is a schematic diagram of the ablation experiment results for the attention cache, a key component provided in this application;

[0069] Figure 7 This is a schematic diagram of the virtual structure of the ultra-high-definition dynamic multi-exposure image fusion device provided in this application;

[0070] Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0071] The present application will be further described below with reference to the accompanying drawings and embodiments.

[0072] The following will clearly and completely describe the concept, specific structure, and resulting technical effects of this application in conjunction with embodiments and accompanying drawings, so as to fully understand the purpose, features, and effects of this application. Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. Other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are all within the scope of protection of this application. Furthermore, all connections / linkages involved in the patent do not simply refer to direct contact between components, but rather to the ability to form a better connection structure by adding or reducing connecting accessories according to specific implementation conditions. The various technical features in this application can be combined interactively without contradicting each other.

[0073] Reference Figure 1 , Figure 1 This is a schematic diagram of the framework for implementing the ultra-high-definition dynamic multi-exposure image fusion method provided in the embodiments of this application. The following is a combination of... Figure 1 A preliminary overview of the multi-exposure image fusion method used in this application is provided:

[0074] First, an ultra-high-resolution multi-exposure image dataset is constructed. Then, three images with different exposure levels are given. The input image is stitched together along the channel dimension. Then, a downsampling unit maps the stitched image onto the feature space to obtain low-level features. Where C, W, and H represent the channel, width, and height, respectively. Then, multiple Feature Integration Blocks (FIBs) are sequentially used from... Generate more refined deep features Feature fusion is performed. Each feature fusion block consists of a Dimensional Attention Enhancement Module (DAEM) and a Dimensional Rolling Transformation Module (DRTM). Finally, the final features... Input to the upsampler, then with Multiply to obtain a fused image .

[0075] The feature integration module includes a dimensionality attention enhancement module and a dimensionality rolling transformation module. The dimensionality attention enhancement module focuses on extracting local features of the image, meticulously capturing local details and texture information. Specifically, it includes a recurrent scanner, attention buffer, and quantization compression. The dimensionality rolling transformation module, on the other hand, captures global features of the image through dimensionality rolling transformation, effectively grasping the overall structure and long-range dependencies of the image. The two modules work together, enabling the network to comprehensively process both local and global information of the image while maintaining computational efficiency, thus supporting complex image processing tasks.

[0076] Reference Figure 2 , Figure 2 This is a flowchart illustrating the ultra-high-definition dynamic multi-exposure image fusion method provided in this application embodiment. The method includes the following steps, and each step is described in detail below:

[0077] In step S1, dynamic scene samples of the video to be processed are extracted, and a dynamic multi-exposure image dataset is constructed based on the dynamic scene samples. The dynamic multi-exposure image dataset consists of low-exposure images, medium-exposure images, and high-exposure images under the same scene.

[0078] Specifically, since the image resolution of existing dynamic multi-exposure datasets does not reach 4K (3840×2160), they cannot meet the training and testing requirements of the model. Therefore, it is necessary to first construct an ultra-high-definition dynamic multi-exposure image dataset. Specifically, a 4K (3840×2160) video is recorded using a mobile phone or other photography device to obtain the video to be processed. The video includes, but is not limited to, moving people, vehicles, and other dynamic elements. Three frames are extracted from the video to obtain dynamic scene samples. Three samples correspond to one scene. The three samples are then combined to obtain low, medium, and high exposure levels to obtain the dynamic multi-exposure image dataset.

[0079] In step S2, the images are stitched together along the image channel dimensions of the low-exposure image, medium-exposure image, and high-exposure image to obtain a stitched image.

[0080] Specifically, the images are stitched together along the channel dimension of three exposure images in the dynamic multi-exposure image dataset. This process can be accomplished using a model to obtain the stitched image.

[0081] In step S3, the stitched image is downsampled to obtain low-level features.

[0082] Specifically, the stitched image is mapped to a feature space using a downsampler to obtain low-level features. The step of downsampling the stitched image to obtain low-level features includes:

[0083] The stitched image is mapped to the feature space using a preset downsampling method to obtain the low-level features. The downsampling includes at least one subpixel convolutional layer and at least one 3×3 convolutional layer;

[0084] in, Indicates channel dimension, Indicates the width dimension. Indicates the height dimension.

[0085] Specifically, refer to Figure 1 The extracted low-level features are easy to process by dividing the data into blocks based on channel, width, and height dimensions, and then output to the slice loop scanner.

[0086] In step S4, dimensional attention enhancement is performed based on the low-level features to obtain enhanced local features, and dimensional rolling transformation is performed based on the enhanced local features to obtain global features.

[0087] Specifically, low-level features are input into the dimensionality attention enhancement module in a pre-defined feature ensemble block for local feature enhancement. The enhanced local features are then input into the dimensionality rolling transformation module in the feature ensemble block for global feature extraction. The specific steps for obtaining enhanced local features are explained below:

[0088] The step of performing dimensionality attention enhancement based on the low-level features to obtain enhanced local features includes:

[0089] The low-level features are divided into blocks along the channel dimension, width dimension, and height dimension, resulting in dimension blocks corresponding to each dimension.

[0090] A preset local feature extractor is used to extract deep features of each dimension block, and the extracted deep features are processed based on the Sigmoid function to generate attention weights corresponding to each dimension in the low-level features.

[0091] Based on the attention weights corresponding to each dimension, attention enhancement is performed on the low-level features to obtain the enhanced local features.

[0092] Specifically, the input data is divided into blocks along the channel, width, and height (i.e., the low-level features obtained in the above steps are divided into blocks). Then, a recurrent scanner is used to capture local features in each dimension block. The recurrent scanner contains a Local Feature Extractor (LFE). The LFE feeds the input blocks into a global average pooling layer and a series of convolutional layers to extract deep features. The Sigmoid function is then used to process these deep features to generate attention weights for the input low-level features.

[0093] More specifically, after obtaining the attention weights of the low-level features, element-wise multiplication is used to apply the weights of the three dimensions to the corresponding low-level features of the input, thereby dynamically adjusting the importance of the channels and improving the expressive power of specific features. This process is repeated until all the dimensional block features of the three dimensions have been extracted by the local feature extractor. The outputs of the three dimensions are then added together to obtain the final output of the dimensional attention enhancement module, which is the enhanced local feature.

[0094] The expression for the local feature extractor is as follows:

[0095] ;

[0096] ;

[0097] ;

[0098] in, These are input features. It is a pooling feature. It is the channel weight. This represents the output features of the local feature extractor. Indicates global average pooling. For the Sigmoid function, For ReLU function, This represents a 1×1 convolution. This represents element-wise multiplication.

[0099] Furthermore, this application employs attention caching technology to accelerate inference across the entire network, as detailed below:

[0100] The steps of extracting deep features from each dimension block using a preset local feature extractor and processing each extracted deep feature based on the Sigmoid function to generate the attention weights corresponding to each dimension in the low-level features include:

[0101]

[0102] in, Representing a dimension block The final local features, This refers to the attention cache. This represents a local feature extractor. This indicates a read operation in the attention cache. This indicates a write operation in the attention cache.

[0103] Specifically, the local feature extractor processes deep features through the Sigmoid function to generate attention weights for the input features. The attention cache can store the local features extracted by the local feature extractor, thereby avoiding redundant calculations and accelerating the inference process.

[0104] More specifically, during training, the intermediate results (feature maps) of the network are obtained through convolution operations. However, during inference, these convolution results are stored as a cache using the method described above, allowing for direct retrieval without recalculation, thus accelerating the inference process of the entire network. It is worth noting that the write operation writes the k value and its corresponding value to the attention cache.

[0105] Furthermore, this application employs quantization compression technology to reduce the memory burden on the device, which is explained in detail below:

[0106] The process of obtaining the corresponding quantization tensor, and the expression for the quantization compression process, is as follows:

[0107]

[0108]

[0109]

[0110] in, It is a scaling factor. It's midnight. It is a quantization tensor. It is a tensor The maximum value in, It is a tensor The minimum value in, It is the maximum value of the quantization range. It is the minimum value of the quantization range. Used to retrieve the maximum value of the comparison. Used to find the minimum value of the comparison values Used to convert floating-point numbers to the nearest integer.

[0111] Specifically, as model training and inference proceed, the memory consumption of the attention cache increases rapidly, significantly exacerbating the burden on device memory. To alleviate this problem, this application employs quantization compression to reduce storage memory. In quantization compression, an encoder is constructed to compress floating-point tensors into quantized tensors.

[0112] Correspondingly, a decoder is used to decompress the quantized tensor. The decompression process is expressed as follows:

[0113]

[0114] in, This represents the inverse quantization tensor.

[0115] The step of performing dimensionality attention enhancement based on the low-level features to obtain enhanced local features can be written as follows:

[0116]

[0117] in, This indicates a quantization compression operation. This indicates a decompression operation.

[0118] Furthermore, after the features have completed the steps described above to obtain enhanced local features, the obtained local features are input into the dimension rolling transformation module in the feature integration block for global feature extraction, which will be explained in detail below:

[0119] Interpolation is used to adjust the resolution of the enhanced local features, and a preset dimensionality transformation step is performed on the adjusted resolution to obtain intermediate features. The intermediate features obtained The process expression is:

[0120]

[0121] ;

[0122] in, and It is an intermediate result. This indicates an interpolation operation. This represents a 1×1 convolution. It is the GELU function. This indicates a dimensional transformation operation; ×3 indicates that the operation is performed three times in sequence.

[0123] Specifically, after local feature extraction, global feature extraction is also required. In this application, the dimension rolling transformation module addresses the problem of global feature loss during cyclic scanning. This application encodes the feature maps of the image from the perspectives of width, height, and channels, and associates and fuses the encoded information in each feature map. Notably, the dimension transformation operation does not introduce additional parameters, which means that by arranging the feature maps from different views, the dimension rolling transformation module achieves efficient modeling of long-distance dependencies.

[0124] More specifically, the input features are first adjusted using interpolation (i.e., local features are enhanced), and then the adjusted input features are subjected to dimensionality transformation. Subsequently, interpolation is used to transform the intermediate feature output. Adjust to its original resolution to estimate the attention map, and adaptively modulate the input via element-wise multiplication based on the estimated attention. The output features (global features) are obtained. Specifically, the global features are obtained, and the process of obtaining the global features is expressed as follows:

[0125]

[0126] in, It is the output feature. This represents element-wise multiplication.

[0127] In general, the processing flow of input features through the feature ensemble block can be described by the following expression:

[0128]

[0129] in, This represents the output of the dimensional attention enhancement module. This indicates the final output.

[0130] In step S5, the global features are upsampled to obtain a high-resolution feature map.

[0131] Specifically, in this embodiment, an upsampler consisting of a sub-pixel convolutional layer and a 3×3 convolutional layer is used to perform upsampling operations on global features. After the low-resolution feature map is upsampled, the spatial resolution of the features will increase, thereby obtaining fine features with higher resolution. These features typically retain rich information from low-level features to high-level features, which can be used for subsequent image reconstruction operations.

[0132] In step S6, the high-resolution feature map and the medium-exposure image are combined to obtain the fused image result.

[0133] Specifically, the final features extracted by the feature ensemble block are processed by an upsampler to generate a high-resolution feature map, which is restored to 4K resolution, consistent with the resolution of the three input exposure images. In this case, based on the constructed dataset and existing non-ultra-high-definition dynamic multi-exposure datasets, it is known that the positions of dynamic elements (such as moving vehicles) in the three input high, medium, and low exposure images may differ in each image. In this embodiment, the original image samples (i.e., the original image samples used to synthesize the medium exposure image) are used as the gold standard (ground truth) before the exposure synthesis. Therefore, pixels containing important dynamic elements in the input medium exposure image are retained, ensuring the accurate representation of these key parts in the final fusion result. The final features extracted by the feature ensemble block contain high-level image feature information, such as texture, edges, and semantic information. These features are multiplied with the medium exposure image, and the pixel values ​​are adaptively adjusted to improve image quality, ultimately obtaining the fused image result.

[0134] Furthermore, the above steps describe the process of multi-exposure image fusion, and the entire process relies on the model obtained through training. The model optimization used in this application embodiment is based on a specific loss function, which is used to guide the training process and ensure that the fused image achieves the best results in terms of detail preservation, brightness balance and visual quality.

[0135] Specifically, this method uses the L1 loss function to calculate the L1 loss value between the fusion result and the gold standard (ground truth) during model training, which can be written as:

[0136]

[0137] in, This refers to the gold standard (ground truth). This represents the output fused image.

[0138] Furthermore, the trained network model is tested to demonstrate the application of this method in the fusion of ultra-high-definition dynamic multi-exposure images.

[0139] In summary, this application first constructs a dynamic multi-exposure dataset for model training and testing. After mapping this dataset to a feature space, it processes the image features using a block-caching-quantization machine learning paradigm. This block-caching-quantization method includes: first, slicing the input stitched image into blocks to reduce the pressure on the model processing the data stream; second, using attention caching technology to process the infinitely large data stream; and third, using quantization compression to reduce the pressure on the device's cache. After feature processing, a high-resolution image is output and fused with a gold-standard sample, thereby achieving efficient processing of multiple ultra-high-definition images with different exposures on a single consumer-grade GPU, ensuring a clear and natural fusion effect in complex dynamic scenes.

[0140] The beneficial effects provided by this application have been verified through the following comparative experiments. Specifically, compared with existing dynamic multi-exposure image fusion methods (such as GDC, AHDRNet, FSHDR, HDR-Transformer, and SCTNet), this method has been experimentally verified on our constructed ultra-high-definition dynamic multi-exposure dataset and publicly available non-ultra-high-definition datasets (such as the Kalantari dataset and the Mobile-HDR dataset).

[0141] Specifically, this application conducts experiments using PyTorch on a single NVIDIA GeForce RTX 4090 GPU. To optimize the network, this application employs the AdamW optimizer with a learning rate of 2×10⁻⁶. The network was trained for 1200 epochs with a batch size of 4, 8 feature ensemble modules (FIBs), and 48 feature channels. It's important to note that many existing methods cannot directly perform full-resolution inference on ultra-high-definition (UHD) images. For these methods, the comparative experiments provided in this application first use bilinear interpolation to downsample the input UHD image to the maximum resolution these algorithms can process on a single GPU (e.g., [missing information]). Figure 3 As shown, Figure 3 This is a list of quantitative comparison results of the ultra-high-definition dynamic multi-exposure dataset provided in the embodiments of this application. After inference is completed, the output results are upsampled back to 4K resolution using bilinear interpolation.

[0142] Furthermore, this application employs three image quality evaluation metrics, such as Figure 3 As shown, the metrics include PSNR, SSIM, and LPIPS. Among them, PSNR (Peak Signal-to-Noise Ratio) is a commonly used metric for measuring image quality. It is mainly used to evaluate the difference between the processed image and the original image. The higher the value, the better the image quality.

[0143] The formula for calculating PSNR is as follows:

[0144] ;

[0145] ;

[0146] Where MSE represents mean squared error, which measures the difference between two images at the pixel level; I(i,j) and K(i,j) represent the pixel values ​​at position (i,j) of the generated fused image and the corresponding gold standard image (ground truth), respectively; m and n represent the width and height of the image, respectively; and MAX represents the maximum value of the image pixels.

[0147] SSIM (Structural Similarity Index Measure) is an indicator used to evaluate the quality of two images. It mainly measures the similarity of images by comparing their brightness, contrast, and structural information. The closer the value is to 1, the better the image quality.

[0148] The formula for calculating SSIM is as follows:

[0149]

[0150]

[0151]

[0152] in, yes average value, Is with Corresponding gold standard The average value, It represents an image and covariance, yes variance yes The variance; L is the dynamic range of pixel values. and This represents the preset hyperparameters, here =0.01, =0.03, and This represents the smoothing parameter.

[0153] The LPIPS (Learned Perceptual Image Patch Similarity) metric is a measure used to evaluate the perceptual similarity between two images. It is based on features extracted by deep convolutional neural networks, rather than simple pixel differences; a lower LPIPS value indicates better image quality.

[0154] The formula for calculating LPIPS is as follows:

[0155] ;

[0156] Where x represents the fused image output by the model, and x0 represents the gold standard image corresponding to x. and This indicates that the first [unclear] extracted through a pre-trained VGG deep neural network... Layer feature map. Is the image x at the th Features of the layer Is the image x0 at the th... Characteristics of the layer. This represents the Euclidean distance (L2 norm) between feature maps. These are feature maps. and The difference metric between two images is used to measure the difference between them at the 1st minute. Similarity in layer features. This represents the weighting coefficients for the feature maps of each layer. Different network layers have different impacts on perceptual similarity, and the weights... This is typically reflected through training and learning. This indicates all layers The results are summed to comprehensively consider the differences in characteristics at different levels.

[0157] Furthermore, referring to Figure 3 The comparative experiments provided in this application also record the highest resolution that each algorithm can process on a single RTX4090 GPU, called the maximum resolution, and the time required to generate the fused image, called the inference time.

[0158] Combination Figure 3 The table shown shows that Figure 3Quantitative comparative results of various methods evaluated on the ultra-high-definition dynamic multi-exposure fusion dataset constructed in this application are presented. The best and second-best values ​​are highlighted in bold and underlined text, respectively. It can be seen that compared to the second-best method, FSHDR, our method improves PSNR and SSIM by approximately 46% and 48%, respectively. Furthermore, the method presented in this application also shows significant improvements on LPIPS, outperforming FSHDR by approximately 88%. Although the method presented in this application is not the fastest in inference time, it still achieves real-time inference at frame rates exceeding 40fps.

[0159] Furthermore, referring to Figure 4 , Figure 4 The table shows the quantitative comparison results of the comparative experiments of this application on the Kalantari dataset and the Mobile-HDR dataset. It can be seen that the method provided in this application performs comparable to the state-of-the-art methods on non-ultra-high-definition datasets. Therefore, the method provided in this application can not only maintain high-quality results in multi-exposure image fusion tasks at different resolutions, but also demonstrate highly competitive performance.

[0160] Furthermore, to demonstrate the effectiveness of each part of the method provided in this application, three corresponding ablation experiments are also disclosed, with each key component evaluated in a completely fair environment. For these experiments, the same architecture and hyperparameters were used, with only one component changed for each ablation. The evaluation of these ablation experiments was conducted on the ultra-high-definition dynamic multi-exposure dataset constructed in this application; please refer to [reference needed] for details. Figure 5 , Figure 5 This is a schematic diagram of the ablation experiment results of the key components DAEM and DRTM provided in the embodiments of this application.

[0161] Combination Figure 5 As can be seen, the model's performance experienced the most severe decline after removing the DAEM module. Results from (b) to (h) demonstrate that all three dimensions of the slice contain crucial information, especially the width and height dimensions, which are essential for the final image fusion.

[0162] Further, please refer to Figure 6 , Figure 6 This is a schematic diagram of the ablation experiment results for the attention cache, a key component provided in this application, combined with... Figure 6 As can be seen, although the performance is slightly reduced due to quantization compression in the attention cache, the inference time is significantly improved. In the embodiment provided in this application, the processing time is reduced from 0.1475 seconds to 0.0441 seconds, a reduction of 70%, which can be considered as achieving real-time inference.

[0163] Reference Figure 7, Figure 7 This is a virtual structural diagram of the ultra-high-definition dynamic multi-exposure image fusion device provided in this application. A second aspect of this application provides an ultra-high-definition dynamic multi-exposure image fusion device, comprising:

[0164] The multi-exposure image set construction module 100 is used to extract dynamic scene samples from the video to be processed and construct a dynamic multi-exposure image dataset based on the dynamic scene samples. The dynamic multi-exposure image dataset consists of low-exposure images, medium-exposure images and high-exposure images in the same scene.

[0165] The multi-exposure image stitching module 200 is used to stitch together the low-exposure image, medium-exposure image and high-exposure image along the image channel dimensions to obtain a stitched image;

[0166] The downsampling module 300 is used to perform downsampling on the stitched image to obtain low-level features;

[0167] The feature integration processing module 400 is used to perform dimensionality attention enhancement based on the low-level features to obtain enhanced local features, and to perform dimensionality rolling transformation based on the enhanced local features to obtain global features;

[0168] The upsampling module 500 is used to perform upsampling on the global features to obtain a high-resolution feature map;

[0169] The image fusion output module 600 is used to combine the high-resolution feature map with the medium-exposure image to obtain the fused image result.

[0170] The ultra-high-definition dynamic multi-exposure image fusion device described in this application embodiment can execute the ultra-high-definition dynamic multi-exposure image fusion method provided in the above embodiments. The ultra-high-definition dynamic multi-exposure image fusion device has the corresponding functional steps and beneficial effects of the ultra-high-definition dynamic multi-exposure image fusion method described in the above embodiments. For details, please refer to the embodiments of the ultra-high-definition dynamic multi-exposure image fusion method described above. The embodiments of this application will not be repeated here.

[0171] This application also provides an electronic device, please refer to... Figure 8 , Figure 8This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device may include a processor and a memory, which can be connected via a bus or other means. The processor may be a Central Processing Unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips. The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the ultra-high-definition dynamic multi-exposure image fusion method in the embodiments of this application. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby realizing the ultra-high-definition dynamic multi-exposure image fusion method in the above method embodiments.

[0172] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. The one or more modules are stored in the memory and, when executed by the processor, perform the ultra-high-definition dynamic multi-exposure image fusion method as described in the above method embodiments. Specific details of the above electronic device can be understood by referring to the corresponding descriptions and effects in the above method embodiments, and will not be repeated here. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it may include the processes of the embodiments of the above methods. The storage medium may be a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD), etc.; the storage medium may also include a combination of the above types of memory.

[0173] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0174] Similarly, it should be understood that, in order to streamline this disclosure and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of this application, various features of this application are sometimes grouped together in a single embodiment, figure, or description thereof. However, this approach to disclosure should not be construed as reflecting an intention that the claimed application requires more features than expressly recited in each claim. Rather, as reflected in the claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0175] It should be noted that the above embodiments are illustrative of this application and not restrictive of this application, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims.

Claims

1. A method for ultra-high-definition dynamic multi-exposure image fusion, characterized in that, The method includes: Extract dynamic scene samples from the video to be processed, and construct a dynamic multi-exposure image dataset based on the dynamic scene samples. The dynamic multi-exposure image dataset consists of low-exposure images, medium-exposure images, and high-exposure images under the same scene. The images are stitched together along the image channel dimensions of the low-exposure image, medium-exposure image, and high-exposure image to obtain a stitched image. The stitched image is downsampled to obtain low-level features; Dimensional attention enhancement is performed based on the low-level features to obtain enhanced local features, and dimensional rolling transformation is performed based on the enhanced local features to obtain global features; the step of performing dimensional attention enhancement based on the low-level features to obtain enhanced local features includes: The low-level features are divided into blocks along the channel dimension, width dimension, and height dimension, resulting in dimension blocks corresponding to each dimension. A preset local feature extractor is used to extract deep features of each dimension block, and the extracted deep features are processed based on the Sigmoid function to generate attention weights corresponding to each dimension in the low-level features. Based on the attention weights corresponding to each dimension, attention enhancement is performed on the low-level features to obtain the enhanced local features; The step of performing a dimensionality rolling transformation based on the enhanced local features to obtain global features includes: Interpolation is used to adjust the resolution of the enhanced local features, and a preset dimensionality transformation step is performed on the adjusted resolution to obtain intermediate features. ; The resolution of the enhanced local features is adjusted using interpolation, and a preset dimensionality transformation step is performed on the adjusted resolution to obtain intermediate features. Following the steps, the method includes: Interpolation is used to transform intermediate features that have undergone dimensionality transformation. The resolution is restored to the original resolution before adjustment, based on the restored intermediate features. The attention map is estimated; Adjust and enhance local features based on the attention map. The global features are obtained. Upsampling is performed on the global features to obtain a high-resolution feature map; The fused image result is obtained by combining the high-resolution feature map with the medium-exposure image.

2. The ultra-high-definition dynamic multi-exposure image fusion method according to claim 1, characterized in that, The step of downsampling the stitched image to obtain low-level features includes: The stitched image is mapped to the feature space using a preset downsampling method to obtain the low-level features. The downsampling includes at least one subpixel convolutional layer and at least one 3×3 convolutional layer; in, Indicates channel dimension, Indicates the width dimension. Indicates the height dimension.

3. The ultra-high-definition dynamic multi-exposure image fusion method according to claim 2, characterized in that, The expression for the local feature extractor is: ； ； ； in, These are input features. It is a pooling feature. It is the channel weight. This represents the output features of the local feature extractor. Indicates global average pooling. For the Sigmoid function, For ReLU function, This represents a 1×1 convolution. This represents element-wise multiplication.

4. The ultra-high-definition dynamic multi-exposure image fusion method according to claim 2, characterized in that, The steps of extracting deep features from each dimension block using a preset local feature extractor and processing each extracted deep feature based on the Sigmoid function to generate the attention weights corresponding to each dimension in the low-level features include: in, Representing a dimension block The final local features, This refers to the attention cache. This represents a local feature extractor. This indicates a read operation in the attention cache.

5. The ultra-high-definition dynamic multi-exposure image fusion method according to claim 4, characterized in that, The method further includes: The floating-point tensor generated during the attention caching process is quantized and compressed to obtain the corresponding quantized tensor. The expression for the quantization and compression process is as follows: in, It is a scaling factor. It's midnight. It is a quantization tensor. It is a tensor The maximum value in, It is a tensor The minimum value in, It is the maximum value of the quantization range. It is the minimum value of the quantization range. Used to retrieve the maximum value of the comparison. Used to find the minimum value of the comparison values Used to convert floating-point numbers to the nearest integer.

6. The ultra-high-definition dynamic multi-exposure image fusion method according to claim 1, characterized in that, The intermediate features are obtained The process expression is: ； in, and It is an intermediate result. This indicates an interpolation operation. This represents a 1×1 convolution. It is the GELU function. This indicates a dimensional transformation operation; ×3 indicates that the operation is performed three times sequentially. It enhances local features.

7. The ultra-high-definition dynamic multi-exposure image fusion method according to claim 6, characterized in that, The process of obtaining global features is expressed as follows: in, It is the output feature. This represents element-wise multiplication.

8. An ultra-high-definition dynamic multi-exposure image fusion device, characterized in that, include: The multi-exposure image set construction module is used to extract dynamic scene samples from the video to be processed and construct a dynamic multi-exposure image dataset based on the dynamic scene samples. The dynamic multi-exposure image dataset consists of low-exposure images, medium-exposure images and high-exposure images under the same scene. The multi-exposure image stitching module is used to stitch together the low-exposure image, medium-exposure image and high-exposure image along the image channel dimensions to obtain a stitched image; The downsampling module is used to perform downsampling on the stitched image to obtain low-level features; The feature integration processing module is used to perform dimensionality attention enhancement based on the low-level features to obtain enhanced local features, and to perform dimensionality rolling transformation based on the enhanced local features to obtain global features; the method used to perform dimensionality attention enhancement based on the low-level features to obtain enhanced local features is as follows: The low-level features are divided into blocks along the channel dimension, width dimension, and height dimension, resulting in dimension blocks corresponding to each dimension. A preset local feature extractor is used to extract deep features of each dimension block, and the extracted deep features are processed based on the Sigmoid function to generate attention weights corresponding to each dimension in the low-level features. Based on the attention weights corresponding to each dimension, attention enhancement is performed on the low-level features to obtain the enhanced local features; The method used to obtain global features by performing dimensional rolling transformation based on the enhanced local features is as follows: Interpolation is used to adjust the resolution of the enhanced local features, and a preset dimensionality transformation step is performed on the adjusted resolution to obtain intermediate features. ; The resolution of the enhanced local features is adjusted using interpolation, and a preset dimensionality transformation step is performed on the adjusted resolution to obtain intermediate features. Following the steps, the method includes: Interpolation is used to transform intermediate features that have undergone dimensionality transformation. The resolution is restored to the original resolution before adjustment, based on the restored intermediate features. The attention map is estimated; Adjust and enhance local features based on the attention map. The global features are obtained. The upsampling module is used to perform upsampling on the global features to obtain a high-resolution feature map; The image fusion output module is used to combine the high-resolution feature map with the medium-exposure image to obtain the fused image result.

9. An electronic device, characterized in that, include: At least one processor and at least one memory, wherein computer-readable instructions are stored on the memory; The computer-readable instructions are executed by one or more of the processors, causing the electronic device to implement the ultra-high-definition dynamic multi-exposure image fusion method as described in any one of claims 1 to 7.

10. A storage medium having computer-readable instructions stored thereon, characterized in that, The computer-readable instructions are executed by a processor to implement the ultra-high-definition dynamic multi-exposure image fusion method as described in any one of claims 1 to 7.

Images