Ultrahigh-definition dynamic multi-exposure image fusion method, device and equipment and storage medium

Abstract

The invention relates to the technical field of image processing, in particular to an ultra-high-definition dynamic multi-exposure image fusion method, device and equipment and a storage medium, and the method comprises the steps: extracting a dynamic scene sample of a to-be-processed video, and constructing a dynamic multi-exposure image data set based on the dynamic scene sample; splicing along the channel dimensions of the three images with the exposure degrees to obtain a spliced image; performing down-sampling on the spliced image to obtain low-level features; performing dimension attention enhancement based on the low-level features to obtain enhanced local features, and performing dimension rolling transformation based on the enhanced local features to obtain global features; performing up-sampling on the global features to obtain a high-resolution feature map; and obtaining a fused image result by combining the high-resolution feature map and the middle exposure image. The method has the beneficial effects that a plurality of ultra-high-definition images with different exposure degrees are efficiently processed on a single consumption-level GPU, and a clear and natural fusion effect is ensured to be obtained in a complex dynamic scene.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and in particular to an ultra-high-definition dynamic multi-exposure image fusion method, device, equipment and storage medium. Background Art

[0002] With the emergence of complex imaging sensors and displays, UHD (Ultra High Definition) imaging technology has made significant progress, leading to a rapid increase in the popularity of UHD images. UHD dynamic multi-exposure image fusion is an advanced image processing technology that generates a high-quality image by fusing multiple images of the same dynamic scene taken at different exposure times. The combination of these two technologies can give full play to their respective advantages and provide users with a more delicate and realistic visual experience.

[0003] However, most existing UHD dynamic multi-exposure image fusion methods are still at the stage of processing low-resolution images, and cannot fully utilize the advantages of UHD imaging technology. These methods are inefficient when processing multiple UHD images with different exposures on devices with limited computing resources, and the quality of the fusion results will be significantly reduced when the exposure difference of the source images is large or the number of images is small. In particular, in dynamic scenes with camera or object motion, traditional methods are prone to obvious ghost patterns.

[0004] In addition, in the existing deep learning-based multi-exposure image fusion methods, they always focus on low-resolution images. When faced with ultra-high-definition images, these methods often encounter problems such as memory overflow and cannot perform full-resolution inference on a single consumer-grade GPU. Even if there are a few methods that can achieve real-time processing of ultra-high-definition images on consumer-grade GPUs (Graphics Processing Units), the ultra-high-definition images generated by these algorithms usually face problems such as insufficient detail preservation or artifacts and low image quality. These limitations make it difficult for current deep learning methods to balance efficiency and image quality in the multi-exposure fusion scenario of ultra-high-definition images.

[0005] In summary, in order to adapt to the development of high-resolution imaging technology, overcome the limitations of existing methods, and make full use of the performance advantages of consumer-grade GPUs, how to efficiently process multiple ultra-high-definition images with different exposures on a single consumer-grade GPU and ensure clear and natural fusion effects in complex dynamic scenes is a technical problem that needs to be solved urgently. Summary of the invention

[0006] In order to overcome the shortcomings of the prior art, the present application provides an ultra-high-definition dynamic multi-exposure image fusion method, device, equipment and storage medium, which can achieve efficient processing of multiple ultra-high-definition images of different exposures on a single consumer-grade GPU, ensuring the beneficial effect of obtaining a clear and natural fusion effect in complex dynamic scenes.

[0007] The technical solution adopted by this application to solve its technical problem is: In a first aspect, the present application provides an ultra-high-definition dynamic multi-exposure image fusion method, the method comprising: Extracting dynamic scene samples of the video to be processed, and constructing a dynamic multi-exposure image dataset based on the dynamic scene samples, wherein the dynamic multi-exposure image dataset includes low-exposure images, medium-exposure images, and high-exposure images of the same scene; Stitching the low-exposure image, the medium-exposure image, and the high-exposure image along an image channel dimension to obtain a stitched image; Downsampling the stitched image to obtain low-level features; Performing dimensional attention enhancement based on the low-level features to obtain enhanced local features, and performing dimensional rolling transformation based on the enhanced local features to obtain global features; Performing upsampling on the global features to obtain a high-resolution feature map; The high-resolution feature map is combined with the medium-exposure image to obtain a fused image result.

[0008] Optionally, the step of performing stitching along the image channel dimension of the low-exposure image, the medium-exposure image, and the high-exposure image to obtain a stitched image includes: The stitched image is mapped to the feature space through a preset downsampler to obtain the low-level feature , the downsampler includes at least one sub-pixel convolution layer and at least one 3×3 convolution layer; in, represents the channel dimension, represents the width dimension, Represents the height dimension.

[0009] Optionally, the step of performing dimensional attention enhancement based on the low-level features to obtain enhanced local features includes: Blocking is performed along the channel dimension, width dimension and height dimension of the low-level features, and after blocking, a dimensional block corresponding to each dimension is obtained; A preset local feature extractor is used to extract the deep features of each dimensional block, and each extracted deep feature is processed based on the Sigmoid function to generate the attention weight corresponding to each dimension in the low-level features; Attention enhancement is performed on the low-level features according to the attention weight corresponding to each dimension to obtain the enhanced local features.

[0010] Optionally, the expression of the local feature extractor is: ; ; ; in, are input features, is the pooling feature, is the channel weight, represents the output features of the local feature extractor, represents global average pooling, is the Sigmoid function, is the ReLU function, represents a 1×1 convolution, Stands for element-wise multiplication.

[0011] Optionally, the step of extracting the deep features of each dimensional block by using a preset local feature extractor, and processing each extracted deep feature based on a Sigmoid function to generate an attention weight corresponding to each dimension in the low-level features includes:

[0012] in, Represents a dimension block The final local features of represents the attention cache, represents the local feature extractor, represents a read operation in the attention buffer, Represents a write operation in the attention buffer.

[0013] Optionally, the corresponding quantized tensor is obtained, and the process expression of the quantization compression is:

[0014]

[0015]

[0016] in, is the scaling factor, It's zero o'clock. is a quantized tensor, is a tensor The maximum value in is a tensor The minimum value in is the maximum value of the quantization range, is the minimum value of the quantization range, Used to get the maximum value of the comparison value. Used to get the minimum value of the comparison value. Used to convert a floating point number to the nearest integer.

[0017] Optionally, the method comprises: A pre-built decoder is used to decompress the quantized tensor. The decompression process is expressed as:

[0018] in, Represents a dequantized tensor.

[0019] Optionally, the step of performing dimensional attention enhancement based on the low-level features to obtain enhanced local features can be written as:

[0020] in, represents the quantization compression operation, Represents a decompression operation.

[0021] Optionally, the step of performing dimensional rolling transformation based on the enhanced local features to obtain global features includes: The resolution of the enhanced local feature is adjusted by interpolation operation, and a preset dimensional transformation step is performed on the adjusted resolution to obtain an intermediate feature , the intermediate features are obtained The process expression is:

[0022] ; in, and is the intermediate result, represents the interpolation operation, represents a 1×1 convolution, is the GELU function, Represents a dimension transformation operation, and ×3 means performing the operation three times in sequence.

[0023] Optionally, the interpolation operation is used to adjust the resolution of the enhanced local feature, and a preset dimensional transformation step is performed on the adjusted resolution to obtain an intermediate feature. After the step of: The intermediate features that have undergone dimension transformation are interpolated The resolution is restored to the original resolution before adjustment, and based on the restored intermediate features Estimate the attention map; Adjust and enhance local features based on the attention map , the global feature is obtained, and the process expression of obtaining the global feature is:

[0024] in, is the output feature, Stands for element-wise multiplication.

[0025] In a second aspect, the present application provides an ultra-high-definition dynamic multi-exposure image fusion device, comprising: A multi-exposure atlas construction module is used to extract dynamic scene samples of the video to be processed, and construct a dynamic multi-exposure image dataset based on the dynamic scene samples, wherein the dynamic multi-exposure image dataset includes low-exposure images, medium-exposure images, and high-exposure images of the same scene; A multi-exposure atlas stitching module, used for stitching the low-exposure image, the medium-exposure image and the high-exposure image along the image channel dimension to obtain a stitched image; A downsampling module, used for performing downsampling on the stitched image to obtain low-level features; A feature integration processing module, used to perform dimensional attention enhancement based on the low-level features to obtain enhanced local features, and perform dimensional rolling transformation based on the enhanced local features to obtain global features; An upsampling module, used to perform upsampling on the global features to obtain a high-resolution feature map; The fused image output module is used to combine the high-resolution feature map with the medium-exposure image to obtain a fused image result.

[0026] In a third aspect, the present application provides an electronic device, comprising: at least one processor and at least one memory, wherein the memory stores computer-readable instructions; The computer-readable instructions are executed by one or more of the processors, so that the electronic device implements the above-mentioned ultra-high-definition dynamic multi-exposure image fusion method.

[0027] In a fourth aspect, the present application provides a storage medium, wherein the computer-readable instructions are executed by a processor to implement the above-mentioned ultra-high-definition dynamic multi-exposure image fusion method.

[0028] The beneficial effects of the present application are as follows: the present application first constructs a dynamic multi-exposure dataset that can be used for model training and testing, maps it to the feature space, and processes the image features through the machine learning paradigm of block-cache-quantization. The block-cache-quantization method includes: the first step is to slice and block the input spliced ​​image to reduce the pressure of the model processing data stream; the second step is to use the attention cache technology to process infinite data streams; the third step is to use quantization compression to reduce the pressure burden of the high-speed cache on the device. After the features are processed, a high-resolution image is output and fused with the gold standard sample, so as to achieve efficient processing of multiple ultra-high-definition images of different exposures on a single consumer-grade GPU, ensuring the beneficial effect of obtaining a clear and natural fusion effect in complex dynamic scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of a framework of a method for realizing ultra-high-definition dynamic multi-exposure image fusion provided in an embodiment of the present application; Figure 2 It is a flowchart of an ultra-high-definition dynamic multi-exposure image fusion method provided in an embodiment of the present application; Figure 3 It is a table diagram of quantitative comparison results of ultra-high-definition dynamic multi-exposure data sets provided in an embodiment of the present application; Figure 4 It is a table diagram of the quantitative comparison results of the comparative experiment of this application on the Kalantari dataset and the quantitative comparison results on the Mobile-HDR dataset; Figure 5 It is a schematic diagram of ablation experiment results of the key components DAEM and DRTM provided in the embodiments of the present application; Figure 6 This is a schematic diagram of the ablation experiment results of the key component attention cache provided by this application; Figure 7 It is a virtual structural schematic diagram of the ultra-high-definition dynamic multi-exposure image fusion device provided by the present application; Figure 8 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The present application is further described below in conjunction with the accompanying drawings and embodiments.

[0031] The following will clearly and completely describe the concept, specific structure and technical effects of the present application in combination with the embodiments and drawings, so as to fully understand the purpose, characteristics and effects of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of them. Based on the embodiments of the present application, other embodiments obtained by technicians in this field without creative work are all within the scope of protection of the present application. In addition, all the connection / connection relationships involved in the patent do not simply refer to the direct connection of components, but refer to the formation of a better connection structure by adding or reducing connection accessories according to the specific implementation situation. The various technical features created in this application can be combined interactively without conflicting with each other.

[0032] Reference Figure 1 , Figure 1 This is a schematic diagram of the framework of the method for realizing ultra-high-definition dynamic multi-exposure image fusion provided in the embodiment of the present application. Figure 1 A preliminary overview of the multi-exposure image fusion method used in this application: First, we construct an ultra-high-definition multi-exposure image dataset, and then give three images with different exposures. , concatenate the input images along the channel dimension. Then, the concatenated images are mapped to the feature space through a downsampler to obtain low-level features , where C, W, and H represent channel, width, and height, respectively. Then, multiple feature integration blocks (FIBs) are used sequentially to Generate more refined deep features To perform feature fusion. Each feature integration block consists of a Dimensional Attention Enhancement Module (DAEM) and a Dimensional Rolling Transformation Module (DRTM). Finally, the final feature The input is fed into the upsampler and then Multiply to get the fused image .

[0033] Among them, the feature integration module includes a dimensional attention enhancement module and a dimensional rolling transformation module. The dimensional attention enhancement module focuses on extracting local features of the image and can capture local details and texture information in the image in detail. Specifically, the dimensional attention enhancement module includes a loop scanner, an attention cache, and quantization compression; the dimensional rolling transformation module captures the global features of the image through dimensional rolling transformation, thereby effectively grasping the overall structure and long-distance dependencies of the image. The two modules work together, allowing the network to fully process the local and global information of the image while maintaining computational efficiency, providing support for complex image processing tasks.

[0034] Reference Figure 2 , Figure 2 : is a flow chart of an ultra-high-definition dynamic multi-exposure image fusion method provided in an embodiment of the present application. The method comprises the following steps, and each implementation step is specifically described below: In step S1, dynamic scene samples of the video to be processed are extracted, and a dynamic multi-exposure image dataset is constructed based on the dynamic scene samples. The dynamic multi-exposure image dataset includes low-exposure images, medium-exposure images, and high-exposure images of the same scene.

[0035] Specifically, since the image resolution of existing dynamic multi-exposure datasets has not reached 4K (3840×2160), it cannot meet the training and testing requirements of the model. Therefore, it is necessary to first build an ultra-high-definition dynamic multi-exposure image dataset. Specifically, a video with a resolution of 4K (3840×2160) is recorded through a mobile phone or other photographic equipment to obtain a video to be processed. The video includes but is not limited to dynamic elements such as moving people and cars. Three frames are extracted from the video to obtain dynamic scene samples. There are three samples corresponding to one scene. The three samples are synthesized into low, medium and high exposure levels to obtain a dynamic multi-exposure image dataset.

[0036] In step S2, the low-exposure image, the medium-exposure image, and the high-exposure image are stitched along the image channel dimension to obtain a stitched image.

[0037] Specifically, the three exposure images in the dynamic multi-exposure image dataset are spliced ​​along the channel dimension. This process can be completed by using a model to obtain a spliced ​​image.

[0038] In step S3, downsampling is performed on the stitched image to obtain low-level features.

[0039] Specifically, the stitched image is mapped to the feature space through a downsampler to obtain low-level features, wherein the step of stitching along the image channel dimension of the low-exposure image, the medium-exposure image, and the high-exposure image to obtain the stitched image includes: The stitched image is mapped to the feature space through a preset downsampler to obtain the low-level feature , the downsampler includes at least one sub-pixel convolution layer and at least one 3×3 convolution layer; in, represents the channel dimension, represents the width dimension, Represents the height dimension.

[0040] Specifically, refer to Figure 1 The extracted low-level features are convenient for block processing according to channel, width and height dimensions and then output to the slice loop scanner.

[0041] In step S4, dimensional attention enhancement is performed based on the low-level features to obtain enhanced local features, and dimensional rolling transformation is performed based on the enhanced local features to obtain global features.

[0042] Specifically, the low-level features are input into the dimension attention enhancement module in the preset feature integration block for local feature enhancement, and the enhanced local features are input into the dimension rolling transformation module in the feature integration block for global feature extraction. The following is a detailed description of the specific steps for obtaining enhanced local features: The step of performing dimensional attention enhancement based on the low-level features to obtain enhanced local features comprises: Blocking is performed along the channel dimension, width dimension and height dimension of the low-level features, and after blocking, a dimensional block corresponding to each dimension is obtained; A preset local feature extractor is used to extract the deep features of each dimensional block, and each extracted deep feature is processed based on the Sigmoid function to generate the attention weight corresponding to each dimension in the low-level features; Attention enhancement is performed on the low-level features according to the attention weight corresponding to each dimension to obtain the enhanced local features.

[0043] Specifically, the input data is divided into blocks along the channel, width and height (that is, the low-level features obtained in the above steps are divided into blocks), and then a circular scanner is used to capture local features of the dimensional blocks of each dimension, wherein a local feature extractor (LFE) is constructed in the circular scanner. The local feature extractor inputs the input block into the global average pooling layer and a series of convolutional layers to extract deep features, and then uses the Sigmoid function to process these deep features to generate attention weights for the input low-level features.

[0044] More specifically, after obtaining the attention weights of the low-level features, the element-by-element multiplication is used to apply the weights of the three dimensions to the low-level features of the corresponding input, thereby dynamically adjusting the importance of the channels and improving the expressiveness of specific features. The operation is repeated until the dimensional block features of the three dimensions are extracted by the local feature extractor. The outputs of the three dimensions are added together to obtain the final output of the dimensional attention enhancement module, that is, the enhanced local features.

[0045] Among them, the expression of the local feature extractor is: ; ; ; in, are input features, is the pooling feature, is the channel weight, represents the output features of the local feature extractor, represents global average pooling, is the Sigmoid function, is the ReLU function, represents a 1×1 convolution, Stands for element-wise multiplication.

[0046] Furthermore, this application uses attention cache technology to achieve accelerated reasoning of the entire network, which is described in detail below: The step of using a preset local feature extractor to extract deep features of each dimensional block, and processing each extracted deep feature based on a Sigmoid function to generate an attention weight corresponding to each dimension in the low-level features includes:

[0047] in, Represents a dimension block The final local features of represents the attention cache, represents the local feature extractor, represents a read operation in the attention buffer, Represents a write operation in the attention buffer.

[0048] Specifically, the local feature extractor processes deep features through the Sigmoid function to generate attention weights for the input features. The attention cache can be used to store the local features extracted by the local feature extractor, thereby avoiding redundant calculations and accelerating the reasoning process.

[0049] More specifically, during the training process, the intermediate results (feature maps) of the network are obtained through convolution operations. However, during the inference process, these convolution operation results are stored as caches through the above method and can be read directly without repeated calculations, thereby achieving accelerated inference of the entire network. It is worth noting that the write operation will write the k value and the corresponding attention cache.

[0050] Furthermore, the present application uses quantization compression technology to reduce the memory burden of the device, which is described in detail below: The corresponding quantized tensor is obtained, and the process expression of the quantization compression is:

[0051]

[0052]

[0053] in, is the scaling factor, It's zero o'clock. is a quantized tensor, is a tensor The maximum value in is a tensor The minimum value in is the maximum value of the quantization range, is the minimum value of the quantization range, Used to get the maximum value of the comparison value. Used to get the minimum value of the comparison value. Used to convert a floating point number to the nearest integer.

[0054] Specifically, as model training and inference proceed, the memory consumption of the attention cache increases rapidly, significantly increasing the burden on device memory. To alleviate this problem, this application uses quantization compression to reduce storage memory. In Quantization Compression, an encoder is constructed to compress floating-point tensors into quantized tensors.

[0055] Correspondingly, a decoder is used to decompress the quantized tensor. The decompression process expression is:

[0056] in, Represents a dequantized tensor.

[0057] The step of performing dimensional attention enhancement based on the low-level features to obtain enhanced local features can be written as:

[0058] in, represents the quantization compression operation, Represents a decompression operation.

[0059] Further, after the feature completes the above-mentioned step of obtaining enhanced local features, the obtained local features are input into the dimensional rolling transformation module in the feature integrated block for global feature extraction, which is specifically described below: The resolution of the enhanced local feature is adjusted by interpolation operation, and a preset dimensional transformation step is performed on the adjusted resolution to obtain an intermediate feature , the intermediate features are obtained The process expression is:

[0060] ; in, and is the intermediate result, represents the interpolation operation, represents a 1×1 convolution, is the GELU function, Represents a dimension transformation operation, and ×3 means performing the operation three times in sequence.

[0061] Specifically, after local feature extraction, global feature extraction is also required. In this application, the dimensional rolling transformation module solves the problem of global feature loss during cyclic scanning. This application encodes the feature map of the image from the perspective of width, height and channel, and associates and fuses the information encoded in each feature map. It is worth noting that the dimensional conversion operation does not introduce additional parameters, which means that by arranging feature maps from different views, the dimensional rolling transformation module achieves effective modeling of long-distance dependencies.

[0062] More specifically, the input features are first adjusted by interpolation (i.e., local features are enhanced), and the adjusted input features are transformed in dimension; then, the intermediate result features are interpolated. Resize to its original resolution to estimate the attention map and adaptively modulate the input by element-wise multiplication according to the estimated attention , get the output features (global features ). Specifically, the global feature is obtained, and the process expression of obtaining the global feature is:

[0063] in, is the output feature, Stands for element-wise multiplication.

[0064] In general, the processing flow of input features through the feature manifold can be described by the following expression:

[0065] in, represents the output of the dimensional attention enhancement module, Represents the final output.

[0066] In step S5, upsampling is performed on the global features to obtain a high-resolution feature map.

[0067] Specifically, in an embodiment of the present application, an upsampler consisting of a sub-pixel convolution layer and a 3×3 convolution layer is used to perform an upsampling operation on the global features. After the low-resolution feature map is upsampled, the spatial resolution of the feature will increase, thereby obtaining fine features with higher resolution. These features usually retain rich information from low-level features to high-level features for subsequent image reconstruction operations.

[0068] In step S6, the high-resolution feature map is combined with the medium-exposure image to obtain a fused image result.

[0069] Specifically, the final features extracted by the feature integration block are processed by the upsampler to generate a high-resolution feature map, that is, the resolution is restored to 4K, which is consistent with the resolution of the three input exposure images. In this case, according to the constructed data set and the existing non-ultra-high-definition dynamic multi-exposure data set, it can be seen from the three input high, medium, and low exposure images that the positions of the dynamic elements (such as moving vehicles) in each image may be different. In the embodiment of the present application, the original image samples (that is, the original image samples used to synthesize the medium exposure image) have been used as the gold standard (ground truth) before the synthetic exposure, so the pixels of the dynamic elements with important information in the input medium exposure image are retained to ensure the accurate presentation of these key parts in the final fusion result. The final features extracted by the feature integration block contain high-level feature information of the image, such as texture, edge, semantic information, etc. These features are multiplied with the medium exposure image, and the image quality is improved by adaptively adjusting the pixel value, and finally the fused image result is obtained.

[0070] Furthermore, the above steps describe the process of multi-exposure image fusion, and the entire process relies on the model obtained through training. The model optimization adopted in the embodiment of the present application is based on a specific loss function, which is used to guide the training process to ensure that the fused image achieves the best results in terms of detail retention, brightness balance and visual quality.

[0071] Specifically, this method uses the L1 loss function to calculate the L1 loss value of the fusion result and the gold standard (ground truth) during the model training process, which can be written as:

[0072] in, represents the gold standard (ground truth), Represents the output fused image.

[0073] Furthermore, the trained network model is tested, so that the method can be applied to fuse ultra-high-definition dynamic multi-exposure images.

[0074] In summary, this application first constructs a dynamic multi-exposure dataset that can be used for model training and testing, maps it to the feature space, and processes the image features through the block-cache-quantization machine learning paradigm. The block-cache-quantization method includes: the first step is to slice and block the input spliced ​​image to reduce the pressure of the model processing data stream; the second step is to use the attention cache technology to process infinite data streams; the third step is to use quantization compression to reduce the pressure burden on the high-speed cache on the device. After the features are processed, high-resolution images are output and fused with gold standard samples, so as to achieve efficient processing of multiple ultra-high-definition images of different exposures on a single consumer-grade GPU, ensuring the beneficial effect of obtaining clear and natural fusion effects in complex dynamic scenes.

[0075] Among them, the beneficial effects provided by this application are verified by the following comparative experiments. Specifically, compared with the existing dynamic multi-exposure image fusion methods (such as GDC, AHDRNet, FSHDR, HDR-Transformer, SCTNet), this method is experimentally verified in the ultra-high-definition dynamic multi-exposure dataset we constructed and the public non-ultra-high-definition datasets (such as the Kalantari dataset and the Mobile-HDR dataset).

[0076] Specifically, this application uses PyTorch to conduct experiments on a single NVIDIA GeForce RTX 4090 GPU. To optimize the network, this application uses the AdamW optimizer with a learning rate of 2× , where the network was trained for 1200 epochs, the batch size was 4, the number of feature integration modules (FIBs) was 8, and the number of feature channels was 48. It should be noted that many existing methods cannot directly perform full-resolution reasoning on ultra-high-definition images (UHD). For these methods, the comparative experiments provided in this application first use bilinear interpolation to downsample the input ultra-high-definition images to the maximum resolution that these algorithms can process on a single GPU (such as Figure 3 As shown, Figure 3 is a list of quantitative comparison results of ultra-high-definition dynamic multi-exposure data sets provided in an embodiment of the present application). After the inference is completed, the output result is upsampled back to 4K resolution using bilinear interpolation.

[0077] Furthermore, this application adopts three image quality assessment indicators, such as Figure 3 As shown in the figure, including PSNR, SSIM and LPIPS, among which PSNR (Peak Signal-to-Noise Ratio) is a common indicator for measuring image quality. It is mainly used to evaluate the difference between the processed image and the original image. The larger the value, the better the image quality. Among them, the calculation formula of PSNR is: ; ; Among them, MSE stands for mean square error, which is used to measure the difference between two images at the pixel level. I(i,j) and K(i,j) represent the pixel values ​​of the generated fused image and the corresponding gold standard image (ground truth) at position (i,j), respectively. m and n represent the width and height of the image, respectively. MAX represents the maximum value of the image pixels.

[0078] SSIM (Structural Similarity Index Measure) is an indicator used to evaluate the quality of two images. It mainly measures the similarity of images by comparing the brightness, contrast and structural information of the images. The closer the value is to 1, the better the image quality. The calculation formula of SSIM is:

[0079]

[0080]

[0081] in, yes average value, is with The corresponding gold standard The average value of It represents an image and The covariance of yes The variance of yes The variance of L is the dynamic range of pixel values. and Represents the preset hyperparameters, here =0.01, =0.03, and represents the smoothing parameter.

[0082] The LPIPS (Learned Perceptual Image Patch Similarity) metric is a metric used to evaluate the perceptual similarity between two images. It is based on features extracted by a deep convolutional neural network rather than simple pixel differences. The smaller the value, the better the image quality. The calculation formula of LPIPS is: ; Among them, x represents the fused image output by the model, x0 represents the gold standard image corresponding to x, and represents the first Layer feature map. is the image x in the The characteristics of the layer, is the image x0 at the Characteristics of the layer. Represents the Euclidean distance (L2 norm) between feature maps. This is the feature map and The difference metric between the two images is used to measure the Similarity in layer characteristics. Represents the weight coefficient of each layer of feature map. Different network layers have different effects on perceived similarity, and the weight Used to reflect this, it is usually learned through training. For all layers The results are summed up to comprehensively consider the feature differences at different levels.

[0083] Further, refer to Figure 3 The comparative experiment provided in the embodiment of the present application also records the highest resolution that each algorithm can process on a single RTX4090 GPU, called the maximum resolution (Maximum Resolution), and the time required to generate a fused image, called the inference time (Inference Time).

[0084] Combination Figure 3 From the table shown, it can be seen Figure 3The quantitative comparison results of various methods evaluated on the ultra-high-definition dynamic multi-exposure fusion dataset constructed by this application are shown, and the optimal and suboptimal values ​​are highlighted in bold and underlined text, respectively. It can be seen that compared with the second best method FSHDR, our method has improved PSNR and SSIM by about 46% and 48%, respectively. In addition, the method provided by this application also has significant improvements on LPIPS, with performance about 88% better than FSHDR. Although the method provided by this application is not the fastest in terms of reasoning time, it can still achieve real-time reasoning with a frame rate of more than 40fps.

[0085] Further, refer to Figure 4 , Figure 4 This is a tabular diagram of the quantitative comparison results of the comparative experiments of this application on the Kalantari dataset and the quantitative comparison results on the Mobile-HDR dataset. It can be seen that the performance of the method provided in this application on the non-ultra-high-definition dataset is comparable to the current most advanced method. It can be seen that the method provided in this application can not only maintain high-quality results in the multi-exposure image fusion task of different resolutions, but also demonstrates highly competitive performance.

[0086] Furthermore, in order to demonstrate the effectiveness of each part of the method provided by this application, this application further discloses three corresponding ablation experiments, in which each key component is evaluated in a completely fair environment. For these experiments, the same architecture and hyperparameters are used, and only one component is changed each time. The evaluation of these ablation experiments is performed on the ultra-high-definition dynamic multi-exposure dataset constructed by this application. For details, please refer to Figure 5 , Figure 5 It is a schematic diagram of the ablation experiment results of the key components DAEM and DRTM provided in the embodiments of the present application.

[0087] Combination Figure 5 It can be seen that after removing the DAEM module, the performance of the model has dropped the most. From the results of (b) to (h), it can be seen that all three dimensions of the slice contain key information, especially the width and height dimensions, which are crucial for the final image fusion.

[0088] For further information, please refer to Figure 6 , Figure 6 This is a schematic diagram of the ablation experiment results of the key component attention cache provided by this application, combined with Figure 6 It can be seen that although the performance is slightly reduced due to quantization compression in the attention buffer, the inference time is significantly improved. In the embodiment provided in this application, the processing time is reduced by 70% from 0.1475 seconds to 0.0441 seconds, which can be considered to achieve real-time inference.

[0089] Reference Figure 7, Figure 7 : is a virtual structural diagram of an ultra-high-definition dynamic multi-exposure image fusion device provided by the present application. The second aspect of the present application provides an ultra-high-definition dynamic multi-exposure image fusion device, including: A multi-exposure atlas construction module 100 is used to extract dynamic scene samples of the video to be processed, and construct a dynamic multi-exposure image dataset based on the dynamic scene samples, wherein the dynamic multi-exposure image dataset is a low-exposure image, a medium-exposure image, and a high-exposure image of the same scene; A multi-exposure atlas stitching module 200, configured to stitch the low-exposure image, the medium-exposure image, and the high-exposure image along an image channel dimension to obtain a stitched image; A downsampling module 300, configured to perform downsampling on the stitched image to obtain low-level features; A feature integration processing module 400 is used to perform dimensional attention enhancement based on the low-level features to obtain enhanced local features, and perform dimensional rolling transformation based on the enhanced local features to obtain global features; An upsampling module 500 is used to perform upsampling on the global features to obtain a high-resolution feature map; The fused image output module 600 is used to combine the high-resolution feature map with the medium-exposure image to obtain a fused image result.

[0090] The ultra-high-definition dynamic multi-exposure image fusion device described in the embodiment of the present application can execute the ultra-high-definition dynamic multi-exposure image fusion method provided in the above embodiment. The ultra-high-definition dynamic multi-exposure image fusion device has the corresponding functional steps and beneficial effects of the ultra-high-definition dynamic multi-exposure image fusion method described in the above embodiment. Please refer to the embodiment of the above-mentioned ultra-high-definition dynamic multi-exposure image fusion method for details. The embodiment of the present application will not be repeated here.

[0091] The present application also provides an electronic device. Figure 8 , Figure 8It is a structural diagram of an electronic device provided in an embodiment of the present application, and the electronic device may include a processor and a memory, wherein the processor and the memory may be connected via a bus or other means. The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components and other chips, or a combination of the above-mentioned types of chips. The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the ultra-high-definition dynamic multi-exposure image fusion method in the embodiment of the present application. The processor executes various functional applications and data processing of the processor by running the non-transient software programs, instructions and modules stored in the memory, that is, the ultra-high-definition dynamic multi-exposure image fusion method in the above method embodiment is realized.

[0092] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. The one or more modules are stored in the memory, and when executed by the processor, the ultra-high-definition dynamic multi-exposure image fusion method as in the above-mentioned method embodiment is executed. The specific details of the above-mentioned electronic device can be understood by corresponding to the corresponding descriptions and effects in the above-mentioned method embodiment, and will not be repeated here. Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and the program, when executed, may include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above types of memory.

[0093] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0094] Similarly, it should be understood that in order to streamline the present disclosure and aid in understanding one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting the following intention: the claimed application requires more features than the features explicitly recited in each claim. More specifically, as reflected in the claims, the inventive aspects lie in less than all the features of the individual embodiments disclosed above. Therefore, the claims that follow the specific embodiment are hereby expressly incorporated into the specific embodiment, with each claim itself serving as a separate embodiment of the present application.

[0095] It should be noted that the above-mentioned embodiments illustrate rather than limit the invention and that those skilled in the art will be able to design alternative embodiments without departing from the scope of the appended claims.

Claims

1. An ultra-high-definition dynamic multi-exposure image fusion method, characterized in that: The method comprises: Extracting dynamic scene samples of the video to be processed, and constructing a dynamic multi-exposure image dataset based on the dynamic scene samples, wherein the dynamic multi-exposure image dataset includes low-exposure images, medium-exposure images, and high-exposure images of the same scene; Stitching the low-exposure image, the medium-exposure image, and the high-exposure image along an image channel dimension to obtain a stitched image; Downsampling the stitched image to obtain low-level features; Performing dimensional attention enhancement based on the low-level features to obtain enhanced local features, and performing dimensional rolling transformation based on the enhanced local features to obtain global features; Performing upsampling on the global features to obtain a high-resolution feature map; The high-resolution feature map is combined with the medium-exposure image to obtain a fused image result.

2. The ultra-high-definition dynamic multi-exposure image fusion method according to claim 1, characterized in that: The step of stitching the low-exposure image, the medium-exposure image, and the high-exposure image along the image channel dimension to obtain a stitched image comprises: The stitched image is mapped to the feature space through a preset downsampler to obtain the low-level feature , the downsampler includes at least one sub-pixel convolution layer and at least one 3×3 convolution layer; in, represents the channel dimension, represents the width dimension, Represents the height dimension; The step of performing dimensional attention enhancement based on the low-level features to obtain enhanced local features comprises: Blocking is performed along the channel dimension, width dimension and height dimension of the low-level features, and after blocking, a dimensional block corresponding to each dimension is obtained; A preset local feature extractor is used to extract the deep features of each dimensional block, and each extracted deep feature is processed based on the Sigmoid function to generate the attention weight corresponding to each dimension in the low-level features; Attention enhancement is performed on the low-level features according to the attention weight corresponding to each dimension to obtain the enhanced local features.

3. The ultra-high-definition dynamic multi-exposure image fusion method according to claim 2, characterized in that: The expression of the local feature extractor is: ； ； ； in, are input features, is the pooling feature, is the channel weight, represents the output features of the local feature extractor, represents global average pooling, is the Sigmoid function, is the ReLU function, represents a 1×1 convolution, Stands for element-wise multiplication.

4. The ultra-high-definition dynamic multi-exposure image fusion method according to claim 2, characterized in that: The step of using a preset local feature extractor to extract deep features of each dimensional block, and processing each extracted deep feature based on a Sigmoid function to generate an attention weight corresponding to each dimension in the low-level features includes: in, Represents a dimension block The final local features of represents the attention cache, represents the local feature extractor, represents a read operation in the attention buffer, Represents a write operation in the attention buffer.

5. The ultra-high-definition dynamic multi-exposure image fusion method according to claim 4, characterized in that: The method further comprises: The floating-point tensor generated in the process of executing the attention cache is quantized and compressed to obtain a corresponding quantized tensor. The process expression of the quantization compression is: in, is the scaling factor, It's zero o'clock. is a quantized tensor, is a tensor The maximum value in is a tensor The minimum value in is the maximum value of the quantization range, is the minimum value of the quantization range, Used to get the maximum value of the comparison value. Used to get the minimum value of the comparison value. Used to convert a floating point number to the nearest integer.

6. The ultra-high-definition dynamic multi-exposure image fusion method according to claim 1, characterized in that: The step of performing dimensional rolling transformation based on the enhanced local features to obtain global features comprises: The resolution of the enhanced local feature is adjusted by interpolation operation, and a preset dimensional transformation step is performed on the adjusted resolution to obtain an intermediate feature , the intermediate features are obtained The process expression is: ； in, and is the intermediate result, represents the interpolation operation, represents a 1×1 convolution, is the GELU function, Represents a dimension transformation operation, and ×3 means performing the operation three times in sequence.

7. The ultra-high-definition dynamic multi-exposure image fusion method according to claim 6, characterized in that: The interpolation operation is used to adjust the resolution of the enhanced local feature, and a preset dimensional transformation step is performed on the adjusted resolution to obtain an intermediate feature After the step of: The intermediate features that have undergone dimension transformation are interpolated The resolution is restored to the original resolution before adjustment, and based on the restored intermediate features Estimate the attention map; Adjust and enhance local features based on the attention map , the global feature is obtained, and the process expression of obtaining the global feature is: in, is the output feature, Stands for element-wise multiplication.

8. An ultra-high-definition dynamic multi-exposure image fusion device, characterized in that: include: A multi-exposure atlas construction module is used to extract dynamic scene samples of the video to be processed, and construct a dynamic multi-exposure image dataset based on the dynamic scene samples, wherein the dynamic multi-exposure image dataset includes low-exposure images, medium-exposure images, and high-exposure images of the same scene; A multi-exposure atlas stitching module, used for stitching the low-exposure image, the medium-exposure image and the high-exposure image along the image channel dimension to obtain a stitched image; A downsampling module, used for performing downsampling on the stitched image to obtain low-level features; A feature integration processing module, used to perform dimensional attention enhancement based on the low-level features to obtain enhanced local features, and perform dimensional rolling transformation based on the enhanced local features to obtain global features; An upsampling module, used to perform upsampling on the global features to obtain a high-resolution feature map; The fused image output module is used to combine the high-resolution feature map with the medium-exposure image to obtain a fused image result.

9. An electronic device, characterized in that: include: at least one processor and at least one memory, wherein the memory has computer-readable instructions stored thereon; The computer-readable instructions are executed by one or more of the processors, so that the electronic device implements the ultra-high-definition dynamic multi-exposure image fusion method as described in any one of claims 1 to 7.

10. A storage medium having computer-readable instructions stored thereon, characterized in that: The computer-readable instructions are executed by a processor to implement the ultra-high-definition dynamic multi-exposure image fusion method as described in any one of claims 1 to 7.

Images