Image generation method, electronic device, and storage medium

Abstract

The present disclosure provides an image generation method, an electronic device and a storage medium. The image generation method comprises: obtaining a composite feature tensor of multiple frames of low dynamic range images with different exposure times in the same scene, wherein the composite feature tensor comprises a nonlinear radiation feature channel and a linear radiation feature channel; extracting a correlation feature representation according to a composite feature sub-tensor of an image block corresponding to a position in each frame of low dynamic range images, the correlation feature representation representing quality features, motion features and inter-frame relationship features of each frame of image blocks; determining a reference frame image block of each frame of image blocks corresponding to the position according to the correlation feature representation; and performing feature alignment and fusion processing according to the composite feature tensor of each frame of low dynamic range images, to generate a high dynamic range image, with the reference frame image block as a reference. The present disclosure realizes adaptive selection of a reference frame according to a scene, improves image quality, and improves generalization ability and robustness.

Description

Technical Field

[0001] This disclosure relates to the fields of computer vision and digital image processing technology, and in particular to an image generation method, an electronic device, and a computer-readable storage medium. Background Technology

[0002] High Dynamic Range (HDR) imaging aims to capture and reproduce a wide range of brightness in a real scene. However, the dynamic range of ordinary camera sensors is limited, and a single shot can usually only capture a Low Dynamic Range (LDR) image. However, LDR images may lose details in bright areas due to overexposure or in dark areas due to underexposure.

[0003] To achieve HDR imaging, a multi-exposure fusion (MEF) approach is typically used. This involves capturing multiple LDR images of the same scene at different exposures, and then generating an HDR image through image alignment, HDR reconstruction, and tone mapping. Image alignment aims to eliminate the effects of slight movement of objects or the camera that may occur during the shooting intervals of the multiple LDR images. HDR reconstruction transforms the aligned LDR image to linear radiative space to generate a linear HDR image. Tone mapping aims to compress the wide brightness range of the linear HDR image to the range that the LDR display device can display, while preserving as much detail and visual quality as possible.

[0004] However, traditional MEF schemes have many inherent defects in alignment and artifact control, resulting in poor quality and weak robustness of the generated HDR images. Summary of the Invention

[0005] This disclosure provides an image generation method, an electronic device, and a computer-readable storage medium.

[0006] In a first aspect, embodiments of this disclosure provide an image generation method, comprising: acquiring a composite feature tensor of multiple frames of low dynamic range images with different exposure times in the same scene, wherein the composite feature tensor includes a nonlinear radiation feature channel and a linear radiation feature channel; extracting associated feature representations based on the composite feature sub-tensor of image blocks corresponding to positions in each frame of the low dynamic range images, wherein the associated feature representations characterize the quality features, motion features, and inter-frame relationship features of the image blocks in each frame; determining reference frame image blocks corresponding to positions in each frame based on the associated feature representations; and performing feature alignment and fusion processing based on the reference frame image blocks and the composite feature tensor of each frame of the low dynamic range images to generate a high dynamic range image.

[0007] In some embodiments, the step of extracting associated feature representations based on the composite feature sub-tensors of image patches corresponding to positions in each frame of the low dynamic range image includes: performing convolutional feature extraction on the nonlinear radial feature channels of each composite feature sub-tensor to obtain a global feature vector of the image patch in each frame; inputting the global feature vector into a multi-head attention module to obtain an attention-weighted feature vector that fuses quality features and inter-frame relationship features; analyzing and determining the differences between the image patches in adjacent frames based on the nonlinear radial feature channels of each composite feature sub-tensor, and extracting motion features to obtain a motion feature vector; wherein, the associated feature representation includes the attention-weighted feature vector and the motion feature vector.

[0008] In some embodiments, inputting the global feature vector into the multi-head attention module to obtain an attention-weighted feature vector that fuses quality features and inter-frame relationship features includes: stacking the global feature vectors of the image blocks in each frame to obtain a stacked feature tensor; inputting the stacked feature tensor into the multi-head attention module, performing self-attention calculation according to a preset number of attention heads, and obtaining the attention-weighted feature vector.

[0009] In some embodiments, the step of estimating motion features based on the nonlinear radiation feature channels of the image blocks in each frame, extracting motion features, and obtaining a motion feature vector includes: concatenating the nonlinear radiation feature channels of any two adjacent frame image blocks and inputting them into a motion estimation network; extracting the difference features between the image blocks in adjacent frames to obtain a difference feature vector, wherein the difference feature vector represents the motion difference information between two adjacent frames; and concatenating multiple difference feature vectors to obtain the motion feature vector.

[0010] In some embodiments, determining the reference frame image block for each frame of the image block corresponding to the location based on the associated feature representation includes: concatenating the attention-weighted feature vector and the motion feature vector and then performing a confidence decision to obtain a confidence vector, wherein the confidence vector consists of weight values ​​that correspond one-to-one with the image blocks in multiple frames, and the weight values ​​represent the confidence of the corresponding image block as a reference frame image block; and determining the reference frame image block based on the confidence vector.

[0011] In some embodiments, determining the reference frame image block based on the confidence vector includes: determining a frame index sequence based on the confidence vector, the frame index sequence representing the arrangement order of the image blocks of each frame corresponding to the position, wherein the image block serving as the reference frame image block corresponds to a preset order in the arrangement order.

[0012] In some embodiments, the step of generating a high dynamic range image by performing feature alignment and fusion processing based on the composite feature tensors of the low dynamic range images of each frame, using the reference frame image blocks as a reference, includes: summarizing the composite feature tensors of the low dynamic range images of each frame according to the arrangement order of the image blocks of each frame corresponding to the positions represented by the frame index sequence, to obtain a tensor to be processed; and performing feature alignment and fusion processing on the tensor to be processed to generate the high dynamic range image.

[0013] In some embodiments, the step of summarizing the composite feature tensors of the low dynamic range images of each frame according to the arrangement order of the image blocks corresponding to the positions represented by the frame index sequence to obtain the tensor to be processed includes: concatenating the composite feature sub-tensors of the image blocks corresponding to the positions in the frame in the channel dimension according to the arrangement order represented by the frame index sequence, and concatenating the composite feature sub-tensors of image blocks at different positions in the spatial dimension to obtain the tensor to be processed.

[0014] In some embodiments, the step of performing feature alignment and fusion processing on the tensor to be processed to generate the high dynamic range image includes: splitting the tensor to be processed in the channel dimension to obtain multiple frames of input feature tensors, wherein the input feature tensors corresponding to the preset order are used as reference frames; using the input feature tensors of the reference frames as a basis, performing feature extraction and attention modulation on the multiple frames of input feature tensors to obtain modulation feature maps; concatenating multiple modulation feature maps and performing deep, multi-scale feature fusion to obtain a fused feature map; performing residual connection between the fused feature map and the initial feature map of the input feature tensor of the reference frames, and reconstructing the high dynamic range image through convolution processing.

[0015] In some embodiments, the step of performing feature extraction and attention modulation on the input feature tensors of multiple frames, using the input feature tensor as a reference frame as a basis, to obtain a modulation feature map includes: convolving the input feature tensors of multiple frames to extract an initial feature map of the input feature tensor; calculating an attention map between the initial feature map of the corresponding non-reference frame and the initial feature map of the corresponding reference frame; superimposing the attention map on the initial feature map of the corresponding reference frame and fusing it with the initial feature map of the corresponding non-reference frame to obtain the modulation feature map.

[0016] In some embodiments, the step of concatenating multiple modulation feature maps and then performing deep, multi-scale feature fusion to obtain a fused feature map includes: concatenating multiple modulation feature maps and inputting them into a hierarchical pyramid network, and performing deep, multi-scale feature fusion through the wavelet transform module and the Transformer module in the hierarchical pyramid network to obtain the fused feature map.

[0017] In some embodiments, before extracting associated feature representations based on the composite feature subtensors of the image blocks corresponding to the low dynamic range image positions in each frame, the image generation method further includes: dividing the low dynamic range image into multiple image blocks.

[0018] In some embodiments, obtaining a composite feature tensor of multiple frames of low dynamic range images with different exposure times in the same scene includes: converting the low dynamic range images to a linear radiometric space to obtain a linear radiometric image; and concatenating the linear radiometric image and the low dynamic range image in the channel dimension to obtain the composite feature tensor, wherein the nonlinear radiometric feature channel of the composite feature tensor corresponds to the low dynamic range image, and the linear radiometric feature channel of the composite feature tensor corresponds to the linear radiometric image.

[0019] Secondly, embodiments of this disclosure provide an electronic device, the electronic device comprising: one or more processors; and a memory storing one or more programs thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the image generation method described in any of the above.

[0020] Thirdly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the image generation method described in any one of the above claims.

[0021] In this embodiment, by extracting the associated feature representations of the quality features, motion features, and inter-frame relationship features of multiple frames / image blocks, a reference frame is determined based on the associated feature representations. Finally, feature alignment and fusion processing are performed based on the reference frame to generate a high dynamic range image. This achieves dynamic adaptive selection of the most suitable reference frame as the alignment and fusion benchmark based on the content quality, motion information, and inter-frame relationship of the input image. This is beneficial for selecting the clearest, most stable, and most information-stable image as the reference frame, providing a better benchmark for feature alignment and fusion processing, generating HDR images with richer details and fewer artifacts, thereby improving the quality of HDR image reconstruction.

[0022] In this embodiment of the disclosure, by extracting the associated feature representations of the quality features, motion features, and inter-frame relationship features of each frame image / image patch from multiple frames of images / image patches, and determining the reference frame based on the associated feature representations, the generalization ability and robustness in handling diverse scenes can be improved. The motion of objects in the selected reference frame is relatively small, which can reduce the motion displacement that needs to be compensated, reduce the difficulty of model learning alignment, and help to resolve artifact problems caused by large-scale motion compensation.

[0023] In this embodiment, the generation of HDR images using existing deep learning-based HDR reconstruction networks is supported. When determining reference frames, simply reordering the reference frames so that the image blocks / image patches serving as reference frames are in the default order of the HDR reconstruction network allows the HDR reconstruction network to perform feature alignment and fusion processing based on the determined reference frames. The reference frame selection method in this embodiment can be implemented as a plug-and-play component, enabling seamless integration with existing HDR reconstruction networks without changing the core structure of the HDR reconstruction network, thus exhibiting good compatibility and scalability. Attached Figure Description

[0024] Figure 1 This is a flowchart of an image generation method according to an embodiment of the present disclosure.

[0025] Figure 2 This is a schematic diagram comparing three low dynamic range images with different exposure values ​​and the output high dynamic range image in an embodiment of this disclosure.

[0026] Figure 3 This is a schematic diagram of the structure of an image generation model in an embodiment of this disclosure.

[0027] Figure 4 Based on the embodiments of this disclosure Figure 3 The diagram shows the image generation process of the image generation model.

[0028] Figure 5 This is a schematic diagram of the composition of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0029] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions of this disclosure will be described in detail below with reference to the accompanying drawings.

[0030] Exemplary embodiments will be described more fully below with reference to the accompanying drawings; however, these exemplary embodiments may be embodied in different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will enable those skilled in the art to fully understand the scope of this disclosure.

[0031] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0032] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0033] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded.

[0034] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0035] Figure 1 This is a flowchart illustrating an image generation method according to an embodiment of the present disclosure.

[0036] In a first aspect, embodiments of this disclosure provide an image generation method, such as... Figure 1 As shown, the image generation method includes the following steps S1 to S4.

[0037] S1. Obtain a composite feature tensor of multiple frames of low dynamic range images with different exposure times in the same scene, wherein the composite feature tensor includes a nonlinear radiation feature channel and a linear radiation feature channel.

[0038] In this step, the camera continuously acquires LDR images of the same scene with different exposure times during a single shot, capturing details in different brightness areas. This embodiment does not impose a special limitation on the number of LDR images. For example, it could be three consecutive LDR images, corresponding to underexposure, normal exposure, and overexposure, respectively capturing details in the shadows, midtones, and highlights of the scene.

[0039] To adapt to human eye perception and display characteristics, cameras convert linear light signals into nonlinear LDR images (such as sRGB format) using the Camera Response Function (CRF). This results in the pixel values ​​of the LDR image being disproportionate to the actual light intensity. To meet the requirement of physical additivity of light intensity for HDR reconstruction, an inverse CRF transform of the LDR image is performed in conjunction with the exposure time, converting the LDR image from a nonlinear color space to a linear radiative space.

[0040] In this step, the above linear transformation process is performed on each frame of the LDR image, generating a composite feature tensor. In computer vision and deep learning, images are represented by tensors; for example, an sRGB image includes three channels: R, G, and B. In this embodiment, the tensor representing the image is called a feature tensor. The feature tensor includes a tensor representing the LDR image and a tensor representing the linear radiometric image after the linear transformation. The composite feature tensor is the tensor obtained by concatenating the above feature tensors. The linear radiometric feature channel in the composite feature tensor corresponds to the linear radiometric image after the linear transformation, and the nonlinear radiometric feature channel corresponds to the original LDR image before the linear transformation. The linear radiometric feature channel is used to characterize the physical illumination intensity information in the scene, and the nonlinear radiometric feature channel is used to preserve the image texture and color information.

[0041] Given the composite feature tensor of each LDR image frame, you can select either the linear or nonlinear radiation feature channel for subsequent processing as needed, or you can perform subsequent processing based on the composite feature tensor.

[0042] By fusing physical illumination intensity information, image texture, and color information from the scene, richer input representations can be provided for subsequent reference frame selection, feature alignment and fusion, and high dynamic range image reconstruction.

[0043] In this embodiment of the disclosure, based on acquiring multiple consecutive frames of LDR images, the composite feature tensor of the LDR images is obtained by preprocessing the LDR images.

[0044] In the preprocessing process, the LDR image can first be transformed to a linear radiometric space through a linear transformation, and then the linear radiometric image and the LDR image can be concatenated to obtain a composite feature tensor. In some embodiments, step S1 includes the following steps S11 to S12.

[0045] S11. Convert the low dynamic range image to linear radiation space to obtain a linear radiation image.

[0046] The embodiments disclosed herein do not impose any special limitations on the method for converting ground dynamic range images to linear radiometric space.

[0047] For example, inverse gamma correction can be used to convert LDR images from a nonlinear color space to a linear radiative space; alternatively, LDR images can be converted from a nonlinear color space to a linear radiative space based on the inverse photoelectric conversion function. The inverse photoelectric conversion function is, for example, the inverse operation of the exact photoelectric conversion function (OETF) defined in the BT.709 standard of the International Telecommunication Union Radiocommunication Sector (ITU-R). The OETF describes the mapping relationship from light intensity to pixel value to meet the requirements of physical illumination reproduction accuracy in application scenarios.

[0048] S12. The linear radiometric image and the low dynamic range image are concatenated in the channel dimension to obtain the composite feature tensor, wherein the nonlinear radiometric feature channel of the composite feature tensor corresponds to the low dynamic range image, and the linear radiometric feature channel of the composite feature tensor corresponds to the linear radiometric image.

[0049] For each frame of LDR image, after calculating the linear radiometric image, the linear radiometric image and the LDR image are concatenated in the channel dimension (dim=0) to obtain a composite feature tensor. For example, if the LDR image is an sRGB image containing three channels: red (R), green (G), and blue (B), the linear radiometric image obtained after linear transformation is a tensor containing three channels. The first three channels of the concatenated composite feature tensor pair are the three channels of the linear radiometric image, and the last three channels are the three channels of the LDR image, forming a 6-channel tensor.

[0050] For example, a low dynamic range image consisting of three sRGB images (I0, I1, and I2) with different exposure times can be represented by the composite feature tensor obtained after preprocessing as follows: S2. Based on the composite feature subtensor of the image patch corresponding to the position in each frame of the low dynamic range image, extract the associated feature representation, which characterizes the quality features, motion features and inter-frame relationship features of the image patch in each frame.

[0051] In this step, the LDR image can be divided into multiple image blocks according to a preset grid strategy, and each image block can be processed to extract the associated feature representation of the image block; alternatively, the LDR image can be processed without dividing it, and the associated feature representation of the LDR image can be extracted. In this case, it is equivalent to dividing the LDR image into one image block, which is consistent with the LDR image.

[0052] When each frame of a multi-frame LDR image is divided into one or more image blocks according to the same preset grid strategy, the image block corresponding to a position refers to the image block whose spatial location and geometric boundary are completely consistent in the multi-frame LDR images. Image blocks corresponding to a position have the same grid index in their respective frames; for example, the first... Frame, First Frame, First The grid index in the frame is ( The image blocks of a location constitute the image blocks corresponding to that location.

[0053] In this step, when there is no image block or only one image block, the composite feature subtensor of the image block, which is the composite feature tensor of the LDR image, is equivalent to extracting the associated feature representation of the LDR image based on the composite feature tensor. The associated feature representation integrates the quality features, motion features, and inter-frame relationship features of each LDR image frame. In the subsequent step S3, one of the multiple LDR images is determined as the reference frame.

[0054] When the LDR image is divided into multiple image blocks, the composite feature subtensor of the image block refers to the slice of the image block in the spatial dimension. For image blocks at different spatial locations, the associated feature representation of the image block is extracted according to its composite feature subtensor. The associated feature representation is a fusion of the quality features, motion features and inter-frame relationship features of image blocks from each frame. In the subsequent step S3, reference frame image blocks are determined for image blocks at different spatial locations.

[0055] In this step, the associated feature representation can be composed of multiple feature vectors; the associated feature representation can also be a single feature vector, such as a feature vector obtained by concatenating multiple feature vectors. This disclosure does not impose any special limitations on this aspect.

[0056] In this step, quality features characterize the image quality and imaging effect of a single-frame LDR image. For example, they reflect content quality-related information such as image noise level, blur degree, edge sharpness, texture richness, local contrast, and effective detail integrity. Motion features characterize the displacement and dynamic changes of image pixels and image blocks within a single-frame LDR image and between adjacent frames. For example, they reflect dynamic information such as the movement amplitude of the camera or objects in the image and the distribution of moving areas. Inter-frame relationship features characterize the matching similarity, exposure detail complementarity, and temporal constraint correlation between pairs of multiple LDR images. For example, they reflect the overall mutual matching and complementary dependence of three frames.

[0057] When using the MEF scheme to achieve HDR imaging, if there is significant object motion, the global or local alignment algorithm is very likely to fail, resulting in severe ghosting, blurring or artifacts. Even if multiple LDR images can be effectively aligned, feature fusion of multiple LDR images may still introduce imperfections that are difficult to eliminate when dealing with complex motions such as rapid non-rigid body motion or objects that suddenly appear / disappear, making artifact control difficult.

[0058] To overcome the aforementioned shortcomings of the MEF scheme, a spatial alignment and feature fusion approach using a reference frame as an "anchor" can be adopted. This involves using one frame from a multi-frame LDR image as the reference frame, with the other frames aligned and fused based on this reference frame. However, taking a three-frame LDR image as an example, if the middle frame is consistently used as the reference frame, it may negatively impact the generated HDR image, as the middle frame is not always the highest quality or most stable. For instance, in extreme lighting conditions, the middle frame might contain both slightly overexposed and underexposed areas, resulting in lower information quality compared to underexposed or overexposed frames. Furthermore, if a moving object happens to be blurred within the middle frame, or if the motion displacement is greatest when the middle frame is used as a reference, forcibly aligning all frames to this middle frame forces the model to learn more difficult and error-prone alignment transformations, thus reducing the quality of the HDR image. This also limits the model's robustness and performance ceiling in handling diverse and complex scenes.

[0059] In this step, by extracting associated feature representations, the content quality, motion information, and inter-frame relationships of multi-frame image patches can be analyzed. This allows the subsequent step S3 to select the optimal reference frame for alignment and fusion in the current scene based on the content quality, motion information, and inter-frame relationships of the multi-frame image patches. In step S4, alignment and fusion are performed using this optimal reference frame as a benchmark to reconstruct the HDR image, thereby improving the quality of the HDR image, achieving scene-adaptive HDR reconstruction, and enhancing the robustness of the model. The selected reference frame has relatively small object motion, which reduces the amount of motion displacement that needs to be compensated, lowers the difficulty of model learning alignment, and helps to mitigate artifacts caused by large-scale motion compensation.

[0060] Furthermore, when dividing the image into blocks, the associated feature representations of each frame image block are extracted for different spatial locations, and the best reference frame image block for each spatial location is selected independently. The reference frame image blocks determined at different spatial locations may correspond to the same LDR image frame or different LDR images, allowing the selection of the most suitable reference frame image block based on the brightness, detail, motion, etc. of different regions of the LDR image. This is beneficial for obtaining better HDR alignment and fusion effects globally, thereby further improving the quality of HDR images and enhancing the robustness of the model.

[0061] In some embodiments, prior to step S2, the image generation method further includes: dividing the low dynamic range image into multiple image blocks.

[0062] This disclosure does not impose any special limitations on how the LDR image is divided. For example, following the same grid strategy, each frame of the LDR image is divided into 6 image blocks of 2 rows × 3 columns.

[0063] In some embodiments, step S2 includes the following steps S21 to S23.

[0064] S21. Perform convolution feature extraction on the nonlinear radiative feature channels of each composite feature sub-tensor to obtain the global feature vector of each frame's image block.

[0065] In this step, depth features of each image patch are extracted by performing convolution operations on the nonlinear radiative feature channels of the composite feature subtensor; then, through pooling and flattening, global feature vectors of each frame image patch are obtained; the global feature vectors fuse complete information of a single frame image patch in the entire space and high-level semantics.

[0066] Pooling aims to downsample and compress the feature map output by convolution, reducing data volume and computational cost while preserving key features and improving feature robustness. For example, the feature map output after convolution is calculated by sliding it across a fixed window and stride, using a single value to represent the features of the entire region within the window. In some embodiments, adaptive average pooling is used in step S21, which takes the average value of the features within the window. Adaptive pooling specifies the output size rather than the pooling window size; the algorithm automatically calculates an appropriate window size and stride based on the size of the input feature map.

[0067] Flattening aims to convert a multidimensional tensor obtained after pooling into a one-dimensional vector. For example, the flattening operation concatenates a multidimensional tensor end to end in channel order to convert it into a one-dimensional vector.

[0068] In some embodiments, convolutional feature extraction is performed using a small convolutional neural network, the structure of which includes: First convolutional layer: 3 input channels, 64 output channels, 3×3 kernel size, and 1 padding; First activation layer: uses the ReLU activation function; Second convolutional layer: 64 input channels, 128 output channels, 3×3 kernel size, 1 padding; Second activation layer: uses the ReLU activation function; Adaptive average pooling layer: pools the output feature map to a fixed spatial size; Flattening layer: Converts the pooled feature map into a one-dimensional feature vector.

[0069] In some embodiments, the adaptive average pooling layer pools the feature map output by the convolution to an 8×8 size, and after flattening, the dimension of each global feature vector is 128×8×8=8192.

[0070] S22. Input the global feature vector into the multi-head attention module to obtain an attention-weighted feature vector that fuses quality features and inter-frame relationship features.

[0071] In this step, the multi-head attention mechanism is used to calculate the interrelationships and dependencies between global feature vectors of multiple image patches.

[0072] The multi-head attention mechanism comprises multiple self-attention heads, using the global feature vectors of multi-frame image patches as queries, keys, and values. Each self-attention head independently computes attention to measure the relevance of features. Specifically, the multi-head attention mechanism calculates the weights of the query and key vectors, scales them by dividing by the square root of their dimensions, and then normalizes them to obtain an attention weight matrix. This weight quantifies the interrelationship and dependency between features of any two image patches. This weight is then used to weight and sum the value vectors to generate new feature representations. Finally, the outputs of all heads are concatenated and linearly transformed again, thereby achieving comprehensive modeling and fusion of complex dependencies of global features across multiple dimensions, including semantics, spatiotemporal dimensions, and texture. After processing by the multi-head attention mechanism, the resulting attention-weighted feature vector integrates the quality features of image patches and inter-frame relationship features.

[0073] In some embodiments, step S22 includes the following steps S221 to S222.

[0074] S221. Stack the global feature vectors of the image blocks in each frame to obtain a stacked feature tensor.

[0075] In this step, the global feature vectors of each frame image block obtained in step S21 are stacked on dimension 1 to obtain a stacked feature tensor, which is used as the input of the multi-head attention module.

[0076] For example, taking three LDR images as an example, the dimension of the global feature vector is 8192. After stacking, it forms a stacked feature tensor with the shape [B, 3, 8192]. Here, B corresponds to the batch dimension, representing the number of samples in one input; 3 corresponds to the number of frames dimension, representing that each group of samples contains 3 frames; and 8192 corresponds to the length of the global feature vector.

[0077] S222. Input the stacked feature tensor into the multi-head attention module, perform self-attention calculation according to the preset number of attention heads, and obtain the attention-weighted feature vector.

[0078] In this step, a multi-head self-attention mechanism is used for processing. Specifically, multiple global feature vectors are input as queries, keys, and values ​​into the multi-head attention module for computation.

[0079] In one embodiment, the feature embedding dimension of the multi-head attention module is consistent with the global feature vector dimension, and the number of attention heads is a preset value, such as 4.

[0080] S23. Based on the nonlinear radiation feature channels of each composite feature sub-tensor, perform motion feature estimation, extract motion features, and obtain motion feature vectors.

[0081] Motion feature estimation refers to the analysis and calculation of multi-dimensional correlation information such as cross-frame pixel grayscale shift, spectral amplitude change, texture deformation, illumination radiation difference, and local occlusion distortion using nonlinear radiation feature channels of composite feature subtensors. It quantifies the dynamic correlation between nonlinear radiation feature channels at different temporal positions, mines motion information such as inter-frame displacement, scaling, rotation, and local jitter hidden at the feature level, and completes the structured extraction and encoding of motion information. It is the core operation process of converting radiation features into standardized motion feature vectors.

[0082] This disclosure does not impose any special limitations on how motion feature estimation is performed. For example, motion feature extraction can be performed based on a motion estimation network, or motion feature estimation can be achieved through optical flow analysis algorithms, block matching iterative operations, feature point matching constraint solving, multidimensional feature difference fitting, temporal feature matrix decomposition, and other schemes.

[0083] In some embodiments, step S23 includes the following steps S231 to S232.

[0084] S231. The nonlinear radiation feature channels of any two adjacent frame image blocks are concatenated and input into the motion estimation network to extract the difference features between the image blocks of the adjacent frames, thereby obtaining the difference feature vector, wherein the difference feature vector represents the motion difference information between two adjacent frames.

[0085] In this step, motion feature estimation is performed based on a motion estimation network. First, the nonlinear radiative feature channels of any two adjacent frame image blocks are concatenated along the channel dimension. For example, taking a three-frame LDR image as an example, the nonlinear radiative feature channels of the composite feature subtensor of the three consecutive frame image blocks are represented as follows: , , ,Will and Perform splicing along the channel dimension, and Stitching along the channel dimension is equivalent to stitching together image patches corresponding to locations in the LDR image along the channel dimension, which then serves as input to the motion estimation network.

[0086] This disclosure does not impose any special limitations on the motion estimation network. For example, the motion estimation network can be a model built on a convolutional neural network, or a lightweight model built on an optical flow network or a recurrent neural network. For example, a motion estimation network based on a convolutional neural network can use backbone networks such as MobileNet, ShuffleNet, GhostNet, and SqueezeNet. It extracts apparent difference information such as gray-level gradients, texture edges, color shifts, and pixel displacements of adjacent image blocks layer by layer, and then compresses the results through global pooling and fully connected layers to obtain motion feature vectors. For example, optical flow network-based models can use optical flow estimation models such as LiteFlowNet and PWC-Net. By matching the pixel correspondence of adjacent image blocks layer by layer through a multi-scale image pyramid, the pixel offset and rotation scaling deformation parameters between consecutive frames are solved, and the multi-scale optical flow amplitude, direction, and residual information are concatenated and encoded into motion feature vectors. Lightweight motion estimation networks based on recurrent neural networks can use lightweight LSTM and GRU structures. They take the shallow visual features extracted from adjacent image blocks as temporal input, iteratively model the dynamic change law between continuous image blocks, capture implicit motion information such as small displacements and local jitter, and finally output motion feature vectors.

[0087] To achieve lightweight motion estimation networks, the number of parameters and computational cost can be reduced through model pruning, quantization, low-rank decomposition, and channel sparsification, thereby constructing lightweight motion estimation networks that are suitable for low-computing scenarios such as mobile terminals and wearable devices.

[0088] Based on the attention-weighted feature vector and motion feature vector, the associated feature representation can include the attention-weighted feature vector and motion feature vector. In subsequent processing, the attention-weighted feature vector and motion feature vector are used as inputs and stacked or concatenated as needed. The associated feature representation can also be a tensor or vector obtained by stacking or concatenating the attention-weighted feature vector and motion feature vector, and it meets the input requirements of subsequent processing.

[0089] In some embodiments, a lightweight motion estimation network is constructed based on a convolutional neural network to obtain motion features, and its structure includes: First convolutional layer: 6 input channels, 32 output channels; First activation layer: uses the ReLU activation function; Second convolutional layer: 32 input channels and 16 output channels; Second activation layer: uses the ReLU activation function; Adaptive average pooling layer: pools the feature map to 1×1; Fully connected layer: outputs low-dimensional motion feature vectors.

[0090] The lightweight motion estimation network outputs difference feature vectors representing the motion differences between adjacent frame image blocks. A difference feature vector, motion01, represents... and The motion difference information between them is represented by another difference feature vector, motion12. and Information on the differences in motion between them.

[0091] S232. Concatenate multiple differential feature vectors to obtain the motion feature vector.

[0092] For example, motion01 and motion12 are 8-dimensional vectors, which are concatenated to obtain a 16-dimensional motion feature vector.

[0093] S3. Determine the reference frame image block of each frame corresponding to the position based on the associated feature representation.

[0094] As mentioned above, the attention-weighted feature vector integrates quality features and inter-frame relationship features, while the motion feature vector integrates motion difference information between adjacent frame image patches. The resulting associated feature representation characterizes the quality features, motion features, and inter-frame relationship features of each frame image patch. In subsequent processing, using this associated feature representation as input, reference frame image patches can be determined based on the quality features, motion features, and inter-frame relationship features of multiple frame image patches. This enables adaptive selection of reference frame image patches by analyzing the content quality, motion information, and inter-frame relationships of multiple frame image patches.

[0095] In this step, if no image blocks are divided or only one image block is divided, one frame from the multiple LDR images is determined as the reference frame.

[0096] When an LDR image is divided into multiple image blocks, for different spatial locations, one frame from the multiple image blocks corresponding to that location is determined as the reference frame image block for that spatial location. In this embodiment of the disclosure, for image blocks at different spatial locations, the same or different reference frame image blocks can be adaptively selected independently based on their respective brightness distribution, texture complexity, and motion. The reference frame image blocks determined at different spatial locations may correspond to the same LDR image frame or different LDR image frames.

[0097] In some embodiments, step S3 includes the following steps S31 to S32.

[0098] S31. After concatenating the attention-weighted feature vector and the motion feature vector, a confidence decision is made to obtain a confidence vector, wherein the confidence vector is composed of weight values ​​that correspond one-to-one with the image blocks in multiple frames, and the weight values ​​represent the confidence of the corresponding image block as a reference frame image block. In this step, the attention-weighted feature vector and the motion feature vector are concatenated along the feature dimension, and then a confidence decision is made. The confidence decision is a processing flow that quantifies the reliability of the target state, motion effectiveness, and feature credibility corresponding to the current sample based on the input, and outputs the judgment result based on the quantified confidence value.

[0099] In this step, the attention-weighted feature vector and motion feature vector are concatenated for confidence decision-making. This approach integrates the quality features and inter-frame relationship features represented by the attention-weighted feature vector, as well as the motion differences represented by the motion feature vector, eliminating the judgment bias caused by a single feature. The output is a confidence score / weight value representing the probability / confidence of the represented image patch as a reference frame image patch, forming a confidence vector. Then, combined with preset rules, the most suitable image patch as the reference frame image patch can be determined. For example, by combining preset thresholds and interval division rules for classification, filtering, correction, or elimination operations, the best reference frame image patch is finally selected, such as choosing the image patch with the largest weight value as the reference frame image.

[0100] This disclosure does not impose any specific limitations on the model and algorithm used to implement confidence decisions. For example, confidence decisions can be implemented using multilayer perceptron (MLP) decision networks, threshold segmentation discrimination, weighted feature distance metrics, support vector classification, clustering confidence scoring, and other schemes.

[0101] In some embodiments, the attention-weighted feature vector and the motion feature vector are concatenated and input into the MLP decision network for confidence decision. The MLP decision network outputs a confidence vector, and each weight value in the confidence vector represents the probability that the corresponding image patch is the best reference frame image patch.

[0102] MLP decision networks rely on stacked fully connected layers to construct a multi-layer nonlinear mapping structure, completing deep correlation mining of fused features, confidence quantification, and decision output, achieving end-to-end confidence scoring and sample discrimination. The MLP decision network consists of an input layer, at least one hidden layer, and an output layer connected in series, with fully connected layers. The number of input nodes in the input layer matches the dimension of the concatenated feature vector, ensuring the complete one-dimensional feature vector is fed into subsequent hidden layers. Before input, the feature vector can undergo preprocessing operations such as normalization, standardization, and outlier truncation. One or more fully connected hidden layers, each configured with several learnable weight and bias parameters, are used. Each layer's output is followed by a nonlinear activation function, such as Sigmoid, ReLU, LeakyReLU, or Tanh, which uses nonlinear transformations to extract the coupling relationship between spatial attention information and temporal motion information in the feature vector, distinguishing the intrinsic feature distribution differences between effective target features, distorted noise features, and false motion features. The output layer is a single fully connected output layer. The number of output nodes is set according to the confidence decision requirements, for example, the same as the number of frames of the LDR image. The output layer is equipped with an adaptive activation function to implement score constraints. For example, Sigmoid activation is used to map the output to the range of 0 to 1. The closer the value is to 1, the higher the confidence of the sample. The closer the value is to 0, the higher the probability that the sample is noise, false target or invalid motion sample.

[0103] In some embodiments, the structure of the MLP decision network includes: an input linear layer, a ReLU activation layer, a Dropout layer, a hidden linear layer, an output linear layer, and a Softmax normalization layer.

[0104] Taking three LDR images as an example, the MLP decision network outputs a confidence vector [w0, w1, w2], where the three weight values ​​w0, w1, and w2 correspond to the image blocks at the three frame positions, representing the probability that each frame image block is the best reference frame image block; Softmax normalization ensures that the sum of the three weight values ​​w0, w1, and w2 is 1; finally, the image block corresponding to the largest of w0, w1, and w2 is determined as the reference frame image.

[0105] S32. Determine the reference frame image block based on the confidence vector.

[0106] In this step, a reference frame image block is determined based on the confidence vector. For example, the image block with the largest weight value is determined as the reference frame image block.

[0107] The output of this step can be a frame index that identifies which LDR image frame the reference frame image block belongs to, which is used to determine which frame of multiple LDR images the reference frame image block originates from.

[0108] The output of this step can also be a sorting result of reordering multiple frame image blocks. For example, the HDR reconstruction module pre-sets images in a preset order among consecutive frames as reference frames. By default, the HDR reconstruction module uses the image / image block located in the preset order as the basis for feature alignment and fusion processing. In the process of determining the reference frame image block, this step reorders the multi-frame image blocks corresponding to the position, outputting a frame index sequence. The frame index sequence represents the arrangement order of the frame image blocks, and ensures that the image block used as the reference frame image block is located in the preset order, so that the subsequent HDR reconstruction module can directly use the reference frame image block as the basis for feature alignment and fusion processing.

[0109] In this embodiment of the disclosure, the frame index is a sequential identifier assigned to each frame of an LDR image in the original continuous LDR image sequence. For example, when the input LDR image sequence consists of 3 LDR images, the frame indices corresponding to each frame are 0, 1, and 2, respectively. The frame indices form a frame index sequence. The preset order refers to the preset arrangement position in the arrangement order of multiple frames / image blocks. For example, in the arrangement of 3 LDR images, the second position is used as the preset order.

[0110] If no image blocks are divided or only one image block is divided, the output of this step is the frame index of the LDR image as a reference frame, or the frame index sequence after reordering multiple LDR image frames.

[0111] When dividing the image into multiple blocks, this step outputs the frame index of the LDR image to which each spatial location reference frame image block belongs, or the frame index sequence after each spatial location image block is reordered.

[0112] S4. Using the reference frame image block as a reference, perform feature alignment and fusion processing based on the composite feature tensor of the low dynamic range image of each frame to generate a high dynamic range image.

[0113] In this embodiment, by extracting the associated feature representations of the quality features, motion features, and inter-frame relationship features of multiple frames / image blocks, a reference frame is determined based on the associated feature representations. Finally, feature alignment and fusion processing are performed based on the reference frame to generate a high dynamic range image. This achieves dynamic adaptive selection of the most suitable reference frame as the alignment and fusion benchmark based on the content quality, motion information, and inter-frame relationship of the input image. This is beneficial for selecting the clearest, most stable, and most information-stable image as the reference frame, providing a better benchmark for feature alignment and fusion processing, generating HDR images with richer details and fewer artifacts, thereby improving the quality of HDR image reconstruction.

[0114] In this embodiment of the disclosure, by extracting the associated feature representations of the quality features, motion features, and inter-frame relationship features of each frame image / image patch from multiple frames of images / image patches, and determining the reference frame based on the associated feature representations, the generalization ability and robustness in handling diverse scenes can be improved. The motion of objects in the selected reference frame is relatively small, which can reduce the motion displacement that needs to be compensated, reduce the difficulty of model learning alignment, and help to resolve artifact problems caused by large-scale motion compensation.

[0115] In some embodiments, in step S4, an HDR image is generated using an existing deep learning-based HDR reconstruction network. When determining the reference frame, simply reordering the reference frames so that the image blocks / image patches serving as reference frames are in the default order of the HDR reconstruction network allows the HDR reconstruction network to perform feature alignment and fusion processing based on the determined reference frames. The reference frame selection method in this embodiment can be implemented as a plug-and-play component, enabling seamless integration with existing HDR reconstruction networks without changing the core structure of the HDR reconstruction network, thus exhibiting good compatibility and scalability.

[0116] In some embodiments, step S32 includes the following step S321.

[0117] S321. Determine a frame index sequence based on the confidence vector. The frame index sequence represents the arrangement order of the image blocks in each frame corresponding to the position, wherein the image block serving as the reference frame image block corresponds to a preset order in the arrangement order.

[0118] In this step, the image blocks corresponding to each location are reordered based on the confidence vector. The determined order is represented by a sequence of frame indices composed of the frame indices of the LDR images to which each image block belongs. When the LDR image is divided into multiple image blocks, the image block sequences at each spatial location are reordered independently. Instead of a global rearrangement of the composite feature tensor of the LDR image, the composite feature sub-tensor corresponding to each image block at each spatial location is rearranged separately.

[0119] For example, taking three LDR images I0, I1, and I2 as an example, their frame indices are 0, 1, and 2 respectively; the HDR reconstruction module uses the second frame as the preset order of the reference frame; the result of reordering the image blocks corresponding to any spatial location is as follows: If the image block belonging to I1 has the largest weight value, the output frame index sequence after reordering is [0, 1, 2], and the image block belonging to I1 is located in the preset order (second frame). If the image block belonging to I0 has the largest weight value, then the output frame index sequence after reordering is [1, 0, 2], and the image block belonging to I0 is located in the preset order; If the image block belonging to I2 has the largest weight value, the output frame index sequence after reordering is [0, 2, 1], and the image block belonging to I2 is located in the preset order.

[0120] In the case of not dividing the image into blocks or dividing only one image block, step S321 is to reorder the multiple LDR images, determine the frame index sequence of the multiple LDR images, and place the LDR image used as the reference frame in a preset order.

[0121] When dividing the image into multiple blocks, the image blocks at different spatial locations are reordered in step S321. After reordering, in step S4, feature alignment and fusion can be performed on the image block sequences at each spatial location, and then the reconstructed HDR image blocks can be stitched together to obtain an HDR image; alternatively, the image blocks can be stitched together according to the frame index sequence of each spatial location, and then input into an HDR model for feature alignment and fusion to obtain a reconstructed HDR image. This disclosure does not impose any special limitations on this embodiment.

[0122] In some embodiments, step S4 includes the following steps S41 to S42.

[0123] S41. Based on the arrangement order of the image blocks in each frame corresponding to the position represented by the frame index sequence, the composite feature tensors of the low dynamic range images in each frame are summarized to obtain the tensor to be processed.

[0124] In this embodiment of the disclosure, the LDR images / image blocks are reordered without creating new data copies in physical storage. In this step, summarizing the composite feature tensors of each LDR image frame after reordering refers to the process of obtaining the composite feature tensor data of the initial LDR images from physical storage and converting it into input data for the HDR reconstruction module.

[0125] In some embodiments, if no image blocks are divided or only one image block is divided, step S41 includes retrieving composite feature tensor data of each LDR image frame from physical storage in chronological order according to the frame index sequence to obtain the tensor to be processed.

[0126] In some embodiments, when no image blocks are divided or only one image block is divided, step S41 includes retrieving the composite feature tensor data of each frame of LDR image from physical storage in sorted order according to the frame index sequence, and concatenating them in the channel dimension to obtain the tensor to be processed. For example, three frames of LDR images I0, I1, and I2 each include 3 channels, and their composite feature tensors x0, x1, and x2 each include 6 channels. The reordered frame index sequence is [1, 0, 2], and the tensor to be processed after concatenation is an 18-channel tensor.

[0127] In some embodiments, when dividing multiple image blocks, step S41 includes sequentially retrieving the composite feature sub-tensor data of each image block from physical storage according to the frame index sequence and the arrangement order of the image blocks at each spatial location, to obtain the tensor to be processed.

[0128] In some embodiments, when dividing multiple image blocks, step S41 includes sequentially retrieving the composite feature sub-tensor data of each image block from physical storage according to the frame index sequence and the arrangement order of the image blocks at each spatial location; corresponding to any spatial location, the composite feature sub-tensors of each image block are concatenated in the channel dimension to obtain the tensor to be processed.

[0129] In some embodiments, when multiple image blocks are divided, step S41 includes sequentially retrieving the composite feature sub-tensor data of each image block from physical storage according to the frame index sequence and the arrangement order of the image blocks at each spatial location, and concatenating them in the channel dimension and spatial dimension to obtain the tensor to be processed. For example, three frames of LDR images have a composite feature tensor with 6 channels. Each LDR image frame is divided into 2×3 blocks, a total of 6 image blocks. After retrieving the data according to the frame index sequence and concatenating them in the channel dimension and spatial dimension, a tensor to be processed with the shape [B, 18, H, W] is obtained, which includes 18 channels.

[0130] S42. Perform feature alignment and fusion processing on the tensor to be processed to generate the high dynamic range image.

[0131] In some embodiments, step S41 includes the following step S411.

[0132] S411. The composite feature sub-tensors of the image blocks corresponding to each position are concatenated in the channel dimension according to the arrangement order represented by the frame index sequence, and the composite feature sub-tensors of the image blocks at different positions are concatenated in the spatial dimension to obtain the tensor to be processed.

[0133] For example, three LDR images have a composite feature tensor with 6 channels. Each LDR image is divided into 2×3 blocks, which are 6 image blocks in total. Data is obtained according to the frame index sequence and then spliced ​​in the channel dimension and spatial dimension to obtain a tensor to be processed with the shape [B, 18, H, W], which includes 18 channels and whose length and width are consistent with the length and width of the LDR image.

[0134] In some embodiments, step S42 includes the following steps S421 to S424.

[0135] S421. The tensor to be processed is split in the channel dimension to obtain a multi-frame input feature tensor, wherein the input feature tensor corresponding to the preset frame number is used as a reference frame.

[0136] In this step, the input to the HDR model is the tensor to be processed, obtained by acquiring data from the frame index sequence and concatenating it in the channel and spatial dimensions. For example, in the case of three LDR images, the tensor to be processed has a shape represented as [B, 18, H, W]. In this step, the tensor to be processed is split into three input feature tensors with shapes represented as [B, 6, H, W] through chunks operations. In the sequence composed of the three input feature tensors, the order of the composite feature sub-tensors of the multi-frame image patches corresponding to any spatial location is consistent with the order represented by the frame index sequence. That is, the three input feature tensors as a whole represent the result of reordering the image patches at each spatial location.

[0137] S422. Using the input feature tensor as a reference frame as a basis, perform feature extraction and attention modulation on the input feature tensor of multiple frames to obtain a modulation feature map.

[0138] In this step, the three input feature tensors are convolved to extract initial feature maps. Then, the attention map of the initial feature map of the non-reference frame is calculated relative to the initial feature map of the reference frame, and attention modulation is applied to the non-reference frame to achieve implicit feature alignment and enhancement based on the reference frame.

[0139] In some embodiments, step S422 includes the following steps S422a to S422c.

[0140] S422a. Convolve the input feature tensor of multiple frames to extract the initial feature map of the input feature tensor.

[0141] S422b: Calculate the attention map between the initial feature map of the corresponding non-reference frame and the initial feature map of the corresponding reference frame.

[0142] S422c. The attention map is superimposed on the initial feature map of the corresponding reference frame and fused with the initial feature map of the corresponding non-reference frame to obtain the modulation feature map.

[0143] In steps S422b to S422c, a spatial attention mechanism is employed, combined with residual modulation. Specifically, the mechanism involves calculating the attention maps of the non-reference frame and the reference frame, then superimposing this attention map onto the initial feature map of the reference frame, and finally implicitly fusing it with the initial feature map of the non-reference frame.

[0144] In some embodiments, implicit feature alignment is achieved by combining spatial attention mechanism with residual modulation, including: First, the spatial attention map is applied as a residual to the initial feature map of the reference frame, adaptively enhancing the initial feature map of the reference frame. Specifically, this can be represented as: Fref_modulated =(1+attn)×F ref Where attn represents the spatial attention map, F ref This represents the initial feature map of the reference frame.

[0145] By using the residual modulation described above, regions with high feature correlation in the reference frame are enhanced, while regions with low correlation remain relatively stable, thereby improving the reference frame's ability to represent key regions.

[0146] Subsequently, the modulated reference frame features are fused with the corresponding non-reference frame features to generate alignment enhancement features.

[0147] In one embodiment, fusion can be performed by adding elements one by one: F aligned =F ref_modulated +F non_ref Among them, F aligned Represents the modulation feature map, F ref_modulated F represents the characteristics of the modulated reference frame. non_re This indicates the characteristics of a non-reference frame.

[0148] In some embodiments, feature fusion can also be performed by convolutional mapping after channel concatenation.

[0149] Through the above modulation and fusion process, the non-reference frame features can be implicitly aligned with the reference frame features in the feature space, while enhancing the semantic association region between the reference frame and the non-reference frame, thereby generating aligned and enhanced features containing complementary information from multiple frames.

[0150] S423. After splicing the multiple modulation feature maps, perform deep, multi-scale feature fusion to obtain a fused feature map.

[0151] In some embodiments, step S423 includes step S423a.

[0152] S423a. After splicing multiple modulation feature maps, input them into a hierarchical pyramid network. Deep-level, multi-scale feature fusion is performed through the wavelet transform module and Transformer module in the hierarchical pyramid network to obtain the fused feature map.

[0153] In this step, a hierarchical pyramid network (HPB) is used to perform deep, multi-scale feature fusion and information extraction. The hierarchical pyramid network includes a wavelet transform module and a Transformer module.

[0154] The pyramid network is a multi-scale hierarchical network architecture. The core idea is to construct a multi-layer image pyramid with progressively increasing resolution from coarse to fine by downsampling the input image / image patch. At different scale levels, it performs operations such as feature extraction, inter-frame alignment, brightness fusion, and artifact suppression. Then, it performs multi-scale information fusion and reconstruction from top to bottom. In the HDR reconstruction process, it can take into account both global large-scale displacement correction and local fine texture restoration.

[0155] The pyramid network in HDR reconstruction includes the following processing steps: Layered downsampling is used to construct a multi-scale pyramid by uniformly scaling multiple LDR images / image patches. All input frames generate corresponding levels synchronously. Processing is performed step by step from coarse to fine. Starting from the lowest resolution at the top level, and using the reference frame image patch as a benchmark, inter-frame feature alignment, motion compensation, and exposure information fusion are completed at each level. The offset and alignment parameters output from the coarse-scale level are passed to the next fine-scale level as prior constraints to narrow the matching search range of the fine-scale level and reduce the amount of computation. Upsampling and fusion are performed from top to bottom. The fine-scale level corrects local minor deformations, occlusions, and exposure distortions based on the alignment results of the coarse-scale level. Then, the fusion features extracted from each scale are upsampled and stitched together layer by layer to integrate the global motion information of the coarse layer with the texture details of the fine layer. The top-level reconstruction output is: after fusing all multi-scale information, the reconstructed HDR image features with a wider dynamic range, no ghosting, and clear details are obtained.

[0156] Using pyramid networks for HDR reconstruction offers the following advantages: It is compatible with large inter-frame motion; the coarse-scale layers of the pyramid can handle significant displacement caused by handheld shooting and lens shake, avoiding alignment failures. It suppresses different types of artifacts layer by layer; the coarse-scale layer corrects global ghosting, while the fine-scale layer repairs edge blurring, color overflow, and noise amplification. It balances computational cost and reconstruction accuracy; the coarse-scale downsampling results in a smaller data volume, significantly reducing the computational cost of frame matching, while the fine-scale layer only corrects details in a small area, balancing inference speed and imaging quality. It adapts to multi-exposure fusion scenarios; different layers can independently process overexposed highlight areas and underexposed shadow areas, extracting effective light and shadow information from LDR images layer by layer, alleviating the highlight truncation and shadow loss problems caused by single-scale fusion.

[0157] This disclosure does not impose any special limitations on the construction method of the pyramid network. For example, the following structures can be selected: multi-scale optical flow pyramid networks, such as PWC-Net and LiteFlowNet pyramid architectures, which focus on inter-frame motion estimation and image patch alignment, and are adapted to the motion feature extraction process described above; feature pyramid fusion networks, which adopt a U-shaped pyramid structure with multi-scale output from the encoder and step-by-step upsampling fusion from the decoder, focusing on HDR brightness and color reconstruction; and lightweight pyramid networks, which compress computation by reducing the number of pyramid levels, replacing ordinary convolutions with depthwise separable convolutions, and using channel clipping, and are adapted to real-time HDR reconstruction scenarios on mobile devices.

[0158] In this embodiment of the disclosure, the pyramid network combines wavelet transform and Transformer, and includes the following processing flow: (1) Feature splicing and joint representation First, the modulation feature maps are spliced ​​together along the channel dimension in a preset order to form a joint feature map, which is then input into a hierarchical pyramid network.

[0159] For example, taking three LDR images as an example, the stitching order is as follows: the modulation feature map of the reference frame is located between the modulation feature maps of the two non-reference frames.

[0160] The joint feature map contains alignment enhancement information of different exposure frames at different spatial locations, providing a basic input for subsequent multi-scale fusion.

[0161] (2) Recursive wavelet decomposition The pyramid network performs recursive layer-by-layer wavelet decomposition on the input joint feature map to achieve multi-scale feature extraction.

[0162] In some embodiments, Haar wavelets are used as the basis functions for wavelet transform. The input joint feature map is decomposed into four frequency sub-bands using Discrete Wavelet Transform (DWT): one low-frequency approximation sub-band (LL) and three high-frequency detail sub-bands (LH, HL, HH). The LL sub-band contains low-frequency contour information of the joint feature map, while the LH, HL, and HH sub-bands contain high-frequency detail information in the horizontal, vertical, and diagonal directions, respectively.

[0163] In each processing stage: the high-frequency subbands LH, HL, and HH are temporarily buffered for subsequent reconstruction and fusion; the low-frequency subband LL serves as the input for the next level and continues to perform the same wavelet decomposition operation (i.e., recursive processing).

[0164] It should be noted that each wavelet transform has a decomposition layer of 1, while the multi-scale characteristic is achieved through the recursive hierarchical structure of HPB.

[0165] As an example, when the number of pyramid levels is 3, the processing flow may include: Level 1: Joint feature map F of the input in Perform Haar wavelet decomposition to obtain LL1, LH1, HL1, and HH1. Send LL1 to Level 2 and buffer LH1, HL1, and HH1. Level 2: Continue wavelet decomposition on LL1 to obtain LL2, LH2, HL2, and HH2. Send LL2 to Level 3 and buffer LH2, HL2, and HH2. Level 3: Use LL2 as the deepest input for feature extraction.

[0166] (3) Global dependency modeling of the deepest low-frequency subband At the deepest level of the pyramid, global dependency modeling is performed on the low-frequency subband features.

[0167] Because the deepest low-frequency features have small spatial resolution and rich semantic information, they are suitable for global context modeling.

[0168] Specifically, the deepest low-frequency features are first transformed through convolutional layers and then processed by the Transformer encoder. The Transformer encoder uses a self-attention mechanism to capture long-range dependencies between spatial locations, thereby obtaining global contextual information.

[0169] In this embodiment, the Transformer encoder may include: two encoder layers; each layer contains a multi-head attention sublayer and a feedforward network sublayer; the number of attention heads is 4; and the feature dimension d model The preset dimension; the feedforward network dimension is dim. feedforward The value is 512.

[0170] In one embodiment, the multiple sub-band features after wavelet decomposition are concatenated along the channel dimension and adjusted to a fixed feature dimension d through a linear projection layer. model After being reshaped into a feature sequence, it is input into the Transformer encoder.

[0171] After processing by the Transformer, the output is enhanced low-frequency features. These enhanced features are then passed through convolutional layers (e.g., conv). tail The process involves upsampling to restore spatial resolution for use in subsequent layer fusion.

[0172] In this embodiment, the upsampling method can be bilinear interpolation.

[0173] (4) Step-by-step reconstruction and integration Based on inverse wavelet transform (IWT), the low-frequency features enhanced by the Transformer are gradually reconstructed and fused with the high-frequency sub-band features of each level of buffer to restore spatial resolution step by step.

[0174] The reconstruction and integration process is executed in a hierarchical order from deep to shallow.

[0175] For example, with levels=3: Level 3 → Level 2 Fusion: The enhanced low-frequency features upsampled from Level 3 are combined with the LH2, HL2, and HH2 high-frequency subbands cached from Level 2 and fused using inverse wavelet transform (IWT) to reconstruct a higher-resolution feature map (shape [B, C, H / 2, W / 2]). This feature map can also be optionally added to and fused with LL1 features processed by bypass convolution.

[0176] Level 2 → Level 1 Fusion: The fused features are further upsampled by a factor of 2 and fused with the LH1, HL1, and HH1 high-frequency subbands cached in Level 1 using inverse wavelet transform to reconstruct a feature map restored to its original spatial resolution (shape [B, C, H, W]). This feature map can also optionally be fused with the original input F. in Features processed by short-connection convolution are then subjected to residual connections.

[0177] In the process of cross-level fusion, a cross-resolution fusion strategy can also be adopted to fuse the reconstructed feature map with other intermediate features in the network (such as features processed by side-channel convolution conv_tail). The fusion method can be element-wise addition or channel concatenation followed by convolution.

[0178] (5) Output multi-scale fused feature maps After the recursive wavelet decomposition, Transformer global modeling, and hierarchical reconstruction and fusion described above, the hierarchical pyramid network finally outputs a joint feature map F with the input. in Multi-scale fusion feature maps F with the same spatial resolution and potentially refined channel count out .

[0179] This multi-scale fusion feature map aggregates multi-scale information from coarse to fine—from the deepest global semantic information to the shallowest fine-grained edge texture information—and contains rich global context, providing powerful feature representation for subsequent HDR image reconstruction.

[0180] Through the processing flow, the embodiments of this disclosure realize multi-scale feature fusion processing based on wavelet decomposition and Transformer global modeling, which helps to simultaneously preserve the global brightness structure and local detail texture of the image, thereby improving the quality of HDR image reconstruction.

[0181] S424. The fused feature map is residually connected to the initial feature map of the input feature tensor, which serves as the reference frame, and the high dynamic range image is reconstructed through convolution processing.

[0182] In this step, the fused feature map is fused with the initial feature map of the reference frame, and the final HDR image is output through convolutional mapping. The specific sub-steps are explained in detail below.

[0183] (1) Channel adaptation of the initial feature map of the reference frame The initial feature map of the reference frame is subjected to channel adaptation processing to match the number of channels with the multi-scale fused feature map.

[0184] The initial feature map F of the reference frame was obtained through convolutional feature extraction. ref Its number of channels is typically a preset feature dimension (e.g., 64 channels). Multi-scale fusion feature map F out The number of channels may be related to F ref They are different. In order to enable residual fusion between the two, channel adaptation is required first.

[0185] In some embodiments, channel adaptation is achieved through 1×1 convolution. By using 1×1 convolution, the number of channels in the initial feature map of the reference frame is adjusted to match the number of channels in the multi-scale fused feature map while maintaining the spatial resolution.

[0186] (2) Residual connection The initial feature map of the reference frame after channel adaptation is residually connected with the multi-scale fused feature map.

[0187] The initial feature map of the reference frame after channel adaptation is added element-wise to the multi-scale fused feature map to obtain the reconstructed features: F reconstruct = F ref_adapted + F out .

[0188] Among them, the multi-scale fusion feature map F out Contains multi-scale complementary information from different exposure frames; reference frame initial feature map F ref_adapted The basic structure and details of the reference frame are preserved. Using the residual fusion method described above, the original detailed features of the reference frame can be preserved while fusing multi-exposure information.

[0189] (3) Convolutional mapping and HDR image output The reconstructed features after residual connection are input into the convolutional mapping module to generate the final HDR image.

[0190] In some embodiments, the convolution mapping module includes two convolutional layers: The first convolutional layer uses a 3×3 convolutional kernel with a stride of 1 and padding of 1. After the convolution, a ReLU activation function is applied to further extract and fuse reconstructed features. The second convolutional layer also uses a 3×3 kernel with a stride of 1 and padding of 1. It is used to map features to the HDR image space and output the final HDR image result.

[0191] In one embodiment, the final output is a 3-channel linear HDR image.

[0192] It should be noted that in this embodiment, the spatial resolution of the input feature map is consistent with the spatial resolution of the target HDR image. Therefore, the convolutional mapping module does not contain an explicit upsampling layer and directly performs pixel-level regression output.

[0193] In some embodiments, when the input feature resolution is inconsistent with the target output resolution, an upsampling operation can be added to the convolutional mapping module, such as bilinear interpolation upsampling, deconvolution, or PixelShuffle.

[0194] After the above convolution mapping, the final HDR image is output. This HDR image can simultaneously retain the brightness details of both bright and dark areas and can be used for subsequent HDR display or tone mapping processing.

[0195] The embodiments disclosed herein implement HDR image reconstruction processing based on multi-scale fusion features and initial features of the reference frame, thereby obtaining a high dynamic range image containing multi-exposure complementary information.

[0196] Figure 2 This is a schematic diagram comparing three low dynamic range images with different exposure values ​​and the output high dynamic range image in an embodiment of this disclosure. Figure 2 As shown, LDR input images with different exposures in the same scene are displayed respectively. The upper left corner is underexposed, the upper right corner is normal, the lower left corner is overexposed, and the lower right corner is the HDR image generated by the method of this disclosure.

[0197] from Figure 2As can be seen, the method disclosed herein can effectively fuse complementary information from three LDR images: the underexposed image retains details in the highlight areas, the overexposed image retains texture information in the shadow areas, and the normally exposed image provides midtone information. Through the aforementioned adaptive reference frame selection, feature alignment, and multi-scale fusion processing, the output HDR image simultaneously retains details in both highlight and shadow areas, and exhibits no obvious motion ghosting artifacts, verifying the robustness and effectiveness of the method disclosed herein in dynamic scenes.

[0198] This disclosure provides an image generation model for generating HDR images based on multiple LDR images with different exposure times. Figure 3 This is a schematic diagram of the structure of the image generation model.

[0199] like Figure 3 As shown, the image generation model includes an input module, a reference frame selection module, and an HDR reconstruction module.

[0200] The input module is used to preprocess multiple frames of LDR images with different exposure times in the same scene to obtain the composite feature tensor of each frame of LDR image.

[0201] The reference frame selection module is used to extract associated feature representations based on the composite feature subtensors of the image blocks corresponding to the positions in each LDR image frame, and to determine the reference frame image blocks for each image block corresponding to the position based on the associated feature representations.

[0202] The HDR reconstruction module is used to generate HDR images by performing feature alignment and fusion processing based on the composite feature tensor of each LDR image frame, using the reference frame image block as a reference.

[0203] like Figure 3 As shown, the reference frame selection module consists of a feature extraction network, a multi-head attention module, a motion estimation network, and an MLP decision network.

[0204] The feature extraction network is used to perform convolutional feature extraction on the nonlinear radiative feature channels of each composite feature sub-tensor to obtain the global feature vector of each frame image block.

[0205] The multi-head attention module takes the global feature vector as input and obtains an attention-weighted feature vector that fuses quality features and inter-frame relationship features.

[0206] The motion estimation network is used to estimate motion features based on the nonlinear radiation feature channels of each composite feature sub-tensor, extract motion features, and obtain motion feature vectors.

[0207] The MLP decision network is used to concatenate the attention-weighted feature vector and the motion feature vector to make a confidence decision, resulting in a confidence vector. The weight values ​​that make up the confidence vector represent the confidence (weight) of the corresponding image patch as a reference frame image patch.

[0208] based on Figure 3 The image generation process of the image generation model shown is as follows: Figure 4 As shown.

[0209] Taking three LDR images as an example, the input consists of LDR image sequences I0, I1, and I2 with different exposure times. After preprocessing such as inverse CRF transform and channel stitching, the input is processed by the reference frame selection module for feature extraction, attention calculation, and motion estimation to select the optimal reference frame. Based on the confidence vector, if I0 has the highest weight, it is reordered as I1, I0, I2; if I1 has the highest weight, it is reordered as I0, I1, I2; and if I2 has the highest weight, it is reordered as I0, I2, I1. Then, the HDR reconstruction module performs feature alignment, fusion, and reconstruction, including shallow feature extraction, attention modulation and alignment using the optimal reference frame as an anchor, multi-scale feature fusion, and HDR image reconstruction decoding. Finally, a high-quality HDR image is output.

[0210] In one embodiment, combined Figure 4 ,like Figure 3 The image generation model shown can be trained and optimized in the following end-to-end manner.

[0211] 1. Training data acquisition Acquire training data, which includes training samples corresponding to multiple scenes. Each training sample includes multiple frames of low dynamic range images with different exposures in the same scene, as well as their corresponding real high dynamic range images.

[0212] In one embodiment, each training sample includes three low dynamic range (LVR) images, corresponding to underexposed, medium-exposed, and overexposed images, respectively, and the corresponding real high dynamic range (HVR) images as supervision labels. The training data can be derived from publicly available HVR image datasets, such as the Kalantari dataset.

[0213] 2. Forward propagation and prediction The low dynamic range images with different exposures of the multiple frames are input into a high dynamic range image generation model constructed using a high dynamic range image generation method to obtain a predicted high dynamic range image.

[0214] Specifically, the multi-frame low dynamic range images undergo the following processes in sequence: linear radiometric spatial transformation and input enhancement construction; adaptive reference frame selection; block-level order rearrangement; feature modulation and alignment; hierarchical pyramid network multi-scale fusion; and HDR image reconstruction; finally, the output is a predicted high dynamic range image H. pred .

[0215] 3. Calculation of joint loss function Based on predicting high dynamic range image H pred Compared with true high dynamic range images H gt Calculate the joint loss function.

[0216] In this embodiment, the joint loss function includes: a pixel-level reconstruction loss based on the μ-Law compressed domain; and a perceptual loss based on a pre-trained feature extraction network. By jointly constraining both, pixel-level reconstruction accuracy and visual perception quality can be simultaneously achieved.

[0217] (1) Pixel-level reconstruction loss Pixel-level reconstruction loss is calculated in the μ-Law compression domain by the L1 distance between the predicted high dynamic range (HDR) image and the true HDR image. The formula for calculating the L1 reconstruction loss in the μ-Law compression domain is not unique. μ-Law compression is a generalized formula framework, the core of which lies in first performing logarithmic dynamic range compression on the HDR data, thereby reducing the impact of extremely high brightness values ​​on the loss calculation.

[0218] For example, pixel-level reconstruction loss can be expressed as: The μ-Law compression function can be expressed as: Here, μ is a compression parameter; the value of μ can be set according to the distribution characteristics of HDR data, such as 255 or 5000. In different embodiments, the μ parameter can also be adjusted according to the importance of the target brightness range or a weighting mechanism can be introduced.

[0219] (2) Perceived loss Perceptual loss is used to constrain the consistency between the predicted HDR image and the real HDR image in the high-level semantic feature space.

[0220] Specifically, predicting HDR image H pred Compared to real HDR images H gt Input a pre-trained visual feature extraction network (e.g., VGG network), extract feature maps of the corresponding layers, and calculate the distance between the feature maps.

[0221] For example, perceptual loss can be expressed as: Among them, F i This represents the feature map output by the i-th layer of the pre-trained network; This represents the weight coefficient of the corresponding layer.

[0222] In different embodiments, features of different layers can be selected according to the target constraint content: shallower features are used to constrain texture details; deeper features are used to constrain overall structural information.

[0223] (3) Total loss function The total loss function is obtained by weighting the pixel-level reconstruction loss and the perceptual loss.

[0224] In one embodiment, the total loss function can be expressed as: Where λ is a hyperparameter (e.g., 0.01 or 0.1) used to balance pixel-level precision and visual perception quality.

[0225] It should be noted that both pixel-level reconstruction loss and perceptual loss are calculated based on the same pair of images, i.e., the predicted HDR image H. pred Compared to real HDR images H gt The pixel-level reconstruction loss directly calculates the pixel-by-pixel difference between the two in the μ-Law compression domain; the perceptual loss first extracts high-level features through a pre-trained feature extraction network before comparison.

[0226] 4. Backpropagation and parameter update Based on the joint loss function, at least some of the trainable parameters in the high dynamic range image generation model are updated through backpropagation.

[0227] In this embodiment, the high dynamic range image generation model includes: a reference frame selection network; and a master HDR reconstruction network.

[0228] The reference frame selection network includes a feature extraction network, a multi-head attention module, a motion estimation network, and an MLP decision network; the main HDR reconstruction network includes a hierarchical pyramid network and an HDR reconstruction module.

[0229] In one embodiment, the system employs an end-to-end joint training approach to uniformly optimize the parameters of the aforementioned modules.

[0230] This disclosure also supports the following training modes: Mode A: Joint Training Mode This mode is the default state. The parameters of both the reference frame selection network and the main HDR reconstruction network participate in the backpropagation update, thereby achieving joint optimization of reference frame selection and HDR reconstruction.

[0231] Mode B: Freeze reference frame selection network mode During training, the parameter gradients of the reference frame selection network are frozen, preventing them from participating in parameter updates; only the parameters of the main HDR reconstruction network are updated. This mode can be used to optimize the main HDR reconstruction network separately under a fixed reference frame selection strategy. This mode can be used to verify the performance lower bound of the main network without adaptive reference frame selection interference, or to accelerate the convergence of the main network.

[0232] Through the above steps, the embodiments of this disclosure achieve training optimization of the high dynamic range image generation model, thereby enabling the model to simultaneously achieve pixel-level reconstruction accuracy and visual perception quality of HDR images.

[0233] Figure 5 This is a schematic diagram of the composition of an electronic device according to an embodiment of the present disclosure.

[0234] Secondly, referring to Figure 5 This disclosure provides an electronic device comprising: One or more processors 701; The memory 702 stores one or more programs that, when executed by one or more processors, enable the one or more processors to implement the high dynamic range image generation method described in the first aspect. One or more I / O interfaces 703 are connected between the processor and the memory and configured to enable information exchange between the processor and the memory.

[0235] Among them, processor 701 is a device with data processing capabilities, including but not limited to central processing unit (CPU); memory 702 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); I / O interface (read-write interface) 703 is connected between processor 701 and memory 702, and can realize information interaction between processor 701 and memory 702, including but not limited to data bus (Bus).

[0236] In some embodiments, the processor 701, memory 702, and I / O interface 703 are interconnected via bus 704, and thus connected to other components of the computing device.

[0237] This disclosure provides a computer-readable storage medium storing a computer program thereon, characterized in that the computer program, when executed by a processor, implements the image generation method described in any one of the above claims.

[0238] This disclosure also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implement any of the above-described image generation methods.

[0239] The high dynamic range image generation method based on adaptive reference frame selection and hierarchical feature fusion proposed in this disclosure has the following beneficial effects: (1) Improve the reconstruction quality of high dynamic range images By adaptively selecting reference frames based on the input image content, a better reference benchmark is provided for subsequent feature alignment and fusion, thereby generating high dynamic range images with richer details and fewer artifacts. Especially in dynamic scenes and scenes with large exposure differences, it can effectively reduce distortions such as motion ghosting and improve image reconstruction quality.

[0240] (2) Enhance the robustness of the model to complex scenarios This disclosure breaks through the limitation of traditional methods that use the intermediate exposure frame as the reference frame, enabling the model to dynamically select the reference frame according to the scene content. This improves the model's adaptability to complex scenes such as motion blur, extreme exposure differences, and image noise, and enhances the model's generalization performance and robustness.

[0241] (3) Improve the alignment effect of the motion area For scenarios with moving targets, this disclosure reduces the amount of motion compensation required for cross-frame feature alignment by selecting images with smaller motion displacements as reference frames, thereby reducing the difficulty for the network to learn alignment relationships, alleviating artifact problems caused by large-scale motion compensation failures, and improving the reconstruction stability of moving regions.

[0242] (4) Possesses good modularity and compatibility The adaptive reference frame selection module proposed in this disclosure is a lightweight, plug-and-play architecture that is compatible with existing deep learning-based high dynamic range reconstruction frameworks. It can replace the original fixed reference frame input mechanism without changing the core structure of the main reconstruction network, thereby achieving performance improvements and exhibiting good backward compatibility and scalability.

[0243] (5) Achieve end-to-end joint optimization The reference frame selection network and the main reconstruction network in this disclosure can be jointly trained end-to-end, enabling the reference frame selection process to be co-optimized with the final high dynamic range image generation target, thereby learning a reference frame selection strategy that is more suitable for high dynamic range image reconstruction tasks and improving the overall reconstruction performance.

[0244] (6) Enhance multi-scale information fusion capabilities By constructing a hierarchical pyramid network combining wavelet transform and Transformer, low-frequency brightness information and high-frequency texture details in images with different exposures are fused at multiple scales. Wavelet transform effectively preserves edge and texture details, while Transformer models global contextual dependencies, thereby improving the fusion effect between different exposure information and enhancing the detail fidelity and visual quality of the generated image.

[0245] In summary, this disclosure effectively solves the problems of fixed reference frames, obvious motion artifacts, and insufficient robustness in complex scenes in existing high dynamic range image generation methods by introducing an adaptive reference frame selection mechanism and a hierarchical multi-scale feature fusion strategy, and provides a more intelligent, higher quality, and more stable high dynamic range image generation scheme.

[0246] To enable those skilled in the art to more clearly understand the technical solutions provided by the embodiments of this disclosure, the technical solutions provided by the embodiments of this disclosure will be described in detail below through specific embodiments: Example 1 The technical solution of this disclosure will be further illustrated below with specific examples. It should be noted that the following embodiments are only used to illustrate the technical solution of this disclosure and are not intended to limit the scope of protection of this disclosure. This example uses the Kalanitari dataset as an example to describe in detail the training and implementation process of the high dynamic range image generation model in the embodiments of this disclosure.

[0247] 1) Data preparation and preprocessing First, we acquire and organize publicly available high dynamic range (HDR) imaging datasets, such as the Kalanitari dataset. Each scene in this dataset can include three HDR images with different exposures: an underexposed image I0, a normally exposed image I1, and an overexposed image I2, along with a corresponding ground truth HDR image H. gt As a supervisory label.

[0248] Run a data preprocessing program to crop the training images. For example, you can set the image patch size to 256×256 pixels and the stride to 128 pixels to generate a large number of fixed-size training samples, thereby increasing the diversity of the training data.

[0249] During the data loading phase, a preprocessing procedure is performed on each input low dynamic range image: first, the image is converted from the sRGB color space to the linear radiometric space through inverse gamma correction; then, the obtained linear radiometric image is stitched with the original low dynamic range image in the channel dimension, thereby generating an enhanced feature tensor containing 6 channels for each frame image, providing input data that integrates radiometric and texture information for subsequent model processing.

[0250] 2) Model Construction and Initialization Instantiate the HDR_HPB model in this embodiment of the disclosure. During model initialization, the following parameters can be set: input channel number nChannel=6, corresponding to the preprocessed enhanced features; internal feature channel number nFeat=64; wavelet transform type wave='haar'; and adaptive reference frame selection function can be enabled.

[0251] A reference frame selection subnetwork can be created and integrated within the model. In one embodiment, this subnetwork includes: A two-layer convolutional feature extractor is used to generate a global feature vector; The multi-head attention module is used to model the relationships between multiple frames; A lightweight motion estimation network is used to extract inter-frame motion difference features; A multilayer perceptron (MLP) decision network is used to output reference frame selection weights.

[0252] For example, the convolutional feature extractor can output 128-channel features, which are then flattened to 8×8 through adaptive average pooling, thereby generating an 8192-dimensional global feature vector; the embedding dimension of the multi-head attention module can be set to 8192, and the number of attention heads can be set to 4; the motion estimation network can output a 16-dimensional motion feature vector; and the MLP decision network finally outputs the reference frame weight vector corresponding to the three frames of images through the Softmax layer.

[0253] 3) Training configuration The Adam optimizer can be used during training, with the learning rate set to 0.0002.

[0254] The loss function can be a weighted combination of joint reconstruction loss and perceptual loss, i.e., a weighted sum of L1Mu loss and VGG perceptual loss. The L1Mu loss calculates the L1 distance between the predicted image and the ground truth image in the μ-Law compression domain; the perceptual loss uses a pre-trained VGG-19 network to extract high-level semantic features and perform distance measurement.

[0255] In terms of training strategies, for example, the total number of training rounds can be set to 100, the batch size to 4, and the adaptive reference frame selection function can be enabled.

[0256] Optionally, the parameters of the reference frame selection sub-network are frozen in the initial stage of training, and only the main HDR reconstruction network is trained; after the main network has initially converged, the reference frame selection sub-network is unfrozen and end-to-end joint training is performed, thereby improving training stability and improving the final reconstruction performance.

[0257] 4) Training process During training, preprocessed image patches are input into the model in batches.

[0258] For each training batch, the reference frame selection subnetwork first receives three low dynamic range image patches and outputs a corresponding 3D reference frame weight vector. Subsequently, the system rearranges the order of the three 6-channel input tensors according to this weight vector, so that the selected best reference frame is located at a preset reference position.

[0259] The rearranged input data is fed into the main HDR reconstruction network and goes through the following processes in sequence: Preliminary feature extraction; Feature modulation and alignment based on spatial attention mechanism; Multi-scale deep feature fusion is performed using the hierarchical pyramid network HPB. The fused features are residually concatenated with the initial features of the reference frame; Predict HDR images by reconstructing the output using convolutional layers.

[0260] The model calculates the joint loss between the predicted HDR image and the real HDR image, and uses the backpropagation algorithm to iteratively update all trainable parameters.

[0261] 5) Testing and Evaluation After the model is trained, its performance is evaluated on an independent test set.

[0262] During the test, the test program can be called to slide the high-resolution test image into multiple image blocks, and input them into the model to perform forward inference; then all the output image blocks are re-stitched into a complete HDR image according to their corresponding spatial positions.

[0263] Evaluation indicators may include: Peak signal-to-noise ratio (PSNR) in the μ-Law compressed domain; SSIM_mu is a structural similarity index in the μ-Law compression domain.

[0264] Experimental results show that the embodiments of this disclosure can achieve high PSNR_mu and SSIM_mu metrics on standard test sets. For example, on the Kalantari dataset, the average PSNR_mu is higher than 44dB and the SSIM_mu is higher than 0.99, thus verifying the effectiveness of the embodiments of this disclosure in high-fidelity HDR reconstruction and visual quality preservation.

[0265] 6) Model Application Once trained, the model can be applied to real-world high dynamic range image generation scenarios.

[0266] When applied, only three low dynamic range images with different exposures acquired in the same scene need to be input. The model can automatically perform reference frame selection, local feature alignment, multi-scale feature fusion, and HDR image reconstruction processes, and finally output a high dynamic range image with rich detail information and effective suppression of motion ghosting.

[0267] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0268] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. An image generation method, characterized in that, include: Obtain a composite feature tensor of low dynamic range images with different exposure times in multiple frames of the same scene, wherein the composite feature tensor includes a nonlinear radiative feature channel and a linear radiative feature channel. Based on the composite feature subtensor of the image patch corresponding to the position in each frame of the low dynamic range image, the associated feature representation is extracted. The associated feature representation characterizes the quality features, motion features and inter-frame relationship features of the image patch in each frame. The reference frame image blocks for each frame corresponding to the position are determined based on the associated feature representation; Based on the reference frame image block, feature alignment and fusion processing are performed according to the composite feature tensor of the low dynamic range image of each frame to generate a high dynamic range image.

2. The image generation method according to claim 1, characterized in that, The step of extracting associated feature representations based on the composite feature sub-tensors of image patches corresponding to positions in each frame of the low dynamic range image includes: Convolution feature extraction is performed on the nonlinear radiative feature channels of each composite feature sub-tensor to obtain the global feature vector of each image block in each frame; The global feature vector is input into the multi-head attention module to obtain an attention-weighted feature vector that fuses quality features and inter-frame relationship features; Motion feature estimation is performed based on the nonlinear radiation feature channels of each composite feature sub-tensor, and motion features are extracted to obtain motion feature vectors; The associated feature representation includes the attention-weighted feature vector and the motion feature vector.

3. The image generation method according to claim 2, characterized in that, The step of inputting the global feature vector into the multi-head attention module to obtain an attention-weighted feature vector that fuses quality features and inter-frame relationship features includes: The global feature vectors of the image blocks in each frame are stacked to obtain a stacked feature tensor; The stacked feature tensor is input into the multi-head attention module, and self-attention calculation is performed according to the preset number of attention heads to obtain the attention-weighted feature vector.

4. The image generation method according to claim 2, characterized in that, The step of estimating motion features based on the nonlinear radiative feature channels of the image blocks in each frame, extracting motion features, and obtaining a motion feature vector includes: The nonlinear radiation feature channels of any two adjacent frame image blocks are concatenated and input into the motion estimation network to extract the difference features between the image blocks of the adjacent frames, thereby obtaining the difference feature vector, wherein the difference feature vector represents the motion difference information between two adjacent frames. The motion feature vector is obtained by concatenating multiple differential feature vectors.

5. The image generation method according to any one of claims 2 to 4, characterized in that, The step of determining the reference frame image block for each frame corresponding to the position based on the associated feature representation includes: The attention-weighted feature vector and the motion feature vector are concatenated and then a confidence decision is made to obtain a confidence vector. The confidence vector is composed of weight values ​​that correspond one-to-one with the image blocks in multiple frames. The weight values ​​represent the confidence of the corresponding image block as a reference frame image block. The reference frame image block is determined based on the confidence vector.

6. The image generation method according to claim 5, characterized in that, Determining the reference frame image block based on the confidence vector includes: The frame index sequence is determined based on the confidence vector. The frame index sequence represents the arrangement order of the image blocks in each frame corresponding to the position, wherein the image block serving as the reference frame image block corresponds to a preset order in the arrangement order.

7. The image generation method according to claim 6, characterized in that, The step of generating a high dynamic range image by using the reference frame image block as a reference and performing feature alignment and fusion processing based on the composite feature tensor of the low dynamic range image of each frame includes: The composite feature tensors of each frame of the low dynamic range image are summarized according to the frame index sequence to obtain the tensor to be processed. The tensor to be processed is subjected to feature alignment and fusion processing to generate the high dynamic range image.

8. The image generation method according to claim 7, characterized in that, The step of summarizing the composite feature tensors of each frame of the low dynamic range image according to the frame index sequence to obtain the tensor to be processed includes: The composite feature sub-tensors of the image blocks corresponding to each position are concatenated in the channel dimension according to the arrangement order represented by the frame index sequence, and the composite feature sub-tensors of image blocks at different positions are concatenated in the spatial dimension to obtain the tensor to be processed.

9. The image generation method according to claim 8, characterized in that, The step of performing feature alignment and fusion processing on the tensor to be processed to generate the high dynamic range image includes: The tensor to be processed is split along the channel dimension to obtain multiple frames of input feature tensors, wherein the input feature tensor corresponding to the preset order is used as a reference frame. Using the input feature tensor as a reference frame as a basis, feature extraction and attention modulation are performed on the input feature tensor of multiple frames to obtain a modulation feature map; After splicing multiple modulation feature maps, deep, multi-scale feature fusion is performed to obtain a fused feature map. The fused feature map is residually concatenated with the initial feature map of the input feature tensor, which serves as the reference frame, and the high dynamic range image is reconstructed through convolution processing.

10. The image generation method according to claim 9, characterized in that, The step of using the input feature tensor as a reference frame as a basis to perform feature extraction and attention modulation on the input feature tensors of multiple frames to obtain a modulated feature map includes: Convolve the input feature tensor across multiple frames to extract the initial feature map of the input feature tensor; Calculate the attention map between the initial feature map of the corresponding non-reference frame and the initial feature map of the corresponding reference frame; The attention map is superimposed on the initial feature map of the corresponding reference frame and fused with the initial feature map of the corresponding non-reference frame to obtain the modulation feature map.

11. The image generation method according to claim 9, characterized in that, The step of concatenating multiple modulation feature maps and then performing deep, multi-scale feature fusion to obtain a fused feature map includes: The multiple modulation feature maps are concatenated and input into a hierarchical pyramid network. Deep, multi-scale feature fusion is performed through the wavelet transform module and the Transformer module in the hierarchical pyramid network to obtain the fused feature map.

12. The image generation method according to any one of claims 1 to 4, characterized in that, Before extracting associated feature representations based on the composite feature sub-tensors of image patches corresponding to the low dynamic range image positions in each frame, the image generation method further includes: The low dynamic range image is divided into multiple image blocks.

13. The image generation method according to any one of claims 1 to 4, characterized in that, The acquisition of the composite feature tensor of multiple frames of low dynamic range images with different exposure times in the same scene includes: The low dynamic range image is converted to linear radiometric space to obtain a linear radiometric image; The linear radiometric image and the low dynamic range image are concatenated along the channel dimension to obtain the composite feature tensor, wherein the nonlinear radiometric feature channel of the composite feature tensor corresponds to the low dynamic range image, and the linear radiometric feature channel of the composite feature tensor corresponds to the linear radiometric image.

14. An electronic device, characterized in that, The electronic device includes: One or more processors; A memory having stored one or more programs that, when executed by one or more processors, cause the one or more processors to implement the image generation method according to any one of claims 1 to 13.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image generation method according to any one of claims 1 to 13.

Images