High dynamic range image reconstruction methods, apparatuses, electronic devices and storage media

Abstract

This application provides a high dynamic range image reconstruction method, apparatus, electronic device, and storage medium. The method includes: globally aligning a linear domain long-exposure image and a linear domain short-exposure image to obtain globally aligned linear domain long-exposure images and linear domain short-exposure images; extracting feature images from the globally aligned linear domain long-exposure images and linear domain short-exposure images to guide the alignment of the linear domain target feature map with reference to the linear domain reference feature map to obtain a linear domain short-exposure target frame feature map; constructing a long-range dependency between the linear domain short-exposure target frame feature map and the linear domain reference feature map to obtain a fused linear domain short-exposure attention map; performing decoding processing based on the linear domain short-exposure attention map and the linear domain reference feature map, and passing the decoded image through an output layer to obtain a reconstructed high dynamic range image. This application solves the ghosting problem caused by large global motion.

Description

Technical Field

[0001] This application relates to the field of image processing, and more particularly to a high dynamic range image reconstruction method, apparatus, electronic device, and storage medium. Background Technology

[0002] Because the range of illuminance variation in natural scenes is usually large, the illuminance ratio can reach 10. 6 1. The human eye is accustomed to perceiving such high dynamic range (HDR) in natural scenes. However, traditional cameras need to set the global exposure time for the entire image and compress its dynamic range to 8-10 bits, resulting in overexposed or completely black areas in the low dynamic range (LDR) images.

[0003] High Dynamic Range (HDR) technology aims to reconstruct images with a higher dynamic range, containing more information from the scene and providing a better visual experience for the human eye. HDR technology has been widely used in fields such as film, medical imaging, and autonomous driving. Traditional HDR technology directly captures high dynamic range images using customized image sensors, which requires higher design and processing costs, making it difficult to promote among ordinary consumers.

[0004] Current high dynamic range (HDR) imaging techniques reconstruct a single HDR image by fusing LDR images taken at different exposure times within the same scene. Traditional cameras capture multiple exposure LDR images containing diverse information about the scene; fusion methods can reconstruct a single HDR image from these multiple LDR frames. This method eliminates the need for custom image sensors, effectively addressing cost issues. However, the resulting HDR images still suffer from problems such as ghosting. Summary of the Invention

[0005] In view of this, embodiments of this application provide a high dynamic range image reconstruction method, apparatus, electronic device, and storage medium, which can solve all or part of the above-mentioned technical problems.

[0006] According to a first aspect of the embodiments of this application, a high dynamic range image reconstruction method is provided. The method includes: globally aligning a linear domain long-exposure image and a linear domain short-exposure image to obtain globally aligned linear domain long-exposure images and linear domain short-exposure images; extracting feature images from the globally aligned linear domain long-exposure images and linear domain short-exposure images, guiding a linear domain target feature map to align with a linear domain reference feature map to obtain a linear domain short-exposure target frame feature map; constructing a long-range dependency between the linear domain short-exposure target frame feature map and the linear domain reference feature map to obtain a fused linear domain short-exposure attention map; performing decoding processing based on the linear domain short-exposure attention map and the linear domain reference feature map, and passing the decoded image through an output layer to obtain a reconstructed high dynamic range image.

[0007] According to a second aspect of the embodiments of this application, a high dynamic range image reconstruction apparatus is provided. The apparatus includes: a global alignment module, configured to globally align received linear domain long exposure images and linear domain short exposure images to obtain globally aligned linear domain long exposure images and linear domain short exposure images; a guided alignment module, configured to extract feature images from the globally aligned linear domain long exposure images and linear domain short exposure images, and guide a linear domain target feature map to align with a linear domain reference feature map to obtain a linear domain short exposure target frame feature map; a fusion processing module, configured to construct a long-range dependency between the linear domain short exposure target frame feature map and the linear domain reference feature map to obtain a fused linear domain short exposure attention map; and a decoding output module, configured to perform decoding processing based on the linear domain short exposure attention map and the linear domain reference feature map, and output the decoded image through an output layer to obtain a reconstructed high dynamic range image.

[0008] According to a third aspect of the embodiments of this application, an electronic device is provided, including: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, which causes the processor to perform an operation corresponding to the method of the first aspect.

[0009] According to a fourth aspect of the embodiments of this application, a storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect.

[0010] According to a fifth aspect of the embodiments of this application, a computer program product is provided that, when executed by a processor, implements the method described in the first aspect.

[0011] This application provides a method, apparatus, electronic device, and storage medium for high dynamic range (HMR) image reconstruction. The method involves globally aligning a long-exposure image and a short-exposure image in the linear domain, extracting feature images from the globally aligned long-exposure and short-exposure images, guiding the alignment of a linear domain target feature map with a reference linear domain feature map, and obtaining a short-exposure target frame feature map in the linear domain. A long-range dependency between the short-exposure target frame feature map and the reference linear domain feature map is constructed to obtain a fused short-exposure attention map in the linear domain. Decoding is performed based on the short-exposure attention map and the reference linear domain feature map, and the decoded image is passed through an output layer to obtain a reconstructed HMR image. This application uses the long-exposure image as the reference frame and the short-exposure image as the target frame. The motion information of the reconstructed HMR image is consistent with that of the long-exposure image and simultaneously includes the dynamic range of the short-exposure image. This application solves the ghosting problem caused by large global motions that deep convolutional neural networks cannot handle by globally aligning the long-exposure and short-exposure images. Alignment of the linear domain target feature map with the linear domain reference feature map and image fusion processing optimize the issues of detail blurring and color cast during the reconstruction of overexposed areas. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0013] Figure 1 A flowchart illustrating a high dynamic range image reconstruction method provided in one embodiment of this application;

[0014] Figure 2 A flowchart of step S2 of a high dynamic range image reconstruction method provided in another embodiment of this application;

[0015] Figure 3 A flowchart of step S21 of a high dynamic range image reconstruction method provided in another embodiment of this application;

[0016] Figure 4 A flowchart of step S23 of a high dynamic range image reconstruction method provided in another embodiment of this application;

[0017] Figure 5 A flowchart of step S24 of a high dynamic range image reconstruction method provided in another embodiment of this application;

[0018] Figure 6 for Figure 5 A schematic diagram illustrating the process of obtaining the linear domain short-exposure target frame feature map in step S24;

[0019] Figure 7 A flowchart of step S3 of a high dynamic range image reconstruction method provided in another embodiment of this application;

[0020] Figure 8 for Figure 7 A schematic diagram of the process of obtaining the linear domain short exposure attention map in step S3;

[0021] Figure 9 A flowchart of step S31 of a high dynamic range image reconstruction method provided in another embodiment of this application;

[0022] Figure 10 for Figure 7 A schematic diagram of the process of obtaining the first attention map in step S31;

[0023] Figure 11 A flowchart of step S32 of a high dynamic range image reconstruction method provided in another embodiment of this application;

[0024] Figure 12 A flowchart of step S4 of a high dynamic range image reconstruction method provided in another embodiment of this application;

[0025] Figure 13 A structural diagram of a high dynamic range image reconstruction apparatus provided in another embodiment of this application;

[0026] Figure 14 A structural diagram of a high dynamic range image reconstruction apparatus provided in another embodiment of this application;

[0027] Figure 15 This is a schematic diagram of the structure of an electronic device in another embodiment of this application. Detailed Implementation

[0028] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings. For ease of explanation, the same reference numerals denote the same components in the embodiments of this application, and for brevity, detailed descriptions of the same components are omitted in different embodiments. It should be understood that the thickness, length, width, and other dimensions of various components in the embodiments of this application shown in the accompanying drawings, as well as the overall thickness, length, width, and other dimensions of the integrated device, are merely illustrative examples and should not constitute any limitation on this application.

[0029] It should be noted that, without conflict, the various embodiments and / or technical features described in this application can be arbitrarily combined with each other, and the resulting technical solutions should also fall within the protection scope of this application.

[0030] It should be understood that the specific examples in the embodiments of this application are only for the purpose of helping those skilled in the art to better understand the embodiments of this application, and are not intended to limit the scope of the embodiments of this application.

[0031] It should also be understood that, in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0032] It should also be understood that the terminology used in the embodiments of this application and the appended claims is for the purpose of describing particular embodiments only and is not intended to limit the embodiments of this application. For example, the singular forms "a," "the," and "the" as used in the embodiments of this application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0033] As used herein, terms such as “first,” “second,” and “third” describe various components, parts, areas, layers, and / or sections, but such components, parts, areas, layers, and / or sections should not be limited by such terms. These terms are used only to distinguish one component, part, area, layer, or section from another. For example, the use of terms such as “first,” “second,” and “third” herein does not imply sequence or order unless explicitly indicated by the context.

[0034] Furthermore, for ease of description, spatial relative terms such as “below,” “under,” “above,” “upper,” and similar terms may be used herein to describe the relationship of one component or member to another component or member illustrated in the figures. In addition to the orientations depicted in the figures, spatial relative terms are intended to cover different orientations of the device during use or operation. The device may be oriented in other ways (rotated 90 degrees or otherwise) and therefore the spatial relative descriptors used herein may be interpreted in the same way.

[0035] In typical high dynamic range (HDR) image reconstruction, one approach is to reduce the weights of regions with relative motion in the input low dynamic range (LMR) image to address ghosting caused by large local motion (e.g., large motion caused by a person waving their hand). However, this approach is prone to ghosting, detail blurring, and color cast in overexposed motion scenarios. Another approach is to align the LMR image using multi-scale alignment, but this is hampered by the insufficient receptive field of convolutional neural networks, making it unable to handle ghosting caused by large global motion due to hand tremors. Yet another approach is to expand the network's receptive field by concatenating non-local blocks, but this still cannot address ghosting caused by large global motion.

[0036] This application uses a long-exposure image in the linear domain as the reference frame and a short-exposure image in the linear domain as the target frame. The motion information of the reconstructed high dynamic range image is consistent with that of the long-exposure image in the linear domain and simultaneously includes the dynamic range of the short-exposure image in the linear domain. This application solves the ghosting problem caused by large global motion that deep convolutional neural networks cannot handle by globally aligning the long-exposure image and the short-exposure image in the linear domain. The alignment of the linear domain target feature map and the linear domain reference feature map, as well as image fusion processing, optimize the detail blurring and color cast problems in the overexposed area reconstruction process.

[0037] See Figure 1 This application provides a high dynamic range image reconstruction method, the method comprising:

[0038] Step S1: Perform global alignment on the linear domain long exposure image and the linear domain short exposure image to obtain the globally aligned linear domain long exposure image and linear domain short exposure image.

[0039] This application embodiment applies to linear domain long exposure images I ref and linear domain short exposure image I tar Perform global alignment and calculate the linear domain short-exposure image I. tar Long exposure image I in the linear domain ref The affine transformation matrix required for alignment; global alignment can effectively solve the problem of large global motion caused by hand shakiness, thereby reducing the ghosting problem caused by large global motion. Among them, the linear domain long exposure image I... ref and linear domain short exposure image I tar These are two low dynamic range images with different exposure times.

[0040] Step S2: Extract feature images from the globally aligned linear domain long-exposure image and linear domain short-exposure image, guide the alignment of the linear domain target feature map with the linear domain reference feature map, and obtain the linear domain short-exposure target frame feature map.

[0041] This application embodiment extracts feature images from globally aligned linear domain long-exposure images and linear domain short-exposure images, and guides the alignment of the linear domain target feature map with reference to the linear domain reference feature map. This enables the alignment of the linear domain reference feature map and the linear domain target feature map, thereby achieving better alignment performance for overexposed motion regions.

[0042] In some specific implementations of the embodiments of this application, see [link to relevant documentation]. Figure 2 Step S2 includes:

[0043] Step S21: Preprocess the globally aligned linear domain long exposure image and linear domain short exposure image to obtain the linear domain reference image, the linear domain target image, the gamma correction reference image, and the gamma correction target image.

[0044] In some specific implementations of the embodiments of this application, see [link to relevant documentation]. Figure 3 Step S21 includes:

[0045] Step S211: Normalize the exposure time of the globally aligned linear domain long exposure image to the globally aligned linear domain short exposure image to obtain the linear domain reference image, and make the globally aligned linear domain short exposure image the linear domain target image.

[0046] Specifically, linear domain long exposure image I ref and linear domain short exposure image I tar The exposure times are t ref t tar This application embodiment aims to make the linear domain long exposure image I... ref and linear domain short exposure image I tar The reconstructed high dynamic range image has the same characteristics as the linear domain long exposure image I. ref Consistent motion information, and simultaneously includes linear domain long exposure images I ref and linear domain short exposure image I tar The dynamic range allows the long-exposure image Iref in the linear domain to be mapped to the short-exposure image I in the linear domain. tar Normalize the exposure time, that is Among them, L ref This is the reference image for the linear domain. L tar =I tar , where L tar The target image is in the linear domain.

[0047] Step S212: Perform gamma calculation on the globally aligned linear domain long exposure image to obtain a gamma-corrected reference image. Normalize the exposure time of the globally aligned linear domain short exposure image to the globally aligned linear domain long exposure image and perform gamma correction to obtain the gamma-corrected target image.

[0048] Specifically, the gamma-corrected reference image G ref Gamma-corrected target image G tar Through respectively The calculation yields γ = 2.2.

[0049] In this embodiment, normalization processing is used to make the exposure levels of the linear domain long exposure image and the linear domain short exposure image consistent. This makes the linear domain reference image, the linear domain target image, the gamma correction reference image, and the gamma correction target image obtained through preprocessing helpful for the alignment of overexposed areas in step S24, which aligns the linear domain target feature map to the linear domain reference feature map.

[0050] Step S22: Extract features from the linear domain reference image and the linear domain target image to obtain the linear domain target feature map and the linear domain reference feature map.

[0051] Specifically, in step S22, the linear domain reference image L ref and linear domain target image L tar Perform equal-size feature extraction to obtain a linear domain reference feature map f. ref and linear domain target feature map f tar .

[0052] Step S23: Perform multi-size feature extraction on the gamma-corrected reference image and the gamma-corrected target image to obtain multi-size gamma-corrected reference feature maps and gamma-corrected target feature maps.

[0053] Specifically, see Figure 4 Step S23 includes:

[0054] Step S231: Perform equal-size feature extraction on the gamma-corrected reference image and the gamma-corrected target image to obtain equal-size gamma-corrected reference feature map and equal-size gamma-corrected target feature map.

[0055] Step S232: Perform 1 / 2 size feature extraction on the gamma correction reference image and the gamma correction target image to obtain a 1 / 2 size gamma correction reference feature map and a 1 / 2 size gamma correction target feature map.

[0056] Step S233: Perform 1 / 4 size feature extraction on the gamma correction reference image and the gamma correction target image to obtain a 1 / 4 size gamma correction reference feature map and a 1 / 4 size gamma correction target feature map.

[0057] Specifically, the gamma-corrected reference image G is processed by cascaded residual blocks and convolutional layers. ref Gamma-corrected target image G tar The residual block, consisting of two convolutional layers, is mapped to a scale-invariant, uniformly sized gamma-corrected reference feature map g. ref1and gamma-corrected target feature map of the same size g tar1 Then compare the gamma-corrected reference feature map g with the same size. ref1 and gamma-corrected target feature map of the same size g tar1 A half-size gamma-corrected reference feature map g is obtained after two cascaded downsampling convolutional blocks. ref2 and 1 / 2 size gamma-corrected target feature map g tar2 and a 1 / 4-size gamma-corrected reference feature map g ref3 and 1 / 4-size gamma-corrected target feature map g tar3 .

[0058] Step S24: Using the gamma-corrected reference feature map and the gamma-corrected target feature map as alignment parameters, the linear domain target feature map is aligned with the linear domain reference feature map to obtain the linear domain short-exposure target frame feature map.

[0059] In some specific implementations of the embodiments of this application, see [link to relevant documentation]. Figure 5 as well as Figure 6 Step S24 includes:

[0060] Step S241: The 1 / 4-size gamma-corrected reference feature map g ref3 and 1 / 4-size gamma-corrected target feature map g tar3 The layers are cascaded together, and after passing through a third offset estimation layer and upsampling, a third offset feature map Offset3 with the same size as the second layer is obtained.

[0061] Step S242: The 1 / 2-size gamma-corrected target feature map g tar2 Concatenated with the offset feature map Offset3 of the third layer, and passed through a deformable convolutional layer, we obtain the aligned linear domain short-exposure target frame feature map g′ of the third layer. tar2 .

[0062] Step S243: Convert the feature map g′ of the short-exposure target frame in the third-layer linear domain. tar2 and 1 / 2 size gamma-corrected reference feature map g ref2 The layers are cascaded together and combined with the offset feature map Offset3 of the third layer. After passing through the second offset estimation layer and upsampling processing, the offset feature map Offset2 of the second layer is obtained.

[0063] Step S244: Transform the gamma-corrected target feature map g into a uniform size. tar1 Concatenated with the offset feature map Offset2 of the second layer, and passed through a deformable convolutional layer, we obtain the aligned second-layer linear domain short-exposure target frame feature map g′. tar1 .

[0064] Step S245: Convert the second-layer linear domain short-exposure target frame feature map g′ tar1 gamma-corrected reference feature map of the same size ref1 They are cascaded together and passed through the first offset estimation layer to obtain the first layer offset feature map Offset1.

[0065] Step S246: Convert the gamma-corrected target feature map f of the same size tar The feature map is concatenated with the offset feature map Offset1 of the first layer and then passed through a deformable convolutional layer to obtain a finely aligned linear domain short-exposure target frame feature map f′. tar .

[0066] Specifically, cascading refers to cascading along the channel dimension.

[0067] Specifically, the first offset estimation layer, the second offset estimation layer, and the third offset estimation layer each employ different offset estimation methods.

[0068] Specifically, the upsampling is bilinear interpolation upsampling.

[0069] The guided multi-scale alignment in step S24 of this embodiment uses a feature map in gamma correction mode to guide the linear domain feature map to achieve more accurate alignment, resulting in better alignment performance for challenging overexposed motion regions. The linear domain short-exposure target frame feature map f′ output in this embodiment is... tar In terms of motion information, it is similar to the linear domain reference feature map f ref Consistent and simultaneously containing linear domain target feature maps f tar The dynamic range.

[0070] S3. Construct the long-range dependency between the linear domain short-exposure target frame feature map and the linear domain reference feature map to obtain the fused linear domain short-exposure attention map.

[0071] In the embodiments of this application, the fusion process establishes a linear domain reference feature map f. ref Feature map f′ of short-exposure target frame in the linear domain tar The relationship between them makes the linear domain reference feature map f ref The region where information is lost due to overexposure and motion is compared with the linear domain short-exposure target frame feature map f′. tar Dependencies are established among relevant pixels to achieve effective and accurate image dynamic range recovery and obtain a short-exposure attention map f″ in the linear domain. tar .

[0072] In some specific implementations of the embodiments of this application, see [link to relevant documentation]. Figure 7 as well as Figure 8 Step S3 includes:

[0073] Step S31: Based on the linear domain short exposure target frame feature map f′ tar and linear domain reference feature map f ref Perform cross-attention processing to obtain the first attention map m1.

[0074] Step S32: Based on the linear domain reference feature map f ref and linear domain short exposure target frame feature map f′ tar The overlay image (f′) with the first attention map m1 tar +m1) performs cross-attention processing to obtain the second attention map m2.

[0075] Step S33: Convert the linear domain short-exposure target frame feature map f′ tar By overlaying the first attention map m1 and the second attention map m2, a short-exposure attention map f″ in the linear domain is obtained. tar .

[0076] The linear domain short exposure attention map f″ in the embodiments of this application tar Includes linear domain reference feature map f ref Feature map f′ of short-exposure target frame in the linear domain tar By analyzing global relevant information, we can further achieve effective and accurate image dynamic range recovery.

[0077] This application employs a serial double-cross attention processing method to construct a linear domain reference feature map f. ref Feature map f′ of short-exposure target frame in the linear domain tar By establishing global connections between pixels and remote dependencies between them, the problems of ghosting, blurring, and color cast that are easily seen in overexposed moving areas can be solved, and the computational cost is significantly reduced.

[0078] Specifically, see Figure 9 as well as Figure 10 Step S31 includes:

[0079] Step S311: Pass the linear domain reference feature map through a convolutional layer to obtain the first matrix, and pass the linear domain short-exposure target frame feature map through two convolutional layers to obtain the second and third matrices respectively.

[0080] Step S312: Model the pixel relationship between the first matrix and the second matrix along the first direction and the second direction to obtain the first modeling matrix along the first direction and the second modeling matrix along the second direction.

[0081] Step S313: Reintegrate the third matrix along the first and second directions to obtain a first integrated matrix along the first direction and a second integrated matrix along the second direction.

[0082] Step S314: Obtain the first sub-attention map in the first direction based on the first modeling matrix and the first integration matrix, and obtain the second sub-attention map in the second direction based on the second modeling matrix and the second integration matrix.

[0083] Step S315: Obtain the first attention map based on the first sub-attention map and the second sub-attention map.

[0084] Specifically, see the linear domain short-exposure target frame feature map f′ in 10. tar and linear domain reference feature map f ref Size information (B: batch size, C: number of channels, H: height, W: width). Linear domain reference feature map f ref The first matrix Q is obtained after a 1×1 convolutional layer, while the linear domain short-exposure target frame feature map f′ is obtained. tar The second matrix K and the third matrix V are obtained after two 1×1 convolutional layers. Pixel relationship modeling involves reorganizing the dimensions of the first matrix Q and the second matrix K along the first direction H and the second direction W to obtain Q. H Q W K H K W Its first modeling matrix, Energy_H, is Q. H and K H The result of the dot product, while the second modeling matrix, Energy_W, is Q. W and K W The dot product results are obtained by recombining the third matrix V along the H and W directions to obtain the first integrated matrix Value_H and the second integrated matrix Value_W, respectively. Then, the first modeling matrix Energy_H is multiplied by the first integrated matrix Value_H, and the second modeling matrix Energy_W is multiplied by the second integrated matrix Value_W to obtain the first sub-attention map Attention_H and the second sub-attention map Attention_W along the first and second directions, respectively. The sum of the first sub-attention map Attention_H and the second sub-attention map Attention_W constitutes the first attention map m1.

[0085] Specifically, see Figure 11 Step S32 includes:

[0086] Step S321: The linear domain reference feature map is passed through a convolutional layer to obtain a fourth matrix, and the superimposed map of the linear domain short exposure target frame feature map and the first attention map is passed through two convolutional layers to obtain a fifth matrix and a sixth matrix, respectively.

[0087] Step S322: Model the pixel relationship between the fourth matrix and the fifth matrix along the first direction and the second direction to obtain the third modeling matrix along the first direction and the fourth modeling matrix along the second direction.

[0088] Step S323: Reintegrate the sixth matrix along the first direction and the second direction to obtain a third integrated matrix along the first direction and a fourth integrated matrix along the second direction.

[0089] Step S324: Obtain the third sub-attention map in the first direction based on the third modeling matrix and the third integration matrix, and obtain the fourth sub-attention map in the second direction based on the fourth modeling matrix and the fourth integration matrix.

[0090] Step S325: Obtain the second attention map based on the third sub-attention map and the fourth sub-attention map.

[0091] Step S32 is performed in the same way as step S31, except that the linear domain short exposure target frame feature map in step S32 is replaced with a superimposed linear domain short exposure target frame feature map and the first attention map.

[0092] The fourth, fifth, and sixth matrices in step S32 can be the same as or different from the first, second, and third matrices.

[0093] The calculation method of the first attention map m1 and the second attention map m2 in this application embodiment only calculates the weight of a certain pixel and the weight of the same row and column, making the computational workload of the global calculation lighter, without having to calculate the weight of all pixels in the entire image.

[0094] S4. Decode the image based on the linear domain short exposure attention map and the linear domain reference feature map, and then pass the decoded image through the output layer to obtain the reconstructed high dynamic range image.

[0095] In some specific implementations of the embodiments of this application, see [link to relevant documentation]. Figure 12 Step S4 includes:

[0096] Step S41: Decode the linear domain short exposure attention map and the linear domain reference feature map to obtain the decoded feature map.

[0097] The decoding process includes employing multiple decoding implementation methods. This application's embodiments utilize multiple decoding implementation methods to ensure a sufficiently large receptive field during decoding, thereby achieving good detail recovery.

[0098] Step S42: The linear domain reference feature map is superimposed on the decoded feature map to obtain the superimposed feature map.

[0099] Among them, the linear domain reference frame feature map f in the embodiment of this application ref By superimposing these features onto the decoded feature map using skip connections, a superimposed feature map can be obtained, which can better reduce the risk of overfitting during network training.

[0100] Step S43: Overlay feature maps for output layer processing to obtain the reconstructed high dynamic range image.

[0101] The process involves overlaying feature maps for output layer processing to obtain a reconstructed high dynamic range image, including:

[0102] The superimposed feature map is mapped into a three-channel high dynamic range image, with the value range of the high dynamic range image between 0 and 1.

[0103] Corresponding to the above method, this application also provides a high dynamic range image reconstruction apparatus, see [link to relevant documentation]. Figure 13 The device includes:

[0104] The global alignment module 131 is used to perform global alignment on the received linear domain long exposure image and linear domain short exposure image to obtain a globally aligned linear domain long exposure image and linear domain short exposure image.

[0105] The alignment module 132 is used to extract feature images from the globally aligned linear domain long-exposure image and linear domain short-exposure image, guide the linear domain target feature map to align with the linear domain reference feature map, and obtain the linear domain short-exposure target frame feature map.

[0106] The fusion processing module 133 is used to construct the remote dependency between the linear domain short exposure target frame feature map and the linear domain reference feature map to obtain the fused linear domain short exposure attention map.

[0107] The decoding output module 134 is used to perform decoding processing based on the linear domain short exposure attention map and the linear domain reference feature map, and to obtain the reconstructed high dynamic range image by passing the decoded image through the output layer.

[0108] This application uses a long-exposure image in the linear domain as the reference frame and a short-exposure image in the linear domain as the target frame. The motion information of the reconstructed high dynamic range image is consistent with that of the long-exposure image in the linear domain and simultaneously includes the dynamic range of the short-exposure image in the linear domain. This application solves the ghosting problem caused by large global motion that deep convolutional neural networks cannot handle by globally aligning the long-exposure image and the short-exposure image in the linear domain. The alignment of the linear domain target feature map and the linear domain reference feature map, as well as image fusion processing, optimize the detail blurring and color cast problems in the overexposed area reconstruction process.

[0109] This application also provides another high dynamic range image reconstruction apparatus, see [link to previous document]. Figure 14 The device includes:

[0110] The global alignment module 141 is used to perform global alignment on the received linear domain long exposure image and linear domain short exposure image to obtain a globally aligned linear domain long exposure image and linear domain short exposure image.

[0111] The preprocessing module 142 is used to preprocess the globally aligned linear domain long exposure image and linear domain short exposure image to obtain a linear domain reference image and a linear domain target image, as well as a gamma-corrected reference image and a gamma-corrected target image.

[0112] The equal-size feature extraction module 143 is used to extract equal-size features from the linear domain reference image and the linear domain target image to obtain a linear domain short-exposure feature map and a linear domain long-exposure feature map.

[0113] The multi-size feature extraction module 144 is used to extract multi-size features from the gamma-corrected reference image and the gamma-corrected target image to obtain multi-size gamma-corrected reference feature maps and gamma-corrected target feature maps.

[0114] The multi-size guided alignment module 145 is used to align the linear domain short exposure feature map to the linear domain long exposure feature map by using the gamma-corrected reference feature map and the gamma-corrected target feature map as alignment parameters, thereby obtaining the linear domain short exposure target frame feature map.

[0115] The fusion processing module 146 is used to construct the remote dependency between the linear domain short exposure target frame feature map and the linear domain reference feature map to obtain the fused linear domain short exposure attention map.

[0116] Decoding module 147 is used to perform decoding processing based on the linear domain short exposure attention map and the linear domain reference feature map.

[0117] Adder 148 is used to superimpose the linear domain reference frame feature map onto the decoded feature map in a skip connection manner.

[0118] Output layer 149 is used to obtain a reconstructed high dynamic range image from the decoded image through the output layer.

[0119] This application embodiment extracts feature images from globally aligned linear domain long-exposure images and linear domain short-exposure images, and guides the alignment of the linear domain target feature map with reference to the linear domain reference feature map. This enables the alignment of the linear domain reference feature map and the linear domain target feature map, thereby achieving better alignment performance for overexposed motion regions.

[0120] The alignment of the linear domain target feature map and the linear domain reference feature map, as well as the image fusion processing in the embodiments of this application, optimize the problems of detail blurring and color cast in the reconstruction process of overexposed areas.

[0121] Based on the method described in Embodiment 1 above, this application also provides an electronic device for executing the method described in the above embodiments, with reference to... Figure 15 This document illustrates a schematic diagram of an electronic device according to an embodiment of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.

[0122] like Figure 15 As shown, the electronic device 150 may include: a processor 1502, a communications interface 1504, a memory 1506, and a communications bus 1508.

[0123] in:

[0124] The processor 1502, communication interface 1504, and memory 1506 communicate with each other via communication bus 1508.

[0125] Communication interface 1504 is used to communicate with other electronic devices or servers.

[0126] The processor 1502 is used to execute program 1510, specifically to perform the relevant steps in the above data processing method embodiment.

[0127] Specifically, program 1510 may include program code that includes computer operation instructions.

[0128] The processor 1502 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The smart device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0129] Memory 1506 is used to store program 1510. Memory 1506 contains SRAM memory.

[0130] Specifically, program 1510 can be used to cause processor 1502 to execute the steps described in any of the methods in the above embodiments. The specific implementation of each step in program 1510 can be found in the corresponding descriptions in the above method embodiments, and will not be repeated here. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the devices and modules described above can be referred to the corresponding process descriptions in the foregoing method embodiments, and will not be repeated here.

[0131] Based on the methods described in the above embodiments, this application provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the methods described in the above embodiments.

[0132] Based on the methods described in the above embodiments, this application provides a computer program product that, when executed by a processor, implements the methods described in the above embodiments.

[0133] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0134] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be processed by software stored on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the navigation methods described herein are implemented. Furthermore, when a general-purpose computer accesses the code used to implement the navigation methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the navigation methods shown herein.

[0135] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0136] This application describes embodiments, but is not intended to limit them. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments. Therefore, all equivalent technical solutions also fall within the scope of the embodiments. The patent protection scope of the embodiments should be defined by the claims.

Claims

1. A high dynamic range image reconstruction method, the method comprising: Global alignment is performed on the linear domain long exposure image and the linear domain short exposure image to obtain the globally aligned linear domain long exposure image and the linear domain short exposure image; Feature images are extracted from the globally aligned linear domain long-exposure image and linear domain short-exposure image. The linear domain target feature map is then aligned with the linear domain reference feature map to obtain the linear domain short-exposure target frame feature map. Construct a long-range dependency between the linear domain short-exposure target frame feature map and the linear domain reference feature map to obtain a fused linear domain short-exposure attention map; The linear domain short exposure attention map and the linear domain reference feature map are used for decoding, and the decoded image is passed through the output layer to obtain the reconstructed high dynamic range image. The step of extracting feature images from the globally aligned linear domain long-exposure image and linear domain short-exposure image, and guiding the alignment of the linear domain target feature map with reference to the linear domain reference feature map to obtain the linear domain short-exposure target frame feature map includes: The globally aligned linear domain long-exposure image and linear domain short-exposure image are preprocessed to obtain a linear domain reference image, a linear domain target image, a gamma-corrected reference image, and a gamma-corrected target image. Feature extraction is performed on the linear domain reference image and the linear domain target image to obtain a linear domain target feature map and a linear domain reference feature map; Multi-size feature extraction is performed on the gamma-corrected reference image and the gamma-corrected target image to obtain multi-size gamma-corrected reference feature maps and gamma-corrected target feature maps; Using the gamma-corrected reference feature map and the gamma-corrected target feature map as alignment parameters, the linear domain target feature map is aligned with the linear domain reference feature map to obtain a linear domain short-exposure target frame feature map.

2. The method according to claim 1, wherein, The preprocessing of the globally aligned linear domain long-exposure image and linear domain short-exposure image to obtain a linear domain reference image, a linear domain target image, a gamma-corrected reference image, and a gamma-corrected target image includes: The exposure time of the globally aligned linear domain long exposure image is normalized to that of the globally aligned linear domain short exposure image to obtain a linear domain reference image, and the globally aligned linear domain short exposure image is designated as the linear domain target image. A gamma-corrected reference image is obtained by performing gamma calculation on the globally aligned linear domain long exposure image. The exposure time of the globally aligned linear domain short exposure image is then normalized and gamma-corrected to the globally aligned linear domain long exposure image to obtain the gamma-corrected target image.

3. The method according to claim 2, wherein, The step of extracting features from the linear domain reference image and the linear domain target image to obtain a linear domain target feature map and a linear domain reference feature map includes: Perform equal-sized feature extraction on the linear domain reference image and the linear domain target image to obtain a linear domain target feature map and a linear domain reference feature map; The step of extracting multi-scale features from the gamma-corrected reference image and the gamma-corrected target image to obtain multi-scale gamma-corrected reference feature maps and gamma-corrected target feature maps includes: The gamma-corrected reference image and the gamma-corrected target image are subjected to equal-sized feature extraction to obtain equal-sized gamma-corrected reference feature map and equal-sized gamma-corrected target feature map; Half-size feature extraction is performed on the gamma-corrected reference image and the gamma-corrected target image to obtain a half-size gamma-corrected reference feature map and a half-size gamma-corrected target feature map; A quarter-size feature extraction is performed on the gamma-corrected reference image and the gamma-corrected target image to obtain a quarter-size gamma-corrected reference feature map and a quarter-size gamma-corrected target feature map.

4. The method according to claim 3, wherein, The step of using the gamma-corrected reference feature map and the gamma-corrected target feature map as alignment parameters to align the linear domain target feature map with the linear domain reference feature map to obtain a linear domain short-exposure target frame feature map includes: The 1 / 4-size gamma-corrected reference feature map and the 1 / 4-size gamma-corrected target feature map are concatenated together, and after passing through the third offset estimation layer and upsampling processing, an offset feature map of the third layer with the same size as the second layer is obtained. The half-size gamma-corrected target feature map is concatenated with the offset feature map of the third layer, and after passing through a deformable convolutional layer, an aligned third-layer linear domain short-exposure target frame feature map is obtained. The third-layer linear domain short-exposure target frame feature map and the 1 / 2-size gamma correction reference feature map are concatenated together and combined with the offset feature map of the third layer. After passing through the second-layer offset estimation layer and upsampling processing, the offset feature map of the second layer is obtained. The equal-sized gamma-corrected target feature map is concatenated with the offset feature map of the second layer, and after passing through a deformable convolutional layer, an aligned second-layer linear domain short-exposure target frame feature map is obtained. The second-layer linear domain short-exposure target frame feature map is concatenated with the same-sized gamma correction reference feature map, and then passed through the first-layer offset estimation layer to obtain the first-layer offset feature map; The equal-sized gamma-corrected target feature map is concatenated with the offset feature map of the first layer and passed through a deformable convolutional layer to obtain a finely aligned linear domain short-exposure target frame feature map.

5. The method according to claim 4, wherein, The cascading refers to cascading along the channel dimension.

6. The method according to claim 5, wherein, The first offset estimation layer, the second offset estimation layer, and the third offset estimation layer each employ different offset estimation methods.

7. The method according to claim 6, wherein, The upsampling is bilinear interpolation upsampling.

8. The method according to claim 1, wherein, The step of constructing the long-range dependency between the linear domain short-exposure target frame feature map and the linear domain reference feature map to obtain the fused linear domain short-exposure attention map includes: A first attention map is obtained by performing cross-attention processing on the linear domain short-exposure target frame feature map and the linear domain reference feature map; A second attention map is obtained by performing cross-attention processing on the linear domain reference feature map, the linear domain short exposure target frame feature map, and the overlay map of the first attention map. The linear domain short exposure target frame feature map is superimposed with the first attention map and the second attention map to obtain the linear domain short exposure attention map.

9. The method according to claim 8, wherein, The step of performing cross-attention processing based on the linear domain short-exposure target frame feature map and the linear domain reference feature map to obtain a first attention map includes: The linear domain reference feature map is passed through a convolutional layer to obtain a first matrix, and the linear domain short-exposure target frame feature map is passed through two convolutional layers to obtain a second matrix and a third matrix, respectively. Model the pixel-to-pixel relationship between the first matrix and the second matrix along a first direction and a second direction to obtain a first modeling matrix along the first direction and a second modeling matrix along the second direction. The third matrix is ​​re-integrated along the first direction and the second direction to obtain a first integrated matrix along the first direction and a second integrated matrix along the second direction; A first sub-attention map in the first direction is obtained based on the first modeling matrix and the first integration matrix, and a second sub-attention map in the second direction is obtained based on the second modeling matrix and the second integration matrix; The first attention map is obtained based on the first sub-attention map and the second sub-attention map.

10. The method according to claim 8, wherein, The step of performing cross-attention processing based on the linear domain reference feature map, the linear domain short-exposure target frame feature map, and the overlay map of the first attention map to obtain the second attention map includes: The linear domain reference feature map is passed through a convolutional layer to obtain a fourth matrix, and the linear domain short exposure target frame feature map and the first attention map superimposed map are passed through two convolutional layers to obtain a fifth matrix and a sixth matrix, respectively. Pixel-to-pixel relationships are modeled on the fourth and fifth matrices along the first and second directions to obtain a third modeling matrix along the first direction and a fourth modeling matrix along the second direction. The sixth matrix is ​​re-integrated along the first direction and the second direction to obtain a third integrated matrix along the first direction and a fourth integrated matrix along the second direction; The third sub-attention map in the first direction is obtained based on the third modeling matrix and the third integration matrix, and the fourth sub-attention map in the second direction is obtained based on the fourth modeling matrix and the fourth integration matrix. The second attention map is obtained based on the third sub-attention map and the fourth sub-attention map.

11. The method according to claim 1, wherein, The step of decoding based on the linear domain short-exposure attention map and the linear domain reference feature map, and then passing the decoded image through the output layer to obtain the reconstructed high dynamic range image, includes: Decoding is performed based on the fused linear domain short exposure attention map and the linear domain reference feature map to obtain a decoded feature map; The linear domain reference feature map is superimposed on the decoded feature map to obtain a superimposed feature map; The superimposed feature map is processed by the output layer to obtain a reconstructed high dynamic range image.

12. The method according to claim 11, wherein, The decoding process includes using various decoding implementation methods.

13. The method according to claim 11, wherein, The linear domain reference feature map is superimposed on the decoded feature map to obtain a superimposed feature map, including: the linear domain reference feature map is superimposed on the decoded feature map in a skip connection manner to obtain a superimposed feature map.

14. The method according to claim 13, wherein, The superimposed feature maps are processed by the output layer to obtain a reconstructed high dynamic range image, including: The superimposed feature map is mapped into a three-channel high dynamic range image, the value range of which is between 0 and 1.

15. A high dynamic range image reconstruction apparatus, the apparatus comprising: The global alignment module is used to globally align the received linear domain long exposure image and linear domain short exposure image to obtain a globally aligned linear domain long exposure image and linear domain short exposure image. The alignment module is used to extract feature images from the globally aligned linear domain long-exposure image and linear domain short-exposure image, and guide the linear domain target feature map to align with the linear domain reference feature map to obtain the linear domain short-exposure target frame feature map. The fusion processing module is used to construct the remote dependency between the linear domain short exposure target frame feature map and the linear domain reference feature map to obtain the fused linear domain short exposure attention map. The decoding output module is used to perform decoding processing based on the fused linear domain short exposure attention map and the linear domain reference feature map, and to obtain a reconstructed high dynamic range image by passing the decoded image through the output layer. The guidance alignment module is specifically used for: The globally aligned linear domain long-exposure image and linear domain short-exposure image are preprocessed to obtain a linear domain reference image, a linear domain target image, a gamma-corrected reference image, and a gamma-corrected target image. Feature extraction is performed on the linear domain reference image and the linear domain target image to obtain a linear domain target feature map and a linear domain reference feature map; Multi-size feature extraction is performed on the gamma-corrected reference image and the gamma-corrected target image to obtain multi-size gamma-corrected reference feature maps and gamma-corrected target feature maps; Using the gamma-corrected reference feature map and the gamma-corrected target feature map as alignment parameters, the linear domain target feature map is aligned with the linear domain reference feature map to obtain a linear domain short-exposure target frame feature map.

16. An electronic device comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation corresponding to the method as described in any one of claims 1-14.

17. A storage medium storing a computer program that, when executed by a processor, implements the method as described in any one of claims 1-14.

18. A computer program product comprising a computer program that, when executed by a processor, implements the method as described in any one of claims 1-14.

Images