A method, a terminal, and an electronic device for reconstructing an HDR image

Abstract

The present disclosure provides a method, a terminal, and an electronic device for reconstructing an HDR image, which are used to ensure the quality of the reconstructed HDR image while reducing the hardware processing cost. The method includes: obtaining multiple original images with the same shooting scene but different exposure degrees; screening out a reference image from the multiple original images, and performing feature alignment processing on the remaining original images according to the reference image to obtain displacement images of the remaining original images; determining an enhanced image according to the reference image and the displacement images of the remaining original images, where the enhanced image is obtained by performing image enhancement processing on a fused image after downsampling, and the fused image is obtained by performing feature fusion on the reference image and the displacement images of the remaining original images; reconstructing an HDR image corresponding to the multiple original images according to the enhanced image.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technologies, and particularly to a method, a terminal, and an electronic device for reconstructing an HDR image. Background Art

[0002] HDR (High Dynamic Range Imaging) is a set of technologies used to achieve a larger exposure dynamic range than ordinary digital image technologies. The purpose of HDR is to correctly represent the range of brightness from direct sunlight to the darkest shadows in the real world. HDR can provide more dynamic range and image details.

[0003] Since the dynamic range of the scene obtained by fixing the exposure of a single image is very limited, it is necessary to restore the actual illuminance data of the actual scene through multiple exposures, so as to obtain an HDR image. Currently, most use multiple ordinary digital images with different exposure degrees to calculate the actual scene brightness, and obtain an HDR high dynamic range image after high-speed computer calculation, and display the HDR image on a low dynamic range (LDR) display device through a compression algorithm.

[0004] However, the current method for generating HDR images has high computational complexity, requires high hardware processing capabilities, high costs, large resource consumption, and poor universality. Summary of the Invention

[0005] The present disclosure provides a method, a terminal, and an electronic device for reconstructing an HDR image, which are used to ensure the quality of the reconstructed HDR image and reduce the hardware processing cost at the same time.

[0006] In a first aspect, a method for reconstructing an HDR image provided by an embodiment of the present disclosure includes:

[0007] Obtain multiple original images with the same shooting scene but different exposure degrees;

[0008] Select a reference image from the multiple original images, and perform feature alignment processing on the remaining original images according to the reference image to obtain displacement images of the remaining original images;

[0009] Determine an enhanced image according to the reference image and the displacement images of the remaining original images, where the enhanced image is obtained by performing image enhancement processing on a fused image after downsampling, and the fused image is obtained by performing feature fusion on the reference image and the displacement images of the remaining original images;

[0010] Reconstruct an HDR image corresponding to the multiple original images according to the enhanced image.

[0011] As an optional implementation manner, after obtaining the displacement image of the remaining original images and before determining the enhanced image according to the reference image and the displacement image of the remaining original images, it further includes:

[0012] Performing at least one downsampling feature alignment process, and each downsampling feature alignment process performs the following steps:

[0013] Performing a downsampling operation on the reference feature image and the displacement image of the remaining original images to obtain the reference feature image and the displacement image after the downsampling operation; wherein the reference feature image is obtained by performing feature extraction on the reference image;

[0014] Performing a feature alignment process on the displacement image after the downsampling operation according to the reference feature image after the downsampling operation to obtain the displacement image after the current feature alignment process.

[0015] As an optional implementation manner, after determining the enhanced image according to the reference image and the displacement image and before reconstructing the HDR image corresponding to the multiple original images according to the enhanced image, it further includes:

[0016] Performing at least one image enhancement operation;

[0017] Wherein each image enhancement operation performs the following steps:

[0018] Taking the enhanced image determined last time as the reference image for this time, and determining the enhanced image for this time according to the reference image for this time and the displacement image.

[0019] As an optional implementation manner, the step of obtaining the displacement image of the remaining original images by performing a feature alignment process on the remaining original images according to the reference image includes:

[0020] Determining the displacement parameter matrix corresponding to the remaining original images according to the feature similarity between the reference image and each of the remaining original images;

[0021] Displacing the features of the remaining original images according to the displacement parameter matrix to obtain the displacement image.

[0022] As an optional implementation manner, the step of obtaining the displacement image of the remaining original images by performing a feature alignment process on the remaining original images according to the reference image includes:

[0023] Performing feature extraction on the reference image to obtain a reference feature image, and performing feature extraction on each of the remaining original images to obtain an original feature image;

[0024] Merge the reference feature image and each original feature image respectively to obtain a first merged image corresponding to each original feature image;

[0025] Input the first merged image and the corresponding original feature image into an attention network to output a displacement image of the original feature image.

[0026] As an optional implementation manner, the attention network is used to determine a displacement parameter matrix of the corresponding original feature image according to the first merged image, and displace the features of the corresponding original feature image by using the displacement parameter matrix.

[0027] As an optional implementation manner, the sampling multiple of the downsampling operation is determined according to the computing power of the hardware platform.

[0028] As an optional implementation manner, determine the fusion image through the following method:

[0029] Merge the reference feature image obtained by feature extraction of the reference image and the displacement images of the remaining original images to obtain a second merged image;

[0030] Reduce the dimension of the second merged image through a convolutional layer to obtain the fusion image.

[0031] As an optional implementation manner, the image enhancement processing for the fusion image after the downsampling operation includes:

[0032] Perform a downsampling operation on the fusion image to obtain a downsampled image;

[0033] Perform image enhancement processing on the downsampled image, where the image enhancement processing is used to align similar features in the downsampled image and enhance the features representing image details in the downsampled image.

[0034] As an optional implementation manner, the image enhancement processing for the downsampled image includes:

[0035] Input the downsampled image into an image enhancement network to output a network image; where the image enhancement network is used to align similar features in the downsampled image and enhance the features representing image details in the downsampled image;

[0036] Perform an upsampling operation on the network image to obtain the enhanced image.

[0037] As an optional implementation manner, reconstruct the HDR image corresponding to the multiple original images according to the enhanced image, including:

[0038] The enhanced image is dimensionally reduced through multiple convolutional layers to obtain the HDR image.

[0039] As an alternative implementation, the reference image selected from multiple original images includes:

[0040] Select the original image with the exposure in the middle from multiple original images as the reference image.

[0041] As an alternative implementation, the obtaining of multiple original images with the same shooting scene and different exposure levels includes:

[0042] In response to a user's shooting instruction, the multiple original images with different exposure levels are continuously shot for the same shooting scene through an imaging component.

[0043] As an alternative implementation, after reconstructing the HDR image corresponding to the multiple original images according to the enhanced image, it further includes:

[0044] Display the reconstructed HDR image on a display.

[0045] In a second aspect, a terminal for reconstructing an HDR image provided by an embodiment of the present disclosure includes a processor and a memory. The memory is used to store a program executable by the processor, and the processor is used to read the program in the memory and execute the following steps:

[0046] Obtain multiple original images with the same shooting scene and different exposure levels;

[0047] Select a reference image from multiple original images, perform feature alignment processing on the remaining original images according to the reference image to obtain a displacement image of the remaining original images;

[0048] Determine an enhanced image according to the reference image and the displacement image of the remaining original images, where the enhanced image is obtained by performing image enhancement processing on a fused image after downsampling operation, and the fused image is obtained by performing feature fusion on the reference image and the displacement image of the remaining original images;

[0049] Reconstruct the HDR image corresponding to the multiple original images according to the enhanced image.

[0050] As an alternative implementation, after obtaining the displacement image of the remaining original images and before determining the enhanced image according to the reference image and the displacement image of the remaining original images, the processor is specifically further configured to execute:

[0051] Perform at least one downsampling feature alignment process, where each downsampling feature alignment process executes the following steps:

[0052] Perform a downsampling operation on the displacement images of the reference feature image and the remaining original images to obtain the reference feature image and the displacement image after the downsampling operation; wherein the reference feature image is obtained by performing feature extraction on the reference image.

[0053] According to the reference feature image after the downsampling operation, perform feature alignment processing on the displacement image after the downsampling operation to obtain the displacement image after the current feature alignment processing.

[0054] As an optional implementation manner, after determining the enhanced image according to the reference image and the displacement image, and before reconstructing the HDR image corresponding to the multiple original images according to the enhanced image, the processor is specifically further configured to perform:

[0055] Perform at least one image enhancement operation, and each image enhancement operation performs the following steps:

[0056] Use the enhanced image determined last time as the reference image this time, and determine the enhanced image this time according to the reference image this time and the displacement image.

[0057] As an optional implementation manner, the processor is specifically configured to perform:

[0058] Determine the displacement parameter matrix corresponding to the remaining original images according to the feature similarity between the reference image and each of the remaining original images;

[0059] Displace the features of the remaining original images according to the displacement parameter matrix to obtain the displacement image.

[0060] As an optional implementation manner, the processor is specifically configured to perform:

[0061] Perform feature extraction on the reference image to obtain the reference feature image, and perform feature extraction on each of the remaining original images to obtain the original feature images;

[0062] Merge the reference feature image and each original feature image respectively to obtain the first merged image corresponding to each original feature image;

[0063] Input the first merged image and the corresponding original feature image into the attention network, and output the displacement image of the original feature image.

[0064] As an optional implementation manner, the attention network is used to determine the displacement parameter matrix of the corresponding original feature image according to the first merged image, and displace the features of the corresponding original feature image by using the displacement parameter matrix.

[0065] As an alternative implementation, the sampling multiple of the downsampling operation is determined according to the computing power of the hardware platform.

[0066] As an alternative implementation, the processor is specifically configured to determine the fused image in the following manner:

[0067] Merge the reference feature image obtained by performing feature extraction on the reference image and the displacement image of the remaining original images to obtain a second merged image;

[0068] Reduce the dimension of the second merged image through a convolutional layer to obtain the fused image.

[0069] As an alternative implementation, the processor is specifically configured to perform:

[0070] Perform a downsampling operation on the fused image to obtain a downsampled image;

[0071] Perform image enhancement processing on the downsampled image, where the image enhancement processing is used to align similar features in the downsampled image and enhance the features representing image details in the downsampled image.

[0072] As an alternative implementation, the processor is specifically configured to perform:

[0073] Input the downsampled image into an image enhancement network to output a network image; where the image enhancement network is used to align similar features in the downsampled image and enhance the features representing image details in the downsampled image;

[0074] Perform an upsampling operation on the network image to obtain the enhanced image.

[0075] As an alternative implementation, the processor is specifically configured to perform:

[0076] Perform dimensionality reduction processing on the enhanced image through multiple convolutional layers to obtain the HDR image.

[0077] As an alternative implementation, the processor is specifically configured to perform:

[0078] Select the original image with the exposure in the middle from multiple original images as the reference image.

[0079] As an alternative implementation, the processor is specifically configured to perform:

[0080] In response to the user's shooting instruction, continuously shoot multiple original images with different exposure degrees for the same shooting scene through the camera component.

[0081] As an alternative implementation, after reconstructing the HDR images corresponding to the multiple original images according to the enhanced image, the processor is further specifically configured to perform:

[0082] Display the reconstructed HDR image on a display.

[0083] In a third aspect, an embodiment of the present disclosure further provides an electronic device for reconstructing an HDR image. The electronic device includes an imaging unit and a control circuit, where:

[0084] The imaging unit is configured to acquire original images with different exposure degrees;

[0085] The control circuit includes a processor and a memory. The memory is used to store programs executable by the processor. The processor is configured to read the programs in the memory and perform the following steps:

[0086] Acquire multiple original images with the same shooting scene and different exposure degrees;

[0087] Select a reference image from the multiple original images, and perform feature alignment processing on the remaining original images according to the reference image to obtain displacement images of the remaining original images;

[0088] Determine an enhanced image according to the reference image and the displacement images of the remaining original images, where the enhanced image is obtained by performing image enhancement processing on a fused image after downsampling, and the fused image is obtained by performing feature fusion on the reference image and the displacement images of the remaining original images;

[0089] Reconstruct the HDR images corresponding to the multiple original images according to the enhanced image.

[0090] In a fourth aspect, an embodiment of the present disclosure further provides a device for reconstructing an HDR image, including:

[0091] An image acquisition unit, configured to acquire multiple original images with the same shooting scene and different exposure degrees;

[0092] A feature alignment unit, configured to select a reference image from the multiple original images, and perform feature alignment processing on the remaining original images according to the reference image to obtain displacement images of the remaining original images;

[0093] A feature enhancement unit, configured to determine an enhanced image according to the reference image and the displacement images of the remaining original images, where the enhanced image is obtained by performing image enhancement processing on a fused image after downsampling, and the fused image is obtained by performing feature fusion on the reference image and the displacement images of the remaining original images;

[0094] A feature reconstruction unit for reconstructing an HDR image corresponding to the multiple original images according to the enhanced image.

[0095] In a fifth aspect, an embodiment of the present disclosure further provides a non-transitory computer storage medium, on which a computer program is stored, and when the program is executed by a processor, it is used to implement the steps of the method described in the first aspect above.

[0096] These aspects or other aspects of the present disclosure will be more clearly understood in the following description of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] To more clearly illustrate the technical solutions in the embodiments of the present disclosure, the following briefly introduces the accompanying drawings required for the description of the embodiments. Obviously, the accompanying drawings in the following description are only some embodiments of the present disclosure. For those of ordinary skill in the art, other accompanying drawings can be obtained based on these drawings without creative efforts.

[0098] Figure 1 FIG. is a flowchart of an embodiment of a method for reconstructing an HDR image provided by an embodiment of the present disclosure;

[0099] Figure 2 FIG. is a schematic diagram of original images with different exposure levels provided by an embodiment of the present disclosure;

[0100] Figure 3 FIG. is a schematic diagram of a scenario applied to terminal shooting provided by an embodiment of the present disclosure;

[0101] Figure 4 FIG. is a schematic diagram of the structure of an attention network provided by an embodiment of the present disclosure;

[0102] Figure 5 FIG. is a schematic diagram of the structure of an image enhancement network provided by an embodiment of the present disclosure;

[0103] Figure 6 FIG. is a schematic diagram of the structure of a BNet network provided by an embodiment of the present disclosure;

[0104] Figure 7 FIG. is a schematic diagram of the structure of an ESA network provided by an embodiment of the present disclosure;

[0105] Figure 8 FIG. is a schematic diagram of a downsampling structure provided by an embodiment of the present disclosure;

[0106] Figure 9 FIG. is a schematic diagram of an upsampling structure provided by an embodiment of the present disclosure;

[0107] Figure 10 FIG. is a schematic diagram of the structure of a feature reconstruction network provided by an embodiment of the present disclosure;

[0108] Figure 11 Flowchart of a supplementary solution for reconstructing an HDR image provided by an embodiment of the present disclosure;

[0109] Figure 12 Flowchart of an enhancement solution for reconstructing an HDR image provided by an embodiment of the present disclosure;

[0110] Figure 13A Schematic diagram of a network architecture for reconstructing an HDR image provided by an embodiment of the present disclosure;

[0111] Figure 13B Method implementation process for reconstructing an HDR image provided by an embodiment of the present disclosure;

[0112] Figure 14A Another schematic diagram of a network architecture for reconstructing an HDR image provided by an embodiment of the present disclosure;

[0113] Figure 14B Another method implementation process for reconstructing an HDR image provided by an embodiment of the present disclosure;

[0114] Figure 15A Another schematic diagram of a network architecture for reconstructing an HDR image provided by an embodiment of the present disclosure;

[0115] Figure 15B Another method implementation process for reconstructing an HDR image provided by an embodiment of the present disclosure;

[0116] Figure 16 Schematic diagram of a terminal for reconstructing an HDR image provided by an embodiment of the present disclosure;

[0117] Figure 17 Schematic diagram of an electronic device for reconstructing an HDR image provided by an embodiment of the present disclosure;

[0118] Figure 18 Schematic diagram of a device for reconstructing an HDR image provided by an embodiment of the present disclosure. Detailed implementation manners

[0119] In order to make the objectives, technical solutions and advantages of the present disclosure clearer, the present disclosure will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present disclosure without creative efforts shall fall within the protection scope of the present disclosure.

[0120] In the embodiments of the present disclosure, the term "and / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after.

[0121] The application scenarios described in the embodiments of the present disclosure are for more clearly explaining the technical solutions of the embodiments of the present disclosure, and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those of ordinary skill in the art know that with the emergence of new application scenarios, the technical solutions provided by the embodiments of the present disclosure are equally applicable to similar technical problems. Among them, in the description of the present disclosure, unless otherwise specified, the meaning of "a plurality of" is two or more.

[0122] Embodiment 1. HDR (High Dynamic Range Imaging) is a set of technologies used to achieve a larger exposure dynamic range than ordinary digital image technologies. The purpose of HDR is to correctly represent the range of brightness from direct sunlight to the darkest shadows in the real world. HDR can provide more dynamic range and image details. Since the dynamic range of the scene obtained by fixing the exposure of a single image is very limited, it is necessary to perform multiple exposures to restore the actual illumination data of the actual scene, thereby obtaining an HDR image. Currently, most use multiple ordinary digital images with different exposure levels to calculate the actual scene brightness. After high-speed computer calculations, an HDR high dynamic range image is obtained, and the HDR image is displayed on a low dynamic range (LDR) display device through a compression algorithm. However, the current method of generating HDR images has high computational complexity, requires high hardware processing capabilities, high costs, large resource consumption, and poor universality.

[0123] An HDR reconstruction method for multiple images with different exposures provided in this embodiment can extract different bright and dark details in the images with different exposures and complement each other. The method for reconstructing an HDR image in this embodiment can also perform image enhancement processing on the fused image after downsampling during image enhancement processing, which can effectively save computing power and ensure the quality of the reconstructed HDR image through feature alignment processing and image enhancement processing.

[0124] As Figure 1 shown, the implementation process of a method for reconstructing an HDR image provided in this embodiment is as follows:

[0125] Step 100: Obtain multiple original images with the same shooting scene but different exposure levels;

[0126] In some embodiments, the multiple original images in this example are the multiple original images with different exposure degrees continuously captured by the shooting component for the same shooting scene. It should be noted that the multiple original images with different exposure degrees are obtained by the camera quickly switching the aperture within an extremely short time duration. As Figure 2 shown, a schematic diagram of original images with different exposure degrees provided in this embodiment, where 3 original images with different exposure degrees are captured for the same scene. From left to right, they are the low-exposure image, the medium-exposure image, and the high-exposure image. The greater the exposure degree, the higher the brightness of the original image, and the smaller the exposure degree, the darker the brightness of the original image. The size of the exposure degree can be determined according to the exposure parameters of the original image. Optionally, the number of original images captured in this embodiment is N, where N is an integer greater than or equal to 3.

[0127] In the scenario applied to a mobile terminal, multiple original images with the same shooting scene and different exposure degrees are obtained through the following method:

[0128] In response to the user's shooting instruction, the multiple original images with different exposure degrees are continuously captured by the camera component for the same shooting scene.

[0129] In implementation, as Figure 3 shown, a schematic diagram of a scenario applied to terminal shooting provided in this embodiment. The user turns on the camera of the terminal and selects whether to enter the HDR mode for shooting. If the HDR mode is not selected for shooting, only the current picture is captured according to the ordinary camera; if the HDR mode is selected for shooting, multiple consecutive original images with different exposure degrees can be quickly obtained after the user clicks the shoot button, and the multiple obtained original images are processed through the following steps in this embodiment to obtain the finally reconstructed HDR image.

[0130] Step 101: Screen out a reference image from the multiple original images, and perform feature alignment processing on the remaining original images according to the reference image to obtain the displacement images of the remaining original images;

[0131] In some embodiments, select the original image with a medium exposure degree from the multiple original images as the reference image. For example, select the medium-exposure original image as the reference image from the low-exposure original image, the medium-exposure original image, and the high-exposure original image.

[0132] In some embodiments, the present embodiment performs the feature alignment processing through the following method:

[0133] Taking the reference image as a benchmark, perform feature alignment processing on the features of each of the remaining original images after screening to obtain the displacement images of the remaining original images.

[0134] In some embodiments, the displacement image of the original image is obtained in the following manner in this embodiment.

[0135] According to the feature similarity between the reference image and each remaining original image after screening, determine the displacement parameter matrix corresponding to the remaining original images;

[0136] Displace the features of the remaining original images according to the displacement parameter matrix to obtain the displacement image.

[0137] In some embodiments, the feature alignment process is performed through an attention network in this embodiment, and the specific implementation process is as follows:

[0138] Extract features from the reference image to obtain a reference feature image, and extract features from each remaining original image to obtain an original feature image;

[0139] Merge the reference feature image and each original feature image respectively to obtain the first merged image corresponding to each original feature image;

[0140] Input the first merged image and the corresponding original feature image into the attention network, and output the displacement image of the original feature image.

[0141] It should be noted that the reference feature image and the original feature image in this embodiment are both essentially a matrix. In some embodiments, the merging of the reference feature image and the original feature image in this embodiment is essentially the merging of two matrices, which is a process of arranging or merging the two matrices without changing the order of the two matrices themselves. For example, perform a concat operation on the reference feature image and the original feature image.

[0142] In some embodiments, extract features from the remaining original images to obtain original feature images, where the original images can be feature-extracted through a feature extraction network. For example, a 3×3 convolutional layer can be used for feature extraction, and the number of feature channels is expanded from 3 channels (i.e., the original image is an RGB image) to nf channels (nf can be taken as 64 or 48, 32, etc.), so as to convert the original image into an original feature image by using this convolutional layer.

[0143] In some embodiments, the attention network in this embodiment is used to determine the displacement parameter matrix of the corresponding original feature image according to the first merged image, and use the displacement parameter matrix to displace the features of the corresponding original feature image.

[0144] In some embodiments, such as Figure 4As shown in the figure, this embodiment provides a schematic structural diagram of an attention network, where the input of the attention network is the first merged image and the original feature image. c represents the input first merged image, and f represents the input original feature image. Optionally, the number of attention networks in this embodiment is determined according to the number of remaining original images. Among them, the S-shaped function is the Sigmoid function, also known as the S-shaped growth curve. In information science, due to its properties such as monotonic increase and monotonic increase of the inverse function, the Sigmoid function is often used as the activation function of neural networks to map variables between 0 and 1.

[0145] Step 102: Determine an enhanced image according to the reference image and the displacement image of the remaining original images, where the enhanced image is obtained by performing image enhancement processing on the fused image after downsampling operation, and the fused image is obtained by performing feature fusion on the reference image and the displacement image of the remaining original images;

[0146] In implementation, first perform feature fusion on the reference image and the displacement images of all remaining original images to obtain a fused image; then perform a downsampling operation on the fused image, and finally perform image enhancement processing on the fused image after the downsampling operation to obtain the final enhanced image.

[0147] In some embodiments, the sampling multiple of the downsampling operation in this embodiment is determined according to the computing power of the hardware platform.

[0148] In some embodiments, the fused image is determined in the following manner:

[0149] The reference feature image obtained by performing feature extraction on the reference image and the displacement images of the remaining original images are merged to obtain a second merged image; the dimension of the second merged image is reduced through a convolutional layer to obtain the fused image.

[0150] In implementation, the displacement image in this embodiment is essentially a matrix, and the reference image after feature extraction is also a matrix. In some embodiments, the merging of the reference feature image after feature extraction and the displacement image in this embodiment is essentially the merging of two matrices, which is a process of arranging or merging the two matrices without changing the order of the two matrices themselves. For example, a concat operation is performed on the displacement image and the reference feature image after feature extraction.

[0151] In some embodiments, the dimension of the second merged image can be reduced through a 3×3 convolutional layer, where the number of features is reduced from 3×nf to nf (nf can be taken as 64 or 48, 32, etc.).

[0152] In some embodiments, the fused image can be first downsampled, and then the downsampled fused image can be subjected to image enhancement processing, where the image enhancement processing is used to align similar features in the downsampled image and enhance the features representing image details in the downsampled image.

[0153] In some embodiments, an image enhancement network is used for image enhancement processing to obtain an enhanced image. The specific implementation process is as follows:

[0154] The downsampled image is input into the image enhancement network to output a network image; the network image is upsampled to obtain the enhanced image. Wherein the image enhancement network is used to align similar features in the downsampled image and enhance the features representing image details in the downsampled image;

[0155] In some embodiments, as Figure 5 shown, this embodiment provides a schematic structural diagram of an image enhancement network, including three BNet network structures, where C in the figure represents the concat operation.

[0156] As Figure 6 shown, this embodiment provides a schematic diagram of a BNet network structure, where conv represents a convolutional layer, k1 represents a convolutional layer size of 1×1, k3 represents a convolutional layer size of 3x3, f represents the number of features. For example, f64->32 means the number of features changes from 64 to 32. Concat represents the operation of merging or arranging matrices.

[0157] As Figure 7 shown, this embodiment provides a schematic diagram of an ESA network structure, where ESA is a spatial self-attention network that only performs self-correction on the currently input features.

[0158] As Figure 8 shown, this embodiment also provides a schematic diagram of a downsampling structure, such as 2-fold Mux downsampling, where a 11 , b 11 , c 11 , d 11 etc. all represent the pixel values of the fused image, where the fused image is a grayscale image. The convolutional layer Conv is k3f(nf×4->nf), which means the convolutional layer size is 3×3, the features change from nf×4 to nf, and the number of features decreases. nf is a positive integer. The upsampling structure uses the same principle DeMux structure. As Figure 9 shown, this embodiment provides a schematic diagram of an upsampling structure.

[0159] Step 103, reconstruct the HDR image corresponding to the multiple original images according to the enhanced image.

[0160] In some embodiments, in this embodiment, the enhanced image is dimensionally reduced through multiple convolutional layers to obtain the HDR image. During implementation, since the enhanced image is a multi-dimensional feature image after feature enhancement, the reconstruction process requires dimensional reduction of the enhanced image to obtain the final HDR image for display.

[0161] As Figure 10 shown, this embodiment also provides a schematic structural diagram of a feature reconstruction network. After inputting the enhanced image into the feature reconstruction network, the HDR image is output. Wherein, conv represents a convolutional layer, k1 represents a convolutional layer with a size of 1×1, k3 represents a convolutional layer with a size of 3×3, f represents the number of features. For example, f(nf->3) represents that the number of features changes from nf to 3.

[0162] In some embodiments, after reconstructing the HDR images corresponding to the multiple original images according to the enhanced image, the reconstructed HDR image can also be displayed on a display.

[0163] In some embodiments, as Figure 11 shown, this embodiment also provides a supplementary scheme for reconstructing the HDR image. The specific implementation process of this scheme is as follows:

[0164] Step 1100: Obtain multiple original images with the same shooting scene but different exposure degrees;

[0165] Step 1101: Screen out a reference image from the multiple original images, and perform feature alignment processing on the remaining original images according to the reference image to obtain the displacement images of the remaining original images;

[0166] Step 1102: Perform at least one downsampling feature alignment process;

[0167] Among them, each downsampling feature alignment process performs the following steps:

[0168] Perform downsampling operations on the reference feature image and the displacement images of the remaining original images to obtain the reference feature image and displacement images after the downsampling operations; wherein the reference feature image is obtained by performing feature extraction on the reference image;

[0169] During implementation, perform downsampling operations on the reference feature image and also perform downsampling operations on each displacement image.

[0170] According to the reference feature image after the downsampling operation, perform feature alignment processing on the displacement image after the downsampling operation to obtain the displacement image after this feature alignment processing. During implementation, based on the same implementation principle of the feature alignment processing, during implementation, each downsampling feature alignment process needs to first perform a downsampling operation and then perform feature alignment processing. The process of the feature alignment processing is as follows:

[0171] Determine the displacement parameter matrix corresponding to the displacement image according to the feature similarity between the reference feature image after the downsampling operation and each displacement image; displace the features of the displacement image again according to the displacement parameter matrix to obtain the displacement image after the current feature alignment process.

[0172] Step 1103: Determine an enhanced image according to the reference feature image and the displacement image after performing at least one downsampled feature alignment process.

[0173] The enhanced image is obtained by performing image enhancement processing on the fused image after the downsampling operation, and the fused image is obtained by performing feature fusion on the reference image and the displacement image; based on the same implementation principle of image enhancement processing, determine the enhanced image according to the reference image and the displacement image after the downsampled feature alignment process.

[0174] Step 1104: Perform at least one upsampling operation on the enhanced image to obtain the enhanced image after the upsampling operation.

[0175] Step 1105: Reconstruct the HDR image corresponding to the multiple original images according to the enhanced image after the upsampling operation.

[0176] In some embodiments, as Figure 12 shown, this embodiment also provides an enhancement scheme for reconstructing the HDR image. This enhancement scheme can be implemented in combination with the above supplementary scheme, and the specific process is as follows:

[0177] Step 1200: Obtain multiple original images with the same shooting scene and different exposure levels.

[0178] Step 1201: Select a reference image from the multiple original images, and perform feature alignment processing on the remaining original images according to the reference image to obtain the displacement images of the remaining original images.

[0179] Step 1202: Perform at least one downsampled feature alignment process.

[0180] Among them, each downsampled feature alignment process performs the following steps:

[0181] Perform a downsampling operation on the reference feature image obtained through feature extraction and the displacement images of the remaining original images to obtain the reference feature image and the displacement images after the downsampling operation; in implementation, perform a downsampling operation on the reference feature image and perform a downsampling operation on each displacement image at the same time.

[0182] Based on the reference feature image after the downsampling operation, perform feature alignment processing on the displacement image after the downsampling operation to obtain the displacement image after this feature alignment processing. In implementation, based on the implementation principle of the same feature alignment processing, in implementation, each downsampling feature alignment processing needs to first perform the downsampling operation and then perform the feature alignment processing. The process of the feature alignment processing is specifically as follows:

[0183] Determine the displacement parameter matrix corresponding to the displacement image according to the feature similarity between the reference feature image after the downsampling operation and each displacement image; displace the features of the displacement image again according to the displacement parameter matrix to obtain the displacement image after this feature alignment processing.

[0184] Step 1203: Determine an enhanced image according to the reference feature image and the displacement image after performing at least one downsampling feature alignment processing, and perform at least one upsampling operation on the enhanced image to obtain the enhanced image after the upsampling operation;

[0185] Wherein the enhanced image is obtained by performing image enhancement processing on the fused image after the downsampling operation, and the fused image is obtained by performing feature fusion on the reference image and the displacement image; based on the same implementation principle of the image enhancement processing, determine the enhanced image according to the reference image and the displacement image after the downsampling feature alignment processing.

[0186] Step 1204: Perform at least one image enhancement operation to obtain the enhanced image of this time, and perform at least one upsampling operation on the enhanced image of this time to obtain the enhanced image after the upsampling operation.

[0187] Wherein each image enhancement operation performs the following steps:

[0188] Take the enhanced image determined last time as the reference image of this time, and determine the enhanced image of this time according to the reference image of this time and the displacement image.

[0189] Step 1205: Reconstruct the HDR image corresponding to the multiple original images according to the enhanced image after the upsampling operation.

[0190] As Figure 13A shown, this embodiment takes three original images with different exposure degrees as an example to provide a schematic diagram of the network architecture for reconstructing the HDR image. As Figure 13B shown, a method for reconstructing the HDR image provided in this embodiment is specifically described based on the network architecture:

[0191] Step 1300: Obtain a low-exposure image, a medium-exposure image, and a high-exposure image with the same shooting scene;

[0192] Step 1301: Use the medium-exposure image as the reference image, and perform feature alignment processing on the low-exposure image and the high-exposure image respectively to obtain the corresponding low-exposure displacement image and high-exposure displacement image;

[0193] Among them, the feature alignment processing specifically includes:

[0194] Extract reference feature images from the reference image, extract low-exposure feature images from the low-exposure image, and extract high-exposure feature images from the high-exposure image;

[0195] Merge the reference feature image and the low-exposure feature image to obtain the first merged image of the low-exposure, and input the first merged image and the low-exposure feature image into the attention network to output the low-exposure displacement image of the low-exposure feature image;

[0196] Similarly, merge the reference feature image and the high-exposure feature image to obtain the first merged image of the high-exposure, and input the first merged image and the high-exposure feature image into the attention network to output the high-exposure displacement image of the high-exposure feature image.

[0197] Step 1302: Input the reference feature image, the low-exposure displacement image, and the high-exposure displacement image obtained by feature extraction of the reference image into the feature fusion network to output the fused image;

[0198] Among them, the feature fusion network is used to perform feature fusion on the reference feature image, the low-exposure displacement image, and the high-exposure displacement image obtained by feature extraction to obtain the fused image; the specific process of feature fusion is as follows:

[0199] Merge the reference feature image, the low-exposure displacement image, and the high-exposure displacement image obtained by feature extraction of the reference image to obtain the second merged image; reduce the dimension of the second merged image through the convolutional layer to obtain the fused image.

[0200] Step 1303: Perform downsampling operation on the fused image to obtain the downsampled image, input the downsampled image into the image enhancement network to output the network image, and perform upsampling operation on the network image to obtain the enhanced image.

[0201] Among them, the image enhancement network is used to align similar features in the downsampled image and enhance the features representing image details in the downsampled image;

[0202] Step 1304: Input the enhanced image into the feature reconstruction network to output the HDR image.

[0203] Among them, the feature reconstruction network is used to perform dimensionality reduction processing on the enhanced image through multiple convolutional layers to obtain the HDR image.

[0204] As Figure 14A shown, in this embodiment, taking three original images with different exposure degrees as an example, a schematic diagram of a network architecture for reconstructing an HDR image is provided. As Figure 14B shown, a method for reconstructing an HDR image provided in this embodiment is specifically described based on the network architecture:

[0205] Step 1400: Obtain a low-exposure image, a medium-exposure image, and a high-exposure image of the same shooting scene;

[0206] Step 1401: Take the medium-exposure image as a reference image, and perform feature alignment processing on the low-exposure image and the high-exposure image respectively to obtain corresponding low-exposure displacement images and high-exposure displacement images;

[0207] Among them, the feature alignment processing specifically includes:

[0208] Extract reference feature images from the reference image, extract low-exposure feature images from the low-exposure image, and extract high-exposure feature images from the high-exposure image;

[0209] Merge the reference feature image and the low-exposure feature image to obtain a first merged image of the low-exposure, and input the first merged image and the low-exposure feature image into the attention network to output the low-exposure displacement image of the low-exposure feature image;

[0210] Similarly, merge the reference feature image and the high-exposure feature image to obtain a first merged image of the high-exposure, and input the first merged image and the high-exposure feature image into the attention network to output the high-exposure displacement image of the high-exposure feature image.

[0211] Step 1402: Perform downsampling operations on the reference feature image, the low-exposure displacement image, and the high-exposure displacement image obtained by feature extraction of the reference image to obtain a downsampled reference feature image, a downsampled low-exposure displacement image, and a downsampled high-exposure displacement image after the downsampling operation respectively;

[0212] Step 1403: Take the downsampled reference feature image as a reference image, and perform feature alignment processing on the downsampled low-exposure displacement image and the downsampled high-exposure displacement image respectively to obtain the low-exposure displacement image and the high-exposure displacement image after this feature alignment processing;

[0213] Among them, the specific feature alignment processing process is as follows:

[0214] Extract features from the downsampled reference feature image, the downsampled low-exposure displacement image, and the downsampled high-exposure displacement image;

[0215] Merge the downsampled reference feature image and the downsampled low-exposure displacement image after feature extraction to obtain the first merged image of the downsampled low-exposure. Input the first merged image and the downsampled low-exposure displacement image into the attention network to output the low-exposure displacement image of the downsampled low-exposure displacement image;

[0216] Similarly, merge the downsampled reference feature image and the downsampled high-exposure feature image after feature extraction to obtain the first merged image of the downsampled high-exposure. Input the first merged image and the downsampled high-exposure feature image into the attention network to output the high-exposure displacement image of the downsampled high-exposure feature image.

[0217] Step 1404: Input the downsampled reference feature image, the low-exposure displacement image, and the high-exposure displacement image into the feature fusion network to output the fused image;

[0218] The feature fusion network is used to perform feature fusion on the downsampled reference feature image, the low-exposure displacement image, and the high-exposure displacement image obtained through feature extraction to obtain the fused image. The specific process of feature fusion is as follows:

[0219] Merge the downsampled reference feature image, the low-exposure displacement image, and the high-exposure displacement image to obtain the second merged image; reduce the dimension of the second merged image through the convolutional layer to obtain the fused image.

[0220] Step 1405: Perform downsampling operation on the fused image to obtain the downsampled image. Input the downsampled image into the image enhancement network to output the network image, and perform upsampling operation on the network image to obtain the enhanced image.

[0221] The image enhancement network is used to align the similar features in the downsampled image and enhance the features representing the image details in the downsampled image;

[0222] Step 1406: Perform upsampling operation on the enhanced image to obtain the enhanced image after upsampling operation;

[0223] Step 1407: Input the enhanced image after upsampling operation into the feature reconstruction network to output the HDR image.

[0224] The feature reconstruction network is used to perform dimensionality reduction processing on the enhanced image through multiple convolutional layers to obtain the HDR image.

[0225] As Figure 15A shown, this embodiment takes three original images with different exposure degrees as an example to provide a schematic diagram of the network architecture for reconstructing the HDR image. As Figure 15B shown, based on the network architecture, a method for reconstructing the HDR image provided in this embodiment is specifically described:

[0226] Step 1500: Obtain a low-exposure image, a medium-exposure image, and a high-exposure image with the same shooting scene;

[0227] Step 1501: Use the medium-exposure image as the reference image, and perform feature alignment processing on the low-exposure image and the high-exposure image respectively to obtain the corresponding low-exposure displacement image and high-exposure displacement image;

[0228] Among them, the feature alignment processing specifically includes:

[0229] Extract features from the reference image to obtain a reference feature image, extract features from the low-exposure image to obtain a low-exposure feature image, and extract features from the high-exposure image to obtain a high-exposure feature image;

[0230] Merge the reference feature image and the low-exposure feature image to obtain a first merged image of the low-exposure, and input the first merged image and the low-exposure feature image into the attention network to output the low-exposure displacement image of the low-exposure feature image;

[0231] Similarly, merge the reference feature image and the high-exposure feature image to obtain a first merged image of the high-exposure, and input the first merged image and the high-exposure feature image into the attention network to output the high-exposure displacement image of the high-exposure feature image.

[0232] Step 1502: Perform downsampling operations on the reference feature image, the low-exposure displacement image, and the high-exposure displacement image obtained by feature extraction of the reference image to obtain the downsampled reference feature image, the downsampled low-exposure displacement image, and the downsampled high-exposure displacement image respectively;

[0233] Step 1503: Use the downsampled reference feature image as the reference image, and perform feature alignment processing on the downsampled low-exposure displacement image and the downsampled high-exposure displacement image respectively to obtain the low-exposure displacement image and the high-exposure displacement image after this feature alignment processing;

[0234] Among them, the specific feature alignment processing process is as follows:

[0235] Extract features from the downsampled reference feature image, the downsampled low-exposure displacement image, and the downsampled high-exposure displacement image;

[0236] Merge the downsampled reference feature image and the downsampled low-exposure displacement image after feature extraction to obtain a first merged image of the downsampled low-exposure, and input the first merged image and the downsampled low-exposure displacement image into the attention network to output the low-exposure displacement image of the downsampled low-exposure displacement image;

[0237] Similarly, the downsampled reference feature image and the downsampled high-exposure feature image after feature extraction are combined to obtain the first combined image of the downsampled high-exposure. The first combined image and the downsampled high-exposure feature image are input into the attention network to output the high-exposure displacement image of the downsampled high-exposure feature image.

[0238] Step 1504: The downsampled reference feature image, the low-exposure displacement image, and the high-exposure displacement image are downsampled again to obtain the first downsampled reference feature image, the first downsampled low-exposure displacement image, and the first downsampled high-exposure displacement image respectively.

[0239] Step 1505: Using the first downsampled reference feature image as the reference image, feature alignment processing is performed on the first downsampled low-exposure displacement image and the first downsampled high-exposure displacement image respectively to obtain the low-exposure displacement image and the high-exposure displacement image after this feature alignment processing.

[0240] Step 1506: The first downsampled reference feature image, the low-exposure displacement image, and the high-exposure displacement image are input into the feature fusion network to output the fused image.

[0241] The feature fusion network is used to perform feature fusion on the first downsampled reference feature image, the low-exposure displacement image, and the high-exposure displacement image to obtain the fused image. The specific process of feature fusion is as follows:

[0242] The first downsampled reference feature image, the low-exposure displacement image, and the high-exposure displacement image are combined to obtain the second combined image; the dimension of the second combined image is reduced through the convolutional layer to obtain the fused image.

[0243] Step 1507: The fused image is downsampled to obtain the downsampled image. The downsampled image is input into the image enhancement network to output the network image, and the network image is upsampled to obtain the enhanced image; the enhanced image is upsampled to obtain the enhanced image after the upsampling operation.

[0244] The image enhancement network is used to align similar features in the downsampled image and enhance the features representing the image details in the downsampled image.

[0245] Step 1508: The fused image is continuously downsampled to obtain the downsampled image. The downsampled image is input into the image enhancement network to output the network image, and the network image is upsampled to obtain the enhanced image; the enhanced image is upsampled to obtain the enhanced image after the upsampling operation.

[0246] Step 1509: The enhanced image after the upsampling operation is input into the feature reconstruction network to output the HDR image.

[0247] Among them, the feature reconstruction network is used to perform dimensionality reduction processing on the enhanced image through multiple convolutional layers to obtain the HDR image.

[0248] Embodiment 2: Based on the same disclosed concept, an embodiment of the present disclosure further provides a terminal for reconstructing an HDR image. Since this terminal is the terminal in the method of the embodiment of the present disclosure, and the principle of this terminal to solve problems is similar to that of this method, the implementation of this terminal can refer to the implementation of the method, and the repeated parts will not be elaborated.

[0249] As Figure 16 shown, the terminal includes a processor 1600 and a memory 1601. The memory 1601 is used to store programs executable by the processor 1600. The processor 1600 is used to read the programs in the memory 1601 and execute the following steps:

[0250] Obtain multiple original images with the same shooting scene but different exposure degrees;

[0251] Select a reference image from the multiple original images, and perform feature alignment processing on the remaining original images according to the reference image to obtain the displacement images of the remaining original images;

[0252] Determine an enhanced image according to the reference image and the displacement images of the remaining original images, where the enhanced image is obtained by performing image enhancement processing on the fused image after downsampling operation, and the fused image is obtained by performing feature fusion on the reference image and the displacement images of the remaining original images;

[0253] Reconstruct the HDR image corresponding to the multiple original images according to the enhanced image.

[0254] As an optional implementation manner, after obtaining the displacement images of the remaining original images and before determining the enhanced image according to the reference image and the displacement images of the remaining original images, the processor 1600 is specifically further configured to execute:

[0255] Perform at least one downsampling feature alignment process, and each downsampling feature alignment process executes the following steps:

[0256] Perform downsampling operation on the reference feature image and the displacement images of the remaining original images to obtain the reference feature image and displacement images after downsampling operation; where the reference feature image is obtained by performing feature extraction on the reference image;

[0257] Perform feature alignment processing on the displacement images after downsampling operation according to the reference feature image after downsampling operation to obtain the displacement images after this feature alignment processing.

[0258] As an alternative implementation, after determining the enhanced image based on the reference image and the displacement image, and before reconstructing the HDR image corresponding to the multiple original images based on the enhanced image, the processor 1600 is specifically further configured to perform:

[0259] Perform at least one image enhancement operation, and each image enhancement operation performs the following steps:

[0260] Use the previously determined enhanced image as the reference image for this time, and determine the enhanced image for this time based on the reference image for this time and the displacement image.

[0261] As an alternative implementation, the processor 1600 is specifically configured to perform:

[0262] Determine the displacement parameter matrix corresponding to the remaining original images according to the feature similarity between the reference image and each of the remaining original images;

[0263] Displace the features of the remaining original images according to the displacement parameter matrix to obtain a displacement image.

[0264] As an alternative implementation, the processor 1600 is specifically configured to perform:

[0265] Extract features from the reference image to obtain a reference feature image, and extract features from each of the remaining original images to obtain an original feature image;

[0266] Merge the reference feature image and each original feature image respectively to obtain a first merged image corresponding to each original feature image;

[0267] Input the first merged image and the corresponding original feature image into an attention network, and output the displacement image of the original feature image.

[0268] As an alternative implementation, the attention network is used to determine the displacement parameter matrix of the corresponding original feature image according to the first merged image, and displace the features of the corresponding original feature image by using the displacement parameter matrix.

[0269] As an alternative implementation, the sampling multiple of the downsampling operation is determined according to the computing power of the hardware platform.

[0270] As an alternative implementation, the processor 1600 is specifically configured to determine the fusion image in the following manner:

[0271] Merge the reference feature image obtained by feature extraction of the reference image and the displacement images of the remaining original images to obtain a second merged image;

[0272] Reduce the dimension of the second merged image through a convolutional layer to obtain the fused image.

[0273] As an alternative implementation, the processor 1600 is specifically configured to execute:

[0274] Perform a downsampling operation on the fused image to obtain a downsampled image;

[0275] Perform image enhancement processing on the downsampled image, where the image enhancement processing is used to align similar features in the downsampled image and enhance the features representing image details in the downsampled image.

[0276] As an alternative implementation, the processor 1600 is specifically configured to execute:

[0277] Input the downsampled image into an image enhancement network to output a network image; where the image enhancement network is used to align similar features in the downsampled image and enhance the features representing image details in the downsampled image;

[0278] Perform an upsampling operation on the network image to obtain the enhanced image.

[0279] As an alternative implementation, the processor 1600 is specifically configured to execute:

[0280] Perform dimensionality reduction processing on the enhanced image through multiple convolutional layers to obtain the HDR image.

[0281] As an alternative implementation, the processor 1600 is specifically configured to execute:

[0282] Select the original image with a medium exposure from multiple original images as the reference image.

[0283] As an alternative implementation, the processor 1600 is specifically configured to execute:

[0284] In response to a user's shooting instruction, continuously shoot multiple original images with different exposure degrees for the same shooting scene through the imaging component.

[0285] As an alternative implementation, after reconstructing the HDR image corresponding to the multiple original images according to the enhanced image, the processor 1600 is specifically further configured to execute:

[0286] Display the reconstructed HDR image on the display.

[0287] Example 3. Based on the same inventive concept, an embodiment of the present disclosure also provides an electronic device for reconstructing an HDR image. Since this electronic device is the same as the one in the method of the embodiment of the present disclosure, and the principle of solving problems by this electronic device is similar to that of the method, the implementation of this electronic device can refer to the implementation of the method, and the repeated parts will not be elaborated.

[0288] As Figure 17 shown, the electronic device includes a camera unit 1700 and a control circuit 1701, where:

[0289] The camera unit 1700 is configured to obtain original images with different exposure levels;

[0290] The control circuit 1701 includes a processor and a memory. The memory is used to store programs executable by the processor, and the processor is configured to read the programs in the memory and execute the following steps:

[0291] Obtain multiple original images with the same shooting scene but different exposure levels;

[0292] Select a reference image from the multiple original images, and perform feature alignment processing on the remaining original images according to the reference image to obtain displacement images of the remaining original images;

[0293] Determine an enhanced image according to the reference image and the displacement images of the remaining original images, where the enhanced image is obtained by performing image enhancement processing on a fused image after downsampling, and the fused image is obtained by performing feature fusion on the reference image and the displacement images of the remaining original images;

[0294] Reconstruct an HDR image corresponding to the multiple original images according to the enhanced image.

[0295] As an optional implementation manner, after obtaining the displacement images of the remaining original images and before determining the enhanced image according to the reference image and the displacement images of the remaining original images, the processor is further specifically configured to execute:

[0296] Perform at least one downsampling feature alignment process, where each downsampling feature alignment process executes the following steps:

[0297] Perform a downsampling operation on a reference feature image and the displacement images of the remaining original images to obtain a downsampled reference feature image and displacement images; where the reference feature image is obtained by performing feature extraction on the reference image;

[0298] Perform feature alignment processing on the downsampled displacement images according to the downsampled reference feature image to obtain displacement images after the current feature alignment process.

[0299] As an alternative implementation, after determining the enhanced image according to the reference image and the displacement image, and before reconstructing the HDR image corresponding to the multiple original images according to the enhanced image, the processor is specifically further configured to perform:

[0300] Perform at least one image enhancement operation, and each image enhancement operation performs the following steps:

[0301] Use the enhanced image determined last time as the reference image this time, and determine the enhanced image this time according to the reference image this time and the displacement image.

[0302] As an alternative implementation, the processor is specifically configured to perform:

[0303] Determine the displacement parameter matrix corresponding to the remaining original images according to the feature similarity between the reference image and each of the remaining original images;

[0304] Displace the features of the remaining original images according to the displacement parameter matrix to obtain displacement images.

[0305] As an alternative implementation, the processor is specifically configured to perform:

[0306] Extract features from the reference image to obtain a reference feature image, and extract features from each of the remaining original images to obtain original feature images;

[0307] Merge the reference feature image and each original feature image respectively to obtain a first merged image corresponding to each original feature image;

[0308] Input the first merged image and the corresponding original feature image into an attention network, and output the displacement image of the original feature image.

[0309] As an alternative implementation, the attention network is used to determine the displacement parameter matrix of the corresponding original feature image according to the first merged image, and use the displacement parameter matrix to displace the features of the corresponding original feature image.

[0310] As an alternative implementation, the sampling multiple of the downsampling operation is determined according to the computing power of the hardware platform.

[0311] As an alternative implementation, the processor is specifically configured to determine the fused image in the following manner:

[0312] The reference feature image obtained by performing feature extraction on the reference image and the displacement image of the remaining original images are merged to obtain a second merged image;

[0313] The dimension of the second merged image is reduced through a convolutional layer to obtain the fused image.

[0314] As an alternative implementation, the processor is specifically configured to execute:

[0315] Perform a downsampling operation on the fused image to obtain a downsampled image;

[0316] Perform image enhancement processing on the downsampled image, where the image enhancement processing is used to align similar features in the downsampled image and enhance the features representing image details in the downsampled image.

[0317] As an alternative implementation, the processor is specifically configured to execute:

[0318] Input the downsampled image into an image enhancement network to output a network image; where the image enhancement network is used to align similar features in the downsampled image and enhance the features representing image details in the downsampled image;

[0319] Perform an upsampling operation on the network image to obtain the enhanced image.

[0320] As an alternative implementation, the processor is specifically configured to execute:

[0321] Perform dimensionality reduction processing on the enhanced image through multiple convolutional layers to obtain the HDR image.

[0322] As an alternative implementation, the processor is specifically configured to execute:

[0323] Select the original image with a medium exposure from multiple original images as the reference image.

[0324] As an alternative implementation, the processor is specifically configured to execute:

[0325] In response to a user's shooting instruction, continuously shoot multiple original images with different exposure degrees for the same shooting scene through a camera component.

[0326] As an alternative implementation, after reconstructing the HDR image corresponding to the multiple original images according to the enhanced image, the processor is specifically further configured to execute:

[0327] Display the reconstructed HDR image on a display.

[0328] Example 4. Based on the same general inventive concept, an embodiment of the present disclosure also provides an apparatus for reconstructing an HDR image. Since this apparatus is the same as the apparatus in the method of the embodiment of the present disclosure, and the principle of solving problems by this apparatus is similar to that of the method, the implementation of this apparatus can refer to the implementation of the method, and the repeated parts will not be elaborated again.

[0329] As Figure 18 shown, the apparatus includes:

[0330] An image acquisition unit 1800, configured to acquire multiple original images with the same shooting scene but different exposure degrees;

[0331] A feature alignment unit 1801, configured to screen out a reference image from the multiple original images, and perform feature alignment processing on the remaining original images according to the reference image to obtain displacement images of the remaining original images;

[0332] A feature enhancement unit 1802, configured to determine an enhanced image according to the reference image and the displacement images of the remaining original images, where the enhanced image is obtained by performing image enhancement processing on a fused image after a downsampling operation, and the fused image is obtained by performing feature fusion on the reference image and the displacement images of the remaining original images;

[0333] A feature reconstruction unit 1803, configured to reconstruct an HDR image corresponding to the multiple original images according to the enhanced image.

[0334] As an optional implementation manner, after obtaining the displacement images of the remaining original images and before determining the enhanced image according to the reference image and the displacement images of the remaining original images, a downsampling alignment processing unit is further included, which is specifically configured to:

[0335] Perform at least one downsampling feature alignment process, and each downsampling feature alignment process performs the following steps:

[0336] Perform a downsampling operation on a reference feature image and the displacement images of the remaining original images to obtain a downsampled reference feature image and displacement images; where the reference feature image is obtained by performing feature extraction on the reference image;

[0337] Perform feature alignment processing on the displacement images after the downsampling operation according to the downsampled reference feature image to obtain displacement images after the current feature alignment process.

[0338] As an optional implementation manner, after determining the enhanced image according to the reference image and the displacement images, and before reconstructing the HDR image corresponding to the multiple original images according to the enhanced image, an image enhancement processing unit is further included, which is specifically configured to:

[0339] Perform at least one image enhancement operation, and each image enhancement operation performs the following steps:

[0340] Use the previously determined enhanced image as the reference image for this time, and determine the enhanced image for this time according to the reference image for this time and the displacement image.

[0341] As an optional implementation manner, the feature alignment unit 1801 is specifically configured to:

[0342] Determine the displacement parameter matrix corresponding to each of the remaining original images according to the feature similarity between the reference image and each of the remaining original images;

[0343] Displace the features of the remaining original images according to the displacement parameter matrix to obtain a displacement image.

[0344] As an optional implementation manner, the feature alignment unit 1801 is specifically configured to:

[0345] Extract features from the reference image to obtain a reference feature image, and extract features from each of the remaining original images to obtain an original feature image;

[0346] Merge the reference feature image and each original feature image respectively to obtain a first merged image corresponding to each original feature image;

[0347] Input the first merged image and the corresponding original feature image into an attention network, and output the displacement image of the original feature image.

[0348] As an optional implementation manner, the attention network is used to determine the displacement parameter matrix of the corresponding original feature image according to the first merged image, and use the displacement parameter matrix to displace the features of the corresponding original feature image.

[0349] As an optional implementation manner, the sampling multiple of the downsampling operation is determined according to the computing power of the hardware platform.

[0350] As an optional implementation manner, the feature enhancement unit 1802 is specifically configured to determine the fused image in the following manner:

[0351] Merge the reference feature image obtained by extracting features from the reference image and the displacement image of the remaining original images to obtain a second merged image;

[0352] Reduce the dimension of the second merged image through a convolutional layer to obtain the fused image.

[0353] As an optional implementation manner, the feature enhancement unit 1802 is specifically configured to:

[0354] Perform a downsampling operation on the fused image to obtain a downsampled image;

[0355] Perform image enhancement processing on the downsampled image, where the image enhancement processing is used to align similar features in the downsampled image and enhance the features representing image details in the downsampled image.

[0356] As an alternative implementation, the feature enhancement unit 1802 is specifically configured to:

[0357] Input the downsampled image into an image enhancement network to output a network image; where the image enhancement network is used to align similar features in the downsampled image and enhance the features representing image details in the downsampled image;

[0358] Perform an upsampling operation on the network image to obtain the enhanced image.

[0359] As an alternative implementation, the feature reconstruction unit 1803 is specifically configured to:

[0360] Perform dimensionality reduction processing on the enhanced image through multiple convolutional layers to obtain the HDR image.

[0361] As an alternative implementation, the feature alignment unit 1801 is specifically configured to:

[0362] Select the original image with the exposure in the middle from multiple original images as the reference image.

[0363] As an alternative implementation, the image acquisition unit 1800 is specifically configured to:

[0364] In response to a user's shooting instruction, continuously shoot the multiple original images with different exposure degrees for the same shooting scene through a camera component.

[0365] As an alternative implementation, after reconstructing the HDR image corresponding to the multiple original images according to the enhanced image, the display unit is further specifically configured to:

[0366] Display the reconstructed HDR image on a display.

[0367] Based on the same inventive concept, an embodiment of the present disclosure further provides a non-transitory computer storage medium, on which a computer program is stored, and when the program is executed by a processor, it is used to implement the following steps:

[0368] Obtain multiple original images with the same shooting scene and different exposure degrees;

[0369] Select a reference image from multiple original images, and perform feature alignment processing on the remaining original images according to the reference image to obtain displacement images of the remaining original images;

[0370] Determine an enhanced image according to the reference image and the displacement images of the remaining original images, where the enhanced image is obtained by performing image enhancement processing on a fused image after downsampling, and the fused image is obtained by performing feature fusion on the reference image and the displacement images of the remaining original images;

[0371] Reconstruct the HDR image corresponding to the multiple original images according to the enhanced image.

[0372] Those skilled in the art should understand that the embodiments of the present disclosure can be provided as a method, a system, or a computer program product. Therefore, the present disclosure can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present disclosure can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories and optical memories, etc.) containing computer-usable program code.

[0373] The present disclosure is described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowcharts and / or block diagrams, and the combination of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the specified functions in one process Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0374] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the specified functions in one process Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0375] These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are performed on the computer or other programmable device to generate a computer-implemented process, and thus the instructions executed on the computer or other programmable device provide for implementing in the processFigure 1 One process or multiple processes and / or boxes Figure 1 Steps of the functions specified in one box or multiple boxes.

[0376] Obviously, those skilled in the art can make various changes and modifications to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalent technologies, the present disclosure is also intended to include these modifications and variations.

Claims

1. A method for reconstructing an HDR image, wherein, The method includes: Obtaining multiple original images with the same shooting scene but different exposure levels; Selecting a reference image from the multiple original images, and performing feature alignment processing on the remaining original images according to the reference image to obtain displacement images of the remaining original images; wherein, from the multiple original images, the original image with a medium exposure level is selected as the reference image; Determining an enhanced image according to the reference image and the displacement images of the remaining original images, wherein the enhanced image is obtained by performing image enhancement processing on a fused image after a downsampling operation, and the fused image is obtained by performing feature fusion on the reference image and the displacement images of the remaining original images; the performing image enhancement processing on the fused image after the downsampling operation includes: Performing a downsampling operation on the fused image to obtain a downsampled image; performing image enhancement processing on the downsampled image, wherein the image enhancement processing is used to align similar features in the downsampled image and enhance the features representing image details in the downsampled image; the performing image enhancement processing on the downsampled image includes: inputting the downsampled image into an image enhancement network to output a network image; wherein the image enhancement network is used to align similar features in the downsampled image and enhance the features representing image details in the downsampled image; performing an upsampling operation on the network image to obtain the enhanced image; Reconstructing an HDR image corresponding to the multiple original images according to the enhanced image.

2. The method according to claim 1, wherein After obtaining the displacement images of the remaining original images and before determining the enhanced image according to the reference image and the displacement images of the remaining original images, it further includes: Performing at least one downsampling feature alignment process, wherein each downsampling feature alignment process performs the following steps: Performing a downsampling operation on the reference feature image and the displacement images of the remaining original images to obtain a downsampled reference feature image and displacement images; wherein the reference feature image is obtained by performing feature extraction on the reference image; Performing feature alignment processing on the downsampled displacement images according to the downsampled reference feature image to obtain the displacement images after the current feature alignment process.

3. The method according to claim 1, wherein, After determining the enhanced image according to the reference image and the displacement images and before reconstructing the HDR image corresponding to the multiple original images according to the enhanced image, it further includes: Performing at least one image enhancement operation; Wherein, each image enhancement operation performs the following steps: Taking the enhanced image determined last time as the reference image of this time, and determining the enhanced image of this time according to the reference image of this time and the displacement images.

4. The method according to any one of claims 1 to 3, wherein, The performing feature alignment processing on the remaining original images according to the reference image to obtain the displacement images of the remaining original images includes: Determining a displacement parameter matrix corresponding to the remaining original images according to the feature similarity between the reference image and each of the remaining original images; Displacing the features of the remaining original images according to the displacement parameter matrix to obtain displacement images.

5. According to the method according to any one of claims 1 to 3, wherein Performing feature alignment processing on the remaining original images according to the reference image to obtain displacement images of the remaining original images includes: Performing feature extraction on the reference image to obtain a reference feature image, and performing feature extraction on each of the remaining original images to obtain an original feature image; Merging the reference feature image and each original feature image respectively to obtain a first merged image corresponding to each original feature image; Inputting the first merged image and the corresponding original feature image into an attention network to output a displacement image of the original feature image.

6. The method according to claim 5, wherein The attention network is used to determine a displacement parameter matrix of the corresponding original feature image according to the first merged image, and use the displacement parameter matrix to displace the features of the corresponding original feature image.

7. The method according to claim 1, wherein The sampling multiple of the downsampling operation is determined according to the computing power of the hardware platform.

8. According to the method described in any one of claims 1 to 3, wherein Determining the fused image in the following manner: Merging the reference feature image obtained by performing feature extraction on the reference image and the displacement images of the remaining original images to obtain a second merged image; Reducing the dimension of the second merged image through a convolutional layer to obtain the fused image.

9. According to the method as claimed in any one of claims 1 to 3, wherein, Reconstructing the HDR image corresponding to the multiple original images according to the enhanced image includes: Performing dimensionality reduction processing on the enhanced image through multiple convolutional layers to obtain the HDR image.

10. The method according to any one of claims 1 to 3, wherein, Obtaining multiple original images with the same shooting scene and different exposure degrees includes: In response to a user's shooting instruction, continuously shooting the multiple original images with different exposure degrees for the same shooting scene through a camera component.

11. According to the method described in any one of claims 1 to 3, wherein, After reconstructing the HDR image corresponding to the multiple original images according to the enhanced image, further includes: Displaying the reconstructed HDR image on a display.

12. A terminal for reconstructing an HDR image, wherein, The terminal includes a processor and a memory, the memory is used to store programs executable by the processor, and the processor is used to read the programs in the memory and execute the steps of the method according to any one of claims 1 to 11.

13. An electronic device for reconstructing an HDR image, wherein, The electronic device includes a camera unit and a control circuit, wherein: The camera unit is used to obtain original images with different exposure degrees; The control circuit includes a processor and a memory, the memory is used to store programs executable by the processor, and the processor is used to read the programs in the memory and execute the steps of the method according to any one of claims 1 to 11.

14. An apparatus for reconstructing an HDR image, wherein, Includes: An image acquisition unit, configured to acquire multiple original images with the same shooting scene and different exposure degrees; A feature alignment unit, configured to select a reference image from multiple original images, and perform feature alignment processing on the remaining original images according to the reference image to obtain displacement images of the remaining original images; A feature enhancement unit, configured to determine an enhanced image according to the reference image and the displacement images of the remaining original images, wherein the enhanced image is obtained by performing image enhancement processing on a fused image after a downsampling operation, and the fused image is obtained by performing feature fusion on the reference image and the displacement images of the remaining original images; Performing image enhancement processing on the fused image after the downsampling operation includes: performing a downsampling operation on the fused image to obtain a downsampled image; performing image enhancement processing on the downsampled image, where the image enhancement processing is used to align similar features in the downsampled image and enhance the features representing image details in the downsampled image; the performing image enhancement processing on the downsampled image includes: inputting the downsampled image into an image enhancement network to output a network image; where the image enhancement network is used to align similar features in the downsampled image and enhance the features representing image details in the downsampled image; performing an upsampling operation on the network image to obtain the enhanced image; A feature reconstruction unit, configured to reconstruct the HDR image corresponding to the multiple original images according to the enhanced image.

15. A non-transitory computer storage medium having a computer program stored thereon, wherein, When the program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.

Images