Image reconstruction method and device and electronic equipment

Abstract

The invention discloses an image reconstruction method and device, electronic equipment, a storage medium and a program product, and belongs to the technical field of image processing. The method comprises the steps that a first image frame sequence is input into a first model, first feature information and at least one piece of second feature information are obtained through the first model, the first image frame sequence comprises a long-exposure first LDR image frame and at least one short-exposure second LDR image frame, the first feature information is the feature information of the first LDR image frame, and the second feature information is the feature information of the second LDR image frame; each piece of second feature information is feature information of one second LDR image frame; performing feature processing on each piece of second feature information based on the first feature information through a first model to obtain third feature information, the first model being obtained based on motion blurred image training; and performing HDR reconstruction based on the first feature information and the third feature information.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to an image reconstruction method, device and electronic equipment. Background Art

[0002] At present, in order to obtain a high dynamic range (HDR) image, an electronic device can obtain a low dynamic range (LDR) image frame with different exposure values ​​in the same scene, and then calculate the motion offset between each LDR image frame and a reference frame through the optical flow method. The LDR image frame and the reference frame are motion aligned through these motion offsets, and then the weight prediction network is used to obtain the fusion weights of the corresponding positions of the motion aligned image. Then, the fusion weights are used to find adjacent frame pixels in the spatial position for fusion to obtain a reconstructed HDR image.

[0003] However, in the above method, in motion scenes, the motion offset obtained by optical flow calculation often has a large error, resulting in the LDR image frame cannot be accurately aligned with the reference frame, thereby causing fusion weight reasoning errors, and then unable to find accurate adjacent frame pixels in spatial position for fusion. As a result, the HDR image reconstructed by the electronic device may have image blurring phenomena, such as image distortion, jitter, and motion ghosting, resulting in poor image quality of the HDR image reconstructed by the electronic device. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image reconstruction method, device and electronic device, which can improve the image quality of HDR images reconstructed by electronic devices in motion scenes.

[0005] In a first aspect, an embodiment of the present application provides an image reconstruction method, which includes: inputting a first image frame sequence into a first model, and obtaining first feature information and at least one second feature information through the first model, wherein the first image frame sequence includes a first LDR image frame with long exposure and at least one second LDR image frame with short exposure, the first feature information is feature information of the first LDR image frame, and each second feature information is feature information of a second LDR image frame; through the first model, based on the first feature information, feature processing is performed on each second feature information to obtain third feature information, and the first model is obtained based on motion blurred image training; based on the first feature information and the third feature information, HDR reconstruction is performed.

[0006] In a second aspect, an embodiment of the present application provides an image reconstruction device, which includes: an acquisition module, a processing module, and an execution module. The acquisition module is used to input a first image frame sequence into a first model, and obtain first feature information and at least one second feature information through the first model. The first image frame sequence includes a first LDR image frame with a long exposure and at least one second LDR image frame with a short exposure. The first feature information is feature information of the first LDR image frame, and each second feature information is feature information of a second LDR image frame. The processing module is used to perform feature processing on each second feature information based on the first feature information through the first model to obtain third feature information, and the first model is obtained based on motion blurred image training. The execution module is used to perform HDR reconstruction based on the first feature information obtained by the acquisition module and the third feature information obtained by the processing module.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0010] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0011] In an embodiment of the present application, a first image frame sequence can be input into a first model, and first feature information and at least one second feature information can be obtained through the first model. The first image frame sequence includes a first low dynamic range LDR image frame with a long exposure and at least one second LDR image frame with a short exposure. The first feature information is feature information of the first LDR image frame, and each second feature information is feature information of a second LDR image frame; then, based on the first feature information, feature processing is performed on each second feature information to obtain third feature information, and the first model is trained based on a motion blurred image; finally, high dynamic range HDR reconstruction is performed based on the first feature information and the third feature information. In this solution, since the first model is trained based on motion blurred images, the first model can learn the characteristic information of motion blurred images, and there will be no motion blur phenomenon in the short exposure LDR image frame, so the first feature information will not contain motion blur feature information; then, when there is motion blur phenomenon in the long exposure LDR image frame, by performing feature processing on the first feature information and each second feature information in at least one second feature information, the first model can identify and remove the characteristic information of the motion blurred image contained in the first feature information through at least one second feature information, so that the third feature information will not contain motion blur feature information, and further, the HDR image reconstructed by the third feature information will not have image blur phenomenon, thereby improving the image quality of the reconstructed HDR image. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is one of the structural schematic diagrams of a model for HDR image reconstruction provided in an embodiment of the present application;

[0013] Figure 2 This is one of the flow charts of an image reconstruction method provided in an embodiment of the present application;

[0014] Figure 3 This is the second flowchart of an image reconstruction method provided in an embodiment of the present application;

[0015] Figure 4 This is the second structural schematic diagram of a model for HDR image reconstruction provided in an embodiment of the present application;

[0016] Figure 5 This is the third flowchart of an image reconstruction method provided in an embodiment of the present application;

[0017] Figure 6 This is a third structural diagram of a model for HDR image reconstruction provided in an embodiment of the present application;

[0018] Figure 7This is a fourth flowchart of an image reconstruction method provided in an embodiment of the present application;

[0019] Figure 8 is a schematic diagram of an upsampling process provided in an embodiment of the present application;

[0020] Fig. 9 This is a fourth structural diagram of a model for HDR reconstruction provided in an embodiment of the present application;

[0021] Fig.10 This is the fifth flowchart of an image reconstruction method provided in an embodiment of the present application;

[0022] Fig.11 is a schematic diagram of a process of obtaining a motion blurred image provided by an embodiment of the present application;

[0023] Fig.12 This is the sixth flowchart of an image reconstruction method provided in an embodiment of the present application;

[0024] Fig.13 This is the seventh flowchart of an image reconstruction method provided in an embodiment of the present application;

[0025] Fig.14 is a schematic diagram of a cosine annealing learning rate decay curve provided in an embodiment of the present application;

[0026] Fig.15 This is a fifth structural diagram of a model for HDR reconstruction provided in an embodiment of the present application;

[0027] Fig.16 This is one of the schematic diagrams of HDR reconstruction effects provided in an embodiment of the present application;

[0028] Fig.17 This is the second schematic diagram of an HDR reconstruction effect provided in an embodiment of the present application;

[0029] Fig.18 This is the third schematic diagram of an HDR reconstruction effect provided in an embodiment of the present application;

[0030] Fig.19 This is a fourth schematic diagram of an HDR reconstruction effect provided in an embodiment of the present application;

[0031] Fig. 20 is a structural schematic diagram of an image reconstruction device provided in an embodiment of the present application;

[0032] Fig.21 This is one of the hardware structure diagrams of an electronic device provided in an embodiment of the present application;

[0033] Fig. 22This is the second schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0035] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0036] The terms "at least one (item)", "at least one of" and the like in the specification and claims of the present application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)".

[0037] The image reconstruction method, device, electronic device, storage medium and program product provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0038] The image reconstruction method, device, and electronic device provided in the embodiments of the present application can be applied in a preview scene, a picture taking scene, or a video taking scene.

[0039] At present, in the field of image processing, the dynamic range of an image refers to the ratio of the highest brightness to the lowest brightness of the image. Compared with LDR images, high dynamic range HDR images can better reflect the visual details of the scene in bright and dark areas. However, the bit depth of the pictures recorded by standard digital imaging devices is not enough to cover the dynamic range of real scenes, resulting in underexposure in dark areas or overexposure in bright areas.

[0040] With the development of high dynamic range technology and neural networks, the traditional method of restoring missing information of low dynamic range images based on statistical features has gradually shifted to reconstructing high dynamic range images through processing time series data by deep learning algorithms. The current mainstream HDR reconstruction method is to capture different low dynamic range image sequences through alternating exposures, and then restore the lost details of each LDR frame based on the adjacent image information of different exposures to reconstruct the HDR image.

[0041] This can be achieved through the following methods 1 and 2:

[0042] Method 1: The electronic device can use the optical flow method to perform explicit motion alignment of the input adjacent LDR image frames with the reference frame, and then use the weight prediction network to obtain the fusion weights of the corresponding positions of the aligned images, and obtain the reconstructed HDR image through weighted averaging and fusion.

[0043] Method 2: The electronic device can implicitly align the features of adjacent image frames through an attention module or deformable convolution, and then input the aligned image features into the reconstruction network to directly output the reconstructed HDR image.

[0044] However, for the above-mentioned method 1, the weighted fusion method is highly dependent on optical flow. In scenes with large-scale motion and object occlusion, there are often large errors in the optical flow calculation, resulting in the inability to accurately align adjacent frames with the reference frame, which in turn causes fusion weight reasoning errors and inability to find suitable adjacent frame pixels for fusion in spatial position. The HDR image reconstructed by the electronic device may have image blurring phenomena, such as image distortion, jitter, and motion ghosting, resulting in poor image quality of the HDR image reconstructed by the electronic device; for the above-mentioned method 2, since the deformable convolution is relatively complex, the computational complexity is too high, and the deformable convolution requires more model parameters. Therefore, the deformable convolution often requires larger computing resources, resulting in larger computing resource consumption of the electronic device.

[0045] In the image reconstruction method, device and electronic device provided in the embodiments of the present application, since the first model is obtained by training based on motion blurred images, the first model can learn the characteristic information of the motion blurred images, and there is no motion blur phenomenon in the short exposure LDR image frame, so the first feature information does not contain motion blur feature information; then, when there is motion blur phenomenon in the long exposure LDR image frame, by performing feature processing on the first feature information and each second feature information in at least one second feature information, the first model can identify and remove the characteristic information of the motion blurred image contained in the first feature information through at least one second feature information, so that the third feature information does not contain motion blur feature information, and further, the HDR image reconstructed by the third feature information will not have image blur phenomenon, thereby improving the image quality of the reconstructed HDR image.

[0046] The image reconstruction method provided in the embodiment of the present application may be executed by an image reconstruction device, which may be an electronic device or a functional module in an electronic device. The technical solution provided in the embodiment of the present application is described below using an electronic device as an example.

[0047] The present application embodiment provides a first model, Figure 1 A schematic diagram of the structure of a first model provided in an embodiment of the present application is shown. Figure 1 As shown, the first model 10 includes: a feature extraction sharing module 20, a feature propagation module 30 and an upsampling fusion module 40. The first model can be used to reconstruct an HDR image.

[0048] In an embodiment of the present application, the feature extraction sharing module 20 can be used to obtain high-dimensional feature information of the first LDR image frame and at least one high-dimensional feature information of at least one second LDR image frame; the optical flow alignment module 30 can be used to perform feature alignment on at least one high-dimensional feature information of at least one second LDR image frame based on the high-dimensional feature information of the first LDR image frame; the feature propagation module 30 can be used to obtain high-dimensional feature information without blurring based on the high-dimensional feature information of the first LDR image frame and at least one high-dimensional feature information of at least one second LDR image frame after feature alignment; the upsampling fusion module 40 can be used to upsample the high-dimensional feature information of the first LDR image frame and the high-dimensional feature information without blurring to obtain high-dimensional feature information with high resolution, and reconstruct the high-dimensional feature information with high resolution to obtain an HDR image.

[0049] The embodiment of the present application proposes an HDR image reconstruction algorithm based on the Raw domain, namely the above-mentioned first model, which can solve the problem of large resource consumption caused by full-size LDR image input under the current HDR image reconstruction method, thereby improving the HDR reconstruction efficiency and increasing the HDR image output frame rate; and can handle the problems of excessive noise and motion blur in HDR reconstructed images, avoiding the phenomenon of poor processing effects on real-scene HDR images, such as image distortion, jitter and motion ghosting.

[0050] Specifically, the electronic device can input low-resolution LDR images with different exposures into the first model, generate high-dimensional feature information of the LDR images through the optical flow alignment module and the shared feature extraction module in the first model, and comprehensively consider the noise similarity and exposure time differences of the multi-exposure images, and use the feature extraction sharing module and the temporal feature propagation module to reduce the noise level of the HDR reconstructed image, and perform HDR reconstruction on the long exposure LDR image frame based on the long exposure LDR image frame as a reference frame. The first model achieves a low noise level without losing the details of the overexposed area and without obvious ghosting.

[0051] The present application embodiment provides an image reconstruction method. Figure 2 FIG. 4 is a flowchart of an image reconstruction method provided by an embodiment of the present application. Figure 2 As shown, the image reconstruction method provided in the embodiment of the present application may include the following steps 201 to 203.

[0052] Step 201: The electronic device inputs a first image frame sequence into a first model, and obtains first feature information and at least one second feature information through the first model.

[0053] In an embodiment of the present application, the above-mentioned first image frame sequence includes a first LDR image frame with long exposure and at least one second LDR image frame with short exposure, the above-mentioned first feature information is feature information of the first LDR image frame, and each second feature information in the above-mentioned at least one second feature information is feature information of a second LDR image frame.

[0054] In the embodiment of the present application, the at least one second LDR image frame corresponds one-to-one to the at least one second feature information.

[0055] In the embodiment of the present application, the first model is obtained based on motion blurred image training.

[0056] In the embodiment of the present application, the first LDR image frame is a reference image, and the at least one second LDR image frame is a non-reference image.

[0057] It should be noted that, since motion blur may only exist in long-exposure LDR image frames, by using long-exposure LDR image frames as reference frames, electronic devices can identify possible blur phenomena in long-exposure LDR image frames based on short-exposure LDR image frames without motion blur, thereby reconstructing an HDR image without blur phenomena.

[0058] In the embodiment of the present application, the first LDR image frame and the at least one second LDR image frame are LDR image frames of the same shooting scene.

[0059] Optionally, in the embodiment of the present application, the above-mentioned shooting scene can be any one of the following: a motion shooting scene, a static shooting scene or a high-exposure shooting scene, etc.

[0060] Optionally, in an embodiment of the present application, the electronic device may collect LDR image frames of different exposures through an image sensor in the electronic device, and then combine the LDR image frames of different exposures to obtain the above-mentioned first image frame sequence.

[0061] Optionally, in an embodiment of the present application, the electronic device may collect LDR image frames of different exposures through the above-mentioned image sensor, and then, respectively divide the number of pixels of the LDR image frames of different exposures by the first exposure coefficient to obtain LDR image frames of different exposures after brightness alignment, and combine the LDR image frames of different exposures after brightness alignment to obtain the above-mentioned first image frame sequence.

[0062] Optionally, in an embodiment of the present application, the above-mentioned first exposure coefficient may be an exposure coefficient of a short-exposure LDR image frame; or, the above-mentioned first exposure coefficient may be an exposure coefficient of a long-exposure LDR image frame; or, the above-mentioned exposure coefficient may be an exposure coefficient preset by the electronic device.

[0063] For example, taking an LDR image frame as an example, the electronic device may divide the number of pixels of the LDR image frame by the first exposure coefficient to obtain the LDR image frame after brightness adjustment; this may be specifically achieved by the following formula (1).

[0064]

[0065] in, is the LDR image frame after brightness adjustment, I i is the number of pixels in an LDR image frame, e i is the first exposure factor.

[0066] It should be noted that the above LDR image frames with different exposures can all be converted into LDR image frames after brightness adjustment through the above formula (1), which will not be described again here to avoid repetition.

[0067] Optionally, in the embodiment of the present application, the image sensor may be a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor.

[0068] Optionally, in an embodiment of the present application, each LDR image frame in the first image frame sequence is an LDR image frame in an original RAW format.

[0069] It should be noted that all feature information provided in the embodiments of the present application may be feature information maps, and the size of each feature information map is the same.

[0070] Exemplarily, if the size of the first characteristic information map is 3×3, the size of the at least one second characteristic information map is also 3×3.

[0071] Optionally, in an embodiment of the present application, the above-mentioned feature information may include at least one of the following: color feature information, texture feature information, shape feature information, spatial relationship feature information, etc.

[0072] Optionally, in the embodiment of the present application, the first model may be stored in an electronic device; or the first model may be stored in a server. The specific method may be determined according to actual use requirements, and the embodiment of the present application does not limit this.

[0073] Optionally, in the embodiment of the present application, combined with Figure 2 ,like Figure 3 As shown, the "obtaining first feature information and at least one second feature information through the first model" in the above step 201 can be specifically implemented through the following steps 201a and 201b.

[0074] Step 201a: The electronic device extracts features from the first LDR image frame using the first model to obtain first feature information, and extracts features from at least one second LDR image frame to obtain at least one sub-feature information.

[0075] In an embodiment of the present application, the electronic device can input each LDR image frame in the first image frame sequence into the feature extraction sharing module in the first model in sequence to obtain first feature information and at least one sub-feature information.

[0076] Optionally, in the embodiment of the present application, the above-mentioned feature extraction sharing module may be a residual block.

[0077] For example, Figure 4As shown, taking the first image frame sequence including three LDR image frames as an example, the three LDR image frames are respectively two short-exposure LDR image frames and one long-exposure LDR image frame, and the arrangement order of the three LDR image frames in the first image frame sequence is short-exposure LDR image frame, long-exposure LDR image frame, and short-exposure LDR image frame.

[0078] The electronic device can input the first short-exposure LDR image frame into the first convolution layer 201 in the feature extraction sharing module 20, and perform convolution processing on the first short-exposure LDR image frame through the first convolution layer 201 to obtain first sub-feature information; input the first sub-feature information into the second convolution layer 202 in the feature extraction sharing module, and perform convolution processing on the first sub-feature information through the second convolution layer 202 to obtain second sub-feature information; input the second sub-feature information into the third convolution layer 203 in the feature extraction sharing module, and perform convolution processing on the second sub-feature information through the third convolution layer 203 to obtain third sub-feature information; then perform feature fusion processing on the third sub-feature information and the first sub-feature information to obtain a first sub-feature information graph.

[0079] Then, the electronic device can input the long exposure LDR image frame into the first convolution layer 201 in the feature extraction sharing module 20, perform convolution processing on the long exposure LDR image frame through the first convolution layer 201, and obtain fourth sub-feature information; input the fourth sub-feature information into the second convolution layer 202 in the feature extraction sharing module, perform convolution processing on the fourth sub-feature information through the second convolution layer 202, and obtain fifth sub-feature information; input the fifth sub-feature information into the third convolution layer 203 in the feature extraction sharing module, perform convolution processing on the fifth sub-feature information through the third convolution layer 203, and obtain sixth sub-feature information; then perform feature fusion processing on the fourth sub-feature information and the sixth sub-feature information to obtain a first feature information map.

[0080] Finally, the electronic device can input the second short exposure LDR image frame into the first convolution layer 201 in the feature extraction sharing module 20, and perform convolution processing on the second short exposure LDR image frame through the first convolution layer 201 to obtain seventh sub-feature information; input the seventh sub-feature information into the second convolution layer 202 in the feature extraction sharing module, and perform convolution processing on the seventh sub-feature information through the second convolution layer 202 to obtain eighth sub-feature information; input the eighth sub-feature information into the third convolution layer 203 in the feature extraction sharing module, and perform convolution processing on the eighth sub-feature information through the third convolution layer 203 to obtain ninth sub-feature information; then perform feature fusion processing on the ninth sub-feature information and the seventh sub-feature information to obtain a second sub-feature information graph.

[0081] Optionally, in the embodiment of the present application, the number of convolution kernels in each convolution layer in the above-mentioned feature extraction shared module is 1, and the convolution dimension of the convolution kernel is 3; or, the number of convolution kernels in each convolution layer in the above-mentioned feature extraction shared module is 2, and the convolution dimension of the convolution kernel is 3. The specific number can be determined according to actual use requirements, and the embodiment of the present application is not limited.

[0082] It should be noted that the above-mentioned first image frame sequence including 3 LDR image frames is only an example listed in the present application. In actual implementation, the above-mentioned first image frame sequence may include more LDR image frames with different exposures, and the characteristic information of each LDR image frame in the first image frame sequence can be obtained through the above-mentioned embodiment; to avoid repetition, it will not be repeated here.

[0083] Optionally, in an embodiment of the present application, the feature fusion may be any one of the following: Concat feature fusion or Addition feature fusion.

[0084] Exemplarily, the above-mentioned concatenated feature fusion is to directly concatenate two feature vectors. For example, if the dimensions of the two features x and y are p and q respectively, the dimension of the output feature z is p+q.

[0085] As another example, the above-mentioned addition feature fusion is to combine two feature vectors into a complex vector. For example, for feature vector x and feature vector y, z=x+iy, where i is an imaginary unit and z is the fused feature vector.

[0086] It can be understood that, when the number of the short-exposure LDR image frames is greater than 2, the electronic device can obtain the feature information corresponding to each short-exposure LDR image frame through the above embodiment.

[0087] Optionally, in the embodiment of the present application, the feature extraction process of the first feature information and at least one sub-feature information can be implemented by the following formula (2).

[0088]

[0089] Among them, F other is a sub-feature information, R other is a second LDR image frame, F ref is the first feature information, R ref is the first LDR image frame, and D is the residual block.

[0090] Step 201b: The electronic device performs feature alignment on at least one sub-feature information based on the first feature information and at least one motion offset matrix to obtain at least one second feature information.

[0091] In an embodiment of the present application, the above-mentioned motion offset matrix is ​​calculated based on pixel point information of the first LDR image frame, and each motion offset matrix in the above-mentioned at least one motion offset matrix represents a motion offset of a second LDR image frame relative to the first LDR image frame.

[0092] Exemplarily, the above pixel point information may be position information of the pixel point.

[0093] In an embodiment of the present application, the electronic device can subtract the position information of each pixel point in each second LDR image frame from the position information of each pixel point in the first LDR image frame through the optical flow alignment module in the first model to obtain the above-mentioned at least one motion offset matrix.

[0094] Exemplarily, taking at least one second LDR image frame as two second LDR image frames as an example, the electronic device may subtract the position information of each pixel point in the first LDR image frame of the two second LDR image frames from the position information of each pixel point in the first LDR image frame through the first model to obtain a first motion offset matrix; then the first model may subtract the position information of each pixel point in the second LDR image frame of the two second LDR image frames from the position information of each pixel point in the first LDR image frame to obtain a second motion offset matrix, so as to obtain the at least one motion offset matrix mentioned above.

[0095] It can be understood that each of the at least one motion offset matrix contains a motion offset value of position information of each pixel in the second LDR image frame relative to position information of each pixel in the first LDR image frame.

[0096] It should be noted that the size of the at least one motion offset matrix is ​​the same as the size of the first feature information map and the at least one second feature information map.

[0097] In an embodiment of the present application, after obtaining at least one motion offset matrix, the electronic device may respectively add the position information of each second sub-feature information in the at least one second sub-feature information and each motion offset in the motion offset matrix corresponding to each second LDR image in the at least one second LDR image to obtain at least one feature information map after feature alignment, that is, the at least one second feature information mentioned above.

[0098] Exemplarily, taking at least one second sub-feature information map as two second sub-feature information maps and at least one motion offset matrix as two motion offset matrices, the first model can add the position information corresponding to each second sub-feature information in the first second sub-feature information map of the two second sub-feature information maps to each motion offset in the first motion offset matrix in accordance with the position information of each motion offset in the first motion offset matrix in the at least one motion offset matrix, to obtain the first feature-aligned second sub-feature information map; then the first model can add the position information corresponding to each second sub-feature information in the second second sub-feature information map of the two second sub-feature information maps to each motion offset in the second motion offset matrix in accordance with the position information of each motion offset in the second motion offset matrix in the at least one motion offset matrix, to obtain the second feature-aligned second sub-feature information map, so as to obtain the above-mentioned at least one second feature information. Specifically, it can be expressed by the following formula (3).

[0099] I(x',y')=I(x+Δ x ,y+Δ y ) (3)

[0100] Wherein, I(x',y') is the second sub-feature information graph of feature alignment, I(x,y) is the second sub-feature information graph without feature alignment, x is the X-axis coordinate corresponding to each feature information of the second sub-feature information graph without feature alignment, y is the y-axis coordinate corresponding to each feature information of the second sub-feature information graph without feature alignment, Δ x is the motion offset on the X-axis, Δ y The movement offset on the Y axis.

[0101] In an embodiment of the present application, the electronic device performs feature alignment on at least one sub-feature information based on the first feature information and at least one motion offset matrix, so that the image content of at least one second LDR image frame corresponds one-to-one with the image content of the first LDR image frame, thereby reconstructing a clear HDR image.

[0102] Step 202: The electronic device performs feature processing on each second feature information in at least one second feature information based on the first feature information by using the first model to obtain third feature information.

[0103] In the embodiment of the present application, the third feature information includes the first feature information after feature processing and at least one second feature information after feature processing.

[0104] In an embodiment of the present application, the electronic device can input the first feature information and each second feature information in the at least one second feature information into the first model in pairs to perform feature processing on the first feature information and each second feature information in the at least one second feature information to obtain third feature information.

[0105] Optionally, in the embodiment of the present application, combined with Figure 2 ,like Figure 5 As shown, the above step 202 can be specifically implemented through the above step 202a and step 202c.

[0106] Step 202a: The electronic device performs convolution processing on the first feature information and the first second feature information through the feature propagation module in the first model to obtain first fused feature information.

[0107] In the embodiment of the present application, the electronic device can perform convolution processing on the first feature information and the first second feature information through the first residual block in the feature propagation module to obtain the first fused feature information.

[0108] Exemplarily, the first model may perform convolution processing on the first feature information and the first second feature information through at least one convolution layer in the first residual block to obtain first fused feature information.

[0109] Step 202b: The electronic device performs convolution processing on the first feature information, the i-th second feature information and the fourth feature information through the feature propagation module in the first model to obtain the i-th fused feature information.

[0110] In the embodiment of the present application, the fourth feature information is the fused feature information corresponding to the i-1th second feature information, i∈[2,N], i is an integer, and N is the number of at least one second feature information.

[0111] In an embodiment of the present application, the electronic device can perform convolution processing on the first feature information, the i-th second feature information and the fourth feature information through the i-th residual block in the feature propagation module to obtain the i-th fused feature information.

[0112] For example, Figure 6As shown, taking the case where the number of at least one second feature information map is 3 second feature information maps as an example, the first model inputs the first feature information map and the first second feature information map into the first convolution layer 3010 in the first residual block 301 of the feature propagation module 30, and performs convolution processing on the first feature information and the first second feature information through the first convolution layer 3011 to obtain the first feature information map; the first feature information map is input into the second convolution layer 3011 in the first residual block 301, and performs convolution processing on the first feature information map through the second convolution layer 3011 to obtain the second feature information map; the second feature information map is input into the third convolution layer 3012 in the first residual block 301, and performs convolution processing on the second feature information map through the third convolution layer 3012 to obtain the third feature information map; then the third feature information map is subjected to feature fusion processing with the first feature information map to obtain the first fused feature information map, i.e., the above-mentioned first fused feature information.

[0113] Then, the first model inputs the first fused feature information map, the second second feature information map and the first feature information map into the first convolution layer 3020 in the second residual block 302 in the feature propagation module 30, and performs convolution processing on the first fused feature information map, the second second feature information map and the first feature information map through the first convolution layer 3020 to obtain a fourth feature information map; inputs the fourth feature information map into the second convolution layer 3021 in the second residual block 302, performs convolution processing on the fourth feature information map to obtain a fifth feature information map, inputs the fifth feature information map into the third convolution layer 3022 in the second residual block 302, performs convolution processing on the fifth feature information map to obtain a sixth feature information map; and fuses the fourth feature information map with the sixth feature information map to obtain a second fused feature information map.

[0114] Finally, the second fused feature information map, the third second feature information map and the first feature information map are input into the first convolution layer 3030 in the third residual block 303 in the feature propagation module 30, and the second fused feature information map, the third second feature information map and the first feature information map are convolved to obtain a seventh feature information map; the seventh feature information map is input into the second convolution layer 3031 in the third residual block 303, and the seventh feature information map is convolved to obtain an eighth feature information map; the eighth feature information map is input into the third convolution layer 3032 in the third residual block 303, and the eighth feature information map is convolved to obtain a ninth feature information map; and the seventh feature information map and the ninth feature information map are feature fused to obtain the third fused feature information.

[0115] In the embodiment of the present application, the number of residual blocks in the above-mentioned feature propagation module is the same as the number of second feature information.

[0116] It should be noted that the number of the at least one second feature information graph mentioned above is 3 second feature information graphs, which is only an example listed in the present application. In actual implementation, the at least one second feature information graph mentioned above may include more second feature information graphs, all of which can obtain the Nth fused feature information through the above-mentioned embodiments; to avoid repetition, they will not be repeated here.

[0117] Step 202c: The electronic device determines the Nth fused feature information as the third feature information.

[0118] Exemplarily, the Nth fused feature information is the third feature information.

[0119] Optionally, in the embodiment of the present application, the process of obtaining the third characteristic information can be implemented by the following formula (4).

[0120]

[0121] in, is the third feature information, R is the residual block in the feature propagation module, and F ref is the first characteristic information, is the second characteristic information, It is a fusion feature information.

[0122] It can be understood that, according to the above embodiment, the third feature information includes the first feature information after feature processing and at least one second feature information after feature processing.

[0123] In an embodiment of the present application, by performing feature processing on the first feature information and each of the second feature information in at least one second feature information, the first model can identify and remove the feature information of the motion blurred image contained in the first feature information through at least one second feature information, so that the third feature information will not contain the feature information of motion blur, and further, the HDR image reconstructed by the third feature information will not have the phenomenon of image blur, thereby improving the image quality of the reconstructed HDR image.

[0124] Step 203: The electronic device performs HDR reconstruction based on the first feature information and the third feature information.

[0125] In an embodiment of the present application, the electronic device fuses the first feature information and the third feature information to obtain fifth feature information, and upsamples the fifth feature information to obtain high-resolution fifth feature information, and then performs HDR reconstruction using the high-resolution fifth feature information to obtain a first HDR image.

[0126] Optionally, in an embodiment of the present application, the resolutions of the LDR image frames in the above-mentioned first image frame sequence are all less than or equal to the first resolution.

[0127] For example, in combination Figure 2 ,like Figure 7 As shown, the above step 203 can be specifically implemented through the following steps 203a to 203c.

[0128] Step 203a: The electronic device performs convolution processing on the first feature information and the third feature information through the residual block in the upsampling fusion module in the first model to obtain fifth feature information.

[0129] Exemplarily, the electronic device can input the first feature information and the third feature information into the first convolution layer in the residual block in the upsampling fusion module through the first model, and perform convolution processing on the first feature information and the third feature information through the first convolution layer to obtain the tenth feature information; then input the tenth feature information into the second convolution layer in the residual block in the upsampling fusion module, and perform convolution processing on the tenth feature information through the second convolution layer to obtain the eleventh feature information; then input the eleventh feature information into the third convolution layer in the residual block in the upsampling fusion module, and perform convolution processing on the eleventh feature information through the third convolution layer to obtain the twelfth feature information; and perform feature fusion processing on the tenth feature information and the twelfth feature information to obtain the above-mentioned fifth feature information.

[0130] Optionally, in the embodiment of the present application, the process of obtaining the fifth characteristic information can be implemented by the following formula (5).

[0131]

[0132] Among them, F con is the fifth feature information, D is the residual block in the upsampling fusion module, and F ref is the first characteristic information, It is the third characteristic information.

[0133] Step 203b: The electronic device performs linear interpolation processing on the fifth feature information through the upsampling layer in the upsampling fusion module to obtain sixth feature information, and performs upsampling processing on the fifth feature information through the pixel_shuffle layer in the upsampling fusion module to obtain seventh feature information.

[0134] In the embodiment of the present application, the resolutions of the sixth characteristic information and the seventy-fourth characteristic information are both greater than the first resolution.

[0135] Optionally, in the embodiment of the present application, the pixel_shuffle layer upsampling the fifth feature information may specifically be inputting the fifth feature information into the pixel_shuffle layer, and the pixel_shuffle layer may convolve the fifth feature information to obtain r 2 The feature map of the channel, r 2 The size of the feature map of the channels is the same as the size of the first LDR image frame, and then 2 The feature graphs of the channels are periodically screened to obtain multiple feature information, and the multiple feature information are inserted into corresponding positions in the fifth feature information according to the position information of the multiple feature information, so as to obtain the seventh feature information, wherein r is the upsampling factor.

[0136] For example, Figure 8 As shown, the electronic device can convolve the fifth feature information map through the pixel_shuffle layer to obtain feature maps of 4 channels, and then periodically filter the feature maps of the 4 channels respectively to obtain multiple feature information, and insert the multiple feature information into corresponding positions in the fifth feature information map according to the position information of the multiple feature information, so as to obtain the seventh feature information map.

[0137] Step 203c: The electronic device performs convolution processing on the sixth feature information and the seventh feature information through a reconstruction layer in an upsampling fusion module to obtain a first HDR image.

[0138] In an embodiment of the present application, the electronic device may input the sixth feature information and the seventh feature information into two convolutional layers in the reconstruction layer, perform convolution processing on the sixth feature information and the seventh feature information, and obtain a first HDR image.

[0139] For example, Fig. 9 As shown, the electronic device can input the first feature information map and the third feature information map into the first convolution layer 4010 in the residual block 401 in the upsampling fusion module 40 through the first model, and perform convolution processing on the first feature information map and the third feature information map through the first convolution layer 4010 to obtain the tenth feature information map; then input the tenth feature information map into the second convolution layer 4011 in the residual block 401 in the upsampling fusion module 40, and perform convolution processing on the tenth feature information map through the second convolution layer 4011 to obtain the eleventh feature information map; then input the eleventh feature information map into the third convolution layer 4012 in the residual block 401 in the upsampling fusion module 40, and perform convolution processing on the eleventh feature information map through the third convolution layer 4012 to obtain the twelfth feature information map; and perform feature fusion processing on the tenth feature information map and the twelfth feature information map to obtain the above-mentioned fifth feature information map.

[0140] Then, the first model inputs the fifth feature information map into the upsampling layer 402 and the pixel_shuffle layer 403 in the above-mentioned sampling fusion module 40 respectively, performs linear interpolation processing on the fifth feature information map through the upsampling layer 402 to obtain the sixth feature information map; and performs upsampling processing on the fifth feature information map through the pixel_shuffle layer 403 to obtain the seventh feature information map.

[0141] Finally, the sixth feature information map and the seventh feature information map are input into the first convolution layer 4040 in the reconstruction layer 404 in the sampling fusion module 40, and the sixth feature information map and the seventh feature information map are convolved to obtain the eighth feature information map; the eighth feature information is then input into the second convolution layer 4041 in the reconstruction layer 404, and the eighth feature information map is convolved to obtain the first HDR image.

[0142] Optionally, in the embodiment of the present application, the process of obtaining the first HDR image can be specifically implemented by the following formula (6).

[0143]

[0144] in, is the first HDR image, Conv is the convolutional layer of the reconstruction layer, UP is the upsampling layer, Shuffle is the pixel_shuffle layer, and F con It is the fifth characteristic information.

[0145] In the embodiment of the present application, since there is less feature information in the low-resolution LDR image, the electronic device can obtain the feature information of the low-resolution LDR image through fewer computing resources, thereby reconstructing the high-resolution HDR image through the low-resolution LDR image, reducing the resource consumption of the electronic device for reconstructing the HDR image.

[0146] In the image reconstruction method provided in the embodiment of the present application, the electronic device can input a first image frame sequence into a first model, and obtain first feature information and at least one second feature information through the first model, wherein the first image frame sequence includes a first low dynamic range LDR image frame with a long exposure and at least one second LDR image frame with a short exposure, the first feature information is feature information of the first LDR image frame, and each second feature information is feature information of a second LDR image frame; then, through the first model, based on the first feature information, feature processing is performed on each second feature information to obtain third feature information, and the first model is trained based on a motion blurred image; finally, high dynamic range HDR reconstruction is performed based on the first feature information and the third feature information. In this solution, since the first model is trained based on motion blurred images, the first model can learn the characteristic information of motion blurred images, and there will be no motion blur phenomenon in the short exposure LDR image frame, so the first feature information will not contain motion blur feature information; then, when there is motion blur phenomenon in the long exposure LDR image frame, by performing feature processing on the first feature information and each second feature information in at least one second feature information, the first model can identify and remove the characteristic information of the motion blurred image contained in the first feature information through at least one second feature information, so that the third feature information will not contain motion blur feature information, and further, the HDR image reconstructed by the third feature information will not have image blur phenomenon, thereby improving the image quality of the HDR image reconstructed by the electronic device.

[0147] Optionally, in the embodiment of the present application, combined with Figure 2 ,like Fig.10 As shown, before the above step 201, the image reconstruction method provided in the embodiment of the present application also includes the following steps 301 to 303.

[0148] Step 301: The electronic device acquires a second image frame sequence.

[0149] In the embodiment of the present application, the second image frame sequence includes long-exposure LDR image frames and short-exposure LDR image frames.

[0150] Optionally, in an embodiment of the present application, the second image frame sequence may be acquired by the electronic device through an image sensor; or the second image frame sequence may be acquired through HDR image frame processing.

[0151] Exemplarily, the second image frame sequence can be obtained by HDR image frame processing, specifically, the electronic device can acquire HDR image frames of different exposures, and then process each of the HDR image frames of different exposures by means of an inverse camera response function (CRF) curve, increasing exposure, adding noise, etc., and degenerate each of the HDR image frames of different exposures into a multiple exposure LDR image frame, and then combine the multiple exposure LDR image frames to obtain the second image frame sequence.

[0152] Step 302: The electronic device performs summing and averaging processing on pixel point information of two adjacent LDR image frames in the M LDR image frames in the second image frame sequence to obtain at least one motion blurred image, where M is an integer greater than 1.

[0153] In the embodiment of the present application, the electronic device can obtain a motion blurred image by using the following formula 7.

[0154] R i =(τ(X i )+τ(X i+1 )) / 2 (7)

[0155] Among them, R i is a motion blurred image, τ(X i ) is the first LDR image frame of two adjacent LDR image frames in the M LDR image frames, τ(X i+1 ) is the second LDR image frame of two adjacent LDR image frames in the M LDR image frames.

[0156] For example, Fig.11 As shown, the electronic device can add the pixel value of each pixel point in the first LDR image frame of two adjacent LDR image frames and the pixel value of each pixel point in the second LDR image frame of two adjacent LDR image frames, and combine the pixel values ​​and pixel points after the addition to obtain a new LDR image frame, and then divide the pixel value of each pixel point in the new LDR image frame by 2 to obtain the above-mentioned motion blurred image.

[0157] It should be noted that, for every two adjacent LDR image frames in the M LDR image frames, the electronic device can obtain the motion blurred image corresponding to every two adjacent LDR image frames through the above formula 7 to obtain at least one motion blurred image. To avoid repetition, it will not be repeated here.

[0158] Step 303: The electronic device performs model training based on the second image frame sequence and at least one motion blurred image to obtain a first model.

[0159] In an embodiment of the present application, the electronic device can input a second image frame sequence and at least one motion blurred image into an initial model to obtain the second HDR image described below, and then train the initial model through the loss value between the second HDR image and the reference HDR image to obtain the above-mentioned first model.

[0160] In an embodiment of the present application, the electronic device can train the first model through motion blurred images, so the first model can learn the characteristic information of the motion blurred images; then, when there is motion blur in the long exposure LDR image frame, by performing feature processing on the first feature information and each second feature information in at least one second feature information, the first model can identify and remove the characteristic information of the motion blurred image contained in the first feature information through at least one second feature information, so that the third feature information will not contain the characteristic information of motion blur, and further, the HDR image reconstructed by the third feature information will not have the phenomenon of image blur, thereby improving the image quality of the reconstructed HDR image.

[0161] Optionally, in the embodiment of the present application, combined with Fig.10 ,like Fig.12 As shown, the above step 303 can be specifically implemented through the following steps 303a to 303c.

[0162] Step 303a: The electronic device performs HDR reconstruction on the second image frame sequence and at least one motion blurred image by using the initial model to obtain a second HDR image.

[0163] Optionally, in an embodiment of the present application, the electronic device may perform brightness alignment processing on the second image frame sequence and at least one motion blurred image to obtain the second image frame sequence after brightness alignment processing and at least one motion blurred image after brightness alignment processing.

[0164] It should be noted that the specific process of the brightness alignment processing can be found in the above embodiments, and will not be described again here to avoid repetition.

[0165] Optionally, in an embodiment of the present application, after obtaining the second image frame sequence after brightness alignment processing and at least one motion blurred image after brightness alignment processing, the electronic device may perform pixel channel splitting processing on the second image frame sequence after brightness alignment processing and the at least one motion blurred image after brightness alignment processing, respectively, to obtain an RGGB first four-channel image corresponding to each LDR image frame in the second image frame sequence after brightness alignment processing and an RGGB second four-channel image corresponding to each motion blurred image in at least one motion blurred image.

[0166] For example, the electronic device may extract the pixel value of each R channel in an LDR image frame after the brightness alignment processing, and splice the pixel value of each R channel into an R channel image; then the electronic device may extract the pixel value of each G channel in an LDR image frame after the brightness alignment processing. r The pixel value of each G r The pixel values ​​of the channels are concatenated into a G r channel image; the electronic device can then extract each G in an LDR image frame after brightness alignment processing b The pixel value of each G b The pixel values ​​of the channels are concatenated into a G b channel image; finally, the electronic device can extract the pixel value of each B channel in an LDR image frame after brightness alignment processing, and splice the pixel value of each B channel into a B channel image to obtain the above-mentioned RGGB first four-channel image.

[0167] It should be noted that each LDR image frame in the above-mentioned second image frame sequence and each motion blurred image in at least one motion blurred image can obtain the RGGB four-channel image corresponding to each image frame through the above-mentioned embodiment. To avoid repetition, it will not be repeated here.

[0168] Optionally, in an embodiment of the present application, the electronic device may input the at least one RGGB first four-channel image and the at least one RGGB second four-channel image into an initial model, and perform image reconstruction processing on the at least one RGGB first four-channel image and the at least one RGGB second four-channel image through the initial model to obtain a second HDR image.

[0169] It should be noted that the process of performing image reconstruction processing on at least one RGGB first four-channel image and at least one RGGB second four-channel image through the initial model to obtain the second HDR image can be found in the above embodiment in detail. To avoid repetition, it will not be repeated here.

[0170] Step 303b: The electronic device calculates a loss value through the initial model based on the first brightness information of the second HDR image and the second brightness information of the reference HDR image.

[0171] In the embodiment of the present application, the first brightness information may be the brightness value of the second HDR image; the second brightness information may be the brightness value of the reference HDR image.

[0172] Optionally, in an embodiment of the present application, the reference HDR image may be an HDR image belonging to the same shooting scene as the second HDR image; or the reference HDR image may be any HDR image.

[0173] Optionally, in the embodiment of the present application, combined with Fig.12 ,like Fig.13 As shown, the above step 303b can be specifically implemented through the following steps 401 and 402.

[0174] Step 401: The electronic device performs tone mapping processing on the second HDR image based on the first brightness information through the initial model to obtain third brightness information, and performs tone mapping processing on the reference HDR image based on the second brightness information to obtain fourth brightness information.

[0175] In the embodiment of the present application, the electronic device can obtain the third brightness information through the following formula (8).

[0176]

[0177] in, is the third brightness information, μ is a constant of 5000, and H is the second HDR image.

[0178] It should be noted that the electronic device can obtain the fourth brightness information through the above formula (8), which will not be described here to avoid repetition.

[0179] Step 402: The electronic device calculates the difference between the third brightness information and the fourth brightness information to obtain a loss value.

[0180] In the embodiment of the present application, the electronic device can obtain the above loss value through the following formula (9).

[0181]

[0182] Among them, Loss is the loss value, is the third brightness information, It is the fourth brightness information.

[0183] In an embodiment of the present application, the electronic device can train the initial model by calculating the above-mentioned loss value, so that the trained initial model can better identify motion blurred images to ensure that there is no motion blur in the reconstructed HDR image.

[0184] Step 303c: The electronic device updates the model parameters of the initial model based on the loss value to obtain a first model.

[0185] In an embodiment of the present application, the electronic device can update the model parameters of the initial model based on the loss value and the learning rate to obtain the first model.

[0186] Exemplarily, the electronic device may multiply the loss value by the learning rate to obtain an updated loss value, and then update the model weights of the initial model using the updated loss value to obtain the first model.

[0187] Optionally, in the embodiment of the present application, the learning rate may be preset by the electronic device. The updating method of the learning rate may be represented by the following formula (10).

[0188]

[0189] Among them, η t is the learning rate, is the maximum value of the learning rate, set to 1 e-4 , is the minimum value of the learning rate, set to 1 e-6 , T cur is the current cycle number, T i is the total number of cycles.

[0190] In the embodiment of the present application, the electronic device can use the Adam optimizer to update the model weights when the gradient is returned. Use the Earlystopping method to supervise the model convergence, and set the validation set loss value to no more than 1 within 3 cycles. e-3 Stop the iteration of the model when , to prevent the model from overfitting. Then, when the model reaches the validation set loss value of no more than 1 within 3 cycles, e-3 The iteration of the model is stopped when , or the iteration of the model is stopped after the total number of cycles is reached, to obtain the first model.

[0191] In an embodiment of the present application, the electronic device can calculate a loss value based on the first brightness information of the second HDR image and the second brightness information of the reference HDR image, and then use the loss value to train the model parameters of the first model, so that the first model can learn the characteristic information of the motion blurred image, and then when there is motion blur in the long exposure LDR image frame, the trained first model can avoid the motion blur in the reconstructed HDR image.

[0192] An embodiment of the present application provides a model training method, and the model training method provided by the embodiment of the present application may include the following steps 501 to 503.

[0193] Step 501: The electronic device acquires a second image frame sequence.

[0194] In the embodiment of the present application, the second image frame sequence includes long-exposure LDR image frames and short-exposure LDR image frames.

[0195] Step 502: The electronic device performs summing and averaging processing on pixel point information of two adjacent LDR image frames in the M LDR image frames in the second image frame sequence to obtain at least one motion blurred image, where M is an integer greater than 1.

[0196] Step 503: The electronic device performs model training based on the second image frame sequence and at least one motion blurred image to obtain a first model.

[0197] In an embodiment of the present application, the electronic device establishes a low-resolution video LDR dataset in the Raw domain that simulates real noise distribution and motion blur, providing a benchmark dataset for the training and evaluation of HDR super-resolution reconstruction. Moreover, by fusing two adjacent LDR image frames to manually create a data enhancement method for motion blur, the generalization and robustness of the HDR reconstruction model in the case of long-frame motion blur are enhanced.

[0198] It should be noted that the specific process of the above model training can be found in the above embodiment, and will not be repeated here to avoid repetition.

[0199] In an embodiment of the present application, the electronic device can train the first model through motion blurred images, so the first model can learn the characteristic information of the motion blurred images; then, when there is motion blur in the long exposure LDR image frame, by performing feature processing on the first feature information and each second feature information in at least one second feature information, the first model can identify and remove the characteristic information of the motion blurred image contained in the first feature information through at least one second feature information, so that the third feature information will not contain the characteristic information of motion blur, and further, the HDR image reconstructed by the third feature information will not have the phenomenon of image blur, thereby improving the image quality of the reconstructed HDR image.

[0200] Optionally, in the embodiment of the present application, the above step 503 can be specifically implemented by the following steps 503a to 503c.

[0201] Step 503a: The electronic device performs HDR reconstruction on the second image frame sequence and at least one motion blurred image by using the initial model to obtain a second HDR image.

[0202] Step 503b: The electronic device calculates a loss value through the initial model based on the first brightness information of the second HDR image and the second brightness information of the reference HDR image.

[0203] Optionally, in the embodiment of the present application, the above step 503b can be specifically implemented through the following steps 601 and 602.

[0204] Step 601: The electronic device performs tone mapping processing on the second HDR image based on the first brightness information through the initial model to obtain third brightness information, and performs tone mapping processing on the reference HDR image based on the second brightness information to obtain fourth brightness information.

[0205] Step 602: The electronic device calculates the difference between the third brightness information and the fourth brightness information to obtain a loss value.

[0206] It should be noted that the specific process of the above step 601 and step 602 can be found in the above embodiment, and will not be described again here to avoid repetition.

[0207] In an embodiment of the present application, the electronic device can train the initial model by calculating the above-mentioned loss value, so that the trained initial model can better identify motion blurred images to ensure that there is no motion blur in the reconstructed HDR image.

[0208] Step 503c: The electronic device updates the model parameters of the initial model based on the loss value to obtain a first model.

[0209] It should be noted that the specific process of the above steps 503a to 503c can be found in the above embodiments, and will not be described again here to avoid repetition.

[0210] In an embodiment of the present application, the electronic device can calculate a loss value based on the first brightness information of the second HDR image and the second brightness information of the reference HDR image, and then use the loss value to train the model parameters of the first model, so that the first model can learn the characteristic information of the motion blurred image, and then when there is motion blur in the long exposure LDR image frame, the trained first model can avoid the motion blur in the reconstructed HDR image.

[0211] By way of example, the model training method provided in the embodiment of the present application is specifically illustrated below through a specific example, which can be implemented through the following steps 20 to 21.

[0212] Step 20. The electronic device uses the RGB image of Vimeo 90K as HDR video data, degrades it into multiple exposure Raw domain LDR image frames through CRF curve, increasing exposure and adding noise, and downsamples the degraded LDR image frames by linear interpolation, and uses the Raw domain HDR real image without adding exposure and noise as data pairs. The whole is split into training set, validation set and test set for HDR super-resolution reconstruction task according to the ratio of 7:2:1.

[0213] Step 21. The electronic device builds an HDR super-resolution reconstruction model, i.e., the initial model mentioned above, and trains the model based on the training set in step 20. The model input frames are 3 Raw images of different exposures and sizes of 256*256. There are 16 groups of sample data pairs in each batch. The initial learning rate is set to 0.0001, and the model is trained using the deep learning framework Pytorch platform. It is iterated 30 times on the entire data set, and the learning rate decay method is set to periodic cosine annealing. The cosine annealing learning rate decay curve is as follows: Fig.14 As shown; the learning rate update method is detailed in the above embodiment. The Adam optimizer is used to update the model weights during gradient backpropagation. The Earlystopping method is used to supervise the model convergence, and the model iteration is stopped when the validation set loss decreases by no more than 1e-3 within 3 cycles to prevent the model from overfitting. The model iteration is stopped when the validation set loss decreases by no more than 1e-3 within 3 cycles, or the model iteration is stopped after the total number of cycles is reached to obtain the final model.

[0214] In an embodiment of the present application, the electronic device can train the initial model by calculating the above-mentioned loss value, so that the trained initial model can better identify motion blurred images to ensure that there is no motion blur in the reconstructed HDR image.

[0215] For example, Fig.15 As shown, the image reconstruction method provided in the embodiment of the present application is specifically described below through a specific example, which can be specifically implemented through the following steps 30 to 36.

[0216] Step 30: The electronic device acquires a multiple-exposure image frame sequence, which may include two short-exposure LDR image frames and one long-exposure LDR image frame.

[0217] Step 31: The electronic device performs brightness alignment on each LDR image frame in the multiple exposure image frame sequence to obtain a brightness aligned multiple exposure image frame sequence.

[0218] Step 32: The electronic device calculates the optical flow information of other exposure LDR images in the multiple exposure image frame sequence using the long exposure LDR image as a reference.

[0219] Step 33: The electronic device inputs the brightness-aligned multiple exposure image frame sequence into a feature extraction sharing module in the image reconstruction model to obtain feature information of the long-exposure LDR image frame and feature information of the short-exposure LDR image frame.

[0220] Step 34: The electronic device uses the image reconstruction model and the optical flow information to perform feature alignment processing on the feature information of the short-exposure LDR image frame and the feature information of the long-exposure LDR image frame to obtain the feature information of the short-exposure LDR image frame after feature alignment.

[0221] Step 35: The electronic device inputs the feature information of the long-exposure LDR image frame and the feature information of the short-exposure LDR image frame after feature alignment into a feature propagation network in the image reconstruction model through the image reconstruction model, and performs convolution fusion processing on the feature information of the long-exposure LDR image frame and the feature information of the short-exposure LDR image frame after feature alignment through the feature propagation network to obtain fused feature information.

[0222] Step 36: The electronic device inputs the fused feature information and the feature information of the long exposure LDR image frame into an upsampling fusion module in the image reconstruction model through the image reconstruction model, and performs image reconstruction processing on the fused feature information and the feature information of the long exposure LDR image frame through the upsampling fusion module to obtain a reconstructed HDR image.

[0223] In one example, if Fig.16As shown, the electronic device can collect two short-exposure LDR image frames and one long-exposure LDR image frame in the same shooting scene, and the two short-exposure LDR image frames and the one long-exposure LDR image frame are a little girl holding a torch; the electronic device can input the two short-exposure LDR image frames and the one long-exposure LDR image frame into a first model, and the first model can perform optical flow calculation on the two short-exposure LDR image frames based on the one long-exposure LDR image frame, respectively, to obtain a first motion offset matrix between the first short-exposure LDR image frame in the two short-exposure LDR image frames and the one long-exposure LDR image frame, and a second motion offset matrix between the second short-exposure LDR image frame in the two short-exposure LDR image frames and the long-exposure LDR image frame; and input the two short-exposure LDR image frames and the long-exposure LDR image frame into a feature extraction sharing module in the first model to obtain a first high-dimensional matrix corresponding to the long-exposure LDR image frame. feature information and the second high-dimensional feature information and the third high-dimensional feature information corresponding to each short-exposure LDR image frame in the two short-exposure LDR image frames; then, the second high-dimensional feature information is feature aligned through the above-mentioned first motion offset matrix and the long-exposure LDR image frame to obtain the second high-dimensional feature information after feature alignment, and the third high-dimensional feature information is feature aligned through the above-mentioned second motion offset matrix and the long-exposure LDR image frame to obtain the third high-dimensional feature information after feature alignment; then, the first model can input the first high-dimensional feature information, the second high-dimensional feature information after feature alignment, and the third high-dimensional feature information after feature alignment into the feature propagation module, perform convolution processing on the first high-dimensional feature information, the second high-dimensional feature information after feature alignment, and the third high-dimensional feature information after feature alignment to obtain the fourth high-dimensional feature information, and input the fourth high-dimensional feature information into the reconstruction layer in the first model to obtain an HDR image without ghosting.

[0224] In another example, Fig.17As shown, the electronic device can collect two short-exposure LDR image frames and one long-exposure LDR image frame in the same shooting scene, and the two short-exposure LDR image frames and the one long-exposure LDR image frame both contain cars; the electronic device can input the two short-exposure LDR image frames and the one long-exposure LDR image frame into a first model, and the first model can perform optical flow calculation on the two short-exposure LDR image frames based on the one long-exposure LDR image frame, respectively, to obtain a first motion offset matrix between the first short-exposure LDR image frame in the two short-exposure LDR image frames and the one long-exposure LDR image frame, and a second motion offset matrix between the second short-exposure LDR image frame in the two short-exposure LDR image frames and the long-exposure LDR image frame; and input the two short-exposure LDR image frames and the long-exposure LDR image frames into a feature extraction sharing module in the first model to obtain a first high-dimensional feature corresponding to the long-exposure LDR image frame. The first model can input the first high-dimensional feature information, the second high-dimensional feature information after feature alignment, and the third high-dimensional feature information after feature alignment into a feature propagation module, perform convolution on the first high-dimensional feature information, the second high-dimensional feature information after feature alignment, and the third high-dimensional feature information after feature alignment, to obtain the fourth high-dimensional feature information, and input the fourth high-dimensional feature information into the reconstruction layer in the first model to obtain a ghost-free HDR image.

[0225] In another example, Fig.18As shown, the electronic device can collect two short-exposure LDR image frames and one long-exposure LDR image frame in the same shooting scene, and the two short-exposure LDR image frames and the one long-exposure LDR image frame both contain wine glasses and wine bottles; the electronic device can input the two short-exposure LDR image frames and the one long-exposure LDR image frame into a first model, and the first model can perform optical flow calculation on the two short-exposure LDR image frames based on the one long-exposure LDR image frame, to obtain a first motion offset matrix between the first short-exposure LDR image frame in the two short-exposure LDR image frames and the one long-exposure LDR image frame, and a second motion offset matrix between the second short-exposure LDR image frame in the two short-exposure LDR image frames and the long-exposure LDR image frame; and input the two short-exposure LDR image frames and the long-exposure LDR image frame into a feature extraction sharing module in the first model to obtain a first high-dimensional feature vector corresponding to the long-exposure LDR image frame. The first model can input the first high-dimensional feature information, the second high-dimensional feature information after feature alignment, and the third high-dimensional feature information after feature alignment into a feature propagation module, perform convolution on the first high-dimensional feature information, the second high-dimensional feature information after feature alignment, and the third high-dimensional feature information after feature alignment, to obtain the fourth high-dimensional feature information, and input the fourth high-dimensional feature information into the reconstruction layer in the first model to obtain a ghost-free HDR image.

[0226] In another example, Fig.19As shown, the electronic device can collect two short-exposure LDR image frames and one long-exposure LDR image frame in the same shooting scene, and the two short-exposure LDR image frames and the one long-exposure LDR image frame both contain multiple light sources; the electronic device can input the two short-exposure LDR image frames and the one long-exposure LDR image frame into a first model, and the first model can perform optical flow calculation on the two short-exposure LDR image frames based on the one long-exposure LDR image frame, respectively, to obtain a first motion offset matrix between the first short-exposure LDR image frame in the two short-exposure LDR image frames and the one long-exposure LDR image frame, and a second motion offset matrix between the second short-exposure LDR image frame in the two short-exposure LDR image frames and the long-exposure LDR image frame; and input the two short-exposure LDR image frames and the long-exposure LDR image frame into a feature extraction sharing module in the first model to obtain a first high-dimensional feature vector corresponding to the long-exposure LDR image frame. The first model can input the first high-dimensional feature information, the second high-dimensional feature information after feature alignment, and the third high-dimensional feature information after feature alignment into a feature propagation module, perform convolution on the first high-dimensional feature information, the second high-dimensional feature information after feature alignment, and the third high-dimensional feature information after feature alignment, to obtain the fourth high-dimensional feature information, and input the fourth high-dimensional feature information into the reconstruction layer in the first model to obtain a ghost-free HDR image.

[0227] The above-mentioned method embodiments, or various possible implementation methods in each method embodiment, can be executed separately, or, under the premise that there is no contradiction, can also be executed in combination with each other. The specific implementation can be determined according to actual usage requirements, and the embodiments of the present application do not limit this.

[0228] It should be noted that the image reconstruction method provided in the embodiment of the present application can be executed by an image reconstruction device. In the embodiment of the present application, the image reconstruction device provided in the embodiment of the present application is described by taking the image reconstruction method executed by the image reconstruction device as an example.

[0229] Fig. 20 A possible structural diagram of an image reconstruction device involved in an embodiment of the present application is shown. Fig. 20 As shown, the image reconstruction device 70 may include: an acquisition module 71 , a processing module 72 and an execution module 73 .

[0230] Among them, the acquisition module 71 is used to input the first image frame sequence into the first model, and obtain the first feature information and at least one second feature information through the first model. The first image frame sequence includes a first LDR image frame with long exposure and at least one second LDR image frame with short exposure. The first feature information is the feature information of the first LDR image frame, and each second feature information is the feature information of a second LDR image frame. The processing module 72 is used to perform feature processing on each second feature information based on the first feature information through the first model to obtain third feature information. The first model is obtained based on motion blurred image training. The execution module 73 is used to perform HDR reconstruction based on the first feature information obtained by the acquisition module 71 and the third feature information obtained by the processing module 72.

[0231] In a possible implementation, the acquisition module 71 is specifically used to perform feature extraction on the first LDR image frame through the first model to obtain first feature information, and to perform feature extraction on at least one second LDR image frame to obtain at least one sub-feature information; and based on the first feature information and at least one motion offset matrix, perform feature alignment on the at least one sub-feature information to obtain at least one second feature information, the motion offset matrix is ​​calculated based on the pixel point information of the first LDR image frame, and each motion offset matrix represents the motion offset of a second LDR image frame relative to the first LDR image frame.

[0232] In a possible implementation, the processing module 72 is specifically used to perform convolution processing on the first feature information and the first second feature information through the feature propagation module in the first model to obtain the first fused feature information; and to perform convolution processing on the first feature information, the i-th second feature information and the fourth feature information through the feature propagation module in the first model to obtain the i-th fused feature information, the fourth feature information is the fused feature information corresponding to the i-1-th second feature information, i∈[2,N], and i is an integer, N is the number of at least one second feature information; and determine the N-th fused feature information as the third feature information.

[0233] In a possible implementation, the resolutions of the LDR image frames in the first image frame sequence are all less than or equal to the first resolution. The execution module 73 is specifically used to perform convolution processing on the first feature information and the third feature information through the residual block in the upsampling fusion module in the first model to obtain the fifth feature information; and perform linear interpolation processing on the fifth feature information through the upsampling layer in the upsampling fusion module to obtain the sixth feature information, and perform upsampling processing on the fifth feature information through the pixel_shuffle layer in the upsampling fusion module to obtain the seventh feature information; and perform convolution processing on the sixth feature information and the seventh feature information through the reconstruction layer in the upsampling fusion module to obtain the first HDR image; wherein the resolutions of the sixth feature information and the seventh feature information are both greater than the first resolution.

[0234] In a possible implementation, the image reconstruction device 70 provided in the embodiment of the present application also includes a training module. The acquisition module 71 is also used to input the first image frame sequence into the first model, and before obtaining the first feature information and at least one second feature information through the first model, obtain the second image frame sequence, wherein the second image frame sequence includes a long-exposure LDR image frame and a short-exposure LDR image frame. The processing module 72 is also used to sum and average the pixel point information of two adjacent LDR image frames in the M LDR image frames in the second image frame sequence to obtain at least one motion blurred image, where M is an integer greater than 1. The training module is used to perform model training based on the second image frame sequence obtained by the acquisition module 71 and the at least one motion blurred image obtained by the processing module 72 to obtain a first model.

[0235] In one possible implementation, the above-mentioned training module is specifically used to perform HDR reconstruction on the second image frame sequence and at least one motion blurred image through an initial model to obtain a second HDR image; and to calculate a loss value based on the first brightness information of the second HDR image and the second brightness information of the reference HDR image through the initial model; and based on the loss value, update the model parameters of the initial model to obtain the first model.

[0236] In a possible implementation, the processing module 72 is specifically used to perform tone mapping processing on the second HDR image based on the first brightness information through the initial model to obtain third brightness information, and to perform tone mapping processing on the reference HDR image based on the second brightness information to obtain fourth brightness information; and calculate the difference between the third brightness information and the fourth brightness information to obtain a loss value.

[0237] An embodiment of the present application provides an image reconstruction device. Since the first model is obtained by training based on a motion blurred image, the first model can learn the characteristic information of the motion blurred image, and there will be no motion blur phenomenon in the short-exposure LDR image frame, so the first feature information will not contain the motion blur feature information; then, when there is motion blur phenomenon in the long-exposure LDR image frame, by performing feature processing on the first feature information and each second feature information in at least one second feature information, the first model can identify and remove the characteristic information of the motion blurred image contained in the first feature information through the at least one second feature information, so that the third feature information will not contain the motion blur feature information, and further, the HDR image reconstructed by the third feature information will not have the image blur phenomenon, thereby improving the image quality of the reconstructed HDR image.

[0238] The image reconstruction device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the mobile electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0239] The image reconstruction device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0240] The image reconstruction device provided in the embodiment of the present application can implement each process implemented in the above embodiment, and will not be described again here to avoid repetition.

[0241] Alternatively, if Fig.21As shown, an embodiment of the present application further provides an electronic device 90, including a processor 91 and a memory 92, wherein the memory 92 stores programs or instructions that can be executed on the processor 91, and when the program or instructions are executed by the processor 91, the various steps of the above-mentioned image reconstruction method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0242] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0243] Fig. 22 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0244] The electronic device 100 includes but is not limited to components such as a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110.

[0245] Those skilled in the art will appreciate that the electronic device 100 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig. 22 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0246] The processor 110 is used to input a first image frame sequence into a first model, and obtain first feature information and at least one second feature information through the first model, wherein the first image frame sequence includes a first LDR image frame with a long exposure and at least one second LDR image frame with a short exposure, the first feature information is feature information of the first LDR image frame, and each second feature information is feature information of a second LDR image frame; and through the first model, based on the first feature information, feature processing is performed on each second feature information to obtain third feature information, and the first model is obtained based on motion blurred image training; and HDR reconstruction is performed based on the first feature information and the third feature information.

[0247] Optionally, in an embodiment of the present application, the processor 110 is specifically used to perform feature extraction on the first LDR image frame through the first model to obtain first feature information, and perform feature extraction on at least one second LDR image frame to obtain at least one sub-feature information; and based on the first feature information and at least one motion offset matrix, perform feature alignment on at least one sub-feature information to obtain at least one second feature information, the motion offset matrix is ​​calculated based on pixel point information of the first LDR image frame, and each motion offset matrix represents a motion offset of a second LDR image frame relative to the first LDR image frame.

[0248] Optionally, in an embodiment of the present application, the above-mentioned processor 110 is specifically used to perform convolution processing on the first feature information and the first second feature information through the feature propagation module in the first model to obtain the first fused feature information; and perform convolution processing on the first feature information, the i-th second feature information and the fourth feature information through the feature propagation module in the first model to obtain the i-th fused feature information, the fourth feature information is the fused feature information corresponding to the i-1-th second feature information, i∈[2,N], and i is an integer, N is the number of at least one second feature information; and determine the N-th fused feature information as the third feature information.

[0249] Optionally, in an embodiment of the present application, the resolution of the LDR image frames in the first image frame sequence is less than or equal to the first resolution. The processor 110 is specifically used to perform convolution processing on the first feature information and the third feature information through the residual block in the upsampling fusion module in the first model to obtain the fifth feature information; and perform linear interpolation processing on the fifth feature information through the upsampling layer in the upsampling fusion module to obtain the sixth feature information, and perform upsampling processing on the fifth feature information through the pixel_shuffle layer in the upsampling fusion module to obtain the seventh feature information; and perform convolution processing on the sixth feature information and the seventh feature information through the reconstruction layer in the upsampling fusion module to obtain the first HDR image; wherein the resolutions of the sixth feature information and the seventh feature information are both greater than the first resolution.

[0250] Optionally, in an embodiment of the present application, the processor 110 is further used to input the first image frame sequence into the first model, and obtain a second image frame sequence before obtaining the first feature information and at least one second feature information through the first model, wherein the second image frame sequence includes long-exposure LDR image frames and short-exposure LDR image frames; and sum and average pixel point information of two adjacent LDR image frames among the M LDR image frames in the second image frame sequence to obtain at least one motion blurred image, where M is an integer greater than 1; and perform model training based on the second image frame sequence and the at least one motion blurred image to obtain the first model.

[0251] Optionally, in an embodiment of the present application, the above-mentioned processor 110 is specifically used to perform HDR reconstruction on the second image frame sequence and at least one motion blurred image through an initial model to obtain a second HDR image; and to calculate a loss value based on the first brightness information of the second HDR image and the second brightness information of the reference HDR image through the initial model; and based on the loss value, update the model parameters of the initial model to obtain the first model.

[0252] Optionally, in an embodiment of the present application, the above-mentioned processor 110 is specifically used to perform tone mapping processing on the second HDR image based on the first brightness information through an initial model to obtain third brightness information, and perform tone mapping processing on the reference HDR image based on the second brightness information to obtain fourth brightness information; and calculate the difference between the third brightness information and the fourth brightness information to obtain a loss value.

[0253] An embodiment of the present application provides an electronic device. Since the first model is obtained by training based on a motion blurred image, the first model can learn the characteristic information of the motion blurred image, and there will be no motion blur phenomenon in the short-exposure LDR image frame, so the first feature information will not contain the motion blur feature information; then, when there is motion blur phenomenon in the long-exposure LDR image frame, by performing feature processing on the first feature information and each second feature information in at least one second feature information, the first model can identify and remove the characteristic information of the motion blurred image contained in the first feature information through the at least one second feature information, so that the third feature information will not contain the motion blur feature information, and further, the HDR image reconstructed by the third feature information will not have the image blur phenomenon, thereby improving the image quality of the reconstructed HDR image.

[0254] The electronic device provided in the embodiment of the present application can implement each process implemented in the above method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described here.

[0255] The beneficial effects of various implementations in this embodiment can be specifically referred to the beneficial effects of the corresponding implementations in the above method embodiment. To avoid repetition, they will not be described again here.

[0256] It should be understood that in the embodiment of the present application, the input unit 104 may include a graphics processor (Graphics Processing Unit, GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 107 includes a touch panel 1071 and at least one of other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0257] The memory 109 can be used to store software programs and various data. The memory 109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 109 may include a volatile memory or a non-volatile memory, or the memory 109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0258] The processor 110 may include one or more processing units; optionally, the processor 110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 110.

[0259] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0260] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0261] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0262] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0263] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image reconstruction method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0264] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0265] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0266] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image reconstruction method, characterized in that: The method comprises: Inputting a first image frame sequence into a first model, and obtaining first feature information and at least one second feature information through the first model, wherein the first image frame sequence includes a first low dynamic range LDR image frame with a long exposure and at least one second LDR image frame with a short exposure, the first feature information is feature information of the first LDR image frame, and each second feature information is feature information of a second LDR image frame; By using the first model, based on the first feature information, feature processing is performed on each of the second feature information to obtain third feature information, wherein the first model is obtained based on motion blurred image training; Based on the first feature information and the third feature information, high dynamic range HDR reconstruction is performed.

2. The method according to claim 1, characterized in that The obtaining, through the first model, first feature information and at least one second feature information includes: By using the first model, feature extraction is performed on the first LDR image frame to obtain the first feature information, and feature extraction is performed on the at least one second LDR image frame to obtain at least one sub-feature information; Based on the first feature information and at least one motion offset matrix, feature alignment is performed on the at least one sub-feature information to obtain the at least one second feature information, the motion offset matrix is ​​calculated based on the pixel point information of the first LDR image frame, and each motion offset matrix represents the motion offset of a second LDR image frame relative to the first LDR image frame.

3. The method according to claim 1, characterized in that The step of performing feature processing on each of the second feature information based on the first feature information by using the first model to obtain third feature information includes: Performing convolution processing on the first feature information and the first second feature information through the feature propagation module in the first model to obtain first fused feature information; Through the feature propagation module in the first model, convolution processing is performed on the first feature information, the i-th second feature information and the fourth feature information to obtain the i-th fused feature information, wherein the fourth feature information is the fused feature information corresponding to the i-1-th second feature information, i∈[2,N], and i is an integer, and N is the number of the at least one second feature information; The Nth fused feature information is determined as the third feature information.

4. The method according to any one of claims 1 to 3, characterized in that The resolutions of the LDR image frames in the first image frame sequence are all less than or equal to the first resolution; The performing HDR reconstruction based on the first feature information and the third feature information includes: Performing convolution processing on the first feature information and the third feature information through the residual block in the upsampling fusion module in the first model to obtain fifth feature information; Performing linear interpolation processing on the fifth feature information through the upsampling layer in the upsampling fusion module to obtain sixth feature information, and performing upsampling processing on the fifth feature information through the pixel_shuffle layer in the upsampling fusion module to obtain seventh feature information; Performing convolution processing on the sixth feature information and the seventh feature information through a reconstruction layer in the upsampling fusion module to obtain a first HDR image; The resolutions of the sixth characteristic information and the seventy-fourth characteristic information are both greater than the first resolution.

5. The method according to claim 1, characterized in that Before inputting the first image frame sequence into the first model and obtaining the first feature information and at least one second feature information through the first model, the method further includes: Acquire a second image frame sequence, wherein the second image frame sequence includes a long-exposure LDR image frame and a short-exposure LDR image frame; Performing sum and average processing on pixel point information of two adjacent LDR image frames in the M LDR image frames in the second image frame sequence to obtain at least one motion blurred image, where M is an integer greater than 1; Model training is performed based on the second image frame sequence and the at least one motion blurred image to obtain the first model.

6. The method according to claim 5, characterized in that The performing model training based on the second image frame sequence and the at least one motion blurred image to obtain the first model comprises: Performing HDR reconstruction on the second image frame sequence and the at least one motion blurred image through the initial model to obtain a second HDR image; By using the initial model, based on the first brightness information of the second HDR image and the second brightness information of the reference HDR image, a loss value is calculated; Based on the loss value, the model parameters of the initial model are updated to obtain the first model.

7. The method according to claim 6, characterized in that The calculating the loss value based on the first brightness information of the second HDR image and the second brightness information of the reference HDR image by the initial model includes: Using the initial model, based on the first brightness information, tone mapping is performed on the second HDR image to obtain third brightness information, and based on the second brightness information, tone mapping is performed on the reference HDR image to obtain fourth brightness information; A difference between the third brightness information and the fourth brightness information is calculated to obtain the loss value.

8. An image reconstruction device, characterized in that: The image reconstruction device comprises: an acquisition module, a processing module and an execution module; The acquisition module is used to input a first image frame sequence into a first model, and acquire first feature information and at least one second feature information through the first model, wherein the first image frame sequence includes a first LDR image frame with a long exposure and at least one second LDR image frame with a short exposure, the first feature information is feature information of the first LDR image frame, and each second feature information is feature information of a second LDR image frame; The processing module is used to perform feature processing on each of the second feature information based on the first feature information through the first model to obtain third feature information, wherein the first model is obtained based on motion blurred image training; The execution module is used to perform HDR reconstruction based on the first feature information acquired by the acquisition module and the third feature information obtained by the processing module.

9. The device according to claim 8, characterized in that The acquisition module is specifically used to perform feature extraction on the first LDR image frame through the first model to obtain the first feature information, and respectively perform feature extraction on the at least one second LDR image frame to obtain at least one sub-feature information; and based on the first feature information and at least one motion offset matrix, perform feature alignment on the at least one sub-feature information to obtain the at least one second feature information, the motion offset matrix is ​​calculated based on the pixel point information of the first LDR image frame, and each motion offset matrix represents the motion offset of a second LDR image frame relative to the first LDR image frame.

10. An electronic device, characterized in that: The method comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein the program or instruction, when executed by the processor, implements the steps of the image reconstruction method according to any one of claims 1 to 7.

Images