Image reconstruction method and device and electronic equipment

Abstract

The invention discloses an image reconstruction method and device and electronic equipment, and belongs to the technical field of image processing. The method comprises the steps that an image frame sequence is acquired, and the image frame sequence comprises a first short-exposure low dynamic range (LDR) image, a long-exposure LDR image and a second short-exposure LDR image which are arranged in sequence; determining a bidirectional optical flow between the first short exposure LDR image and the second short exposure LDR image; determining an initialized optical flow corresponding to a first prediction frame insertion image between the first short exposure LDR image and the second short exposure LDR image based on the bidirectional optical flow; refining the initialized optical flow by taking the long-exposure LDR image as a motion alignment target to obtain a target optical flow; determining a frame insertion image based on the target optical flow, the first short exposure LDR image and the second short exposure LDR image; and performing high dynamic range (HDR) image reconstruction based on the long exposure LDR image and the frame insertion image to obtain an HDR image.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and specifically relates to an image reconstruction method, device and electronic equipment. Background Art

[0002] In the field of image processing, the dynamic range of an image refers to the ratio of the highest brightness to the lowest brightness of the image. Compared with low dynamic range (LDR) images, high dynamic range (HDR) images can better reflect the visual details of the scene in bright and dark areas. However, the bit depth of the images recorded by standard digital imaging devices is not enough to cover the dynamic range of real scenes, resulting in underexposure in dark areas or overexposure in bright areas.

[0003] With the development of high dynamic range technology and neural networks, the current method for HDR image reconstruction is usually: first, the short-exposure LDR image and the long-exposure LDR image are explicitly motion aligned by optical flow, and then the fusion weights of the corresponding positions of the aligned images are determined by the weighted prediction network, and the reconstructed HDR image is obtained by weighted averaging and fusion.

[0004] In order to achieve explicit motion alignment of short-exposure LDR images with long-exposure LDR images through optical flow, the first prerequisite is to ensure the brightness of the two images is consistent. However, in real scenes, it is usually impossible to ensure the consistency of image brightness, which seriously affects the accuracy of optical flow and the inability to achieve motion alignment of images, resulting in low generalization and robustness of HDR image reconstruction. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide an image reconstruction method, device, electronic device, readable storage medium and computer program product, which can solve the problem of low generalization and robustness of HDR image reconstruction.

[0006] In a first aspect, an embodiment of the present application provides an image reconstruction method, the method comprising:

[0007] Acquire an image frame sequence, wherein the image frame sequence includes a first short-exposure low dynamic range LDR image, a long-exposure LDR image, and a second short-exposure LDR image arranged in sequence;

[0008] determining a bidirectional optical flow between the first short-exposure LDR image and the second short-exposure LDR image;

[0009] Determine, based on the bidirectional optical flow, an initialization optical flow corresponding to a first predicted interpolation frame image between the first short-exposure LDR image and the second short-exposure LDR image;

[0010] Taking the long-exposure LDR image as a motion alignment target, refining the initialized optical flow to obtain a target optical flow;

[0011] determining an interpolated image based on the target optical flow, the first short-exposure LDR image, and the second short-exposure LDR image;

[0012] A high dynamic range HDR image is reconstructed based on the long exposure LDR image and the interpolated frame image to obtain an HDR image.

[0013] In a second aspect, an embodiment of the present application provides an image reconstruction device, the device comprising:

[0014] An acquisition module, configured to acquire an image frame sequence, wherein the image frame sequence includes a first short-exposure low dynamic range LDR image, a long-exposure LDR image, and a second short-exposure LDR image arranged in sequence;

[0015] a determining module, configured to determine a bidirectional optical flow between the first short-exposure LDR image and the second short-exposure LDR image;

[0016] The determining module is further used to determine, based on the bidirectional optical flow, an initialization optical flow corresponding to a first predicted interpolation frame image between the first short-exposure LDR image and the second short-exposure LDR image;

[0017] A refinement module, used to refine the initialization optical flow using the long exposure LDR image as a motion alignment target to obtain a target optical flow;

[0018] The determination module is further configured to determine an interpolated image based on the target optical flow, the first short-exposure LDR image, and the second short-exposure LDR image;

[0019] A reconstruction module is used to perform high dynamic range HDR image reconstruction based on the long exposure LDR image and the interpolated frame image to obtain an HDR image.

[0020] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0021] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0022] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0023] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0024] In the embodiment of the present application, in the process of determining the interpolated image between the first short-exposure LDR image and the second short-exposure LDR image, the bidirectional optical flow between the first short-exposure LDR image and the second short-exposure LDR image is first determined, and the initialization optical flow corresponding to the first predicted interpolated image between the first short-exposure LDR image and the second short-exposure LDR image is determined based on the bidirectional optical flow, and then the initialization optical flow is refined with the long-exposure LDR image as the motion alignment target to obtain the target optical flow, and then the interpolated image is determined based on the target optical flow, the first short-exposure LDR image and the second short-exposure LDR image, so that the interpolated image motion-aligned with the long-exposure LDR image can be obtained. In this way, by reconstructing a high dynamic range HDR image based on the long-exposure LDR image and the interpolated image, an HDR image can be obtained. The method of reconstructing the HDR image in the embodiment of the present application is not limited by the exposure parameters, which enhances the generalization and robustness of the HDR image reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a flowchart of an image reconstruction method provided in an embodiment of the present application;

[0026] Figure 2 It is a schematic diagram of estimating a first initialization optical flow and a second initialization optical flow by using an initialization optical flow module provided by an embodiment of the present application;

[0027] Figure 3 is a schematic diagram of an optical flow refinement process provided in an embodiment of the present application;

[0028] Figure 4 is a schematic diagram of performing frame insertion and fusion through a frame insertion and fusion module provided in an embodiment of the present application;

[0029] Figure 5 is a schematic diagram of performing HDR image reconstruction through an HDR reconstruction module provided in an embodiment of the present application;

[0030] Figure 6 It is a flowchart of an HDR image reconstruction method implemented by an HDR image reconstruction model provided in an embodiment of the present application;

[0031] Figure 7is a schematic diagram of an initialization optical flow module provided in an embodiment of the present application;

[0032] Figure 8 is a structural schematic diagram of an image reconstruction device provided in an embodiment of the present application;

[0033] Fig. 9 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application;

[0034] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0036] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0037] As described in the background technology section, in order to achieve explicit motion alignment of short-exposure LDR images and long-exposure LDR images by means of optical flow, the first prerequisite is to ensure that the brightness of the two images is consistent. However, in real scenes, due to the inconsistent brightness and noise levels of images with different exposures, the accuracy of the optical flow is seriously affected during the HDR reconstruction process, and it is difficult to accurately align; especially when the moving object is completely in the overexposed area or is blocked, due to the lack of reference features, there are no reference features when calculating the optical flow, resulting in errors in the optical flow calculation, and motion alignment cannot be performed, resulting in fusion ghosts. On the one hand, it is difficult to achieve motion alignment in real scenes with existing HDR image reconstruction methods, resulting in low generalization and robustness of HDR image reconstruction. On the other hand, the HDR images reconstructed by existing HDR image reconstruction methods are prone to fusion ghosts and low image quality.

[0038] Based on this, in order to solve the problems of the prior art, the embodiments of the present application provide an image reconstruction method, device, electronic device, readable storage medium and computer program product. The image reconstruction method can be applied to reconstruct an HDR image from an LDR image to obtain an HDR image scene.

[0039] The image reconstruction method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0040] Figure 1 FIG. 1 is a flow chart of an image reconstruction method provided by an embodiment of the present application. Figure 1 As shown, the image reconstruction method provided in the embodiment of the present application includes the following steps S110-S160, and each step is explained in detail below.

[0041] S110 , acquiring an image frame sequence, where the image frame sequence includes a first short-exposure low dynamic range LDR image, a long-exposure LDR image, and a second short-exposure LDR image arranged in sequence.

[0042] Here, the images in the image frame sequence may be images actually captured in a real scene, or may be images obtained from an LDR image set, which is not limited here. The LDR image set may include multiple LDR images. The multiple LDR images may have different exposure parameters. The first short-exposure LDR image, the long-exposure LDR image, and the second short-exposure LDR image may be determined from the multiple LDR images through the exposure parameters. The long-exposure LDR image has problems of overexposure and motion blur. The short-exposure LDR image has large noise and limited dynamic range. The first short-exposure LDR image and the second short-exposure LDR image may have the same or similar exposure parameters.

[0043] It should be noted that, in the embodiment of the present application, the number of long-exposure LDR images may be one or more, which is not limited here.

[0044] S120, determining a bidirectional optical flow between the first short-exposure LDR image and the second short-exposure LDR image.

[0045] The bidirectional optical flow may include a first optical flow from the first short exposure LDR image to the second short exposure LDR image, and a second optical flow from the second short exposure LDR image to the first short exposure LDR image.

[0046] Assuming that the image frame sequence includes three frames, that is, the first frame is the first short-exposure LDR image, the second frame is the long-exposure LDR image, and the third frame is the second short-exposure LDR image, then the first optical flow can be recorded as The second optical flow can be expressed as

[0047] The embodiment of the present application can calculate the above-mentioned bidirectional optical flow based on algorithms such as dense inverse search (DIS) algorithm, sparse optical flow estimation (Lucas-Kanade) algorithm, multi-level optical flow estimation (Farneback) algorithm, and deep matching (DeepFlow) algorithm, or can calculate the above-mentioned bidirectional optical flow through an optical flow estimation module in an HDR image reconstruction model, which is not limited here. Among them, the HDR image reconstruction model can be a trained model for performing HDR image reconstruction on an LDR image to obtain an HDR image. The optical flow estimation module can be used to calculate the bidirectional optical flow between two frames of images.

[0048] Based on this, in order to ensure the accuracy of the bidirectional optical flow between the first short-exposure LDR image and the second short-exposure LDR image, and thus ensure the effect of subsequent HDR image reconstruction, in some embodiments, the above S120 may specifically include:

[0049] Calculating brightness information of the first short-exposure LDR image and the second short-exposure LDR image respectively to obtain first brightness information and second brightness information;

[0050] Perform noise reduction processing on the first brightness information and the second brightness information respectively to obtain third brightness information and fourth brightness information;

[0051] The third brightness information and the fourth brightness information are input into an optical flow estimation module of the HDR image reconstruction model, so that the first optical flow and the second optical flow are calculated based on the third brightness information and the fourth brightness information through the optical flow estimation module.

[0052] Here, the first brightness information may be brightness information of the first short-exposure LDR image, the second brightness information may be brightness information of the second short-exposure LDR image, the third brightness information may be brightness information obtained by performing noise reduction processing on the first brightness information, and the fourth brightness information may be brightness information obtained by performing noise reduction processing on the second brightness information.

[0053] In addition, the brightness information may be Y channel information, and the brightness information of the image may be calculated by the following formula (1):

[0054] Y=0.2989*R+0.587*G+0.114*B (1)

[0055] Based on this, the first brightness information can be recorded as Y1, and the second brightness information can be recorded as Y2.

[0056] Considering that the noise of the short exposure image is large, the two short exposure LDR images can be subjected to denoising by Gaussian filtering to ensure the accuracy of the brightness information. The two short exposure LDR images are subjected to denoising by Gaussian filtering, and specifically, the brightness information of the two short exposure LDR images is subjected to denoising by Gaussian filtering to obtain the third brightness information and the fourth brightness information. If the Gaussian filter is denoted as G(·), the third brightness information can be expressed as Y3=G(Y1). Similarly, the fourth brightness information can be expressed as Y4=G(Y2).

[0057] In addition, the optical flow estimation module in the embodiment of the present application can be implemented based on the DIS algorithm. If the optical flow estimation module is implemented based on the DIS algorithm, after the third brightness information and the fourth brightness information are input into the optical flow estimation module, the optical flow estimation module can calculate the first optical flow and the second optical flow by the following formula (2):

[0058]

[0059] In the above formula (2), DIS(·) can represent the DIS optical flow calculation algorithm.

[0060] Assuming that the image frame sequence includes a long exposure LDR image, through the above formula (2), if i is set to 1 and j is set to 3, the first optical flow can be calculated: If i is set to 3 and j is set to 1, the second optical flow can be calculated.

[0061] The embodiment of the present application calculates the first optical flow and the second optical flow by using the optical flow estimation module in the pre-trained HDR image reconstruction model, so as to ensure the accuracy of the bidirectional optical flow between the first short-exposure LDR image and the second short-exposure LDR image, thereby ensuring the effect of subsequent HDR image reconstruction.

[0062] S130 , determining an initialization optical flow corresponding to a first predicted interpolation frame image between the first short-exposure LDR image and the second short-exposure LDR image based on the bidirectional optical flow.

[0063] Here, the first predicted interpolated image may be a short exposure LDR image estimated to be inserted between the first short exposure LDR image and the second short exposure LDR image. The number of first predicted interpolated images may be the same as the number of long exposure LDR images. In addition, the initialization optical flow corresponding to the first predicted interpolated image may include a first initialization optical flow from the first short exposure LDR image to the first predicted interpolated image, and a second initialization optical flow from the second short exposure LDR image to the first predicted interpolated image. The first initialization optical flow may be determined based on the first optical flow, and the second initialization optical flow may be determined based on the second optical flow.

[0064] As a simple example, if the first optical flow is A, and the number of the first predicted interpolated frame images is one, the first initialized optical flow can be roughly recorded as A / 2.

[0065] Based on this, in order to ensure the accuracy of the first initialized optical flow and the second initialized optical flow, and thus ensure the effect of subsequent HDR image reconstruction, in some embodiments, the above S130 may further specifically include:

[0066] The first short-exposure LDR image, the second short-exposure LDR image, and the bidirectional optical flow are spliced ​​to obtain a first vector;

[0067] The first vector is input into an initialization optical flow estimation module of the HDR image reconstruction model, so as to estimate a first initialization optical flow and a second initialization optical flow based on the first vector through a residual block in the initialization optical flow estimation module.

[0068] Here, the HDR image reconstruction model may further include an initialization optical flow estimation module. The initialization optical flow estimation module may be used to estimate a first initialization optical flow and a second initialization optical flow based on the first short exposure LDR image, the second short exposure LDR image and the bidirectional optical flow. In addition, the initialization optical flow estimation module may include at least one residual block. Each residual block may include multiple (such as three) continuous convolutional layers and skip-level connections, and ensure that the output and input formats are the same.

[0069] Assume that the first short-exposure LDR image is denoted as I1, the second short-exposure LDR image is denoted as I3, and the first initialized optical flow is denoted as The second initialized optical flow is recorded as Then the process of estimating the first initialization optical flow and the second initialization optical flow based on the first short exposure LDR image, the second short exposure LDR image and the bidirectional optical flow can be defined as the following formula (3):

[0070]

[0071] In the above formula (3), D(·) may represent a residual block.

[0072] A schematic diagram of estimating a first initialization optical flow and a second initialization optical flow by initializing an optical flow module provided in an embodiment of the present application can be as follows: Figure 2 shown.

[0073] In this way, by splicing the first short-exposure LDR image, the second short-exposure LDR image and the bidirectional optical flow into a first vector, and using the initialized optical flow estimation module in the pre-trained HDR image reconstruction model, the first initialized optical flow and the second initialized optical flow are estimated based on the first vector, so that the accuracy of the first initialized optical flow and the second initialized optical flow can be guaranteed, thereby ensuring the effect of subsequent HDR image reconstruction.

[0074] S140, taking the long exposure LDR image as the motion alignment target, refining the initialized optical flow to obtain the target optical flow.

[0075] Here, the target optical flow can be regarded as the accurate optical flow. That is, the alignment image obtained by motion aligning the corresponding short-exposure LDR images using the target optical flow is more accurate than the alignment image obtained by motion aligning the corresponding short-exposure LDR images using the initialization optical flow.

[0076] The target optical flow may include a first target optical flow from the first short-exposure LDR image to the first predicted interpolation image, and a second target optical flow from the second short-exposure LDR image to the first predicted interpolation image. The first target optical flow may be an optical flow obtained by refining the first initialization optical flow, and the second target optical flow may be an optical flow obtained by refining the second initialization optical flow.

[0077] As an example, the process of refining the initialized optical flow may be a process of taking the long-exposure LDR image as a motion alignment target and continuously adjusting the initialized optical flow to obtain the target optical flow.

[0078] As a more specific example, the process of refining the initialized optical flow can be a process of continuously adjusting the refinement scale to adjust the optical flow. The refinement scale can be understood as the ability to capture details in the optical flow estimation. The initialized optical flow can be regarded as the target optical flow when the refinement scale is 1. In addition, each time the refinement scale is adjusted, the long exposure LDR image can be used as the motion alignment target.

[0079] By taking the long exposure LDR image as the motion alignment target in each process of adjusting the initialized optical flow, not only the accuracy of the target optical flow can be continuously improved, but also the first predicted interpolated frame image can be continuously aligned with the long exposure LDR image in motion, thereby obtaining a first predicted interpolated frame image that is consistent with the long exposure LDR image in motion.

[0080] In order to refine the initialized optical flow and obtain the target optical flow, the corresponding short-exposure LDR image can be firstly motion-aligned using the initialized optical flow, and the aligned image can be used as the motion alignment target with the long-exposure LDR image, and the initialized optical flow can be refined based on the aligned image.

[0081] Based on this, in order to improve the accuracy of the target optical flow and thus improve the effect of subsequent HDR image reconstruction, in some embodiments, the above S140 may specifically include:

[0082] Performing motion alignment on the first short-exposure LDR image based on the first initialization optical flow to obtain a first aligned image, and performing motion alignment on the second short-exposure LDR image based on the second initialization optical flow to obtain a second aligned image;

[0083] The long-exposure LDR image and the optical flow refinement data are input into the optical flow refinement module of the HDR image reconstruction model, and the optical flow refinement module is used to take the long-exposure LDR image as the motion alignment target, and the optical flow refinement is performed based on the optical flow refinement data to obtain a first target optical flow and a second target optical flow, wherein the optical flow refinement data includes the first short-exposure LDR image, the second short-exposure LDR image, the first initialized optical flow, the second initialized optical flow, the first aligned image, and the second aligned image.

[0084] Here, the HDR image reconstruction model may further include an optical flow refinement module. The optical flow refinement module may be used to refine the initialization optical flow to obtain the target optical flow.

[0085] In order to refine the initialized optical flow and obtain the target optical flow, the first short-exposure LDR image can be firstly motion-aligned based on the first initialized optical flow to obtain the first aligned image, and the second short-exposure LDR image can be motion-aligned based on the second initialized optical flow to obtain the second aligned image, and then the long-exposure LDR image, the first short-exposure LDR image, the second short-exposure LDR image, the first initialized optical flow, the second initialized optical flow, the first aligned image and the second aligned image are input into the optical flow refinement module, so that the optical flow refinement module can refine the optical flow based on the long-exposure LDR image, the first short-exposure LDR image, the second short-exposure LDR image, the first initialized optical flow, the second initialized optical flow, the first aligned image and the second aligned image to obtain the first target optical flow and the second target optical flow. Among them, the long-exposure LDR image can be used as a global reference to supervise the refinement process of the optical flow, which can not only continuously improve the accuracy of the target optical flow, but also enable the first predicted interpolation frame image to continuously motion-align with the long-exposure LDR image, so as to obtain the first predicted interpolation frame image that is consistent with the motion of the long-exposure LDR image.

[0086] The embodiment of the present application refines the initialization optical flow by using the optical flow refinement module in the pre-trained HDR image reconstruction model to obtain the target optical flow, which can improve the accuracy of the target optical flow and further improve the effect of subsequent HDR image reconstruction.

[0087] Based on this, in order to further improve the accuracy of the target optical flow, in some embodiments, the long-exposure LDR image is used as the motion alignment target, and the optical flow is refined based on the optical flow refinement data to obtain the first target optical flow and the second target optical flow, which may specifically include:

[0088] Taking the long exposure LDR image as the motion alignment target, the optical flow is refined based on the initial refinement scale and the optical flow refinement data to obtain the third optical flow and the fourth optical flow;

[0089] When the initial refinement scale is larger than the target refinement scale, the following operations are performed repeatedly until the initial refinement scale is smaller than or equal to the target resolution:

[0090] Reduce the value of the initial refinement scale, use the third optical flow as the first initialization optical flow, use the fourth optical flow as the second initialization optical flow, use the long exposure LDR image as the motion alignment target, perform optical flow refinement based on the initial refinement scale and optical flow refinement data, and obtain the third optical flow and the fourth optical flow;

[0091] The third optical flow is determined as the first target optical flow, and the fourth optical flow is determined as the second target optical flow.

[0092] Here, the refinement scale may include the resolution of the image. The refinement scale and the resolution of the image may have a negative correlation. For example, at high resolution, since the image is rich in details, the optical flow estimation requires a finer scale to match these details, so the refinement scale is smaller.

[0093] As an example, the initial refinement scale may be 1, and the target refinement scale may be 1 / 8. The initial refinement scale may be reduced by reducing the initial refinement scale to half of the original scale each time until the initial refinement scale is reduced to 1 / 8 of the original scale.

[0094] Based on the above embodiments, a schematic diagram of an optical flow refinement process provided in an embodiment of the present application can be shown as follows: Figure 3 shown.

[0095] The embodiment of the present application calculates the optical flow offset at each refinement scale and uses the long-exposure LDR image as a global reference to supervise the refinement process of the optical flow, thereby further improving the accuracy of the target optical flow.

[0096] S150 , determining an interpolated image based on the target optical flow, the first short-exposure LDR image, and the second short-exposure LDR image.

[0097] Here, the interpolated image may be an actually determined short-exposure LDR image that is consistent with the motion of the long-exposure LDR image. The number of interpolated images may be the same as the number of long-exposure LDR images. More specifically, the interpolated image may correspond one-to-one to the long-exposure LDR image.

[0098] As an example, after determining the first target optical flow corresponding to the first short-exposure LDR image and the second target optical flow corresponding to the second short-exposure LDR image, an aligned image can be obtained by using the first target optical flow to perform motion alignment on the first short-exposure LDR image; another aligned image can be obtained by using the second target optical flow to perform motion alignment on the second short-exposure LDR image. By performing interpolation fusion on the two aligned images, an interpolated image with consistent motion with the long-exposure LDR image can be obtained.

[0099] Based on this, in order to ensure the accuracy of the interpolated frame image, in some embodiments, the above S150 may specifically include:

[0100] Performing motion alignment on the first short-exposure LDR image based on the first target optical flow to obtain a third aligned image, and performing motion alignment on the second short-exposure LDR image based on the second target optical flow to obtain a fourth aligned image;

[0101] The third aligned image and the fourth aligned image are interpolated and fused to obtain an interpolated image.

[0102] Here, the third aligned image and the fourth aligned image may be interpolated and fused by first determining the fusion weights corresponding to the third aligned image and the fourth aligned image, and then interpolating and fusion the third aligned image and the fourth aligned image based on the fusion weights to obtain the interpolated image.

[0103] As a more specific example, the embodiment of the present application can determine the interpolated image by the following formulas (4)-(6):

[0104]

[0105]

[0106] In the above formula (4), Warp(·) represents the alignment operation of the images using the optical flow. Therefore, S1 can represent the third aligned image, and S3 can represent the fourth aligned image.

[0107] In the above formula (5), Inter(·) can represent the weight prediction method, Weight1 can represent the fusion weight corresponding to the third aligned image, and Weight3 can represent the fusion weight corresponding to the fourth aligned image. The embodiment of the present application does not limit the weight prediction method, and any method that can determine the fusion weight is within the protection scope of the present application.

[0108] In the above formula (6), Can represent interpolated images.

[0109] In the embodiment of the present application, a third aligned image is obtained by first performing motion alignment on the first short exposure LDR image based on the first target optical flow, and a fourth aligned image is obtained by performing motion alignment on the second short exposure LDR image based on the second target optical flow; and then the third aligned image and the fourth aligned image are interpolated and fused to obtain an interpolated image, thereby ensuring the accuracy of the interpolated image.

[0110] Based on this, in order to further improve the accuracy of the interpolated frame image, in some embodiments, the interpolated frame fusion of the third aligned image and the fourth aligned image to obtain the interpolated frame image may specifically include:

[0111] Inputting the third aligned image and the fourth aligned image into an interpolation fusion module of the HDR image reconstruction model, so as to determine the fusion weights corresponding to the third aligned image and the fourth aligned image respectively through the interpolation fusion module;

[0112] Based on the fusion weights corresponding to the third aligned image and the fourth aligned image respectively, interpolation fusion is performed on the third aligned image and the fourth aligned image to obtain an interpolation image.

[0113] Here, the HDR image reconstruction model may further include a frame insertion fusion module. The frame insertion fusion module may include a convolution layer.

[0114] As an example, the second vector can be obtained by splicing the third aligned image and the fourth aligned image. After the second vector is input into the interpolation fusion module, the interpolation fusion module can determine and output the fusion weight map corresponding to the third aligned image and the fourth aligned image through the convolution layer, and perform pixel fusion on the third aligned image and the fourth aligned image based on the fusion weight map corresponding to the third aligned image and the fourth aligned image, so as to obtain the interpolation image.

[0115] The implementation logic of the interpolation fusion module may be the same as or different from the above formulas (4)-(6), and this is not limited here. If the implementation logic of the interpolation fusion module is the same as the above formulas (4)-(6), then Inter(·) in the above formula (5) may represent the interpolation fusion module.

[0116] A schematic diagram of performing frame insertion fusion by using a frame insertion fusion module provided in an embodiment of the present application can be as follows: Figure 4 shown.

[0117] The embodiment of the present application can further improve the accuracy of the interpolated frame image by performing interpolated frame fusion using an interpolated frame fusion module in a pre-trained HDR image reconstruction model.

[0118] S160, reconstructing a high dynamic range HDR image based on the long exposure LDR image and the interpolated frame image to obtain an HDR image.

[0119] Considering that the interpolated image is a fusion of two short-exposure LDR images, it has a high noise level and a limited dynamic range, and the long-exposure LDR image also has problems of overexposure and motion blur. Therefore, by performing pixel fusion on the long-exposure LDR image and the interpolated image, it can not only make up for the problems of high noise level and limited dynamic range of the interpolated image, but also make up for the problems of overexposure and motion blur of the long-exposure LDR image, and reconstruct a high-quality HDR image.

[0120] Based on this, in order to improve the image quality of the HDR image, in some embodiments, the above S160 may specifically include:

[0121] The long-exposure LDR image and the interpolated frame image are input into an HDR reconstruction module of an HDR image reconstruction model, so as to determine the fusion weights corresponding to the long-exposure LDR image and the interpolated frame image respectively through the HDR reconstruction module;

[0122] Based on the long-exposure LDR image, the interpolated image, and the corresponding fusion weights of the long-exposure LDR image and the interpolated image, high dynamic range HDR reconstruction is performed to obtain an HDR image.

[0123] Here, the HDR image reconstruction model may further include an HDR reconstruction module. The HDR reconstruction module may be used to perform pixel fusion on the long-exposure LDR image and the interpolated image to obtain an HDR image. The HDR reconstruction module may include a convolution layer.

[0124] As an example, after the long exposure LDR image and the interpolated image are input into the HDR reconstruction module, the HDR reconstruction module can first determine and output the fusion weights corresponding to the long exposure LDR image and the interpolated image through the convolution layer, and then perform pixel fusion on the long exposure LDR image and the interpolated image based on the fusion weights corresponding to the long exposure LDR image and the interpolated image, so as to compensate for the defects of the long exposure LDR image and the interpolated image and reconstruct the final HDR image.

[0125] A schematic diagram of performing HDR image reconstruction by using an HDR reconstruction module provided in an embodiment of the present application can be as follows: Figure 5 shown.

[0126] The embodiment of the present application can improve the image quality of the HDR image by performing HDR image reconstruction using an HDR reconstruction module in a pre-trained HDR image reconstruction model.

[0127] In addition, as described above, there may be multiple long-exposure LDR images in the image frame sequence, and there may be a one-to-one correspondence between the interpolated frame image and the long-exposure LDR image. Therefore, when there are multiple long-exposure LDR images in the image frame sequence, the embodiment of the present application can reconstruct multiple HDR images at one time, thereby improving the reconstruction efficiency of the HDR image, reducing the consumption of computing resources, and generating a high frame rate HDR video.

[0128] Therefore, in the embodiment of the present application, in the process of determining the interpolated image between the first short-exposure LDR image and the second short-exposure LDR image, the bidirectional optical flow between the first short-exposure LDR image and the second short-exposure LDR image is first determined, and the initialization optical flow corresponding to the first predicted interpolated image between the first short-exposure LDR image and the second short-exposure LDR image is determined based on the bidirectional optical flow, and then the initialization optical flow is refined with the long-exposure LDR image as the motion alignment target to obtain the target optical flow, and then the interpolated image is determined based on the target optical flow, the first short-exposure LDR image and the second short-exposure LDR image, so that the interpolated image motion-aligned with the long-exposure LDR image can be obtained. In this way, by reconstructing a high dynamic range HDR image based on the long-exposure LDR image and the interpolated image, an HDR image can be obtained. The method of reconstructing the HDR image in the embodiment of the present application is not limited by the exposure parameters, which enhances the generalization and robustness of the HDR image reconstruction.

[0129] In order to better describe the above solution, some specific examples are given based on the above embodiments.

[0130] For example, the HDR image reconstruction method in the embodiment of the present application can be implemented by a pre-trained HDR image reconstruction model, wherein the HDR image reconstruction model may include an optical flow estimation module, an initialization optical flow module, an optical flow refinement module, an interpolation fusion module, and an HDR reconstruction module.

[0131] like Figure 6 As shown, an HDR image reconstruction method implemented by an HDR image reconstruction model provided in an embodiment of the present application may include the following steps:

[0132] S61, acquiring an image frame sequence; wherein the image frame sequence includes a first short-exposure LDR image, a long-exposure LDR image, and a second short-exposure LDR image;

[0133] S62, inputting the image frame sequence into the HDR image reconstruction model;

[0134] S63, calculating a first optical flow and a second optical flow between the first short-exposure LDR image and the second short-exposure LDR image by an optical flow estimation module;

[0135] S64, estimating a first initialized optical flow and a second initialized optical flow based on the first optical flow, the second optical flow, the first short-exposure LDR image, and the second short-exposure LDR image through an initialized optical flow module;

[0136] S65, using the optical flow refinement module, taking the long-exposure LDR image as the motion alignment target, performing optical flow refinement based on the first initialization optical flow, the second initialization optical flow, the first short-exposure LDR image, and the second short-exposure LDR image, to obtain a first target optical flow and a second target optical flow;

[0137] S66, determining an interpolation frame image based on the first short-exposure LDR image, the second short-exposure LDR image, the first target optical flow, and the second target optical flow through an interpolation frame fusion module;

[0138] S67. Perform HDR image reconstruction based on the interpolated image and the long-exposure LDR image through an HDR reconstruction module to obtain an HDR image.

[0139] On this basis, in order to realize HDR image reconstruction through various modules in the HDR image reconstruction model, in some embodiments, before the first vector is input into the initialization optical flow estimation module of the HDR image reconstruction model, the method may further include:

[0140] Acquire image frame sequence samples and their corresponding reference HDR images, the image frame sequence samples comprising a first short-exposure LDR image sample, a long-exposure LDR image sample, and a second short-exposure LDR image sample arranged in sequence;

[0141] The image frame sequence samples are input into the initial HDR image reconstruction model, so as to perform the following operations through the initial HDR image reconstruction model: determining the predicted bidirectional optical flow between the first short-exposure LDR image sample and the second short-exposure LDR image sample; determining the predicted initialization optical flow corresponding to the second predicted interpolation frame image between the first short-exposure LDR image sample and the second short-exposure LDR image sample based on the predicted bidirectional optical flow; taking the long-exposure LDR image sample as the motion alignment target, refining the predicted initialization optical flow to obtain the predicted target optical flow; determining the third predicted interpolation frame image based on the predicted target optical flow, the first short-exposure LDR image sample and the second short-exposure LDR image sample; performing HDR reconstruction based on the long-exposure LDR image sample and the third predicted interpolation frame image to obtain the predicted HDR image;

[0142] Determine a first loss function value according to the similarity between the predicted HDR image and the reference HDR image;

[0143] The model parameters of the initial HDR image reconstruction model are adjusted based on the first loss function value, and the HDR image reconstruction model is obtained through training.

[0144] Here, the image in the image frame sequence sample may be an image obtained from an LDR image set. The LDR image set may include multiple LDR images. The multiple LDR images may have different exposure parameters. The first short-exposure LDR image sample, the long-exposure LDR image sample, and the second short-exposure LDR image sample may be determined from the multiple LDR images by the exposure parameters. The long-exposure LDR image sample has problems of overexposure and motion blur. The short-exposure LDR image sample has large noise and limited dynamic range. The first short-exposure LDR image sample and the second short-exposure LDR image sample may have the same or similar exposure parameters.

[0145] It should be noted that, in the embodiment of the present application, the number of long-exposure LDR image samples may be one or more, which is not limited here.

[0146] In addition, in the embodiments of the present application, the image processing process in the model training process can refer to the process of using the model to reconstruct the HDR image described above, and will not be repeated here.

[0147] In addition, the first loss function value can be determined by the L2Loss loss function. The L2Loss loss function can be shown in the following formula (7):

[0148]

[0149] In the above formula (7), γ may be a parameter of the Ma curve, and is set to 2.2 by default. It can be expressed as the predicted HDR image, can represent the reference HDR image, and T(·) can represent the tone mapping function. The tone mapping function can be expressed as follows:

[0150]

[0151] In the above formula (8), H can represent the input image (including the predicted HDR image and the reference HDR image), and μ can be set to 5000.

[0152] The embodiment of the present application calculates the first loss function value of the predicted HDR image and the reference HDR image after Gamma and tone mapping transformation, and based on the first loss function value, uses the Adam optimizer to update the model parameters when the gradient is returned, so as to train the HDR image reconstruction model, thereby realizing HDR image reconstruction through each module in the HDR image reconstruction model.

[0153] Based on this, in order to obtain image frame sequence samples and their corresponding reference HDR images under ideal conditions, thereby ensuring the accuracy of model training, in some embodiments, the above-mentioned acquisition of image frame sequence samples and their corresponding reference HDR images may specifically include:

[0154] Get HDR video data;

[0155] Performing exposure processing and adding noise processing on multiple HDR image frames in the HDR video data to obtain multiple exposure images corresponding to each of the multiple HDR image frames;

[0156] Extracting multiple frame images from multiple multi-exposure images;

[0157] Determine a first frame image in the multiple frame images as a first short-exposure LDR image sample, and determine a last frame image in the multiple frame images as a second short-exposure LDR image sample;

[0158] Performing long exposure processing on the remaining images except the first frame and the last frame in the multiple-frame image to obtain long exposure LDR image samples;

[0159] Determine an image frame sequence sample according to the first short-exposure LDR image sample, the second short-exposure LDR image sample, the long-exposure LDR image sample and the relative position relationship between them;

[0160] An HDR image in the HDR video data corresponding to the long-exposure LDR image sample is determined as a reference HDR image.

[0161] Here, the HDR video data may be standard color (Standard Red Green Blue, sRGB) image data. HDR video data can be obtained from large-scale, high-quality video datasets such as Vimeo 90K and Kalantaril13. By performing exposure processing and adding noise processing on the sRGB image in the HDR video data, the sRGB image can be degraded into a multiple exposure image, and a multiple exposure image corresponding to the real HDR image can be obtained.

[0162] After determining a plurality of multiple exposure images, a plurality of groups of image frame sequences can be extracted from the plurality of multiple exposure images. Each group of image frame sequences can include a plurality of frames. The first frame image can be determined as a first short exposure LDR image sample, and the last frame image can be determined as a second short exposure LDR image sample. In addition, by performing long exposure processing on the remaining images except the first frame and the last frame in the plurality of frames, a long exposure LDR image sample can be obtained. The number of long exposure LDR image samples can be one or more. The method of performing long exposure processing on an image to obtain a long exposure LDR image sample can be as follows: multiplying the image by a random light ratio (such as 8, 16, 32...) and limiting and truncating it as a long exposure LDR image.

[0163] Afterwards, the image frame sequence samples may be determined according to the first short-exposure LDR image samples, the second short-exposure LDR image samples, the long-exposure LDR image samples and the relative position relationship therebetween in each group of image frame sequences.

[0164] In addition, since the long exposure LDR image samples are obtained by performing long exposure processing on the images in the image frame sequence, and the images in the image frame sequence are obtained by performing exposure processing and noise adding processing on the HDR images in the HDR video data, after determining the long exposure LDR image samples, according to the above correspondence, the HDR image in the HDR video data corresponding to the long exposure LDR image samples can be determined as the reference HDR image.

[0165] In addition, after obtaining multiple groups of image frame sequence samples, the number of image frame sequence samples can be increased by performing data enhancement on the images by using methods such as inversion, rotation and cropping.

[0166] Multiple groups of image frame sequence samples can be split into training set, validation set and test set for HDR image reconstruction task in a ratio of 7:2:1.

[0167] The embodiment of the present application generates image frame sequence samples for HDR video data, thereby obtaining image frame sequence samples in an ideal state and their corresponding reference HDR images, thereby ensuring the accuracy of model training.

[0168] In addition, corresponding to the HDR image reconstruction model, the initial HDR image reconstruction model may include an optical flow estimation module, an initialization optical flow module, an optical flow refinement module, an interpolation fusion module, and an HDR reconstruction module. Among them, the optical flow refinement module may include a student module and a teacher module. During model training, the student module and the teacher module are turned on; during model inference, the student module may be turned on, but the teacher module may not be turned on. In addition, the first loss function value may include a second loss function value.

[0169] Based on this, in order to further improve the accuracy of model training, in some embodiments, the method may also include:

[0170] Inputting the long exposure LDR image sample and the optical flow refinement data sample into the student module, so as to obtain a first predicted optical flow and a second predicted optical flow by using the long exposure LDR image sample as a motion alignment target and performing optical flow refinement based on the optical flow refinement data sample through the student module;

[0171] Inputting the reference HDR image and the optical flow refinement data samples into the teacher module, so as to obtain a third predicted optical flow and a fourth predicted optical flow by performing optical flow refinement based on the optical flow refinement data samples with the reference HDR image as a motion alignment target through the teacher module;

[0172] Determining a second loss function value based on a similarity between the first predicted optical flow and the third predicted optical flow, and a similarity between the second predicted optical flow and the fourth predicted optical flow;

[0173] Among them, the optical flow refinement data samples include a first short exposure LDR image sample, a second short exposure LDR image sample, a first predicted initialization optical flow from the first short exposure LDR image sample to a third predicted interpolation image, a second predicted initialization optical flow from the second short exposure LDR image sample to the third predicted interpolation image, a first predicted alignment image and a second predicted alignment image, the first predicted alignment image is determined based on the first short exposure LDR image sample and the first predicted initialization optical flow; the second predicted alignment image is determined based on the second short exposure LDR image sample and the second predicted initialization optical flow.

[0174] Here, both the student module and the teacher module can be composed of residual blocks, but the global reference input of the teacher module is a real HDR image, that is, a reference HDR image, so the teacher module can provide assistance to the student module and guide the student module to converge better. The process of determining the first predicted optical flow and the second predicted optical flow based on the student module, and the process of determining the third predicted optical flow and the fourth predicted optical flow based on the teacher module can refer to the process of determining the first initialized optical flow and the second initialized optical flow based on the optical flow refinement module above, which will not be described in detail here.

[0175] A schematic diagram of initializing an optical flow module provided in an embodiment of the present application can be as follows: Figure 7 As shown. Figure 7In the embodiment, the first short frame alignment image may represent the first alignment image or the third alignment image, the third short frame alignment image may represent the second alignment image or the fourth alignment image, the 1->2 optical flow may represent the first initialization optical flow or the third optical flow, the 3->2 optical flow may represent the second initialization optical flow or the fourth optical flow, the first short frame image may represent the first short exposure LDR image, the third short frame image may represent the second short exposure LDR image, the second long frame image may represent the long exposure LDR image, and the second frame real HDR image may represent the reference HDR image.

[0176] In the process of model training in the embodiment of the present application, the teacher module can guide the student module to converge better, thereby further improving the accuracy of model training.

[0177] In addition, in the model training process of the present application, 16 sets of sample data pairs (including image frame sequence samples and their corresponding reference HDR images) can be input each time. In addition, the initial learning rate can be set to 0.001, and the model is trained using the deep learning framework Pytorch platform, iterating 50 epochs on the entire data set, and setting the learning rate decay method to periodic cosine annealing. Among them, the learning rate update method can be shown in the following formula (9):

[0178]

[0179] In the above formula (9), It can represent the maximum learning rate, which is set to 1e-4; It can represent the minimum learning rate, which is set to 1e-6. cur It can be the current cycle number, T i is the total number of cycles.

[0180] By using the Adam optimizer to update the model parameters during gradient backpropagation, the final model can be trained.

[0181] In summary, the embodiments of the present application first generate an interpolated image between two short-exposure LDR images and motion-align the interpolated image with the long-exposure LDR image, thereby avoiding direct estimation of the optical flow from the short frame to the long frame, and avoiding the problem of optical flow calculation errors caused by inconsistent brightness and noise distribution, thereby enhancing the generalization and robustness of the HDR reconstruction model in scenes with large exposure ratios and poor linearity, and improving the quality of the reconstructed HDR image.

[0182] In addition, by taking the long-exposure LDR image as the global reference and using the teacher model for supervision, accurate optical flow information can be obtained, and short-exposure interpolated images with consistent motion with the long-exposure LDR image can be further generated, which are then fused with the long-exposure LDR image to obtain a ghost-free HDR image, effectively reducing ghosts in HDR reconstruction.

[0183] The HDR reconstruction method in the embodiment of the present application is not limited by exposure parameters, and can still restore details without obvious ghosting when the motion is in a completely overexposed area, thus achieving dynamic range expansion of low dynamic range videos in difficult scenes such as high exposure ratios and large motions.

[0184] The image reconstruction method provided in the embodiment of the present application may be executed by an image reconstruction device. In the embodiment of the present application, the image reconstruction method performed by the image reconstruction device is taken as an example to illustrate the image reconstruction device provided in the embodiment of the present application.

[0185] Figure 8 FIG. 1 is a schematic diagram showing the structure of an image reconstruction device provided in an embodiment of the present application. Figure 8 As shown, the image reconstruction device 800 provided in the embodiment of the present application may include:

[0186] An acquisition module 810 is used to acquire an image frame sequence, where the image frame sequence includes a first short-exposure low dynamic range LDR image, a long-exposure LDR image, and a second short-exposure LDR image arranged in sequence;

[0187] A determination module 820, configured to determine a bidirectional optical flow between the first short-exposure LDR image and the second short-exposure LDR image;

[0188] The determination module 820 is further configured to determine, based on the bidirectional optical flow, an initialization optical flow corresponding to a first predicted interpolation frame image between the first short-exposure LDR image and the second short-exposure LDR image;

[0189] A refinement module 830 is used to refine the initialization optical flow using the long exposure LDR image as a motion alignment target to obtain a target optical flow;

[0190] The determination module 820 is further configured to determine an interpolated image based on the target optical flow, the first short-exposure LDR image, and the second short-exposure LDR image;

[0191] The reconstruction module 840 is used to reconstruct a high dynamic range HDR image based on the long exposure LDR image and the interpolated frame image to obtain an HDR image.

[0192] The image reconstruction device 800 is described in detail below, as follows:

[0193] In some of the embodiments, the initialization optical flow includes a first initialization optical flow from the first short-exposure LDR image to the first predicted interpolation frame image, and a second initialization optical flow from the second short-exposure LDR image to the first predicted interpolation frame image.

[0194] Based on this, the determination module 820 may specifically include:

[0195] A stitching submodule, used for stitching the first short-exposure LDR image, the second short-exposure LDR image and the bidirectional optical flow to obtain a first vector;

[0196] The estimation submodule is used to input the first vector into the initialization optical flow estimation module of the HDR image reconstruction model, so as to estimate the first initialization optical flow and the second initialization optical flow based on the first vector through the residual block in the initialization optical flow estimation module.

[0197] In some of the embodiments, the target optical flow includes a first target optical flow from the first short-exposure LDR image to the first predicted interpolation frame image, and a second target optical flow from the second short-exposure LDR image to the first predicted interpolation frame image.

[0198] Based on this, the refinement module 830 may specifically include:

[0199] an alignment submodule, configured to perform motion alignment on the first short-exposure LDR image based on the first initialization optical flow to obtain a first aligned image, and to perform motion alignment on the second short-exposure LDR image based on the second initialization optical flow to obtain a second aligned image;

[0200] The refinement submodule is further used to input the long-exposure LDR image and the optical flow refinement data into the optical flow refinement module of the HDR image reconstruction model, so as to obtain the first target optical flow and the second target optical flow by taking the long-exposure LDR image as the motion alignment target through the optical flow refinement module and performing optical flow refinement based on the optical flow refinement data, wherein the optical flow refinement data includes the first short-exposure LDR image, the second short-exposure LDR image, the first initialized optical flow, the second initialized optical flow, the first aligned image and the second aligned image.

[0201] In some embodiments, the refinement submodule includes:

[0202] A refinement unit, used for taking the long exposure LDR image as a motion alignment target, performing optical flow refinement based on an initial refinement scale and optical flow refinement data, and obtaining a third optical flow and a fourth optical flow;

[0203] The execution unit is used for, when the initial refinement scale is larger than the target refinement scale, cyclically performing the following operations until the initial refinement scale is smaller than or equal to the target refinement scale: reducing the value of the initial refinement scale, taking the third optical flow as the first initialization optical flow, taking the fourth optical flow as the second initialization optical flow, taking the long exposure LDR image as the motion alignment target, performing optical flow refinement based on the initial refinement scale and the optical flow refinement data, and obtaining the third optical flow and the fourth optical flow;

[0204] The determining unit is used to determine the third optical flow as the first target optical flow and determine the fourth optical flow as the second target optical flow.

[0205] In some embodiments, the determination module 820 may further include:

[0206] The alignment submodule is further used to perform motion alignment on the first short-exposure LDR image based on the first target optical flow to obtain a third aligned image, and to perform motion alignment on the second short-exposure LDR image based on the second target optical flow to obtain a fourth aligned image;

[0207] The fusion submodule is used to perform interpolation fusion on the third aligned image and the fourth aligned image to obtain an interpolation image.

[0208] In some embodiments, the fusion submodule may specifically include:

[0209] A determination unit, configured to input the third aligned image and the fourth aligned image into an interpolation fusion module of an HDR image reconstruction model, so as to determine fusion weights corresponding to the third aligned image and the fourth aligned image respectively through the interpolation fusion module;

[0210] The fusion unit is used to perform interpolation fusion on the third aligned image and the fourth aligned image based on the fusion weights corresponding to the third aligned image and the fourth aligned image respectively, so as to obtain an interpolation image.

[0211] In some embodiments, the reconstruction module 840 may specifically include:

[0212] A determination submodule, used for inputting the long-exposure LDR image and the interpolated frame image into an HDR reconstruction module of an HDR image reconstruction model, so as to determine the fusion weights corresponding to the long-exposure LDR image and the interpolated frame image respectively through the HDR reconstruction module;

[0213] The fusion submodule is used to perform high dynamic range HDR reconstruction based on the long exposure LDR image, the interpolated image, and the fusion weights corresponding to the long exposure LDR image and the interpolated image to obtain an HDR image.

[0214] In some of these embodiments, the bidirectional optical flow includes a first optical flow from the first short exposure LDR image to the second short exposure LDR image, and a second optical flow from the second short exposure LDR image to the first short exposure LDR image.

[0215] Based on this, the determination module 820 may further specifically include:

[0216] A calculation submodule, used to calculate the brightness information of the first short-exposure LDR image and the second short-exposure LDR image respectively, to obtain first brightness information and second brightness information;

[0217] A processing submodule, configured to perform noise reduction processing on the first brightness information and the second brightness information respectively to obtain third brightness information and fourth brightness information;

[0218] The calculation submodule is further used to input the third brightness information and the fourth brightness information into the optical flow estimation module of the HDR image reconstruction model, so as to calculate the first optical flow and the second optical flow based on the third brightness information and the fourth brightness information through the optical flow estimation module.

[0219] In some embodiments, the image reconstruction device 800 may further include:

[0220] The acquisition module 810 is further used to acquire image frame sequence samples and their corresponding reference HDR images before inputting the first vector into the initialization optical flow estimation module of the HDR image reconstruction model, wherein the image frame sequence samples include a first short-exposure LDR image sample, a long-exposure LDR image sample, and a second short-exposure LDR image sample arranged in sequence;

[0221] An execution module is used to input the image frame sequence samples into the initial HDR image reconstruction model, so as to perform the following operations through the initial HDR image reconstruction model: determine the predicted bidirectional optical flow between the first short-exposure LDR image sample and the second short-exposure LDR image sample; determine the predicted initialization optical flow corresponding to the second predicted interpolation frame image between the first short-exposure LDR image sample and the second short-exposure LDR image sample based on the predicted bidirectional optical flow; refine the predicted initialization optical flow with the long-exposure LDR image sample as the motion alignment target to obtain the predicted target optical flow; determine the third predicted interpolation frame image based on the predicted target optical flow, the first short-exposure LDR image sample and the second short-exposure LDR image sample; perform HDR reconstruction based on the long-exposure LDR image sample and the third predicted interpolation frame image to obtain the predicted HDR image;

[0222] The determination module 820 is further configured to determine a first loss function value according to a similarity between the predicted HDR image and the reference HDR image;

[0223] The training module is used to adjust the model parameters of the initial HDR image reconstruction model based on the first loss function value, and train the HDR image reconstruction model.

[0224] In some embodiments, the initial HDR image reconstruction model includes an optical flow refinement module, the optical flow refinement module includes a student module and a teacher module, and the first loss function value includes a second loss function value. Based on this, the image reconstruction device 800 may also include:

[0225] The refinement module 830 is further configured to input the long exposure LDR image sample and the optical flow refinement data sample into the student module, so as to obtain a first predicted optical flow and a second predicted optical flow by performing optical flow refinement based on the optical flow refinement data sample with the long exposure LDR image sample as a motion alignment target through the student module;

[0226] The refinement module 830 is further used to input the reference HDR image and the optical flow refinement data sample into the teacher module, so as to obtain a third predicted optical flow and a fourth predicted optical flow by performing optical flow refinement based on the optical flow refinement data sample with the reference HDR image as a motion alignment target through the teacher module;

[0227] The determination module 820 is further configured to determine a second loss function value based on a similarity between the first predicted optical flow and the third predicted optical flow, and a similarity between the second predicted optical flow and the fourth predicted optical flow;

[0228] Among them, the optical flow refinement data samples include a first short exposure LDR image sample, a second short exposure LDR image sample, a first predicted initialization optical flow from the first short exposure LDR image sample to a third predicted interpolation image, a second predicted initialization optical flow from the second short exposure LDR image sample to the third predicted interpolation image, a first predicted alignment image and a second predicted alignment image, the first predicted alignment image is determined based on the first short exposure LDR image sample and the first predicted initialization optical flow; the second predicted alignment image is determined based on the second short exposure LDR image sample and the second predicted initialization optical flow.

[0229] In some embodiments, the acquisition module 810 may specifically include:

[0230] The acquisition submodule is used to obtain HDR video data;

[0231] A processing submodule performs exposure processing and noise adding processing on multiple HDR image frames in the HDR video data to obtain multiple exposure images corresponding to each of the multiple HDR image frames;

[0232] An extraction submodule, used for extracting multiple frame images from multiple multi-exposure images;

[0233] A determination submodule, configured to determine a first frame image in the multiple frames of images as a first short-exposure LDR image sample, and to determine a last frame image in the multiple frames of images as a second short-exposure LDR image sample;

[0234] The processing submodule is further used to perform long exposure processing on the remaining images except the first frame and the last frame in the multiple frames of images to obtain long exposure LDR image samples;

[0235] The determination submodule is further used to determine the image frame sequence samples according to the first short-exposure LDR image samples, the second short-exposure LDR image samples, the long-exposure LDR image samples and the relative position relationship between them; and determine the HDR image in the HDR video data corresponding to the long-exposure LDR image samples as the reference HDR image.

[0236] In the embodiment of the present application, in the process of determining the interpolated image between the first short-exposure LDR image and the second short-exposure LDR image, the bidirectional optical flow between the first short-exposure LDR image and the second short-exposure LDR image is first determined, and the initialization optical flow corresponding to the first predicted interpolated image between the first short-exposure LDR image and the second short-exposure LDR image is determined based on the bidirectional optical flow, and then the initialization optical flow is refined with the long-exposure LDR image as the motion alignment target to obtain the target optical flow, and then the interpolated image is determined based on the target optical flow, the first short-exposure LDR image and the second short-exposure LDR image, so that the interpolated image motion-aligned with the long-exposure LDR image can be obtained. In this way, by reconstructing a high dynamic range HDR image based on the long-exposure LDR image and the interpolated image, an HDR image can be obtained. The method of reconstructing the HDR image in the embodiment of the present application is not limited by the exposure parameters, which enhances the generalization and robustness of the HDR image reconstruction. The image reconstruction device in the embodiment of the present application can be an electronic device, or a component in an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or a device other than a terminal. Exemplarily, the electronic device may be a mobile phone, a tablet computer, a laptop computer, a PDA, an in-vehicle electronic device, a mobile Internet device (MID), an augmented reality (AR) / virtual reality (VR) device, a robot, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc. It may also be a server, a network attached storage (NAS), a personal computer (PC), a television (TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiments of the present application.

[0237] The image reconstruction device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0238] The image reconstruction device provided in the embodiment of the present application can achieve Figures 1 to 7 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0239] Alternatively, if Fig. 9As shown, an embodiment of the present application further provides an electronic device 900, including a processor 901 and a memory 902, wherein the memory 902 stores programs or instructions that can be executed on the processor 901, and when the program or instructions are executed by the processor 901, the various steps of the above-mentioned image reconstruction method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0240] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0241] Fig.10 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0242] The electronic device 1000 includes but is not limited to: a radio frequency unit 1001, a network module 1002, an audio output unit 1003, an input unit 1004, a sensor 1005, a display unit 1006, a user input unit 1007, an interface unit 1008, a memory 1009, and a processor 1010 and other components.

[0243] Those skilled in the art will appreciate that the electronic device 1000 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1010 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.10 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0244] The processor 1010 is used to acquire an image frame sequence, where the image frame sequence includes a first short-exposure low dynamic range LDR image, a long-exposure LDR image, and a second short-exposure LDR image arranged in sequence; determine a bidirectional optical flow between the first short-exposure LDR image and the second short-exposure LDR image; determine an initialization optical flow corresponding to a first predicted interpolation frame image between the first short-exposure LDR image and the second short-exposure LDR image based on the bidirectional optical flow; refine the initialization optical flow with the long-exposure LDR image as a motion alignment target to obtain a target optical flow; determine an interpolation frame image based on the target optical flow, the first short-exposure LDR image, and the second short-exposure LDR image; and reconstruct a high dynamic range HDR image based on the long-exposure LDR image and the interpolation frame image to obtain an HDR image.

[0245] In the embodiment of the present application, in the process of determining the interpolated image between the first short-exposure LDR image and the second short-exposure LDR image, the bidirectional optical flow between the first short-exposure LDR image and the second short-exposure LDR image is first determined, and the initialization optical flow corresponding to the first predicted interpolated image between the first short-exposure LDR image and the second short-exposure LDR image is determined based on the bidirectional optical flow, and then the initialization optical flow is refined with the long-exposure LDR image as the motion alignment target to obtain the target optical flow, and then the interpolated image is determined based on the target optical flow, the first short-exposure LDR image and the second short-exposure LDR image, so that the interpolated image motion-aligned with the long-exposure LDR image can be obtained. In this way, by reconstructing a high dynamic range HDR image based on the long-exposure LDR image and the interpolated image, an HDR image can be obtained. The method of reconstructing the HDR image in the embodiment of the present application is not limited by the exposure parameters, which enhances the generalization and robustness of the HDR image reconstruction.

[0246] Optionally, the processor 1010 is further configured to:

[0247] The first short-exposure LDR image, the second short-exposure LDR image and the bidirectional optical flow are spliced ​​to obtain a first vector; the first vector is input into an initialization optical flow estimation module of the HDR image reconstruction model to estimate a first initialization optical flow and a second initialization optical flow based on the first vector through a residual block in the initialization optical flow estimation module.

[0248] Optionally, the processor 1010 is further configured to:

[0249] The first short-exposure LDR image is motion aligned based on the first initialized optical flow to obtain a first aligned image, and the second short-exposure LDR image is motion aligned based on the second initialized optical flow to obtain a second aligned image; the long-exposure LDR image and the optical flow refinement data are input into the optical flow refinement module of the HDR image reconstruction model, so that the optical flow refinement module takes the long-exposure LDR image as the motion alignment target, and the optical flow refinement is performed based on the optical flow refinement data to obtain a first target optical flow and a second target optical flow, wherein the optical flow refinement data includes the first short-exposure LDR image, the second short-exposure LDR image, the first initialized optical flow, the second initialized optical flow, the first aligned image, and the second aligned image.

[0250] Optionally, the processor 1010 is further configured to:

[0251] Taking the long exposure LDR image as the motion alignment target, performing optical flow refinement based on the initial refinement scale and the optical flow refinement data to obtain the third optical flow and the fourth optical flow; when the initial refinement scale is larger than the target refinement scale, looping the following operations until the initial refinement scale is smaller than or equal to the target refinement scale: reducing the value of the initial refinement scale, taking the third optical flow as the first initialization optical flow, taking the fourth optical flow as the second initialization optical flow, and taking the long exposure LDR image as the motion alignment target, performing optical flow refinement based on the initial refinement scale and the optical flow refinement data to obtain the third optical flow and the fourth optical flow; determining the third optical flow as the first target optical flow, and determining the fourth optical flow as the second target optical flow.

[0252] Optionally, the processor 1010 is further configured to:

[0253] The first short-exposure LDR image is motion-aligned based on the first target optical flow to obtain a third aligned image, and the second short-exposure LDR image is motion-aligned based on the second target optical flow to obtain a fourth aligned image; the third aligned image and the fourth aligned image are interpolated and fused to obtain an interpolated image.

[0254] Optionally, the processor 1010 is further configured to:

[0255] The third aligned image and the fourth aligned image are input into an interpolation fusion module of an HDR image reconstruction model, so as to determine the fusion weights corresponding to the third aligned image and the fourth aligned image respectively through the interpolation fusion module; based on the fusion weights corresponding to the third aligned image and the fourth aligned image respectively, the third aligned image and the fourth aligned image are interpolated and fused to obtain an interpolated image.

[0256] Optionally, the processor 1010 is further configured to:

[0257] The long-exposure LDR image and the interpolated image are input into the HDR reconstruction module of the HDR image reconstruction model to determine the fusion weights corresponding to the long-exposure LDR image and the interpolated image through the HDR reconstruction module; based on the long-exposure LDR image, the interpolated image, and the fusion weights corresponding to the long-exposure LDR image and the interpolated image, high dynamic range HDR reconstruction is performed to obtain an HDR image.

[0258] Optionally, the processor 1010 is further configured to:

[0259] The brightness information of the first short-exposure LDR image and the second short-exposure LDR image are calculated respectively to obtain the first brightness information and the second brightness information; the first brightness information and the second brightness information are subjected to noise reduction processing respectively to obtain the third brightness information and the fourth brightness information; the third brightness information and the fourth brightness information are input into the optical flow estimation module of the HDR image reconstruction model, so as to calculate the first optical flow and the second optical flow based on the third brightness information and the fourth brightness information through the optical flow estimation module.

[0260] Optionally, the processor 1010 is further configured to:

[0261] Before inputting the first vector into the initialization optical flow estimation module of the HDR image reconstruction model, obtaining image frame sequence samples and their corresponding reference HDR images, the image frame sequence samples comprising a first short-exposure LDR image sample, a long-exposure LDR image sample, and a second short-exposure LDR image sample arranged in sequence; inputting the image frame sequence samples into the initial HDR image reconstruction model, so as to perform the following operations through the initial HDR image reconstruction model: determining a predicted bidirectional optical flow between the first short-exposure LDR image sample and the second short-exposure LDR image sample; determining the first short-exposure LDR image sample and the second short-exposure LDR image sample based on the predicted bidirectional optical flow; The method comprises the following steps: first, refining the predicted initialization optical flow corresponding to the second predicted interpolation frame image between the two images; taking the long exposure LDR image samples as the motion alignment target, refining the predicted initialization optical flow, and obtaining the predicted target optical flow; determining the third predicted interpolation frame image based on the predicted target optical flow, the first short exposure LDR image samples, and the second short exposure LDR image samples; performing HDR reconstruction based on the long exposure LDR image samples and the third predicted interpolation frame image, and obtaining the predicted HDR image; determining the first loss function value according to the similarity between the predicted HDR image and the reference HDR image; adjusting the model parameters of the initial HDR image reconstruction model based on the first loss function value, and training the HDR image reconstruction model.

[0262] Optionally, the processor 1010 is further configured to:

[0263] The long exposure LDR image samples and the optical flow refinement data samples are input into the student module, and the long exposure LDR image samples are used as motion alignment targets through the student module, and the optical flow refinement is performed based on the optical flow refinement data samples to obtain a first predicted optical flow and a second predicted optical flow; the reference HDR image and the optical flow refinement data samples are input into the teacher module, and the reference HDR image is used as the motion alignment target through the teacher module, and the optical flow refinement is performed based on the optical flow refinement data samples to obtain a third predicted optical flow and a fourth predicted optical flow; based on the similarity between the first predicted optical flow and the third predicted optical flow, and the similarity between the second predicted optical flow and the fourth predicted optical flow The second loss function value is determined based on the similarity between the first and second short exposure LDR image samples; wherein the optical flow refinement data samples include the first short exposure LDR image samples, the second short exposure LDR image samples, the first predicted initialization optical flow from the first short exposure LDR image samples to the third predicted interpolation frame image, the second predicted initialization optical flow from the second short exposure LDR image samples to the third predicted interpolation frame image, the first predicted alignment image and the second predicted alignment image, the first predicted alignment image is determined based on the first short exposure LDR image samples and the first predicted initialization optical flow; the second predicted alignment image is determined based on the second short exposure LDR image samples and the second predicted initialization optical flow.

[0264] Optionally, the processor 1010 is further configured to:

[0265] Acquire HDR video data; perform exposure processing and noise addition processing on multiple HDR image frames in the HDR video data to obtain multiple exposure images corresponding to each of the multiple HDR image frames; extract multiple image frames from the multiple multiple exposure images; determine the first image frame in the multiple image frames as a first short-exposure LDR image sample, and determine the last image frame in the multiple image frames as a second short-exposure LDR image sample; perform long-exposure processing on the remaining images in the multiple image frames except the first frame and the last frame to obtain long-exposure LDR image samples; determine image frame sequence samples according to the first short-exposure LDR image sample, the second short-exposure LDR image sample, the long-exposure LDR image sample and the relative position relationship between them; determine the HDR image in the HDR video data corresponding to the long-exposure LDR image sample as a reference HDR image.

[0266] The embodiment of the present application first generates an interpolated image between two short-exposure LDR images, and motion-aligns the interpolated image with the long-exposure LDR image, thereby avoiding direct estimation of the optical flow from the short frame to the long frame, and avoiding the problem of optical flow calculation errors caused by inconsistent brightness and noise distribution, thereby enhancing the generalization and robustness of the HDR reconstruction model in scenes with large exposure ratios and poor linearity, and improving the quality of the reconstructed HDR image.

[0267] In addition, by taking the long-exposure LDR image as the global reference and using the teacher model for supervision, accurate optical flow information can be obtained, and short-exposure interpolated images with consistent motion with the long-exposure LDR image can be further generated, which are then fused with the long-exposure LDR image to obtain a ghost-free HDR image, effectively reducing ghosts in HDR reconstruction.

[0268] The HDR reconstruction method in the embodiment of the present application is not limited by exposure parameters, and can still restore details without obvious ghosting when the motion is in a completely overexposed area, thus achieving dynamic range expansion of low dynamic range videos in difficult scenes such as high exposure ratios and large motions.

[0269] It should be understood that in the embodiment of the present application, the input unit 1004 may include a graphics processor (Graphics Processing Unit, GPU) 10041 and a microphone 10042, and the graphics processor 10041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1006 may include a display panel 10061, and the display panel 10061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1007 includes a touch panel 10071 and at least one of other input devices 10072. The touch panel 10071 is also called a touch screen. The touch panel 10071 may include two parts: a touch detection device and a touch controller. Other input devices 10072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0270] The memory 1009 can be used to store software programs and various data. The memory 1009 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1009 may include a volatile memory or a non-volatile memory, or the memory 1009 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1009 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0271] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1010.

[0272] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned image reconstruction method embodiment are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0273] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0274] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image reconstruction method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0275] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0276] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image reconstruction method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0277] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0278] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0279] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image reconstruction method, characterized in that: include: Acquire an image frame sequence, wherein the image frame sequence includes a first short-exposure low dynamic range LDR image, a long-exposure LDR image, and a second short-exposure LDR image arranged in sequence; determining a bidirectional optical flow between the first short-exposure LDR image and the second short-exposure LDR image; Determine, based on the bidirectional optical flow, an initialization optical flow corresponding to a first predicted interpolation frame image between the first short-exposure LDR image and the second short-exposure LDR image; Taking the long-exposure LDR image as a motion alignment target, refining the initialized optical flow to obtain a target optical flow; determining an interpolated image based on the target optical flow, the first short-exposure LDR image, and the second short-exposure LDR image; A high dynamic range HDR image is reconstructed based on the long exposure LDR image and the interpolated frame image to obtain an HDR image.

2. The method according to claim 1, characterized in that: The initialized optical flow comprises a first initialized optical flow from the first short-exposure LDR image to the first predicted interpolated frame image, and a second initialized optical flow from the second short-exposure LDR image to the first predicted interpolated frame image; The determining, based on the bidirectional optical flow, an initialization optical flow corresponding to a first predicted interpolation frame image between the first short-exposure LDR image and the second short-exposure LDR image, comprises: splicing the first short-exposure LDR image, the second short-exposure LDR image, and the bidirectional optical flow to obtain a first vector; The first vector is input into an initialization optical flow estimation module of an HDR image reconstruction model, so as to estimate the first initialization optical flow and the second initialization optical flow based on the first vector through a residual block in the initialization optical flow estimation module.

3. The method according to claim 2, characterized in that The target optical flow includes a first target optical flow from the first short-exposure LDR image to the first predicted interpolation frame image, and a second target optical flow from the second short-exposure LDR image to the first predicted interpolation frame image; The step of taking the long-exposure LDR image as a motion alignment target and refining the initialized optical flow to obtain a target optical flow includes: Performing motion alignment on the first short-exposure LDR image based on the first initialization optical flow to obtain a first aligned image, and performing motion alignment on the second short-exposure LDR image based on the second initialization optical flow to obtain a second aligned image; The long-exposure LDR image and the optical flow refinement data are input into an optical flow refinement module of an HDR image reconstruction model, so that the optical flow refinement module takes the long-exposure LDR image as a motion alignment target, performs optical flow refinement based on the optical flow refinement data, and obtains the first target optical flow and the second target optical flow, wherein the optical flow refinement data includes the first short-exposure LDR image, the second short-exposure LDR image, the first initialized optical flow, the second initialized optical flow, the first aligned image, and the second aligned image.

4. The method according to claim 3, characterized in that The method of taking the long-exposure LDR image as a motion alignment target and performing optical flow refinement based on the optical flow refinement data to obtain the first target optical flow and the second target optical flow includes: Taking the long exposure LDR image as a motion alignment target, performing optical flow refinement based on an initial refinement scale and the optical flow refinement data to obtain a third optical flow and a fourth optical flow; When the initial refinement scale is greater than the target refinement scale, the following operations are performed in a loop until the initial refinement scale is less than or equal to the target refinement scale: reducing the value of the initial refinement scale, taking the third optical flow as the first initialization optical flow, taking the fourth optical flow as the second initialization optical flow, taking the long exposure LDR image as a motion alignment target, performing optical flow refinement based on the initial refinement scale and the optical flow refinement data, and obtaining a third optical flow and a fourth optical flow; The third optical flow is determined as the first target optical flow, and the fourth optical flow is determined as the second target optical flow.

5. The method according to claim 3, characterized in that: The determining of the interpolated image based on the target optical flow, the first short-exposure LDR image, and the second short-exposure LDR image comprises: performing motion alignment on the first short-exposure LDR image based on the first target optical flow to obtain a third aligned image, and performing motion alignment on the second short-exposure LDR image based on the second target optical flow to obtain a fourth aligned image; The third aligned image and the fourth aligned image are interpolated and fused to obtain the interpolated frame image.

6. The method according to claim 5, characterized in that The performing interpolation fusion on the third aligned image and the fourth aligned image to obtain the interpolation image includes: Inputting the third aligned image and the fourth aligned image into an interpolation fusion module of an HDR image reconstruction model, so as to determine the fusion weights corresponding to the third aligned image and the fourth aligned image respectively through the interpolation fusion module; Based on the fusion weights corresponding to the third aligned image and the fourth aligned image respectively, interpolation fusion is performed on the third aligned image and the fourth aligned image to obtain the interpolation image.

7. The method according to any one of claims 1 to 6, characterized in that: The reconstructing a high dynamic range HDR image based on the long exposure LDR image and the interpolated frame image to obtain the HDR image comprises: Inputting the long-exposure LDR image and the interpolated frame image into an HDR reconstruction module of an HDR image reconstruction model, so as to determine the fusion weights corresponding to the long-exposure LDR image and the interpolated frame image respectively through the HDR reconstruction module; Based on the long-exposure LDR image, the interpolated frame image, and the fusion weights corresponding to the long-exposure LDR image and the interpolated frame image, high dynamic range HDR reconstruction is performed to obtain an HDR image.

8. The method according to any one of claims 1 to 6, characterized in that: The bidirectional optical flow includes a first optical flow from the first short-exposure LDR image to the second short-exposure LDR image, and a second optical flow from the second short-exposure LDR image to the first short-exposure LDR image; The determining of the bidirectional optical flow between the first short-exposure LDR image and the second short-exposure LDR image comprises: Respectively calculating brightness information of the first short-exposure LDR image and the second short-exposure LDR image to obtain first brightness information and second brightness information; Performing noise reduction processing on the first brightness information and the second brightness information respectively to obtain third brightness information and fourth brightness information; The third brightness information and the fourth brightness information are input into an optical flow estimation module of an HDR image reconstruction model, so that the first optical flow and the second optical flow are calculated based on the third brightness information and the fourth brightness information through the optical flow estimation module.

9. An image reconstruction device, characterized in that: include: An acquisition module, configured to acquire an image frame sequence, wherein the image frame sequence includes a first short-exposure low dynamic range LDR image, a long-exposure LDR image, and a second short-exposure LDR image arranged in sequence; a determining module, configured to determine a bidirectional optical flow between the first short-exposure LDR image and the second short-exposure LDR image; The determining module is further used to determine, based on the bidirectional optical flow, an initialization optical flow corresponding to a first predicted interpolation frame image between the first short-exposure LDR image and the second short-exposure LDR image; A refinement module, used to refine the initialization optical flow using the long exposure LDR image as a motion alignment target to obtain a target optical flow; The determination module is further configured to determine an interpolated image based on the target optical flow, the first short-exposure LDR image, and the second short-exposure LDR image; A reconstruction module is used to perform high dynamic range HDR image reconstruction based on the long exposure LDR image and the interpolated frame image to obtain an HDR image.

10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image reconstruction method according to any one of claims 1 to 8 are implemented.

Images