Method and device for generating high dynamic range (HDR) image and electronic equipment

Abstract

The invention relates to a method and device for generating a high dynamic range (HDR) image and electronic equipment, and belongs to the technical field of image processing. The method comprises the following steps: acquiring a plurality of first image frames; based on the exposure degree, determining a fusion sequence of the plurality of first image frames; according to the fusion sequence, performing fusion processing on the plurality of first image frames to obtain a fused image frame; and generating an HDR image based on the fused image frame. Therefore, the fusion sequence of the image frames can be determined in consideration of the exposure degree, flexible determination of the fusion sequence can be realized, and fusion processing is performed on the plurality of image frames in sequence according to the fusion sequence, so that compared with fusion processing performed on all image frames together in the prior art, the image fusion method can more accurately fuse the image, and the image fusion efficiency is improved. The motion blur and ghosting phenomena of the HDR image can be reduced, so that the quality of the HDR image is improved, and the calculation complexity is also reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technologies, and in particular, to a method, apparatus, electronic device, and storage medium for generating a high dynamic range (HDR) image. Background Art

[0002] Currently, HDR (High Dynamic Range) images have advantages such as a wide dynamic range, rich details and colors, and are widely used in fields such as photography, film and television production, and game development. However, in related technologies, most HDR images have motion blur and ghosting phenomena, resulting in low-quality HDR images. Summary of the Invention

[0003] The present disclosure provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for generating a high dynamic range (HDR) image, so as to at least solve the problem that most HDR images in related technologies have motion blur and ghosting phenomena, resulting in low-quality HDR images. The technical solutions of the present disclosure are as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, a method for generating a high dynamic range (HDR) image is provided, including: obtaining a plurality of first image frames; determining a fusion order of the plurality of first image frames based on an exposure degree; performing a fusion process on the plurality of first image frames according to the fusion order to obtain a fused image frame; and generating an HDR image based on the fused image frame.

[0005] According to a second aspect of an embodiment of the present disclosure, an apparatus for generating an HDR image is provided, including: an obtaining module configured to obtain a plurality of first image frames; a determining module configured to determine a fusion order of the plurality of first image frames based on an exposure degree; a fusion module configured to perform a fusion process on the plurality of first image frames according to the fusion order to obtain a fused image frame; and a generating module configured to generate an HDR image based on the fused image frame.

[0006] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, including a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the steps of the method according to the first aspect of the embodiments of the present disclosure.

[0007] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which computer program instructions are stored, and when the program instructions are executed by a processor, the steps of the method according to the first aspect of the embodiments of the present disclosure are implemented.

[0008] According to a fifth aspect of the embodiments of the present disclosure, there is provided a computer program product, including a computer program, characterized in that when the computer program is executed by a processor of an electronic device, it implements the steps of the method described in the first aspect of the embodiments of the present disclosure.

[0009] The technical solutions provided by the embodiments of the present disclosure at least bring the following beneficial effects: The fusion order of image frames can be determined in consideration of the exposure degree, the flexible determination of the fusion order can be achieved, and according to the fusion order, multiple image frames are sequentially fused. Compared with the related art in which all image frames are fused together, the images can be fused more precisely, which helps to reduce the motion blur and ghosting phenomena of HDR images, thereby improving the quality of HDR images and also reducing the computational complexity.

[0010] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure and do not constitute an improper limitation to the present disclosure.

[0012] Figure 1 is a flowchart of a method for generating an HDR image shown according to an exemplary embodiment.

[0013] Figure 2 is a flowchart of a method for generating an HDR image shown according to another exemplary embodiment.

[0014] Figure 3 is a flowchart of a method for generating an HDR image shown according to another exemplary embodiment.

[0015] Figure 4 is a flowchart of a method for generating an HDR image shown according to another exemplary embodiment.

[0016] Figure 5 is a schematic diagram of an image generation model shown according to an exemplary embodiment.

[0017] Figure 6 is a flowchart of a method for generating an HDR image shown according to another exemplary embodiment.

[0018] Figure 7 is a block diagram of an apparatus for generating an HDR image shown according to an exemplary embodiment.

[0019] Figure 8 is a block diagram of an electronic device shown according to an exemplary embodiment. Detailed implementation manners

[0020] In order to enable those of ordinary skill in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0021] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned accompanying drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that such used data can be interchanged under appropriate circumstances so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the present disclosure. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0022] In the technical solutions of the present disclosure, the acquisition, storage, use, processing, etc. of data all comply with the provisions of relevant laws and regulations.

[0023] Figure 1 is a flowchart of a method for generating an HDR image shown according to an exemplary embodiment. As Figure 1 shown, the method for generating an HDR (High Dynamic Range) image in the embodiments of the present disclosure includes the following steps.

[0024] S101, obtain a plurality of first image frames.

[0025] It should be noted that the execution subject of the method for generating an HDR image in the embodiments of the present disclosure is an electronic device, such as a mobile phone, a notebook, a desktop computer, a vehicle-mounted terminal, a smart home appliance, a wearable device, etc. Among them, the wearable device may include a wrist-worn device (such as a smart watch, a smart bracelet), a head-mounted device, a foot-worn device, etc. The method for generating an HDR image in the embodiments of the present disclosure can be executed by the device for generating an HDR image in the embodiments of the present disclosure. The device for generating an HDR image in the embodiments of the present disclosure can be configured in any electronic device to execute the method for generating an HDR image in the embodiments of the present disclosure.

[0026] It should be noted that there are no excessive limitations on the first image frame. For example, it may include an exposure frame, an image frame after exposure frame processing, etc., and may include RGB, HSV, HSL, YCbCr, Lab, YUV images, etc. There are no excessive limitations on the number of types of exposure degrees of the first image frame. For example, the exposure degrees of each first image frame are the same. In this case, the exposure degrees of multiple first image frames are 1 type. Or, at least two first image frames have different exposure degrees. In this case, the exposure degrees of multiple first image frames are at least 2 types. There are no excessive limitations on the number of first image frames for each exposure degree.

[0027] It should be noted that any exposure degree in the related art can be adopted for the exposure degree in the embodiments of the present disclosure. For example, the exposure degree is the product of the exposure time and the exposure gain.

[0028] For example, multiple first image frames include ev11, ev12, ev13, ev21, ev22, ev31. Among them, the exposure time of ev11, ev12, and ev13 is t1, and the exposure gain is gain1. The exposure time of ev21 and ev22 is t2, and the exposure gain is gain2. The exposure time of ev31 is t3, and the exposure gain is gain3. That is, the exposure degrees of ev11, ev12, and ev13 are all gain1 * t1, the exposure degrees of ev21 and ev22 are all gain2 * t2, and the exposure degree of ev31 is gain3 * t3.

[0029] If gain1 * t1 < gain2 * t2 < gain3 * t3, then the exposure degrees of the above 6 first image frames are 3 types. The first image frames of the first type of exposure degree include ev11, ev12, and ev13. The first image frames of the second type of exposure degree include ev21 and ev22. The first image frame of the third type of exposure degree includes ev31. The exposure degrees of the first to third types increase in sequence.

[0030] It should be noted that to obtain multiple first image frames, any shooting method in the related art can be used to achieve this, and no excessive limitations are imposed here. Among them, the shooting method includes bracketing exposure.

[0031] S102. Determine the fusion order of multiple first image frames based on the exposure degree.

[0032] In the embodiments of the present disclosure, determining the fusion order of multiple first image frames based on the exposure degree includes the following possible implementation manners:

[0033] Method 1. Identify that the exposure degree of the second image frame is less than the exposure degree of the third image frame, and determine that the fusion order of the second image frame is before the fusion order of the third image frame.

[0034] In this embodiment, the multiple first image frames include a second image frame and a third image frame. The fusion order of the first image frames is negatively correlated with the exposure degree of the first image frames, that is, the smaller the exposure degree of a first image frame, the earlier the fusion order of this first image frame.

[0035] For example, continuing with ev11, ev12, ev13, ev21, ev22, ev31 as an example, if the exposure degrees of ev11, ev12, and ev13 are all smaller than those of ev21, ev22, and ev31, then the fusion orders of ev11, ev12, and ev13 are all before the fusion orders of ev21, ev22, and ev31. If the exposure degrees of ev21 and ev22 are both smaller than that of ev31, then the fusion orders of ev21 and ev22 are both before the fusion order of ev31.

[0036] Method 2: Identify that the exposure degree of the second image frame is smaller than that of the third image frame, and determine that the fusion order of the second image frame is after the fusion order of the third image frame.

[0037] In this embodiment, the multiple first image frames include a second image frame and a third image frame. The fusion order of the first image frames is positively correlated with the exposure degree of the first image frames, that is, the larger the exposure degree of a first image frame, the earlier the fusion order of this first image frame.

[0038] For example, continuing with ev11, ev12, ev13, ev21, ev22, ev31 as an example, if the exposure degrees of ev11, ev12, and ev13 are all smaller than those of ev21, ev22, and ev31, then the fusion orders of ev11, ev12, and ev13 are all after the fusion orders of ev21, ev22, and ev31. If the exposure degrees of ev21 and ev22 are both smaller than that of ev31, then the fusion orders of ev21 and ev22 are both after the fusion order of ev31.

[0039] Method 3: Identify that the exposure degrees of the multiple fourth image frames are the same, and determine that the fusion orders of the multiple fourth image frames are consecutive.

[0040] In this embodiment, the multiple first image frames include multiple fourth image frames, and the fusion orders of the multiple first image frames with the same fusion degree are consecutive.

[0041] For example, continuing with ev11, ev12, ev13, ev21, ev22, ev31 as an example, if the exposure degrees of ev11, ev12, and ev13 are the same, then the fusion orders of ev11, ev12, and ev13 are consecutive. If the exposure degrees of ev21 and ev22 are the same, then the fusion orders of ev21 and ev22 are consecutive.

[0042] Method 4: Obtain the minimum exposure degree among the exposure degrees of multiple first image frames. If the minimum exposure degree is greater than the set exposure degree, determine that the fusion order of the first image frame with the minimum exposure degree is the first priority. Or, if the minimum exposure degree is less than or equal to the set exposure degree, determine that the fusion order of the first image frame with the third exposure degree is the first priority, where the third exposure degree is greater than the minimum exposure degree.

[0043] In this solution, if the minimum exposure degree is relatively large, the image frame with the minimum exposure degree can be used as the image frame with the first priority.

[0044] It can be understood that if the minimum exposure degree is too small, the signal-to-noise ratio of the image frame with the minimum exposure degree in the dark is too low (i.e., many details in the shadow area are lost). If the image frame with the minimum exposure degree is used as the image frame with the first priority, it will affect the image quality of the HDR image in the dark. In this solution, to solve the above problem, if the minimum exposure degree is too small, the image frame with an exposure degree greater than the minimum exposure degree can be used as the image frame with the first priority, which helps to improve the image quality of the HDR image in the dark.

[0045] It should be noted that there are no excessive restrictions on the set exposure degree and the third exposure degree. For example, the third exposure degree is the normal exposure degree.

[0046] For example, continuing with ev11, ev12, ev13, ev21, ev22, ev31 as an example, if gain1*t1 < gain2*t2 < gain3*t3, then the minimum exposure degree is gain1*t1. If gain1*t1 is greater than the set exposure degree, determine that the fusion order of any one of ev11, ev12, and ev13 is the first priority. Or, if gain1*t1 is less than or equal to the set exposure degree, determine that the fusion order of any one of ev21, ev22, and ev31 is the first priority.

[0047] For example, if gain1*t1 is less than or equal to the set exposure degree and gain2*t2 is the normal exposure degree, then it can be determined that the fusion order of any one of ev21 and ev22 is the first priority.

[0048] S103: Perform a fusion process on multiple first image frames according to the fusion order to obtain a fused image frame.

[0049] It can be understood that performing a fusion process on multiple first image frames according to the fusion order to obtain a fused image frame includes performing at least one fusion process on multiple first image frames according to the fusion order to obtain a fused image frame. There are no excessive restrictions on the number of image frames fused in each fusion process.

[0050] In one implementation, multiple first image frames are fused in a fusion order to obtain a fused image frame, including performing at least one fusion process on multiple first image frames in the fusion order to obtain a fused image frame, where only two image frames are fused in each fusion process. Thus, only two image frames are fused in each fusion process, that is, pairwise fusion, which can fuse images more precisely, helps reduce motion blur and ghosting phenomena in the HDR image, thereby improving the quality of the HDR image and also reducing the computational complexity. Among them, the ghosting phenomenon is also called artifact or double image.

[0051] In some examples, multiple first image frames are fused in a fusion order to obtain a fused image frame, including fusing the first image frame in the first order and the first image frame in the second order to obtain a fused image frame, taking the first image frame in the third order as the current image frame, and fusing the fused image frame and the current image frame to update the fused image frame.

[0052] For example, after fusing the fused image frame and the current image frame to update the fused image frame, it further includes fusing the fused image frame and the next image frame of the current image frame to update the fused image frame. Among them, before fusing the fused image frame and the next image frame of the current image frame, it further includes identifying that the first end condition is not satisfied.

[0053] For example, after fusing the fused image frame and the current image frame to update the fused image frame, it further includes updating the current image frame to the first image frame in the next order of the current image frame, and fusing the fused image frame and the current image frame to update the fused image frame. Among them, before updating the current image frame to the first image frame in the next order of the current image frame, it further includes identifying that the first end condition is not satisfied.

[0054] It should be noted that fusing the fused image frame and the current image frame to update the fused image frame means fusing the fused image frame and the current image frame to obtain a new image frame, and updating the fused image frame to the new image frame.

[0055] It should be noted that the first end condition is not overly limited. For example, it may include the end of the fusion process of the first image frame in the last order, and / or the number of fusion processes reaches a set number, etc.

[0056] For example, continuing with the examples of ev11, ev12, ev13, ev21, ev22, and ev31, if the fusion order of ev11, ev12, ev13, ev21, ev22, and ev31 is ev11, ev12, ev13, ev21, ev22, ev31, that is, the fusion order of ev11 is the first order, and the fusion order of ev31 is the last order.

[0057] Fusion processing can be performed on ev11 and ev12 to obtain a fused image frame ev41. Take ev13 as the current image frame, and perform fusion processing on ev41 and ev13 to update ev41. Update the current image frame to ev21, and perform fusion processing on ev41 and ev21 to update ev41. Update the current image frame to ev22, and perform fusion processing on ev41 and ev22 to update ev41. Update the current image frame to ev31, and perform fusion processing on ev41 and ev31 to update ev41.

[0058] In one implementation, before performing fusion processing on multiple first image frames according to the fusion order, it further includes preprocessing the first image frames to update the first image frames. It should be noted that no excessive limitation is imposed on the preprocessing. For example, the preprocessing may include noise reduction, alignment, etc. Among them, alignment may include brightness alignment, spatial alignment, etc.

[0059] In some examples, before performing fusion processing on multiple first image frames according to the fusion order, it further includes performing noise reduction processing on the first image frame in the first order to update the first image frame in the first order.

[0060] In some examples, before performing fusion processing on multiple first image frames according to the fusion order, it further includes obtaining a candidate exposure ratio between multiple first image frames based on the exposure degrees of the multiple first image frames, and performing brightness alignment on the multiple first image frames based on the candidate exposure ratio.

[0061] It should be noted that the candidate exposure ratio between any two first image frames is the ratio of the exposure degrees of the above-mentioned any two first image frames.

[0062] For example, continuing with the examples of ev11, ev12, ev13, ev21, ev22, and ev31, if gain1 * t1 < gain2 * t2 < gain3 * t3, then the candidate exposure ratio 1 between ev31 and ev11, ev12, ev13 is (gain3 * t3) / (gain1 * t1), and the candidate exposure ratio 2 between ev31 and ev21, ev22 is (gain3 * t3) / (gain2 * t2).

[0063] The product of the original brightness of the pixel points in EV11 and the candidate exposure ratio 1 can be used as the brightness of the pixel points in EV11 after alignment. The product of the original brightness of the pixel points in EV12 and the candidate exposure ratio 1 can be used as the brightness of the pixel points in EV12 after alignment. The product of the original brightness of the pixel points in EV13 and the candidate exposure ratio 1 can be used as the brightness of the pixel points in EV13 after alignment. Thus, the brightness of EV11, EV12, and EV13 can be aligned to EV31.

[0064] The product of the original brightness of the pixel points in EV21 and the candidate exposure ratio 2 can be used as the brightness of the pixel points in EV21 after alignment. The product of the original brightness of the pixel points in EV22 and the candidate exposure ratio 2 can be used as the brightness of the pixel points in EV22 after alignment. Thus, the brightness of EV21 and EV22 can be aligned to EV31.

[0065] S104. Generate an HDR image based on the fused image frame.

[0066] It should be noted that generating an HDR image based on the fused image frame can be implemented by using any method for generating an HDR image in related technologies, and no further limitation is imposed here.

[0067] In one embodiment, generating an HDR image based on the fused image frame includes performing demosaicing on the fused image frame to obtain the HDR image. It should be noted that terms such as demosaicing and de-mosaicing can be used interchangeably.

[0068] The method for generating an HDR image provided by the embodiments of the present disclosure includes obtaining a plurality of first image frames, determining the fusion order of the plurality of first image frames based on the exposure degree, performing fusion processing on the plurality of first image frames in accordance with the fusion order to obtain a fused image frame, and generating an HDR image based on the fused image frame. Thus, the fusion order of the image frames can be determined considering the exposure degree, the flexible determination of the fusion order can be achieved, and the plurality of image frames can be sequentially fused in accordance with the fusion order. Compared with the related technology of fusing all image frames together, the images can be fused more precisely, which helps to reduce the motion blur and ghosting phenomena in the HDR image, thereby improving the quality of the HDR image and reducing the computational complexity.

[0069] Figure 2 It is a flowchart of a method for generating an HDR image shown in another exemplary embodiment, as Figure 2 shown, the method for generating an HDR image of the embodiments of the present disclosure includes the following steps.

[0070] S201. Obtain a plurality of first image frames.

[0071] For the relevant content of step S201, reference can be made to the above embodiments, and details are not described herein again.

[0072] S202. Obtain the target exposure ratio between multiple first image frames based on the exposure levels of the multiple first image frames.

[0073] It should be noted that the multiple candidate exposure ratios include the target exposure ratio, and the number of target exposure ratios is at least one. For the relevant content of the candidate exposure ratios, reference can be made to the above embodiments and will not be elaborated here.

[0074] In one implementation, obtaining the target exposure ratio between multiple first image frames based on the exposure levels of the multiple first image frames includes obtaining the first exposure level and the minimum exposure level among the exposure levels of the multiple first image frames, where the first exposure level is greater than the minimum exposure level, and obtaining the ratio between the first exposure level and the minimum exposure level as the target exposure ratio. Thus, the ratio between the first exposure level greater than the minimum exposure level and the minimum exposure level can be obtained as the target exposure ratio.

[0075] It should be noted that there are no excessive limitations on the first exposure level. For example, the first exposure level includes the normal exposure level, the maximum exposure level, etc.

[0076] In one implementation, obtaining the target exposure ratio between multiple first image frames based on the exposure levels of the multiple first image frames includes obtaining the fourth exposure level and the maximum exposure level among the exposure levels of the multiple first image frames, where the fourth exposure level is less than the maximum exposure level, and obtaining the ratio between the maximum exposure level and the fourth exposure level as the target exposure ratio. Thus, the ratio between the maximum exposure level and the fourth exposure level less than the maximum exposure level can be obtained as the target exposure ratio.

[0077] It should be noted that there are no excessive limitations on the fourth exposure level. For example, the fourth exposure level includes the normal exposure level, the minimum exposure level, etc.

[0078] In one implementation, obtaining the target exposure ratio between multiple first image frames based on the exposure levels of the multiple first image frames includes obtaining the maximum exposure level and the minimum exposure level among the exposure levels of the multiple first image frames, and obtaining the ratio between the maximum exposure level and the minimum exposure level as the target exposure ratio. Thus, the ratio between the maximum exposure level and the minimum exposure level can be obtained as the target exposure ratio.

[0079] S203. Determine the fusion order based on the target exposure ratio.

[0080] In one implementation, determining the fusion order based on the target exposure ratio includes inputting the target exposure ratio into an order determination model, and the order determination model outputs the fusion order. It should be noted that there are no excessive limitations on the order determination model. For example, it can include a data-driven model, a mechanism model, etc.

[0081] In one embodiment, determining the fusion order based on the target exposure ratio includes determining a reference image frame from multiple first image frames based on the target exposure ratio, and determining that the fusion order of the reference image frame is the first order. Thus, the reference image frame can be determined in consideration of the target exposure ratio, and the reference image frame is used as the first image frame in the first order.

[0082] In some examples, obtaining the target exposure ratio between multiple first image frames based on the exposure levels of the multiple first image frames includes obtaining a fourth exposure level and a maximum exposure level among the exposure levels of the multiple first image frames, where the fourth exposure level is less than the maximum exposure level, and obtaining the ratio between the maximum exposure level and the fourth exposure level as the target exposure ratio.

[0083] Determining a reference image frame from multiple first image frames based on the target exposure ratio includes, if the target exposure ratio is less than or equal to a set threshold, using the first image frame with the maximum exposure level as the reference image frame, and if the target exposure ratio is greater than the set threshold, using the first image frame with a fifth exposure level as the reference image frame, where the fifth exposure level is less than the maximum exposure level. Thus, in the case where the target exposure ratio is the ratio between the maximum exposure level and the fourth exposure level, if the target exposure ratio is small, the image frame with the maximum exposure level can be used as the reference image frame. If the target exposure ratio is too large, the signal-to-noise ratio of the image frame with the maximum exposure level in the bright area is too low (i.e., many details in the highlight area are lost). If the image frame with the maximum exposure level is used as the reference image frame, it will affect the image quality of the HDR image in the bright area. In this solution, to solve the above problem, if the target exposure ratio is too large, an image frame with an exposure level less than the maximum exposure level can be used as the reference image frame, which helps to improve the image quality of the HDR image in the bright area.

[0084] It should be noted that the set threshold and the fifth exposure level are not overly limited. For example, the fifth exposure level includes a normal exposure level, a minimum exposure level, etc. For example, the first image frame with the fifth exposure level is a normally exposed image frame. For example, the fourth exposure level is the same as the fifth exposure level.

[0085] For example, continuing to take ev11, ev12, ev13, ev21, ev22, ev31 as an example, if gain1*t1 < gain2*t2 < gain3*t3, then the maximum exposure level is gain3*t3, and the fourth exposure level and the fifth exposure level are gain1*t1 or gain2*t2.

[0086] If the fourth exposure level is gain1 * t1, the target exposure ratio = (gain3 * t3) / (gain1 * t1); or, if the fourth exposure level is gain2 * t2, the target exposure ratio = (gain3 * t3) / (gain2 * t2).

[0087] If the target exposure ratio is less than or equal to the set threshold, ev31 is used as the reference image frame.

[0088] If the target exposure ratio is greater than the set threshold and the fifth exposure level is gain1 * t1, any one of ev11, ev12, and ev13 is used as the reference image frame.

[0089] If the target exposure ratio is greater than the set threshold and the fifth exposure level is gain2 * t2, any one of ev21 and ev22 is used as the reference image frame.

[0090] S204. Perform a fusion process on multiple first image frames in accordance with the fusion order to obtain a fused image frame.

[0091] S205. Generate an HDR image based on the fused image frame.

[0092] For the relevant content of steps S204 - S205, reference can be made to the above embodiments and will not be elaborated here.

[0093] The method for generating an HDR image provided by the embodiments of the present disclosure obtains the target exposure ratio between multiple first image frames based on the exposure levels of the multiple first image frames, and determines the fusion order based on the target exposure ratio. Thus, the exposure ratio can be obtained based on the exposure level, and the fusion order of the image frames can be determined considering the exposure ratio, enabling flexible determination of the fusion order.

[0094] Figure 3 is a flowchart of a method for generating an HDR image shown according to another exemplary embodiment. As Figure 3 shown, the method for generating an HDR image of the embodiments of the present disclosure includes the following steps.

[0095] S301. Obtain multiple first image frames.

[0096] S302. Obtain the first exposure level and the minimum exposure level among the exposure levels of the multiple first image frames, where the first exposure level is greater than the minimum exposure level.

[0097] S303. Obtain the ratio between the first exposure level and the minimum exposure level as the target exposure ratio.

[0098] For the relevant content of steps S301 - S303, reference can be made to the above embodiments and will not be elaborated here

[0099] S304, if the target exposure ratio is less than or equal to the set threshold, use the first image frame with the minimum exposure level as the reference image frame.

[0100] S305, if the target exposure ratio is greater than the set threshold, use the first image frame with the second exposure level as the reference image frame, where the second exposure level is greater than the minimum exposure level.

[0101] In this solution, if the target exposure ratio is small, the image frame with the minimum exposure level can be used as the reference image frame.

[0102] It can be understood that if the target exposure ratio is too large, the signal-to-noise ratio of the image frame with the minimum exposure level in the dark is too low (i.e., a lot of details in the shadow area are lost). If the image frame with the minimum exposure level is used as the reference image frame, it will affect the image quality of the HDR image in the dark. To solve the above problem in this solution, if the target exposure ratio is too large, the image frame with an exposure level greater than the minimum exposure level can be used as the reference image frame, which helps to improve the image quality of the HDR image in the dark.

[0103] It should be noted that there are no excessive restrictions on the set threshold and the second exposure level. For example, the second exposure level includes the normal exposure level, the maximum exposure level, etc. For example, the first image frame with the second exposure level is a normally exposed image frame. For example, the first exposure level is the same as the second exposure level.

[0104] For example, continuing with ev11, ev12, ev13, ev21, ev22, ev31 as an example, if gain1*t1 < gain2*t2 < gain3*t3, then the minimum exposure level is gain1*t1, and the first exposure level and the second exposure level are gain2*t2 or gain3*t3.

[0105] If the first exposure level is gain2*t2, then the target exposure ratio = (gain2*t2) / (gain1*t1), or if the first exposure level is gain3*t3, then the target exposure ratio = (gain3*t3) / (gain1*t1).

[0106] If the target exposure ratio is less than or equal to the set threshold, use any one of ev11, ev12, ev13 as the reference image frame.

[0107] If the target exposure ratio is greater than the set threshold and the second exposure level is gain2*t2, use any one of ev21, ev22 as the reference image frame.

[0108] If the target exposure ratio is greater than the set threshold and the second exposure level is gain3*t3, use ev31 as the reference image frame.

[0109] S306. Determine that the fusion order of the reference image frame is the first priority.

[0110] S307. According to the fusion order, perform fusion processing on multiple first image frames to obtain a fused image frame.

[0111] S308. Generate an HDR image based on the fused image frame.

[0112] For the relevant content of steps S306 - S308, reference can be made to the above - mentioned embodiments and will not be elaborated here.

[0113] In the method for generating an HDR image provided by the embodiments of the present disclosure, when the target exposure ratio is the ratio between the first exposure level and the minimum exposure level, if the target exposure ratio is small, the image frame with the minimum exposure level can be used as the reference image frame. If the target exposure ratio is too large, the image frame with an exposure level greater than the minimum exposure level can be used as the reference image frame, which helps to improve the image quality of the HDR image in the dark.

[0114] Based on any of the above - mentioned embodiments, performing fusion processing on multiple first image frames according to the fusion order to obtain a fused image frame includes inputting multiple first image frames into the fusion network in the image generation model, and through the fusion network, performing fusion processing on multiple first image frames according to the fusion order to obtain a fused image frame.

[0115] It should be noted that no excessive limitations are imposed on the image generation model and the fusion network. For example, the image generation model is a non - linear mapping model. For example, the image generation model may include Unet network, Transform network, convolutional neural network (such as residual network), recurrent neural network, fully - connected network, generative adversarial network, etc. For example, the Unet network is a residual architecture.

[0116] It should be noted that for the relevant content of performing fusion processing on multiple first image frames according to the fusion order through the fusion network to obtain a fused image frame, reference can be made to the relevant content of step S103 in the above - mentioned embodiments and will not be elaborated here.

[0117] In one implementation, as Figure 5 shown, the image generation model further includes a noise reduction network. Before performing fusion processing on multiple first image frames according to the fusion order, it further includes inputting the first image frame in the first priority into the noise reduction network, and performing noise reduction processing on the first image frame in the first priority through the noise reduction network to update the first image frame in the first priority.

[0118] In one implementation, as Figure 5As shown, the image generation model further includes a demosaicing network. Based on the fused image frames, an HDR image is generated, including inputting the fused image frames into the demosaicing network, and performing demosaicing processing on the fused image frames through the demosaicing network to obtain the HDR image.

[0119] It should be noted that the fusion network, the noise reduction network, and the demosaicing network are not overly limited. For example, they are all Unet networks.

[0120] Figure 4 is a flowchart of a method for generating an HDR image shown according to another exemplary embodiment. As Figure 4 shown, the method for generating an HDR image according to the embodiments of the present disclosure includes the following steps.

[0121] S401, Obtain a plurality of first image frames.

[0122] S402, Based on the exposure degree, determine the fusion order of the plurality of first image frames.

[0123] S403, Input the plurality of first image frames into the fusion network in the image generation model.

[0124] For the relevant content of steps S401 - S403, reference can be made to the above embodiments, and details will not be repeated here.

[0125] S404, Through the first fusion sub - network, according to the fusion order, perform fusion processing on the plurality of first image frames with the first exposure degree to obtain a fused image frame.

[0126] S405, Through the i - th fusion sub - network, according to the fusion order, perform fusion processing on the fused image frame and the first image frame with the i - th exposure degree to update the fused image frame, where i is a positive integer not greater than N.

[0127] S406, If the fusion processing of each first image frame with the i - th exposure degree is completed, through the j - th fusion sub - network, according to the fusion order, perform fusion processing on the fused image frame and the first image frame with the j - th exposure degree to update the fused image frame, where j is a positive integer not greater than N.

[0128] In this embodiment, the exposure degrees of the plurality of first image frames are N types, and the fusion network includes N fusion sub - networks, where N is a positive integer. For example, as Figure 5 shown, taking N = 3 as an example, the fusion network includes 3 fusion sub - networks, namely the first to the third fusion sub - networks.

[0129] For example, the initial value of i is 2, and the initial value of j is 3. For example, j = i + 1.

[0130] In one embodiment, before the fused image frame and the first image frame with the second exposure level are fused through the second fusion sub-network in the fusion order, it further includes that if there is a first image frame with the first exposure level that has not been fused, the first fusion sub-network is used to fuse the fused image frame and the first image frame with the first exposure level that has not been fused in the fusion order to update the fused image frame. Thus, when the number of frames of the first image frame with the first exposure level is greater than or equal to 3, multiple fusion processes can be performed through the first fusion sub-network until all the first image frames with the first exposure level are fused.

[0131] In one embodiment, before the fused image frame and the first image frame with the j-th exposure level are fused through the j-th fusion sub-network in the fusion order to update the fused image frame, it further includes that if there is a first image frame with the i-th exposure level that has not been fused, the i-th fusion sub-network is used to fuse the fused image frame and the first image frame with the i-th exposure level that has not been fused in the fusion order to update the fused image frame. Thus, when the number of frames of the first image frame with the i-th exposure level is greater than or equal to 2, multiple fusion processes can be performed through the i-th fusion sub-network until each first image frame with the i-th exposure level is fused.

[0132] For example, continuing with ev11, ev12, ev13, ev21, ev22, ev31 as an example, if the fusion order of ev11, ev12, ev13, ev21, ev22, ev31 is ev11, ev12, ev13, ev21, ev22, ev31, that is, the fusion order of ev11 is the first position and the fusion order of ev31 is the last position, and gain1*t1 < gain2*t2 < gain3*t3.

[0133] As Figure 5 shown, the first fusion sub-network can be used to fuse ev11 and ev12 to obtain the fused image frame ev41, and the first fusion sub-network can be used to fuse ev41 and ev13 to update ev41.

[0134] When i = 2, the second fusion sub-network can be used to fuse ev41 and ev21 to update ev41, and the second fusion sub-network can be used to fuse ev41 and ev22 to update ev41.

[0135] When i = 3, the third fusion sub-network can be used to fuse ev41 and ev31 to update ev41.

[0136] S407. Generate an HDR image based on the fused image frame.

[0137] For the relevant content of step S407, refer to the above embodiments, which will not be elaborated here.

[0138] The method for generating an HDR image provided by the embodiment of the present disclosure uses the first fusion sub-network to perform fusion processing on multiple first image frames with the first exposure degree in the fusion order to obtain a fused image frame, and uses the i-th fusion sub-network to perform fusion processing on the fused image frame and the first image frame with the i-th exposure degree in the fusion order to update the fused image frame, where i is a positive integer not greater than N. If the fusion processing of each first image frame with the i-th exposure degree is completed, the j-th fusion sub-network is used to perform fusion processing on the fused image frame and the first image frame with the j-th exposure degree in the fusion order to update the fused image frame. Thus, multiple fusion sub-networks in the image generation model can be used to perform fusion processing on image frames with multiple exposure degrees in the fusion order respectively.

[0139] Figure 6 is a flowchart of a method for generating an HDR image shown according to an exemplary embodiment. As Figure 6 shown, the method for generating an HDR image in the embodiment of the present disclosure includes the following steps.

[0140] S601: Keep the model parameters of the fusion network unchanged and train the noise reduction network and the demosaicing network.

[0141] S602: Keep the model parameters of the noise reduction network and the remaining fusion sub-networks other than the m-th fusion sub-network unchanged, and train the m-th fusion sub-network and the demosaicing network, where m is a positive integer not greater than N.

[0142] S603: Keep the model parameters of the noise reduction network and the remaining fusion sub-networks other than the n-th fusion sub-network unchanged, and train the n-th fusion sub-network and the demosaicing network, where n is a positive integer not greater than N.

[0143] In this embodiment, the fusion network includes N fusion sub-networks, N is a positive integer, the image generation model further includes a noise reduction network and a demosaicing network. The training process of the demosaicing network includes N + 1 stages. After the training of the current stage of the demosaicing network is completed, the demosaicing network inherits the model parameters obtained from the training of the current stage.

[0144] It should be noted that in step S601, during the training of the noise reduction network and the demosaicing network, the model parameters of the fusion network are fixed. In step S602, during the training of the m-th fusion sub-network and the demosaicing network, the model parameters of the noise reduction network are fixed, and the model parameters of the remaining fusion sub-networks other than the m-th fusion sub-network are fixed. In step S603, during the training of the n-th fusion sub-network and the demosaicing network, the model parameters of the noise reduction network are fixed, and the model parameters of the remaining fusion sub-networks other than the n-th fusion sub-network are fixed.

[0145] It should be noted that the training process of the demosaicing network includes N + 1 stages. In step S601, the demosaicing network is trained in the first stage. In steps S602 and S603, the demosaicing network is trained in the 2nd to N + 1st stages. After the current stage training of the demosaicing network ends, the demosaicing network inherits the model parameters obtained from the current stage training, which refers to the model parameters obtained from the current stage training of the demosaicing network, as the initial model parameters for the next stage training of the demosaicing network.

[0146] For example, the initial value of m is 1, and the initial value of n is 2. For example, n = m + 1.

[0147] In one implementation, before training the m-th fusion sub-network and the demosaicing network, it also includes identifying the fusion sub-networks that have not completed training. Before training the n-th fusion sub-network and the demosaicing network, it also includes identifying the fusion sub-networks that have not completed training.

[0148] For example, continuing with Figure 5 as an example, the model parameters of the fusion network are fixed, and the noise reduction network and the demosaicing network are trained. This step trains the demosaicing network in the first stage. After the first stage training of the demosaicing network ends, the demosaicing network inherits the model parameters obtained from the first stage training.

[0149] The model parameters of the noise reduction network, the 2nd fusion sub-network, and the 3rd fusion sub-network are fixed, and the 1st fusion sub-network and the demosaicing network are trained. It should be noted that the demosaicing network is trained in the second stage. After the second stage training of the demosaicing network ends, the demosaicing network inherits the model parameters obtained from the second stage training.

[0150] The model parameters of the noise reduction network, the 1st fusion sub-network, and the 3rd fusion sub-network are fixed, and the 2nd fusion sub-network and the demosaicing network are trained. It should be noted that this step trains the demosaicing network in the third stage. After the second stage training of the demosaicing network ends, the demosaicing network inherits the model parameters obtained from the third stage training.

[0151] The model parameters of the fixed noise reduction network, the first fusion sub-network, and the second fusion sub-network remain unchanged, and the third fusion sub-network and the demosaicing network are trained. It should be noted that the fourth-stage training of the demosaicing network is carried out. After the fourth-stage training of the demosaicing network is completed, the demosaicing network inherits the model parameters obtained from the fourth-stage training.

[0152] It should be noted that for the training of each network of the image generation model, any model training method in the related art can be used to implement it, and no excessive limitation is made here. For example, taking step S601 as an example, before training the noise reduction network and the demosaicing network, the output of the noise reduction network can be used as the input of the demosaicing network.

[0153] In one implementation, the training of the noise reduction network and the demosaicing network includes obtaining a plurality of sample first image frames and a sample HDR image, determining the fusion order of the plurality of sample first image frames based on the exposure degree, inputting the plurality of sample first image frames into the image generation model, outputting a predicted HDR image by the image generation model, and training the noise reduction network and the demosaicing network based on the predicted HDR image and the sample HDR image.

[0154] In some examples, the training of the noise reduction network and the demosaicing network based on the predicted HDR image and the sample HDR image includes obtaining the loss functions of the noise reduction network and the demosaicing network based on the predicted HDR image and the sample HDR image, and training the noise reduction network and the demosaicing network based on the loss functions. It should be noted that no excessive limitation is made on the loss functions. For example, it may include CE (Cross Entropy), MSE (Mean-Square Error), KL (Kullback-Leibler) divergence, contrast loss function, etc. Among them, CE may include multi-class cross entropy.

[0155] It should be noted that for the training of the m-th fusion sub-network and the demosaicing network, the relevant content of the training of the n-th fusion sub-network and the demosaicing network can refer to the relevant content of the training of the noise reduction network and the demosaicing network in the above embodiments, and will not be elaborated here.

[0156] In some examples, the method further includes obtaining a sample HDR image, performing noise addition processing on the sample HDR image to obtain a plurality of sample first image frames. It should be noted that the noise addition processing can be implemented by any image noise addition method in the related art, and no excessive limitation is made here. The noises added to different sample first image frames may be different, and no excessive limitation is made on the noises. For example, it may include Gaussian noise, uniform noise, salt-and-pepper noise, etc.

[0157] The method for generating an HDR image provided by an embodiment of the present disclosure keeps the model parameters of the fixed fusion network unchanged, trains the denoising network and the demosaicing network, keeps the model parameters of the denoising network and the remaining fusion sub-networks other than the m-th fusion sub-network unchanged, and trains the m-th fusion sub-network and the demosaicing network, where m is a positive integer not greater than N, keeps the model parameters of the denoising network and the remaining fusion sub-networks other than the n-th fusion sub-network unchanged, and trains the n-th fusion sub-network and the demosaicing network, where n is a positive integer not greater than N. The training process of the demosaicing network includes N + 1 stages. After the training of the current stage of the demosaicing network ends, the demosaicing network inherits the model parameters obtained from the training of the current stage. Thus, multi-stage decoupled training can be performed on the image generation model, which helps to reduce the training complexity, improve the training efficiency, and improve the model performance.

[0158] Figure 7 is a block diagram of a device for generating an HDR image shown according to an exemplary embodiment. Referring to Figure 7 The device 100 for generating an HDR image according to an embodiment of the present disclosure includes: an acquisition module 110, a determination module 120, a fusion module 130, and a generation module 140.

[0159] The acquisition module 110 is configured to acquire a plurality of first image frames;

[0160] The determination module 120 is configured to determine the fusion order of the plurality of first image frames based on the exposure degree;

[0161] The fusion module 130 is configured to perform a fusion process on the plurality of first image frames according to the fusion order to obtain a fused image frame;

[0162] The generation module 140 is configured to generate an HDR image based on the fused image frame.

[0163] In an embodiment of the present disclosure, the fusion module 130 is further configured to: perform at least one fusion process on the plurality of first image frames according to the fusion order to obtain the fused image frame, where only two image frames are fused in each fusion process.

[0164] In an embodiment of the present disclosure, the fusion module 130 is further configured to: perform a fusion process on the first image frame in the first order and the first image frame in the second order to obtain the fused image frame; use the first image frame in the third order as the current image frame, and perform a fusion process on the fused image frame and the current image frame to update the fused image frame.

[0165] In one embodiment of the present disclosure, when multiple first image frames include a second image frame and a third image frame, the determining module 120 is further configured to: identify that the exposure degree of the second image frame is less than that of the third image frame; determine that the fusion order of the second image frame is before that of the third image frame.

[0166] In one embodiment of the present disclosure, when multiple first image frames include multiple fourth image frames, the determining module 120 is further configured to: identify that the exposure degrees of the multiple fourth image frames are the same; determine that the fusion orders of the multiple fourth image frames are consecutive.

[0167] In one embodiment of the present disclosure, the determining module 120 is further configured to: obtain a target exposure ratio between multiple first image frames based on the exposure degrees of the multiple first image frames; determine the fusion order based on the target exposure ratio.

[0168] In one embodiment of the present disclosure, the determining module 120 is further configured to: determine a reference image frame from multiple first image frames based on the target exposure ratio; determine that the fusion order of the reference image frame is the first position.

[0169] In one embodiment of the present disclosure, the determining module 120 is further configured to: obtain a first exposure degree and a minimum exposure degree among the exposure degrees of multiple first image frames, where the first exposure degree is greater than the minimum exposure degree; obtain a ratio between the first exposure degree and the minimum exposure degree as the target exposure ratio; if the target exposure ratio is less than or equal to a set threshold, use the first image frame with the minimum exposure degree as the reference image frame; if the target exposure ratio is greater than the set threshold, use the first image frame with a second exposure degree as the reference image frame, where the second exposure degree is greater than the minimum exposure degree.

[0170] In one embodiment of the present disclosure, the first image frame with the second exposure degree is a normally exposed image frame.

[0171] In one embodiment of the present disclosure, the fusion module 130 is further configured to: input multiple first image frames into a fusion network in an image generation model; through the fusion network, perform a fusion process on multiple first image frames according to the fusion order to obtain the fused image frame.

[0172] In one embodiment of the present disclosure, when the exposure degrees of multiple first image frames are N types, the fusion network includes N fusion sub-networks, where N is a positive integer;

[0173] The fusion module 130 is further configured to: through the first fusion sub-network, fuse multiple first image frames of the first exposure level in accordance with the fusion order to obtain the fused image frame; through the i-th fusion sub-network, fuse the fused image frame and the first image frame of the i-th exposure level in accordance with the fusion order to update the fused image frame, where i is a positive integer not greater than N; if the fusion process of each first image frame of the i-th exposure level is completed, through the j-th fusion sub-network, fuse the fused image frame and the first image frame of the j-th exposure level in accordance with the fusion order to update the fused image frame, where j is a positive integer not greater than N.

[0174] In an embodiment of the present disclosure, before the step of fusing the fused image frame and the first image frame of the j-th exposure level through the j-th fusion sub-network in accordance with the fusion order to update the fused image frame, the fusion module 130 is further configured to: if there is a first image frame of the i-th exposure level that has not been fused, fuse the fused image frame and the unfused first image frame of the i-th exposure level through the i-th fusion sub-network in accordance with the fusion order to update the fused image frame.

[0175] In an embodiment of the present disclosure, the image generation model further includes a noise reduction network. Before fusing multiple first image frames in accordance with the fusion order, the fusion module 130 is further configured to: input the first image frame in the first order into the noise reduction network; perform noise reduction processing on the first image frame in the first order through the noise reduction network to update the first image frame in the first order.

[0176] In an embodiment of the present disclosure, the image generation model further includes a demosaicing network. The generation module 140 is further configured to: input the fused image frame into the demosaicing network; perform demosaicing processing on the fused image frame through the demosaicing network to obtain the HDR image.

[0177] In an embodiment of the present disclosure, the fusion network includes N fusion sub-networks, where N is a positive integer, and the image generation model further includes a noise reduction network and a demosaicing network;

[0178] The apparatus 100 further includes: a training module, configured to: keep the model parameters of the fusion network unchanged and train the denoising network and the demosaicking network; keep the model parameters of the denoising network and the remaining fusion sub-networks other than the m-th fusion sub-network unchanged and train the m-th fusion sub-network and the demosaicking network, where m is a positive integer not greater than N; keep the model parameters of the denoising network and the remaining fusion sub-networks other than the n-th fusion sub-network unchanged and train the n-th fusion sub-network and the demosaicking network, where n is a positive integer not greater than N; wherein, the training process of the demosaicking network includes N + 1 stages, and after the training of the current stage of the demosaicking network ends, the demosaicking network inherits the model parameters obtained from the training of the current stage.

[0179] Regarding the apparatus in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.

[0180] The HDR image generation apparatus provided by an embodiment of the present disclosure obtains a plurality of first image frames, determines the fusion order of the plurality of first image frames based on the exposure degree, and performs a fusion process on the plurality of first image frames according to the fusion order to obtain a fused image frame, and generates an HDR image based on the fused image frame. Thus, the fusion order of the image frames can be determined considering the exposure degree, the flexible determination of the fusion order can be achieved, and the plurality of image frames are sequentially fused according to the fusion order. Compared with the related art in which all image frames are fused together, the images can be fused more precisely, which helps to reduce the motion blur and ghosting phenomena in the HDR image, thereby improving the quality of the HDR image and reducing the computational complexity.

[0181] Figure 8 It is a block diagram of an electronic device shown according to an exemplary embodiment.

[0182] As Figure 8 shown, the above electronic device 200 includes:

[0183] a memory 210 and a processor 220, and a bus 230 connecting different components (including the memory 210 and the processor 220), where the memory 210 stores a computer program, and when the processor 220 executes the program, the HDR image generation method described in the embodiments of the present disclosure is implemented.

[0184] Bus 230 represents one or more of several types of bus architectures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. By way of example, such architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0185] Electronic device 200 typically includes a variety of electronic device-readable media. These media can be any available media that can be accessed by electronic device 200, including volatile and nonvolatile media, removable and non-removable media.

[0186] Memory 210 may also include computer system-readable media in the form of volatile memory, such as random access memory (RAM) 240 and / or cache memory 250. Electronic device 200 may further include other removable / non-removable, volatile / nonvolatile computer system storage media. By way of example only, storage system 260 can be used for reading and writing non-removable, nonvolatile magnetic media ( Figure 8 not shown and typically called a "hard disk drive"). Although Figure 8 not shown in the figure, a disk drive for reading and writing a removable nonvolatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing a removable nonvolatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) can be provided. In these cases, each drive can be connected to bus 230 via one or more data media interfaces. Memory 210 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the embodiments of the present disclosure.

[0187] A program / utility 280 having a set (at least one) of program modules 270 can be stored, for example, in memory 210, and such program modules 270 include—but are not limited to—an operating system, one or more application programs, other program modules, and program data, and an implementation of a network environment may be included in each or some combination of these examples. Program modules 270 generally perform the functions and / or methods in the embodiments described in the present disclosure.

[0188] The electronic device 200 can also communicate with one or more external devices 290 (such as a keyboard, a pointing device, a display 291, etc.), and can also communicate with one or more devices that enable a user to interact with the electronic device 200, and / or communicate with any device that enables the electronic device 200 to communicate with one or more other computing devices (such as a network card, a modem, etc.). Such communication can be carried out through an input / output (I / O) interface 292. Moreover, the electronic device 200 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter 293. As Figure 8 shown, the network adapter 293 communicates with other modules of the electronic device 200 through a bus 230. It should be understood that although not shown in the figure, other hardware and / or software modules can be used in combination with the electronic device 200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0189] The processor 220 executes various functional applications and data processing by running programs stored in the memory 210.

[0190] It should be noted that for the implementation process and technical principle of the electronic device in this embodiment, refer to the foregoing explanation of the method for generating an HDR image in the embodiments of the present disclosure, and details are not described herein again.

[0191] The electronic device provided in the embodiments of the present disclosure can execute the method for generating an HDR image as described above, obtain a plurality of first image frames, determine the fusion order of the plurality of first image frames based on the exposure degree, perform fusion processing on the plurality of first image frames according to the fusion order to obtain a fused image frame, and generate an HDR image based on the fused image frame. Thus, the fusion order of the image frames can be determined considering the exposure degree, the flexible determination of the fusion order can be realized, and the plurality of image frames are sequentially fused according to the fusion order. Compared with the related art in which all image frames are fused together, the images can be fused more precisely, which helps to reduce the motion blur and ghost phenomena in the HDR image, thereby improving the quality of the HDR image and also reducing the computational complexity.

[0192] To implement the above embodiments, the present disclosure also proposes a computer-readable storage medium, on which computer program instructions are stored, and when the program instructions are executed by a processor, the steps of the method for generating an HDR image provided by the present disclosure are implemented.

[0193] Optionally, the computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0194] To implement the above embodiments, the present disclosure also provides a computer program product, including a computer program, characterized in that when the computer program is executed by a processor of an electronic device, it implements the method for generating an HDR image as described above.

[0195] After considering the specification and practicing the invention disclosed herein, those skilled in the art will readily conceive of other embodiments of the present disclosure. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure, which follow the general principles of the present disclosure and include known common general knowledge or conventional technical means in the technical field not disclosed by the present disclosure. The specification and examples are only regarded as exemplary, and the true scope and spirit of the present disclosure are pointed out by the following claims.

[0196] It should be understood that the present disclosure is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present disclosure is only limited by the appended claims.

Claims

1. A method for generating a high dynamic range (HDR) image, characterized in that, Including: Obtain a plurality of first image frames; Determine the fusion order of the plurality of first image frames based on the exposure degree; Perform a fusion process on the plurality of first image frames according to the fusion order to obtain a fused image frame; Generate an HDR image based on the fused image frame.

2. The method according to claim 1, characterized in that, The performing a fusion process on the plurality of first image frames according to the fusion order to obtain a fused image frame includes: Perform at least one fusion process on the plurality of first image frames according to the fusion order to obtain the fused image frame, wherein only two image frames are fused in each fusion process.

3. The method according to claim 2, wherein The performing at least one fusion process on the plurality of first image frames according to the fusion order to obtain the fused image frame includes: Fuse the first image frame in the first order and the first image frame in the second order to obtain the fused image frame; Use the first image frame in the third order as the current image frame, and fuse the fused image frame and the current image frame to update the fused image frame.

4. The method according to claim 1, wherein When the plurality of first image frames include a second image frame and a third image frame, the determining the fusion order of the plurality of first image frames based on the exposure degree includes: Identify that the exposure degree of the second image frame is less than the exposure degree of the third image frame; Determine that the fusion order of the second image frame is before the fusion order of the third image frame.

5. The method according to claim 1, characterized in that, When the plurality of first image frames include a plurality of fourth image frames, the determining the fusion order of the plurality of first image frames based on the exposure degree includes: Identify that the exposure degrees of the plurality of fourth image frames are the same; Determine that the fusion orders of the plurality of fourth image frames are consecutive.

6. The method according to claim 1, characterized in that The determining the fusion order of the plurality of first image frames based on the exposure degree includes: Obtain a target exposure ratio between the plurality of first image frames based on the exposure degrees of the plurality of first image frames; Determine the fusion order based on the target exposure ratio.

7. The method according to claim 6, wherein The determining the fusion order based on the target exposure ratio includes: Determine a reference image frame from the plurality of first image frames based on the target exposure ratio; Determine that the fusion order of the reference image frame is the first order.

8. The method according to claim 7, characterized in that, The obtaining a target exposure ratio between the plurality of first image frames based on the exposure degrees of the plurality of first image frames includes: Obtain a first exposure degree and a minimum exposure degree among the exposure degrees of the plurality of first image frames, wherein the first exposure degree is greater than the minimum exposure degree; Obtain the ratio between the first exposure degree and the minimum exposure degree as the target exposure ratio; The determining a reference image frame from the plurality of first image frames based on the target exposure ratio includes: If the target exposure ratio is less than or equal to a set threshold, use the first image frame with the minimum exposure degree as the reference image frame; If the target exposure ratio is greater than the set threshold, use the first image frame with a second exposure degree as the reference image frame, wherein the second exposure degree is greater than the minimum exposure degree.

9. The method according to claim 8, wherein The first image frame with the second exposure degree is a normally exposed image frame.

10. The method according to any one of claims 1-9, characterized in that, Performing fusion processing on the multiple first image frames according to the fusion order to obtain a fused image frame, including: Inputting the multiple first image frames into a fusion network in an image generation model; Performing fusion processing on the multiple first image frames according to the fusion order through the fusion network to obtain the fused image frame.

11. The method according to claim 10, wherein When there are N types of exposure degrees for the multiple first image frames, the fusion network includes N fusion sub-networks, where N is a positive integer; The performing fusion processing on the multiple first image frames according to the fusion order through the fusion network to obtain the fused image frame includes: Performing fusion processing on the multiple first image frames with the first type of exposure degree according to the fusion order through the first fusion sub-network to obtain the fused image frame; Performing fusion processing on the fused image frame and the first image frames with the i-th type of exposure degree according to the fusion order through the i-th fusion sub-network to update the fused image frame, where i is a positive integer not greater than N; If the fusion processing of each first image frame with the i-th type of exposure degree is completed, performing fusion processing on the fused image frame and the first image frames with the j-th type of exposure degree according to the fusion order through the j-th fusion sub-network to update the fused image frame, where j is a positive integer not greater than N.

12. The method according to claim 11, wherein Before the performing fusion processing on the fused image frame and the first image frames with the j-th type of exposure degree according to the fusion order through the j-th fusion sub-network to update the fused image frame, it further includes: If there are first image frames with the i-th type of exposure degree that have not been fusion-processed, performing fusion processing on the fused image frame and the un-fusion-processed first image frames with the i-th type of exposure degree according to the fusion order through the i-th fusion sub-network to update the fused image frame.

13. The method according to claim 10, wherein The image generation model further includes a noise reduction network. Before performing fusion processing on the multiple first image frames according to the fusion order, it further includes: Inputting the first first image frame into the noise reduction network; Performing noise reduction processing on the first first image frame through the noise reduction network to update the first first image frame.

14. The method according to claim 10, wherein The image generation model further includes a demosaicing network. Generating an HDR image based on the fused image frame includes: Inputting the fused image frame into the demosaicing network; Performing demosaicing processing on the fused image frame through the demosaicing network to obtain the HDR image.

15. The method according to claim 10, characterized in that, The fusion network includes N fusion sub-networks, where N is a positive integer. The image generation model further includes a noise reduction network and a demosaicing network; The method further includes: Fixing the model parameters of the fusion network unchanged and training the noise reduction network and the demosaicing network; Fixing the model parameters of the noise reduction network and the remaining fusion sub-networks other than the m-th fusion sub-network unchanged and training the m-th fusion sub-network and the demosaicing network, where m is a positive integer not greater than N; Keep the model parameters of the remaining fusion sub-networks other than the noise reduction network and the nth fusion sub-network unchanged, and train the nth fusion sub-network and the demosaicing network, where n is a positive integer not greater than N; among them, The training process of the demosaicing network includes N + 1 stages. After the training of the current stage of the demosaicing network ends, the demosaicing network inherits the model parameters obtained from the training of the current stage.

16. An apparatus for generating an HDR image, characterized in that, Including: An acquisition module, configured to acquire a plurality of first image frames; A determination module, configured to determine the fusion order of the plurality of first image frames based on the exposure degree; A fusion module, configured to perform a fusion process on the plurality of first image frames according to the fusion order to obtain a fused image frame; A generation module, configured to generate an HDR image based on the fused image frame.

17. An electronic device, characterized in that, Including: A processor; A memory for storing instructions executable by the processor; Among them, the processor is configured to: Implement the steps of the method according to any one of claims 1-15.

18. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the program instructions are executed by the processor, the steps of the method according to any one of claims 1-15 are implemented.

Images