Image processing methods, apparatus and electronic devices

Abstract

This application proposes an image processing method, apparatus, and electronic device. The method includes: after acquiring an image frame sequence for the same shooting scene, fusing the first SDR image frame and the i-th SDR image frame in the image frame sequence by calling a dual exposure fusion engine to obtain a high dynamic range (HDR) image; then, obtaining an unfused SDR image from the image frame sequence as the SDR image to be fused, and fusing the unfused SDR image frame with the HDR image by calling the dual exposure fusion engine, and using the fused image as the HDR image; repeating the step of obtaining an unfused SDR image frame from the image frame sequence until an SDR image frame is obtained from the image frame sequence. Thus, without adding hardware, the dual exposure fusion engine enables the fusion processing of multiple image frames.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to an image processing method, device and electronic equipment. BACKGROUND

[0002] Compared with a standard dynamic range (SDR) image, a high dynamic range (HDR) image can provide more dynamic range and image details. In the related art, for a scene with a lower dynamic range requirement, a dual-exposure fusion engine in an image signal processor of an electronic device fuses two SDR images with different exposure times of the same scene to obtain an HDR image to meet the requirement of the scene. However, in some scenes with a higher dynamic range requirement, more than two SDR images may need to be fused to obtain an HDR image that meets the scene with a high dynamic range requirement. If another processing device that can fuse more than two images is added to the electronic device, the cost of the electronic device is relatively high. Therefore, how to fuse more than two SDR images without increasing the cost of the electronic device is a technical problem that needs to be solved at present. SUMMARY

[0003] The present application aims to at least partially solve the technical problems in the related art.

[0004] According to a first aspect of the present application, an image processing method is provided, which is applied in a controller of an image signal processor, the image signal processor further comprising a dual-exposure fusion engine, and the method comprises: acquiring an image frame sequence for a same shooting scene, wherein the image frame sequence comprises N SDR image frames, the exposure degrees of the N image frames are different, and N is an integer greater than 2; calling the dual-exposure fusion engine to perform fusion processing on a first SDR image frame and an i-th SDR image frame in the image frame sequence to obtain an HDR image, wherein i is an integer greater than or equal to 2 and less than or equal to N; acquiring an SDR image frame that has not been fused from the image frame sequence as a to-be-fused SDR image frame; calling the dual-exposure fusion engine to perform fusion processing on the to-be-fused SDR image frame and the HDR image, and taking the image obtained through the fusion processing as the HDR image; and repeatedly performing the step of acquiring an SDR image frame that has not been fused from the image frame sequence until the SDR image frames in the image frame sequence have all been fused.

[0005] In an embodiment of the present application, the obtaining an SDR image frame that has not been fused from the sequence of image frames as the SDR image frame to be fused includes: when the initial value of i is 2, performing plus 1 processing on i; obtaining an i-th SDR image frame from the sequence of image frames, and taking the i-th SDR image frame as the SDR image frame to be fused.

[0006] In an embodiment of the present application, the image signal processor further includes a post-processing module, and the method further includes: obtaining a target HDR image obtained after performing fusion processing on an N-th SDR image frame in the sequence of image frames; and calling the post-processing module to perform post-processing on the target HDR image to obtain a red-green-blue (RGB) image.

[0007] In an embodiment of the present application, the image signal processor further includes a pre-processing module, and before the calling the dual-exposure fusion engine to perform fusion processing on the first SDR image frame and the i-th SDR image frame in the sequence of image frames to obtain a high dynamic range (HDR) image, the method further includes: calling the pre-processing module to perform pre-processing on each SDR image frame in the sequence of image frames respectively.

[0008] In an embodiment of the present application, before the calling the dual-exposure fusion engine to perform fusion processing on the first SDR image frame and the i-th SDR image frame in the sequence of image frames to obtain a high dynamic range (HDR) image, the method further includes: setting exposure of the first SDR image frame and the i-th SDR image frame in the sequence of image frames for the dual-exposure fusion engine, so that the dual-exposure fusion engine performs fusion processing on the first SDR image frame and the i-th SDR image frame based on the set exposure.

[0009] In an embodiment of the present application, before the calling the dual-exposure fusion engine to perform fusion processing on the SDR image frame to be fused and the HDR image, and taking the image obtained after the fusion processing as the HDR image, the method further includes: setting exposure of the HDR image and the SDR image frame to be fused for the dual-exposure fusion engine, so that the dual-exposure fusion engine performs fusion processing on the HDR image and the SDR image frame to be fused based on the set exposure.

[0010] The image processing method provided in the embodiments of the present application, after obtaining an image frame sequence for the same shooting scene, performs fusion processing on the first SDR image frame and the i-th SDR image frame in the image frame sequence by calling a double-exposure fusion engine to obtain a high dynamic range (HDR) image, then obtains an SDR image that has not been fused from the image frame sequence as a to-be-fused SDR image, and performs fusion processing on the to-be-fused SDR image frame and the HDR image by calling the double-exposure fusion engine, and takes the image obtained through the fusion processing as the HDR image, and repeatedly performs the step of obtaining an SDR image frame that has not been fused from the image frame sequence until the SDR image frames in the image frame sequence are fused. In this way, the double-exposure fusion engine is called multiple times to perform fusion processing on multiple image frames, so that the double-exposure fusion engine is used to perform fusion processing on multiple image frames without increasing hardware devices, and the requirement for fusion processing on more than two image frames is met.

[0011] According to a second aspect of the embodiments of the present application, an image processing device is provided, which is applied in a controller in an image signal processor, and the image signal processor further includes a double-exposure fusion engine. The device includes: an obtaining unit configured to obtain an image frame sequence for the same shooting scene, wherein the image frame sequence includes N SDR (standard dynamic range) image frames, the exposure degrees of the N image frames are different, and N is an integer greater than 2; and an algorithm logic unit configured to: call the double-exposure fusion engine to perform fusion processing on the first SDR image frame and the i-th SDR image frame in the image frame sequence to obtain a high dynamic range (HDR) image, wherein i is an integer greater than or equal to 2 and less than or equal to N; obtain an SDR image frame that has not been fused from the image frame sequence as a to-be-fused SDR image frame; call the double-exposure fusion engine to perform fusion processing on the to-be-fused SDR image frame and the HDR image, and take the image obtained through the fusion processing as the HDR image; and repeatedly perform the step of obtaining an SDR image frame that has not been fused from the image frame sequence until the SDR image frames in the image frame sequence are fused.

[0012] In an embodiment of the present application, the algorithm logic unit obtains an SDR image frame that has not been fused from the image frame sequence as a to-be-fused SDR image frame, and the specific process is as follows: when the initial value of i is 2, i is incremented by 1; the i-th SDR image frame is obtained from the image sequence, and the i-th SDR image frame is taken as the to-be-fused SDR image frame.

[0013] In an embodiment of the present application, the image signal processor further comprises a post-processing module, and the algorithm logic unit is further configured to: acquire a target HDR image obtained after performing fusion processing on an Nth SDR image frame in the image frame sequence; and call the post-processing module to perform post-processing on the target HDR image to obtain a red-green-blue (RGB) image.

[0014] In an embodiment of the present application, the image signal processor further comprises a pre-processing module, and the algorithm logic unit is further configured to: before the calling of the dual-exposure fusion engine to perform fusion processing on the first SDR image frame and the i th SDR image frame in the image frame sequence to obtain the high dynamic range (HDR) image, call the pre-processing module to perform pre-processing on each SDR image frame in the image frame sequence respectively.

[0015] In an embodiment of the present application, the algorithm logic unit is further configured to: before the calling of the dual-exposure fusion engine to perform fusion processing on the first SDR image frame and the i th SDR image frame in the image frame sequence to obtain the high dynamic range (HDR) image, set exposure of the first SDR image frame and the i th SDR image frame in the image frame sequence for the dual-exposure fusion engine, so that the dual-exposure fusion engine performs fusion processing on the first SDR image frame and the i th SDR image frame based on the set exposure.

[0016] In an embodiment of the present application, the algorithm logic unit is further configured to: after the i is incremented by 1, and before the calling of the dual-exposure fusion engine to perform fusion processing on the HDR image and an i th SDR image frame in the image frame sequence, and to take the image obtained after the fusion processing as the HDR image, set exposure of the HDR image and the i th SDR image frame in the image frame sequence for the dual-exposure fusion engine, so that the dual-exposure fusion engine performs fusion processing on the HDR image and the i th SDR image frame in the image frame sequence based on the set exposure.

[0017] The image processing apparatus provided by the embodiments of the present application, after obtaining the image frame sequence for the same shooting scene, performs fusion processing on the first SDR image frame and the i-th SDR image frame in the image frame sequence by calling the dual-exposure fusion engine to obtain a high dynamic range (HDR) image, then obtains an SDR image that has not been fused from the image frame sequence as a to-be-fused SDR image, and performs fusion processing on the to-be-fused SDR image frame and the HDR image by calling the dual-exposure fusion engine, and takes the image obtained by the fusion processing as the HDR image, and repeatedly performs the step of obtaining an SDR image frame that has not been fused from the image frame sequence until the SDR image frames in the image frame sequence. In this way, the dual-exposure fusion engine is called multiple times to perform fusion processing on multiple image frames, so that the dual-exposure fusion engine is used to perform fusion processing on multiple image frames without increasing hardware devices, and the demand for fusion processing on more than two image frames is met.

[0018] According to a third aspect of the present application, an electronic device is provided, comprising: a memory; an image signal processor, wherein the image signal processor comprises a dual-exposure fusion engine and a controller; a computer program stored on the memory and executable on the controller, and the controller implements the image processing method provided by the first aspect of the present application when executing the program.

[0019] According to a fourth aspect of the present application, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the image processing method provided by the first aspect of the present application.

[0020] According to a fifth aspect of the present application, a computer program product is provided, comprising a computer program, and the computer program is executed by a processor to implement the image processing method provided by the first aspect of the present application.

[0021] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0022] The above and / or additional aspects and advantages of the present application will become apparent and be readily appreciated from the following description, taken in conjunction with the following drawings, in which:

[0023] Figure 1 is a flowchart of an image processing method according to an exemplary embodiment;

[0024] Figure 2 is a flowchart of an image processing method according to another exemplary embodiment;

[0025] Figure 3is an example diagram of a structural relationship between internal components in an image signal processor;

[0026] Figure 4 is a flow diagram of an image processing method according to another example embodiment;

[0027] Figure 5 is a flow diagram of an image processing method according to another example embodiment;

[0028] Figure 6 is a structural diagram of an image processing apparatus according to an example embodiment;

[0029] Figure 7 is a block diagram of an electronic device for implementing an image processing method according to an example embodiment. DETAILED DESCRIPTION

[0030] Embodiments of the present application are described in detail below with reference to the accompanying drawings, in which the same or like reference numerals in different drawings denote the same or like elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to explain the present application, and are not to be understood as limiting the present application.

[0031] The image processing method, apparatus and electronic device provided by the present application are described in detail below with reference to the accompanying drawings.

[0032] Figure 1 is a flow diagram of an image processing method according to an example embodiment. It should be noted that the execution subject of the image processing method is an image processing apparatus, which can be a controller of an image signal processor, or can also be configured in the controller of the image signal processor, and the image signal processor can also include a dual-exposure fusion engine. The image signal processor in the present example can be provided in an electronic device. In some examples, the electronic device can also be provided in a vehicle and can process images collected by an on-board camera in the vehicle to obtain an HDR image. For example, the electronic device can be a car device in the vehicle, and the electronic device is not limited in the embodiment.

[0033] As shown in Figure 1 , the image processing method can include the following steps:

[0034] Step 101, acquiring an image frame sequence for the same shooting scene, wherein the image frame sequence includes N standard dynamic range (SDR) image frames, the exposure degrees of the N image frames are different, and N is an integer greater than 2.

[0035] In the example embodiment, the N SDR image frames with different exposure times can be obtained by an image acquisition device on an electronic device for generating high dynamic range images, for example, the image signals are first obtained by a CCD, CMOS or other light-sensitive element arranged in a camera on a mobile phone, and then the image signals are transmitted to an image signal processor (ISP) in the mobile phone, and a controller in the image signal processor generates the image frame sequence for the same shooting scene.

[0036] The image frames in the image frame sequence can be sorted in the order of generation time of the image frames.

[0037] Step 102, calling a double-exposure fusion engine to perform fusion processing on the first SDR image frame and the i-th SDR image frame in the image frame sequence to obtain a high dynamic range (HDR) image, where i is an integer greater than or equal to 2 and less than or equal to N.

[0038] The double-exposure fusion engine refers to a semiconductor integrated circuit that can fuse two input SDR images into one HDR image.

[0039] Step 103, obtaining an SDR image frame that has not been fused from the image frame sequence as a to-be-fused SDR image frame.

[0040] Step 104, calling the double-exposure fusion engine to perform fusion processing on the to-be-fused SDR image frame and the HDR image, and taking the image obtained by the fusion processing as the HDR image.

[0041] Step 105, determining whether there is an SDR image frame that has not been fused in the image frame sequence, if yes, jumping to step 103, if no, ending.

[0042] That is, in the example, the step of obtaining an SDR image frame that has not been fused from the image frame sequence is repeatedly executed until all the SDR image frames in the image frame sequence have been fused.

[0043] In the present example, the first SDR image and the jth SDR image in the image frame sequence are first fused by the dual-exposure fusion engine to obtain an HDR image, and then the fused HDR image and an SDR image in the image frame sequence that has not been fused are fused by the dual-exposure fusion engine, and the fused image is taken as the HDR image, and the above process is repeatedly continued until all SDR images in the image frame sequence are fused. As can be seen, the dual-exposure fusion engine is called multiple times in the present application, so that the dual-exposure fusion engine is used to fuse multiple image frames without increasing hardware devices, thereby meeting the requirement of fusing more than two image frames.

[0044] The image processing method provided in the embodiments of the present application obtains an image frame sequence for the same shooting scene, fuses the first SDR image frame and the ith SDR image frame in the image frame sequence by calling the dual-exposure fusion engine to obtain an HDR image, then obtains an SDR image that has not been fused from the image frame sequence as a to-be-fused SDR image, fuses the to-be-fused SDR image frame and the HDR image by calling the dual-exposure fusion engine, takes the fused image as the HDR image, and repeatedly executes the step of obtaining an SDR image frame that has not been fused from the image frame sequence until the SDR image frames in the image frame sequence are fused. Thus, the dual-exposure fusion engine is called multiple times to fuse multiple image frames, so that the dual-exposure fusion engine is used to fuse multiple image frames without increasing hardware devices, thereby meeting the requirement of fusing more than two image frames.

[0045] Figure 2 is a flowchart of an image processing method according to another exemplary embodiment. It should be noted that the process of fusing SDR images in the image frame sequence by calling the dual-exposure fusion engine multiple times is exemplarily described in the present example with the initial value of i being 2 as an example.

[0046] As Figure 2 shown, the image processing method can include the following steps:

[0047] In step 201, an image frame sequence for the same shooting scene is obtained, wherein the image frame sequence includes N SDR image frames, the exposure degrees of the N image frames are different, and N is an integer greater than 2.

[0048] In the example embodiment, the N SDR image frames with different exposure times can be obtained by an image acquisition device on an electronic device for generating a high dynamic range image, for example, the image signals are first obtained by a CCD, CMOS or other photosensitive element arranged in a camera on a mobile phone, and then the image signals are transmitted to an image signal processor (ISP) in the mobile phone, and the controller in the image signal processor generates the image frame sequence for the same shooting scene.

[0049] The image frames in the image frame sequence can be sorted according to the generation time of the image frames.

[0050] In step 202, a double-exposure fusion engine is called to fuse the first SDR image frame and the i-th SDR image frame in the image frame sequence to obtain a high dynamic range (HDR) image, where the initial value of i is 2.

[0051] The double-exposure fusion engine refers to a semiconductor integrated circuit that can fuse two input SDR images into one HDR image.

[0052] In step 203, i is incremented by 1.

[0053] In step 204, the double-exposure fusion engine is called to fuse the HDR image and the i-th SDR image frame in the image frame sequence, and the image obtained by the fusion processing is used as the HDR image.

[0054] In step 205, it is determined whether all SDR image frames in the image frame sequence have been fused, and if not, the step of incrementing i by 1 is performed.

[0055] That is, in this example, if i is less than or equal to N, the step of incrementing i by 1 is performed until all SDR image frames in the image frame sequence have been fused.

[0056] In this example, the double-exposure fusion engine is first used to fuse the first two SDR image frames in the image frame sequence, then the double-exposure fusion engine is used to fuse the HDR image obtained by the fusion and the third SDR image frame in the image frame sequence, and the double-exposure fusion engine is used to fuse the fused HDR image and the fourth SDR image frame in the image frame sequence, and the above process is repeated until the last SDR image frame in the image frame sequence is also fused. As can be seen, the present application calls the double-exposure fusion engine multiple times, thereby achieving fusion processing of multiple image frames by the double-exposure fusion engine without increasing hardware devices, and meeting the demand for fusion processing of more than two image frames.

[0057] The image processing method provided in the embodiments of the present application, after obtaining the image frame sequence for the same shooting scene, performs fusion processing on the first SDR image frame and the i-th SDR image frame in the image frame sequence by calling the dual-exposure fusion engine to obtain a high dynamic range (HDR) image, where the initial value of i is 2, then i is incremented by 1, and the dual-exposure fusion engine is called to perform fusion processing on the HDR image and the i-th SDR image frame in the image frame sequence, and the image obtained by the fusion processing is taken as the HDR image, and the step of incrementing i by 1 is continued until all the SDR image frames in the image frame sequence are fused. Thus, the dual-exposure fusion engine is called multiple times to perform fusion processing on two input images, so that the dual-exposure fusion engine is used to perform fusion processing on multiple image frames without increasing hardware devices, and the demand for fusion processing on more than two image frames is met.

[0058] The image processing method provided in the embodiments of the present application will be further described below in combination with Figure 3 and Figure 4 It should be noted that Figure 3 The internal component structure relationship diagram of the image signal processor is shown in the above Figure 3 It can be seen that the image signal processor in this example can include a pre-processing module 301, a dual-exposure fusion engine 302, a post-processing module 303, a storage module 304, and a controller 305. It should be noted that the image processing method in this example is exemplarily described by taking the controller 305 as an example.

[0059] Figure 4 is a flowchart of an image processing method according to another exemplary embodiment.

[0060] As shown in Figure 4 The image processing method includes the following steps:

[0061] In step 401, an image frame sequence for the same shooting scene is obtained, where the image frame sequence includes N SDR image frames, the exposure degrees of the N image frames are different, and N is an integer greater than 2.

[0062] It should be noted that the specific description of step 401 can be referred to the related description of the embodiments of the present application, which will not be repeated here.

[0063] In step 402, the pre-processing module is called to perform pre-processing on each SDR image frame in the image frame sequence.

[0064] In some example embodiments, in order to improve the effect of the subsequently fused HDR image, the pre-processing module can be invoked to perform pre-processing on each SDR image frame in the image frame sequence respectively to obtain a processed SDR image frame, and store the processed SDR image frame into the storage module 304.

[0065] In the actual application, the pre-processing can be set according to actual needs, for example, the pre-processing can include but is not limited to black level correction, bad point removal and the like. That is, in some examples, for each SDR image frame in the image sequence, the pre-processing module can perform black level correction, bad point removal and the like on the SDR image frame.

[0066] Step 403, invoke the dual-exposure fusion engine to perform fusion processing on the first SDR image frame and the i-th SDR image frame in the image frame sequence to obtain a high dynamic range (HDR) image, where the initial value of i is 2.

[0067] In an embodiment of the present application, in order to enable the dual-exposure fusion engine to accurately fuse the input two images and improve the effect of the fused HDR image, the exposure of the first SDR image frame and the i-th SDR image frame in the image frame sequence can also be set for the dual-exposure fusion engine, so that the dual-exposure fusion engine performs fusion processing on the first SDR image frame and the i-th SDR image frame based on the set exposure.

[0068] Step 404, increment i by 1.

[0069] Step 405, invoke the dual-exposure fusion engine to perform fusion processing on the HDR image and the i-th SDR image frame in the image frame sequence, and take the image obtained by the fusion processing as the HDR image.

[0070] In an embodiment of the present application, in order to enable the dual-exposure fusion engine to accurately fuse the input two images, the exposure of the HDR image and the i-th SDR image frame in the image frame sequence can also be set for the dual-exposure fusion engine, so that the dual-exposure fusion engine performs fusion processing on the HDR image and the i-th SDR image frame in the image frame sequence based on the set exposure.

[0071] The exposure of the HDR image can be obtained by exposure analysis on the HDR image. As an example, the HDR image can be input into a pre-trained exposure analysis model to obtain the exposure of the HDR image through the exposure analysis model. As another example, the exposure of the HDR image can be determined according to the exposure of the SDR image corresponding to the HDR image. For example, assuming that the HDR image is obtained by fusion processing on the jth image frame in the image sequence, the exposures of the first j image frames in the image sequence can be summed, and the summed exposure is taken as the exposure of the HDR image, where j is greater than or equal to 2 and less than N-1.

[0072] In step 406, it is determined whether all SDR image frames in the image frame sequence have been fused. If not, the step of increasing i by 1 is performed. If yes, steps 407 and 408 are performed.

[0073] That is, in the present example, when i is less than or equal to N, the step of increasing i by 1 is performed until all SDR image frames in the image frame sequence have been fused.

[0074] In step 407, a target HDR image obtained by fusion processing on the Nth SDR image frame in the image frame sequence is obtained.

[0075] In step 408, a post-processing module is called to post-process the target HDR image to obtain an RGB image.

[0076] In some example embodiments, the post-processing module in the present example can generally convert the HDR image into an RGB image conforming to the viewing habits of the human eye.

[0077] In some example embodiments, the post-processing module can also perform color correction and Gamma curve correction on the converted RGB image to improve the effect of the obtained RGB image.

[0078] It can be understood that, in some example embodiments, the converted RGB image can also be saved to the storage module 304 for subsequent viewing of the RGB image.

[0079] The storage module 304 in the present example can be a dynamic random access memory (DRAM) or a static random access memory (SRAM) or the like, and functions to store the processed image.

[0080] In the present example, after obtaining the image sequence for the same scene, each SDR image frame in the image sequence is pre-processed by the pre-processing module, and then the processed image sequence is fused by multiple invocations of the double-exposure fusion engine, and the target HDR image obtained after the fusion processing of the last SDR image frame in the image sequence is obtained, and the post-processing module is invoked to convert the target HDR image into an RGB image. Thus, through the cooperation of each internal component in the image processor, the fusion processing of multiple image frames can be accurately realized without increasing the hardware cost, and a good-quality RGB image can be obtained.

[0081] In order to clearly understand the present application, the following will be described in combination with Figure 3 and Figure 5 the image processing method of the embodiment is exemplarily illustrated.

[0082] Figure 5 is a flowchart of an image processing method according to another exemplary embodiment. It should be noted that the execution subject of the image processing method in the present example is the controller 305.

[0083] As Figure 5 indicated, the image processing method can include:

[0084] Step 501, invoking the pre-processing module 301 to process the input SDR image to obtain a processed SDR image.

[0085] Step 502, storing the SDR image obtained by the pre-processing module 301 in the storage module 304.

[0086] Step 503, confirming whether the pre-processing of all SDR images has been completed, if yes, entering step 504, otherwise returning to step 501 to pre-process the next frame of SDR image, repeating steps 501-503 until all SDR images have been pre-processed.

[0087] Step 504, controlling the double-exposure fusion engine 302 to read the first frame of SDR image and the second frame of SDR image from the storage module 304.

[0088] That is, the double-exposure fusion engine 302 is controlled to read the first SDR image frame and the second SDR image frame in the image sequence from the storage module 304.

[0089] Step 505, parameter configuration is performed on the double-exposure fusion engine 302, and important parameters include the exposure of the two frames.

[0090] For example, the exposure of the first frame SDR image is 1.0, and the exposure of the subsequent N-1 frames SDR images are E1, E2, E3, E4,..., EN-1 respectively. N-1 .

[0091] Step 506, calling the dual-exposure fusion engine 302 to fuse the two frames of input images to generate an HDR image.

[0092] As an exemplary embodiment, the dual-exposure fusion engine 302 generally fuses according to the exposure ratio configured in step 505. For example, assuming that the N frames of SDR images are IMG0, IMG1,..., IMGN respectively, and the exposure of the first frame IMG0 is 1.0, and the exposure of the second frame IMG1 is E1, then the dual-exposure fusion engine 302 first fuses the first frame and the second frame according to the following formula N-1 .

[0093]

[0094] Step 507, controlling the dual-exposure fusion engine 302 to save the synthesized HDR image to the storage module 304.

[0095] Step 508, confirming whether the fusion processing of all SDR images has been completed, if yes, entering step 510, otherwise executing step 509 and returning to step 505 for the next HDR fusion.

[0096] Step 509, controlling the dual-exposure fusion engine 302 to read the HDR image saved in step 507 from the storage module 304, and reading the third frame SDR image from the storage module 304 for the next HDR fusion.

[0097] In some exemplary embodiments, the third frame SDR image is fused according to the following formula, and the fused HDR image is updated:

[0098]

[0099] It can be understood that each subsequent frame (Mth frame SDR image) is fused according to the following formula:

[0100]

[0101] It should be noted that steps 505-509 can be repeated until all SDR images are fused.

[0102] Step 510, calling the post-processing module 303 to obtain the target HDR image obtained by fusing the last frame SDR image from the storage module 304.

[0103] At step 511, the post-processing module 303 is controlled to post-process the target HDR image to obtain an RGB image in which the target HDR image conforms to the habit of human eye observation.

[0104] At step 512, the post-processing module 303 is controlled to store the obtained RGB image to the storage module 304.

[0105] Based on the above description, it can be seen that the multiple SDR image frames that need to be fused can be fused by multiple invocations of the dual-exposure fusion engine without increasing the hardware cost, and the original dual-exposure fusion engine is extended.

[0106] Figure 6 FIG. 1 is a structural schematic diagram of an image processing apparatus according to an example embodiment. It should be noted that the apparatus in this example is applied to a controller in an image signal processor, and the image signal processor further includes a dual-exposure fusion engine.

[0107] As shown in FIG. 6, the image processing apparatus 60 includes an acquisition unit 601 and an algorithm logic unit 602, where: Figure 6 The acquisition unit 601 is configured to acquire an image frame sequence for a same photographed scene, where the image frame sequence includes N standard dynamic range (SDR) image frames, the exposure degrees corresponding to the N image frames are different, and N is an integer greater than 2.

[0108] The algorithm logic unit 601 is configured to: invoke the dual-exposure fusion engine to perform fusion processing on a first SDR image frame and an i-th SDR image frame in the image frame sequence to obtain a high dynamic range (HDR) image, where i is an integer greater than or equal to 2 and less than or equal to N; acquire an SDR image frame that has not been subjected to fusion processing from the image frame sequence as a to-be-fused SDR image frame; invoke the dual-exposure fusion engine to perform fusion processing on the to-be-fused SDR image frame and the HDR image, and take the image obtained through the fusion processing as an HDR image; and repeatedly perform the step of acquiring an SDR image frame that has not been subjected to fusion processing from the image frame sequence until the SDR image frames in the image frame sequence have all been subjected to fusion processing.

[0109] In an embodiment of the present application, the algorithm logic unit 602 acquires an SDR image frame that has not been subjected to fusion processing from the image frame sequence as the to-be-fused SDR image frame, and the specific process is as follows: when the initial value of i is 2, i is incremented by 1; the i-th SDR image frame is acquired from the image sequence, and the i-th SDR image frame is taken as the to-be-fused SDR image frame.

[0110]

[0111] ​In an embodiment of the present application, the image signal processor further comprises a post-processing module, and the algorithm logic unit 602 is further configured to: acquire a target HDR image obtained after fusion processing of the Nth SDR image frame in the image frame sequence; and call the post-processing module to post-process the target HDR image to obtain a red-green-blue (RGB) image.

[0112] In an embodiment of the present application, the image signal processor further comprises a pre-processing module, and the algorithm logic unit 602 is further configured to: call the pre-processing module to pre-process each SDR image frame in the image frame sequence before calling the dual-exposure fusion engine to perform fusion processing on the 1st SDR image frame and the ith SDR image frame in the image frame sequence to obtain the high dynamic range (HDR) image.

[0113] In an embodiment of the present application, the algorithm logic unit 602 is further configured to: before calling the dual-exposure fusion engine to perform fusion processing on the 1st SDR image frame and the ith SDR image frame in the image frame sequence to obtain the high dynamic range (HDR) image, set the exposure of the 1st SDR image frame and the ith SDR image frame in the image frame sequence for the dual-exposure fusion engine, so that the dual-exposure fusion engine performs fusion processing on the 1st SDR image frame and the ith SDR image frame based on the set exposure.

[0114] In an embodiment of the present application, the algorithm logic unit 602 is further configured to: after the i is incremented by 1, and before calling the dual-exposure fusion engine to perform fusion processing on the HDR image and the ith SDR image frame in the image frame sequence and taking the image obtained after the fusion processing as the HDR image, set the exposure of the HDR image and the ith SDR image frame in the image frame sequence for the dual-exposure fusion engine, so that the dual-exposure fusion engine performs fusion processing on the HDR image and the ith SDR image frame in the image frame sequence based on the set exposure.

[0115] It should be noted that the above description of the image processing method is also applicable to the image processing device in this embodiment, and the present embodiment will not be repeated here.

[0116] The image processing apparatus of the embodiment of the present application, after obtaining the image frame sequence for the same shooting scene, performs fusion processing on the 1st SDR image frame and the i-th SDR image frame in the image frame sequence by calling the dual-exposure fusion engine to obtain a high dynamic range (HDR) image, then obtains an SDR image that has not been fused from the image frame sequence as a to-be-fused SDR image, and performs fusion processing on the to-be-fused SDR image frame and the HDR image by calling the dual-exposure fusion engine, and takes the image obtained by the fusion processing as the HDR image, and repeatedly performs the step of obtaining an SDR image frame that has not been fused from the image frame sequence until the SDR image frames in the image frame sequence. Thus, the dual-exposure fusion engine is called multiple times to perform fusion processing on multiple image frames, so that the dual-exposure fusion engine is used to perform fusion processing on multiple image frames without increasing hardware devices, and the demand for fusion processing on more than two image frames is met.

[0117] To implement the above embodiment, the present application further provides an electronic device, as shown in Figure 7 Figure 7 is a block diagram of an electronic device for implementing the image processing method according to an exemplary embodiment. As shown in Figure 7

[0118] The memory 710 and the image signal processor 720 are connected by the bus 730 connecting different components (including the memory 710 and the image signal processor 720), wherein the image signal processor 720 includes a dual-exposure fusion engine 721 and a controller 722, the memory 710 stores a computer program, and the controller 722 implements the image processing method of the embodiment of the present application when executing the program.

[0119] The bus 730 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of a variety of bus structures. For example, these architectures include but are not limited to industry standard architecture (ISA) bus, micro channel architecture (MAC) bus, enhanced ISA bus, video electronics standards association (VESA) local bus, and peripheral component interconnect (PCI) bus.

[0120] The electronic device 700 typically includes a variety of computer readable media. These media can be any available media that can be accessed by the electronic device 700 and includes both volatile and nonvolatile media, removable and non-removable media.

[0121] ​​Memory 710 can also include other removable / non-removable, volatile / nonvolatile computer system storage media. By way of example only, storage system 760 can be provided for reading from and writing to a non-removable, nonvolatile magnetic media (not shown and typically called a "hard drive"). Figure 7 Although not specifically shown, one or more mass storage devices 760 can be used to read from and write to a non-removable, nonvolatile magnetic media (e.g., a "hard drive"). Figure 7 Although not specifically shown, one or more mass storage devices 760 can be used to read from and write to a non-removable, nonvolatile magnetic media (e.g., a "hard drive").

[0122] Program / utility 780, having a set (at least one) of program modules 770, can be stored in memory 710 by way of example, and not limitation, as well as an operating system, one or more application programs, other program modules, and program data, each or some combination thereof, can include implementation of a networking environment. Program modules 770 generally carry out the functions and / or methodologies of embodiments of the application as described herein.

[0123] Electronic device 700 can also communicate with one or more external devices 790 such as a keyboard or pointing device, a display 791, etc.; one or more devices that enable a user to interact with electronic device 700; and / or one or more devices that enable electronic device 700 to communicate with one or more other computing devices. Such communication can be via input / output (I / O) interfaces 792. Similarly, communication can be enabled via a network adapter 793 that can be used to connect electronic device 700 to an intranet, LAN, WAN, or the Internet, for example. Figure 7 As shown, network adapter 793 communicates with the other components of electronic device 700 via bus 730. It should be understood that although not specifically shown: Other hardware and / or software modules are contemplated. For example, a mass storage device such as a disk drive, RAID array, etc., can be used with electronic device 700. Figure 7

[0124] ​The image signal processor 720 executes various function applications and data processing by running programs stored in the memory 710.

[0125] It should be noted that, for the image signal processor 720 in the present example, only the dual-exposure fusion engine and the controller in the image signal processor 720 are exemplified in the present example, and other internal components in the image signal processor 720 can be referred to the image processing method of the embodiments of the present application. Figure 2 The embodiments do not limit the same.

[0126] It should be noted that the implementation process and technical principles of the electronic device of the present embodiment can be referred to the foregoing explanation and description of the image processing method of the embodiments of the present application, which will not be repeated here.

[0127] The electronic device provided by the embodiments of the present application, after obtaining the image frame sequence for the same shooting scene, performs fusion processing on the first SDR image frame and the i-th SDR image frame in the image frame sequence by calling the dual-exposure fusion engine to obtain a high dynamic range HDR image, then obtains an SDR image that has not been fused from the image frame sequence as a to-be-fused SDR image, and performs fusion processing on the to-be-fused SDR image frame and the HDR image by calling the dual-exposure fusion engine, and takes the image obtained by the fusion processing as the HDR image, and repeatedly performs the step of obtaining an SDR image frame that has not been fused from the image frame sequence until the SDR image frame in the image frame sequence. Thus, by calling the dual-exposure fusion engine to perform fusion processing on multiple image frames multiple times, the dual-exposure fusion engine is used to perform fusion processing on multiple image frames without increasing hardware devices, and the demand for fusion processing on more than two image frames is met.

[0128] In order to realize the above-mentioned embodiments, the embodiments of the present application further provide a computer readable storage medium.

[0129] When the instructions in the computer readable storage medium are executed by the processor of the electronic device, the electronic device can execute the image processing method as described above.

[0130] In order to realize the above-mentioned embodiments, the present application further provides a computer program product, which is executed by the processor of the electronic device, so that the electronic device can execute the image processing method as described above.

[0131] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. It is intended that the specification and examples be considered as exemplary only, with the true scope and spirit of the application being indicated by the following claims.

[0132] It is to be understood that the application is not limited to the precise construction herein disclosed and shown in the drawings, and that various changes in shape, size and arrangements of parts can be made without departing from the scope of the application. The scope of the application is limited only by the claims that follow.

Claims

1. An image processing method, characterized by, The method is applied in a controller of an image signal processor, the image signal processor further comprising a dual-exposure fusion engine, and the method comprises: obtaining an image frame sequence for a same shooting scene, wherein the image frame sequence comprises N standard dynamic range (SDR) image frames, the exposure degrees of the N image frames are different, and N is an integer greater than 2; calling the dual-exposure fusion engine to perform fusion processing on a first SDR image frame and an i-th SDR image frame in the image frame sequence to obtain a high dynamic range (HDR) image, wherein i is an integer greater than or equal to 2 and less than or equal to N; obtaining an SDR image frame that has not been fused from the image frame sequence as a to-be-fused SDR image frame; calling the dual-exposure fusion engine to perform fusion processing on the to-be-fused SDR image frame and the HDR image, and taking the image obtained through the fusion processing as the HDR image; repeating the step of obtaining an SDR image frame that has not been fused from the image frame sequence until all the SDR image frames in the image frame sequence have been fused.

2. The method of claim 1, wherein, The step of obtaining an SDR image frame that has not been fused from the image frame sequence as a to-be-fused SDR image frame comprises: when the initial value of i is 2, performing plus 1 processing on i; obtaining an i-th SDR image frame from the image sequence and taking the i-th SDR image frame as the to-be-fused SDR image frame.

3. The method of claim 1, wherein, The image signal processor further comprises a post-processing module, and the method further comprises: obtaining a target HDR image obtained through fusion processing on an N-th SDR image frame in the image frame sequence; calling the post-processing module to perform post-processing on the target HDR image to obtain a red-green-blue (RGB) image.

4. The method of claim 1, wherein, The image signal processor further comprises a pre-processing module, and before the step of calling the dual-exposure fusion engine to perform fusion processing on a first SDR image frame and an i-th SDR image frame in the image frame sequence to obtain a high dynamic range (HDR) image, the method further comprises: calling the pre-processing module to perform pre-processing on each SDR image frame in the image frame sequence respectively.

5. The method of claim 1, wherein, Before the step of calling the dual-exposure fusion engine to perform fusion processing on a first SDR image frame and an i-th SDR image frame in the image frame sequence to obtain a high dynamic range (HDR) image, the method further comprises: setting the exposure degrees of the first SDR image frame and the i-th SDR image frame in the image frame sequence for the dual-exposure fusion engine, so that the dual-exposure fusion engine performs fusion processing on the first SDR image frame and the i-th SDR image frame based on the set exposure degrees.

6. The method of claim 2, wherein, Before the step of calling the dual-exposure fusion engine to perform fusion processing on the to-be-fused SDR image frame and the HDR image, and taking the image obtained through the fusion processing as the HDR image, the method further comprises: set exposure of the HDR image and the to-be-fused SDR image frame for the dual-exposure fusion engine, so that the dual-exposure fusion engine performs fusion processing on the HDR image and the to-be-fused SDR image frame based on the set exposure.

7. An image processing apparatus characterized by comprising: The device is applied to a controller in an image signal processor, and the image signal processor further comprises a dual-exposure fusion engine, and the device comprises: an acquisition unit, configured to acquire an image frame sequence for a same shooting scene, wherein the image frame sequence comprises N standard dynamic range (SDR) image frames, exposure of the N image frames is different, and N is an integer greater than 2; an algorithm logic unit, configured to: invoke the dual-exposure fusion engine to perform fusion processing on the first SDR image frame and the i-th SDR image frame in the image frame sequence to obtain a high dynamic range (HDR) image, wherein i is an integer greater than or equal to 2 and less than or equal to N; acquire an SDR image frame that has not been fused from the image frame sequence as a to-be-fused SDR image frame; invoke the dual-exposure fusion engine to perform fusion processing on the to-be-fused SDR image frame and the HDR image, and take an image obtained through the fusion processing as the HDR image; repeat the step of acquiring an SDR image frame that has not been fused from the image frame sequence until all SDR image frames in the image frame sequence have been fused.

8. The apparatus of claim 7, wherein, The algorithm logic unit acquires an SDR image frame that has not been fused from the image frame sequence as a to-be-fused SDR image frame, and the specific process is as follows: when the initial value of i is 2, perform i+1 processing on i; acquire the i-th SDR image frame from the image sequence and take the i-th SDR image frame as the to-be-fused SDR image frame.

9. The apparatus of claim 7, wherein, The image signal processor further comprises a post-processing module, and the algorithm logic unit is further configured to: acquire a target HDR image obtained through fusion processing on the N-th SDR image frame in the image frame sequence; invoke the post-processing module to perform post-processing on the target HDR image to obtain a red-green-blue (RGB) image.

10. The apparatus of claim 7, wherein, The image signal processor further comprises a pre-processing module, and the algorithm logic unit is further configured to: before the step of invoking the dual-exposure fusion engine to perform fusion processing on the first SDR image frame and the i-th SDR image frame in the image frame sequence to obtain the HDR image, invoke the pre-processing module to perform pre-processing on each SDR image frame in the image frame sequence respectively.

11. The apparatus of claim 7, wherein, The algorithm logic unit is further configured to: set, for the dual-exposure fusion engine, exposure of the first SDR image frame and the i-th SDR image frame in the image frame sequence, so that the dual-exposure fusion engine performs fusion processing on the first SDR image frame and the i-th SDR image frame based on the set exposure, before the calling of the dual-exposure fusion engine to perform fusion processing on the first SDR image frame and the i-th SDR image frame in the image frame sequence to obtain the high dynamic range HDR image.

12. The apparatus of claim 7, wherein, The algorithm logic unit is further configured to: set, for the dual-exposure fusion engine, exposure of the HDR image and the SDR image frame to be fused, so that the dual-exposure fusion engine performs fusion processing on the HDR image and the SDR image frame to be fused based on the set exposure.

13. An electronic device, comprising: The apparatus comprises: a memory; an image signal processor, wherein the image signal processor comprises a dual-exposure fusion engine and a controller; a computer program stored on the memory and executable on the controller, the controller implementing the image processing method of any one of claims 1-6 when executing the program.

14. A computer readable storage medium having stored thereon a computer program, characterized in that, The program is executed by a processor to implement the image processing method of any one of claims 1-6.

Images