Image processing method, device and equipment and readable storage medium

Abstract

The invention discloses an image processing method, device and equipment and a readable storage medium, and belongs to the technical field of video processing. The method is executed by a terminal and comprises the following steps: acquiring multiple frames of images continuously acquired by an image sensor; processing the multiple frames of images based on at least one fusion strategy to obtain target images, wherein the target images are a group of images used for high dynamic range image processing; wherein the fusion strategy is used for indicating at least one of the following items: a reference image; fusing the auxiliary images; fusing the number of images contained in the target; and outputting the frame rate.

Description

Technical Field

[0001] The present application belongs to the field of video processing technology, and specifically relates to an image processing method, device, equipment and readable storage medium. Background Art

[0002] With the development of terminal devices, more users use mobile terminal devices to take photos and prefer to watch photos and preview videos on mobile terminal devices. In real scenes, the color of objects has a very high dynamic range. Most terminal devices can support 24 bits of information to represent colors, but the pixel value of a single channel is 8 bits of data. Therefore, the dynamic range that can be covered by a single photo taken by an ordinary terminal device is extremely limited. Using too short an exposure time will make the low-brightness parts of the scene dark and produce noise, while using too long an exposure time will cause the high-brightness parts of the scene to be overexposed and lose details. In order to make the image present rich color details and brightness levels, High Dynamic Range (HDR) technology that combines alternate exposure of multiple frames of pictures has become a mainstream solution.

[0003] Among them, the traditional HDR video synthesis technology fuses the alternating images with different exposure levels and outputs the HDR video at a fixed frame rate. In this way, the video output frame rate will be reduced due to multi-frame fusion, and the front and back HDR video frames are prone to jitter and strobe. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image processing method, apparatus, device and readable storage medium to achieve the purpose of improving the HDR video frame rate and meeting the HDR image quality requirements.

[0005] In a first aspect, an embodiment of the present application provides an image processing method, which is executed by a terminal and includes:

[0006] Acquire multiple frames of images continuously collected by an image sensor;

[0007] The multiple frames of images are processed based on at least one fusion strategy to obtain a target image, where the target image is a group of images used for high dynamic range image processing; wherein the fusion strategy is used to indicate at least one of the following:

[0008] Baseline image;

[0009] Fusion of auxiliary images;

[0010] The number of images contained in the fusion target;

[0011] Output frame rate.

[0012] In a second aspect, an embodiment of the present application provides an image processing device, including:

[0013] A first processing module is used to obtain multiple frames of images continuously collected by an image sensor;

[0014] The second processing module is used to process the multiple frames of images based on at least one fusion strategy to obtain a target image, where the target image is a group of images used for high dynamic range image processing; wherein the fusion strategy is used to indicate at least one of the following:

[0015] Baseline image;

[0016] Fusion of auxiliary images;

[0017] The number of images contained in the fusion target;

[0018] Output frame rate.

[0019] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0020] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0021] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0022] In a sixth aspect, a computer program / program product is provided, wherein the computer program / program product is stored in a storage medium and is executed by at least one processor to implement the steps of the method described in the first aspect.

[0023] In an embodiment of the present application, the terminal can adopt at least one fusion strategy to determine a group of images for high dynamic range image processing by acquiring multiple frames of images continuously collected by the image sensor, thereby providing stable and effective front-end long and short frame inputs for subsequent processing, achieving the best balance in power consumption, frame rate, and ghost elimination, and achieving the purpose of improving the HDR video frame rate and meeting HDR image quality requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flowchart of an image processing method according to an embodiment of the present application;

[0025] Figure 2It is a schematic diagram of the arrangement order of long exposure images and short exposure images;

[0026] Figure 3 It is a schematic diagram of motion state judgment;

[0027] Figure 4 This is a schematic diagram of the 2-in-1 half-frame rate output strategy;

[0028] Figure 5 This is a schematic diagram of the 2-in-1 equal frame rate output strategy;

[0029] Figure 6 This is a schematic diagram of the 3-in-1 frame rate reduction output strategy;

[0030] Figure 7 This is a schematic diagram of the 3-in-1 equal frame rate output strategy;

[0031] Figure 8 It is a schematic diagram of the long and short frame exposure strategy;

[0032] Fig. 9 It is a linear schematic diagram of the acquisition gap-motion index;

[0033] Fig.10 It is a schematic diagram of target strategy, collection interval, and ISO determination for 9 scenarios;

[0034] Fig.11 It is a schematic diagram of the terminal hardware structure;

[0035] Fig.12 This is a schematic diagram of the HDR-MV-LITE module structure;

[0036] Fig.13 It is a schematic diagram of the terminal interface;

[0037] Fig.14 This is a schematic diagram of the HDR video processing flow;

[0038] Fig.15 This is a schematic diagram of the offline HDR video processing process;

[0039] Fig.16 is a schematic diagram of the module structure of the image processing device according to an embodiment of the present application;

[0040] Fig.17 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application;

[0041] Fig.18 It is a schematic diagram of the structure of an electronic device according to another embodiment of the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0043] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, but not to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited, for example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, for example, the protection scope of "A or B" covers at least three schemes, that is, scheme one: including A and excluding B; scheme two: including B and excluding A; scheme three: including both A and B. In addition, the terms "A and / or B", "at least one of A and B", and "at least one of A or B" also cover at least the above three schemes respectively. The character " / " generally indicates that the objects associated before and after are in an "or" relationship.

[0044] The term "indication" in this application can be a direct indication (or explicit indication) or an indirect indication (or implicit indication). A direct indication can be understood as the sender explicitly informing the receiver of specific information, operations to be performed, or request results in the sent indication; an indirect indication can be understood as the receiver determining the corresponding information according to the indication sent by the sender, or making a judgment and determining the operation to be performed or the request result according to the judgment result.

[0045] The image processing method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0046] like Figure 1 As shown, an image processing method in an embodiment of the present application is executed by a terminal, including:

[0047] Step 101: Acquire multiple frames of images continuously captured by an image sensor.

[0048] Here, the multiple frames of images continuously captured by the image sensor may be alternating long exposure images and short exposure images. For example, when a user turns on the HDR video mode on the terminal, the image sensor alternately captures long and short exposure images in real time, such as Figure 2As shown in FIG. 1 , L1 to LN / 2 represent long exposure images, and S1 to SN / 2 represent short exposure images.

[0049] Step 102: Process the multiple frames of images based on at least one fusion strategy to obtain a target image, where the target image is a group of images used for high dynamic range image processing; wherein the fusion strategy is used to indicate at least one of the following:

[0050] Baseline image;

[0051] Fusion of auxiliary images;

[0052] The number of images contained in the fusion target;

[0053] Output frame rate.

[0054] Here, the fusion strategy is used to indicate at least one of the reference image, the fusion auxiliary image, the number of images included in the target image, and the output frame rate. According to step 102, the multiple frames of images obtained in step 101 can be processed using at least one content indicated by the fusion strategy to obtain a group of images for high dynamic range image processing, that is, the target image.

[0055] In this way, the terminal executes the method of the embodiment of the present application, and according to the above steps 101 and 102, by acquiring multiple frames of images continuously captured by the image sensor, it can adopt at least one fusion strategy to determine a group of images for high dynamic range image processing, thereby providing stable and effective front-end long and short frame inputs for subsequent processing, achieving the best balance effect in terms of power consumption, frame rate, and ghost elimination, and achieving the purpose of improving the HDR video frame rate and meeting HDR image quality requirements.

[0056] If the fusion strategy indicates a reference image, the reference image may be indicated as a long exposure image or a short exposure image. If the fusion strategy indicates a fusion auxiliary image, the fusion auxiliary image may be indicated as a long exposure image or a short exposure image. Generally speaking, the fusion auxiliary image corresponds to the reference image. For example, if the reference image is a long exposure image, the fusion auxiliary image is a short exposure image, and the short exposure image has a specific association with the long exposure image, such as the short exposure image is the first short exposure image after the long exposure image, or the short exposure image adjacent to the long exposure image; if the reference image is a short exposure image, the fusion auxiliary image is a long exposure image, and the short exposure image has a specific association with the long exposure image, such as the long exposure image is the first long exposure image after the short exposure image, or the long exposure image adjacent to the short exposure image. The fusion target is the target object fused in multiple frames of images. If the fusion strategy indicates the number of fusion targets, it can be indicated separately for the number of long exposure images and short exposure images, or the total number can be indicated without distinguishing between long exposure images and short exposure images. If the fusion strategy indicates the output frame rate, it can be indicated based on the video frame rate captured by the image sensor.

[0057] Generally speaking, the terminal may include a mobile phone, a pad, a smart screen, a VR glasses, etc., which not only supports real-time online processing and offline processing, but also supports uploading videos to the cloud and sending them to the mobile phone after processing in the cloud.

[0058] In this embodiment, the multiple frames of images continuously captured by the image sensor can be understood as original images (RAW domain images). RAW domain images have a higher dynamic range, a wider range of light intensity, and a higher color depth, and can retain more highlights and shadow details. In addition, each pixel on the RAW domain image is linear data, which is proportional to the exposure intensity. Compared with RGB domain images, HDR fusion in the RAW domain can generate higher quality HDR images.

[0059] Optionally, in this embodiment, the long exposure image is also called a long frame (i.e., long exposure frame) image, and the short exposure image is also called a short frame (i.e., short exposure frame) image. As an implementation method, in order to reduce power consumption, the long frame image acquisition size is full size, such as 1920*980; the short frame image acquisition size is half size, such as 960*490. Short frame images have high noise, and using bilinear interpolation to restore the short frame size to full size can reduce noise intensity and reduce power consumption of storing and transmitting short frame images.

[0060] Optionally, the fusion strategy is the first strategy, the second strategy, the third strategy or the fourth strategy;

[0061] The first strategy is used to indicate that the number of images included in the fusion target is 2, the output frame rate is half of the video frame rate acquired by the image sensor, the reference image is a long exposure image, and the fusion auxiliary image is the first short exposure image after the long exposure image;

[0062] The second strategy is used to indicate that the number of images included in the fusion target is 2, the output frame rate is equal to the video frame rate collected by the image sensor, the reference images are a long exposure image and a short exposure image, and, if the reference image is a long exposure image, the fusion auxiliary image is the first short exposure image after the long exposure image; if the reference image is a short exposure image, the fusion auxiliary image is the first long exposure image after the short exposure image;

[0063] The third strategy is used to indicate that the number of images included in the fusion target is 3, the output frame rate is less than the video frame rate acquired by the image sensor, the reference image is a long exposure image, and the fusion auxiliary image is a short exposure image adjacent to the long exposure image;

[0064] The fourth strategy is used to indicate that the number of images included in the fusion target is 3, the output frame rate is equal to the video frame rate captured by the image sensor, the reference images are a long exposure image and a short exposure image, and, if the reference image is a long exposure image, the fusion auxiliary image is the short exposure image adjacent to the long exposure image; if the reference image is a short exposure image, the fusion auxiliary image is the long exposure image adjacent to the short exposure image.

[0065] Here, the first strategy can also be called a 2-in-1 half-frame rate output strategy, the second strategy can also be called a 2-in-1 equal frame rate output strategy, the third strategy can also be called a 3-in-1 reduced frame rate output strategy, and the fourth strategy can also be called a 3-in-1 equal frame rate output strategy.

[0066] like Figure 4 As shown in the figure, according to the 2-in-1 half-frame rate output strategy, two consecutive long and short exposure images are used as fusion targets. The subsequent fusion can be based on a long exposure image, and the first short exposure image after it is used to align the long exposure image, and the fused HDR images are output in chronological order to improve the fusion speed. For example, in the first group, the long exposure image (L1) is used as the reference, and the short exposure image (S1) is used to align L1, and the fusion result is S1-L1 HDR.

[0067] like Figure 5As shown in the figure, according to the 2-in-1 equal frame rate output strategy, two consecutive long and short exposure images are used as fusion targets. The subsequent fusion can use the long and short exposure images in each group of N frames as the reference in turn, and align the image with the subsequent images. At this time, the output frame rate is consistent with the video frame rate collected by the image sensor. For example, in the first group, the long exposure image (L1) is used as the reference, and the short exposure image (S1) is used to align L1, and the fusion result is S1-L1 HDR; the short exposure image (S1) is used as the reference, and the long exposure image (L2) is used to align S1, and the fusion result is L2-S1 HDR. In this way, multiple HDR images obtained after fusion are output to form an HDR video.

[0068] like Figure 6 As shown in the figure, according to the 3-in-1 frame rate reduction output strategy, three consecutive frames of long and short exposure images are used as fusion targets. The subsequent fusion can be based on the long exposure image in each group, and the long exposure image is aligned with the short exposure images before and after it. At this time, the output frame rate is lower than the video frame rate collected by the image sensor. For example, taking the long exposure image (L1) as the reference, the short exposure images (S0 (not shown in the figure) and S1) are used to align L1, and the fusion result is S0-L1-S1 HDR; then taking the next long exposure image (L2) as the reference, the short exposure images (S1, S2) are used to align L2, and the fusion result is S1-L2-S2 HDR. In this way, multiple HDR images obtained after fusion are output to form an HDR video. This strategy can solve the double occlusion problem in motion scenes.

[0069] like Figure 7 As shown in the figure, according to the 3-in-1 equal frame rate output strategy, three consecutive frames of long and short exposure images are used as fusion targets. The subsequent fusion can use the long and short exposure images in each group of N frames as the reference in turn, and align the image with the images before and after it. At this time, the output frame rate is consistent with the video frame rate collected by the image sensor. For example, taking the long exposure image (L1) as the reference, align L1 with the short exposure images (S0 (not shown in the figure) and S1), and the fusion result is S0-L1-S1 HDR; taking the short exposure image (S1) as the reference, align S1 with the long exposure images (L1 and L2), and the fusion result is L1-S1-L2 HDR. In this way, multiple HDR images obtained after fusion are output to form an HDR video.

[0070] Of course, in addition to the above-mentioned first strategy, second strategy, third strategy and fourth strategy, the fusion strategy may also include other strategies, such as the number of fusion targets is 2M+1, M is less than N, the reference image is a long exposure image, and the fusion auxiliary image is the M short exposure images before and after the long exposure image, so as to improve the long-term stability of HDR video.

[0071] Optionally, in this embodiment, before the processing of the multiple frames of images based on at least one fusion strategy to obtain a target image, the method further includes:

[0072] Determining, according to at least part of the multiple frames of images, types of shooting scenes corresponding to the multiple frames of images;

[0073] The at least one fusion strategy is determined according to the type of the shooting scene.

[0074] Here, at least part of the images in the multiple frames can be selected based on a set time length or number of images. For example, the type of shooting scene corresponding to the multiple frames is determined based on the first two frames of the multiple frames or the frames collected in the first 5 seconds. Among them, the type of shooting scene (scene type) can be pre-determined into multiple types based on the brightness state and the motion state. For example, the motion state is divided into static, motion and large motion, and the brightness state is divided into extremely dark, normal brightness and high brightness. Then the scene type includes 9 categories, as shown in the following table:

[0075]

[0076] After determining the type of shooting scene, the terminal can determine at least one fusion strategy for the type of shooting scene, so as to adopt an applicable fusion strategy to obtain the target image, provide stable and effective front-end long and short frame inputs for subsequent processing based on the characteristics of different scenes, and achieve the best balance in power consumption, frame rate, and ghost elimination, so as to achieve the purpose of improving the HDR video frame rate and meeting the HDR image quality requirements.

[0077] Here, multiple target images may be determined, that is, multiple frames of images may be divided into multiple groups of target images for HDR image processing. The HDR image processing may output multiple fused HDR images based on the multiple groups of target images to form an HDR video. Optionally, in this embodiment, determining the type of shooting scene corresponding to the multiple frames of images based on at least part of the images in the multiple frames of images includes:

[0078] Determining the brightness state of the shooting scene by light intensity detection;

[0079] Determining the motion state of the shooting scene by object motion detection;

[0080] The type of the shooting scene is determined based on the brightness state and the motion state.

[0081] The light intensity detection can be implemented by a light sensor configured in the terminal. For example, the terminal is configured with a light sensor, and the brightness state is classified according to the light intensity detected by the light sensor and the size of the predetermined light intensity threshold. Assuming that the predetermined light intensity threshold includes a first value and a second value (the first value is less than the second value), if the detected light intensity is less than the first value, the brightness state is extremely dark; if the detected light intensity is greater than the first value and less than the second value, the brightness state is normal brightness; if the detected light intensity is greater than the second value, the brightness state is high brightness.

[0082] Among them, object motion detection can be achieved by a lightweight optical flow module configured in the terminal. The lightweight optical flow module is used to detect the object motion amplitude when the camera is turned on. For example, the terminal is configured with a fast optical flow module based on dense inverse search (Fast Optical Flow using Dense Inverse Search), namely the DIS optical flow module. The DIS optical flow module can run on the CPU and calculate the optical flow on a 1920*960 image in milliseconds. In order to reduce the delay in motion amplitude detection, detection can be performed only on 1 / 2-size low-exposure size images. The lightweight optical flow module can output the motion information of the image, thereby counting the area of ​​the moving area and the area of ​​the static area, as well as the average and variance of the optical flow values ​​in the moving area, and comparing them with the threshold value after normalization to distinguish the motion state of the image into static, moving and large motion. Specifically, such as Figure 3 As shown in the figure, the optical flow is first calculated to obtain motion information, and the moving area and the static area are calculated based on the motion information. The static area mask and the moving area mask are distinguished by the optical flow value and threshold 1. Based on the moving area mask, the mask area, the mean and variance of the motion amplitude, the deviation coefficient (motion deviation coefficient) and the motion index are calculated. Finally, the motion index is compared with thresholds 2 and 3 to determine whether the motion state is static, moving or large motion.

[0083] Among them, the calculation formula of the motion index m is: c i represents the motion dispersion coefficient (mean and variance ratio) within motion region i, u i represents the mean value of the motion amplitude of all moving points in the motion area i, δ i represents the variance of the motion amplitude of all moving points in motion area i, s i represents the size of the motion region i, and S represents the size of the original image.

[0084] In other words, the main idea of ​​object motion detection is to determine the ratio of all statistical values ​​of the motion amplitude of the moving area in the image to the area of ​​the moving area. The smaller the ratio, the fewer moving objects in the image and the smaller the motion amplitude, so it is judged as static. The larger the ratio, the larger the area of ​​the moving object in the image and the larger the motion amplitude, so it is judged as motion or large motion.

[0085] When the scene type is determined to be multiple based on the brightness state and the motion state in advance, after the brightness state and the motion state of the shooting scene are determined, the specific scene type of the shooting scene can be determined based on the determined states.

[0086] In addition, optionally, in this embodiment, determining the at least one fusion strategy according to the type of the shooting scene includes:

[0087] Based on a preset mapping relationship between scene types and fusion strategies, at least one fusion strategy corresponding to the type of the shooting scene is determined.

[0088] That is, the mapping relationship between scene type and fusion strategy can be preset, such as scene type 1 and strategy 1, scene type 2 and strategy 2, and so on. After determining the type of shooting scene, it is possible to query at least one fusion strategy corresponding to the type of shooting scene based on the mapping relationship to process the collected multiple frames of images.

[0089] As an optional implementation method, when the motion state of the shooting scene is static, the HDR fusion effect is better and ghosting problems are not likely to occur. It only needs point-to-point synthesis and fusion of short frame detail information on the basis of long frames. Therefore, a 2-in-1 frame rate reduction output strategy can be adopted, and the default acquisition interval can be selected. When the motion state of the shooting scene is moving, the HDR fusion result is prone to double occlusion and ghosting problems. Three frames of information can be used for fusion. The three frames of information jointly capture the motion characteristics of the object to reduce the impact of double occlusion. Therefore, a 3-in-1 frame rate reduction output is adopted. When the motion state of the shooting scene is large motion, in order to better capture the details of the object's motion, more frame information is used for fusion, and a 3-in-1 frame rate output can be adopted.

[0090] As an optional implementation method, when the brightness state of the shooting scene is extremely dark, the amount of light entering is insufficient and the overall RAW image is too dark, so it is necessary to increase the ISO value to increase light sensitivity. In addition, since noise is dominant in extremely dark scenes, HDR fusion uses the long frame as the reference image for fusion. When the brightness state of the shooting scene is normal brightness, the light entering is sufficient. HDR fusion can adopt the output strategy of long frame as the reference image or long and short frames as the reference image. The detailed selection needs to be determined according to the amplitude of the object's motion. When the brightness state of the shooting scene is high brightness, due to the sufficient amount of light entering, the overexposed area of ​​the long frame image is high, and the output strategy of long and short frames as the reference image is adopted.

[0091] In this way, the mapping relationship between the scene type and the fusion strategy can be preset according to the above scheme.

[0092] It should also be known that in this embodiment, the long and short frame exposure strategy used by the image sensor to collect images is as follows: Figure 8 As shown in the figure, the long frame exposure time, short frame exposure time, acquisition interval (gap), and exposure time interval are included. In this long-short exposure strategy, the short exposure image is next to the long exposure image within an exposure time interval (sampling interval), the long exposure size is the original size (1920*960), and the short exposure size is half size (960*480). In a sampling interval, in addition to the long-short exposure time and the short frame exposure time, there is also some remaining time (acquisition gap).

[0093] Optionally, in this embodiment, the method further includes:

[0094] Determine the acquisition interval according to the motion state of the shooting scene; wherein the acquisition interval is the remaining time in a sampling interval except the long frame exposure time and the short frame exposure time;

[0095] The image sensor is adjusted according to the acquisition interval, and the multiple frames of images are updated.

[0096] In this way, the acquisition gap can be dynamically adjusted according to the motion state of the shooting scene to adapt to different scenes. After determining the applicable acquisition gap, the image sensor is adjusted, the image acquisition is performed again, and the multiple frames of images to be processed are updated. Subsequently, the target image can be determined based on the re-acquired multiple frames of images.

[0097] For example, when the motion state is static, the acquisition gap can be increased to reduce the power consumption of RAW domain image acquisition. When the motion state is moving, the acquisition gap can be reduced to reduce the severity of ghosting in HDR fusion images. The acquisition gap adjustment strategy uses a linear transition solution based on the magnitude of the motion, such as Fig. 9As shown. When the motion index is less than θ1, the motion state is still, and the acquisition gap is set to gap1, which reduces power consumption while ensuring the HDR fusion effect. When the motion index is between θ1 and θ2, the acquisition gap is reduced to reduce the exposure time interval between long and short frames, thereby reducing the ghosting effect. Due to the cost and technical limitations of the terminal camera, when the motion index is greater than θ2, the acquisition gap is reduced to gap2.

[0098] Of course, in the long and short exposure strategies, the sampling interval and the long and short exposure times can be set to be fixed.

[0099] Optionally, the method further comprises:

[0100] Determining the photosensitivity according to the brightness state of the shooting scene;

[0101] The image sensor is adjusted according to the sensitivity, and the multiple frames of images are updated.

[0102] That is, after determining the brightness state of the shooting scene, the sensitivity (IOS) is adjusted according to the brightness state of the shooting scene to adapt to different scenes. After determining the applicable IOS, the image sensor is adjusted, the image is recaptured, and the multiple frames of images to be processed are updated. Subsequently, the target image can be determined based on the recaptured multiple frames of images.

[0103] Among them, the IOS of the brightness state of different scenes can be pre-set. For example, when the brightness state is extremely dark, the IOS is set to I1; when the brightness state is normal, the IOS is set to I2; when the brightness state is high brightness, the IOS is set to I3.

[0104] As an optional implementation, in this embodiment, if Fig.10 As shown in the figure, if the actual shooting scene is divided into 9 categories according to brightness and motion, the 2-in-1 frame rate reduction (half frame rate) strategy is adopted when the motion state is static, so as to reduce the power consumption of long and short frame acquisition. In the high-brightness large motion scene, because the object moves quickly and the high-exposure area is prone to cool occlusion, the 3-in-1 equal frame rate strategy is adopted to capture more motion details; in the normal brightness large motion scene, the 3-in-1 frame rate reduction strategy is adopted to capture motion details while reducing the acquisition power consumption; in the extremely dark large motion scene, the 2-in-1 equal frame rate strategy is adopted to reduce power consumption and solve ghosting. In the high-brightness normal motion scene, the 3-in-1 frame rate reduction strategy is used, and the 2-in-1 equal frame rate strategy is used in other brightness scenes. Among them, the difference between the 2-in-1 and 3-in-1 strategies is the ghosting problem caused by double occlusion when the motion state is static and moving. The difference between equal frame rate and unequal frame rate is the problem of motion detail recovery when the motion state is static and moving.

[0105] The method of the embodiment of the present application can be run on a dedicated processor chip. According to the characteristics of the actual shooting scene, the fusion strategy, ISO, and acquisition interval used can be automatically adjusted to provide reliable long and short frame inputs after achieving a balance between power consumption, frame rate, and ghost elimination. At the same time, the noise intensity and exposure ratio information can be input to the back-end processing to assist the back-end processing in better fusing the long and short frame images.

[0106] It should be noted that in this embodiment, the hardware components of the terminal shown are as follows Fig.11 As shown, it includes (a) audio / video system, such as: camera (image sensor), audio equipment, display equipment; (b) chip processing system, such as central processing unit (CPU), neural network processor (NPU), graphics processing unit (GPU); (c) memory, such as system memory, local cache, etc.; (d) communication bus. Of course, other hardware parts such as communication interface can also be included. Software components include kernel layer, system layer, application framework layer, and application layer. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register, etc. The storage medium is located in the memory, and the processor executes the instructions in the memory, combined with its hardware completion.

[0107] The camera converts the optical image into an electronic signal (image Raw data), and the processor (CPU, NPU, GPU) is used for system coordination, scheduling, management, image-related processing flow and AI neural network processing flow, providing systematic computing support for the algorithm model. The system memory is used to temporarily store the calculation data in the CPU and exchange cache data with the CPU. The local cache is a buffer for CPU data exchange, which is used to solve the speed difference problem between the CPU speed and the memory speed. The processor, memory, and communication structure are connected through a communication bus; the memory is used to store execution instructions, the processor executes calculation instructions, and the required previous and next frame image data. The display device is used to present the digital image signal on the display to ensure the normal operation of the LCD display. The chip processing system is responsible for processing the data captured by the camera in the HDR video (High Dynamic Range Video Module-Lite, HDR-VM-Lite) module to output HDR-fused RGB images and HDR videos. The details of the HDR-MV-LITE module are as follows: Fig.12As shown in the figure, it mainly includes four submodules: (a) HDR video frame acquisition submodule; (b) HDR single frame synthesis submodule; (c) storage submodule; (d) HDR video single frame processing submodule. The HDR-MV-LITE module inputs continuous long and short exposure RAW domain images and outputs RAW domain HDR synthesis images. The HDR video acquisition strategy (the strategy used by the image sensor to acquire images) is located in the HDR video frame acquisition submodule.

[0108] The HDR-VM-Lite module is used for terminal photo taking, video recording, video calling, live broadcasting and other functions.

[0109] Among them, the HDR single-frame synthesis submodule includes (a) optical flow estimation unit 1, (b) long and short exposure registration unit, and (c) long and short exposure synthesis unit.

[0110] The optical flow estimation unit 1 is used to estimate the optical flow motion vector between the long and short exposure frames.

[0111] The long-short exposure registration unit is used to align the pixels of the long-short exposure images according to the optical flow motion information, and can be calculated using the following formula I(x',y')=I(x+Δx,y+Δy). Where I represents the pixel value, (x',y') represents the pixel position of the target image; (x,y) represents the pixel position of the original image, and (Δx,Δy) represents the optical flow motion vector calculated by the optical flow estimation unit 1.

[0112] The long and short exposure synthesis unit processes the long and short exposure images separately and outputs a fusion weight map. The fusion weight can be calculated based on factors such as exposure ratio, pixel brightness, regional similarity, and time similarity, and can also be output through an AI neural network.

[0113] The HDR video single frame processing submodule includes an optical flow estimation unit 2, a preprocessing unit, an HDR multi-frame feature extraction unit, an HDR multi-frame feature association unit, an HDR single-frame feature generation unit, and an artifact elimination unit. The optical flow estimation result is used for HDR multi-frame feature fusion. After extracting the HDR video frame features, an alignment operation is performed on the feature layer. Finally, a jitter-free, color-stable HDR video is output.

[0114] Among them, the HDR video frame acquisition submodule automatically adjusts the fusion strategy, acquisition interval, and sensitivity according to the light intensity of the external environment and the movement of objects to ensure the optimal front-end long and short frame input in different scenarios.

[0115] Specifically, when a user uses a terminal to enable HDR video mode, such as clicking Fig.13 The HDR mode function button shown in the figure, the terminal camera will capture video frame images with a fixed frame rate and alternating long and short exposures. After that, the HDR video processing flow is as follows Fig.14 As shown:

[0116] The terminal applies at least one fusion strategy (also known as HDR frame fusion strategy) in the embodiments of the present application to obtain the target image, and inputs the target image into the HDR single-frame synthesis submodule. The HDR single-frame synthesis submodule selects alternating long and short exposure frames from the DDR (a) memory for corresponding HDR single-frame fusion; at the same time, the HDR single-frame synthesis submodule calculates the optical flow information between the input long and short exposure frames, and stores the calculation results in the DDR (c) memory; the HDR single-frame synthesis submodule saves a copy of the long and short exposure fusion results in the DDR (b) memory for subsequent use by the HDR video single-frame processing submodule, and directly inputs a copy into the HDR video single-frame processing submodule. The HDR video single-frame processing submodule reads the original long and short exposure data, optical flow data, and HDR fused image, and ultimately outputs an HDR image that is consistent with the texture, color, motion and other features of the historical frame as a frame in the HDR output video.

[0117] In the HDR video processing flow, optical flow calculation runs on the SoC GPU, and other functions of the HDR single-frame synthesis submodule are implemented on the SoC CPU; other functions of the HDR video single-frame processing submodule are implemented on the SoC NPU. The performance of each SoC chip is fully utilized to output HDR video data in real time while ensuring the stability of HDR video.

[0118] In this way, the real-time HDR video output function can be completed on the terminal side, combining the optical flow information of the HDR video frame and relying on the AI ​​neural network model to improve the front and back stability of the HDR video frame, so as to achieve real-time online HDR video output. This solution can be used to reduce the jitter phenomenon of HDR video, and HDR video truly reflects the color range of real scenes.

[0119] Or, specifically, after using the terminal for HDR shooting, the user can set the HDR offline processing function. After the setting is completed, the terminal background starts the HDR video synthesis effect when the current process task is not busy. At this time, the original long and short exposure data collected by the front-end camera has been cleared from the DDR, so it is necessary to perform frame insertion processing on the synthesized HDR video, and at the same time call the optical flow model to calculate the optical flow vector information of the front and back frames of the HDR video. The offline processing process is as follows Fig.15 As shown:

[0120] The HDR video is first deframed, and the result is stored in the DDR (a) memory; the deframed image enters the optical flow estimation unit 1 to output the optical flow motion vector information (referred to as optical flow information) between the previous and next frames; the HDR video single frame processing submodule is called, and the HDR video single frame processing submodule performs self-training on the HDR video. After the training is completed, the trained model is used to read the HDR deframe data to regenerate each frame in the HDR video to remove the HDR video jitter. In this way, the HDR video offline output function can be completed on the terminal side, and self-training can be performed on the HDR video to make the model more compatible with the HDR video content and generate a more stable HDR video.

[0121] The image processing method provided in the embodiment of the present application can be executed by an image processing device. In the embodiment of the present application, an image processing device executing the image processing method is taken as an example to illustrate the image processing device provided in the embodiment of the present application.

[0122] like Fig.16 As shown, an image processing device 1600 according to an embodiment of the present application includes:

[0123] The first processing module 1610 is used to obtain multiple frames of images continuously acquired by the image sensor;

[0124] The second processing module 1620 is configured to process the multiple frames of images based on at least one fusion strategy to obtain a target image, where the target image is a group of images used for high dynamic range image processing; wherein the fusion strategy is used to indicate at least one of the following:

[0125] Baseline image;

[0126] Fusion of auxiliary images;

[0127] The number of images contained in the fusion target;

[0128] Output frame rate.

[0129] Optionally, the fusion strategy is the first strategy, the second strategy, the third strategy or the fourth strategy;

[0130] The first strategy is used to indicate that the number of images included in the fusion target is 2, the output frame rate is half of the video frame rate acquired by the image sensor, the reference image is a long exposure image, and the fusion auxiliary image is the first short exposure image after the long exposure image;

[0131] The second strategy is used to indicate that the number of images included in the fusion target is 2, the output frame rate is equal to the video frame rate collected by the image sensor, the reference images are a long exposure image and a short exposure image, and, if the reference image is a long exposure image, the fusion auxiliary image is the first short exposure image after the long exposure image; if the reference image is a short exposure image, the fusion auxiliary image is the first long exposure image after the short exposure image;

[0132] The third strategy is used to indicate that the number of images included in the fusion target is 3, the output frame rate is less than the video frame rate acquired by the image sensor, the reference image is a long exposure image, and the fusion auxiliary image is a short exposure image adjacent to the long exposure image;

[0133] The fourth strategy is used to indicate that the number of images included in the fusion target is 3, the output frame rate is equal to the video frame rate captured by the image sensor, the reference images are a long exposure image and a short exposure image, and, if the reference image is a long exposure image, the fusion auxiliary image is the short exposure image adjacent to the long exposure image; if the reference image is a short exposure image, the fusion auxiliary image is the long exposure image adjacent to the short exposure image.

[0134] Optionally, the device further comprises:

[0135] A third processing module, configured to determine the type of shooting scenes corresponding to the multiple frames of images according to at least part of the multiple frames of images;

[0136] The fourth processing module is used to determine the at least one fusion strategy according to the type of the shooting scene.

[0137] Optionally, the third processing module is further used for:

[0138] Determining the brightness state of the shooting scene by light intensity detection;

[0139] Determining the motion state of the shooting scene by object motion detection;

[0140] The type of the shooting scene is determined based on the brightness state and the motion state.

[0141] Optionally, the fourth processing module is further used for:

[0142] Based on a preset mapping relationship between scene types and fusion strategies, at least one fusion strategy corresponding to the type of the shooting scene is determined.

[0143] Optionally, the device further comprises:

[0144] A fifth processing module, configured to determine a collection interval according to the motion state of the shooting scene; wherein the collection interval is the remaining time in a sampling interval excluding the long frame exposure time and the short frame exposure time;

[0145] The sixth processing module is used to adjust the image sensor according to the acquisition interval and update the multiple frames of images.

[0146] Optionally, the device further comprises:

[0147] A seventh processing module, used to determine the photosensitivity according to the brightness state of the shooting scene;

[0148] An eighth processing module is used to adjust the image sensor according to the sensitivity and update the multiple frames of images.

[0149] The image processing device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0150] The image processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0151] The image processing device provided in the embodiment of the present application can achieve Figure 1-Figure 15 To avoid repetition, the various processes implemented in the embodiment of the image processing method are not described again here.

[0152] Alternatively, if Fig.17As shown, an embodiment of the present application further provides an electronic device 1700, including a processor 1701 and a memory 1702, wherein the memory 1702 stores programs or instructions that can be executed on the processor 1701, and when the program or instructions are executed by the processor 1701, the various steps of the above-mentioned image processing method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, they are not described here.

[0153] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0154] Fig.18 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0155] The electronic device 1800 includes but is not limited to: a radio frequency unit 1801, a network module 1802, an audio output unit 1803, an input unit 1804, a sensor 1805, a display unit 1806, a user input unit 1807, an interface unit 1808, a memory 1809, and a processor 1810 and other components.

[0156] Those skilled in the art will appreciate that the electronic device 1800 may also include a power source (such as a battery) for supplying power to each component, and the power source may be logically connected to the processor 1810 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.18 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0157] The processor 1810 is used to obtain multiple frames of images continuously acquired by the image sensor;

[0158] The multiple frames of images are processed based on at least one fusion strategy to obtain a target image, where the target image is a group of images used for high dynamic range image processing; wherein the fusion strategy is used to indicate at least one of the following:

[0159] Baseline image;

[0160] Fusion of auxiliary images;

[0161] The number of images contained in the fusion target;

[0162] Output frame rate.

[0163] Optionally, the fusion strategy is the first strategy, the second strategy, the third strategy or the fourth strategy;

[0164] The first strategy is used to indicate that the number of images included in the fusion target is 2, the output frame rate is half of the video frame rate acquired by the image sensor, the reference image is a long exposure image, and the fusion auxiliary image is the first short exposure image after the long exposure image;

[0165] The second strategy is used to indicate that the number of images included in the fusion target is 2, the output frame rate is equal to the video frame rate collected by the image sensor, the reference images are a long exposure image and a short exposure image, and, if the reference image is a long exposure image, the fusion auxiliary image is the first short exposure image after the long exposure image; if the reference image is a short exposure image, the fusion auxiliary image is the first long exposure image after the short exposure image;

[0166] The third strategy is used to indicate that the number of images included in the fusion target is 3, the output frame rate is less than the video frame rate acquired by the image sensor, the reference image is a long exposure image, and the fusion auxiliary image is a short exposure image adjacent to the long exposure image;

[0167] The fourth strategy is used to indicate that the number of images included in the fusion target is 3, the output frame rate is equal to the video frame rate captured by the image sensor, the reference images are a long exposure image and a short exposure image, and, if the reference image is a long exposure image, the fusion auxiliary image is the short exposure image adjacent to the long exposure image; if the reference image is a short exposure image, the fusion auxiliary image is the long exposure image adjacent to the short exposure image.

[0168] Optionally, the processor 1810 is further configured to:

[0169] Determining, according to at least part of the multiple frames of images, types of shooting scenes corresponding to the multiple frames of images;

[0170] The at least one fusion strategy is determined according to the type of the shooting scene.

[0171] Optionally, the processor 1810 is further configured to:

[0172] Determining the brightness state of the shooting scene by light intensity detection;

[0173] Determining the motion state of the shooting scene by object motion detection;

[0174] The type of the shooting scene is determined based on the brightness state and the motion state.

[0175] Optionally, the processor 1810 is further configured to:

[0176] Based on a preset mapping relationship between scene types and fusion strategies, at least one fusion strategy corresponding to the type of the shooting scene is determined.

[0177] Optionally, the processor 1810 is further configured to:

[0178] Determine the acquisition interval according to the motion state of the shooting scene; wherein the acquisition interval is the remaining time in a sampling interval except the long frame exposure time and the short frame exposure time;

[0179] The image sensor is adjusted according to the acquisition interval, and the multiple frames of images are updated.

[0180] Optionally, the processor 1810 is further configured to:

[0181] Determining the photosensitivity according to the brightness state of the shooting scene;

[0182] The image sensor is adjusted according to the sensitivity, and the multiple frames of images are updated.

[0183] It should be understood that in the embodiment of the present application, the input unit 1804 may include a graphics processor (Graphics Processing Unit, GPU) 18041 and a microphone 18042, and the graphics processor 18041 processes the image data of the static picture or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 1806 may include a display panel 18061, and the display panel 18061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 1807 includes a touch panel 18071 and at least one of other input devices 18072. The touch panel 18071 is also called a touch screen. The touch panel 18071 may include two parts: a touch detection device and a touch controller. Other input devices 18072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0184] The memory 1809 can be used to store software programs and various data. The memory 1809 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 1809 may include a volatile memory or a non-volatile memory, or the memory x09 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 1809 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0185] The processor 1810 may include one or more processing units; optionally, the processor 1810 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 1810.

[0186] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned image processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0187] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0188] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned image processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0189] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0190] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned image processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0191] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0192] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0193] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An image processing method, characterized in that: Executed by the terminal, including: Acquire multiple frames of images continuously collected by an image sensor; The multiple frames of images are processed based on at least one fusion strategy to obtain a target image, where the target image is a group of images used for high dynamic range image processing; wherein the fusion strategy is used to indicate at least one of the following: Baseline image; Fusion of auxiliary images; The number of images contained in the fusion target; Output frame rate.

2. The method according to claim 1, characterized in that: The fusion strategy is the first strategy, the second strategy, the third strategy or the fourth strategy; The first strategy is used to indicate that the number of images included in the fusion target is 2, the output frame rate is half of the video frame rate acquired by the image sensor, the reference image is a long exposure image, and the fusion auxiliary image is the first short exposure image after the long exposure image; The second strategy is used to indicate that the number of images included in the fusion target is 2, the output frame rate is equal to the video frame rate collected by the image sensor, the reference images are a long exposure image and a short exposure image, and, if the reference image is a long exposure image, the fusion auxiliary image is the first short exposure image after the long exposure image; if the reference image is a short exposure image, the fusion auxiliary image is the first long exposure image after the short exposure image; The third strategy is used to indicate that the number of images included in the fusion target is 3, the output frame rate is less than the video frame rate acquired by the image sensor, the reference image is a long exposure image, and the fusion auxiliary image is a short exposure image adjacent to the long exposure image; The fourth strategy is used to indicate that the number of images included in the fusion target is 3, the output frame rate is equal to the video frame rate captured by the image sensor, the reference images are a long exposure image and a short exposure image, and, if the reference image is a long exposure image, the fusion auxiliary image is the short exposure image adjacent to the long exposure image; if the reference image is a short exposure image, the fusion auxiliary image is the long exposure image adjacent to the short exposure image.

3. The method according to claim 1, characterized in that Before the processing of the multiple frames of images based on at least one fusion strategy to obtain the target image, the method further includes: determining the type of shooting scenes corresponding to the multiple frames of images according to at least part of the multiple frames of images; The at least one fusion strategy is determined according to the type of the shooting scene.

4. The method according to claim 3, characterized in that The determining, based on at least part of the multiple frames of images, types of shooting scenes corresponding to the multiple frames of images includes: Determining the brightness state of the shooting scene by light intensity detection; Determining the motion state of the shooting scene by object motion detection; The type of the shooting scene is determined based on the brightness state and the motion state.

5. The method according to claim 3, characterized in that: The determining, according to the type of the shooting scene, the at least one fusion strategy includes: Based on a preset mapping relationship between scene types and fusion strategies, at least one fusion strategy corresponding to the type of the shooting scene is determined.

6. The method according to claim 4, characterized in that Also includes: Determine the acquisition interval according to the motion state of the shooting scene; wherein the acquisition interval is the remaining time in a sampling interval except the long frame exposure time and the short frame exposure time; The image sensor is adjusted according to the acquisition interval, and the multiple frames of images are updated.

7. The method according to claim 4, characterized in that Also includes: Determining the photosensitivity according to the brightness state of the shooting scene; The image sensor is adjusted according to the sensitivity, and the multiple frames of images are updated.

8. An image processing device, characterized in that: include: A first processing module is used to obtain multiple frames of images continuously collected by an image sensor; The second processing module is used to process the multiple frames of images based on at least one fusion strategy to obtain a target image, where the target image is a group of images used for high dynamic range image processing; wherein the fusion strategy is used to indicate at least one of the following: Baseline image; Fusion of auxiliary images; The number of images contained in the fusion target; Output frame rate.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the image processing method according to any one of claims 1 to 7 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the image processing method according to any one of claims 1 to 7 are implemented.

Images