Image determination method and device, image processing method and device and electronic equipment

Abstract

The image determination method comprises the following steps: determining the type of an image sensor; the image sensor type is supporting a digital overlapping wide dynamic range or not supporting the digital overlapping wide dynamic range (202); acquiring ambient brightness and motion state information (204); in response to a photographing operation, an image set for fusion is determined according to the image sensor type, the ambient brightness, and the motion state information (206).

Description

Image determination method and device, image processing method and device, and electronic device Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image determination method and apparatus, an image processing method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the development of computer technology, most electronic devices such as mobile phones are equipped with cameras, and the camera is used to achieve the shooting function.

[0003] However, during the shooting process, the camera often encounters situations such as object movement and insufficient ambient light, resulting in blurred images.

[0004] Summary of the Invention

[0005] Embodiments of the present application provide an image determination method and apparatus, an image processing method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the clarity of captured images.

[0006] In a first aspect, the present application provides an image determination method. The image determination method includes:

[0007] Determine an image sensor type; the image sensor type is one that supports digital overlap wide dynamic range or does not support digital overlap wide dynamic range;

[0008] Get ambient brightness and motion status information;

[0009] In response to a shooting operation, an image set for fusion is determined according to the image sensor type, the ambient brightness, and the motion state information.

[0010] A second aspect of the present application provides an image processing method. The image processing method includes:

[0011] Acquire at least two sets of brightness image sets with different brightness;

[0012] For each group of brightness image sets, synthesizing the image frames to be processed in the group of brightness image sets to obtain corresponding synthesized frames;

[0013] determining a fusion reference frame from the plurality of synthesized frames;

[0014] Taking the fusion reference frame as a reference, a plurality of the synthesized frames are fused to obtain a target image.

[0015] A third aspect of the present application provides an image determination device. The image determination device includes:

[0016] A type determination module, configured to determine an image sensor type; the image sensor type is one that supports digital overlap wide dynamic range or does not support digital overlap wide dynamic range;

[0017] Information acquisition module, used to obtain environmental brightness and motion status information;

[0018] The image frame determination module is configured to determine, in response to a shooting operation, an image frame for fusion according to the image sensor type, the ambient brightness, and the motion state information.

[0019] A fourth aspect of the present application provides an image processing device. The image processing device includes:

[0020] An image acquisition module, configured to acquire at least two sets of brightness image sets with different brightness;

[0021] A synthesis module, for synthesizing, for each luminance image set, the image frames to be processed in the luminance image set to obtain a corresponding synthesized frame;

[0022] A reference frame determining module, for determining a fusion reference frame from the plurality of synthesized frames;

[0023] The fusion module is used to fuse the multiple synthetic frames based on the fusion reference frame to obtain a target image.

[0024] In a fifth aspect, the present application provides an electronic device. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of the image determination method provided in the first aspect or the steps of the image processing method provided in the second aspect.

[0025] In a sixth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the image determination method provided in the first aspect, or the steps of the image processing method provided in the second aspect.

[0026] In a seventh aspect, the present application provides a computer program product, which includes a computer program, and when executed by a processor, implements the steps of the image determination method provided in the first aspect, or implements the steps of the image processing method provided in the second aspect.

[0027] The above-mentioned image determination method, device and electronic device, computer-readable storage medium and computer program product obtain the image sensor type, ambient brightness and motion status information. After the shooting operation, a comprehensive decision is made based on the image sensor type, ambient brightness and motion status information to determine the image set for fusion. The images in the image set are more consistent with the image sensor type, ambient brightness and motion status information, and the fused image is clearer.

[0028] The above-mentioned image processing method, device and electronic device, computer-readable storage medium and computer program product obtain at least two groups of brightness image sets. For each group of brightness image sets, the individual image frames to be processed in each group of brightness image sets are first synthesized to obtain corresponding synthesized frames, and then a fusion reference frame is determined from the synthesized frames. Based on the fusion reference frame, multiple synthesized frames are fused to obtain a target image, thereby generating a more accurate and clearer image. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, without paying any creative work, they can also obtain drawings of other embodiments based on these drawings.

[0030] FIG1 is a schematic diagram of an application scenario of an image determination method and an image processing method in one embodiment.

[0031] FIG2 is a schematic flow chart of an image determination method in one embodiment.

[0032] FIG3 is a schematic diagram of a flow chart of determining the type of an image sensor according to an embodiment.

[0033] FIG. 4 is a comparison diagram of exposure outputs in a digital overlap wide dynamic range mode and a normal mode according to an embodiment.

[0034] FIG5 is a schematic flow chart of an image processing method in one embodiment.

[0035] FIG6 is a schematic diagram of a process of acquiring at least two sets of brightness image sets with different brightnesses in one embodiment.

[0036] FIG7 is a flow chart showing a process of performing high dynamic range fusion on a plurality of synthesized frames based on the fusion reference frame to obtain a target image in one embodiment.

[0037] FIG8 is a schematic diagram of a system architecture for applying an image determination method and an image processing method in one embodiment.

[0038] FIG9 is a schematic diagram of an algorithm flow of an image processing method in one embodiment.

[0039] FIG10 is a detailed diagram of an image processing method according to an embodiment.

[0040] FIG11 is a structural block diagram of an image determination device in one embodiment.

[0041] FIG12 is a structural block diagram of an image processing apparatus in one embodiment.

[0042] FIG13 is a structural block diagram of an image processing apparatus in another embodiment.

[0043] FIG14 is a schematic diagram of the internal structure of a fusion module in one embodiment.

[0044] FIG15 is a structural block diagram of an image processing apparatus in another embodiment.

[0045] FIG16 is a block diagram of the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION

[0046] To facilitate understanding of the present application, a more comprehensive description of the present application will be provided below with reference to the accompanying drawings. The accompanying drawings illustrate preferred embodiments of the present application. However, the present application may be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and comprehensive understanding of the disclosure of the present application.

[0047] It will be understood that the terms "first," "second," and the like used herein may be used to describe various elements herein, but these elements are not limited by these terms. These terms are merely used to distinguish a first element from another element. For example, a first image frame may be referred to as a second image frame, and similarly, a second image frame may be referred to as a first image frame without departing from the scope of this application. Both the first and the second are image frames, but they are not the same image frame. A plurality refers to at least two, such as two, three, or four, but is not limited thereto.

[0048] Figure 1 is a schematic diagram of an application scenario for an image determination method and an image processing method according to one embodiment. As shown in Figure 1 , in this application scenario, electronic device 102 previews scene 104 using its own camera and then, upon receiving a capture command, takes a picture. The image set to be fused is determined based on the image sensor type, ambient brightness, and motion state information. Appropriate images are then fused to produce the target image, improving image clarity. Furthermore, dynamic capture is combined with dynamic motion capture to achieve what you see is what you get.

[0049] FIG2 is a flow chart of an image determination method according to an embodiment. As shown in FIG2 , an image determination method includes steps 202 to 206 .

[0050] Step 202 : Determine the type of image sensor; the image sensor type is either supporting digital overlap wide dynamic range or not supporting digital overlap wide dynamic range.

[0051] When previewing or shooting, electronic devices use the corresponding camera to capture images based on the zoom ratio. Different zoom ratio ranges correspond to different cameras. For example, if an electronic device includes a wide-angle camera, a main camera, and a telephoto camera, the zoom ratio range [0.6, 1) corresponds to the wide-angle camera, the zoom ratio range [1, 2) corresponds to the main camera, and the zoom ratio range [2, 20] corresponds to the telephoto camera. Image sensors can include those that support DOL-HDR (Digital Overlap-High Dynamic Range) and those that do not. Image sensors generally support binning. Image sensors that support DOL-HDR can capture two pairs of real-time image data within the same exposure time, with different brightness levels.

[0052] Optionally, during real-time preview or shooting, the electronic device can determine the corresponding camera according to the zoom ratio, and then determine the type of the image sensor according to the image sensor of the camera.

[0053] Step 204: Acquire ambient brightness and motion status information.

[0054] The ambient brightness refers to the brightness of the environment in which the scene being previewed or captured by the electronic device is located. In one possible embodiment, the ambient brightness can be determined based on the ISO (International Standards Organization, sensitivity) when the image is captured. The smaller the brightness gain value represented by the ISO, the greater the ambient brightness.

[0055] Motion state information is used to characterize the scene captured by the electronic device and whether the electronic device itself is in motion. Motion state information can be categorized as motion state and stationary state, or as global motion state, local motion state, and stationary state, as needed. This motion state information can be obtained during real-time preview of the electronic device or during a capture operation.

[0056] Step 206 : In response to the shooting operation, determine an image set for fusion according to the image sensor type, the ambient brightness, and the motion state information.

[0057] The electronic device detects a trigger operation generated by a user triggering a capture button. In response to the capture operation, the electronic device then selects, based on the image sensor type, ambient brightness, and motion status information, image frames captured in a preview before the capture operation and image frames captured after the capture operation to determine an image set for fusion. The image set for fusion may include at least one of the image frames before and after the capture operation, depending on the circumstances. The image data in the image set may be RAW domain image data or RGB domain image data, etc.

[0058] The above-mentioned image determination method obtains the image sensor type, ambient brightness and motion status information. After the shooting operation, it makes a comprehensive decision based on the image sensor type, ambient brightness and motion status information to determine the image set for fusion. The images in the image set are more consistent with the image sensor type, ambient brightness and motion status information, the fused image is clearer, and the shooting operation and preview are consistent, ensuring that what you see is what you get.

[0059] In some exemplary embodiments, as shown in FIG3 , determining the image sensor type includes:

[0060] Step 302: Obtain the zoom factor.

[0061] The electronic device obtains the zoom factor input by the user in the camera application interface.

[0062] Step 304 : Determine a target image sensor based on the relationship between the zoom factor and the zoom factor switching threshold, and determine whether the target image sensor supports digital overlap wide dynamic range.

[0063] The zoom switching threshold is set based on lens parameters. For example, a zoom ratio range of [0.6, 1) corresponds to a wide-angle camera, a zoom ratio range of [1, 2) corresponds to a main camera, and a zoom ratio range of [2, 20] corresponds to a telephoto camera. Comparing the input zoom ratio with the zoom switching threshold determines the target image sensor (main, wide-angle, or telephoto). Based on the target image sensor, DOL-HDR mode is determined. Furthermore, the target image sensor can be used to determine whether binning mode, remosaic (Bayer image regeneration) mode, or insensor zoom mode is used.

[0064] Step 306 : If the target image sensor supports digital overlap wide dynamic range, determine that the image sensor type supports digital overlap wide dynamic range;

[0065] Step 308 : If the target image sensor supports non-digital overlap wide dynamic range, determine that the image sensor type supports non-digital overlap wide dynamic range.

[0066] Determine the target sensor by zooming in and out, which is simple and convenient.

[0067] In some exemplary embodiments, obtaining the motion state information includes: obtaining gyroscope data during the shooting operation, and brightness difference statistics of corresponding pixel points in two frames of images captured by the shooting operation; and determining the motion state information based on the gyroscope data and the brightness difference statistics.

[0068] Gyroscope data is collected by the gyroscope within the electronic device. The gyroscope collects data in real time during preview mode, and this gyroscope data is cumulative. When the electronic device receives a capture command, it obtains gyroscope data from the capture operation and uses this gyroscope data to determine whether the electronic device experienced any shaking during the capture. The electronic device also obtains statistical values ​​of the brightness difference between corresponding pixels in the two captured image frames to determine whether any objects in the captured scene are moving. The electronic device uses this combined gyroscope data and the brightness difference statistics to determine the motion state information during the capture operation.

[0069] In some exemplary embodiments, determining the motion state information based on the gyroscope data and the brightness difference statistics includes: if the gyroscope data is in a first motion threshold range and the brightness difference statistics are in the first brightness difference threshold range, determining that the motion state information is a stationary state; if the gyroscope data is in a second motion threshold range and the brightness difference statistics are in the second brightness difference threshold range, performing main area motion detection; if the gyroscope data is in a third motion threshold range and the brightness difference statistics are in the third brightness difference threshold range, determining that the motion state information is a global motion state; wherein the upper limit value of the first motion threshold range is less than the lower limit value of the second motion threshold range, and the upper limit value of the second motion threshold range is less than the lower limit value of the third motion threshold range; the upper limit value of the first brightness difference threshold range is less than the lower limit value of the second brightness difference threshold range, and the upper limit value of the second brightness difference threshold range is less than the lower limit value of the third brightness difference threshold range.

[0070] For example, the first motion threshold range is (0, 0.1), the second motion threshold range is [0.1, 0.3), the third motion threshold range is [0.3, ∞), the first brightness difference threshold range is (0, 0.1), the second brightness difference threshold range is [0.1, 0.3), and the third brightness difference threshold range is [0.3, ∞).

[0071] In some exemplary embodiments, the subject area motion detection includes: obtaining brightness difference statistics of pixel points corresponding to the subject area in the two frames of images; if the brightness difference statistics corresponding to the subject area is greater than or equal to a brightness difference threshold, determining that the motion state information is a global motion state; if the brightness difference statistics corresponding to the subject area is less than the brightness difference threshold, determining that the motion state information is a local motion state.

[0072] The brightness difference threshold is set as needed, for example, it can be 8, 9, 10, etc.

[0073] In some exemplary embodiments, when the image sensor type supports DOL and the ambient brightness is greater than a first brightness threshold, determining that the image set for fusion includes at least one image pair data before a capture operation. The determining of the image set for fusion based on the image sensor type, the ambient brightness, and the motion state information includes: if the image sensor type supports digital overlap wide dynamic range and the ambient brightness is greater than the first brightness threshold, determining that the image set for fusion includes at least one image pair data before the capture operation based on the motion state information.

[0074] The first brightness threshold is set as needed. Ambient brightness greater than the first brightness threshold indicates a bright environment. An image pair data set refers to two frames of image data captured within a normal exposure time using digital overlap wide dynamic range technology. The two frames of image data have different brightness. The image pair data set before the capture operation can be image pair data obtained using ZSL (Zero Shutter Lag).

[0075] When the image sensor type supports digital overlapping wide dynamic range and the ambient brightness is greater than the first brightness threshold, the image set for fusion is determined based on the motion state information to include at least one image pair data before the shooting operation. Using the image pair data before the shooting operation for fusion can improve image clarity.

[0076] Figure 4 is a comparison of exposure output in digital overlap wide dynamic range mode and normal mode in one embodiment. As shown in Figure 4, in digital overlap wide dynamic range mode, the camera can output long and short exposure data within a single frame. This means the time interval between the long and short exposures is very short, minimizing motion ghosting. In contrast, in normal mode, the long and short exposures are output in separate frames. In Figure 4, the ratio between the exposure duration of the long and short exposures in both digital overlap wide dynamic range mode and normal mode is 2:1. Consequently, the camera outputs a frame rate of 30 FPS (Frames Per Second) for both long and short exposures in digital overlap wide dynamic range mode, while the camera outputs a frame rate of 60 FPS for both long and short exposures in normal mode.

[0077] It should be noted that in digital overlap wide dynamic range mode, the camera can support the output of long exposure data and short exposure data within a single frame, and can also support the output of 3-4 or more different exposure data within a single frame, without limitation here. It is understood that the two image frames output by the camera within a normal frame exposure time constitute an image pair data. In addition, it should be noted that the frame rate output by the camera in digital overlap wide dynamic range mode and in normal mode can be different or the same.

[0078] In some exemplary embodiments, when the motion state information is a global motion state, a local motion state, or a stationary state, and at least one image pair data prior to the shooting operation included in the image set for fusion differs, if the image sensor type supports a digital overlap wide dynamic range and the ambient brightness is greater than a first brightness threshold, determining, based on the motion state information, that the image set for fusion includes the at least one image pair data prior to the shooting operation includes:

[0079] If the image sensor type supports digital overlap wide dynamic range, the ambient brightness is greater than a first brightness threshold, and the motion state information is a local motion state or a stationary state, determining that the image set for fusion includes at least one first image pair data before the shooting operation, the first image pair data including data of a first baseline exposure frame and a first exposure brightness frame; the brightness of the first baseline exposure frame is greater than the brightness of the first exposure brightness frame;

[0080] If the image sensor type supports digital overlap wide dynamic range, the ambient brightness is greater than the first brightness threshold, and the motion state information is a global motion state, then it is determined that the image set for fusion includes at least one second image pair data before the shooting operation, and the second image pair data includes data of a first exposure duration frame and a second exposure brightness frame; the first exposure duration frame has the same brightness as the first reference exposure frame, and the exposure duration of the first exposure duration frame is less than the first reference exposure frame; the brightness of the first exposure duration frame is greater than the brightness of the second exposure brightness frame.

[0081] Among them, the motion amplitudes corresponding to the global motion state, local motion state and static state gradually decrease.

[0082] The first image pair data (EV0n, EV-) represents image pair data captured in DOL-HDR mode, with ambient light brightness greater than a first brightness threshold, and in a static state within a normal exposure duration of one frame. In (EV0n, EV-), "EV0n" represents a first baseline exposure frame, and "EV-" represents a first exposure brightness frame. The first baseline exposure frame EV0n represents static exposure EV0, i.e., an image frame obtained by normal exposure in a static state when DOL-HDR mode is supported and ambient light brightness is greater than the first brightness threshold. The first exposure brightness frame-represents an image frame obtained by underexposure in a static state when DOL-HDR mode is supported and ambient light brightness is greater than the first brightness threshold. The brightness of the first baseline exposure frame is greater than that of the first exposure brightness frame.

[0083] Among them, EV (Exposure Values) is a quantity that reflects the amount of exposure. When the sensitivity is ISO 100, the aperture coefficient is F1, and the exposure time is 1 second, the exposure is defined as 0. If the exposure is reduced by one stop (the shutter time is reduced by half or the aperture is reduced by one stop), the exposure is EV-1; if the exposure is increased by one stop (the shutter time is doubled or the aperture is increased by one stop), the exposure is EV+1.

[0084] The second image pair data (EV0sg, EV-) represents image pair data captured within a normal exposure frame in DOL-HDR mode, with ambient light brightness greater than a first brightness threshold, and in a global motion state. The "EV0sg" in (EV0sg, EV-) represents the first exposure duration frame, and the "EV-" in (EV0sg, EV-) represents the second exposure brightness frame. The first exposure duration frame EV0sg represents the global motion shorter exposure EV0, i.e., an image frame obtained with short exposure in a global motion state when the DOL-HDR mode is supported and the ambient light brightness is greater than the first brightness threshold. The second exposure brightness frame represents an image frame obtained with underexposure in a global motion state when the DOL-HDR mode is supported and the ambient light brightness is greater than the first brightness threshold.

[0085] In a global motion state, excessively long exposure times can easily blur images. Therefore, the first exposure time frame has the same brightness as the first baseline exposure time frame and is shorter than the first baseline exposure time frame. The brightness of the first exposure time frame is greater than the brightness of the second exposure brightness frame. The exposure time of the second exposure brightness frame matches that of the first exposure time frame and is also shorter than that of the first baseline exposure time frame.

[0086] For the global motion state, the local motion state and the stationary state, the image frames before the shooting operation used for fusion are different, which are more consistent with the motion state information, thereby improving the clarity of the fused image.

[0087] In one possible implementation, the number of first reference exposure frames and the number of first exposure duration frames are both positively correlated with the brightness gain. The number of first reference exposure frames and the number of first exposure duration frames increase as the brightness gain increases.

[0088] Taking the relationship between the number of first-baseline exposure frames and the brightness gain as an example, when the brightness gain is within the first brightness gain range, the number of first-baseline exposure frames is within the first number range; when the brightness gain is within the second brightness gain range, the number of first-baseline exposure frames is within the second number range; and when the brightness gain is within the third brightness gain range, the number of first-baseline exposure frames is within the third number range. The upper limit of the first number range is less than the upper limit of the second number range, the upper limit of the second number range is less than the upper limit of the third number range, the lower limit of the first number range is less than the lower limit of the second number range, and the lower limit of the second number range is less than the lower limit of the third number range.

[0089] For example, when the brightness gain is in the first brightness gain range [1,8], the first number range is 2 to 4 frames; when the brightness gain is in the second brightness gain range (8,32), the second number range is 3 to 6 frames; and when the brightness gain is in the third brightness gain range [32,∞), the third number range is 5 to 8 frames. With other conditions remaining unchanged, the greater the brightness gain, the darker the ambient brightness, and the more first baseline exposure frames or first exposure duration frames required for fusion to avoid information loss and improve the clarity and information richness of the fused image.

[0090] It can be understood that the relationship between the number of first exposure duration frames and the brightness gain is the same as the relationship between the number of first reference exposure frames and the brightness gain described above.

[0091] In some exemplary embodiments, the method further includes: acquiring a shooting mode during the shooting operation.

[0092] The shooting mode can be HDR (High Dynamic Range) mode or non-HDR mode.

[0093] In one possible implementation, the electronic device may detect whether the HDR switch is turned on, or whether the fill light or the flash is turned on. If it is detected that the HDR switch is turned off, or the fill light is turned on, or the flash is turned on, the shooting mode is determined to be non-HDR mode.

[0094] In another possible embodiment, the electronic device detects that at least one of AI Colorful, 50M High Resolution Mode, Timed Shooting Function, Beauty, Filter, 1 Billion Colors and HEIE (High Efficiency Image File Format) is turned on, and then determines that the shooting mode is HDR mode.

[0095] Further, determining the image set for fusion based on the image sensor type, the ambient brightness and the motion state information includes: determining the image set for fusion based on the image sensor type, the ambient brightness, the shooting mode and the motion state information.

[0096] The image set determined to be used for fusion also includes images related to the shooting mode.

[0097] The image set used for fusion determined jointly by the image sensor type, ambient brightness, shooting mode and motion status information is more consistent with the shooting scene, and the resulting image is clearer and more accurate.

[0098] In some exemplary embodiments, when the capture mode is a high dynamic range mode, the image set for fusion includes not only images before the capture operation but also images after the capture operation. The method of determining the image set for fusion based on the image sensor type, the ambient brightness, the capture mode, and the motion state information includes: if the image sensor type supports digital overlap wide dynamic range, the ambient brightness is greater than a first brightness threshold, and the capture mode is a high dynamic range mode, determining, based on the motion state information, that the image set for fusion includes at least one image pair data before the capture operation and at least one image pair data after the capture operation.

[0099] The image pair data before the shooting operation may be the image pair data obtained by ZSL, and the image pair data after the shooting operation may be the image pair data obtained by PSL (Post Shutter Lag).

[0100] When the image sensor supports DOL mode, the ambient brightness is greater than the first brightness threshold, and the shooting mode is high dynamic range mode, the image set determined for fusion includes image pair data before the shooting operation and image pair data after the shooting operation, which can improve the dynamic range of the image and the clarity of the image.

[0101] In a possible implementation, an image pair data before the shooting operation and an image pair data after the shooting operation are both a frame of normal exposure time data, and the brightness of the two frames of images in the one image pair data is different.

[0102] In some exemplary embodiments, if the image sensor type supports digital overlap wide dynamic range, the ambient brightness is greater than a first brightness threshold, and the shooting mode is a high dynamic range mode, determining, based on the motion state information, that an image set for fusion includes at least one image pair data before the shooting operation and at least one image pair data after the shooting operation includes:

[0103] If the image sensor type supports digital overlap wide dynamic range, the ambient brightness is greater than a first brightness threshold, the shooting mode is a high dynamic range mode, and the motion state information is a local motion state, then the image set determined for fusion includes at least one of at least one first image pair data before the shooting operation and at least one third image pair data and at least one fourth image pair data after the shooting operation; the first image pair data includes data of a first baseline exposure frame and a first exposure brightness frame; the brightness of the first baseline exposure frame is greater than the brightness of the first exposure brightness frame; the third image pair data includes data of a second exposure duration frame and a third exposure brightness frame; the second exposure duration frame has the same brightness as the first baseline exposure frame, and the exposure duration of the second exposure duration frame is less than the exposure duration of the first baseline exposure frame; the brightness of the third exposure brightness frame is less than the brightness of the first exposure brightness frame; the fourth image pair data includes a third exposure duration frame and a third exposure brightness frame; the third exposure duration frame has the same brightness as the first baseline exposure frame, and the exposure duration of the third exposure duration frame is greater than the exposure duration of the first baseline exposure frame.

[0104] If the image sensor type supports digital overlapping wide dynamic range, the ambient brightness is greater than a first brightness threshold, the shooting mode is a high dynamic range mode, and the motion state information is a static state, then the image set determined for fusion includes at least one first image pair data before the shooting operation and at least one fourth image pair data after the shooting operation; the first image pair data includes data of a first baseline exposure frame and a first exposure brightness frame; the brightness of the first baseline exposure frame is greater than the brightness of the first exposure brightness frame; the fourth image pair data includes a third exposure brightness frame and a third exposure duration frame; the third exposure duration frame has the same brightness as the first baseline exposure frame, and the exposure duration of the third exposure duration frame is greater than the exposure duration of the first baseline exposure frame.

[0105] If the image sensor type supports digital overlap wide dynamic range, the ambient brightness is greater than the first brightness threshold, the shooting mode is high dynamic range mode and the motion state information is global motion state, then the image set determined for fusion includes at least one second image pair data before the shooting operation and at least one fifth image pair data after the shooting operation; the second image pair data includes data of a first exposure duration frame and a second exposure brightness frame; the fifth image pair data includes data of a fourth exposure brightness frame and a fourth exposure duration frame; the first exposure duration frame has the same brightness as the first baseline exposure frame, and the exposure duration of the first exposure duration frame is less than the first baseline exposure frame; the brightness of the first exposure duration frame is greater than the brightness of the second exposure brightness frame; the brightness of the second exposure brightness frame is greater than the brightness of the fourth exposure brightness frame; the brightness of the fourth exposure duration frame is the same as the brightness of the first exposure duration frame, and the exposure duration of the fourth exposure duration frame is greater than the exposure duration of the first exposure duration frame.

[0106] The third image pair data (EV0s1, EV--) represents image pair data captured within a normal exposure time frame in DOL-HDR mode, with ambient light brightness greater than the first brightness threshold, and with local motion. In (EV0s1, EV--), "EV0s1" represents the second exposure time frame, and "EV--" represents the third exposure brightness frame. The second exposure time frame EV0s1 represents the local motion shorter exposure EV0, i.e., an image frame obtained with a shorter exposure time under local motion when DOL-HDR mode is supported and ambient light brightness is greater than the first brightness threshold. The third exposure brightness frame EV-- represents an image frame obtained with a two-stop reduction in exposure from the exposure parameter corresponding to EV0s1 when DOL-HDR mode is supported and ambient light brightness is greater than the first brightness threshold, with local motion.

[0107] The fourth image pair data (EV--, EV0L) represents image pair data collected in a static state within a normal exposure time frame in the DOL-HDR mode, with the ambient light brightness greater than the first brightness threshold. In (EV--, EV0L), "EV--" represents a third exposure brightness frame, and "EV0L" represents a third exposure time frame. The third exposure time frame EV0L is an image frame obtained by long exposure in a static state when the DOL-HDR mode is supported and the ambient brightness is greater than the first brightness threshold. It will be understood that the third exposure brightness frame is used in both the static state and the partial motion state.

[0108] The fifth image pair data (EV--', EV0L') represents image pair data captured in DOL-HDR mode, with ambient light brightness greater than the first brightness threshold, and in a stationary state within one frame of normal exposure duration. In (EV--', EV0L'), "EV--'" represents the fourth exposure brightness frame, and "EV0L'" represents the fourth exposure duration frame. The fourth exposure brightness frame EV--' represents an image frame obtained by reducing the exposure parameters corresponding to EV0sg by two exposure stops in a global motion state, when the DOL-HDR mode is supported and the ambient brightness is greater than the first brightness threshold. The fourth exposure duration frame EV0L' is an image frame obtained by long exposure in a global motion state, when the DOL-HDR mode is supported and the ambient brightness is greater than the first brightness threshold.

[0109] According to different motion states, the image set for fusion is determined in the high dynamic range mode to include image pair data corresponding to the motion state information, so that the fused image is clearer and has a high dynamic range.

[0110] It is understandable that in order to obtain an image with a high dynamic range, it is sufficient to obtain one frame by reducing the image frames by two levels from the image set for fusion. Multiple frames may also be used, which is not limited here.

[0111] In some exemplary embodiments, the exposure duration of the first exposure duration frame is shorter than the exposure duration of the second exposure duration frame. The exposure duration of the first exposure duration frame in a global motion state is shorter than the exposure duration of the second exposure duration frame in a local motion state. Shortening the exposure duration reduces the impact of motion on imaging and reduces image blur.

[0112] In some exemplary embodiments, when the shooting mode is a non-high dynamic range mode, a ZSL method is used to take frames. The image set for fusion is determined based on the image sensor type, the ambient brightness, the shooting mode and the motion state information, including: if the image sensor type supports digital overlapping wide dynamic range, the ambient brightness is greater than a first brightness threshold and the shooting mode is a non-high dynamic range mode, the image set for fusion is determined to include at least one image pair data before the shooting operation based on the motion state information. In the low to medium dynamic range mode, it is determined that the image set for fusion includes the image pair data before the shooting operation, and then the fusion is performed to obtain a clearer image, thereby improving the efficiency of image fusion.

[0113] In some exemplary embodiments, if the image sensor type supports digital overlap wide dynamic range, the ambient brightness is greater than a first brightness threshold, and the shooting mode is not a high dynamic range mode, determining, based on the motion state information, that the image set for fusion includes at least one image pair data before the shooting operation includes:

[0114] If the image sensor type supports digital overlap wide dynamic range, the ambient brightness is greater than a first brightness threshold, the shooting mode is a non-high dynamic range mode, and the motion state information is a global motion state, the image set determined for fusion includes at least one second image pair data before the shooting operation; the second image pair data includes data of a first exposure duration frame and a second exposure brightness frame; the first exposure duration frame has the same brightness as the first reference exposure frame, and the exposure duration of the first exposure duration frame is less than the first reference exposure frame; and the brightness of the first exposure duration frame is greater than the brightness of the second exposure brightness frame;

[0115] If the image sensor type supports digital overlap wide dynamic range, the ambient brightness is greater than a first brightness threshold, the shooting mode is a non-high dynamic range mode, and the motion state information is a local motion state, then the image set determined for fusion includes at least one first image pair data before the shooting operation and at least one frame of a second exposure time length after the shooting operation; the first image pair data includes data of a first baseline exposure frame and a first exposure brightness frame; the brightness of the first baseline exposure frame is greater than the brightness of the first exposure brightness frame;

[0116] If the image sensor type supports digital overlapping wide dynamic range, the ambient brightness is greater than a first brightness threshold, the shooting mode is a non-high dynamic range mode, and the motion state information is a static state, then the image set determined for fusion includes at least one first image pair data before the shooting operation.

[0117] When supporting DOL-HDR mode, the ambient brightness is greater than the first brightness threshold, and the shooting mode is non-high dynamic range mode, the global motion state is fused using the short-exposure image frame before the shooting operation to reduce the impact of motion and improve image clarity. The local motion state has a certain range of motion compared to the static state. The first image pair data before the shooting operation is combined with the second exposure long frame after the shooting operation to reduce the impact of motion and improve image clarity.

[0118] In some exemplary embodiments, when the capture mode is high dynamic range mode, the number of first baseline exposure frames and the number of first exposure duration frames are both positively correlated with the luminance gain. The above-mentioned image determination method further includes: obtaining the product of DRC (Dynamic Range Compression) and DarkBoost (dark enhancement), and determining the number of image frames of different brightness in the image set for fusion based on the product, the luminance gain, and the motion state.

[0119] Taking the brightness gain in the first brightness gain range as an example, when the product is in the first threshold range, the number of first base exposure frames is in the first number range; when the product is in the second threshold range, the number of first base exposure frames is in the first number range, and the number of first exposure brightness frames can be at least one frame; when the product is in the third threshold range, the motion state information is a global motion state or a local motion state, the number of first base exposure frames is in the first number range, and the number of first exposure brightness frames can be at least one frame; when the brightness gain is in the first brightness gain range and the product is in the third threshold range, the motion state information is a static state, the number of first base exposure frames is in the first number range, and the number of first exposure brightness frames can be at least one frame, and the number of third exposure brightness frames can be at least one frame. The upper limit of the first threshold range is less than the lower limit of the second threshold range, and the upper limit of the second threshold range is less than the lower limit of the third threshold range. For example, the first threshold range is (1, 2), the second threshold range is [2, 5], and the third threshold range is (5, ∞).

[0120] It can be understood that the processing when the brightness gain is in the second brightness gain range and the third brightness gain range is the same as that when the brightness gain is in the first brightness gain range.

[0121] In some exemplary embodiments, determining the image set for fusion based on the image sensor type, the ambient brightness and the motion status information includes: if the image sensor type supports digital overlap wide dynamic range and the ambient brightness is not greater than a first brightness threshold, then determining the image set for fusion based on the motion status information includes an image sequence after the shooting operation, and the image sequence includes image frames with different brightness.

[0122] In DOL-HDR mode, when the ambient brightness is no greater than the first brightness threshold, that is, in a dark environment, in order to take into account the image quality, EV0 will lengthen the exposure and adopt a post-frame binning mode to ensure the image quality in dark light.

[0123] In some exemplary embodiments, if the image sensor type supports digital overlap wide dynamic range and the ambient brightness is not greater than a first brightness threshold, determining, based on the motion state information, that the image set for fusion includes an image sequence after the capture operation, includes:

[0124] If the image sensor type supports a digital overlap wide dynamic range, the ambient brightness is not greater than a first brightness threshold, and the motion state information indicates a stationary state, determining that the image set for fusion includes a first image sequence after the shooting operation, the first image sequence including a second baseline exposure frame, a fifth exposure brightness frame, and a sixth exposure brightness frame after the shooting operation, the second baseline exposure frame being a plurality of frames; and the brightness of the second baseline exposure frame, the fifth exposure brightness frame, and the sixth exposure brightness frame decreases in sequence;

[0125] If the image sensor type supports digital overlap wide dynamic range, the ambient brightness is not greater than the first brightness threshold, and the motion state information is a motion state, then it is determined that the image set for fusion includes a second image sequence after the shooting operation, the second image sequence includes a fifth exposure brightness frame and a sixth exposure brightness frame after the shooting operation, and the second image sequence also includes at least one of a second base exposure frame and a fifth exposure duration frame, and the second base exposure frame and the fifth exposure duration frame are multiple frames; the brightness of the second base exposure frame, the fifth exposure brightness frame, and the sixth exposure brightness frame decreases successively; the fifth exposure duration frame has the same brightness as the second base exposure frame, and the exposure duration of the fifth exposure duration frame is less than the exposure duration of the second base exposure frame.

[0126] Optionally, the first baseline exposure frame in the first image sequence is multiple frames, and the fifth exposure brightness frame and the sixth exposure brightness frame are each one frame. Furthermore, in the first image sequence, the fifth exposure brightness frame and the sixth exposure brightness frame are located in the middle of the first baseline exposure frame. Taking the first image sequence including 6 frames of second baseline exposure frames, 1 frame of fifth exposure brightness frame, and 1 frame of sixth exposure brightness frame as an example, the first image sequence is (EV0, EV0, EV0, EV-, EV--, EV0, EV0, EV0), where EV0 is a frame of normally exposed image data, the brightness of the second baseline exposure frame EV0 is greater than the brightness EV- of the fifth exposure brightness frame, and the brightness of the fifth exposure brightness frame EV- is greater than the brightness of the sixth exposure brightness frame EV--.

[0127] For example, the second image sequence may be (EV0, EV0, EV0, EV-, EV--, EV0, EV0, EV0), or (EV0, EV0, EV0s, EV-, EV--, EV0s, EV0, EV0), or (EV0s, EV0s, EV0s, EV-, EV--, EV0s, EV0s, EV0s). The brightness of the fifth exposure duration frame EV0s is the same as that of the second reference exposure frame EV0, and the exposure duration of the fifth exposure duration frame EV0s is shorter than that of the second reference exposure frame EV0.

[0128] In the D0L-HDR mode, since the ambient brightness is no greater than the first brightness threshold, the post-frame method is used to perform binning mode image fusion, ensuring image quality in dark environments. When the motion state information is in motion, the use of short-exposure images can further reduce image noise and improve image clarity.

[0129] In some exemplary embodiments, determining the image set for fusion based on the image sensor type, the ambient brightness and the motion status information includes: if the image sensor type does not support digital overlap wide dynamic range and the ambient brightness is greater than a second brightness threshold, then determining, based on the motion status information, that the image set for fusion includes at least one image frame before the shooting operation.

[0130] In some exemplary embodiments, the method further includes: acquiring a shooting mode during the shooting operation.

[0131] In one possible implementation, the electronic device may detect whether the HDR switch is turned on, or whether the fill light or the flash is turned on. If it is detected that the HDR switch is turned off, or the fill light is turned on, or the flash is turned on, the shooting mode is determined to be non-HDR mode.

[0132] In another possible embodiment, the electronic device detects that at least one of AI Colorful, 50M High Resolution Mode, Timed Shooting Function, Beauty, Filter, 1 Billion Colors and HEIE (High Efficiency Image File Format) is turned on, and then determines that the shooting mode is HDR mode.

[0133] Further, if the image sensor type does not support digital overlapping wide dynamic range and the ambient brightness is greater than a second brightness threshold, then the image set for fusion is determined according to the motion state information to include at least one image frame before the shooting operation, including: if the image sensor type does not support digital overlapping wide dynamic range and the ambient brightness is greater than a second brightness threshold, then the image set for fusion is determined according to the motion state information and the shooting mode to include at least one image frame before the shooting operation and an exposure frame set after the shooting operation.

[0134] When the DOL-HDR mode is not supported and the ambient brightness is greater than the second brightness threshold, the image frame before the shooting operation and the exposure frame set after the shooting operation are obtained and subsequently fused, which can improve the image clarity and ensure that what you see is what you get.

[0135] In some exemplary embodiments, the image determination method further includes: acquiring the sensitivity during the shooting operation; and determining the number of image frames before the shooting operation based on a relationship between the sensitivity and a sensitivity threshold.

[0136] In some exemplary embodiments, determining the number of image frames before the shooting operation based on the relationship between the sensitivity and the sensitivity threshold includes: if the sensitivity is greater than or equal to the sensitivity threshold, determining the number of image frames before the shooting operation to be a first number; if the sensitivity is less than the sensitivity threshold, determining the number of image frames before the shooting operation to be a second number; the first number is greater than the second number.

[0137] The sensitivity threshold is set as needed, for example, 800. The first and second numbers are set as needed. For example, the first number is 6 frames and the second number is 4 frames. If ISO < 800, the number of image frames before the capture operation is 4 frames; if ISO ≥ 800, the number of image frames before the capture operation is 6 frames.

[0138] In some exemplary embodiments, if the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, and the motion state information is a stationary state or a local motion state, determining that the image set for fusion includes at least one first exposure gain frame before the shooting operation, and the brightness of the first exposure gain frame is less than the brightness of a third reference exposure frame;

[0139] If the image sensor type does not support digital overlapping wide dynamic range, the ambient brightness is greater than the second brightness threshold and the motion state information is a global motion state, then it is determined that the image set for fusion includes at least one second exposure gain frame before the shooting operation, and the exposure duration of the second exposure gain frame is less than the exposure duration of the first exposure gain frame.

[0140] The third reference exposure frame EV0n' is image frame data that does not support digital overlap wide dynamic range, has ambient brightness greater than a second brightness threshold, and has motion state information indicating normal exposure in a static state. The third reference exposure frame EV0n' may be the same as or different from the first reference exposure frame EV0n.

[0141] The brightness of the first exposure gain frame EV0nd is less than the brightness of the third reference exposure frame. The actual exposure of EV0nd is EV0n' / drcGain parameter. The drcGain parameter is a ratio calculated by the Qualcomm platform based on the bright area information in the current scene.

[0142] The exposure duration of the second exposure gain frame EVOsgd is shorter than the exposure duration of the first exposure gain frame. The actual exposure of EVOsgd is the parameter EVOn' / drcGain.

[0143] The first exposure gain frame and the second exposure gain frame can achieve an exposure equivalent to EV0n' by multiplying the relevant drcGain parameters.

[0144] In some exemplary embodiments, if the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, and the motion state information is a static state, then the exposure frame set is a first exposure frame set, and the first exposure frame set includes a third reference exposure frame.

[0145] If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than the second brightness threshold and the motion state information is a local motion state, then the exposure frame set is a second exposure frame set, and the second exposure frame set includes a sixth exposure duration frame, and the exposure duration of the sixth exposure duration frame is less than the exposure duration of the third benchmark exposure frame; wherein the sixth exposure duration frame is EV0sl'.

[0146] If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, and the motion state information is a global motion state, the exposure frame set is a third exposure frame set, the third exposure frame set includes a seventh exposure duration frame, and the exposure duration of the seventh exposure duration frame is less than the exposure duration of the third reference exposure frame. The seventh exposure duration frame is EVOsg′.

[0147] In the case where digital overlapping wide dynamic range is not supported and the ambient brightness is greater than the second brightness threshold, the corresponding exposure frame image is obtained according to the motion state information, so as to improve the clarity of the image.

[0148] In some exemplary embodiments, if the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, the motion state information indicates a stationary state, and the shooting mode is a high dynamic range mode, the first exposure frame set includes a third base exposure frame and a seventh exposure brightness frame;

[0149] If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, the motion state information is a stationary state, and the shooting mode is a non-high dynamic range mode, then the first exposure frame set includes a third reference exposure frame;

[0150] If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, the motion state information is a local motion state, and the shooting mode is a high dynamic range mode, the second exposure frame set includes a sixth exposure duration frame and an eighth exposure brightness frame;

[0151] If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, the motion state information is a local motion state, and the shooting mode is a non-high dynamic range mode, the second exposure frame set includes a sixth exposure duration frame;

[0152] If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, the motion state information is a global motion state, and the shooting mode is a high dynamic range mode, the third exposure frame set includes a seventh exposure duration frame and a ninth exposure brightness frame;

[0153] If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than the second brightness threshold, the motion state information is the global motion state, and the shooting mode is a non-high dynamic range mode, then the third exposure frame set includes a seventh exposure duration frame.

[0154] In high dynamic range mode, you can take an extra underexposed image frame for fusion to improve the high dynamic range and clarity of the image.

[0155] In some exemplary embodiments, if the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, the motion state information indicates a stationary state, the shooting mode is a high dynamic range mode, and the minimum exposure value is less than an exposure value threshold, then the first exposure frame set includes a third base exposure frame, a seventh exposure brightness frame, and a tenth exposure brightness frame, and the brightness of the tenth exposure brightness frame is greater than the brightness of the seventh exposure brightness frame;

[0156] If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than the second brightness threshold, the motion state information indicates a stationary state, the shooting mode is a high dynamic range mode, and the minimum exposure value is greater than or equal to the exposure value threshold, then the first exposure frame set includes a third reference exposure frame and a seventh exposure brightness frame;

[0157] If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, the motion state information is a local motion state, the shooting mode is a high dynamic range mode, and the minimum exposure value is less than the exposure value threshold, then the second exposure frame set includes a sixth exposure duration frame, an eighth exposure brightness frame, and an eleventh exposure brightness frame; the brightness of the eleventh exposure brightness frame is greater than the brightness of the eighth exposure brightness frame;

[0158] If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than the second brightness threshold, the motion state information is a local motion state, the shooting mode is a high dynamic range mode, and the minimum exposure value is greater than or equal to the exposure value threshold, then the second exposure frame set includes a sixth exposure duration frame and an eighth exposure brightness frame;

[0159] If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than the second brightness threshold, the motion state information is a global motion state, the shooting mode is a high dynamic range mode, and the minimum exposure value is less than the exposure value threshold, then the third exposure frame set includes a seventh exposure duration frame, a ninth exposure brightness frame, and a twelfth exposure brightness frame; the brightness of the twelfth exposure brightness frame is greater than the brightness of the ninth exposure brightness frame;

[0160] If the image sensor type does not support digital overlapping wide dynamic range, the ambient brightness is greater than the second brightness threshold, the motion state information is a global motion state, the shooting mode is a high dynamic range mode and the minimum exposure value is greater than or equal to the exposure value threshold, then the third exposure frame set includes a seventh exposure duration frame and a ninth exposure brightness frame.

[0161] The seventh, eighth, and ninth exposure brightness frames can be represented by -aEV, and the tenth, eleventh, and twelfth exposure brightness frames can be represented by -a / 2EV. a represents the EV value, which can be calculated by the AEC module. The maximum value of -a is 12.

[0162] The minimum exposure value, EVminus, and the exposure value threshold are set as needed, for example, -5*6. When EVminus < -5*6, the values ​​are -aEV, -a / 2EV, or EV0X. When EVminus ≥ -5*6, the values ​​are -aEV or EV0X. EV0X can be the third reference exposure frame, EV0n', the sixth exposure frame, EV0s1', or the seventh exposure frame, EV0sg'.

[0163] In some exemplary embodiments, the exposure parameters of the seventh exposure brightness frame, the eighth exposure brightness frame, the ninth exposure brightness frame, the tenth exposure brightness frame, the eleventh exposure brightness frame, and the twelfth exposure brightness frame are adjustable. The exposure parameter, i.e., the value of a, is adjustable, for example, and can be any value between 0 and 12.

[0164] In some exemplary embodiments, determining the image set for fusion based on the image sensor type, the ambient brightness and the motion status information includes: if the image sensor type does not support digital overlapping wide dynamic range, and the ambient brightness is not greater than a second brightness threshold, then determining, based on the motion status information, that the image set for fusion includes an exposure frame sequence after the shooting operation, the exposure frame sequence including exposure frames of different brightness.

[0165] In some exemplary embodiments, if the image sensor type does not support digital overlap wide dynamic range and the ambient brightness is not greater than a second brightness threshold, determining, based on the motion state information, that the image set for fusion includes an exposure frame sequence after the shooting operation, includes:

[0166] If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is not greater than a second brightness threshold, and the motion state information indicates a stationary state, determining that the image set for fusion includes a first exposure frame sequence after the shooting operation, the first exposure frame sequence includes a fourth reference exposure frame, a thirteenth exposure brightness frame, and a fourteenth exposure brightness frame after the shooting operation, the fourth reference exposure frame being a plurality of frames; and the brightness of the fourth reference exposure frame, the thirteenth exposure brightness frame, and the fourteenth exposure brightness frame decreases in sequence;

[0167] If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is not greater than the second brightness threshold, and the motion state information is a motion state, then it is determined that the image set for fusion includes a second exposure frame sequence after the shooting operation, the second exposure frame sequence includes the fifteenth exposure brightness frame and the sixteenth exposure brightness frame after the shooting operation, and the second exposure frame sequence also includes at least one of a fourth base exposure frame and an eighth exposure time frame, and the fourth base exposure frame and the eighth exposure time frame are multiple frames; the brightness of the fourth base exposure frame, the fifteenth exposure brightness frame, and the sixteenth exposure brightness frame decreases successively; the eighth exposure time frame has the same brightness as the fourth base exposure frame, and the exposure time of the eighth exposure time frame is less than the exposure time of the fourth base exposure frame.

[0168] The fourth baseline exposure frame in the first exposure frame sequence may be multiple frames, and the thirteenth exposure brightness frame and the fourteenth exposure brightness frame are each one frame. Furthermore, in the first image sequence, the fifth exposure brightness frame and the sixth exposure brightness frame are located in the middle of the first baseline exposure frame. Taking the first exposure frame sequence including six fourth baseline exposure frames, one thirteenth exposure brightness frame, and one fourteenth exposure brightness frame as an example, the first exposure frame sequence is (EV0', EV0', EV0', EV-', EV--', EV0', EV0', EV0'), where EV0' is a frame of normally exposed image data, the brightness of the fourth baseline exposure frame EV0' is greater than the brightness of the thirteenth exposure brightness frame EV-', and the brightness of the thirteenth exposure brightness frame EV-' is greater than the brightness of the fourteenth exposure brightness frame EV--'.

[0169] For example, the second exposure frame sequence may be (EV0, EV0, EV0, EV-, EV--, EV0, EV0, EV0), or (EV0, EV0, EV0s, EV-, EV--, EV0s, EV0, EV0), or (EV0s, EV0s, EV0s, EV-, EV--, EV0s, EV0s, EV0s). The brightness of the fifth exposure duration frame EV0s is the same as that of the second reference exposure frame EV0, and the exposure duration of the fifth exposure duration frame EV0s is shorter than that of the second reference exposure frame EV0.

[0170] In some exemplary embodiments, in the case of global motion, the exposure duration is reduced by a factor of N, and in the case of local motion, the exposure duration is reduced by a factor of N / 2, where N is a positive integer. Since the motion amplitude is greater in the global motion state than in the local motion state, reducing the exposure duration further in the global motion state can avoid motion artifacts and improve image quality.

[0171] In some exemplary embodiments, in the case of global motion, the exposure duration is reduced by a factor of N, and the corresponding brightness gain is increased by a factor of N. In the case of local motion, the exposure duration is reduced by a factor of N / 2, and the corresponding brightness gain is increased by a factor of N / 2. After reducing the exposure duration in the global and local motion states, the corresponding brightness gain is increased to ensure that the image brightness is similar to that of normal exposure.

[0172] FIG5 is a flow chart of an image processing method according to an embodiment. As shown in FIG5 , an image processing method includes steps 502 to 508 .

[0173] Step 502: Acquire at least two sets of brightness image sets with different brightness.

[0174] Optionally, the electronic device detects the brightness of each image frame to be processed and groups the image frames to be processed within the same brightness range into the same brightness image set. The image brightness of each image frame in the same brightness image set can be the same or different. The difference between the upper brightness limit and the lower brightness limit within the same brightness range is within a preset range, that is, the brightness difference between each image frame to be processed in each brightness image set is within a preset range. The preset range can be set as needed.

[0175] Optionally, the electronic device may obtain the image set for fusion obtained by the above-mentioned image determination method, and form the image frames to be processed within the same brightness range into the same group of brightness image sets.

[0176] Step 504 : For each luminance image set, synthesize the image frames to be processed in the luminance image set to obtain a corresponding synthesized frame.

[0177] A composite frame is an image frame obtained by synthesizing the images to be processed in a luminance image set.

[0178] Optionally, for each luminance image set, the electronic device performs noise reduction processing and color interpolation processing on each to-be-processed image frame in the luminance image set to obtain a corresponding synthesized frame.

[0179] Furthermore, for each luminance image set, each image frame to be processed in the luminance image set is subjected to noise reduction processing and color interpolation processing through an interpolation network to obtain a corresponding synthesized frame.

[0180] The interpolation network may be an AI (Artificial Intelligence) model implemented using a CNN (Convolutional Neural Networks). For example, a network structure such as UNET may be used to implement a fusion network for multi-frame or single-frame noise reduction and color interpolation. In some exemplary embodiments, the interpolation network is implemented by a neural-network processing unit (NPU).

[0181] The image frame to be processed is a RAW domain image. After color interpolation processing, the RAW domain image can be converted into an RGB (Red Green Blue) domain image, that is, the synthesized frame is an RGB domain image.

[0182] Each image frame in the luminance image set is a RAW-domain image. The interpolation network features noise reduction (NR) and demosaicing, fully leveraging the advantages of RAW-domain noise reduction. Furthermore, an interpolation network that integrates noise reduction and color interpolation functions implements color interpolation on the algorithm side, eliminating the platform-side color interpolation bottleneck that hinders clarity in traditional technologies. This facilitates obtaining clearer composite frames and, ultimately, a clearer target image.

[0183] In some exemplary embodiments, the bit width of the image input to the interpolation network is smaller than the bit width of the image output by the interpolation network. Bit width refers to the amount of data that can be transferred at one time by memory or video memory. For example, if the output format of an image is 10-bit, the 10 bits in the image represent that the brightness or chrominance values ​​of that frame are stored in a variable with a bit width of 10 bits, so the brightness or chrominance values ​​range from 0 to 1023. For a 16-bit image, the brightness or chrominance values ​​range from 0 to 65536.

[0184] Step 506: Determine a fusion reference frame from the multiple synthesized frames.

[0185] The synthesized frame may be a plurality of synthesized frames of different brightness, and a fusion reference frame for fusion is determined from the synthesized frames of different brightness.

[0186] Step 508 : Taking the fusion reference frame as a reference, fuse the multiple synthesized frames to obtain a target image.

[0187] Using the fusion reference frame as a reference, the composite frame is aligned to the fusion reference frame and fused to obtain the target image. The target image can be a YUV (Y: Luminance or Luma, U, V: Chrominance or Chroma) domain image.

[0188] In this embodiment, at least two groups of brightness image sets are obtained. For each group of brightness image sets, the individual image frames to be processed in each group of brightness image sets are first synthesized to obtain a corresponding synthesized frame. Then, a fusion reference frame is determined from the synthesized frames. Based on the fusion reference frame, multiple synthesized frames are fused to obtain a target image, thereby generating a more accurate and clearer image.

[0189] In some exemplary embodiments, as shown in FIG6 , acquiring at least two sets of brightness image sets with different brightness includes:

[0190] Step 602: Acquire a plurality of image pair data, each image pair data including a first image frame and a second image frame.

[0191] Each image pair can be two frames of image data of different brightness acquired using DOL-HDR technology within a normal exposure time of one frame. Taking eight image pairs as an example, multiple image pairs are (EV0, EV-), (EV0, EV-), (EV0, EV-), (EV0, EV-), (EV0, EV-), (EV0, EV-), (EV0, EV-), (EV0, EV-). EV (Exposure Values) is a quantity that reflects the amount of exposure. When the sensitivity is ISO 100, the aperture coefficient is F1, and the exposure time is 1 second, the exposure is defined as 0. A decrease of one stop (shutter time reduced by half or aperture closed by one stop) is EV-1; an increase of one stop (shutter time doubled or aperture increased by one stop) is EV+1. EV0 represents the first image frame, and EV- represents the second image frame.

[0192] Step 604: Determine a first reference frame from the plurality of first image frames.

[0193] The clarity of the plurality of first image frames EV0 is compared, and the first image frame with the highest or second highest clarity is selected as the first reference frame. For example, EV0 in the third image pair data is selected as the first reference frame.

[0194] Step 606: Acquire the second image frame in the image pair data where the first reference frame is located as the second reference frame.

[0195] EV0 in the third image pair data (EV0, EV-) is used as the first reference frame, and the corresponding EV- in the image pair data is used as the second reference frame.

[0196] Step 608 : Align the remaining first image frames except the first reference frame with the first reference frame, and obtain a first luminance image set based on the first reference frame and the aligned first image frames.

[0197] Among the above 8 image pair data, EV0 in the 1st, 2nd, 4th, 5th, 6th, 7th and 8th image pair data are aligned with EV0 in the 3rd image pair data, and then EV0 in the aligned 1st, 2nd, 3rd, 4th, 5th, 6th, 7th and 8th image pair data constitute the first brightness image set.

[0198] Step 610 : Align the remaining second image frames except the second reference frame with the second reference frame, and obtain a second luminance image set based on the second reference frame and the aligned second image frames.

[0199] Among the above 8 image pair data, the EV- in the 1st, 2nd, 4th, 5th, 6th, 7th and 8th image pair data are aligned with the EV- in the 3rd image pair data, and then the EV- in the aligned 1st, 2nd, 3rd, 4th, 5th, 6th, 7th and 8th image pair data constitute the second brightness image set.

[0200] It is understood that when the multiple image pair data include a third image frame, the third image frame can be aligned with the first reference frame and processed, and a third luminance image set can be formed based on the aligned third image frame. Optionally, when the third image frame is multiple frames, a third image frame similar to the first reference frame can be selected from the multiple third image frames as the third reference frame, and the multiple third image frames can be aligned with the third reference frame. Then, the third luminance image set can be obtained based on the third reference frame and the aligned third image frame. Optionally, when the third image frame is a single frame, the third image frame can also be directly used as the third luminance image set.

[0201] The above image frames with different brightness respectively find corresponding reference frames, and then align other image frames to the corresponding reference frames to obtain aligned image frames, which are convenient for subsequent synthesis to obtain clear images.

[0202] In some exemplary embodiments, the image processing method further includes: determining a first reference frame; determining a reference frame mark based on the first reference frame and an image frame to be processed having a brightness different from that of the first reference frame, wherein the reference frame mark is used to characterize a fused reference frame of high dynamic range fusion.

[0203] A frame is selected from at least two brightness image sets as a first reference frame, and then overexposed areas are compared with the first reference frame and image frames to be processed in other brightness sets except the first reference frame to determine a reference frame mark.

[0204] Furthermore, determining the fused reference frame from the plurality of synthesized frames includes: determining the fused reference frame from the plurality of synthesized frames according to the reference frame tags. Determining the fused reference frame according to the reference frame tags uses a small amount of information.

[0205] In some exemplary embodiments, the at least two groups of brightness image sets include at least two of a first brightness image set, a second brightness image set, and a third brightness image set, the first brightness image set includes multiple first image frames, the second brightness image set includes at least one second image frame, the third brightness image set includes at least one third image frame, the brightness of the first image frame is greater than the brightness of the second image frame, and the brightness of the second image frame is greater than the brightness of the third image frame.

[0206] Correspondingly, the determining the first reference frame includes: determining the first reference frame from multiple frames of the first image frames.

[0207] Correspondingly, determining the reference frame mark based on the first reference frame and the image frame to be processed having a brightness different from that of the reference frame includes: determining the reference frame mark based on the first reference frame, the second image frame and the third image frame, and the reference frame mark is used to characterize the fused reference frame of high dynamic range fusion.

[0208] In some exemplary embodiments, determining the reference frame mark based on the first reference frame, the second image frame and the third image frame includes: determining a motion mask image based on the first reference frame, the second image frame and the third image frame; obtaining the overlapping area between the overexposed areas corresponding to each of the first reference frame, the second image frame and the third image frame and the motion mask image, and determining the reference frame mark based on the respective overlapping areas; if the reference frame mark is a first mark, it indicates that high dynamic range fusion is performed with the synthetic frame corresponding to the first image frame as the fusion reference frame; if the reference frame mark is a second mark, it indicates that high dynamic range fusion is performed with the synthetic frame corresponding to the second image frame as the fusion reference frame; if the reference frame mark is a third mark, it indicates that high dynamic range fusion is performed with the synthetic frame corresponding to the third image frame as the fusion reference frame.

[0209] The electronic device performs optical flow calculation on the first reference frame, the second image frame and the third image frame through an optical flow algorithm to obtain a motion mask image.

[0210] Optionally, the electronic device obtains the corresponding overexposed areas in the first reference frame, the second image frame, and the third image frame. Optionally, the electronic device can obtain the overexposed areas of each of the three image frames based on pixels whose brightness values ​​are greater than a brightness threshold. Further, the electronic device compares the overexposed area of ​​the first reference frame with the motion mask image to obtain a first overlapping area, compares the overexposed area of ​​the second image frame with the motion mask image to obtain a second overlapping area, and compares the overexposed area of ​​the third image frame with the motion mask image to obtain a third overlapping area; compares the areas of the first overlapping area, the second overlapping area, and the third overlapping area to obtain the overlapping area with the smallest area, and obtains the reference frame mark based on the image frame corresponding to the overlapping area with the smallest area. For example, if the overlapping area with the smallest area is the first overlapping area, the reference frame mark is determined to be the first mark based on the first reference frame corresponding to the first overlapping area; if the overlapping area with the smallest area is the second overlapping area, the reference frame mark is determined to be the second mark based on the second image frame corresponding to the second overlapping area; if the overlapping area with the smallest area is the third overlapping area, the reference frame mark is determined to be the third mark based on the third image frame corresponding to the third overlapping area. By determining the reference frame markers using the overexposed area and the motion mask image, the resulting fused reference frame can be made clearer and contain richer information.

[0211] In some exemplary embodiments, the process of determining the first reference frame and the process of determining the benchmark frame marker are implemented by a central processing unit (CPU).

[0212] In some exemplary embodiments, for each group of luminance image sets, each image frame to be processed in the luminance image set is subjected to noise reduction processing and color interpolation processing through an interpolation network to obtain a corresponding synthetic frame, including: if the luminance image set has one frame of image frame to be processed, each image frame to be processed in the luminance image set is subjected to noise reduction processing and color interpolation processing through a single-frame interpolation network to obtain a synthetic frame; if the luminance image set has multiple frames of image frame to be processed, each image frame to be processed in the luminance image set is subjected to noise reduction processing and color interpolation processing through a multi-frame interpolation network to obtain a synthetic frame.

[0213] The multi-frame interpolation network can be MF AI Demosaic (Multi-Frame Artificial Intelligence Demosaic), and the single-frame interpolation network can be SF AI Demosaic (Single-Frame Artificial Intelligence Demosaic). The multi-frame interpolation network and the single-frame interpolation network are called by the neural network processor.

[0214] For the first luminance image set, each 10-bit first image frame in the RAW domain of the first luminance image set is subjected to noise reduction and color interpolation processing through a multi-frame interpolation network to obtain a 16-bit synthetic frame EV0 in the RGB domain; for the second luminance image set, each 10-bit second image frame in the RAW domain of the second luminance image set is subjected to noise reduction and color interpolation processing through a multi-frame interpolation network or a single-frame interpolation network to obtain a 16-bit synthetic frame EV- in the RGB domain; for the third luminance image set, each 10-bit third image frame in the RAW domain of the third luminance image set is subjected to noise reduction and color interpolation processing through a multi-frame interpolation network or a single-frame interpolation network to obtain a 16-bit synthetic frame EV-- in the RGB domain.

[0215] The RAW domain image frames are processed through the interpolation network to prevent the loss of image frame information. The synthesized image frames are richer in information, and the high bit width can ensure a high dynamic range and support multi-frame super-resolution and HDR tasks.

[0216] In some exemplary embodiments, the image processing method further includes: performing at least one of removing black level, applying dynamic range correction gain, lens shading correction (LSC), and automatic white balance (AWB) on the synthesized frame obtained by multi-frame interpolation network processing to obtain an adjusted synthesized frame.

[0217] The image clarity is improved by removing black level, increasing dynamic range adjustment gain, shading correction, white balance and other processes. These processes are implemented by the graphics processor and do not require the image signal processor (ISP), reducing the loss of clarity caused by the ISP.

[0218] In some exemplary embodiments, the image processing method further includes: performing inverse gamma transformation on the synthesized frame obtained by the single-frame interpolation network processing to obtain an adjusted synthesized frame.

[0219] In some exemplary embodiments, the image processing method further includes: if the image to be processed in the luminance image set is a single frame, performing black level removal processing, shading correction, white balancing, and gamma conversion on the image to be processed to obtain an adjusted image to be processed, and inputting the adjusted image to be processed into the single-frame interpolation network to obtain a synthesized frame. The image clarity is improved by performing black level removal, shading correction, white balancing, and gamma conversion, and these processing processes are implemented by a graphics processor, not an image signal processor (ISP), thereby reducing the loss of clarity caused by the ISP.

[0220] In some exemplary embodiments, performing black level removal, white balancing, and gamma transformation on the image to be processed to obtain an adjusted image to be processed includes performing black level removal, noise reduction (noise map computation), shading correction, white balancing, and gamma transformation on the image to be processed to obtain the adjusted image to be processed. In addition to performing black level removal, shading correction, white balancing, and gamma transformation on the image, noise reduction is also performed to further improve image clarity.

[0221] In some exemplary embodiments, the image processing method further includes performing black level removal and noise reduction processing on the multiple frames of image frames to be processed that are input to the multi-frame interpolation network, thereby obtaining adjusted image frames to be processed. The black level removal and noise reduction processing of the image frames to be processed that are input to the multi-frame interpolation network reduces image noise, improves the quality of the input images, and facilitates subsequent fusion to obtain a clearer image.

[0222] In some exemplary embodiments, the image processing method further includes: after obtaining the reference frame marker, performing an affine transformation on the image frame to be processed based on the first reference frame to obtain an affine-transformed image frame to be processed, and inputting the affine-transformed image frame to be processed into a multi-frame interpolation network for processing. The affine transformation is a RAW-domain affine transformation, which allows the image frame to be processed to be more closely aligned with the first reference frame, facilitating subsequent synthesis and improving image clarity.

[0223] In some exemplary embodiments, the image processing method further includes: performing affine transformation on the synthesized frame based on the fused reference frame to obtain an affine transformed synthesized frame.

[0224] Among them, the affine transformation is RGB Warp (to HDR base), that is, based on the fusion reference frame, other synthetic frames are affine transformed to the fusion reference frame in the RGB domain.

[0225] Furthermore, the target image is obtained by fusing the plurality of synthesized frames based on the fused reference frame, including fusing the plurality of synthesized frames that have undergone affine transformation based on the fused reference frame to obtain the target image. By performing affine transformation on the synthesized frames, the synthesized frames are more closely aligned with the fused reference frame, and the target image obtained by subsequent fusion is more accurate and clear.

[0226] In some exemplary embodiments, the fusing of the plurality of synthesized frames based on the fusion reference frame to obtain the target image includes: performing high dynamic range fusing on the plurality of synthesized frames based on the fusion reference frame to obtain the target image.

[0227] By performing high dynamic range fusion on the synthesized frames, the resulting target image has a wider dynamic range, that is, a greater difference between light and dark, and can show more brightness information.

[0228] In some exemplary embodiments, the image processing method further includes: performing target segmentation processing on the fused reference frame to obtain a subject area and a non-subject area.

[0229] The electronic device uses a multi-class segmentation module to perform object segmentation on the fused reference frame to obtain subject regions and non-subject regions. Subject regions may include portrait regions, face regions, animals, vehicles, etc. Non-subject regions may include sky, grass, etc. In some exemplary embodiments, the object segmentation process is implemented by a neural network processor.

[0230] Correspondingly, as shown in FIG7 , the high dynamic range fusion is performed on the plurality of synthesized frames based on the fusion reference frame to obtain a target image, including steps 702 to 708 .

[0231] Step 702 : Taking the fusion reference frame as a reference, perform high dynamic range fusion on the plurality of synthesized frames to obtain a first fused image.

[0232] The graphics processor can perform high dynamic range linear fusion on multiple synthesized frames based on the fusion reference frame to obtain a first fused image.

[0233] Step 704: Fuse the subject area with the first fused image to obtain a second fused image.

[0234] Step 706: Perform tone mapping processing on the second fused image to obtain a third fused image.

[0235] The tone mapping may include at least one of global tone mapping (GTM) and local tone mapping (LTM). Optionally, the graphics processor may perform at least one of global tone mapping and local tone mapping on the second fused image to obtain a third fused image. If the tone mapping includes global tone mapping and local tone mapping, the graphics processor may first perform global tone mapping on the second fused image and then perform local tone mapping to obtain the third fused image. Alternatively, the graphics processor may first perform local tone mapping on the second fused image and then perform global tone mapping to obtain the third fused image.

[0236] Step 708: Adjust the third fused image according to the subject area and the non-subject area to obtain a target image.

[0237] Optionally, the graphics processor may adjust the third fused image based on the subject area and the non-subject area obtained by target segmentation, so as to make the subject area more obvious and obtain a target image.

[0238] The subject area and the non-subject area are obtained by performing target segmentation on the fusion reference frame, and then the subject area is fused with the first fused image to obtain a second fused image, so that the subject area in the second fused image is more prominent. The second fused image is then tone mapped to obtain a third fused image, so that the high dynamic range is adapted to the low dynamic range to make the image more natural. The third fused image is then adjusted based on the subject area and the non-subject area to make the subject area in the obtained target image more prominent and more natural.

[0239] In some exemplary embodiments, the target segmentation processing of the fused reference frame to obtain the main area and the non-main area includes: compressing and scaling the fused reference frame to obtain an adjusted fused reference frame; and target segmentation processing of the adjusted fused reference frame to obtain the main area and the non-main area.

[0240] Optionally, the graphics processor uses a compression and downscaling module to compress and downscale the fused reference frame in the 16-bit RGB domain, generating an adjusted fused reference frame in the 8-bit RGB domain. The neural network processor uses a multi-class segmentation module to segment the adjusted fused reference frame, generating subject and non-subject regions.

[0241] In some exemplary embodiments, fusing the main body area with the first fused image to obtain a second fused image includes: performing brightness adaptive segmentation processing on the first fused image, and then fusing the main body area with the adaptively segmented first fused image to obtain the second fused image.

[0242] The graphics processor calls the adaptive segmentation module (Adaptive multi EV split) to perform brightness adaptive segmentation processing on the first fused image step by step according to the brightness value to obtain HDR images with different brightness ranges. The required HDR image is selected from the HDR images with different brightness ranges as the first fused image after adaptive segmentation, and then the main area is fused with the first fused image after adaptive segmentation to obtain the second fused image.

[0243] In some exemplary embodiments, adjusting the third fused image according to the main area and the non-main area to obtain a target image includes: performing local contrast enhancement processing on the third fused image to obtain a locally enhanced third fused image; performing subject protection processing on the locally enhanced third fused image according to the main area and the non-main area to obtain a fourth fused image; and obtaining the target image based on the fourth fused image.

[0244] The graphics processor calls a local contrast enhancement (LCE) algorithm to perform local contrast enhancement processing on the third fused image to obtain a locally enhanced third fused image; performs subject protection (Portrait Protection) processing on the locally enhanced third fused image according to the subject area and the non-subject area to obtain a fourth fused image; and then performs color space conversion processing on the fourth fused image to obtain a target image.

[0245] In some exemplary embodiments, obtaining the target image based on the fourth fused image includes: performing a first color space conversion on the fourth fused image to obtain a fifth fused image; and performing a second color space conversion on the fifth fused image to obtain the target image.

[0246] The first color space conversion converts the RGB domain into the HSV (Hue, Saturation, Value) domain. The second color space conversion converts the HSV domain into the YUV domain. Optionally, the fourth fused image is an RGB domain image, the fifth fused image is an HSV domain image, and the target image is a YUV domain image. The first color space conversion and the second color space conversion processes can be implemented by a graphics processor.

[0247] In some exemplary embodiments, performing a first color space conversion on the fourth fused image to obtain a fifth fused image includes: performing a color correction matrix, a gamma transform, and a first color space conversion on the fourth fused image to obtain the fifth fused image.

[0248] The graphics processor invokes a color correction matrix (CCM) algorithm to perform a color correction on the fourth fused image, then invokes a gamma transform algorithm (Gamma) to perform a gamma transform on the fourth fused image after the color correction, and then performs an HSV color space conversion on the gamma-transformed fourth fused image to obtain a fifth fused image. Optionally, the fourth fused image before the gamma transform is a 16-bit RGB domain fused image, and the fourth fused image after the gamma transform is a 10-bit RGB domain fused image.

[0249] In some exemplary embodiments, performing a second color space conversion on the fifth fused image to obtain a target image includes: performing brightness adjustment processing and a second color space conversion on the fifth fused image to obtain the target image.

[0250] The graphics processor invokes a highlight adjustment algorithm to perform brightness adjustment processing and a second color space conversion on the fifth fused image to obtain a target image. The target image may be an image in a 10-bit YUV domain. The target image is input to the image signal processor.

[0251] In some exemplary embodiments, the step of performing high dynamic range fusion on the plurality of synthetic frames based on the fusion reference frame to obtain a first fused image includes: acquiring a motion mask image; and performing high dynamic range fusion on the motion mask image and the plurality of synthetic frames based on the fusion reference frame to obtain a first fused image.

[0252] The motion mask image and multiple synthetic frames are fused in high dynamic range, and the obtained first fused image eliminates ghosting, makes the image clearer, and is easy to fuse.

[0253] In some exemplary embodiments, acquiring the motion mask image includes: determining a first reference frame; and processing the first reference frame and an image frame to be processed having a brightness different from that of the first reference frame using an optical flow algorithm to obtain the motion mask image.

[0254] In some exemplary embodiments, the at least two groups of brightness image sets include at least two of a first brightness image set, a second brightness image set, and a third brightness image set, the first brightness image set includes multiple first image frames, the second brightness image set includes at least one second image frame, the third brightness image set includes at least one third image frame, the brightness of the first image frame is greater than the brightness of the second image frame, and the brightness of the second image frame is greater than the brightness of the third image frame.

[0255] Correspondingly, the determining the first reference frame includes: determining the first reference frame from multiple frames of the first image frames.

[0256] Correspondingly, the first reference frame and the image frame to be processed having a brightness different from that of the first reference frame are processed by an optical flow algorithm to obtain a motion mask map, including: processing the first reference frame, the second image frame and the third image frame by an optical flow algorithm to obtain a motion mask map.

[0257] In some exemplary embodiments, the image to be processed in the brightness image set is subjected to at least one of black level removal processing, dynamic range adjustment gain increase, lens shading correction, affine transformation, high dynamic range fusion, brightness adaptive segmentation, tone mapping, local contrast enhancement, subject protection processing, color correction matrix, gamma transformation, first color space conversion, brightness adjustment processing, and second color space conversion to obtain a target image, wherein the black level removal processing, dynamic range adjustment gain increase, lens shading correction, affine transformation, high dynamic range fusion, brightness adaptive segmentation, tone mapping, local contrast enhancement, subject protection processing, color correction matrix, gamma transformation, first color space conversion, brightness adjustment processing, and second color space conversion are implemented by a graphics processor.

[0258] It is understandable that the processing process of obtaining the target image is not implemented in the ISP. After obtaining the target image, the target image is input into the ISP. This avoids the loss of RAW data caused by the ISP, improves the clarity of the image, and the color and brightness adjustment are not dependent on the platform, reducing the linkage cost, and linear brightness, ghosting and fusion processing are easier.

[0259] In some exemplary embodiments, the image determination method and image processing method are applied to the system architecture of Figure 8. As shown in Figure 8, the system architecture includes an image sensor, a hybrid RAW algorithm (Hybrid Raw) module, and an image processing engine (IPE). The image sensor is used to collect RAW domain image data and transmit the RAW domain image data to the hybrid RAW algorithm module. The hybrid RAW algorithm module fuses the RAW domain image data using the above-mentioned image processing method to obtain a target image P10, and then transmits the target image data to the image processing engine.

[0260] In some exemplary embodiments, the image processing method is applied to the algorithm flow of Figure 9. As shown in Figure 9, the image processing method includes: obtaining a first image frame EV0, a ​​second image frame EV-, and a third image frame EV-- in the RAW domain, selecting a reference frame from the multiple frames EV0, and then performing Align processing, affine transformation Warp processing, and Deghost processing on the multiple frames EV0, EV-, and EV-- to the reference frame to obtain processed multiple frames EV0, EV-, and EV--, and performing optical flow calculation on EV0, EV-, and EV-- through an optical flow algorithm to obtain a motion mask, and determine a reference frame marker; performing AI processing on the multiple frames EV0, EV-, and EV-- The Demosaic network performs color interpolation and denoising to produce a composite frame EV0 in the RGB domain, a composite frame EV- in the RGB domain, and a composite frame EV-- in the RGB domain. A fusion reference frame base is determined from the three composite frames based on the reference frame markers. The fused reference frame is segmented to obtain the portrait and non-portrait regions (including the sky and background). Using the fused reference frame as a reference, high dynamic range fusion (HDR fusion) is performed on the composite frames EV0, EV-, EV-- in the RGB domain, the motion mask, and the portrait region to produce a high-bit RGB fused image. Tone mapping is performed on the high-bit RGB fused image, and then adjustments are made based on the portrait and non-portrait regions obtained from the target segmentation to produce the target image P010. P010 is a YUV domain image.

[0261] FIG10 is a detailed process of an image processing method in an embodiment. As shown in FIG10 , the image processing method includes:

[0262] (1) obtaining input image frames (Input Frames), wherein the input image frames include a plurality of first image frames EV0, at least one second image frame EV-, and at least one third image frame EV--;

[0263] (2) The CPU selects the best frame (Pick EV0 Best Frame) from the multiple frames EV0 as the reference frame EV0 (EV0base), and uses the reference frame as a benchmark to align the multiple frames EV0, at least one frame EV-, and at least one frame EV-- to obtain the aligned multiple frames EV0, at least one frame EV-, and at least one frame EV--. The Dense Inverse Search-based (DIS) algorithm is used to calculate the reference frame, one frame EV-, and one frame EV--, and remove artifacts to obtain a motion mask image.

[0264] (3) The CPU obtains the corresponding overexposed areas in EV0, EV- and EV-- through the reference frame mark determination module (Compose Deghost Base), compares the overexposed area of ​​EV0 with the motion mask image to obtain the first overlapping area, compares the overexposed area of ​​EV- with the motion mask image to obtain the second overlapping area, and compares the overexposed area of ​​EV-- with the motion mask image to obtain the third overlapping area; compares the areas of the first overlapping area, the second overlapping area and the third overlapping area to obtain the overlapping area with the smallest area, obtains the reference frame mark according to the image frame corresponding to the overlapping area with the smallest area, and transmits the reference frame mark to the affine transformation (RAW Warp) or remove black level (Remove Black Level).

[0265] (4) The GPU performs black level removal and noise reduction processing (Compute Noise Map) on the aligned multi-frame EV0, and then inputs them into the multi-frame interpolation network (MF AI Demosaic) for synthesis to obtain the synthetic frame EV0 in the RGB domain corresponding to the multi-frame EV0;

[0266] When EV- is multi-frame, the reference frame EV0base is used as the benchmark, and the multi-frame EV- is affine transformed to the reference frame EV0, and the multi-frame EV- after affine transformation is input into the multi-frame interpolation network (MF AI Demosaic) for synthesis to obtain the EV- of a synthetic frame in the RGB domain;

[0267] When EV-- is a multi-frame, the reference frame EV0base is used as the benchmark, and the multi-frame EV-- is affine transformed to the reference frame EV0, and the multi-frame EV-- after affine transformation is input into the multi-frame interpolation network (MF AI Demosaic) for synthesis to obtain the EV-- of a synthetic frame in the RGB domain;

[0268] Among them, the image input to the multi-frame interpolation network is a 10-bit image in the RAW domain, and the image output after processing by the multi-frame interpolation network is a 16-bit image in the RGB domain.

[0269] (5) The GPU removes black level, applies dynamic range correction gain, performs lens shading correction (LSC) and automatic white balance (AWB) on the composite frame output by the multi-frame interpolation network to obtain the adjusted composite frame. The fusion reference frame (HDR Base Frame) is determined from the adjusted composite frame according to the reference frame mark, and the fusion reference frame and the other two composite frames are input into the RGB domain affine transformation (RGB Warp to HDR base). With the fusion reference frame as the reference, the other two composite frames are affine transformed to the fusion reference frame to obtain the other two fusion frames after affine transformation. The fusion reference frame and the other two composite frames after affine transformation are then input into the HDR Fusion Linear for high dynamic range fusion processing.

[0270] For example, taking the adjusted synthetic frame EV0 as the fusion reference frame, the adjusted synthetic frame EV0, synthetic frame EV- and synthetic frame EV-- are input into the affine transformation of the RGB domain, and the adjusted EV- and adjusted synthetic frame EV-- are affine transformed to the fusion reference frame EV0.

[0271] (6) The GPU calls the Companding & Down Scale module to compress and downscale the 16-bit RGB domain fusion reference frame to obtain the adjusted 8-bit RGB domain fusion reference frame, and transmits the adjusted fusion reference frame to the NPU. The NPU calls the Multi-class Segmentation module to perform target segmentation on the adjusted fusion reference frame to obtain the subject area and the non-subject area, and transmits the subject area to the Extended Exposure Fusion module, and transmits the subject area and the non-subject area to the Portrait Protection module for processing.

[0272] (7) The GPU calls the HDR Fusion Linear module to perform high dynamic range fusion processing on the motion mask image, the fusion reference frame and the other two synthetic frames to obtain a first fused image, calls the adaptive segmentation module (Adaptive multi Ev split) to perform brightness adaptive segmentation processing, and then calls the long exposure fusion module to fuse the subject area with the first fused image after adaptive segmentation to obtain a second fused image; the GPU continues to call global tone mapping (GTM) and local tone mapping (LTM) to perform tone mapping on the second fused image to obtain a third fused image, and then performs local contrast enhancement (LCE) on the third fused image to obtain a locally enhanced third fused image; performs portrait protection processing on the locally enhanced third fused image according to the subject area and the non-subject area to obtain a fourth fused image, and then performs color correction matrix (CCM), gamma transformation (Gamma) (converting the 16-bit RGB domain image to the 10-bit RGB domain image), first color space transformation (Color Space Transform), brightness adjustment algorithm (Highlight adjustment) and second color space transformation (Color Space Transform) on the fourth fused image. The target image is obtained by performing a color space transform (output P010), which is then transmitted to the ISP for processing. The first color space conversion converts the RGB domain to the HSV (Hue, Saturation, Value) domain. The second color space conversion converts the HSV domain to the YUV domain.

[0273] (8) When EV0 is multiple frames and EV- or EV-- is one frame, first select one frame EV0 from the multiple frames EV0 as the reference frame, then calculate the motion mask map and the reference frame mark according to the reference frame, EV- and EV--. When the reference frame mark is EV- or EV--, the reference frame mark determination module (Compose Deghost Base) transmits the reference frame mark to the single frame interpolation network; the multiple frames EV0 go through the multi-frame interpolation network to obtain the synthetic frame EV0, and perform black level removal processing (remove black level), dynamic range adjustment gain (Apply Dynamic Range Correction gain), shadow correction (LSC) and white balance (AWB) processing to obtain the adjusted synthetic frame EV0, and input the adjusted synthetic frame EV0 into the affine transformation (RGB Warp) of the RGB domain; for a single frame EV- or EV--, the GPU performs black level removal processing (remove black level) on the frame to be processed. The GPU performs black level removal, noise reduction, shading correction (LSC), white balance (AWB) and gamma transformation (gamma) on the frame of image to be processed to obtain the adjusted image to be processed, and inputs the adjusted image to be processed into the single-frame interpolation network. The NPU calls the single-frame network to perform color interpolation on the adjusted image to be processed EV- to obtain a synthesized frame EV-, and performs color interpolation on the adjusted image to be processed EV-- to obtain a synthesized frame EV--; performs inverse gamma transformation on the synthesized frame obtained by the single-frame interpolation network to obtain an adjusted synthesized frame, and transfers the adjusted synthesized frames EV- and EV-- to the affine transformation (RGB Warp) in the RGB domain. When the reference frame mark indicates that the synthetic frame EV0 is used as the fusion reference frame, the synthetic frame EV0 is input into the Companding&down scale for compression and scaling. When the reference frame mark indicates that the synthetic frame EV- or EV-- obtained by the single-frame network is used as the fusion reference frame, the single-frame interpolation network inputs the obtained synthetic frame EV- or EV-- into the Companding&down scale for compression and scaling, and then processes it according to the description in (6) and (7).

[0274] The image processing method of this embodiment is compatible with the image quality and high dynamic scenes from single frame to multi-frame, as well as the snapshot function, and covers the basic image quality and dynamic range optimization of the entire focal length and the entire scene. It uses multi-frame RAW images as input, and has richer available information, high bit width and can ensure high dynamic range. It can support multi-frame, super-resolution, and HDR tasks, and can perform linear brightness adjustment, ghosting and fusion more easily. It combines target segmentation to achieve semantic segmentation, realizes adjustment for specific areas, supports quick shooting, and the processing process is not processed by the ISP, reducing the loss of clarity in the ISP process. There is no jump in the boundary scene switching between multiple algorithm adaptation scenes. The algorithm processing process does not depend on the ISP platform, which is convenient for subsequent flexible modification and adaptation to personalized needs.

[0275] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0276] Based on the same inventive concept, embodiments of the present application further provide an image determination device for implementing the aforementioned image determination method, and an image processing device for implementing the aforementioned image processing method. The implementation solutions provided by these devices are similar to the implementation solutions described in the aforementioned methods. Therefore, the specific limitations in the embodiments of one or more image determination devices and image processing devices provided below can be found in the above-mentioned limitations on the image determination method and image processing method, and will not be further elaborated here.

[0277] FIG11 is a block diagram of an image determination device according to an embodiment. As shown in FIG11 , an image determination device includes a type determination module 1102, an information acquisition module 1104, and an image frame determination module 1106.

[0278] The type determination module 1102 is used to determine the type of the image sensor; the image sensor type is one that supports digital overlap wide dynamic range or does not support digital overlap wide dynamic range.

[0279] The information acquisition module 1104 is used to acquire ambient brightness and motion status information.

[0280] The image frame determination module 1106 is configured to determine, in response to a shooting operation, an image frame for fusion according to the image sensor type, the ambient brightness, and the motion state information.

[0281] In some exemplary embodiments, the image frame determination module 1106 is further used to determine, based on the motion state information, that the image set for fusion includes at least one image pair data before the shooting operation if the image sensor type supports digital overlap wide dynamic range and the ambient brightness is greater than a first brightness threshold.

[0282] In some exemplary embodiments, the image frame determination module 1106 is further configured to, if the image sensor type supports a digital overlap wide dynamic range, the ambient brightness is greater than a first brightness threshold, and the motion state information is a local motion state or a stationary state, determine that the image set for fusion includes at least one first image pair data before the shooting operation, the first image pair data including data of a first baseline exposure frame and a first exposure brightness frame; the brightness of the first baseline exposure frame is greater than the brightness of the first exposure brightness frame;

[0283] If the image sensor type supports digital overlap wide dynamic range, the ambient brightness is greater than the first brightness threshold, and the motion state information is a global motion state, then it is determined that the image set for fusion includes at least one second image pair data before the shooting operation, and the second image pair data includes data of a first exposure duration frame and a second exposure brightness frame; the first exposure duration frame has the same brightness as the first reference exposure frame, and the exposure duration of the first exposure duration frame is less than the first reference exposure frame; the brightness of the first exposure duration frame is greater than the brightness of the second exposure brightness frame.

[0284] In some exemplary embodiments, the number of the first reference exposure frames and the number of the first exposure duration frames are both positively correlated with brightness gain.

[0285] In some exemplary embodiments, the image determination apparatus further includes a shooting mode acquisition module, configured to acquire a shooting mode during the shooting operation.

[0286] The image frame determination module 1106 is further configured to determine an image set for fusion according to the image sensor type, the ambient brightness, the shooting mode, and the motion state information.

[0287] In some exemplary embodiments, the image frame determination module 1106 is further used to determine, based on the motion state information, that the image set for fusion includes at least one image pair data before the shooting operation and at least one image pair data after the shooting operation if the image sensor type supports digital overlapping wide dynamic range, the ambient brightness is greater than a first brightness threshold and the shooting mode is a high dynamic range mode.

[0288] The one image pair data is one frame of normal exposure duration data, and the brightness of the two frames of images in the one image pair data is different.

[0289] In some exemplary embodiments, the image frame determination module 1106 is further configured to, if the image sensor type supports digital overlap wide dynamic range, the ambient brightness is greater than a first brightness threshold, the shooting mode is a high dynamic range mode, and the motion state information is a local motion state, determine the image set for fusion including at least one of at least one first image pair data before the shooting operation and at least one third image pair data and at least one fourth image pair data after the shooting operation; the first image pair data including data of a first baseline exposure frame and a first exposure brightness frame; the brightness of the first baseline exposure frame is greater than the brightness of the first exposure brightness frame; the third image pair data including data of a second exposure duration frame and a third exposure brightness frame; the second exposure duration frame having the same brightness as the first baseline exposure frame, and the exposure duration of the second exposure duration frame being less than the exposure duration of the first baseline exposure frame; the brightness of the third exposure brightness frame being less than the brightness of the first exposure brightness frame; the fourth image pair data including a third exposure brightness frame and a third exposure duration frame; the third exposure duration frame having the same brightness as the first baseline exposure frame, and the exposure duration of the third exposure duration frame being greater than the exposure duration of the first baseline exposure frame;

[0290] If the image sensor type supports digital overlap wide dynamic range, the ambient brightness is greater than a first brightness threshold, the shooting mode is a high dynamic range mode, and the motion state information is a static state, then the image set determined for fusion includes at least one first image pair data before the shooting operation and at least one fourth image pair data after the shooting operation; the first image pair data includes data of a first baseline exposure frame and a first exposure brightness frame; the brightness of the first baseline exposure frame is greater than the brightness of the first exposure brightness frame; the fourth image pair data includes a third exposure brightness frame and a third exposure duration frame; the third exposure duration frame has the same brightness as the first baseline exposure frame, and the exposure duration of the third exposure duration frame is greater than the exposure duration of the first baseline exposure frame;

[0291] If the image sensor type supports digital overlap wide dynamic range, the ambient brightness is greater than the first brightness threshold, the shooting mode is high dynamic range mode and the motion state information is global motion state, then the image set determined for fusion includes at least one second image pair data before the shooting operation and at least one fifth image pair data after the shooting operation; the second image pair data includes data of a first exposure duration frame and a second exposure brightness frame; the fifth image pair data includes data of a fourth exposure brightness frame and a fourth exposure duration frame; the first exposure duration frame has the same brightness as the first baseline exposure frame, and the exposure duration of the first exposure duration frame is less than the first baseline exposure frame; the brightness of the first exposure duration frame is greater than the brightness of the second exposure brightness frame; the brightness of the second exposure brightness frame is greater than the brightness of the fourth exposure brightness frame; the brightness of the fourth exposure duration frame is the same as the brightness of the first exposure duration frame, and the exposure duration of the fourth exposure duration frame is greater than the exposure duration of the first exposure duration frame.

[0292] In some exemplary embodiments, the image frame determination module 1106 is further used to determine, based on the motion state information, that the image set for fusion includes at least one image pair data before the shooting operation if the image sensor type supports digital overlapping wide dynamic range, the ambient brightness is greater than a first brightness threshold, and the shooting mode is a non-high dynamic range mode.

[0293] In some exemplary embodiments, if the image sensor type supports digital overlap wide dynamic range, the ambient brightness is greater than a first brightness threshold, the shooting mode is a non-high dynamic range mode, and the motion state information is a global motion state, the image set determined for fusion includes at least one second image pair data before the shooting operation; the second image pair data includes data of a first exposure duration frame and a second exposure brightness frame; the first exposure duration frame has the same brightness as the first reference exposure frame, and the exposure duration of the first exposure duration frame is less than the first reference exposure frame; the brightness of the first exposure duration frame is greater than the brightness of the second exposure brightness frame;

[0294] If the image sensor type supports digital overlap wide dynamic range, the ambient brightness is greater than a first brightness threshold, the shooting mode is a non-high dynamic range mode, and the motion state information is a local motion state, then the image set determined for fusion includes at least one first image pair data before the shooting operation and at least one frame of a second exposure time length after the shooting operation; the first image pair data includes data of a first baseline exposure frame and a first exposure brightness frame; the brightness of the first baseline exposure frame is greater than the brightness of the first exposure brightness frame;

[0295] If the image sensor type supports digital overlapping wide dynamic range, the ambient brightness is greater than a first brightness threshold, the shooting mode is a non-high dynamic range mode, and the motion state information is a static state, then the image set determined for fusion includes at least one first image pair data before the shooting operation.

[0296] In some exemplary embodiments, if the image sensor type supports digital overlap wide dynamic range and the ambient brightness is not greater than a first brightness threshold, the image set for fusion is determined based on the motion state information to include an image sequence after the shooting operation, and the image sequence includes image frames with different brightness.

[0297] In some exemplary embodiments, if the image sensor type supports digital overlap wide dynamic range, the ambient brightness is not greater than a first brightness threshold, and the motion state information indicates a stationary state, then determining that the image set for fusion includes a first image sequence after the shooting operation, the first image sequence includes a second baseline exposure frame, a fifth exposure brightness frame, and a sixth exposure brightness frame after the shooting operation, the second baseline exposure frame being a plurality of frames; and the brightness of the second baseline exposure frame, the fifth exposure brightness frame, and the sixth exposure brightness frame decreases in sequence;

[0298] If the image sensor type supports digital overlap wide dynamic range, the ambient brightness is not greater than the first brightness threshold, and the motion state information is a motion state, then it is determined that the image set for fusion includes a second image sequence after the shooting operation, the second image sequence includes a fifth exposure brightness frame and a sixth exposure brightness frame after the shooting operation, and the second image sequence also includes at least one of a second base exposure frame and a fifth exposure duration frame, and the second base exposure frame and the fifth exposure duration frame are multiple frames; the brightness of the second base exposure frame, the fifth exposure brightness frame, and the sixth exposure brightness frame decreases successively; the fifth exposure duration frame has the same brightness as the second base exposure frame, and the exposure duration of the fifth exposure duration frame is less than the exposure duration of the second base exposure frame.

[0299] In some exemplary embodiments, the image frame determination module 1106 is further used to determine, based on the motion state information, that the image set for fusion includes at least one image frame before the shooting operation if the image sensor type does not support digital overlapping wide dynamic range and the ambient brightness is greater than a second brightness threshold.

[0300] In some exemplary embodiments, the shooting mode acquisition module is used to acquire the shooting mode during the shooting operation;

[0301] In some exemplary embodiments, the image frame determination module 1106 is further used to determine, based on the motion state information and the shooting mode, that the image set for fusion includes at least one image frame before the shooting operation and an exposure frame set after the shooting operation if the image sensor type does not support digital overlapping wide dynamic range and the ambient brightness is greater than a second brightness threshold.

[0302] In some exemplary embodiments, the image determination apparatus further includes a sensitivity acquisition module and a quantity determination module. The sensitivity acquisition module is configured to acquire the sensitivity during the shooting operation; and the quantity determination module is configured to determine the number of image frames before the shooting operation based on a relationship between the sensitivity and a sensitivity threshold.

[0303] The quantity determination module is also used to determine that the number of image frames before the shooting operation is a first number if the sensitivity is greater than or equal to the sensitivity threshold; if the sensitivity is less than the sensitivity threshold, the number of image frames before the shooting operation is a second number; the first number is greater than the second number.

[0304] In some exemplary embodiments, if the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, and the motion state information is a stationary state or a local motion state, it is determined that the image set for fusion includes at least one first exposure gain frame before the shooting operation, and the brightness of the first exposure gain frame is less than the brightness of a third reference exposure frame;

[0305] If the image sensor type does not support digital overlapping wide dynamic range, the ambient brightness is greater than the second brightness threshold and the motion state information is a global motion state, then it is determined that the image set for fusion includes at least one second exposure gain frame before the shooting operation, and the exposure duration of the second exposure gain frame is less than the exposure duration of the first exposure gain frame.

[0306] In some exemplary embodiments, if the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, and the motion state information is a static state, the exposure frame set is a first exposure frame set, and the first exposure frame set includes a third reference exposure frame;

[0307] If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, and the motion state information is a local motion state, the exposure frame set is a second exposure frame set, the second exposure frame set includes a sixth exposure duration frame, and the exposure duration of the sixth exposure duration frame is less than the exposure duration of the third reference exposure frame;

[0308] If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than the second brightness threshold, and the motion state information is a global motion state, then the exposure frame set is a third exposure frame set, and the third exposure frame set includes a seventh exposure duration frame, and the exposure duration of the seventh exposure duration frame is less than the exposure duration of the third benchmark exposure frame.

[0309] In some exemplary embodiments, if the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, the motion state information indicates a stationary state, and the shooting mode is a high dynamic range mode, then the first exposure frame set includes a third base exposure frame and a seventh exposure brightness frame;

[0310] If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, the motion state information is a stationary state, and the shooting mode is a non-high dynamic range mode, then the first exposure frame set includes a third reference exposure frame;

[0311] If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, the motion state information is a local motion state, and the shooting mode is a high dynamic range mode, the second exposure frame set includes a sixth exposure duration frame and an eighth exposure brightness frame;

[0312] If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, the motion state information is a local motion state, and the shooting mode is a non-high dynamic range mode, the second exposure frame set includes a sixth exposure duration frame;

[0313] If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, the motion state information is a global motion state, and the shooting mode is a high dynamic range mode, the third exposure frame set includes a seventh exposure duration frame and a ninth exposure brightness frame;

[0314] If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than the second brightness threshold, the motion state information is the global motion state, and the shooting mode is a non-high dynamic range mode, then the third exposure frame set includes a seventh exposure duration frame.

[0315] In some exemplary embodiments, if the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, the motion state information indicates a stationary state, the shooting mode is a high dynamic range mode, and the minimum exposure value is less than an exposure value threshold, then the first exposure frame set includes a third base exposure frame, a seventh exposure brightness frame, and a tenth exposure brightness frame, and the brightness of the tenth exposure brightness frame is greater than the brightness of the seventh exposure brightness frame;

[0316] If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than the second brightness threshold, the motion state information indicates a stationary state, the shooting mode is a high dynamic range mode, and the minimum exposure value is greater than or equal to the exposure value threshold, then the first exposure frame set includes a third reference exposure frame and a seventh exposure brightness frame;

[0317] If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, the motion state information is a local motion state, the shooting mode is a high dynamic range mode, and the minimum exposure value is less than the exposure value threshold, then the second exposure frame set includes a sixth exposure duration frame, an eighth exposure brightness frame, and an eleventh exposure brightness frame; the brightness of the eleventh exposure brightness frame is greater than the brightness of the eighth exposure brightness frame;

[0318] If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than the second brightness threshold, the motion state information is a local motion state, the shooting mode is a high dynamic range mode, and the minimum exposure value is greater than or equal to the exposure value threshold, then the second exposure frame set includes a sixth exposure duration frame and an eighth exposure brightness frame;

[0319] If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than the second brightness threshold, the motion state information is a global motion state, the shooting mode is a high dynamic range mode, and the minimum exposure value is less than the exposure value threshold, then the third exposure frame set includes a seventh exposure duration frame, a ninth exposure brightness frame, and a twelfth exposure brightness frame; the brightness of the twelfth exposure brightness frame is greater than the brightness of the ninth exposure brightness frame;

[0320] If the image sensor type does not support digital overlapping wide dynamic range, the ambient brightness is greater than the second brightness threshold, the motion state information is a global motion state, the shooting mode is a high dynamic range mode and the minimum exposure value is greater than or equal to the exposure value threshold, then the third exposure frame set includes a seventh exposure duration frame and a ninth exposure brightness frame.

[0321] In some exemplary embodiments, exposure parameters of the seventh exposure brightness frame, the eighth exposure brightness frame, the ninth exposure brightness frame, the tenth exposure brightness frame, the eleventh exposure brightness frame, and the twelfth exposure brightness frame are adjustable.

[0322] In some exemplary embodiments, if the image sensor type does not support digital overlap wide dynamic range and the ambient brightness is not greater than a second brightness threshold, the image set for fusion is determined based on the motion state information to include a sequence of exposure frames after the shooting operation, and the exposure frame sequence includes exposure frames of different brightness.

[0323] In some exemplary embodiments, if the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is not greater than a second brightness threshold, and the motion state information indicates a stationary state, then determining that the image set for fusion includes a first exposure frame sequence after the shooting operation, the first exposure frame sequence including a fourth base exposure frame, a thirteenth exposure brightness frame, and a fourteenth exposure brightness frame after the shooting operation, the fourth base exposure frame being a plurality of frames; the brightness of the fourth base exposure frame, the thirteenth exposure brightness frame, and the fourteenth exposure brightness frame decreases in sequence;

[0324] If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is not greater than the second brightness threshold, and the motion state information is a motion state, then it is determined that the image set for fusion includes a second exposure frame sequence after the shooting operation, the second exposure frame sequence includes the fifteenth exposure brightness frame and the sixteenth exposure brightness frame after the shooting operation, and the second exposure frame sequence also includes at least one of a fourth base exposure frame and an eighth exposure time frame, and the fourth base exposure frame and the eighth exposure time frame are multiple frames; the brightness of the fourth base exposure frame, the fifteenth exposure brightness frame, and the sixteenth exposure brightness frame decreases successively; the eighth exposure time frame has the same brightness as the fourth base exposure frame, and the exposure time of the eighth exposure time frame is less than the exposure time of the fourth base exposure frame.

[0325] In some exemplary embodiments, the type determination module 1102 is further used to obtain a zoom factor, determine a target image sensor based on the relationship between the zoom factor and the zoom factor switching threshold, and determine whether the target image sensor supports a digital overlapping wide dynamic range; if the target image sensor supports a digital overlapping wide dynamic range, determine the image sensor type as supporting a digital overlapping wide dynamic range; if the target image sensor supports a non-digital overlapping wide dynamic range, determine the image sensor type as supporting a non-digital overlapping wide dynamic range.

[0326] In some exemplary embodiments, the information acquisition module 1104 is also used to obtain gyroscope data during the shooting operation, as well as the brightness difference statistics of corresponding pixel points in the two frames of images captured by the shooting operation, and determine the motion state information based on the gyroscope data and the brightness difference statistics.

[0327] In some exemplary embodiments, the information acquisition module 1104 is also used to determine that the motion state information is a static state if the gyroscope data is in a first motion threshold range and the brightness difference statistics are in the first brightness difference threshold range; if the gyroscope data is in a second motion threshold range and the brightness difference statistics are in the second brightness difference threshold range, perform main area motion detection; if the gyroscope data is in a third motion threshold range and the brightness difference statistics are in the third brightness difference threshold range, determine that the motion state information is a global motion state; wherein the upper limit value of the first motion threshold range is less than the lower limit value of the second motion threshold range, and the upper limit value of the second motion threshold range is less than the lower limit value of the third motion threshold range; the upper limit value of the first brightness difference threshold range is less than the lower limit value of the second brightness difference threshold range, and the upper limit value of the second brightness difference threshold range is less than the lower limit value of the third brightness difference threshold range.

[0328] In some exemplary embodiments, the information acquisition module 1104 is further used to obtain the brightness difference statistics of the pixel points corresponding to the main area in the two frames of images; if the brightness difference statistics corresponding to the main area is greater than or equal to the brightness difference threshold, the motion state information is determined to be a global motion state; if the brightness difference statistics corresponding to the main area is less than the brightness difference threshold, the motion state information is determined to be a local motion state.

[0329] In some exemplary embodiments, in the case of a global motion state, the exposure duration is controlled to be reduced by N times, and in the case of a local motion state, the exposure duration is controlled to be reduced by N / 2 times, where N is a positive integer.

[0330] In some exemplary embodiments, in the case of a global motion state, the exposure duration is reduced by N times, and the corresponding brightness gain is increased by N times; in the case of a local motion state, the exposure duration is controlled to be reduced by N / 2 times, and the corresponding brightness gain is increased by N / 2 times.

[0331] FIG12 is a block diagram of the structure of an image processing device in one embodiment. As shown in FIG12 , the device includes an image acquisition module 1202, a synthesis module 1204, a reference frame determination module 1206, and a fusion module 1208.

[0332] The image acquisition module 1202 is configured to acquire at least two sets of brightness image sets with different brightness.

[0333] The synthesis module 1204 is configured to synthesize, for each luminance image set, the image frames to be processed in the luminance image set to obtain a corresponding synthesized frame.

[0334] The reference frame determining module 1206 is configured to determine a fusion reference frame from the plurality of synthesized frames.

[0335] The fusion module 1208 is used to fuse the multiple synthesized frames based on the fusion reference frame to obtain a target image, which is input into an image signal processor.

[0336] In this embodiment, at least two groups of brightness image sets are obtained. For each group of brightness image sets, the individual image frames to be processed in each group of brightness image sets are first synthesized to obtain a corresponding synthesized frame. Then, a fusion reference frame is determined from the synthesized frames. Based on the fusion reference frame, multiple synthesized frames are fused to obtain a target image, thereby generating a more accurate and clearer image.

[0337] FIG13 is a block diagram of the structure of an image processing apparatus in another embodiment. As shown in FIG13 , the apparatus further includes a reference frame determination module 1210 and a marker determination module 1212 .

[0338] The reference frame determining module 1210 is configured to determine a first reference frame.

[0339] The tag determination module 1212 is configured to determine a reference frame tag according to the first reference frame and an image frame to be processed having a brightness different from that of the first reference frame, wherein the reference frame tag is used to characterize a fused reference frame of high dynamic range fusion.

[0340] The reference frame determining module 1206 is further configured to determine a fusion reference frame from the plurality of synthesized frames according to the reference frame tag.

[0341] In some exemplary embodiments, the at least two groups of brightness image sets include at least two of a first brightness image set, a second brightness image set, and a third brightness image set, the first brightness image set includes multiple first image frames, the second brightness image set includes at least one second image frame, the third brightness image set includes at least one third image frame, the brightness of the first image frame is greater than the brightness of the second image frame, and the brightness of the second image frame is greater than the brightness of the third image frame.

[0342] Correspondingly, the reference frame determining module 1210 is further configured to determine a first reference frame from the plurality of first image frames.

[0343] The label determination module 1212 is further configured to determine a reference frame label according to the first reference frame, the second image frame, and the third image frame, where the reference frame label is used to characterize a fused reference frame of high dynamic range fusion.

[0344] The reference frame determination module 1210 and the marker determination module 1212 are called by a central processing unit.

[0345] In some exemplary embodiments, the device also includes: at least one of an alignment module, an affine transformation module and a de-artifact module called by a central processing unit; the alignment module is used to align the images in the at least two groups of brightness image sets; the affine transformation module is used to perform affine transformation on the images in the at least two groups of brightness image sets; and the de-artifact module is used to perform de-artifact on the images in the at least two groups of brightness image sets.

[0346] In some exemplary embodiments, the image acquisition module 1202 is also used to acquire multiple image pair data, each image pair data includes a first image frame and a second image frame, determine a first reference frame from the multiple first image frames, acquire the second image frame in the image pair data where the first reference frame is located, and use it as the second reference frame; align the remaining first image frames except the first reference frame with the first reference frame, and obtain a first brightness image set based on the first reference frame and the aligned first image frame; and align the remaining second image frames except the second reference frame with the second reference frame, and obtain a second brightness image set based on the second reference frame and the aligned second image frame.

[0347] In some embodiments, the synthesis module 1204 is further configured to perform noise reduction and color interpolation processing on each to-be-processed image frame in each luminance image set through an interpolation network to obtain a corresponding synthesized frame.

[0348] In some embodiments, the synthesis module 1204 includes a single-frame interpolation network module and a multi-frame interpolation network module.

[0349] The single-frame interpolation network module is called by the neural network processor and is used to perform noise reduction and color interpolation processing on each image frame to be processed in the brightness image set through the single-frame interpolation network if there is one image frame to be processed in the brightness image set to obtain a corresponding synthetic frame.

[0350] The multi-frame interpolation network module is called by the neural network processor and is used to perform noise reduction and color interpolation processing on each of the image frames to be processed in the brightness image set through the multi-frame interpolation network if there are multiple frames in the brightness image set to obtain corresponding synthetic frames.

[0351] In some embodiments, as shown in FIG14 , the fusion module 1208 includes:

[0352] The object segmentation module 1402 is configured to perform object segmentation processing on the fused reference frame to obtain a main body region and a non-main body region.

[0353] The high dynamic range fusion module 1404 is configured to perform high dynamic range fusion on the plurality of synthesized frames based on the fusion reference frame to obtain a first fused image.

[0354] The enhanced fusion module 1406 is configured to fuse the subject area with the first fused image to obtain a second fused image.

[0355] The tone mapping module 1408 is configured to perform tone mapping on the second fused image to obtain a third fused image.

[0356] The subject protection module 1410 is configured to adjust the third fused image according to the subject area and the non-subject area to obtain a target image.

[0357] In some embodiments, the tone mapping module 1408 may include at least one of a global tone mapping module and a local tone mapping module. The global tone mapping module is configured to perform global tone mapping on the second fused image. The local tone mapping module is configured to perform local tone mapping on the second fused image.

[0358] In some embodiments, the target segmentation module 1402 is called by the neural network processor; the reference frame determination module 1206, the high dynamic range fusion module 1404, the enhanced fusion module 1406, the tone mapping module 1408 and the subject protection module 1410 are called by the image processor.

[0359] In some embodiments, as shown in FIG14 , the fusion module 1208 further includes a compression and scaling module 1401 for compressing and scaling the fused reference frame to obtain an adjusted fused reference frame. An object segmentation module 1402 is further configured to segment the adjusted fused reference frame to obtain a subject region and a non-subject region.

[0360] The compression and scaling module 1401 is called by the graphics processor.

[0361] In some embodiments, as shown in FIG14 , the fusion module 1208 further includes an adaptive segmentation module 1405 , which is called by the graphics processor and is configured to perform brightness adaptive segmentation on the first fused image. The enhanced fusion module 1406 is further configured to fuse the subject region with the adaptively segmented first fused image to obtain a second fused image.

[0362] In some embodiments, as shown in FIG14 , the fusion module 1208 further includes:

[0363] The local contrast enhancement module 1409 is used to perform local contrast enhancement processing on the third fused image to obtain a locally enhanced third fused image. The subject protection module 1410 is also used to perform subject protection processing on the locally enhanced third fused image according to the subject area and the non-subject area to obtain a fourth fused image. The color correction module 1412 is used to perform color correction matrix processing on the fourth fused image. The first gamma transformation module 1414 is used to perform gamma transformation processing on the fourth fused image after color correction. The first color space conversion module 1416 is used to perform a first color space conversion on the fourth fused image after gamma transformation processing to obtain a fifth fused image. The brightness adjustment module 1418 is used to perform brightness adjustment processing on the fifth fused image. The second color space conversion module 1420 is used to perform a second color space conversion on the fifth fused image after brightness adjustment to obtain a target image.

[0364] In some embodiments, the local contrast enhancement module, the color correction module 1412 , the first gamma conversion module 1414 , the first color space conversion module 1416 , the brightness adjustment module 1418 , and the second color space conversion module 1420 are all called by the graphics processor.

[0365] In some embodiments, the fourth fused image is an RGB domain image, the fifth fused image is an HSV domain image, and the target image is a YUV domain image.

[0366] In some embodiments, the at least two groups of luminance image sets include at least two of a first luminance image set, a second luminance image set, and a third luminance image set, the first luminance image set includes multiple first image frames, the second luminance image set includes at least one second image frame, and the third luminance image set includes at least one third image frame, the luminance of the first image frame is greater than the luminance of the second image frame, and the luminance of the second image frame is greater than the luminance of the third image frame; the apparatus further includes a reference frame determination module 1210 and a motion mask determination module. The reference frame determination module 1210 is configured to determine a first reference frame from the multiple first image frames. The motion mask determination module is configured to process the first reference frame, the second image frame, and the third image frame using an optical flow algorithm to obtain a motion mask map.

[0367] The marking determination module 1212 is also used to obtain the overlapping areas of the over-exposed areas corresponding to the first reference frame, the second image frame and the third image frame and the motion mask image, and determine the reference frame mark according to the respective overlapping areas; if the reference frame mark is the first mark, it means that the synthetic frame corresponding to the first image frame is used as the fusion reference frame for high dynamic range fusion; if the reference frame mark is the second mark, it means that the synthetic frame corresponding to the second image frame is used as the fusion reference frame for high dynamic range fusion; if the reference frame mark is the third mark, it means that the synthetic frame corresponding to the third image frame is used as the fusion reference frame for high dynamic range fusion.

[0368] In some exemplary embodiments, the marker determination module 1212 is further configured to determine overexposed regions of each of the three image frames based on pixels whose brightness values ​​are greater than a brightness threshold. Furthermore, the electronic device compares the overexposed region of the first reference frame with the motion mask image to determine a first overlapping region, compares the overexposed region of the second image frame with the motion mask image to determine a second overlapping region, and compares the overexposed region of the third image frame with the motion mask image to determine a third overlapping region. The electronic device compares the areas of the first overlapping region, the second overlapping region, and the third overlapping region to determine the overlapping region with the smallest area, and determines the reference frame marker based on the image frame corresponding to the overlapping region with the smallest area. For example, if the overlapping region with the smallest area is the first overlapping region, the reference frame marker is determined as the first marker based on the first reference frame corresponding to the first overlapping region; if the overlapping region with the smallest area is the second overlapping region, the reference frame marker is determined as the second marker based on the second image frame corresponding to the second overlapping region; and if the overlapping region with the smallest area is the third overlapping region, the reference frame marker is determined as the third marker based on the third image frame corresponding to the third overlapping region.

[0369] The high dynamic range fusion module 1404 is further configured to perform high dynamic range fusion on the motion mask image and multiple synthesized frames based on the fusion reference frame to obtain a first fused image.

[0370] In some embodiments, as shown in FIG15 , the image processing apparatus further includes at least one of the following modules: a first black level removal processing module 1502 , a dynamic range enhancement module 1504 , a first shading correction module 1506 , and a first white balance module 1508 .

[0371] The first black level removal processing module 1502 is configured to perform black level removal processing on the synthesized frame obtained through the multi-frame interpolation network processing.

[0372] The dynamic range enhancement module 1504 is configured to perform a dynamic range adjustment gain process on the synthesized frame after the black level process is removed.

[0373] The first shadow correction module 1506 is configured to perform shadow correction processing on the synthesized frame after gain adjustment.

[0374] The first white balance module 1508 is configured to perform white balance processing on the composite frame after shadow correction to obtain an adjusted composite frame.

[0375] The first black level removal processing module 1502 , the dynamic range enhancement module 1504 , the first shadow correction module 1506 , and the first white balance module 1508 are called by a graphics processor.

[0376] In some embodiments, as shown in FIG15 , the apparatus further includes an inverse gamma transform module 1510. The inverse gamma transform module 1510 is called by the graphics processor to perform inverse gamma transform processing on the synthesized frame obtained by the single-frame interpolation network processing to obtain an adjusted synthesized frame.

[0377] In some embodiments, as shown in FIG15 , the apparatus further includes: a second black level removal processing module 1512 for performing black level removal processing on the image to be processed if the image to be processed in the luminance image set is one frame. A second shadow correction module 1514 for performing shadow correction processing on the image to be processed after the black level removal processing. A second white balance module 1516 for performing white balance processing on the image to be processed after the shadow correction processing. A second gamma transform processing module 1518 for performing gamma transform processing on the image to be processed after the white balance processing to obtain an adjusted image to be processed, and the adjusted image to be processed is used to input the single-frame interpolation network to obtain a synthesized frame. The second black level removal processing module 1514, the second shadow correction module 1516, the second white balance module 1516, and the second gamma transform 1518 are called by the graphics processor.

[0378] In some embodiments, as shown in FIG15 , the apparatus further includes a first noise reduction module 1515. The first noise reduction module 1515 is called by the graphics processor to perform noise reduction on the image to be processed after the black level removal process.

[0379] In some embodiments, the image processing device also includes an RGB domain affine transformation module, and the RGB domain affine transformation module is used to perform RGB domain affine transformation processing on the adjusted synthetic frame to the fusion reference frame to obtain the synthetic frame after affine transformation processing, and then transmit the fusion reference frame and the synthetic frame after affine transformation to the high dynamic range fusion module for fusion processing.

[0380] In some embodiments, the image processing apparatus further includes a third black level removal processing module and a second noise reduction module. The third black level removal processing module is configured to perform black level removal processing on multiple frames of image frames to be processed that are input to the multi-frame interpolation network. The second noise reduction module is configured to perform noise reduction processing on the multiple frames of image frames to be processed that have undergone black level removal processing, thereby obtaining adjusted multiple frames of image frames to be processed for input to the multi-frame interpolation network. The black level removal and noise reduction processing of the image frames to be processed that are input to the multi-frame interpolation network reduces image noise, improves the quality of the input image, and facilitates subsequent fusion to obtain a clearer image.

[0381] Each module in the aforementioned image determination device and image processing device may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in the form of hardware, or may be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0382] In some exemplary embodiments, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in Figure 16. The computer device includes a processor, memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, memory, and input / output interface are connected via a system bus, and the communication interface, display unit, and input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via wired or wireless communication, and the wireless communication may be achieved via Wi-Fi, a mobile cellular network, NFC (near-field communication), or other technologies. When executed by the processor, the computer program implements an image determination method or an image processing method. The display unit of the computer device is used to form a visually visible image and may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse.

[0383] Those skilled in the art will understand that the structure shown in Figure 16 is merely a block diagram of a portion of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0384] The present application also provides a computer-readable storage medium, one or more non-volatile computer-readable storage media containing computer-executable instructions, which, when executed by one or more processors, cause the processors to perform the steps of the image determination method or the image processing method.

[0385] An embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute an image determination method or an image processing method.

[0386] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0387] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processor involved in the various embodiments provided herein may be, but are not limited to, a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, and the like.

[0388] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0389] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. An image determination method, characterized in that: The method comprises: Determine an image sensor type; the image sensor type is one that supports digital overlap wide dynamic range or does not support digital overlap wide dynamic range; Get ambient brightness and motion status information; In response to a shooting operation, an image set for fusion is determined according to the image sensor type, the ambient brightness, and the motion state information.

2. The method according to claim 1, characterized in that The determining of the image set for fusion according to the image sensor type, the ambient brightness, and the motion state information includes: If the image sensor type supports digital overlap wide dynamic range and the ambient brightness is greater than a first brightness threshold, the image set for fusion is determined according to the motion state information to include at least one image pair data before the shooting operation.

3. The method according to claim 2, characterized in that If the image sensor type supports digital overlap wide dynamic range and the ambient brightness is greater than a first brightness threshold, determining, based on the motion state information, that the image set for fusion includes at least one image pair data before the shooting operation, including: If the image sensor type supports digital overlap wide dynamic range, the ambient brightness is greater than a first brightness threshold, and the motion state information is a local motion state or a stationary state, determining that the image set for fusion includes at least one first image pair data before the shooting operation, the first image pair data including data of a first baseline exposure frame and a first exposure brightness frame; the brightness of the first baseline exposure frame is greater than the brightness of the first exposure brightness frame; If the image sensor type supports digital overlap wide dynamic range, the ambient brightness is greater than the first brightness threshold, and the motion state information is a global motion state, then it is determined that the image set for fusion includes at least one second image pair data before the shooting operation, and the second image pair data includes data of a first exposure duration frame and a second exposure brightness frame; the first exposure duration frame has the same brightness as the first reference exposure frame, and the exposure duration of the first exposure duration frame is less than the first reference exposure frame; the brightness of the first exposure duration frame is greater than the brightness of the second exposure brightness frame.

4. The method according to claim 3, characterized in that The number of the first reference exposure frames and the number of the first exposure duration frames are both positively correlated with the brightness gain.

5. The method according to claim 4, characterized in that The method further comprises: When the brightness gain is within the first brightness gain range, the number of the first reference exposure frames and the first exposure time length frames are both within the first number range; When the brightness gain is within the second brightness gain range, the number of the first reference exposure frames and the first exposure time length frames are both within the second number range; When the brightness gain is within a third brightness gain range, the number of the first reference exposure frames and the first exposure time length frames are both within a third number range; Among them, the upper limit value of the first number range is smaller than the upper limit value of the second number range, the upper limit value of the second number range is smaller than the upper limit value of the third number range, the lower limit value of the first number range is smaller than the lower limit value of the second number range, and the lower limit value of the second number range is smaller than the lower limit value of the third number range.

6. The method according to claim 1, characterized in that The method further comprises: Acquire the shooting mode during the shooting operation; The determining of the image set for fusion according to the image sensor type, the ambient brightness, and the motion state information includes: An image set for fusion is determined according to the image sensor type, the ambient brightness, the shooting mode, and the motion state information.

7. The method according to claim 6, characterized in that The determining of the image set for fusion according to the image sensor type, the ambient brightness, the shooting mode, and the motion state information includes: If the image sensor type supports digital overlapping wide dynamic range, the ambient brightness is greater than the first brightness threshold and the shooting mode is a high dynamic range mode, the image set for fusion is determined according to the motion state information to include at least one image pair data before the shooting operation and at least one image pair data after the shooting operation.

8. The method according to claim 7, characterized in that The one image pair data is one frame of normal exposure duration data, and the brightness of the two frames of images in the one image pair data is different.

9. The method according to claim 8, characterized in that If the image sensor type supports digital overlap wide dynamic range, the ambient brightness is greater than a first brightness threshold, and the shooting mode is a high dynamic range mode, determining, according to the motion state information, that the image set for fusion includes at least one image pair data before the shooting operation and at least one image pair data after the shooting operation, including: If the image sensor type supports digital overlap wide dynamic range, the ambient brightness is greater than a first brightness threshold, the shooting mode is a high dynamic range mode, and the motion state information is a local motion state, then the image set determined for fusion includes at least one of at least one first image pair data before the shooting operation, at least one third image pair data after the shooting operation, and at least one fourth image pair data; the first image pair data includes data of a first baseline exposure frame and a first exposure brightness frame; the brightness of the first baseline exposure frame is greater than the brightness of the first exposure brightness frame; the third image pair data includes data of a second exposure duration frame and a third exposure brightness frame; the second exposure duration frame has the same brightness as the first baseline exposure frame, and the exposure duration of the second exposure duration frame is less than the exposure duration of the first baseline exposure frame; the brightness of the third exposure brightness frame is less than the brightness of the first exposure brightness frame; the fourth image pair data includes a third exposure brightness frame and a third exposure duration frame; the third exposure duration frame has the same brightness as the first baseline exposure frame, and the exposure duration of the third exposure duration frame is greater than the exposure duration of the first baseline exposure frame; If the image sensor type supports digital overlap wide dynamic range, the ambient brightness is greater than a first brightness threshold, the shooting mode is a high dynamic range mode, and the motion state information is a static state, then the image set determined for fusion includes at least one first image pair data before the shooting operation and at least one fourth image pair data after the shooting operation; If the image sensor type supports digital overlap wide dynamic range, the ambient brightness is greater than the first brightness threshold, the shooting mode is high dynamic range mode and the motion state information is global motion state, then the image set determined for fusion includes at least one second image pair data before the shooting operation and at least one fifth image pair data after the shooting operation; the second image pair data includes data of a first exposure duration frame and a second exposure brightness frame; the fifth image pair data includes data of a fourth exposure brightness frame and a fourth exposure duration frame; the first exposure duration frame has the same brightness as the first baseline exposure frame, and the exposure duration of the first exposure duration frame is less than the first baseline exposure frame; the brightness of the first exposure duration frame is greater than the brightness of the second exposure brightness frame; the brightness of the second exposure brightness frame is greater than the brightness of the fourth exposure brightness frame; the brightness of the fourth exposure duration frame is the same as the brightness of the first exposure duration frame, and the exposure duration of the fourth exposure duration frame is greater than the exposure duration of the first exposure duration frame.

10. The method according to any one of claims 6 to 9, characterized in that The determining of the image set for fusion according to the image sensor type, the ambient brightness, the shooting mode, and the motion state information includes: If the image sensor type supports digital overlapping wide dynamic range, the ambient brightness is greater than the first brightness threshold and the shooting mode is non-high dynamic range mode, the image set for fusion is determined according to the motion state information to include at least one image pair data before the shooting operation.

11. The method according to claim 10, characterized in that If the image sensor type supports digital overlap wide dynamic range, the ambient brightness is greater than a first brightness threshold, and the shooting mode is a non-high dynamic range mode, determining, according to the motion state information, that the image set for fusion includes at least one image pair data before the shooting operation, including: If the image sensor type supports digital overlap wide dynamic range, the ambient brightness is greater than a first brightness threshold, the shooting mode is a non-high dynamic range mode, and the motion state information is a global motion state, the image set determined for fusion includes at least one second image pair data before the shooting operation; the second image pair data includes data of a first exposure duration frame and a second exposure brightness frame; the first exposure duration frame has the same brightness as the first reference exposure frame, and the exposure duration of the first exposure duration frame is less than the first reference exposure frame; and the brightness of the first exposure duration frame is greater than the brightness of the second exposure brightness frame; If the image sensor type supports digital overlap wide dynamic range, the ambient brightness is greater than a first brightness threshold, the shooting mode is a non-high dynamic range mode, and the motion state information is a local motion state, then the image set determined for fusion includes at least one first image pair data before the shooting operation and at least one frame of a second exposure time length after the shooting operation; the first image pair data includes data of a first baseline exposure frame and a first exposure brightness frame; the brightness of the first baseline exposure frame is greater than the brightness of the first exposure brightness frame; If the image sensor type supports digital overlapping wide dynamic range, the ambient brightness is greater than a first brightness threshold, the shooting mode is a non-high dynamic range mode, and the motion state information is a static state, then the image set determined for fusion includes at least one first image pair data before the shooting operation.

12. The method according to claim 1, characterized in that The determining of the image set for fusion according to the image sensor type, the ambient brightness, and the motion state information includes: If the image sensor type supports digital overlap wide dynamic range and the ambient brightness is not greater than a first brightness threshold, the image set for fusion is determined according to the motion state information to include an image sequence after the shooting operation, and the image sequence includes image frames with different brightness.

13. The method according to claim 12, characterized in that If the image sensor type supports digital overlap wide dynamic range and the ambient brightness is not greater than a first brightness threshold, determining, based on the motion state information, that the image set for fusion includes an image sequence after the shooting operation, including: If the image sensor type supports a digital overlap wide dynamic range, the ambient brightness is not greater than a first brightness threshold, and the motion state information indicates a stationary state, determining that the image set for fusion includes a first image sequence after the shooting operation, the first image sequence including a second baseline exposure frame, a fifth exposure brightness frame, and a sixth exposure brightness frame after the shooting operation, the second baseline exposure frame being a plurality of frames; and the brightness of the second baseline exposure frame, the fifth exposure brightness frame, and the sixth exposure brightness frame decreases in sequence; If the image sensor type supports digital overlap wide dynamic range, the ambient brightness is not greater than the first brightness threshold, and the motion state information is a motion state, then it is determined that the image set for fusion includes a second image sequence after the shooting operation, the second image sequence includes a fifth exposure brightness frame and a sixth exposure brightness frame after the shooting operation, and the second image sequence also includes at least one of a second base exposure frame and a fifth exposure duration frame, and the second base exposure frame and the fifth exposure duration frame are multiple frames; the brightness of the second base exposure frame, the fifth exposure brightness frame, and the sixth exposure brightness frame decreases successively; the fifth exposure duration frame has the same brightness as the second base exposure frame, and the exposure duration of the fifth exposure duration frame is less than the exposure duration of the second base exposure frame.

14. The method according to claim 1, wherein The determining of the image set for fusion according to the image sensor type, the ambient brightness, and the motion state information includes: If the image sensor type does not support digital overlap wide dynamic range and the ambient brightness is greater than a second brightness threshold, the image set for fusion is determined according to the motion state information to include at least one image frame before the shooting operation.

15. The method according to claim 14, characterized in that The method further comprises: Acquire the shooting mode during the shooting operation; If the image sensor type does not support digital overlap wide dynamic range and the ambient brightness is greater than a second brightness threshold, determining, based on the motion state information, that the image set for fusion includes at least one image frame before the shooting operation, including: If the image sensor type does not support digital overlap wide dynamic range and the ambient brightness is greater than a second brightness threshold, the image set for fusion is determined based on the motion state information and the shooting mode to include at least one image frame before the shooting operation and an exposure frame set after the shooting operation.

16. The method according to claim 14 or 15, characterized in that The method further comprises: Acquiring the light sensitivity during the shooting operation; The number of image frames before the shooting operation is determined according to the relationship between the sensitivity and the sensitivity threshold.

17. The method according to claim 16, characterized in that The determining, based on the relationship between the sensitivity and the sensitivity threshold, the number of image frames before the shooting operation includes: If the sensitivity is greater than or equal to the sensitivity threshold, determining the number of image frames before the shooting operation is a first number; If the sensitivity is less than the sensitivity threshold, the number of image frames before the shooting operation is determined to be a second number; and the first number is greater than the second number.

18. The method according to claim 14 or 15, characterized in that If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, and the motion state information is a stationary state or a local motion state, determining that the image set for fusion includes at least one first exposure gain frame before the shooting operation, and the brightness of the first exposure gain frame is less than the brightness of a third reference exposure frame; If the image sensor type does not support digital overlapping wide dynamic range, the ambient brightness is greater than the second brightness threshold and the motion state information is a global motion state, then it is determined that the image set for fusion includes at least one second exposure gain frame before the shooting operation, and the exposure duration of the second exposure gain frame is less than the exposure duration of the first exposure gain frame.

19. The method according to claim 15, characterized in that If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, and the motion state information is a static state, the exposure frame set is a first exposure frame set, and the first exposure frame set includes a third reference exposure frame; If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, and the motion state information is a local motion state, the exposure frame set is a second exposure frame set, the second exposure frame set includes a sixth exposure duration frame, and the exposure duration of the sixth exposure duration frame is less than the exposure duration of the third reference exposure frame; If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than the second brightness threshold, and the motion state information is a global motion state, then the exposure frame set is a third exposure frame set, and the third exposure frame set includes a seventh exposure duration frame, and the exposure duration of the seventh exposure duration frame is less than the exposure duration of the third benchmark exposure frame.

20. The method according to claim 19, characterized in that If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, the motion state information indicates a stationary state, and the shooting mode is a high dynamic range mode, then the first exposure frame set includes a third baseline exposure frame and a seventh exposure brightness frame; If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, the motion state information is a stationary state, and the shooting mode is a non-high dynamic range mode, then the first exposure frame set includes a third reference exposure frame; If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, the motion state information is a local motion state, and the shooting mode is a high dynamic range mode, the second exposure frame set includes a sixth exposure duration frame and an eighth exposure brightness frame; If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, the motion state information is a local motion state, and the shooting mode is a non-high dynamic range mode, the second exposure frame set includes a sixth exposure duration frame; If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, the motion state information is a global motion state, and the shooting mode is a high dynamic range mode, the third exposure frame set includes a seventh exposure duration frame and a ninth exposure brightness frame; If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than the second brightness threshold, the motion state information is the global motion state, and the shooting mode is a non-high dynamic range mode, then the third exposure frame set includes a seventh exposure duration frame.

21. The method according to claim 20, characterized in that If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, the motion state information indicates a stationary state, the shooting mode is a high dynamic range mode, and the minimum exposure value is less than an exposure value threshold, then the first exposure frame set includes a third base exposure frame, a seventh exposure brightness frame, and a tenth exposure brightness frame, and the brightness of the tenth exposure brightness frame is greater than the brightness of the seventh exposure brightness frame; If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than the second brightness threshold, the motion state information indicates a stationary state, the shooting mode is a high dynamic range mode, and the minimum exposure value is greater than or equal to the exposure value threshold, then the first exposure frame set includes a third reference exposure frame and a seventh exposure brightness frame; If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than a second brightness threshold, the motion state information is a local motion state, the shooting mode is a high dynamic range mode, and the minimum exposure value is less than the exposure value threshold, then the second exposure frame set includes a sixth exposure duration frame, an eighth exposure brightness frame, and an eleventh exposure brightness frame; the brightness of the eleventh exposure brightness frame is greater than the brightness of the eighth exposure brightness frame; If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than the second brightness threshold, the motion state information is a local motion state, the shooting mode is a high dynamic range mode, and the minimum exposure value is greater than or equal to the exposure value threshold, then the second exposure frame set includes a sixth exposure duration frame and an eighth exposure brightness frame; If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is greater than the second brightness threshold, the motion state information is a global motion state, the shooting mode is a high dynamic range mode, and the minimum exposure value is less than the exposure value threshold, then the third exposure frame set includes a seventh exposure duration frame, a ninth exposure brightness frame, and a twelfth exposure brightness frame; The brightness of the twelfth exposure brightness frame is greater than the brightness of the ninth exposure brightness frame; If the image sensor type does not support digital overlapping wide dynamic range, the ambient brightness is greater than the second brightness threshold, the motion state information is a global motion state, the shooting mode is a high dynamic range mode and the minimum exposure value is greater than or equal to the exposure value threshold, then the third exposure frame set includes a seventh exposure duration frame and a ninth exposure brightness frame.

22. The method according to claim 21, characterized in that Exposure parameters of the seventh exposure brightness frame, the eighth exposure brightness frame, the ninth exposure brightness frame, the tenth exposure brightness frame, the eleventh exposure brightness frame, and the twelfth exposure brightness frame are adjustable.

23. The method according to claim 1, wherein The determining of the image set for fusion according to the image sensor type, the ambient brightness, and the motion state information includes: If the image sensor type does not support digital overlap wide dynamic range and the ambient brightness is not greater than a second brightness threshold, the image set for fusion is determined based on the motion state information to include an exposure frame sequence after the shooting operation, and the exposure frame sequence includes exposure frames of different brightness.

24. The method according to claim 23, wherein If the image sensor type does not support digital overlap wide dynamic range and the ambient brightness is not greater than a second brightness threshold, determining, based on the motion state information, that the image set for fusion includes an exposure frame sequence after the shooting operation, including: If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is not greater than a second brightness threshold, and the motion state information indicates a stationary state, determining that the image set for fusion includes a first exposure frame sequence after the shooting operation, the first exposure frame sequence includes a fourth reference exposure frame, a thirteenth exposure brightness frame, and a fourteenth exposure brightness frame after the shooting operation, the fourth reference exposure frame being a plurality of frames; and the brightness of the fourth reference exposure frame, the thirteenth exposure brightness frame, and the fourteenth exposure brightness frame decreases in sequence; If the image sensor type does not support digital overlap wide dynamic range, the ambient brightness is not greater than the second brightness threshold, and the motion state information is a motion state, then it is determined that the image set for fusion includes a second exposure frame sequence after the shooting operation, the second exposure frame sequence includes the fifteenth exposure brightness frame and the sixteenth exposure brightness frame after the shooting operation, and the second exposure frame sequence also includes at least one of a fourth base exposure frame and an eighth exposure time frame, and the fourth base exposure frame and the eighth exposure time frame are multiple frames; the brightness of the fourth base exposure frame, the fifteenth exposure brightness frame, and the sixteenth exposure brightness frame decreases successively; the eighth exposure time frame has the same brightness as the fourth base exposure frame, and the exposure time of the eighth exposure time frame is less than the exposure time of the fourth base exposure frame.

25. The method according to claim 1, wherein The determining the image sensor type includes: Get the zoom factor; Determining a target image sensor based on the relationship between the zoom factor and the zoom factor switching threshold, and determining whether the target image sensor supports a digital overlap wide dynamic range; If the target image sensor supports digital overlap wide dynamic range, determining the image sensor type as supporting digital overlap wide dynamic range; If the target image sensor supports non-digital overlap wide dynamic range, the image sensor type is determined to support non-digital overlap wide dynamic range.

26. The method according to claim 1, wherein The acquiring of motion state information includes: Obtaining gyroscope data during the shooting operation and statistical values ​​of brightness differences between corresponding pixels in two frames of images captured by the shooting operation; Motion state information is determined based on the gyroscope data and the brightness difference statistics.

27. The method according to claim 26, characterized in that The determining of the motion state information according to the gyroscope data and the brightness difference statistical value includes: If the gyroscope data is within a first motion threshold range and the brightness difference statistics are within the first brightness difference threshold range, determining that the motion state information is a stationary state; If the gyroscope data is within a second motion threshold range and the brightness difference statistics are within the second brightness difference threshold range, performing subject area motion detection; If the gyroscope data is within a third motion threshold range and the brightness difference statistics are within the third brightness difference threshold range, determining that the motion state information is a global motion state; Among them, the upper limit value of the first motion threshold range is smaller than the lower limit value of the second motion threshold range, and the upper limit value of the second motion threshold range is smaller than the lower limit value of the third motion threshold range; the upper limit value of the first brightness difference threshold range is smaller than the lower limit value of the second brightness difference threshold range, and the upper limit value of the second brightness difference threshold range is smaller than the lower limit value of the third brightness difference threshold range.

28. The method according to claim 27, characterized in that The performing of main body area motion detection includes: Obtaining brightness difference statistics of pixel points corresponding to the subject area in the two frames of image; If the brightness difference statistics corresponding to the main body area is greater than or equal to the brightness difference threshold, determining that the motion state information is a global motion state; If the brightness difference statistics corresponding to the main body area is less than the brightness difference threshold, it is determined that the motion state information is a local motion state.

29. The method according to claim 28, characterized in that In the case of a global motion state, the exposure duration is controlled to be reduced by N times, and in the case of a local motion state, the exposure duration is controlled to be reduced by N / 2 times, where N is a positive integer.

30. The method according to claim 29, wherein In the case of global motion, the exposure time is reduced by N times, and the corresponding brightness gain is increased by N times; in the case of local motion, the exposure time is controlled to be reduced by N / 2 times, and the corresponding brightness gain is increased by N / 2 times.

31. An image processing method, characterized in that: include: Acquire at least two sets of brightness image sets with different brightness; For each group of brightness image sets, synthesizing the image frames to be processed in the group of brightness image sets to obtain corresponding synthesized frames; determining a fusion reference frame from the plurality of synthesized frames; Taking the fusion reference frame as a reference, a plurality of the synthesized frames are fused to obtain a target image.

32. The method according to claim 31, characterized in that The method further comprises: determining a first reference frame; determining a reference frame mark according to the first reference frame and an image frame to be processed having a brightness different from that of the first reference frame, wherein the reference frame mark is used to represent a fused reference frame of high dynamic range fusion; Determining a fusion reference frame from the plurality of synthesized frames includes: A fusion reference frame is determined from the plurality of synthesized frames according to the reference frame label.

33. The method according to claim 32, characterized in that The at least two groups of luminance image sets include at least two of a first luminance image set, a second luminance image set, and a third luminance image set, the first luminance image set includes a plurality of first image frames, the second luminance image set includes at least one second image frame, the third luminance image set includes at least one third image frame, the luminance of the first image frame is greater than the luminance of the second image frame, and the luminance of the second image frame is greater than the luminance of the third image frame; The determining of the first reference frame includes: determining a first reference frame from the plurality of first image frames; The determining of the reference frame mark according to the first reference frame and the image frame to be processed having a brightness different from that of the reference frame includes: A reference frame tag is determined according to the first reference frame, the second image frame, and the third image frame, where the reference frame tag is used to characterize a fused reference frame of high dynamic range fusion.

34. The method according to claim 33, wherein The determining of a reference frame marker according to the first reference frame, the second image frame, and the third image frame includes: determining a motion mask image according to the first reference frame, the second image frame, and the third image frame; Obtaining overlapping areas between the overexposed areas corresponding to each of the first reference frame, the second image frame, and the third image frame and the motion mask image, and determining reference frame markers according to the overlapping areas; If the reference frame mark is the first mark, it means that the synthetic frame corresponding to the first image frame is used as the fusion reference frame for high dynamic range fusion; If the reference frame mark is the second mark, it means that the synthetic frame corresponding to the second image frame is used as the fusion reference frame for high dynamic range fusion; If the reference frame mark is the third mark, it means that the synthetic frame corresponding to the third image frame is used as the fusion reference frame for high dynamic range fusion.

35. The method according to claim 32, wherein The process of determining the first reference frame and the process of determining the benchmark frame mark are implemented by a central processing unit.

36. The method according to claim 31, wherein The acquiring of at least two sets of brightness image sets with different brightness comprises: Acquire a plurality of image pair data, each image pair data including a first image frame and a second image frame; determining a first reference frame from the plurality of first image frames; Acquire a second image frame in the image pair data where the first reference frame is located as a second reference frame; aligning the remaining first image frames except the first reference frame with the first reference frame, and obtaining a first luminance image set based on the first reference frame and the aligned first image frames; The remaining second image frames except the second reference frame are aligned with the second reference frame, and a second luminance image set is obtained according to the second reference frame and the aligned second image frames.

37. The method according to claim 31, wherein The step of synthesizing the image frames to be processed in each luminance image set to obtain a corresponding synthesized frame includes: For each luminance image set, each image frame to be processed in the luminance image set is subjected to noise reduction processing and color interpolation processing through an interpolation network to obtain a corresponding synthetic frame.

38. The method according to claim 37, wherein For each luminance image set, performing noise reduction and color interpolation processing on each to-be-processed image frame in the luminance image set through an interpolation network to obtain a corresponding synthesized frame, including: If the image frame to be processed in the luminance image set is one frame, performing noise reduction and color interpolation processing on each image frame to be processed in the luminance image set through a single frame interpolation network to obtain a composite frame; If there are multiple frames of image frames to be processed in the brightness image set, noise reduction processing and color interpolation processing are performed on each of the image frames to be processed in the brightness image set through a multi-frame interpolation network to obtain a synthesized frame.

39. The method according to claim 31, wherein The step of fusing the plurality of synthesized frames based on the fusion reference frame to obtain a target image includes: Taking the fusion reference frame as a reference, high dynamic range fusion is performed on the multiple synthetic frames to obtain a target image.

40. The method according to claim 39, wherein The method further comprises: Performing target segmentation processing on the fused reference frame to obtain a main body area and a non-main body area; The step of performing high dynamic range fusion on the plurality of synthesized frames based on the fusion reference frame to obtain a target image includes: Taking the fusion reference frame as a reference, performing high dynamic range fusion on the plurality of synthesized frames to obtain a first fused image; fusing the subject area with the first fused image to obtain a second fused image; performing tone mapping processing on the second fused image to obtain a third fused image; The third fused image is adjusted according to the subject area and the non-subject area to obtain a target image.

41. The method according to claim 40, wherein The target segmentation processing is performed on the fused reference frame to obtain the main body area and the non-main body area, including: compressing and scaling the fused reference frame to obtain an adjusted fused reference frame; The adjusted fusion reference frame is subjected to target segmentation processing to obtain a main body area and a non-main body area.

42. The method according to claim 40, wherein The fusing the subject area with the first fused image to obtain a second fused image includes: The first fused image is subjected to brightness adaptive segmentation processing, and then the main body area is fused with the adaptively segmented first fused image to obtain a second fused image.

43. The method according to claim 40, wherein The adjusting the third fused image according to the subject area and the non-subject area to obtain a target image includes: performing local contrast enhancement processing on the third fused image to obtain a locally enhanced third fused image; performing subject protection processing on the locally enhanced third fused image according to the subject area and the non-subject area to obtain a fourth fused image; A target image is obtained according to the fourth fused image.

44. The method according to claim 43, wherein Obtaining a target image according to the fourth fused image includes: Performing a first color space conversion on the fourth fused image to obtain a fifth fused image; The fifth fused image is converted into a second color space to obtain a target image.

45. The method according to claim 44, wherein The step of performing a first color space conversion on the fourth fused image to obtain a fifth fused image includes: Performing color correction matrix, gamma transformation, and first color space transformation on the fourth fused image to obtain a fifth fused image.

46. ​​The method according to claim 44, wherein The step of performing a second color space conversion on the fifth fused image to obtain a target image includes: Perform brightness adjustment processing and second color space transformation on the fifth fused image to obtain a target image.

47. The method according to claim 44, characterized in that The fourth fused image is an RGB domain image, the fifth fused image is an HSV domain image, and the target image is a YUV domain image.

48. The method according to claim 40, wherein The step of performing high dynamic range fusion on the plurality of synthesized frames based on the fusion reference frame to obtain a first fused image includes: Get the motion mask image; Taking the fusion reference frame as a reference, the motion mask image and the plurality of synthesized frames are fused in a high dynamic range to obtain a first fused image.

49. The method according to claim 48, characterized in that The acquiring of the motion mask image comprises: determining a first reference frame; The first reference frame and the image frame to be processed with a brightness different from that of the first reference frame are processed by an optical flow algorithm to obtain a motion mask image.

50. The method according to claim 49, wherein The at least two groups of luminance image sets include at least two of a first luminance image set, a second luminance image set, and a third luminance image set, the first luminance image set includes a plurality of first image frames, the second luminance image set includes at least one second image frame, the third luminance image set includes at least one third image frame, the luminance of the first image frame is greater than the luminance of the second image frame, and the luminance of the second image frame is greater than the luminance of the third image frame; The determining of the first reference frame includes: determining a first reference frame from the plurality of first image frames; The step of processing the first reference frame and the image frame to be processed having a brightness different from that of the first reference frame by an optical flow algorithm to obtain a motion mask image includes: The first reference frame, the second image frame, and the third image frame are processed by an optical flow algorithm to obtain a motion mask image.

51. The method according to claim 38, wherein The method further comprises: The synthesized frame obtained by the multi-frame interpolation network processing is subjected to at least one of black level removal processing, dynamic range adjustment gain increase, shadow correction and white balance processing to obtain an adjusted synthesized frame.

52. The method according to claim 38, wherein The method further comprises: The synthesized frame obtained by the single-frame interpolation network processing is subjected to inverse gamma transformation processing to obtain an adjusted synthesized frame.

53. The method according to claim 38, wherein The method further comprises: If the image to be processed in the luminance image set is one frame, the image to be processed is subjected to black level removal processing, shadow correction, white balance and gamma transformation processing to obtain an adjusted image to be processed, and the adjusted image to be processed is used to input the single-frame interpolation network to obtain a synthesized frame.

54. The method according to claim 53, wherein The step of performing black level removal processing, white balance processing, and gamma conversion processing on the image to be processed to obtain an adjusted image to be processed includes: The image to be processed is subjected to black level removal processing, noise reduction processing, shadow correction, white balance and gamma transformation processing to obtain an adjusted image to be processed.

55. The method according to claim 37, wherein The bit width of the image input to the interpolation network is smaller than the bit width of the image output by the interpolation network.

56. The method according to claim 37, wherein The interpolation network is implemented by a neural network processor.

57. The method according to claim 40, wherein The target segmentation processing process is implemented by a neural network processor.

58. The method according to claim 31, wherein The image frames to be processed in the brightness image set are RAW domain images, the synthesized frames are RGB domain images, and the target images are YUV domain images.

59. The method according to claim 31, wherein The image to be processed in the brightness image set is subjected to at least one of black level removal processing, dynamic range adjustment gain increase, lens shading correction, affine transformation, high dynamic range fusion, brightness adaptive segmentation, tone mapping, local contrast enhancement, subject protection processing, color correction matrix, gamma transformation, first color space conversion, brightness adjustment processing, and second color space conversion to obtain a target image, wherein the black level removal processing, dynamic range adjustment gain increase, lens shading correction, affine transformation, high dynamic range fusion, brightness adaptive segmentation, tone mapping, local contrast enhancement, subject protection processing, color correction matrix, gamma transformation, first color space conversion, brightness adjustment processing, and second color space conversion are implemented by a graphics processor.

60. The method according to any one of claims 31 to 59, characterized in that The target image is input to an image signal processor.

61. An image determination device, characterized in that: The device comprises: A type determination module, configured to determine an image sensor type; the image sensor type is one that supports digital overlap wide dynamic range or does not support digital overlap wide dynamic range; Information acquisition module, used to obtain environmental brightness and motion status information; The image frame determination module is configured to determine, in response to a shooting operation, an image frame for fusion according to the image sensor type, the ambient brightness, and the motion state information.

62. An image processing device, characterized in that The device comprises: An image acquisition module, configured to acquire at least two sets of brightness image sets with different brightness; A synthesis module, for synthesizing, for each luminance image set, the image frames to be processed in the luminance image set to obtain a corresponding synthesized frame; A reference frame determining module, for determining a fusion reference frame from the plurality of synthesized frames; The fusion module is used to fuse the multiple synthetic frames based on the fusion reference frame to obtain a target image.

63. The device according to claim 62, characterized in that The device further comprises: A reference frame determining module, configured to determine a first reference frame; a tag determining module, configured to determine a reference frame tag based on the first reference frame and an image frame to be processed having a brightness different from that of the first reference frame, wherein the reference frame tag is used to characterize a fused reference frame of high dynamic range fusion; The reference frame determination module is further configured to determine a fusion reference frame from the plurality of synthesized frames according to the reference frame mark.

64. The device according to claim 63, characterized in that The at least two groups of luminance image sets include at least two of a first luminance image set, a second luminance image set, and a third luminance image set, the first luminance image set includes a plurality of first image frames, the second luminance image set includes at least one second image frame, the third luminance image set includes at least one third image frame, the luminance of the first image frame is greater than the luminance of the second image frame, and the luminance of the second image frame is greater than the luminance of the third image frame; the apparatus includes: The reference frame determination module is further configured to determine a first reference frame from a plurality of first image frames; The label determination module is further configured to determine a reference frame label according to the first reference frame, the second image frame, and the third image frame, wherein the reference frame label is configured to characterize a fused reference frame of high dynamic range fusion.

65. The device according to claim 64, characterized in that The reference frame determination module and the marker determination module are called by a central processing unit.

66. The device according to claim 62, characterized in that The apparatus further comprises: at least one of an alignment module, an affine transformation module, and a de-artifacting module called by a central processing unit; The alignment module is used to align the images in the at least two groups of brightness image sets; The affine transformation module is used to perform affine transformation processing on the images in the at least two groups of brightness image sets; The artifact removal module is configured to perform artifact removal processing on the images in the at least two groups of brightness image sets.

67. The device according to claim 62, characterized in that The synthesis module is further configured to perform noise reduction and color interpolation processing on each to-be-processed image frame in each luminance image set through an interpolation network to obtain a corresponding synthesized frame.

68. The device according to claim 67, characterized in that The synthesis module includes: a single-frame interpolation network module, called by the neural network processor, for performing noise reduction and color interpolation processing on each of the image frames to be processed in the luminance image set through the single-frame interpolation network to obtain a corresponding composite frame if the image frame to be processed in the luminance image set is one frame; The multi-frame interpolation network module is called by the neural network processor and is used to perform noise reduction and color interpolation processing on each of the image frames to be processed in the brightness image set through the multi-frame interpolation network if there are multiple frames in the brightness image set to obtain corresponding synthetic frames.

69. The device according to claim 62, characterized in that The fusion module includes: An object segmentation module is used to perform object segmentation processing on the fused reference frame to obtain a main body area and a non-main body area; a high dynamic range fusion module, configured to perform high dynamic range fusion on the plurality of synthesized frames based on the fusion reference frame to obtain a first fused image; an enhanced fusion module, configured to fuse the subject area with the first fused image to obtain a second fused image; a tone mapping module, configured to perform tone mapping processing on the second fused image to obtain a third fused image; The subject protection module is used to adjust the third fused image according to the subject area and the non-subject area to obtain a target image.

70. The device according to claim 69, characterized in that The target segmentation module is called by a neural network processor; the reference frame determination module, the high dynamic range fusion module, the enhanced fusion module, the tone mapping module and the subject protection module are called by an image processor.

71. The device according to claim 69, characterized in that The fusion module also includes: a compression and scaling module, configured to compress and scale the fused reference frame to obtain an adjusted fused reference frame; The object segmentation module is further configured to perform object segmentation processing on the adjusted fusion reference frame to obtain a main body area and a non-main body area.

72. The device according to claim 71, characterized in that The compression and scaling module is called by a graphics processor.

73. The device according to claim 69, characterized in that The fusion module also includes: an adaptive segmentation module, called by a graphics processor, for performing brightness adaptive segmentation processing on the first fused image; The enhanced fusion module is further configured to fuse the main body region with the adaptively segmented first fused image to obtain a second fused image.

74. The device according to claim 69, characterized in that The fusion module also includes: a local contrast enhancement module, configured to perform local contrast enhancement processing on the third fused image to obtain a locally enhanced third fused image; The subject protection module is further configured to perform subject protection processing on the locally enhanced third fused image according to the subject area and the non-subject area to obtain a fourth fused image; A color correction module, configured to perform color correction matrix processing on the fourth fused image; a first gamma transformation module, configured to perform gamma transformation processing on the fourth fused image after color correction; a first color space conversion module, configured to perform a first color space conversion on the fourth fused image that has undergone gamma transformation to obtain a fifth fused image; a brightness adjustment module, configured to perform brightness adjustment processing on the fifth fused image; The second color space conversion module is used to perform a second color space conversion on the fifth fused image after brightness adjustment to obtain a target image.

75. The device according to claim 74, characterized in that The local contrast enhancement module, the color correction module, the gamma transformation module, the first color space conversion module, the brightness adjustment module, and the second color space conversion module are all called by a graphics processor.

76. The device according to claim 74, characterized in that The fourth fused image is an RGB domain image, the fifth fused image is an HSV domain image, and the target image is a YUV domain image.

77. The device according to claim 69, characterized in that The at least two groups of luminance image sets include at least two of a first luminance image set, a second luminance image set, and a third luminance image set, the first luminance image set includes a plurality of first image frames, the second luminance image set includes at least one second image frame, the third luminance image set includes at least one third image frame, the luminance of the first image frame is greater than the luminance of the second image frame, and the luminance of the second image frame is greater than the luminance of the third image frame; The device further comprises: A reference frame determining module, configured to determine a first reference frame from a plurality of first image frames; a motion mask determination module, configured to process the first reference frame, the second image frame, and the third image frame using an optical flow algorithm to obtain a motion mask image; The high dynamic range fusion module is further configured to perform high dynamic range fusion on the motion mask image and multiple synthetic frames based on the fusion reference frame to obtain a first fused image.

78. The device according to claim 69, characterized in that The device further comprises at least one of the following modules: A first black level removal processing module is used to remove the black level of the synthesized frame obtained by the multi-frame interpolation network processing; An enhanced dynamic range module, for performing a dynamic range adjustment gain process on the synthesized frame after black level processing is removed; A first shadow correction module, configured to perform shadow correction processing on the synthesized frame after gain adjustment; a first white balance module, configured to perform white balance processing on the composite frame after shadow correction to obtain an adjusted composite frame; The first black level removal processing module, the dynamic range enhancement module, the first shadow correction module, and the first white balance module are called by a graphics processor.

79. The device according to claim 69, characterized in that The device further comprises: The inverse gamma transform module is called by the graphics processor to perform inverse gamma transform processing on the synthesized frame obtained by the single-frame interpolation network processing to obtain an adjusted synthesized frame.

80. The method according to claim 69, wherein The device further comprises: a second black level removal processing module, configured to perform black level removal processing on the image to be processed if the image to be processed in the luminance image set is one frame; A second shadow correction module is used to perform shadow correction processing on the image to be processed after the black level processing is removed; A second white balance module is used to perform white balance processing on the image to be processed after the shadow correction processing; a second gamma transform process for performing gamma transform processing on the image to be processed that has undergone white balance processing to obtain an adjusted image to be processed, wherein the adjusted image to be processed is input into the single-frame interpolation network to obtain a synthesized frame; The second black level removal processing module, the second shadow correction module, the second white balance module, and the second gamma conversion processing are called by a graphics processor.

81. The device according to claim 80, characterized in that The device further comprises: The noise reduction module is called by the graphics processor to perform noise reduction on the image to be processed after the black level is removed.

82. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the computer program is executed by the processor, the processor is caused to perform the steps of the image determination method according to any one of claims 1 to 30, or perform the steps of the image processing method according to any one of claims 31 to 61.

83. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the computer program implements the steps of the image determination method according to any one of claims 1 to 30, or implements the steps of the image processing method according to any one of claims 31 to 61.

84. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the steps of the image determination method according to any one of claims 1 to 30, or implements the steps of the image processing method according to any one of claims 31 to 61.

Images