Video processing method and electronic equipment

Abstract

The invention provides a video processing method and electronic equipment, and belongs to the technical field of terminals. The method comprises the following steps: receiving a first operation input by a user; in response to the first operation, acquiring a first image according to a preset default HDR exposure mode; according to the first image, environment brightness is obtained, a target HDR exposure mode is determined according to a preset strategy, and the preset strategy comprises a dynamic range corresponding to video shooting, a stroboscopic state and a corresponding relation between the environment brightness and the target HDR exposure mode; and when the target HDR exposure mode is different from the default HDR exposure mode, switching the default HDR exposure mode into the target HDR exposure mode, and continuing video shooting. According to the method, seamless automatic switching is carried out among multiple exposure modes according to the actual video shooting condition, so that efficient HDR processing is carried out in a video shooting scene, and the video image quality is improved.

Description

Video processing method and electronic device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on August 9, 2022, with application number 202210950356.6 and application name “Video Processing Method and Electronic Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of terminal technology, and in particular to a video processing method and electronic device. Background Art

[0003] Exposure is the process by which light entering the lens is received by a photosensitive device to form an image. During the shooting process, the brightness and darkness of the background or subject will change. Too much external light can easily lead to overexposure, resulting in an image that is too bright and lacks depth and detail. Too little external light can easily lead to underexposure, resulting in an image that is too dark and fails to reflect true color.

[0004] In practical applications, image brightness is often limited by dynamic range. Dynamic range refers to the ratio of the device's maximum and minimum output signals, or the grayscale ratio of the image's upper and lower brightness limits. If the ambient brightness exceeds the upper limit of the dynamic range, the captured image will appear brighter; if the ambient brightness falls below the lower limit, the captured image will appear darker.

[0005] Currently, factors influencing dynamic range include the size of the device and the image sensor. A larger sensor provides a larger photosensitive surface area, providing more area for receiving light information during the exposure time. This increases the number of pixels and results in a wider dynamic range. However, with the advancement of technology, the physical miniaturization of devices has limited sensor size, resulting in a limited dynamic range.

[0006] Summary of the Invention

[0007] An embodiment of the present application provides a video processing method, which realizes efficient HDR processing in video shooting scenarios and improves video quality by seamlessly and automatically switching between multiple exposure modes according to the actual video shooting situation.

[0008] In a first aspect, a frequency processing method is provided, which is applied to an electronic device, and the method includes:

[0009] receiving a first operation input by a user, where the first operation is used to enable a video capture function of the electronic device;

[0010] In response to the first operation, acquiring a first image according to a preset default high dynamic range (HDR) exposure mode;

[0011] Acquire ambient brightness based on the first image, and determine a target HDR exposure mode according to a preset strategy, the preset strategy including dynamic range information corresponding to the video shooting, a stroboscopic state, and a correspondence between the ambient brightness and the target HDR exposure mode;

[0012] When the target HDR exposure mode is different from the default HDR exposure mode, the default HDR exposure mode is switched to the target HDR exposure mode, and the video shooting is continued to obtain a second image.

[0013] In a possible implementation, the dynamic range information here may include dynamic range and / or dynamic range compression gain.

[0014] According to the video processing method provided by this implementation, by seamlessly switching between multiple types of HDR processing schemes according to changes in factors such as ambient brightness, required dynamic range, and flicker detection, it is possible to use the HDR scheme adapted to the actual shooting environment and image quality requirements for image processing, thereby effectively expanding the dynamic range in the recording scene and improving the image quality in the recording scene.

[0015] With reference to the first aspect, in some implementations of the first aspect, the target HDR exposure mode includes at least a first HDR exposure mode and a second HDR exposure mode, the first HDR exposure mode is a single-frame mode, and the second HDR exposure mode is a dual-frame mode;

[0016] When the target HDR exposure mode is the second HDR exposure mode, the images input in the dual-frame mode are fused.

[0017] In one possible implementation, the single-frame mode can be a binning exposure method, where the image sensor outputs a single frame after exposure. The dual-frame mode can be an exposure method such as SHDR, DCG, or DXG, where two frames are read from a single exposure and then fused and adjusted for HDR; or two frames are acquired using long and short exposures and then fused and adjusted for HDR.

[0018] In combination with the first aspect, in some implementations of the first aspect, the first HDR exposure mode is a binning mode, and the second HDR exposure mode includes a staggered high dynamic range mode SHDR and DXG, and the DXG is a mode in which a dual conversion gain mode DCG and a dual analog gain DAG are superimposed.

[0019] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes:

[0020] Presetting target parameters corresponding to the image obtained through the video shooting;

[0021] adjusting initial parameters corresponding to the first image acquired based on the first HDR exposure mode to the target parameters; and / or,

[0022] Adjusting initial parameters corresponding to the second image acquired based on the second HDR exposure mode to the target parameters.

[0023] In a possible implementation, the target parameters here may include image size, bit depth, etc.

[0024] It should be understood that adjusting the parameters of images corresponding to different exposure modes to target parameters can ensure that images of different exposure modes have consistent image parameters, thereby avoiding image jumps when switching between different exposure modes.

[0025] In combination with the first aspect, in some implementations of the first aspect, the electronic device includes an automatic exposure control AEC module, an image sensor, and a perception module.

[0026] The step of acquiring the first image according to a preset default HDR exposure mode in response to the first operation specifically includes:

[0027] In response to the first operation, the AEC module sends first indication information to the image sensor, where the first indication information is used to instruct the image sensor to capture an image using the default HDR exposure mode;

[0028] In response to the first indication information, the image sensor uses the default HDR exposure mode and acquires the first image.

[0029] In combination with the first aspect, in some implementations of the first aspect, the electronic device includes an AEC module, an image sensor, and a perception module.

[0030] The step of obtaining the ambient brightness according to the first image and determining the target HDR exposure mode according to a preset strategy specifically includes:

[0031] The image sensor sends the first image to the perception module;

[0032] The perception module obtains the ambient brightness according to the first image and indicates the ambient brightness to the AEC module;

[0033] The AEC module determines the target HDR exposure mode according to the ambient brightness and a preset strategy.

[0034] In combination with the first aspect, in some implementations of the first aspect, the electronic device includes an AEC module, an image sensor, a perception module, and a fusion module.

[0035] When the target HDR exposure mode is the second HDR exposure mode, fusing the dual-frame input images specifically includes:

[0036] When the target HDR exposure mode is the second HDR exposure mode, the image sensor transmits the image in the dual-frame mode to the fusion module;

[0037] The fusion module fuses the images in the dual-frame mode; or,

[0038] When the target HDR exposure mode is the second HDR exposure mode, the image sensor fuses the images in the dual-frame mode.

[0039] In combination with the first aspect, in certain implementations of the first aspect, when the fusion module fuses the images in the dual-frame mode, target sensitivity ratios between the dual-frame input images in the DCG mode and the dual-frame input images in the DAG mode are determined according to the sensitivity ratio required by the DXG mode.

[0040] The DCG mode double-frame input image and the DAG mode double-frame input image are superimposed respectively according to the target sensitivity ratio to obtain the superimposed double-frame input image that meets the DXG mode sensitivity ratio.

[0041] In combination with the first aspect, in certain implementations of the first aspect, when the image sensor fuses the image in the dual-frame mode, the dual-frame input image in the DCG mode and the dual-frame input image in the DAG mode are superimposed according to a preset sensitivity ratio.

[0042] In conjunction with the first aspect, in certain implementations of the first aspect, determining the target HDR exposure mode according to a preset strategy specifically includes:

[0043] Obtaining a dynamic range corresponding to the video shooting;

[0044] When the dynamic range information is less than a first threshold, determining that the target HDR exposure mode is the binning mode;

[0045] When the dynamic range information is greater than or equal to the first threshold, detecting whether there is stroboscopic light;

[0046] When stroboscopic light is present, determining that the target HDR exposure mode is the binning mode;

[0047] When there is no stroboscopic light, the target HDR exposure mode is determined according to the ambient brightness; wherein,

[0048] When the ambient brightness is greater than a second threshold, determining that the target HDR exposure mode is the SHDR mode;

[0049] When the ambient brightness is less than a third threshold, the target HDR exposure mode is determined to be the DXG mode.

[0050] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes:

[0051] Detecting whether there is a thermal escape phenomenon in the video shooting;

[0052] When the thermal escape phenomenon occurs, the target HDR exposure mode is determined to be the binning mode.

[0053] In combination with the first aspect, in some implementations of the first aspect, the electronic device supports the first HDR video mode, and the first HDR video mode includes HDR10 or HDR10+.

[0054] In conjunction with the first aspect, in certain implementations of the first aspect, when switching between an HDR camera and a non-HDR camera occurs during video capture, the method further includes:

[0055] During the switching, the exposure mode of the HDR camera is the first HDR exposure mode, and the exposure mode corresponding to the non-HDR camera is the second HDR exposure mode, and the first dynamic range gain corresponding to the first HDR exposure mode is adjusted so that the first dynamic range gain is closest to the second dynamic range gain corresponding to the second HDR exposure mode.

[0056] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes:

[0057] A first interface is displayed, where the first interface includes a first control, and the first control is used to enable a function of automatically switching the HDR exposure mode.

[0058] In a second aspect, an electronic device is provided, comprising: one or more processors; one or more memories; the one or more memories storing one or more computer programs, the one or more computer programs comprising instructions, which, when executed by the one or more processors, enable the electronic device to execute the method described in any implementation of the first or second aspect above.

[0059] In a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable program instructions, and when the computer-executable program instructions are executed on a computer, the computer executes the method as described in any implementation of the first aspect or the second aspect above.

[0060] In a fourth aspect, a computer program product is provided, comprising computer program code, which, when executed on a computer, causes the computer to execute the method as described in any one of the implementations of the first or second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] FIG1 is a schematic diagram of an image captured by an interlaced high dynamic range mode according to an embodiment of the present application.

[0062] 2A and 2B are schematic diagrams of images captured by the DCG mode and the DAG mode provided in an embodiment of the present application.

[0063] FIG3 is a schematic structural diagram of an electronic device 100 provided in an embodiment of the present application.

[0064] FIG4 is a software structure block diagram of an electronic device 100 provided in an embodiment of the present application.

[0065] FIG5 is a schematic diagram showing differences in related parameters corresponding to a plurality of exposure modes provided in an embodiment of the present application.

[0066] 6A to 6D are schematic diagrams of GUIs that may be involved in some video processing processes provided in embodiments of the present application.

[0067] FIG7 is a schematic diagram of a video processing method provided in an embodiment of the present application.

[0068] FIG8 is a schematic diagram of another video processing method provided in an embodiment of the present application.

[0069] FIG9 is a schematic diagram of another video processing method provided in an embodiment of the present application.

[0070] FIG10 is a schematic diagram of another video processing method provided in an embodiment of the present application.

[0071] FIG11 is a schematic diagram of another video processing method provided in an embodiment of the present application.

[0072] FIG12 is a schematic diagram of another video processing method provided in an embodiment of the present application.

[0073] FIG13 is a schematic diagram of adjusting image parameters during a video processing process provided in an embodiment of the present application.

[0074] FIG14 is a schematic diagram of adjusting image parameters during a video processing process provided in an embodiment of the present application.

[0075] FIG15 is a schematic diagram of another video processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0076] It should be noted that the terms used in the implementation methods section of the embodiments of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is merely a way to describe the association relationship of associated obstacles, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two, "at least one" and "one or more" mean one, two or more than two.

[0077] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly specifying the number of the technical features indicated. Therefore, the definition of "first" and "second" features may explicitly or implicitly include one or more of the features.

[0078] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0079] The term "user interface" as used in the specification, claims, and drawings of this application refers to the media interface for interaction and information exchange between an application or operating system and a user. A common form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operations that is displayed graphically. It can be an interface element such as an icon, window, or control displayed on the display screen of an electronic device. Controls can include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.

[0080] As described in the background, high dynamic range imaging (HDR) technology has emerged to overcome the problem of low image quality (small differences in brightness and darkness) caused by limited dynamic range. HDR technology is a set of technologies that can achieve a larger exposure dynamic range (i.e., a larger difference in brightness and darkness). Its purpose is to accurately represent the brightness range in the real world, from direct sunlight to the darkest shadows. In a specific implementation of HDR technology, the subject currently being photographed can be exposed by setting multiple sets of exposure values ​​(EV). These include exposure using the normal EV value obtained by the current metering calculation, exposure using EV values ​​below normal (EV-n), and exposure using EV values ​​above normal (EV+n). The multiple exposed photos are then fused, so that dark objects are partially exposed using a high EV exposure, while bright objects are partially exposed using a low EV exposure, ensuring that the entire scene in the photo is neither too bright nor too dark.

[0081] The above-mentioned method of expanding the dynamic range by EV bracketing and fusing multiple photos involves a more complex image processing algorithm, which requires a sufficient interval between two frames as calculation time. Generally speaking, in photo shooting scenarios, the time interval between acquiring two consecutive frames of images is long enough to achieve dynamic range expansion according to the traditional HDR exposure method. However, in video recording scenarios, in order to ensure the smoothness of the picture, images are generally acquired and processed at a specific frame rate, and the frame interval is very short, that is, each frame corresponds to only a very short calculation time (for example, when the frame rate is 30fps, the calculation time corresponding to each frame is less than 33ms on average). This makes it impossible to use the more complex traditional method to expand the dynamic range in video recording scenarios, but it is necessary to customize an adaptive HDR solution based on the characteristics of video recording.

[0082] In view of this, an embodiment of the present application provides a video processing method, which seamlessly switches between multiple types of HDR exposure modes according to changes in factors such as ambient brightness, dynamic range, and flicker, thereby utilizing an HDR solution adapted to the actual shooting state and image quality requirements for image acquisition and processing, thereby effectively expanding the dynamic range in the recording scene and improving the image quality in the recording scene.

[0083] In order to better understand the video processing method provided in the embodiments of the present application, some term definitions that may be involved in this article are first introduced.

[0084] 1. Stagger high-dynamic range (SHDR)

[0085] Different manufacturers may have different names for technologies similar to SHDR. For example, Sony's related technology is called digital overlap HDR (DOL-HDR). SHDR technology increases the sensor's frame rate, allowing it to capture multiple frames of images with different exposure times within a single acquisition cycle. Multi-frame fusion then fuses the corresponding long-exposure and short-exposure frames into a single frame to produce an image with a high dynamic range. The short-exposure frame (or "short-exposure image") captures bright area information, while the long-exposure frame (or "long-exposure image") captures dark area information, and the long-exposure frame has excellent noise control. Fusion of the two frames yields gains in both highlights and shadows.

[0086] Exemplarily, an image captured using the SHDR technology may be as shown in FIG1 , including a frame of long-exposure images (such as the frame marked with the letter “L” in FIG1 ) and a frame of short-exposure images (such as the frame marked with the letter “S” in FIG1 ) that are periodically captured.

[0087] It should be noted that, affected by different exposure periods and exposure times, SHDR may not be able to obtain ideal shooting effects in some scenes. For example, compared with non-SHDR exposure shooting scenes, SHDR scenes need to collect images under multiple exposure parameters (such as exposure time), so the blank frame interval between the two frames will be shorter, which is not suitable for scenes where the image processing algorithm is too complex. In addition, the two frames used by SHDR to fuse into one image come from different exposure periods and correspond to different exposure times. The longer the exposure time, the more likely it is to produce smearing (or ghosting), and motion ghosting will inevitably occur during fusion (only in high-brightness scenes where the exposure time of both frames is very short can a more ideal effect be achieved).

[0088] 2. Binning

[0089] Binning is an image readout mode that adds together the charges induced in adjacent pixels and reads them out in a one-pixel mode. For example, when an electronic device is shooting an image, the light reflected by the target object is collected by the camera so that the reflected light is transmitted to the image sensor. The image sensor includes multiple photosensitive elements, and the charge collected by each photosensitive element is a pixel, and a binning operation is performed on the pixel information. Specifically, binning can merge n×n pixels into one pixel. For example, binning can combine adjacent 2×2 pixels into one pixel, that is, the colors of adjacent 2×2 pixels are presented in the form of one pixel.

[0090] 3. Dual conversion gain (DCG)

[0091] DCG is also a method of image readout, which can be understood as the ability to perform two reads within a pixel element circuit, or as having two capacitors storing photon energy within the photosensitive unit corresponding to a pixel. The DCG involved in the embodiments of this application can specifically refer to performing two conversion gain readouts on an image based on a single exposure operation.

[0092] In an embodiment of the present application, DCG can be used to expand the dynamic range. The implementation principle is as follows: an image sensor with dual conversion gain DCG capability has two potential wells per pixel. The two potential wells correspond to different full well capacities and different conversion gains CG. The large full well capacity corresponds to low conversion gain (LCG) and low sensitivity, and the small full well capacity corresponds to high conversion gain (HCG) and high sensitivity. In this way, the sensor can use two potential wells (two sensitivities) and two conversion gains in the same scene to acquire two images in one exposure: an image in high-sensitivity mode and an image in low-sensitivity mode. The electronic device then synthesizes the two acquired images into one image, which is HDR technology.

[0093] As an example, based on the same short exposure, two DCG readouts can be shown in Figure 2A. The LCG frame is read out using the LCG gain signal, which protects highlights from overexposure. The HCG frame is read out using the HCG gain signal, which enhances shadow brightness while controlling noise. The two frames are then fused to capture the gains in highlights and shadows, yielding a single image with optimized dynamic range.

[0094] 4. Dual analog gain (DAG)

[0095] As another image readout method, DAG, similar to the dual-conversion-gain (DCG) described above, also uses two analog signals to read two frames of image data from a single exposure. The difference lies in the readout method. DAG readout uses two analog gains for image readout: low analog gain (LAG) and high analog gain (HAG). LAG protects highlights from overexposure, while HAG brightens shadows while controlling noise.

[0096] As an example, based on the same short exposure, two DAG readout images can be shown in Figure 2B, where the LAG frame is an image frame readout using LAG, and the HAG frame is an image frame readout using HAG. The two images are then fused to capture highlights and shadows, expanding the dynamic range. Compared to conversion gain (CG), analog gain (AG) offers better noise control capabilities.

[0097] It should be noted that since the two frames of DCG and DAG above are from different readouts of a single exposure, there will be no ghosting problem after the two frames are fused, making it suitable for a wide range of use in various recording scenarios.

[0098] It should also be noted that in the embodiment of the present application, the above-mentioned DCG and DAG technologies can also be superimposed to expand the dynamic range. For the sake of ease of description, the embodiment of the present application refers to the method of superimposing DCG and DAG as DXG. For example, DCG and DAG can be superimposed according to the sensitivity ratio. For example, if the sensitivity ratio between the image read out by the high gain signal and the image read out by the low gain signal is required to be 1:16, then DCG can be used to obtain two frames of images with a sensitivity ratio of 1:2, and then DAG can be used to obtain two frames of images with a sensitivity ratio of 1:8. After multiplying the sensitivity ratios of the two, a sensitivity ratio image of 1:16 is obtained.

[0099] 5. Single-frame dynamic range DXG (intra-scene DXG, iDXG)

[0100] For ease of distinction, in the embodiment of the present application, the method of fusing two frames acquired through the DXG method into one frame in the image sensor is referred to as iDXG.

[0101] 6. RAW images (or RAW domain images)

[0102] A raw image is a data stream processed by the image sensor of a digital camera, scanner, or film scanner. A RAW image contains the original image information and has not undergone nonlinear processing during image signal processing (ISP).

[0103] 7. HDR10 video

[0104] HDR10 video is configured according to static metadata. For example, the HDR10 PQ conversion curve is fixedly mapped according to the display's baseline brightness. The HDR10 video has a bit depth of 10 bits. This static metadata can meet the definition in SMPTE ST2086 or other standards.

[0105] 8. HDR10+ video

[0106] HDR10+ is an improvement on HDR. It supports dynamic metadata, which means that HDR10+ can adjust or enhance image brightness, contrast, and color saturation based on different scenes in the video, so that each frame in the HDR10+ video has an independently adjusted HDR effect. The bit depth of HDR10+ video is 12 bits, and this dynamic metadata can meet the definitions in SMPTE ST2094 or other standards.

[0107] 9. Brightness scene

[0108] The brightness scene can also be referred to as the brightness level. In the embodiment of the present application, the brightness scene can be used to determine the exposure mode (DXG or SHDR) of the image acquisition. Among them, the brightness scene can include: high-light scene, medium scene and low-light scene, etc.

[0109] For example, brightness scenes can correspond to different brightness ranges, and the device can distinguish different brightness levels based on the intensity of light reflected by the subject. For example, the brightness range corresponding to a bright scene can be greater than 50,000 lux, the brightness range corresponding to a medium brightness scene can be 50,000 lux-10 lux, and the brightness range corresponding to a dark scene can be 10 lux-0 lux.

[0110] It should be noted that the brightness levels described in the embodiments of this application are not limited to the three above. Moreover, the brightness ranges corresponding to these three brightness scenes are only used as examples. The values ​​of the brightness ranges corresponding to different brightness scenes can also be other values, and the embodiments of this application do not limit this.

[0111] For example, as shown in FIG3 , it is a schematic structural diagram of an electronic device 100 provided in an embodiment of the present application.

[0112] 3 shows a schematic structural diagram of an electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0113] It should be understood that the structure illustrated in the embodiments of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0114] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0115] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0116] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0117] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0118] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0119] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0120] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0121] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0122] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0123] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with a network and other devices through wireless communication technologies. The wireless communication technologies may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0124] The audio module 170 is used to convert digital audio information into analog audio signals for output, and is also used to convert analog audio input into digital audio signals.

[0125] The speaker 170A, also called a "horn", is used to convert audio electrical signals into sound signals.

[0126] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals.

[0127] The microphone 170C, also called a "microphone" or "speaker", is used to convert sound signals into electrical signals. The electronic device 100 may be provided with at least one microphone 170C.

[0128] The headphone jack 170D is used to connect a wired headphone.

[0129] The sensor module 180 may include one or more sensors, which may be of the same type or different types. The sensor module 180 may include a pressure sensor, a gyroscope sensor, an acceleration sensor, a distance sensor, a proximity sensor, a fingerprint sensor, a touch sensor, an ambient light sensor, and the like.

[0130] The buttons 190 include a power button, a volume button, and the like. The buttons 190 may be mechanical buttons or touch buttons. The electronic device 100 may receive key inputs and generate key signal inputs related to user settings and function control of the electronic device 100.

[0131] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), organic light-emitting diode (OLED), active-matrix organic light-emitting diode or active-matrix organic light-emitting diode (AMOLED), flexible light-emitting diode (FLED), Mini LED, Micro LED, Micro-OLED, quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N display screens 194, where N is a positive integer greater than 1.

[0132] Electronic device 100 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0133] The electronic device 100 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0134] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy. The video codec is used to compress or decompress digital video. The NPU is a neural network (NN) computing processor that quickly processes input information by drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, and can also continuously self-learn.

[0135] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0136] The internal memory 121 can be used to store computer executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image and video playback function, etc.). The data storage area can store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.).

[0137] In some embodiments, the internal memory 121 may store code related to automatic exposure control (AEC) and exposure mode switching methods. The processor may implement the automatic exposure control process by running the AEC-related code, and may implement the exposure mode switching of the image sensor in the camera by running the exposure mode switching method-related code.

[0138] In some embodiments, the AEC module may include an AEC algorithm module and an AEC statistics module. The AEC statistics module is configured to perform statistical analysis on parameters in the captured image, such as image brightness. The AEC algorithm module may automatically adjust the camera's exposure parameters based on the statistical results and may also estimate ambient light brightness based on the statistical results.

[0139] The electronic device 100 can realize the function of acquiring images through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0140] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image or video. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0141] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then passed to the ISP for conversion into a digital image or video signal. The ISP outputs the digital image or video signal to the DSP for processing. The DSP converts the digital image or video signal into an image or video signal in a standard format such as RGB or YUV.

[0142] In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1. For example, in some embodiments, the electronic device 100 may use the N cameras 193 to acquire images with multiple exposure coefficients. Then, in video post-processing, the electronic device 100 may synthesize an HDR image based on the images with multiple exposure coefficients using a High Dynamic Range (HDR) technique.

[0143] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. This allows electronic device 100 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0144] The NPU is a neural network (NN) computing processor that rapidly processes input information and continuously self-learns by drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain. The NPU enables intelligent cognitive applications in electronic device 100, such as image recognition, face recognition, speech recognition, and text comprehension.

[0145] The SIM card interface 195 is used to connect a SIM card. A SIM card can be connected to and disconnected from the electronic device 100 by inserting or removing it from the SIM card interface 195. The electronic device 100 may support one or N SIM card interfaces, where N is a positive integer greater than 1. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, or embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0146] For example, the software system of the electronic device 100 can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system with a layered architecture is used as an example to illustrate the software structure of the electronic device 100.

[0147] For example, as shown in FIG4 , it is a software structure block diagram of an electronic device 100 provided in an embodiment of the present application.

[0148] In some embodiments, a layered architecture divides software into several layers, each with distinct roles and divisions of labor. Layers communicate with each other via software interfaces. In embodiments of the present application, the system may specifically include, from top to bottom, an application layer, a hardware abstraction layer, and a kernel layer. Between the application layer and the hardware abstraction layer may also include an application framework layer, system libraries, a runtime, and the like.

[0149] The application layer may include a series of application packages. As shown in FIG4 , the application package may include applications (or applications) such as camera, gallery, music, video, and call.

[0150] The hardware abstraction layer (HAL) shields the differences between different hardware devices and provides a standard interface for the system. As shown in Figure 4, the HAL transmits data to the kernel layer through a standard HAL interface and receives data uploaded by the kernel layer. The HAL can contain multiple library modules, each of which implements a set of interfaces for a specific type of hardware component, such as a wireless fidelity (Wi-Fi) / Bluetooth module or a camera module. When the application framework layer's application program interface requests access to device hardware, the system loads the corresponding library module for that hardware.

[0151] In some embodiments, the hardware abstraction layer may further include a camera module. The camera module may include an automatic exposure control (AEC) module and an image processing pipeline. The AEC module may be used to implement automatic exposure control. Specifically, the AEC module may obtain exposure parameters from a system configuration file and configure the exposure parameters to the image sensor. The image sensor captures images according to the exposure parameters. The AEC statistics module in the AEC module may statistically analyze the parameters in the captured image, such as image brightness. The AEC module may also estimate the dynamic range compression gain (adrc gain) based on the above-mentioned exposure parameters and image brightness used when capturing the image. The exposure mode control module may be used to control the exposure mode of the camera's image sensor according to the estimated ambient light measurement. Specifically, in bright light scenes, the image sensor may be controlled to operate in a low-sensitivity mode, thereby extending the exposure time so that the exposure time meets the exposure time requirements for shooting video.

[0152] The kernel layer is the layer between hardware and software. It includes at least the display driver, camera driver, sensor driver, sensor front-end processor, image processor front-end, and image processor back-end. The exposure mode control module can control the exposure mode of the camera's image sensor through the camera driver, and the AEC module can configure exposure parameters for the camera through the camera driver.

[0153] The following describes the workflow of the software and hardware of the electronic device 100 in conjunction with a video recording scenario.

[0154] When the touch sensor receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, touch operation timestamp, and other information). The raw input event is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a single-click operation and the control affected by the single-click operation is the control of the camera application icon, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer, and captures the video image through the camera 193. The display screen 194 displays the camera preview interface, and the captured video image is displayed in the preview interface.

[0155] It should be understood that since dynamic range, strobe, ambient brightness, etc. all have an impact on image quality, an HDR solution adapted to the actual shooting situation needs to be used for image exposure and processing in different scenarios. In order to effectively expand the dynamic range and obtain high-quality video images in video recording scenarios, the video processing method provided in the embodiment of the present application establishes a correspondence between specific influencing factors and exposure methods, and then seamlessly switches between multiple HDR solutions according to the actual conditions of the influencing factors during the video shooting process, thereby realizing a processing method adapted to the high dynamic range influencing factors and effectively improving the extended dynamic range in video recording scenarios.

[0156] In order to better understand the video processing method provided in the embodiments of the present application, the differences between the several exposure methods involved in the present application and the situations in which the several exposure methods are used are introduced below.

[0157] Exemplarily, as shown in FIG5 , there is a timing diagram of the image acquisition process using the SHDR exposure mode, binning exposure mode, and DXG exposure mode, respectively, provided in an embodiment of the present application.

[0158] SHDR obtains long and short frame images by changing the shutter speed (that is, the exposure conditions), which can save the computing power of subsequent image reading and processing. However, when the exposure time is long, SHDR is prone to causing ghosting in the image, so it is suitable for scenes with short exposure time (high ambient brightness).

[0159] Binning exposure is a single-frame exposure method. Compared to DXG exposure, its exposure time is longer, for example, it can be longer than the long exposure time of SHDR exposure. Taking the several exposure methods shown in Figure 5 as an example, the exposure time corresponding to binning exposure can be the longest. The specific exposure time value can be set as needed and is not limited in this embodiment of the application.

[0160] The DXG exposure method simultaneously exposes two gain signals (conversion gain signal and analog gain signal) to obtain two frames of images with different gains (conversion gain or analog gain). Although the image acquisition process is relatively complicated, this method does not cause ghosting problems and has a short exposure time. Therefore, this exposure method can be used in scenes with medium and low ambient brightness.

[0161] Considering the characteristics of each of the aforementioned exposure methods, binning single-frame exposure is suitable for scenes with a narrow dynamic range, where the image's brightness and darkness differ minimally. For scenes with a wide dynamic range, further determination can be made regarding flicker, known as flicker detection. If flicker is present, binning is used, as its longer exposure time effectively mitigates the flicker caused by the periodic fluctuations of AC power. If flicker is absent, the ambient brightness can be further adjusted to determine whether to use SHDR or DXG exposure. SHDR is preferred for high ambient brightness, while DXG is preferred for low ambient brightness.

[0162] For example, Table 1 below shows examples of exposure methods corresponding to different influencing factors:

[0163] Table 1

[0164] It should be noted that the influencing factors described above are only examples. In actual applications, the types of influencing factors can be flexibly set as needed. For example, in some cases, only the dynamic range and ambient brightness in the shooting scene can be considered, while ignoring the flicker factor. The embodiments of the present application are not limited to this.

[0165] The video processing methods provided in the embodiments of the present application can be applied to a variety of scenarios requiring HDR processing, and can be particularly applied to HDR processing scenarios during video recording. For example, as shown in Figures 6A to 6D , there are schematic diagrams of graphical user interfaces (GUIs) that may be involved in implementing some of the video processing methods provided in the embodiments of the present application. For ease of understanding, the GUI interface in a mobile phone is used as an example for description.

[0166] It should be noted that in the embodiments of the present application, there are multiple ways to trigger the video processing process. For example, the video processing process can be triggered by inputting a click operation on the smart HDR switch control set in the mobile phone; for example, when the mobile phone is turned on, the smart HDR function can be turned on by default, that is, the video processing process can be triggered when the mobile phone is turned on, etc.

[0167] In one possible implementation, the switch control of Smart HDR can be set in the settings application of the mobile phone. For example, as shown in Figure 6A, it is a schematic diagram of a settings interface including a Smart HDR switch control. The settings interface 601 includes a shooting parameter setting area, a video parameter setting area, and a general settings area. Among them, the shooting parameter setting area can specifically include a photo ratio setting bar, a voice-controlled shooting function setting bar, and a smile capture setting bar; the video parameter setting area can specifically include a video resolution setting bar, a video frame rate setting bar, and a Smart HDR setting bar; the general settings area can specifically include a reference line function setting bar, a level function setting bar (not shown in the figure), and a timed shooting function setting bar (not shown in the figure). Among them, the Smart HDR setting bar includes a switch control for Smart HDR. When the user turns on the Smart HDR function through the switch control, the mobile phone can intelligently enable HDR mode according to the shooting scene during video recording.

[0168] In another possible implementation, the switch control of Smart HDR can also be set in the camera application of the mobile phone. For example, as shown in Figures 6B and 6C, it is a GUI schematic diagram that may be involved when turning on the Smart HDR function through the camera application. Specifically, when the mobile phone receives the user's opening operation for the camera application (such as a click operation on the camera icon in the main interface or lock screen interface), the camera main interface 603 as shown in Figure 6B can be displayed. Exemplarily, the camera main interface 603 may include an image preview area, a mode setting area, and shooting controls. Among them, the mode setting area 605 may include a viewfinder, an album icon, a shooting control 604, and a rotation control.

[0169] The viewfinder is used to capture the preview image and display it in real time. The album icon is used to quickly access the album. When the phone detects that the user has clicked the album icon, the captured photos or videos can be displayed on the touch screen. The shooting control 604 is used for shooting or recording. When the phone detects that the user has clicked the shooting control, the phone performs the shooting operation and saves the captured photo; or, when the phone is in recording mode, when the user clicks the shooting control, the phone performs the recording operation and saves the recorded video. The camera rotation control is used to control the switching between the front camera and the rear camera.

[0170] In addition, the camera main interface 603 also includes functional controls for setting shooting modes, such as the aperture shooting mode, night shooting mode, portrait shooting mode, video mode, photo mode, and more as shown in Figure 6B. As shown in Figure 6C, the more mode may also include slow motion mode, panoramic mode, black and white art mode, dual view recording mode, filter mode, smart HDR mode, etc.

[0171] In some embodiments, when a user shoots a video through a camera application, a specific operation can be input for the Smart HDR icon shown in FIG6C to enable the Smart HDR function.

[0172] In some embodiments, when Smart HDR is on, artificial intelligence (AI) prompts can be enabled during video preview (before recording begins). For example, in a bright, high-dynamic-range scene, the corresponding exposure mode is SHDR. In this case, the preview interface may display a prompt: "Backlight HDR." Another example is in a low-brightness, high-dynamic-range scene, the corresponding exposure mode is DXG. When the ambient brightness falls below a certain threshold, the preview interface may display a prompt: "Night HDR."

[0173] It should be noted that the positions of the Smart HDR switch controls shown in Figures 6A to 6C are only examples. In actual applications, the Smart HDR switch controls are not limited to being set in the settings application and the camera application. In addition, the above embodiments only introduce the example of turning on the Smart HDR function on the user's mobile phone, but in other embodiments, there may be multiple ways to turn on Smart HDR. For example, when the mobile phone is turned on, the Smart HDR function may be turned on by default; for example, when the mobile phone detects a preset trigger event, the Smart HDR function may be automatically turned on. In some other embodiments, if the electronic device does not support the Smart HDR function (such as due to hardware limitations), the control icon of the HDR switch may be grayed out. This application is not limited to this.

[0174] In some embodiments, when the electronic device detects that the scene brightness in the current shooting environment is significantly different and the Smart HDR function is turned off, it can also prompt the user to turn on the Smart HDR function in various ways. For example, when a user is shooting a cave against the light, the brightness level of the backlit part is higher, while the brightness level of the cave part is lower, and the difference between the two is large. At this time, the shooting device can prompt the user to turn on the Smart HDR function through (artificial intelligence, AI), such as displaying a prompt message as shown in Figure 6D on the shooting interface: "The current shooting picture has a large difference in brightness and darkness. You can click the Smart HDR function in the "More" option to improve the image quality."

[0175] In some embodiments, when the Smart HDR function is turned on, the mobile phone can perform exposure and image processing according to the HDR scheme that matches the current actual shooting situation during the shooting process to obtain a video picture with a high dynamic range. The shooting picture with Smart HDR turned on can more clearly show the details of the bright and dark areas, and avoid the situation where the details are blurred due to excessive brightness or too low brightness.

[0176] It is worth noting that the preset HDR scheme mentioned here refers to the HDR scheme determined by the mobile phone according to factors such as the dynamic range, stroboscopic state, and brightness of the shooting environment corresponding to the actual shooting environment. The difference between different HDR schemes mainly lies in the different exposure methods and image processing methods. In actual applications, the mobile phone can seamlessly switch between multiple supported exposure methods according to the actual shooting conditions. Combined with the introduction above, the exposure methods available in the embodiment of the present application may include, for example: SHDR, DXG, and binning. It should be noted that in different scenarios, the exposure methods available for selection may be different. For example, due to limitations on the hardware performance of the device, for devices that do not support the DXG exposure method, the available exposure methods may also include only SHDR and binning.

[0177] In some embodiments, the underlying implementation of the Smart HDR switch control may include: the HAL reports multi-state video HDR switching capabilities to differentiate between products. The app configures the "Smart HDR" switch based on the reported capabilities. The app determines the switch status (on / off / grayed out) based on the mode and scenario. Based on the switch status, the app sends commands to the HAL to enable and disable HDR capabilities. When disabled, only binning is used. The HAL reports the AEC HDR status tag to support AI prompts and recommendations.

[0178] The above introduces the principles of different exposure methods and how to enable Smart HDR. In order to better understand the video processing method provided in the embodiment of this application, the specific functional modules for executing the method and the underlying implementation process of the method are introduced below.

[0179] For example, as shown in FIG7 , it is a schematic diagram of data flow interaction between relevant modules when implementing a video processing method provided in an embodiment of the present application.

[0180] As shown in Figure 7, the electronic device 100 includes an image sensor (camera sensor), an automatic exposure control (AEC) module, a sensor front-end processor (or fusion module), an image processor front-end, a multi-camera smooth switching module, an image processor back-end and a local tone mapping (LTM) module included therein, etc.

[0181] The image sensor may correspond to the camera 193 shown in FIG3 , and is mainly used to capture video images. In some embodiments, the image sensor captures image frames by exposure, and when the reflected light of the subject passes through the lens, it converges on the image sensor; the image sensor may convert the light signal into an analog electrical signal, and transmit the analog electrical signal to the fusion module. Among them, the image sensor outputs the original digital image it captures, that is, the original (RAW) image. In some embodiments, in the initial stage of shooting (when the camera is turned on), the image sensor may capture images in accordance with the default exposure mode, and subsequently, after determining the exposure mode that is suitable for the current shooting conditions, the image sensor may shoot in a target mode under the instruction of the AEC module. Among them, the default exposure mode may be, for example, a binning exposure mode, but the embodiments of the present application are not limited to this.

[0182] In some embodiments, when capturing images using different exposure methods, the image sensor may output different numbers of frames for the same scene at the same moment. For example, when shooting using the binning exposure method, the image sensor outputs a single frame (such as the long exposure frame in Figure 1); when shooting using the DXG exposure method, the image sensor outputs two frames.

[0183] In addition, in embodiments of the present application, the image sensor can also perform image fusion. For example, in DXG exposure mode, after acquiring two frames of image, the image sensor can fuse the two frames into a single frame. The image sensor can also be used to adjust image parameters (such as image size and bit depth), such as adjusting the size of images acquired by different exposure modes to a consistent target size; or adjusting the bit depth of images acquired by different exposure modes to a consistent target bit depth.

[0184] The sensor front-end processor (also known as the fusion module) is used to fuse the RAW image frames collected by the image sensor, and can achieve single-frame input and single-frame output, or two-frame input and single-frame output.

[0185] For example, the SHDR exposure method can capture two frames with different exposure times. Therefore, when shooting with the SHDR exposure method, the initial image input from the image sensor to the fusion module can be two frames with a certain frame rate (such as 30fps) and different exposure times, that is, dual-frame mode. After acquiring the initial image, the fusion module fuses the two frames with different exposure times into a single frame and adjusts the parameters of the fused image to the preset target parameters, such as adjusting the initial input image format RAW10 to the target format RAW14.

[0186] For another example, the DXG exposure method is to obtain two frames of images based on one exposure. Therefore, when shooting with the DXG exposure method, one exposure of the image sensor can correspond to inputting two frames of initial images to the fusion module, which is also a dual-frame mode. The two initial image frames can be images read respectively through high conversion gain and low conversion gain, or the two frames of images can also be images read respectively through high analog gain and low analog gain, or the two frames of images can also be images after superimposing the images read through conversion gain and analog gain. The initial format of the input image can be RAW10, for example, and the frame rate can be 30fps, for example. After the fusion module obtains the initial image, the two frames of images read out with different gains are fused into one frame, and the parameters of the fused image can be adjusted to the preset target parameters, such as adjusting the input initial format RAW10 to the target format RAW14.

[0187] For example, binning exposure captures a single image frame. When using binning exposure, a single exposure from the image sensor corresponds to a single initial image input to the fusion module. This is known as single-frame mode. After the fusion module acquires the initial image, it doesn't need to perform image fusion. However, it can adjust image parameters as needed, such as converting the initial input format of RAW10 to the target format of RAW14.

[0188] It should be noted that the purpose of format adjustment of the initial image is to unify the images obtained by different exposure methods into the same format (size, bit depth, etc.) so that the screen size will not jump when the video is played later.

[0189] In some embodiments, if the image sensor fuses the image, a single frame can be input into the fusion module. In this case, the fusion module does not need to perform fusion processing on the image, nor does it need to adjust the image size. Instead, a single frame is output to the image processor front end for further processing.

[0190] The image processor front end (IPF) is used to process the image to obtain an initial preview image. The IPF can specifically include a statistics module (STATS) and a GTM module. The GTM module is used to globally brighten dark areas of the image to improve image quality.

[0191] In some embodiments, the image output by the image processor front end can be divided into the following two transmission paths:

[0192] (1) One transmission path is from the front end of the image processor to the back end of the image processor for back end processing. The back end of the image processor may include a local tone mapping (LTM) module, which is used to locally brighten the dark part of the image. Afterwards, the back end of the image processor continues to transmit the image processed by the back end to the perception module, which can obtain the ambient brightness based on the image and indicate the ambient brightness to the automatic exposure control AEC module. Then, the AEC module determines the exposure mode according to the ambient brightness and instructs the image sensor to switch the exposure mode. Alternatively, the perception module can obtain the ambient brightness based on the image, determine the exposure mode according to the ambient brightness, and then indicate the exposure mode to the AEC module. Afterwards, the AEC module determines the exposure mode according to the ambient brightness and instructs the image sensor to switch the exposure mode.

[0193] In some embodiments, the way in which the perception module obtains the ambient brightness based on the image may include: comprehensively judging the ambient brightness based on the current state of the image sensor, exposure parameters, and brightness channel information of the image frame.

[0194] It's important to note that while the AEC module can be used to control exposure, it doesn't inherently determine whether SHDR or DXG exposure is enabled. Instead, it relies on the ambient brightness detection results from the perception module. When ambient brightness is greater than X, SHDR exposure is used, and when ambient brightness is less than Y, DXG exposure is used. X and Y can be set based on actual needs, but to prevent ping-pong, a dual-threshold design can be used, where X is greater than Y, for example, X = 500 lux and Y = 300 lux.

[0195] (2) Another transmission process is to transmit from the front end of the image processor to the multi-camera smooth switching module, which is used to realize the process of smooth switching of images from one camera to another camera using a multi-camera smooth switching algorithm (such as the SAT (spatial alignment transform) algorithm). After the multi-camera smooth switching module performs smooth image switching, the processed image can be transmitted to the back end of the image processor. The back end of the image processor can locally brighten the dark part of the image through the LTM module included therein. Afterwards, a preview video image can be output at a specific frame rate (such as 30fps), or a video type (vedio) file can be generated and stored at a specific frame rate (such as 30fps).

[0196] It should be understood that the processes of the image processor front-end and the image processor back-end performing front-end processing and back-end processing on the image respectively can refer to the existing process and will not be described in detail here.

[0197] It should be noted that, in the embodiment of the present application, the smooth switching operation performed by the multi-camera smooth switching module is mainly used to realize the switching between images captured by the HDR camera and the non-HDR camera respectively. Specifically, when there is a switch between the HDR camera and the non-HDR camera during the video shooting process, the multi-camera smooth switching module can be used to smoothly switch the image of the HDR camera to the image of the non-HDR camera, or smoothly switch the image of the non-HDR camera to the image of the HDR camera, that is, to realize the smooth switching between the images captured by the two different types of cameras, so as to ensure the smooth playback of the video and avoid the problem of image jump caused by direct switching.

[0198] It should also be noted that the HDR camera in the embodiment of the present application may be a main camera, and the non-HDR camera may be a non-main camera, such as a wide-angle camera. Optionally, the HDR camera and non-HDR camera in the embodiment of the present application may not specifically refer to two different cameras, but may refer to a camera using an HDR mode and a camera not using an HDR mode. For example, a camera may be considered an HDR camera when it uses an HDR solution to capture images, and may be considered a non-HDR camera when it does not use an HDR solution to capture images.

[0199] For example, as shown in FIG8 , a schematic flow chart of a video processing method provided in an embodiment of the present application is provided. The execution subject of the method may be the electronic device 100 described above, and may be implemented by various functional modules in the electronic device. The method may include the following steps:

[0200] S801: The AEC module determines dynamic range information and obtains a first result.

[0201] The dynamic range information here may be a dynamic range or a dynamic range compression gain. The dynamic range compression gain may be calculated based on the dynamic range of the current image and a preset standard dynamic range.

[0202] In some embodiments, before executing step S801, the video processing method provided in the embodiments of the present application may further include: upon receiving a first operation input by a user for starting shooting, in response to the first operation, the AEC module of the electronic device may instruct the image sensor to capture an image according to a default exposure method; after the AEC module obtains the image captured by the image sensor, the dynamic range of the current image may be obtained through a histogram. The specific process of obtaining the dynamic range of the current image based on the histogram can be referred to in the existing process and will not be described in detail in the embodiments of the present application.

[0203] In some embodiments, during video capture, as the image sensor captures more and more frames of images, the AEC module can obtain the current dynamic range based on the multiple frames of images, such as obtaining the average dynamic range based on the histogram corresponding to each frame of image.

[0204] S802: Determine whether the first result is greater than a preset first threshold.

[0205] Among them, the first threshold can be flexibly set according to actual conditions, and the embodiments of the present application do not limit this.

[0206] In some embodiments, the current dynamic range level can be determined based on the correlation between the first result and the first threshold. For example, when the first result is greater than the first threshold, it indicates that the current image has a large difference in brightness and darkness, and the current dynamic range can be determined to be a high dynamic range. When the first result is not greater than the first threshold, it indicates that the current image has a small difference in brightness and darkness, and the current dynamic range can be determined to be a medium-low dynamic range.

[0207] It should be noted that different dynamic ranges can correspond to different exposure methods. For example, when it is a high dynamic range, the exposure method is determined to be binning. When it is a medium or low dynamic range, the exposure method needs to be further determined based on the flicker situation, ambient brightness and preset judgment method.

[0208] S803: When the first result is not greater than the first threshold, the AEC module instructs the image sensor to adopt a binning exposure mode.

[0209] S804: When the first result is greater than the first threshold, the AEC module determines whether the image has stroboscopic motion and obtains a second result.

[0210] S805 , when the second result indicates that there is stroboscopic light, the AEC module instructs the image sensor to adopt a binning exposure mode.

[0211] In some embodiments, during video capture, flicker may occur in the captured image due to the use of AC power by all devices. Based on the flickering results, it can be determined whether to use binning exposure. If flicker is present, binning exposure is used. If flicker is absent, SHDR or DXG exposure can be further determined based on the ambient brightness.

[0212] It should be noted that the exposure time corresponding to the binning exposure mode in the embodiment of the present application can be greater than or equal to the AC power cycle. For example, when the AC power frequency is 50 Hz, the exposure time of the binning exposure mode can be 10 ms.

[0213] S806: When the second result indicates that there is no flicker, the perception module determines the ambient brightness.

[0214] In some embodiments, the perception module may obtain an image captured by an image sensor and obtain the ambient brightness based on the image.

[0215] S807: When the ambient brightness is greater than the second threshold, the AEC module instructs the image sensor to adopt the SHDR exposure mode.

[0216] Among them, the second threshold can be flexibly set according to actual conditions, and the embodiments of the present application do not limit this.

[0217] S808 , when the ambient brightness is less than the third threshold and greater than the fourth threshold, the AEC module instructs the image sensor to adopt the DXG exposure mode.

[0218] The third threshold value is smaller than the second threshold value. The third threshold value and the fourth threshold value can be flexibly set according to actual conditions, and are not limited in this embodiment of the present application.

[0219] S809: When the ambient brightness is less than a fourth threshold, the AEC module instructs the image sensor to adopt a binning exposure mode.

[0220] According to the video processing method provided in the embodiment of the present application, by seamlessly switching between multiple types of HDR processing schemes according to changes in factors such as ambient brightness, required dynamic range, and flicker detection, it is possible to use an HDR scheme adapted to the actual shooting situation and image quality requirements for image processing, thereby effectively expanding the dynamic range in the recording scene and improving the image quality in the recording scene.

[0221] It should be noted that, in other embodiments, if the image sensor of the electronic device does not support certain exposure modes (such as DXG), then the video processing method provided in the embodiment of the present application can also switch between SHDR and binning exposure modes. In this case, the exposure modes corresponding to different influencing factors can be as shown in Table 2:

[0222] Table 2

[0223] Alternatively, in some other embodiments, if the image sensor of the electronic device does not support certain exposure modes (such as SHDR), then the video processing method provided in the embodiment of the present application can also switch between DCG and binning exposure modes. In this case, the exposure modes corresponding to different influencing factors can be as shown in Table 3:

[0224] Table 3

[0225] When switching between different exposure modes, a seamless switching effect can be achieved. The specific means of achieving the seamless switching effect will be described in detail below and will not be elaborated here.

[0226] For example, as shown in Figure 9, a schematic flow chart of another video processing method provided by an embodiment of the present application is shown. Figure 9 shows the interaction process between various functional modules, which may specifically include the following steps:

[0227] S901: The automatic exposure control module sends first instruction information to the image sensor.

[0228] In some embodiments, the automatic exposure module may preset a default exposure mode, which may be binning, for example.

[0229] In some embodiments, before executing step S901, the video processing method provided in embodiments of the present application may further include: upon receiving a first user input for starting shooting, in response to the first operation, the AEC module of the electronic device may instruct the image sensor to capture an image using a default exposure method. Exemplarily, the first operation may be for starting video recording, such as clicking on a video image in the camera main interface as shown in FIG6C .

[0230] S902: The image sensor captures an image using a default exposure mode in response to the first instruction information.

[0231] It should be noted that, in the embodiment of the present application, the image sensor is only used to capture original images, such as images in RAW10 format, without performing image fusion processing or adjusting the image size.

[0232] S903: The image sensor sends the first image to the fusion module.

[0233] The first image may be an original image captured by an image sensor, such as an image in RAW10 format.

[0234] Optionally, when the default exposure mode is SHDR or DXG, capturing a dual-frame image, step S904 may be performed, where the fusion module performs multi-frame fusion on the first image. When the image sensor captures a single-frame image using a binning exposure mode, the image can be input to the fusion module in single-frame mode. In this case, the fusion module does not need to perform fusion processing and can only adjust image parameters.

[0235] S905: The fusion module sends the second image to the image processor front end.

[0236] The second image may be an image processed by a fusion module. The fusion module may be used to perform fusion processing on the acquired images and may also adjust the image size according to the location.

[0237] S906: The image processor front end sends the third image to the image processor back end.

[0238] The third image may be an image obtained by front-end processing of the second image by the front-end of an image processor.

[0239] In some embodiments, after the image processor front end performs front-end processing on the image, it can output a third image according to a two-way transmission process. One transmission process is to transmit the third image to the multi-camera smooth switching module, and then the multi-camera smooth switching module transmits it to the image processor back end. After the image processor back end receives the image, it can execute the following step S907A; the other transmission process is to transmit the third image to the image processor back end, and then the image processor back end sends the image to the perception module, that is, execute the next step S907B.

[0240] S907A: The image processor backend performs backend processing to obtain the preview video and the stored video file.

[0241] S907B, the image processor backend sends the fourth image to the perception module.

[0242] The fourth image may be an image obtained after the image processor performs back-end processing on the third image.

[0243] S908: The perception module obtains the ambient brightness based on the fourth image.

[0244] S909: The perception module sends the ambient brightness to the automatic exposure control module.

[0245] S910: The automatic exposure control module determines an exposure mode according to preset influencing factors, where the influencing factors include ambient brightness.

[0246] S911: The automatic exposure control module sends second instruction information to the image sensor.

[0247] The second instruction information is used to instruct the image sensor to capture an image using a target exposure mode.

[0248] S912: The image sensor continues to use the default exposure mode or other exposure modes in response to the second indication information.

[0249] After the image sensor captures the original image (such as the RAW10 image) using the target exposure mode, the real-time exposure mode can be determined according to the above steps S903 to S911, and the image can be captured and processed using the target exposure mode.

[0250] According to the video processing method provided in the embodiment of the present application, by seamlessly switching between multiple types of HDR processing schemes according to changes in factors such as ambient brightness, required dynamic range, and flicker detection, it is possible to use an HDR scheme adapted to the actual shooting environment and image quality requirements for image processing, thereby effectively expanding the dynamic range in the recording scene and improving the image quality in the recording scene.

[0251] It should be noted that in the video processing method provided in the embodiments of this application, when the image sensor has the functions of image fusion and image resizing, the image sensor can perform the operations of fusing multiple frames of images in the DXG exposure mode and resizing the images. The following describes the video processing process under this method with reference to the accompanying drawings.

[0252] For example, as shown in FIG10 , it is a schematic diagram of data flow interaction between relevant modules when implementing another video processing method provided in an embodiment of the present application.

[0253] In some embodiments, when video shooting is turned on, the automatic exposure control module can send exposure indication information to the image sensor, instructing the image sensor to capture an image using a default exposure mode (such as binning exposure mode). The image sensor collects images (such as RAW10) according to the default exposure mode, fuses the images (under DXG or SHDR exposure mode) and adjusts image parameters (such as image size, bit depth, etc.), and then outputs the image (such as RAW14) to the fusion module. The fusion module transmits the acquired image to the image processor front end, which performs front-end processing on the image and then outputs the image according to two transmission paths: (1) One transmission path is that the image processor front end transmits the image to the image processor back end, which can perform back-end processing on the image and then transmit the image to the perception module. The perception module can obtain the ambient brightness based on the image and transmit the ambient brightness to the decision module. The decision module is used to determine the target exposure mode based on the ambient brightness, wherein the target exposure mode is an exposure mode that matches the actual conditions such as the current dynamic range, strobe state, and ambient brightness; then, the decision module sends exposure mode indication information to the image sensor, instructing the image sensor to switch to the target exposure mode. (2) Another transmission path is that the image processor front end transmits the image to the multi-camera smooth switching module. Optionally, the multi-camera smooth switching module can use a multi-camera smooth switching algorithm to achieve smooth switching of images from one camera to another. After the multi-camera smooth switching module performs smooth image switching, it can transmit the processed image to the image processor back end. Afterwards, the image processor back end can locally brighten the dark part of the image through the LTM module it includes, and then output a preview video image at a specific frame rate (such as 30fps), or generate and store a video type (vedio) file at a specific frame rate (such as 30fps).

[0254] It should be understood that the processes of the image processor front-end and the image processor back-end performing front-end processing and back-end processing on the image respectively can refer to the existing process and will not be described in detail here.

[0255] It should be noted that, in the embodiment of the present application, the smooth switching operation performed by the multi-camera smooth switching module is mainly used to realize the switching between images captured by the HDR camera and the non-HDR camera respectively. Specifically, when there is a switch between the HDR camera and the non-HDR camera during the video shooting process, the multi-camera smooth switching module can be used to smoothly switch the image of the HDR camera to the image of the non-HDR camera, or smoothly switch the image of the non-HDR camera to the image of the HDR camera, that is, to realize the smooth switching between the images captured by the two different types of cameras, so as to ensure the smooth playback of the video and avoid the problem of image jump caused by direct switching.

[0256] It should also be noted that the HDR camera in the embodiment of the present application may be a main camera, and the non-HDR camera may be a non-main camera, such as a wide-angle camera. Optionally, the HDR camera and non-HDR camera in the embodiment of the present application may not specifically refer to two different cameras, but may refer to a camera using an HDR mode and a camera not using an HDR mode. For example, a camera may be considered an HDR camera when it uses an HDR solution to capture images, and may be considered a non-HDR camera when it does not use an HDR solution to capture images.

[0257] According to the video processing method provided in the embodiment of the present application, by seamlessly switching between multiple types of HDR processing schemes according to changes in factors such as ambient brightness, required dynamic range, and flicker detection, it is possible to use an HDR scheme adapted to the actual shooting environment and image quality requirements for image processing, thereby effectively expanding the dynamic range in the recording scene and improving the image quality in the recording scene.

[0258] For example, as shown in Figure 11, a schematic flow chart of another video processing method provided by an embodiment of the present application is provided. Figure 11 shows the interaction process between various functional modules, which may specifically include the following steps:

[0259] S1101: The automatic exposure control module sends first instruction information to the image sensor.

[0260] In some embodiments, the automatic exposure module may preset a default exposure mode, which may be binning, for example.

[0261] In some embodiments, before executing step S1101, the video processing method provided in the embodiments of the present application may further include: upon receiving a first user input for starting shooting, in response to the first operation, the AEC module of the electronic device may instruct the image sensor to capture an image using a default exposure method. Exemplarily, the first operation may be for starting video recording, such as clicking on a video image in the camera main interface as shown in FIG6C .

[0262] S1102: In response to the first instruction information, the image sensor captures an image using a default exposure method and performs image processing according to a preset strategy.

[0263] It should be noted that, unlike the embodiment of FIG. 9 above, the image sensor in the embodiment of the present application can have the ability to fuse images, adjust image size, and perform image processing according to a preset strategy. For example, when the default exposure mode is DCG, DAG, or DXG, and the image is acquired using a dual-frame module, the image sensor can fuse the first image of the dual frame. When the image sensor captures images using a binning exposure mode, the single-frame mode is input to the fusion module. In this case, the fusion module does not need to perform fusion processing and can only adjust the image size.

[0264] For example, for ease of description, the embodiment of the present application will obtain two frames of images in the image sensor through the DXG method and record the method of fusing them as iDXG. Referring to Figure 11, the method of fusing two frames of images by the image sensor may include: obtaining two frames of images with a preset sensitivity ratio, and then fusing the two frames of images to obtain an image with a larger dynamic range. Due to the limitations of the hardware performance of the image sensor, the sensitivity ratio of the two frames of images obtained by the iDXG method can be pre-set, such as iDXG1:4, iDXG1:8 or iDXG1:16 and other fixed values.

[0265] Different switching conditions can correspond to different sensitivity ratios. For example, in high-dynamic scenes, it can correspond to iDXG1:4 (that is, using iDXG exposure method, and the sensitivity ratio is 1:4); in extremely high-dynamic scenes, it can correspond to iDXG 1:16 (that is, using iDXG exposure method, and the sensitivity ratio is 1:16); in medium and low dynamic scenes, it can correspond to binning exposure method; in thermal escape scenes, it can correspond to binning exposure method.

[0266] Among them, the extremely high dynamic range, high dynamic range and medium-low dynamic range can be divided according to needs, and this application does not make specific limitations on each dynamic range.

[0267] In contrast, in the embodiment of Figure 9, when the fusion module performs dual-frame image fusion, the sensitivity ratio of the two frames can be flexibly set and is not limited to a few fixed values. Therefore, the fusion method of the embodiment of Figure 9 facilitates flexible adjustment of image brightness, while the fusion method of the embodiment of Figure 11 can achieve HDR processing of images while ensuring the performance of the image sensor.

[0268] S1103: The image sensor sends the fifth image to the image processor front end.

[0269] Specifically, the image sensor may first transmit the fifth image to the fusion module, and then the fusion module transparently transmits the fifth image to the image processor front end.

[0270] S1104: The image processor front end sends the sixth image to the image processor back end.

[0271] The sixth image may be an image obtained by performing front-end processing on the fifth image by the front-end of an image processor.

[0272] In some embodiments, after the image processor front end performs front-end processing on the image, it can output the sixth image according to a two-way transmission process. One transmission process is to transmit the sixth image to the multi-camera smooth switching module, and then the multi-camera smooth switching module transmits it to the image processor back end. After the image processor back end receives the image, it can execute the following step S1105A; the other transmission process is to directly transmit the sixth image to the image processor back end, and then the image processor back end sends the image to the perception module, that is, execute the next step S1105B.

[0273] S1105A: The image processor backend performs backend processing to obtain the preview video and the stored video file.

[0274] S1105B, the image processor backend sends the seventh image to the perception module.

[0275] S1106: The perception module obtains the ambient brightness based on the seventh image.

[0276] S1107, the perception module sends the ambient brightness to the decision module.

[0277] It should be noted that the decision module here may be an additional module, or may be a submodule in the automatic exposure control, and the embodiment of the present application does not limit this.

[0278] S1108: The perception module determines the target exposure method according to a preset strategy.

[0279] The preset strategy refers to a strategy for determining the target exposure mode based on the current dynamic range, whether there is flicker, ambient brightness, etc., which may correspond to the strategy for determining the target exposure mode described in the embodiment of FIG. 8 above.

[0280] S1109: The decision module sends second instruction information to the image sensor.

[0281] S1110 : The image sensor captures an image using a target exposure mode in response to the second instruction information.

[0282] After the image sensor captures the original image (such as the RAW10 image) using the target exposure mode, the real-time exposure mode can be determined according to the above steps S1103 to S1109, and the image can be captured and processed using the target exposure mode.

[0283] According to the video processing method provided in the embodiment of the present application, by seamlessly switching between multiple types of HDR processing schemes according to changes in factors such as ambient brightness, required dynamic range, and flicker detection, it is possible to use an HDR scheme adapted to the actual shooting environment and image quality requirements for image processing, thereby effectively expanding the dynamic range in the recording scene and improving the image quality in the recording scene.

[0284] In the video processing method provided in the embodiment of the present application, during video shooting, HDR mode and non-HDR mode can be switched between each other, wherein HDR mode is a video shooting mode processed by HDR technology, and non-HDR mode is a video shooting mode not processed by HDR technology. Generally speaking, the main camera and the secondary camera of an electronic device are included in a logical camera, wherein the main camera has HDR capability, and the secondary camera may have other capabilities (such as ultra-wide-angle capability) but not HDR capability. Therefore, when switching between HDR mode and non-HDR mode is required during video shooting, it can be achieved by switching between the main camera and the secondary camera.

[0285] For example, as shown in FIG12 , it is a schematic flowchart of a method for video processing in a scenario where a main camera and a secondary camera are switched, provided in an embodiment of the present application.

[0286] When the main camera and the secondary camera are switched, in order to reduce the image jump after the switch, the following methods can be used:

[0287] (1) In low-dynamic scenes, or scenes with flicker or thermal runaway, the exposure method used is binning. At this time, the system's dynamic range gain remains flat and the image effect does not jump.

[0288] (2) In high-dynamic scenarios where there is no flicker or thermal runaway, the main camera can capture images according to the target exposure method determined by the method described above, and the secondary camera uses the binning exposure method and maintains the maximum dynamic range, trying to be close to the dynamic range of the main camera to reduce image jumps.

[0289] As shown in FIG12 , in some embodiments, the main camera can capture images using various exposure modes, including SHDR, DXG, and binning. After capturing an image, the main camera can transmit the initial image (e.g., a RAW10 image) to the fusion module. When input into the fusion module in dual-frame mode, the fusion module fuses the two frames and transmits the fused image (RAW14 image) to the image processor front end 0.

[0290] The secondary camera may capture an image using a binning exposure method, and then transmit the captured initial image (eg, a RAW image) to the image processor front end 1 .

[0291] Image Processor Front-End 0 and Image Processor Front-End 1 can transmit their respective processed images to the multi-camera smooth switching module. This module can process the images corresponding to different cameras according to the multi-camera smooth switching algorithm, achieving smooth switching between images captured by two different types of cameras, ensuring smooth video playback and avoiding image jumps caused by direct switching.

[0292] Afterwards, the fusion module can continue to transmit the image to the image processor backend, which can perform stabilization on the image through the preview stream stabilization module (such as EIS2.0 and EIS3.0), and then further process the image through the image affine transformation module, color conversion matrix (CCM) module, and gamma transformation (Gamma) module. The specific processing process can be referred to the existing process and principles, which will not be repeated here.

[0293] After the back-end processing of the above-mentioned image processor, a preview image with a certain frame rate (such as 30fps) and a video storage file with a certain frame rate (such as 30fps) can be obtained.

[0294] According to the video processing method provided in the embodiment of the present application, by seamlessly switching between multiple types of HDR processing schemes according to changes in factors such as ambient brightness, required dynamic range, and flicker detection, it is possible to use an HDR scheme adapted to the actual shooting environment and image quality requirements for image processing, thereby effectively expanding the dynamic range in the recording scene and improving the image quality in the recording scene.

[0295] In the video processing method provided in the embodiment of the present application, multiple HDR capabilities can be supported, such as HDR10+ video capabilities, that is, electronic devices can apply HDR10+ technology for video recording or playback. In some embodiments, HDR10+ capabilities can mainly act on the back-end part of the image processor, including adjusting the color conversion matrix module to BT.2020, adjusting the gamma transform to the PQ conversion curve, generating dynamic metadata, etc. That is, HDR10+ technology can be used simultaneously with seamless switching of different exposure modes. Among them, the path for switching exposure modes using HDR10+ technology can be seen in Figure 9, which will not be described in detail here.

[0296] In some embodiments, when an electronic device supports multi-state HDR capabilities, a corresponding switch control can be set. In addition, when an electronic device supports other HDR technologies (for example, HDR10+), a switch control corresponding to HDR10+ can also be set. For electronic devices that do not support multi-state HDR or HDR10+, the switch control corresponding to multi-state HDR or HDR10+ on the electronic device can be grayed out.

[0297] The following describes the process of adjusting image parameters corresponding to different exposure modes by taking the adjustment of bit depth as an example with reference to the accompanying drawings.

[0298] For example, as shown in FIG13 , it is a schematic diagram of a method for adjusting image parameters provided in an embodiment of the present application.

[0299] In some embodiments, adjusting image parameters may include padding the low-order bits with zeros using Pad0, so that images corresponding to different exposure modes have the same bit depth. For example, as shown in FIG13 , an image corresponding to the iDCG 1:4 exposure mode has the low-order 2 bits padded with zeros, where 1:4 is the sensitivity ratio between two iDCG image frames with different gains; and an image corresponding to the binning exposure mode has the low-order 4 bits padded with zeros. After the low-order 0s are padded, the images obtained by the iDCG and binning exposure modes have the same bit depth.

[0300] For example, as shown in FIG14 , it is a schematic diagram of a method for adjusting image parameters provided in an embodiment of the present application.

[0301] In some embodiments, an image sensor configuration that supports seamless switching of multiple exposure modes (e.g., binning, DXG, and SHDR) can unify the bit depth, such as to RAW 14. For images with a bit depth less than 14 bits, they can be unified to RAW 14 by padding.

[0302] It should be noted that in the video processing methods provided in the embodiments of this application, the unified bit depth can be based on the bit depth of iDXG images. This is because, among the several exposure methods provided in this application, the iDXG exposure method corresponds to the highest bit depth of images. In actual applications, when other exposure methods are used, the unified bit depth can be flexibly set based on the type of exposure method, and this embodiment of the application does not limit this.

[0303] Taking 14 bits as an example when the bit depth is unified, the method for adjusting the bit depth can be shown in Figure 13, including: padding the image captured by binning exposure from 10 bits to 14 bits; padding the image obtained by iDXG exposure with a sensitivity ratio of 1:4 and the fusion of two frames from 12 bits to 14 bits; the image obtained by iDXG exposure with a sensitivity ratio of 1:16 and the fusion of two frames is 14 bits, so no padding is required.

[0304] It should also be noted that in the video processing method provided in the embodiments of the present application, different exposure modes can correspond to different sensitivity ratios. It should be noted that the sensitivity ratio can be a combination of exposure time (exp) and system gain (gain). The system gain can be set in multiple groups, such as gain = 1, gain = 4, gain = 16, and so on.

[0305] Assume that there are three exposure modes: binning, iDXG1:4, and iDXG1:16, and that the three exposure modes correspond to different exposure times and system gains. The sensitivity ratios corresponding to the different exposure modes can be obtained based on the exposure time and system gain. For example, the principle of obtaining the sensitivity ratio may include: (1) when the system dynamic range compression gain is 3.99 (i.e., adrc gain = 3.99), the exposure mode is binning mode, the exposure time is a ms (exp = a ms), the system gain is 1, and the sensitivity ratio after the exposure time and the system gain are combined is 1; (2) when the system dynamic range compression gain is 4 (i.e., adrc gain = 4), the exposure mode is iDXG1:4, the exposure time is a ms (exp = a ms), the system gain is 4, and the sensitivity ratio after the exposure time and the system gain are combined is 4; (3) when the system dynamic range compression gain is 4 (i.e., adrc gain = 4), the exposure mode is iDXG1:16, the exposure time is a ms (exp = a ms), the system gain is 16, and the sensitivity ratio after the exposure time and the system gain are combined is 16. For example, the sensitivity ratios corresponding to the various exposure modes can be shown in Table 4:

[0306] Table 4

[0307] Under different exposure modes, the AEC can instruct the image sensor to capture images using the gain corresponding to that exposure mode. Therefore, images captured using different exposure modes may correspond to different sensitivities. To avoid image brightness jumps caused by different sensitivities between images corresponding to different exposure modes when switching between different exposure modes, the video processing method provided in the embodiments of the present application solves this problem by adjusting the sensitivity ratio corresponding to the exposure mode.

[0308] For example, the three exposure modes and sensitivity ratios given in Table 4 above are used as examples. In Table 4 above, the binning exposure mode corresponds to a sensitivity ratio of 1, and the iDXG1:4 corresponds to a sensitivity ratio of 4. When the exposure mode is switched from binning to iDXG1:4, the iDXG1:4 exposure mode can be increased to 4 times the original value; or, in Table 4 above, the binning exposure mode corresponds to a sensitivity ratio of 1, and the iDXG1:16 corresponds to a sensitivity ratio of 16. When the exposure mode is switched from binning to iDXG1:16, the iDXG1:16 exposure mode can be increased to 16 times the original value; or, in Table 4 above, the iDXG1:4 exposure mode corresponds to a sensitivity ratio of 4, and the iDXG1:16 corresponds to a sensitivity ratio of 16. When the exposure mode is switched from iDXG1:4 to iDXG1:16, the iDXG1:16 exposure mode can be increased to the original value of 4. It should be noted that the several sensitivity ratios given in Table 4 above are only examples. In actual applications, the sensitivity ratios corresponding to different exposure modes can also be other specific values, which is not limited in the embodiments of the present application.

[0309] By adjusting the sensitivity of the image corresponding to the switching of different exposure modes, the jump in image brightness can be avoided. When switching between different exposure modes, the video image can also be switched smoothly to ensure image quality.

[0310] For example, as shown in FIG15 , there is a schematic flow chart of another video processing method provided in an embodiment of the present application. This method may include the following steps:

[0311] S1501: Receive a first operation input by a user, where the first operation is used to enable a video recording function of an electronic device.

[0312] The first operation may correspond to the operation of clicking the record button in FIG. 6C ; or, the first operation may correspond to the operation of clicking to turn on the smart HDR function.

[0313] In some embodiments, when the video recording function is turned on, the electronic device may display a first interface, wherein the first interface includes a first control for turning on the automatic switching of the HDR exposure mode function. The first interface may correspond to the interface shown in FIG6A or FIG6D , for example.

[0314] S1502: In response to a first operation, acquire a first image according to a preset default high dynamic range (HDR) exposure mode.

[0315] The exposure modes in the embodiments of the present application may include: binning exposure mode, SHDR exposure mode, DXG exposure mode, etc. According to the number of image frames corresponding to different exposure modes, they can be divided into a first HDR exposure mode and a second HDR exposure mode. The first HDR exposure mode is a single-frame mode, that is, one frame of image can be acquired based on a single exposure. The first HDR exposure mode may be a binning mode. The second HDR exposure mode is a dual-frame mode, that is, two frames of image can be read based on a single exposure, such as the DXG mode, or two frames of image can be acquired by performing two long and short exposures, such as the SHDR mode.

[0316] In some embodiments, a default HDR exposure mode may be pre-set. When the first operation is received, the AEC module may instruct the image sensor to capture an image using the default HDR exposure mode. For example, the default HDR exposure mode may be a binning mode.

[0317] S1503: Acquire ambient brightness based on the first image, and determine a target HDR exposure mode according to a preset strategy, where the preset strategy includes dynamic range information corresponding to the video shooting, a stroboscopic state, and a correspondence between ambient brightness and the target HDR exposure mode.

[0318] The dynamic range compression information here may include the dynamic range and / or the dynamic range compression gain. The embodiment of the present application is introduced by taking the dynamic range as an example.

[0319] In some embodiments, determining the target HDR exposure mode according to a preset strategy may include: obtaining a dynamic range corresponding to the video shooting; when the dynamic range is less than a first threshold, determining that the target HDR exposure mode is the binning mode; when the dynamic range is greater than or equal to the first threshold, detecting whether there is flicker; when there is flicker, determining that the target HDR exposure mode is the binning mode; when there is no flicker, determining the target HDR exposure mode according to the ambient brightness; wherein, when the ambient brightness is greater than a second threshold, determining that the target HDR exposure mode is the SHDR mode; when the ambient brightness is less than a third threshold and greater than a fourth threshold, determining that the target HDR exposure mode is the DXG mode, wherein the third threshold is less than the second threshold; when the ambient brightness is less than the fourth threshold, determining that the target HDR exposure mode is the binning mode.

[0320] It should be noted that in addition to determining the exposure mode based on the dynamic range, it can also be determined based on the dynamic range compression gain. The process of determining the exposure mode based on the dynamic range compression gain is similar to the above process (dynamic range can be replaced with dynamic range compression gain), and this will not be repeated here. When dynamic range is replaced with dynamic range compression gain, the specific value of the first threshold value can also change accordingly. That is, the specific values ​​of the first threshold value corresponding to the dynamic range and the first threshold value corresponding to the dynamic range compression gain can be different.

[0321] In some embodiments, determining the target HDR exposure mode according to a preset strategy may further include: detecting whether there is a thermal escape phenomenon in the video shooting; when the thermal escape phenomenon exists, determining that the target HDR exposure mode is the binning mode.

[0322] In some embodiments, when there is a switch between an HDR camera and a non-HDR camera during video shooting, the method further includes: when the switch occurs, the exposure mode of the HDR camera is the first HDR exposure mode, and the exposure mode corresponding to the non-HDR camera is the second HDR exposure mode, then adjusting the first dynamic range gain corresponding to the first HDR exposure mode so that the first dynamic range gain is closest to the second dynamic range gain corresponding to the second HDR exposure mode.

[0323] S1504: When the target HDR exposure mode is different from the default HDR exposure mode, switch the default HDR exposure mode to the target HDR exposure mode, and continue the video shooting to obtain a second image.

[0324] In some embodiments, when the target HDR exposure mode is the second HDR exposure mode, the images in the dual-frame mode are fused.

[0325] In some embodiments, when the target HDR exposure mode is the second HDR exposure mode, the image sensor transmits the image in dual-frame mode to the fusion module; the fusion module fuses the image in dual-frame mode; or, when the target HDR exposure mode is the second HDR exposure mode, the image sensor fuses the image in dual-frame mode.

[0326] In some embodiments, when the fusion module fuses the image in the dual-frame mode, the target sensitivity ratio between the dual-frame input image in the DCG mode and the dual-frame input image in the DAG mode is determined according to the sensitivity ratio required by the DXG mode; the DCG mode dual-frame input image and the DAG mode dual-frame input image are superimposed accordingly according to the target sensitivity ratio to obtain the superimposed dual-frame input image that meets the sensitivity ratio of the DXG mode.

[0327] In some embodiments, when the image sensor fuses the dual-frame mode images, the dual-frame input images in the DCG mode and the dual-frame input images in the DAG mode are superimposed according to a preset sensitivity ratio. Taking iDXG as an example, the sensitivity ratio is, for example, 1:4 or 1:16.

[0328] In some embodiments, for images acquired through different exposure modes, the image parameters corresponding to these exposure modes can be adjusted to be consistent. The specific process may include: presetting target parameters corresponding to the image acquired through the video shooting; adjusting the initial parameters corresponding to the first image acquired based on the first HDR exposure mode to the target parameters; and / or adjusting the initial parameters corresponding to the second image acquired based on the second HDR exposure mode to the target parameters.

[0329] In some embodiments, the electronic device supports the first HDR video mode, and the first HDR video mode includes HDR10 or HDR10+.

[0330] According to the video processing method provided in the embodiment of the present application, by seamlessly switching between multiple types of HDR processing schemes according to changes in factors such as ambient brightness, required dynamic range, and flicker detection, it is possible to use an HDR scheme adapted to the actual shooting environment and image quality requirements for image processing, thereby effectively expanding the dynamic range in the recording scene and improving the image quality in the recording scene.

[0331] Based on the same technical concept, an embodiment of the present application also provides an electronic device, including one or more processors; one or more memories; the one or more memories store one or more computer programs, and the one or more computer programs include instructions. When the instructions are executed by the one or more processors, the computer or processor executes one or more steps in any of the above methods.

[0332] Based on the same technical concept, an embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable program instructions. When the computer-executable program instructions are executed on a computer, the computer or processor executes one or more steps in any of the above methods.

[0333] Based on the same technical concept, an embodiment of the present application also provides a computer program product containing instructions, wherein the computer program product includes computer program code. When the computer program code is run on a computer, the computer or processor executes one or more steps in any of the above methods.

[0334] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via a wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) method. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)).

[0335] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

[0336] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A video processing method, characterized in that: Applied to electronic equipment, the method includes: receiving a first operation input by a user, where the first operation is used to enable a recording function of the electronic device; In response to the first operation, acquiring a first image according to a preset default high dynamic range (HDR) exposure mode; Acquire ambient brightness based on the first image, and determine a target HDR exposure mode according to a preset strategy, the preset strategy including dynamic range information corresponding to the video shooting, a stroboscopic state, and a correspondence between the ambient brightness and the target HDR exposure mode; When the target HDR exposure mode is different from the default HDR exposure mode, the default HDR exposure mode is switched to the target HDR exposure mode, and the video shooting is continued to obtain a second image.

2. The method according to claim 1, characterized in that The target HDR exposure mode includes at least a first HDR exposure mode and a second HDR exposure mode, the first HDR exposure mode is a single-frame mode, and the second HDR exposure mode is a dual-frame mode; When the target HDR exposure mode is the second HDR exposure mode, the images input in the dual-frame mode are fused.

3. The method according to claim 2, characterized in that The first HDR exposure mode is a binning mode, and the second HDR exposure mode includes a staggered high dynamic range mode SHDR and DXG, wherein the DXG is a mode in which a dual conversion gain mode DCG and a dual analog gain DAG are superimposed.

4. The method according to claim 2 or 3, characterized in that The method further comprises: Presetting target parameters corresponding to the image obtained through the video shooting; Adjusting initial parameters corresponding to the first image acquired based on the first HDR exposure mode to the target parameters; and / or, Adjusting initial parameters corresponding to the second image acquired based on the second HDR exposure mode to the target parameters.

5. The method according to any one of claims 1 to 4, characterized in that The electronic device includes an automatic exposure control AEC module, an image sensor, and a perception module. The step of acquiring the first image according to a preset default HDR exposure mode in response to the first operation specifically includes: In response to the first operation, the AEC module sends first indication information to the image sensor, where the first indication information is used to instruct the image sensor to capture an image using the default HDR exposure mode; In response to the first indication information, the image sensor uses the default HDR exposure mode and acquires the first image.

6. The method according to any one of claims 1 to 5, characterized in that The electronic device includes an AEC module, an image sensor, and a perception module. The step of obtaining the ambient brightness according to the first image and determining the target HDR exposure mode according to a preset strategy specifically includes: The image sensor sends the first image to the perception module; The perception module obtains the ambient brightness according to the first image and indicates the ambient brightness to the AEC module; The AEC module determines the target HDR exposure mode according to the ambient brightness and a preset strategy.

7. The method according to any one of claims 2 to 6, characterized in that The electronic device includes an AEC module, an image sensor, a perception module and a fusion module. When the target HDR exposure mode is the second HDR exposure mode, fusing the dual-frame input images specifically includes: When the target HDR exposure mode is the second HDR exposure mode, the image sensor transmits the image in the dual-frame mode to the fusion module; The fusion module fuses the images in the dual-frame mode; or, When the target HDR exposure mode is the second HDR exposure mode, the image sensor fuses the images in the dual-frame mode.

8. The method according to claim 7, characterized in that When the fusion module fuses the images in the dual-frame mode, the target sensitivity ratio between the dual-frame input images in the DCG mode and the dual-frame input images in the DAG mode is determined according to the sensitivity ratio required by the DXG mode; The DCG mode double-frame input image and the DAG mode double-frame input image are superimposed respectively according to the target sensitivity ratio to obtain the superimposed double-frame input image that meets the DXG mode sensitivity ratio.

9. The method according to claim 7, characterized in that When the image sensor fuses the images in the dual-frame mode, the dual-frame input images in the DCG mode and the dual-frame input images in the DAG mode are superimposed according to a preset sensitivity ratio.

10. The method according to any one of claims 1 to 9, characterized in that Determining the target HDR exposure mode according to the preset strategy specifically includes: Obtaining a dynamic range corresponding to the video shooting; When the dynamic range information is less than a first threshold, determining that the target HDR exposure mode is the binning mode; When the dynamic range information is greater than or equal to the first threshold, detecting whether there is stroboscopic light; When stroboscopic light is present, determining that the target HDR exposure mode is the binning mode; When there is no stroboscopic light, the target HDR exposure mode is determined according to the ambient brightness; wherein, When the ambient brightness is greater than a second threshold, determining that the target HDR exposure mode is the SHDR mode; When the ambient brightness is less than a third threshold and greater than a fourth threshold, determining that the target HDR exposure mode is the DXG mode, wherein the third threshold is less than the second threshold; When the ambient brightness is less than the fourth threshold, the target HDR exposure mode is determined to be the binning mode.

11. The method according to claim 10, characterized in that The method further comprises: Detecting whether there is a thermal escape phenomenon in the video shooting; When the thermal escape phenomenon occurs, the target HDR exposure mode is determined to be the binning mode.

12. The method according to any one of claims 1 to 11, characterized in that The electronic device supports the first HDR video mode, and the first HDR video mode includes HDR10 or HDR10+.

13. The method according to any one of claims 3 to 12, characterized in that When the video is captured, switching between an HDR camera and a non-HDR camera occurs, and the method further includes: When the switch is made, the exposure mode of the HDR camera is the first HDR exposure mode, and the exposure mode corresponding to the non-HDR camera is the second HDR exposure mode, then the first HDR exposure mode is adjusted. The first dynamic range gain corresponding to the second HDR exposure mode is configured so that the first dynamic range gain is closest to the second dynamic range gain corresponding to the second HDR exposure mode.

14. The method according to any one of claims 1 to 13, characterized in that The method further comprises: A first interface is displayed, where the first interface includes a first control, and the first control is used to enable a function of automatically switching the HDR exposure mode.

15. An electronic device, characterized in that: include: one or more processors; one or more memories; The one or more memories store one or more computer programs, and the one or more computer programs include instructions. When the instructions are executed by the one or more processors, the electronic device performs the method according to any one of claims 1 to 14.

16. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-executable program instructions, which, when executed on a computer, enable the computer to perform the method according to any one of claims 1 to 14.

Images