Video processing method and electronic device
By dynamically switching multiple HDR exposure methods according to environmental brightness and strobe changes in the recording scene, the problem of low image quality caused by limited dynamic range in the recording scene is solved, and efficient image processing and image quality improvement is achieved.
Patent Information
- Application Number
- CN202210950356.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-09
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2042-08-09
AI Technical Summary
In the video scene, the prior art has low image quality due to the limitation of dynamic range in the dynamic range of video, which cannot effectively expand the exposure dynamic range, resulting in the problem of excessive brightness or darkness of the image.
By seamlessly switching various HDR exposure methods based on changes in environmental brightness, dynamic range and strobe, we use an adaptive HDR solution to perform image processing, including binning, SHDR, DXG and other exposure methods to achieve high image efficiency HDR processing.
Effectively expand the dynamic range in the recording scene, improve the image quality, and ensure the balance of light and darkness and detailed performance of the image under different light conditions.
Smart Images

Figure CN117651221B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of terminal technology, and in particular to a video processing method and electronic device. Background Art
[0002] Exposure is the process of forming an image by receiving light from the lens through a photosensitive device. During the shooting process, the brightness of the background or subject will change. If the external light is too strong, it will easily lead to overexposure, resulting in an image that is too bright and lacks layers and details; if the external light is too weak, it will easily lead to underexposure, resulting in an image that is too dark and cannot reflect the true color.
[0003] In practical applications, the brightness of an image is usually limited by the dynamic range (dynamic imaging). The dynamic range refers to the ratio of the maximum output signal to the minimum output signal supported by the device, or the grayscale ratio of the upper and lower brightness limits of the image. If the ambient brightness is greater than the upper limit of the dynamic range, the captured image will be brighter; if the ambient brightness is less than the lower limit of the dynamic range, the captured image will be darker.
[0004] Currently, factors affecting dynamic range include the size of the device and the image sensor (camera sensor). The larger the sensor size, the larger the photosensitive surface area, which can provide more area for receiving light information during the exposure time. The more pixels, the larger the dynamic range. However, with the development of technology, the miniaturization of the physical space of the device limits the size of the sensor, resulting in a limited dynamic range. Summary of the invention
[0005] The embodiment of the present application provides a method for video processing, which realizes efficient HDR processing in video shooting scenarios and improves video quality by seamlessly and automatically switching between multiple exposure modes according to actual video shooting conditions.
[0006] In a first aspect, a frequency processing method is provided, which is applied to an electronic device, and the method includes:
[0007] Receiving a first operation input by a user, where the first operation is used to turn on a video shooting function of the electronic device;
[0008] In response to the first operation, acquiring a first image according to a preset default high dynamic range (HDR) exposure mode;
[0009] Acquire the ambient brightness according to the first image, and determine a target HDR exposure mode according to a preset strategy, wherein the preset strategy includes dynamic range information corresponding to the video shooting, a stroboscopic state, and a correspondence between the ambient brightness and the target HDR exposure mode;
[0010] When the target HDR exposure mode is different from the default HDR exposure mode, the default HDR exposure mode is switched to the target HDR exposure mode, and the video shooting is continued to obtain a second image.
[0011] In a possible implementation, the dynamic range information here may include a dynamic range and / or a dynamic range compression gain.
[0012] According to the video processing method provided by the present implementation, by seamlessly switching between various types of HDR processing schemes according to changes in factors such as ambient brightness, required dynamic range, and flicker detection, it is possible to use an HDR scheme adapted to the actual shooting environment and image quality requirements for image processing, thereby effectively expanding the dynamic range in the recording scene and improving the image quality in the recording scene.
[0013] In combination with the first aspect, in some implementations of the first aspect, the target HDR exposure mode includes at least a first HDR exposure mode and a second HDR exposure mode, the first HDR exposure mode is a single-frame mode, and the second HDR exposure mode is a dual-frame mode;
[0014] When the target HDR exposure mode is the second HDR exposure mode, the image input in the dual-frame mode is fused.
[0015] In a possible implementation, the single-frame mode may be a binning exposure mode, that is, the image sensor outputs a single-frame image after exposure. The dual-frame mode may be an exposure mode such as SHDR, DCG, DXG, etc., that is, a mode of reading two frames of images based on one exposure, and then fusing and adjusting the HDR of the two frames of images; or a mode of obtaining two frames of images by long exposure and short exposure, and then fusing and adjusting the HDR of the two frames of images.
[0016] In combination with the first aspect, in certain implementations of the first aspect, the first HDR exposure mode is a binning mode, and the second HDR exposure mode includes a staggered high dynamic range mode SHDR and DXG, and the DXG is a mode in which a dual conversion gain mode DCG and a dual analog gain DAG are superimposed.
[0017] In combination with the first aspect, in some implementations of the first aspect, the method further includes:
[0018] Presetting target parameters corresponding to the image acquired through the video shooting;
[0019] adjusting initial parameters corresponding to the first image acquired based on the first HDR exposure method to the target parameters; and / or,
[0020] The initial parameters corresponding to the second image acquired based on the second HDR exposure method are adjusted to the target parameters.
[0021] In a possible implementation, the target parameters here may include image size, bit depth, etc.
[0022] It should be understood that adjusting the parameters of images corresponding to different exposure modes to target parameters can make the images of different exposure modes have consistent image parameters, thereby avoiding image jumps when switching between different exposure modes.
[0023] In combination with the first aspect, in some implementations of the first aspect, the electronic device includes an automatic exposure control AEC module, an image sensor, and a perception module.
[0024] In response to the first operation, acquiring the first image according to a preset default HDR exposure mode specifically includes:
[0025] In response to the first operation, the AEC module sends first indication information to the image sensor, where the first indication information is used to indicate that the image is captured using the default HDR exposure mode;
[0026] In response to the first indication information, the image sensor uses the default HDR exposure mode and acquires the first image.
[0027] In combination with the first aspect, in some implementations of the first aspect, the electronic device includes an AEC module, an image sensor, and a perception module.
[0028] The step of acquiring the ambient brightness according to the first image and determining the target HDR exposure mode according to a preset strategy specifically includes:
[0029] The image sensor sends the first image to the perception module;
[0030] The perception module acquires the ambient brightness according to the first image, and indicates the ambient brightness to the AEC module;
[0031] The AEC module determines the target HDR exposure mode according to the ambient brightness and a preset strategy.
[0032] In combination with the first aspect, in some implementations of the first aspect, the electronic device includes an AEC module, an image sensor, a perception module, and a fusion module.
[0033] When the target HDR exposure mode is the second HDR exposure mode, fusing the double-frame input images specifically includes:
[0034] When the target HDR exposure mode is the second HDR exposure mode, the image sensor transmits the image in the dual-frame mode to the fusion module;
[0035] The fusion module fuses the images in the dual-frame mode; or,
[0036] When the target HDR exposure mode is the second HDR exposure mode, the image sensor fuses the image in the dual-frame mode.
[0037] In combination with the first aspect, in some implementations of the first aspect, when the fusion module fuses the image in the dual-frame mode, a target sensitivity ratio between the dual-frame input image in the DCG mode and the dual-frame input image in the DAG mode is determined according to the sensitivity ratio required by the DXG mode;
[0038] The DCG mode double-frame input image and the DAG mode double-frame input image are superimposed respectively according to the target sensitivity ratio to obtain a superimposed double-frame input image that meets the DXG mode sensitivity ratio.
[0039] In combination with the first aspect, in certain implementations of the first aspect, when the image sensor fuses the image in the dual-frame mode, the dual-frame input image in the DCG mode and the dual-frame input image in the DAG mode are superimposed according to a preset sensitivity ratio.
[0040] In combination with the first aspect, in some implementations of the first aspect, determining the target HDR exposure mode according to a preset strategy specifically includes:
[0041] Obtaining a dynamic range corresponding to the video shooting;
[0042] When the dynamic range information is less than a first threshold, determining that the target HDR exposure mode is the binning mode;
[0043] When the dynamic range information is greater than or equal to the first threshold, detecting whether there is stroboscopic light;
[0044] When there is stroboscopic light, determining that the target HDR exposure mode is the binning mode;
[0045] When there is no stroboscopic light, the target HDR exposure mode is determined according to the ambient brightness; wherein,
[0046] When the ambient brightness is greater than a second threshold, determining that the target HDR exposure mode is the SHDR mode;
[0047] When the ambient brightness is less than a third threshold, the target HDR exposure mode is determined to be the DXG mode.
[0048] In combination with the first aspect, in some implementations of the first aspect, the method further includes:
[0049] Detecting whether there is a thermal escape phenomenon in the video shooting;
[0050] When the thermal escape phenomenon occurs, the target HDR exposure mode is determined to be the binning mode.
[0051] In combination with the first aspect, in some implementations of the first aspect, the electronic device supports the first HDR video mode, and the first HDR video mode includes HDR10 or HDR10+.
[0052] In combination with the first aspect, in some implementations of the first aspect, when there is a switch between an HDR camera and a non-HDR camera during video shooting, the method further includes:
[0053] When the switch occurs, the exposure mode of the HDR camera is the first HDR exposure mode, and the exposure mode corresponding to the non-HDR camera is the second HDR exposure mode, then the first dynamic range gain corresponding to the first HDR exposure mode is adjusted so that the first dynamic range gain is closest to the second dynamic range gain corresponding to the second HDR exposure mode.
[0054] In combination with the first aspect, in some implementations of the first aspect, the method further includes:
[0055] A first interface is displayed, wherein the first interface includes a first control, and the first control is used to enable a function of automatically switching the HDR exposure mode.
[0056] In a second aspect, an electronic device is provided, comprising: one or more processors; one or more memories; the one or more memories storing one or more computer programs, the one or more computer programs comprising instructions, which, when executed by the one or more processors, enable the electronic device to execute a method as described in any implementation of the first aspect or the second aspect above.
[0057] In a third aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer-executable program instructions, and when the computer-executable program instructions are executed on a computer, the computer executes a method as described in any implementation of the first aspect or the second aspect.
[0058] In a fourth aspect, a computer program product is provided, the computer program product comprising a computer program code, and when the computer program code is run on a computer, the computer is caused to execute the method as described in any implementation of the first aspect or the second aspect above. BRIEF DESCRIPTION OF THE DRAWINGS
[0059] Figure 1 A schematic diagram of an image captured by an interlaced high dynamic range mode provided in an embodiment of the present application.
[0060] Figure 2A and Figure 2B A schematic diagram of images captured by the DCG mode and the DAG mode provided in an embodiment of the present application.
[0061] Figure 3 A schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application.
[0062] Figure 4 A software structure block diagram of an electronic device 100 provided in an embodiment of the present application.
[0063] Figure 5 A schematic diagram of the differences in related parameters corresponding to a plurality of exposure modes provided in an embodiment of the present application.
[0064] FIG. 6A to FIG. 6D GUI schematic diagrams that may be involved in some video processing processes provided in embodiments of the present application.
[0065] Figure 7 A schematic diagram of a video processing method provided in an embodiment of the present application.
[0066] Figure 8 A schematic diagram of another video processing method provided in an embodiment of the present application.
[0067] Fig. 9 A schematic diagram of another video processing method provided in an embodiment of the present application.
[0068] Fig.10 A schematic diagram of another video processing method provided in an embodiment of the present application.
[0069] Fig.11 A schematic diagram of another video processing method provided in an embodiment of the present application.
[0070] Fig.12 A schematic diagram of another video processing method provided in an embodiment of the present application.
[0071] Fig.13 A schematic diagram of adjusting image parameters during video processing provided by an embodiment of the present application.
[0072] Fig.14 A schematic diagram of adjusting image parameters during video processing provided by an embodiment of the present application.
[0073] Fig.15 A schematic diagram of another video processing method provided in an embodiment of the present application. DETAILED DESCRIPTION
[0074] It should be noted that the terms used in the implementation method part of the embodiments of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of associated obstacles, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two, "at least one" and "one or more" mean one, two or more than two.
[0075] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, it is defined that the "first" and "second" features may explicitly or implicitly include one or more of the features.
[0076] References to "one embodiment" or "some embodiments" etc. described in this specification mean that a particular feature, structure or characteristic described in conjunction with the embodiment is included in one or more embodiments of the present application. Thus, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing at different places in this specification do not necessarily all refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in another way. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in another way.
[0077] The term "user interface" in the specification, claims and drawings of this application refers to the medium interface for interaction and information exchange between an application or operating system and a user. A common form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operations displayed in a graphical manner. It can be an interface element such as an icon, window, or control displayed on a display screen of an electronic device, wherein a control may include visual interface elements such as icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, and widgets.
[0078] Combined with the introduction in the background technology, in order to overcome the problem of low image quality (small difference between light and dark) caused by limited dynamic range, high dynamic range imaging (HDR) technology came into being. HDR technology is a group of technologies that can achieve a larger exposure dynamic range (i.e., a larger difference between light and dark), and its purpose is to correctly represent the brightness range from direct sunlight to the darkest shadow in the real world. When the HDR technology is specifically implemented, the current object being photographed can be exposed by setting multiple sets of exposure values (EV), including exposure using the EV value obtained by the current metering calculation under normal conditions, and exposure using EV values lower than the normal value (EV value of EV-n) and EV values higher than the normal value (EV value of EV+n), and then the multiple photos after exposure are fused, so that the dark object uses a local photo with a high EV exposure, and the bright object uses a local photo with a low EV exposure, so that the scene of the entire photo is not too bright or too dark.
[0079] The above-mentioned method of expanding the dynamic range by EV surrounding and fusing multiple photos involves a more complex image processing algorithm, which requires a sufficient interval between two frames as calculation time. Generally speaking, in the photo scene, the time interval between obtaining two consecutive frames of images is long enough to achieve the expansion of the dynamic range according to the traditional HDR exposure method. However, in the video recording scene, in order to ensure the smoothness of the picture, the image is generally collected and processed at a specific frame rate (frame rate), and the frame interval is very short, that is, each frame (frame) corresponds to only a very short calculation time (such as when the frame rate is 30fps, the calculation time corresponding to each frame is less than 33ms on average). This makes it impossible to use the more complex traditional method to expand the dynamic range in the video recording scene, but it is necessary to customize an adaptive HDR solution based on the characteristics of the video.
[0080] In view of this, an embodiment of the present application provides a video processing method, which seamlessly switches between multiple types of HDR exposure modes according to changes in factors such as ambient brightness, dynamic range, and flicker, thereby utilizing an HDR solution that is adapted to the actual shooting state and image quality requirements for image acquisition and processing, thereby effectively expanding the dynamic range in the recording scene and improving the image quality in the recording scene.
[0081] In order to better understand the video processing method provided in the embodiments of the present application, some definitions of terms that may be involved in this article are first introduced.
[0082] 1. Stagger high-dynamic range (SHDR)
[0083] Different manufacturers may have different names for technologies similar to SHDR. For example, Sony's related technology is called digital overlap HDR (DOL-HDR). SHDR technology can collect multiple frames of images with different exposure times in one acquisition cycle by increasing the frame rate of the sensor, and then fuse the corresponding long exposure frames and short exposure frames into one frame through multi-frame fusion technology to obtain images with high dynamic range. Among them, the short exposure frame (or "short exposure image") can obtain bright information; the long exposure frame (or "long exposure image") can obtain dark information, and the long exposure frame has excellent noise control; the fusion of the two frames of images can obtain the gain of highlights and shadows.
[0084] For example, the image captured by SHDR technology can be as follows Figure 1 As shown, it includes a frame of long exposure image (such as Figure 1 The frame marked with the letter “L” in the figure) and a short exposure image (such as Figure 1 frame marked with the letter “S” in the image).
[0085] It should be noted that, affected by different exposure periods and exposure times, SHDR may not be able to obtain ideal shooting effects in some scenes. For example, compared with non-SHDR exposure shooting scenes, SHDR scenes need to collect images under multiple exposure parameters (such as exposure time), so the blank frame interval between two frames will be shorter, which is not suitable for scenes where the image processing algorithm is too complex. In addition, the two frames used by SHDR to fuse into one image come from different exposure periods and correspond to different exposure times. The longer the exposure time, the more likely it is to produce smear (or ghosting), and motion ghosting will inevitably occur during fusion (only in high-brightness scenes where the exposure time of both frames is very short can a more ideal effect be achieved).
[0086] 2. Binning
[0087] Binning is an image readout mode that adds together the charges induced in adjacent pixels and reads them out in a one-pixel mode. For example, when an electronic device is shooting an image, the light reflected by the target object is collected by the camera so that the reflected light is transmitted to the image sensor. The image sensor includes multiple photosensitive elements, and the charge collected by each photosensitive element is a pixel, and a binning operation is performed on the pixel information. Specifically, binning can merge n×n pixels into one pixel. For example, binning can combine adjacent 2×2 pixels into one pixel, that is, the colors of adjacent 2×2 pixels are presented in the form of one pixel.
[0088] 3. Dual conversion gain (DCG)
[0089] DCG is also a method of image readout, which can be understood as the ability to perform two readouts in a pixel unit circuit, or can be understood as two capacitors storing photon energy in the photosensitive unit corresponding to a pixel. The DCG involved in the embodiments of the present application can specifically refer to performing two conversion gain readouts on an image based on a single exposure operation.
[0090] In the embodiment of the present application, DCG can be used to expand the dynamic range. The implementation principle is as follows: an image sensor with dual conversion gain DCG capability has two potential wells for one pixel. The two potential wells correspond to different full well capacities and different conversion gains CG. The large full well capacity corresponds to low conversion gain (LCG) and low sensitivity, and the small full well capacity corresponds to high conversion gain (HCG) and high sensitivity. In this way, the sensor can use two potential wells (two sensitivities) and two conversion gains in the same scene to acquire two images in one exposure: an image in high-sensitivity mode and an image in low-sensitivity mode. The electronic device then synthesizes the two acquired images into one image, which is HDR technology.
[0091] As an example, based on the same short exposure, the images read out by DCG twice can be respectively as follows: Figure 2A As shown in the figure, the LCG frame is an image frame read out by applying the LCG gain signal, which can protect the highlight area from overexposure; the HCG frame is an image frame read out by applying the HCG gain signal, which can increase the shadow brightness and control the noise at the same time. Afterwards, the two frames are fused to obtain the gains of highlights and shadows, and obtain an image with optimized dynamic range.
[0092] 4. Dual analog gain (DAG)
[0093] As another image readout method, similar to the dual conversion gain DCG described above, DAG also reads two frames of images through two analog signals based on one exposure, but the difference is that the two readout methods are different. The DAG readout mode uses two analog gains for image readout, including low analog gain (LAG) and high analog gain (HAG). LAG can protect the highlight area from overexposure; HAG can brighten the shadows and control noise at the same time.
[0094] As an example, based on the same short exposure, the images read out by DAG twice can be as follows: Figure 2B As shown in the figure, the LAG frame is an image frame read out by applying LAG, and the HAG frame is an image frame read out by applying HAG. Afterwards, the two frames are fused to obtain the gains of highlights and shadows and expand the dynamic range. Compared with conversion gain (CG), analog gain (AG) has better noise control ability.
[0095] It should be noted that since the two frames of images of DCG and DAG are from different readouts of one exposure, there will be no ghosting problem after the two frames of images are fused, which is suitable for wide use in various video recording scenes.
[0096] It should also be noted that in the embodiment of the present application, the above-mentioned DCG and DAG technologies can also be superimposed to expand the dynamic range. For ease of description, the embodiment of the present application refers to the method of superimposing DCG and DAG as DXG. Exemplarily, DCG and DAG can be superimposed according to the sensitivity ratio. For example, if the sensitivity ratio between the image read out by the high gain signal and the image read out by the low gain signal is 1:16, then DCG can be used to obtain two frames of images with a sensitivity ratio of 1:2, and then DAG can be used to obtain two frames of images with a sensitivity ratio of 1:8. After multiplying the sensitivity ratios of the two, a sensitivity ratio image of 1:16 is obtained.
[0097] 5. Single-frame dynamic range DXG (intra-scene DXG, iDXG)
[0098] For the sake of distinction, the embodiment of the present application refers to the method of fusing two frames acquired by the DXG method into one frame in the image sensor as iDXG.
[0099] 6. RAW images (or RAW domain images)
[0100] The original image contains the data processed by the image sensor of a digital camera, scanner or film scanner. The RAW image contains the original information of the image without nonlinear processing in the image signal processing (ISP) process.
[0101] 7. HDR10 video
[0102] The HDR10 video is configured according to static metadata, for example, the PQ conversion curve of the HDR10 is fixedly mapped according to the reference display brightness of the display. The bit depth of the HDR10 video is 10 bits; the static metadata may meet the definition in SMPTE ST2086 or other standards.
[0103] 8. HDR10+ video
[0104] HDR10+ is an improvement on HDR. It supports dynamic metadata, which means that HDR10+ can adjust or enhance the brightness, contrast and color saturation of the image according to different scenes in the video, so that each frame in the HDR10+ video has an independently adjusted HDR effect. The bit depth of the HDR10+ video is 12 bits; the dynamic metadata can meet the definition in SMPTE ST2094 or other standards.
[0105] 9. Brightness scene
[0106] The brightness scene may also be referred to as a brightness level. In an embodiment of the present application, the brightness scene may be used to determine the exposure mode (DXG or SHDR) of image acquisition. The brightness scene may include: a high-brightness scene, a medium-brightness scene, and a low-light scene.
[0107] For example, brightness scenes may correspond to different brightness ranges, and the device may distinguish different brightness levels according to the intensity of light reflected by the object being photographed. For example, the brightness range corresponding to a bright scene may be greater than 50,000 lux, the brightness range corresponding to a medium brightness scene may be 50,000 lux-10 lux, and the brightness range corresponding to a dark scene may be 10 lux-0 lux.
[0108] It should be noted that the brightness levels described in the embodiments of the present application are not limited to the above three. Moreover, the brightness ranges corresponding to the three brightness scenes are only used as examples, and the values of the brightness ranges corresponding to different brightness scenes can also be other values, which are not limited in the embodiments of the present application.
[0109] For example, Figure 3, which is a schematic diagram of the structure of an electronic device 100 provided in an embodiment of the present application.
[0110] Figure 1 The structure diagram of the electronic device 100 is shown. The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.
[0111] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0112] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.
[0113] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.
[0114] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.
[0115] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.
[0116] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.
[0117] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.
[0118] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.
[0119] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.
[0120] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.
[0121] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).
[0122] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals.
[0123] The speaker 170A, also called a "horn", is used to convert audio electrical signals into sound signals.
[0124] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals.
[0125] The microphone 170C, also called a "microphone" or "speaker", is used to convert a sound signal into an electrical signal. The electronic device 100 may be provided with at least one microphone 170C.
[0126] The earphone jack 170D is used to connect a wired earphone.
[0127] The sensor module 180 may include one or more sensors, which may be of the same type or of different types. The sensor module 180 may include a pressure sensor, a gyroscope sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a touch sensor, an ambient light sensor, and the like.
[0128] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.
[0129] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active matrix organic light-emitting diode or an active matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Mini LED, Micro LED, Micro-OLED, quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.
[0130] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.
[0131] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.
[0132] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy. The video codec is used to compress or decompress digital video. The NPU is a neural network (NN) computing processor that quickly processes input information by drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, and can also continuously self-learn.
[0133] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.
[0134] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image and video playback function, etc.), etc. The data created during the use of the electronic device 100 (such as audio data, a phone book, etc.) can be stored in the data storage area.
[0135] In some embodiments, the internal memory 121 may store codes related to automatic exposure control (AEC) and exposure mode switching methods. The processor may implement the automatic exposure control process by running the AEC-related codes, and may implement the switching of the exposure mode of the image sensor in the camera by running the exposure mode switching method-related codes.
[0136] In some embodiments, the AEC module may include an AEC algorithm module and an AEC statistics module. The AEC statistics module is used to statistically analyze the parameters in the collected image, such as image brightness, etc. The AEC algorithm module can automatically adjust the exposure parameters of the camera according to the statistical results, and the AEC algorithm module can also estimate the ambient light brightness according to the statistical results.
[0137] The electronic device 100 can realize the function of acquiring images through ISP, camera 193, video codec, GPU, display screen 194 and application processor.
[0138] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image or video visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.
[0139] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the ISP to convert it into a digital image or video signal. The ISP outputs the digital image or video signal to the DSP for processing. The DSP converts the digital image or video signal into an image or video signal in a standard RGB, YUV or other format.
[0140] In some embodiments, the electronic device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1. For example, in some embodiments, the electronic device 100 may use N cameras 193 to obtain images with multiple exposure coefficients, and then, in video post-processing, the electronic device 100 may synthesize an HDR image based on the images with multiple exposure coefficients using a High Dynamic Range (HDR) technology.
[0141] The video codec is used to compress or decompress digital video. The electronic device 100 may support one or more video codecs. In this way, the electronic device 100 can play or record videos in multiple coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.
[0142] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.
[0143] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to or separated from the electronic device 100 by inserting it into or removing it from the SIM card interface 195. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The electronic device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.
[0144] Exemplarily, the software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device 100.
[0145] For example, Figure 4 , which is a software structure block diagram of an electronic device 100 provided in an embodiment of the present application.
[0146] In some embodiments, the layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In the embodiments of the present application, specifically, the system may include an application layer, a hardware abstraction layer, and a kernel layer from top to bottom. Among them, between the application layer and the hardware abstraction layer may also include an application framework layer, a system library, a runtime, etc.
[0147] The application layer can include a series of application packages. Figure 4 As shown, the application package may include applications (or applications) such as camera, gallery, music, video, and call.
[0148] The hardware abstraction layer shields the differences between different hardware devices and provides a standard interface for the system, such as Figure 4 As shown, the hardware abstraction layer transmits data to the kernel layer through the standard HAL interface and accepts data uploaded by the kernel layer. The hardware abstraction layer can contain multiple library modules, each of which implements a set of interfaces for a specific type of hardware component, such as wireless fidelity (Wi-Fi) / Bluetooth modules and camera modules. When the application framework layer application interface requests access to device hardware, the system will load the corresponding library module for the hardware.
[0149] In some embodiments, the hardware abstraction layer may also include a camera module. The camera module may include an automatic exposure control (AEC) module and an image processing pipeline. Among them, the AEC module can be used to implement automatic exposure control. Specifically, the AEC module can obtain exposure parameters from a system configuration file and configure the exposure parameters to the image sensor. The image sensor captures an image according to the exposure parameters. The AEC statistics module in the AEC module can statistically analyze the parameters in the captured image, such as image brightness, etc. The AEC module can also estimate the dynamic range compression gain (adrc gain) based on the above-mentioned exposure parameters used when capturing images and the image brightness. The exposure mode control module can be used to control the exposure mode of the image sensor of the camera according to the estimated ambient light measurement. Specifically, in a bright light scene, the image sensor can be controlled to work in a low-sensitivity mode, thereby extending the exposure time so that the exposure time meets the exposure time requirements for shooting video.
[0150] The kernel layer is a layer between hardware and software. The kernel layer at least includes display driver, camera driver, sensor driver, sensor front-end processor, image processor front-end, image processor back-end, etc. Among them, the above-mentioned exposure mode control module can control the exposure mode of the camera's image sensor through the camera driver, and the AEC module can configure exposure parameters to the camera through the camera driver.
[0151] The following is an illustrative description of the workflow of the software and hardware of the electronic device 100 in conjunction with a video recording scenario.
[0152] When the touch sensor receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into an original input event (including touch coordinates, timestamp of the touch operation, and other information). The original input event is stored in the kernel layer. The application framework layer obtains the original input event from the kernel layer and identifies the control corresponding to the input event. For example, if the touch operation is a single-click operation and the control acted on by the single-click operation is the control of the camera application icon, the camera application calls the interface of the application framework layer to start the camera application, and then starts the camera driver by calling the kernel layer, and captures the video image through the camera 193. The display screen 194 displays the preview interface of the camera, and the captured video image is displayed in the preview interface.
[0153] It should be understood that since dynamic range, strobe, ambient brightness, etc. all have an impact on image quality, an HDR solution adapted to the actual shooting situation needs to be used for image exposure and processing in different scenarios. In order to effectively expand the dynamic range in the video recording scene and obtain high-quality video images, the video processing method provided in the embodiment of the present application establishes a corresponding relationship between specific influencing factors and exposure methods, and then seamlessly switches between multiple HDR solutions according to the actual situation of the influencing factors during the video shooting process, thereby realizing a processing method adapted to the high dynamic range influencing factors and effectively improving the extended dynamic range in the video recording scene.
[0154] In order to better understand the video processing method provided in the embodiments of the present application, the differences between the several exposure methods involved in the present application and the situations in which the several exposure methods are used are introduced below.
[0155] For example, Figure 5 As shown, it is a timing diagram of the process of capturing images through SHDR exposure mode, binning exposure mode and DXG exposure mode respectively provided in the embodiments of the present application.
[0156] SHDR obtains long and short frame images by changing the shutter speed (that is, the exposure conditions), which can save the computing power of subsequent image reading and processing. However, when the exposure time is long, SHDR is prone to cause ghosting in the image, so it is suitable for scenes with short exposure time (high ambient brightness).
[0157] Binning exposure is a single-frame exposure method. Compared with DXG exposure, its exposure time is longer, for example, it can be longer than the long exposure time in SHDR exposure method. Figure 5 Taking the several exposure modes shown as examples, the exposure time corresponding to the binning exposure mode may be the longest, and the specific exposure time value may be set as needed, which is not limited in the embodiments of the present application.
[0158] The DXG exposure method is to expose two gain signals (conversion gain signal and analog gain signal) at the same time to obtain two frames of images with different gains (conversion gain or analog gain). Although the image acquisition process is relatively complicated, this method will not cause ghosting problems and the exposure time is short. Therefore, this exposure method can be applied to scenes with medium and low ambient brightness.
[0159] Based on the characteristics of the above exposure methods, for scenes with a small dynamic range, the image brightness difference is small, and the binning single-frame exposure method can be used. For scenes with a large dynamic range, it is possible to further determine whether the current scene has flicker, that is, to perform flicker judgment. If flicker exists, the binning exposure method is used because the binning exposure method has a longer exposure time and can effectively overcome the flicker phenomenon caused by the periodic change of AC power; if flicker does not exist, the SHDR or DXG exposure method can be further determined based on the ambient brightness. When the ambient brightness is high, the SHDR exposure method is used, and when the ambient brightness is low, the DXG exposure method is used.
[0160] For example, the following Table 1 shows examples of exposure methods corresponding to different influencing factors:
[0161] Table 1
[0162] Influencing factors Exposure method High ambient brightness + high dynamic range SHDR Low brightness + high dynamic range in the environment DXG Low dynamic range binning There is flickering phenomenon binning Thermal escape binning
[0163] It should be noted that the above-mentioned influencing factors are only examples. In practical applications, the types of influencing factors can be flexibly set as needed. For example, in some cases, only the dynamic range and ambient brightness in the shooting scene can be considered, while ignoring the stroboscopic factor, etc. The embodiments of the present application are not limited to this.
[0164] The video processing method provided in the embodiment of the present application can be applied to various scenarios where HDR processing is required, and can be particularly applied to HDR processing scenarios during video recording. FIG. 6A to FIG. 6D , which is a schematic diagram of a graphical user interface (GUI) that may be involved in the implementation of some video processing methods provided in the embodiments of the present application. For ease of understanding, the GUI interface in a mobile phone is used as an example for introduction.
[0165] It should be noted that in the embodiments of the present application, there may be multiple ways to trigger the video processing process. For example, the video processing process may be triggered by inputting a click operation on the smart HDR switch control set in the mobile phone; for example, when the mobile phone is turned on, the smart HDR function may be turned on by default, that is, the video processing process may be triggered when the mobile phone is turned on, and so on.
[0166] In one possible implementation, the switch control of Smart HDR can be set in the settings application of the mobile phone. Fig. 6AAs shown, it is a schematic diagram of a setting interface including a smart HDR switch control. The setting interface 601 includes a shooting parameter setting area, a video parameter setting area, and a general setting area. Among them, the shooting parameter setting area may specifically include a photo ratio setting bar, a voice-controlled shooting function setting bar, and a smile capture setting bar; the video parameter setting area may specifically include a video resolution setting bar, a video frame rate setting bar, and a smart HDR setting bar; the general setting area may specifically include a reference line function setting bar, a level function setting bar (not shown in the figure), and a timed shooting function setting bar (not shown in the figure). Among them, the smart HDR setting bar includes a switch control for smart HDR. When the user turns on the smart HDR function through the switch control, the mobile phone can intelligently enable the HDR mode according to the shooting scene during video recording.
[0167] In another possible implementation, the switch control of Smart HDR can also be set in the camera application of the mobile phone. Figure 6B and Figure 6C , which is a schematic diagram of the GUI that may be involved when the smart HDR function is enabled through the camera application. Specifically, when the mobile phone receives the user's opening operation for the camera application (such as a click operation on the camera icon in the main interface or lock screen interface), the following may be displayed: Figure 6B The camera main interface 603 is shown. Exemplarily, the camera main interface 603 may include an image preview area, a mode setting area, and shooting controls, etc. Among them, the mode setting area 605 may include a viewfinder, an album icon, a shooting control 604, and a rotation control, etc.
[0168] Among them, the viewfinder is used to obtain the preview image of the shooting and display the preview image in real time. The album icon is used to quickly enter the album. When the mobile phone detects that the user clicks the album icon, the photos or videos that have been taken can be displayed on the touch screen. The shooting control 604 is used for shooting or recording. When the mobile phone detects that the user clicks the shooting control, the mobile phone performs the shooting operation and saves the captured photos; or, when the mobile phone is in the recording mode, after the user clicks the shooting control, the mobile phone performs the recording operation and saves the recorded video. The camera rotation control is used to control the switching of the front camera and the rear camera.
[0169] In addition, the camera main interface 603 also includes functional controls for setting the shooting mode, such as Figure 6B The aperture shooting mode, night scene shooting mode, portrait shooting mode, video mode, photo mode and more are shown. Figure 6C As shown, more modes may include slow motion mode, panoramic mode, black and white art mode, dual-view recording mode, filter mode, smart HDR mode, etc.
[0170] In some embodiments, when a user takes a video through the camera application, Figure 6C Enter specific operations using the Smart HDR icon shown to turn on the Smart HDR function.
[0171] In some embodiments, when the smart HDR switch is turned on, artificial intelligence (AI) prompt capabilities can be supported in the video preview scene (recording has not started). For example, in a high-brightness and high-dynamic scene, the corresponding exposure mode is SHDR state, at this time, the preview interface can display a prompt message: "Backlight HDR"; for another example, in a low-brightness and high-dynamic scene, the corresponding exposure mode is DXG state, when the ambient brightness is less than a certain threshold, the preview interface can display a prompt message: "Night scene HDR".
[0172] It should be noted that FIG. 6A to FIG. 6C The location of the Smart HDR switch control shown is only an example. In actual applications, the Smart HDR switch control is not limited to being set in the settings application and the camera application. In addition, the above embodiment only introduces the example of turning on the Smart HDR function on the user's mobile phone, but in other embodiments, there may be multiple ways to turn on Smart HDR. For example, when the mobile phone is turned on, the Smart HDR function may be turned on by default; for example, when the mobile phone detects that a preset trigger event has occurred, the Smart HDR function may be automatically turned on. In some other embodiments, if the electronic device does not support the Smart HDR function (such as due to hardware limitations), the control icon of the HDR switch may be grayed out. This application is not limited to this.
[0173] In some embodiments, when the electronic device detects that the scene brightness in the current shooting environment is quite different and the Smart HDR function is turned off, it can also prompt the user to turn on the Smart HDR function in a variety of ways. For example, when a user is shooting a cave against the light, the brightness level of the backlit part is higher, while the brightness level of the cave part is lower, and the difference between the two is large. At this time, the shooting device can prompt the user to turn on the Smart HDR function through artificial intelligence (AI), such as displaying a prompt on the shooting interface. Fig.6D The prompt message shown is "The brightness and darkness of the current shooting picture are quite different. You can click the Smart HDR function in the "More" option to improve the picture quality."
[0174] In some embodiments, when the smart HDR function is turned on, the mobile phone can perform exposure and image processing according to the HDR scheme that matches the current actual shooting situation during the shooting process to obtain a video picture with a high dynamic range. The picture shot with smart HDR turned on can more clearly show the details of the bright and dark areas, and the details will not be blurred due to excessive brightness or too low brightness.
[0175] It is worth noting that the preset HDR scheme mentioned here refers to the HDR scheme determined by the mobile phone according to factors such as the dynamic range, stroboscopic state, and brightness of the shooting environment corresponding to the actual shooting environment. The difference between different HDR schemes lies mainly in the different exposure methods and image processing methods. In actual applications, the mobile phone can seamlessly switch between multiple supported exposure methods according to the actual shooting conditions. Combined with the introduction above, the exposure methods available in the embodiments of the present application may include, for example: SHDR, DXG, and binning. It should be noted that in different scenarios, the exposure methods available for selection may be different. For example, due to the hardware performance of the device, for devices that do not support the DXG exposure method, the available exposure methods may also include only SHDR and binning.
[0176] In some embodiments, the underlying implementation process of controlling the smart HDR switch control may include: HAL reports multi-state switching video HDR capabilities to distinguish products. APP configures the "Smart HDR" switch based on the reported capabilities. APP confirms the switch status: on / off / grayed out based on the mode and scene. APP sends commands to HAL to turn on and off the HDR capability based on the switch status. When turned off, only binning capabilities are used. HAL reports the AECHDR status tag to support AI prompts and recommendations.
[0177] The above introduces the principles of different exposure methods and how to enable smart HDR. In order to better understand the video processing method provided in the embodiment of the present application, the specific functional modules for executing the method and the underlying implementation process of the method are introduced below.
[0178] For example, Figure 7 As shown, it is a schematic diagram of data flow interaction between relevant modules when a video processing method provided by an embodiment of the present application is implemented.
[0179] like Figure 7As shown, the electronic device 100 includes an image sensor (camera sensor), an automatic exposure control (AEC) module, a sensor front-end processor (or fusion module), an image processor front-end, a multi-camera smooth switching module, an image processor back-end and a local tone mapping (LTM) module included therein, etc.
[0180] Image sensor, which can correspond to Figure 3 The camera 193 shown is mainly used to capture video images. In some embodiments, the image sensor captures image frames by exposure. When the reflected light of the photographed object passes through the lens, it converges on the image sensor; the image sensor can convert the optical signal into an analog electrical signal and transmit the analog electrical signal to the fusion module. Among them, the image sensor outputs the original digital image captured by it, that is, the original (RAW) image. In some embodiments, in the initial stage of shooting (when the camera is turned on), the image sensor can capture images according to the default exposure mode. Later, after determining the exposure mode that is suitable for the current shooting conditions, the image sensor can shoot in a target mode under the instruction of the AEC module. Among them, the default exposure mode can be, for example, a binning exposure mode, but the embodiments of the present application are not limited to this.
[0181] In some embodiments, when different exposure modes are used to capture images, the image sensor may output different numbers of frames for the same image at the same time. For example, when a binning exposure mode is used to capture images, the image sensor may output a single frame (e.g. Figure 1 When shooting in DXG exposure mode, the image sensor outputs double frames.
[0182] In addition, in the embodiment of the present application, the image sensor can also perform image fusion. For example, in the DXG exposure mode, after the image sensor acquires two frames of images, the two frames of images can be fused into one frame. The image sensor can also be used to adjust image parameters (such as image size, bit depth, etc.), such as adjusting the image sizes acquired by different exposure modes to a consistent target size; or adjusting the image bit depths acquired by different exposure modes to a consistent target bit depth, etc.
[0183] The sensor front-end processor (also known as the fusion module) is used to fuse the RAW image frames collected by the image sensor, and can achieve single-frame input and single-frame output, or two-frame input and single-frame output.
[0184] For example, the SHDR exposure method can obtain two frames of images with different exposure times. Therefore, when shooting with the SHDR exposure method, the initial image input by the image sensor to the fusion module can be two frames of images with a certain frame rate (such as 30fps) and different exposure times, that is, a dual-frame mode. After the fusion module obtains the initial image, it fuses the two frames of images with different exposure times into one frame, and adjusts the parameters of the fused image to the preset target parameters, such as adjusting the initial format RAW10 of the input image to the target format RAW14.
[0185] For another example, the DXG exposure method is to obtain two frames of images based on one exposure. Therefore, when shooting with the DXG exposure method, one exposure of the image sensor can correspond to inputting two frames of initial images to the fusion module, which is also a dual-frame mode. The two initial image frames can be images read through high conversion gain and low conversion gain respectively, or the two frames of images can also be images read through high analog gain and low analog gain respectively, or the two frames of images can also be images after superimposing the images read through conversion gain and analog gain. The initial format of the input image can be RAW10, for example, and the frame rate can be 30fps, for example. After the fusion module obtains the initial image, the two frames of images read out with different gains are fused into one frame, and the parameters of the fused image can be adjusted to the preset target parameters, such as adjusting the input initial format RAW10 to the target format RAW14.
[0186] For example, the binning exposure method can obtain a single-frame image. When the binning exposure method is used for shooting, one exposure of the image sensor can correspond to the input of a single-frame initial image to the fusion module, that is, the single-frame mode. After the fusion module obtains the initial image, it does not need to perform image fusion, but can adjust the image parameters according to the needs, such as adjusting the input initial format RAW10 to the target format RAW14.
[0187] It should be noted that the purpose of adjusting the format of the initial image is to unify the images obtained by different exposure methods into the same format (size, bit depth, etc.) so that the screen size will not jump when the video is played later.
[0188] In some embodiments, if the image sensor fuses the image, a single frame can be input into the fusion module. In this case, the fusion module does not need to perform fusion processing on the image or adjust the image size, but outputs a single frame to the image processor front end for further processing.
[0189] The image processor front end (image processor front end) is used to perform front-end processing on the image to obtain an initial preview image. The image processor front end may specifically include a statistics module (STATS) and a GTM module, wherein the GTM module is used to globally brighten the dark part of the image to improve the image quality.
[0190] In some embodiments, the image output by the image processor front end can be divided into the following two transmission paths:
[0191] (1) One transmission path is from the front end of the image processor to the back end of the image processor for back end processing. Among them, the back end of the image processor may include a local tone mapping (LTM) module, which is used to locally brighten the dark part of the image. Afterwards, the back end of the image processor continues to transmit the image processed by the back end to the perception module, and the perception module can obtain the ambient brightness based on the image, and indicate the ambient brightness to the automatic exposure control AEC module. Then, the AEC module determines the exposure mode according to the ambient brightness, and instructs the image sensor to switch the exposure mode. Alternatively, the perception module can obtain the ambient brightness based on the image, determine the exposure mode according to the ambient brightness, and then indicate the exposure mode to the AEC module. Afterwards, the AEC module determines the exposure mode according to the ambient brightness, and instructs the image sensor to switch the exposure mode.
[0192] In some embodiments, the way in which the perception module obtains the ambient brightness based on the image may include: comprehensively judging the ambient brightness based on the current state of the image sensor, exposure parameters, and brightness channel information of the image frame.
[0193] It should be noted that although the AEC module can be used to control the exposure mode, the system's AEC module itself does not have the enabling conditions for determining the SHDR and DXG exposure modes. Instead, it needs to be combined with the ambient brightness detection results of the perception module. When the ambient brightness is greater than X, the SHDR exposure mode is used, and when the ambient brightness is less than Y, the DXG exposure mode is used. X and Y can be set according to actual needs, but in order to prevent ping-pong, a dual threshold design can be used, that is, X is greater than Y, such as X = 500lux, Y = 300lux.
[0194] (2) Another transmission process is from the image processor front end to the multi-camera smooth switching module, which is used to use a multi-camera smooth switching algorithm (such as SAT (spatial alignment transform) algorithm) to achieve a smooth image switching process from one camera to another. After the multi-camera smooth switching module performs image smooth switching, the processed image can be transmitted to the image processor back end. Among them, the image processor back end can locally brighten the dark part of the image through the LTM module it includes. Afterwards, a preview video image can be output at a specific frame rate (such as 30fps), or a video type (vedio) file can be generated and stored at a specific frame rate (such as 30fps).
[0195] It should be understood that the process of the image processor front end and the image processor back end performing front-end processing and back-end processing on the image respectively can refer to the existing flow, which will not be described in detail here.
[0196] It should be noted that in the embodiment of the present application, the smooth switching operation performed by the multi-camera smooth switching module is mainly used to realize the switching between the images captured by the HDR camera and the non-HDR camera respectively. Specifically, when there is a switch between the HDR camera and the non-HDR camera during the video shooting process, the multi-camera smooth switching module can be used to smoothly switch the image of the HDR camera to the image of the non-HDR camera, or to smoothly switch the image of the non-HDR camera to the image of the HDR camera, that is, to realize the smooth switching between the images captured by the two different types of cameras, so as to ensure the smooth playback of the video and avoid the problem of image jump caused by direct switching.
[0197] It should also be noted that the HDR camera in the embodiment of the present application may be a main camera, and the non-HDR camera may be a non-main camera, such as a wide-angle camera, etc. Optionally, the HDR camera and the non-HDR camera in the embodiment of the present application may not specifically refer to two different cameras, but may refer to a camera using an HDR mode and a camera not using an HDR mode. For example, a camera is regarded as an HDR camera when it uses an HDR solution to capture images, and is regarded as a non-HDR camera when it does not use an HDR solution to capture images.
[0198] For example, Figure 8 FIG. 1 is a schematic flow chart of a video processing method provided in an embodiment of the present application. The method may be implemented by the electronic device 100 described above, and may be implemented by various functional modules in the electronic device. The method may include the following steps:
[0199] S801: The AEC module determines dynamic range information and obtains a first result.
[0200] The dynamic range information here may be a dynamic range or a dynamic range compression gain. The dynamic range compression gain may be calculated based on the dynamic range of the current image and a preset standard dynamic range.
[0201] In some embodiments, before executing step S801, the video processing method provided by the embodiment of the present application may further include: when receiving a first operation input by a user for starting shooting, in response to the first operation, the AEC module of the electronic device may instruct the image sensor to capture an image in a default exposure mode; after the AEC module obtains the image captured by the image sensor, the dynamic range of the current image may be obtained through a histogram. The specific process of obtaining the dynamic range of the current image according to the histogram can refer to the existing process, and the embodiment of the present application will not be described in detail.
[0202] In some embodiments, during video capture, as the image sensor captures more and more frames of images, the AEC module may obtain a current dynamic range based on the multiple frames of images, such as obtaining an average dynamic range based on a histogram corresponding to each frame of image.
[0203] S802: Determine whether the first result is greater than a preset first threshold.
[0204] Among them, the first threshold can be flexibly set according to actual conditions, and the embodiments of the present application do not limit this.
[0205] In some embodiments, the level of the current dynamic range can be determined based on the correlation between the first result and the first threshold. For example, when the first result is greater than the first threshold, it means that the brightness difference of the current shot is large, and the current dynamic range can be determined to be a high dynamic range; when the first result is not greater than the first threshold, it means that the brightness difference of the current shot is small, and the current dynamic range can be determined to be a medium-low dynamic range.
[0206] It should be noted that different dynamic ranges can correspond to different exposure methods. For example, when it is a high dynamic range, the exposure method is determined to be binning. When it is a medium or low dynamic range, the exposure method needs to be further determined based on the flicker situation, ambient brightness and preset judgment method.
[0207] S803: When the first result is not greater than the first threshold, the AEC module instructs the image sensor to adopt a binning exposure mode.
[0208] S804: When the first result is greater than the first threshold, the AEC module determines whether the image has stroboscopic light and obtains a second result.
[0209] S805: When the second result indicates that there is flicker, the AEC module instructs the image sensor to adopt a binning exposure mode.
[0210] In some embodiments, during the video shooting process, since the devices all use AC power, flicker may occur in the captured image. According to the flicker result, it can be determined whether to use the binning exposure method, wherein when flicker exists, it is determined to use the binning exposure method; when flicker does not exist, it can be further determined to use the SHDR or DXG exposure method according to the ambient brightness.
[0211] It should be noted that the exposure time corresponding to the binning exposure method in the embodiment of the present application can be greater than or equal to the AC power cycle. For example, when the AC power frequency is 50 Hz, the exposure time of the binning exposure method can be 10 ms.
[0212] S806: When the second result indicates that there is no flicker, the perception module determines the ambient brightness.
[0213] In some embodiments, the perception module can obtain an image captured by an image sensor and obtain the ambient brightness based on the image.
[0214] S807: When the ambient brightness is greater than the second threshold, the AEC module instructs the image sensor to adopt the SHDR exposure mode.
[0215] Among them, the second threshold can be flexibly set according to actual conditions, and the embodiments of the present application do not limit this.
[0216] S808: When the ambient brightness is less than the third threshold and greater than the fourth threshold, the AEC module instructs the image sensor to adopt the DXG exposure mode.
[0217] The third threshold is smaller than the second threshold. The third threshold and the fourth threshold can be flexibly set according to actual conditions, and the embodiment of the present application does not limit this.
[0218] S809: When the ambient brightness is less than a fourth threshold, the AEC module instructs the image sensor to adopt a binning exposure mode.
[0219] According to the video processing method provided in the embodiment of the present application, by seamlessly switching between various types of HDR processing schemes according to changes in factors such as ambient brightness, required dynamic range, and flicker detection, it is possible to use an HDR scheme that is adapted to the actual shooting situation and image quality requirements for image processing, thereby effectively expanding the dynamic range in the recording scene and improving the image quality in the recording scene.
[0220] It should be noted that, in other embodiments, if the image sensor of the electronic device does not support certain exposure modes (such as DXG), then the video processing method provided in the embodiment of the present application can also be switched according to the SHDR and binning exposure modes. At this time, the exposure modes corresponding to different influencing factors can be as shown in Table 2:
[0221] Table 2
[0222] Influencing factors Exposure method High ambient brightness + high dynamic range SHDR Low brightness + high dynamic range in the environment binning Low dynamic range binning There is flickering phenomenon binning Thermal escape binning
[0223] Alternatively, in some other embodiments, if the image sensor of the electronic device does not support certain exposure modes (such as SHDR), the video processing method provided in the embodiment of the present application can also be switched according to the DCG and binning exposure modes. At this time, the exposure modes corresponding to different influencing factors can be as shown in Table 3:
[0224] Table 3
[0225] Influencing factors Exposure method High ambient brightness + high dynamic range DCG Low brightness + high dynamic range in the environment DCG Low dynamic range binning Thermal escape binning
[0226] When switching between different exposure modes, a seamless switching effect can be achieved. The specific means for achieving the seamless switching effect will be described in detail below and will not be described here.
[0227] For example, Fig. 9 , which is a schematic flow chart of another video processing method provided in an embodiment of the present application. Fig. 9 The interaction process between various functional modules is shown, which may specifically include the following steps:
[0228] S901, the automatic exposure control module sends first indication information to the image sensor.
[0229] In some embodiments, the automatic exposure module may preset a default exposure mode, and the default exposure mode may be binning, for example.
[0230] In some embodiments, before executing step S901, the video processing method provided in the embodiment of the present application may further include: when receiving a first operation input by a user for starting shooting, in response to the first operation, the AEC module of the electronic device may instruct the image sensor to capture an image according to a default exposure mode. Exemplarily, the first operation may be used to start video recording, for example, by clicking on Figure 6C The operations of recording images in the main interface of the camera are shown.
[0231] S902: The image sensor captures an image using a default exposure mode in response to the first indication information.
[0232] It should be noted that, in the embodiment of the present application, the image sensor is only used to capture original images, such as images in RAW10 format, without performing image fusion processing or adjusting the image size.
[0233] S903: The image sensor sends the first image to the fusion module.
[0234] The first image may be an original image captured by an image sensor, such as an image in RAW10 format.
[0235] Optionally, when the default exposure mode is SHDR or DXG exposure mode to capture the image in the dual-frame mode, step S904 can also be performed, that is, the fusion module performs multi-frame fusion on the first image. When the image sensor uses the binning exposure mode to capture the image in the single-frame mode, the fusion module can be input in the single-frame mode. At this time, the fusion module does not need to perform fusion processing and can only adjust the image parameters.
[0236] S905, the fusion module sends the second image to the image processor front end.
[0237] The second image may be an image processed by a fusion module. The fusion module may be used to perform fusion processing on the acquired images and may also adjust the image size according to the location.
[0238] S906: The image processor front end sends the third image to the image processor back end.
[0239] The third image may be an image obtained by front-end processing of the second image by an image processor front-end.
[0240] In some embodiments, after the image processor front end performs front-end processing on the image, the third image can be output according to a two-way transmission process. One transmission process is to transmit the third image to the multi-camera smooth switching module, and then the multi-camera smooth switching module transmits it to the image processor back end. After the image processor back end receives the image, the following step S907A can be executed; the other transmission process is to transmit the third image to the image processor back end, and then the image processor back end sends the image to the perception module, that is, execute the next step S907B.
[0241] S907A, the image processor backend performs backend processing to obtain the preview video and the stored video file.
[0242] S907B, the image processor backend sends the fourth image to the perception module.
[0243] The fourth image may be an image obtained after the image processor performs back-end processing on the third image.
[0244] S908, the perception module obtains the ambient brightness based on the fourth image.
[0245] S909, the perception module sends the ambient brightness to the automatic exposure control module.
[0246] S910, the automatic exposure control module determines an exposure mode according to preset influencing factors, where the influencing factors include ambient brightness.
[0247] S911, the automatic exposure control module sends second indication information to the image sensor.
[0248] The second indication information is used to instruct the image sensor to capture an image using a target exposure method.
[0249] S912: The image sensor continues to use a default exposure mode or other exposure modes in response to the second indication information.
[0250] After the image sensor acquires the original image (such as RAW10 image) using the target exposure mode, the real-time exposure mode can be determined according to the above steps S903 to S911, and the image can be captured and processed using the target exposure mode.
[0251] According to the video processing method provided in the embodiment of the present application, by seamlessly switching between various types of HDR processing schemes according to changes in factors such as ambient brightness, required dynamic range, and flicker detection, it is possible to use an HDR scheme that is adapted to the actual shooting environment and image quality requirements for image processing, thereby effectively expanding the dynamic range in the recording scene and improving the image quality in the recording scene.
[0252] It should be noted that in the video processing method provided in the embodiment of the present application, when the image sensor has the function of fusing images and adjusting the image size, the image sensor can perform the operation of fusing multiple frames of images in the DXG exposure mode and adjusting the image size. The video processing process under this mode is introduced below in conjunction with the accompanying drawings.
[0253] For example, Fig.10 As shown, it is a schematic diagram of data flow interaction between relevant modules when implementing another video processing method provided by an embodiment of the present application.
[0254] In some embodiments, when video shooting is turned on, the automatic exposure control module can send exposure indication information to the image sensor, instructing the image sensor to capture images using a default exposure mode (such as a binning exposure mode). The image sensor collects images (such as RAW10) according to the default exposure mode, and performs image fusion (under DXG or SHDR exposure mode) and image parameters (such as image size, bit depth, etc.) on the images, and then outputs the images (such as RAW14) to the fusion module. The fusion module transmits the acquired image to the image processor front end, and after the image processor front end performs front-end processing on the image, the image is output according to two transmission paths: (1) One transmission path is that the image processor front end transmits the image to the image processor back end, and the image processor back end can perform back-end processing on the image, and then transmits the image to the perception module, which can obtain the ambient brightness according to the image and transmit the ambient brightness to the decision module, which is used to determine the target exposure mode according to the ambient brightness, wherein the target exposure mode is an exposure mode that matches the actual conditions such as the current dynamic range, strobe state, and ambient brightness; then, the decision module sends exposure mode indication information to the image sensor, instructing the image sensor to switch to the target exposure mode. (2) Another transmission path is that the image processor front end transmits the image to the multi-camera smooth switching module. Optionally, the multi-camera smooth switching module can use a multi-camera smooth switching algorithm to achieve smooth switching of images from one camera to another. After the multi-camera smooth switching module performs smooth image switching, the processed image can be transmitted to the image processor back end. Afterwards, the image processor back end module can locally brighten the dark part of the image through the LTM module it includes, and then output a preview video image at a specific frame rate (such as 30fps), or generate and store a video type (vedio) file at a specific frame rate (such as 30fps).
[0255] It should be understood that the process of the image processor front end and the image processor back end performing front-end processing and back-end processing on the image respectively can refer to the existing flow, which will not be described in detail here.
[0256] It should be noted that in the embodiment of the present application, the smooth switching operation performed by the multi-camera smooth switching module is mainly used to realize the switching between the images captured by the HDR camera and the non-HDR camera respectively. Specifically, when there is a switch between the HDR camera and the non-HDR camera during the video shooting process, the multi-camera smooth switching module can be used to smoothly switch the image of the HDR camera to the image of the non-HDR camera, or to smoothly switch the image of the non-HDR camera to the image of the HDR camera, that is, to realize the smooth switching between the images captured by the two different types of cameras, so as to ensure the smooth playback of the video and avoid the problem of image jump caused by direct switching.
[0257] It should also be noted that the HDR camera in the embodiment of the present application may be a main camera, and the non-HDR camera may be a non-main camera, such as a wide-angle camera, etc. Optionally, the HDR camera and the non-HDR camera in the embodiment of the present application may not specifically refer to two different cameras, but may refer to a camera using an HDR mode and a camera not using an HDR mode. For example, a camera is regarded as an HDR camera when it uses an HDR solution to capture images, and is regarded as a non-HDR camera when it does not use an HDR solution to capture images.
[0258] According to the video processing method provided in the embodiment of the present application, by seamlessly switching between various types of HDR processing schemes according to changes in factors such as ambient brightness, required dynamic range, and flicker detection, it is possible to use an HDR scheme that is adapted to the actual shooting environment and image quality requirements for image processing, thereby effectively expanding the dynamic range in the recording scene and improving the image quality in the recording scene.
[0259] For example, Fig.11 , which is a schematic flow chart of another video processing method provided in an embodiment of the present application. Fig.11 The interaction process between various functional modules is shown, which may specifically include the following steps:
[0260] S1101, the automatic exposure control module sends first indication information to the image sensor.
[0261] In some embodiments, the automatic exposure module may preset a default exposure mode, and the default exposure mode may be binning, for example.
[0262] In some embodiments, before executing step S1101, the video processing method provided by the embodiment of the present application may further include: when receiving a first operation input by the user for starting shooting, in response to the first operation, the AEC module of the electronic device may instruct the image sensor to capture an image according to a default exposure mode. Exemplarily, the first operation may be used to start video recording, for example, by clicking on Figure 6C The operations of recording images in the main interface of the camera are shown.
[0263] S1102: In response to the first indication information, the image sensor captures an image using a default exposure method and performs image processing according to a preset strategy.
[0264] It should be noted that Fig. 9The difference between the embodiments is that the image sensor in the embodiments of the present application can have the ability to fuse images, adjust image size, etc., and can process images according to preset strategies. For example, when the default exposure mode is DCG, DAG, DXG or other dual-frame modules to acquire images, the image sensor can fuse the dual-frame first image. When the image sensor uses the binning exposure mode to capture images, the single-frame mode is input into the fusion module. At this time, the fusion module does not need to perform fusion processing and can only adjust the image size.
[0265] For example, for the convenience of description, the embodiment of the present application will obtain two frames of images in the image sensor through the DXG method and record the method of fusing them as iDXG. Fig.11 , the image sensor may fuse two frames of images by: acquiring two frames of images with a preset sensitivity ratio, and then fusing the two frames of images to acquire an image with a larger dynamic range. Limited by the hardware performance of the image sensor, the sensitivity ratio of the two frames of images acquired by the iDXG method can be preset, such as iDXG1:4, iDXG1:8 or iDXG1:16 and other fixed values.
[0266] Different switching conditions can correspond to different sensitivity ratios, for example: in high dynamic scenes, it can correspond to iDXG1:4 (that is, using iDXG exposure method, and the sensitivity ratio is 1:4); in extremely high dynamic scenes, it can correspond to iDXG 1:16 (that is, using iDXG exposure method, and the sensitivity ratio is 1:16); in medium and low dynamic scenes, it can correspond to binning exposure method; in thermal escape scenes, it can correspond to binning exposure method.
[0267] Among them, the extremely high dynamic range, high dynamic range and medium and low dynamic range can be divided according to needs, and this application does not make specific limitations on each dynamic range.
[0268] In contrast, Fig. 9 In the embodiment, when the fusion module performs dual-frame image fusion, the sensitivity ratio of the two frames of images can be flexibly set and is not limited to a few sets of fixed values. Fig. 9 The fusion method of the embodiment can facilitate flexible adjustment of the brightness of the image. Fig.11 The fusion method of the embodiment can realize HDR processing of the image while ensuring the performance of the image sensor.
[0269] S1103, the image sensor sends the fifth image to the image processor front end.
[0270] Specifically, the image sensor may first transmit the fifth image to the fusion module, and then the fusion module transparently transmits the fifth image to the front end of the image processor.
[0271] S1104: The image processor front end sends a sixth image to the image processor back end.
[0272] The sixth image may be an image obtained after the image processor front end performs front-end processing on the fifth image.
[0273] In some embodiments, after the image processor front end performs front-end processing on the image, the sixth image can be output according to a two-way transmission process. One transmission process is to transmit the sixth image to the multi-camera smooth switching module, and then the multi-camera smooth switching module transmits it to the image processor back end. After the image processor back end receives the image, the following step S1105A can be executed; the other transmission process is to directly transmit the sixth image to the image processor back end, and then the image processor back end sends the image to the perception module, that is, execute the next step S1105B.
[0274] S1105A, the image processor backend performs backend processing to obtain the preview video and the stored video files.
[0275] S1105B, the image processor back end sends the seventh image to the perception module.
[0276] S1106, the perception module obtains the ambient brightness based on the seventh image.
[0277] S1107, the perception module sends the ambient brightness to the decision module.
[0278] It should be noted that the decision module here may be an additionally provided module, or may be a submodule in the automatic exposure control, which is not limited in the embodiment of the present application.
[0279] S1108, the perception module determines the target exposure method according to a preset strategy.
[0280] The preset strategy refers to the strategy for determining the target exposure mode based on the current dynamic range, whether there is flicker, ambient brightness, etc., which can correspond to the above Figure 8 The strategy for determining the target exposure mode is introduced in the embodiment.
[0281] S1109, the decision module sends second indication information to the image sensor.
[0282] S1110: The image sensor captures an image in a target exposure mode in response to the second indication information.
[0283] After the image sensor acquires the original image (such as RAW10 image) using the target exposure mode, the real-time exposure mode can be determined according to the above steps S1103 to S1109, and the image can be captured and processed using the target exposure mode.
[0284] According to the video processing method provided in the embodiment of the present application, by seamlessly switching between various types of HDR processing schemes according to changes in factors such as ambient brightness, required dynamic range, and flicker detection, it is possible to use an HDR scheme that is adapted to the actual shooting environment and image quality requirements for image processing, thereby effectively expanding the dynamic range in the recording scene and improving the image quality in the recording scene.
[0285] In the video processing method provided in the embodiment of the present application, during video shooting, HDR mode and non-HDR mode can be switched between each other, wherein HDR mode is a video shooting mode processed by HDR technology, and non-HDR mode is a video shooting mode not processed by HDR technology. Generally speaking, the main camera and the secondary camera of an electronic device are included in a logical camera, wherein the main camera has HDR capability, and the secondary camera may have other capabilities (such as ultra-wide-angle capability) but not HDR capability. Therefore, when switching between HDR mode and non-HDR mode is required during video shooting, it can be achieved by switching between the main camera and the secondary camera.
[0286] For example, Fig.12 As shown, it is a schematic flowchart of a method for video processing in a main camera and a secondary camera switching scenario provided in an embodiment of the present application.
[0287] Among them, when the main camera and the secondary camera are switched, in order to reduce the image jump after the switch, the following methods can be used:
[0288] (1) In low-dynamic scenes, or scenes with flicker or thermal runaway, binning is used as the exposure method. At this time, the system's dynamic range gain remains flat and the image effect does not jump.
[0289] (2) In high-dynamic scenarios where there is no flicker or thermal runaway, the main camera can capture images according to the target exposure method determined by the method described above, and the secondary camera uses the binning exposure method and maintains the maximum dynamic range, trying to be close to the dynamic range of the main camera to reduce image jumps.
[0290] See also Fig.12 As shown, in some embodiments, the main camera can capture images in multiple exposure modes, including SHDR, DXG and binning exposure modes. After the main camera captures the image, the initial image (such as RAW10 image) can be transmitted to the fusion module. When the fusion module is input in double-frame mode, the fusion module fuses the double-frame images and transmits the fused image (RAW14 image) to the image processor front end 0.
[0291] The secondary camera may capture an image by a binning exposure method, and then transmit the captured initial image (eg, a RAW image) to the image processor front end 1 .
[0292] Image processor front end 0 and image processor front end 1 can transmit the images processed by their respective front ends to the multi-camera smooth switching module. The multi-camera smooth switching module can process the images corresponding to different cameras according to the multi-camera smooth switching algorithm, and realize smooth switching between images collected by two different types of cameras to ensure smooth video playback and avoid the problem of image jump caused by direct switching.
[0293] Afterwards, the fusion module can continue to transmit the image to the image processor backend, which can perform stabilization on the image through the preview stream stabilization module (such as EIS2.0 and EIS3.0), and then further process the image through the image affine transformation module, color conversion matrix (CCM) module, and gamma transformation (Gamma) module. The specific processing process can refer to the existing process and principles, which will not be repeated here.
[0294] After being processed by the image processor backend, a preview image with a certain frame rate (such as 30fps) and a video storage file with a certain frame rate (such as 30fps) can be obtained.
[0295] According to the video processing method provided in the embodiment of the present application, by seamlessly switching between various types of HDR processing schemes according to changes in factors such as ambient brightness, required dynamic range, and flicker detection, it is possible to use an HDR scheme that is adapted to the actual shooting environment and image quality requirements for image processing, thereby effectively expanding the dynamic range in the recording scene and improving the image quality in the recording scene.
[0296] In the video processing method provided in the embodiment of the present application, multiple HDR capabilities can be supported, such as HDR10+ video capabilities, that is, electronic devices can apply HDR10+ technology to record or play videos. In some embodiments, HDR10+ capabilities can mainly act on the back-end part of the image processor, including adjusting the color conversion matrix module to BT.2020, adjusting the gamma transform to the PQ conversion curve, generating dynamic metadata, etc. That is, HDR10+ technology can be used simultaneously with seamless switching of different exposure modes. Among them, the exposure mode switching path using HDR10+ technology can be referred to. Fig. 9 , which will not be described in detail here.
[0297] In some embodiments, when the electronic device supports multi-state HDR capability, a corresponding switch control can be set. In addition, when the electronic device supports other HDR technologies (taking HDR10+ as an example), the switch control corresponding to HDR10+ can also be set. For electronic devices that do not support multi-state HDR or HDR10+, the switch control corresponding to multi-state HDR or HDR10+ on the electronic device can be grayed out.
[0298] The following describes the process of adjusting image parameters corresponding to different exposure modes by taking the adjustment of bit depth as an example in conjunction with the accompanying drawings.
[0299] For example, Fig.13 , which is a schematic diagram of a method for adjusting image parameters provided in an embodiment of the present application.
[0300] In some embodiments, the method of adjusting the image parameters may include: padding the low bits with 0 by means of Pad0 so that the images corresponding to different exposure modes have the same bit depth. Fig.13 As shown, the image corresponding to the iDCG1:4 exposure mode has 0s added to the lower 2 bits, where 1:4 is the sensitivity ratio between two frames of images with different gains of iDCG; the image corresponding to the binning exposure mode has 0s added to the lower 4 bits. After the low bits are filled with 0s, the images obtained by the iDCG and binning exposure modes have the same bit depth.
[0301] For example, Fig.14 , which is a schematic diagram of a method for adjusting image parameters provided in an embodiment of the present application.
[0302] In some embodiments, an image sensor setting that supports seamless switching of multiple exposure modes (such as binning, DXG, and SHDR) can unify the bit depth, such as unifying the bit depth to RAW 14. Among them, for images with a bit depth less than 14 bits, they can be unified to RAW14 by padding.
[0303] It should be noted that in the video processing method provided in the embodiment of the present application, the unified bit depth can be based on the bit depth of the iDXG image, because among the several exposure methods provided in the present application, the iDXG exposure method corresponds to the image with the largest bit depth. When other exposure methods are used in actual applications, the unified bit depth can also be flexibly set according to the type of exposure method, and the embodiment of the present application does not limit this.
[0304] Taking the unified bit depth of 14 bits as an example, the bit depth can be adjusted as follows: Fig.13As shown, it includes: padding the image captured by binning exposure from 10bit to 14bit; padding the image obtained by iDXG exposure method with a sensitivity ratio of 1:4 and the fusion of two frames from 12bit to 14bit; the image obtained by iDXG exposure method with a sensitivity ratio of 1:16 and the fusion of two frames is 14bit, and no padding is required at this time.
[0305] It should also be noted that in the video processing method provided in the embodiment of the present application, different exposure modes may correspond to different sensitivity ratios. It should be noted that the sensitivity ratio may be a combination of exposure time (exp) and system gain (gain). The system gain may be set in multiple groups, such as gain=1, gain=4, gain=16, and so on.
[0306] Assume that there are currently three exposure modes: binning, iDXG1:4, and iDXG1:16, and the three exposure modes correspond to different exposure times and system gains. Then, the sensitivity ratios corresponding to different exposure modes can be obtained based on the exposure time and system gain. For example, the principle of obtaining the sensitivity ratio may include: (1) when the system dynamic range compression gain is 3.99 (i.e., adrc gain=3.99), the exposure mode is binning mode, the exposure time is a ms (exp=a ms), the system gain is 1, and the sensitivity ratio after the exposure time and the system gain are combined is 1; (2) when the system dynamic range compression gain is 4 (i.e., adrc gain=4), the exposure mode is iDXG1:4, the exposure time is a ms (exp=a ms), the system gain is 4, and the sensitivity ratio after the exposure time and the system gain are combined is 4; (3) when the system dynamic range compression gain is 4 (i.e., adrc gain=4), the exposure mode is iDXG1:16, the exposure time is a ms (exp=a ms), the system gain is 16, and the sensitivity ratio after the exposure time and the system gain are combined is 16. Exemplarily, the sensitivity ratios corresponding to the various exposure modes may be shown in Table 4:
[0307] Table 4
[0308] Exposure method Sensitivity ratio binning 1 iDXG 1:4 4 iDXG 1:16 16
[0309] Under different exposure modes, AEC can instruct the image sensor to capture images using the gain corresponding to the exposure mode. Therefore, images captured by different exposure modes may correspond to different sensitivities. In order to avoid image brightness jumps caused by different sensitivities between images corresponding to different exposure modes when switching between different exposure modes, the video processing method provided in the embodiment of the present application solves the above problem by adjusting the sensitivity ratio corresponding to the exposure mode.
[0310] Exemplarily, the exposure modes and sensitivity ratios in 3 given in Table 4 above are used as examples. The binning exposure mode in Table 4 above corresponds to a sensitivity ratio of 1, and the sensitivity ratio of iDXG1:4 corresponds to 4. When the exposure mode is switched from binning to iDXG1:4, the iDXG1:4 exposure mode can be increased to 4 times the original; or, the binning exposure mode in Table 4 above corresponds to a sensitivity ratio of 1, and the sensitivity ratio of iDXG1:16 corresponds to 16. When the exposure mode is switched from binning to iDXG1:16, the iDXG1:16 exposure mode can be increased to 16 times the original; or, the iDXG1:4 exposure mode in Table 4 above corresponds to a sensitivity ratio of 4, and the sensitivity ratio of iDXG1:16 corresponds to 16. When the exposure mode is switched from iDXG1:4 to iDXG1:16, the iDXG1:16 exposure mode can be increased to the original 4. It should be noted that the several sensitivity ratios given in Table 4 above are only examples. In actual applications, the sensitivity ratios corresponding to different exposure modes may also be other specific values, which are not limited in the embodiments of the present application.
[0311] By adjusting the sensitivity of the image corresponding to the switching of different exposure modes, the jump of image brightness can be avoided. When switching between different exposure modes, the video image can also be switched smoothly to ensure the image quality.
[0312] For example, Fig.15 FIG. 1 is a schematic flow chart of another video processing method provided in an embodiment of the present application. The method may include the following steps:
[0313] S1501, receiving a first operation input by a user, where the first operation is used to enable a recording function of an electronic device.
[0314] The first operation may correspond to the above Figure 6C Alternatively, the first operation may also correspond to the operation of clicking to turn on the smart HDR function.
[0315] In some embodiments, when the video recording function is turned on, the electronic device may display a first interface, wherein the first interface includes a first control, and the first control is used to turn on the automatic switching of the HDR exposure mode function. The first interface may correspond to, for example, Fig. 6A or Fig.6D The interface shown.
[0316] S1502: In response to a first operation, a first image is acquired according to a preset default high dynamic range (HDR) exposure mode.
[0317] Among them, the exposure mode in the embodiment of the present application may include: binning exposure mode, SHDR exposure mode, DXG exposure mode, etc. Among them, according to the number of image frames corresponding to different exposure modes, it can be divided into a first HDR exposure mode and a second HDR exposure mode. The first HDR exposure mode is a single-frame mode, that is, one frame of image can be obtained based on one exposure, and the first HDR exposure mode can be a binning mode; the second HDR exposure mode is a double-frame mode, that is, two frames of image can be read based on one exposure, such as DXG mode, or two long and short exposures are performed to obtain two frames of image, such as SHDR mode.
[0318] In some embodiments, a default HDR exposure mode may be preset, and when the first operation is received, the AEC module may instruct the image sensor to capture an image using the default HDR exposure mode. Exemplarily, the default HDR exposure mode may be, for example, a binning mode.
[0319] S1503, obtaining the ambient brightness according to the first image, and determining the target HDR exposure mode according to a preset strategy, where the preset strategy includes dynamic range information corresponding to the video shooting, a stroboscopic state, and a correspondence between the ambient brightness and the target HDR exposure mode.
[0320] The dynamic range compression information here may include a dynamic range and / or a dynamic range compression gain. The embodiment of the present application is introduced by taking the dynamic range as an example.
[0321] In some embodiments, determining the target HDR exposure mode according to a preset strategy may include: obtaining a dynamic range corresponding to the video shooting; when the dynamic range is less than a first threshold, determining that the target HDR exposure mode is the binning mode; when the dynamic range is greater than or equal to the first threshold, detecting whether there is flicker; when there is flicker, determining that the target HDR exposure mode is the binning mode; when there is no flicker, determining the target HDR exposure mode according to the ambient brightness; wherein, when the ambient brightness is greater than a second threshold, determining that the target HDR exposure mode is the SHDR mode; when the ambient brightness is less than a third threshold and greater than a fourth threshold, determining that the target HDR exposure mode is the DXG mode, wherein the third threshold is less than the second threshold; when the ambient brightness is less than the fourth threshold, determining that the target HDR exposure mode is the binning mode.
[0322] It should be noted that, in addition to determining the exposure mode based on the dynamic range, the exposure mode can also be determined based on the dynamic range compression gain. The process of determining the exposure mode based on the dynamic range compression gain is similar to the above process (the dynamic range can be replaced by the dynamic range compression gain), and this will not be repeated. When the dynamic range is replaced by the dynamic range compression gain, the specific value of the first threshold value can also change accordingly, that is, the specific values of the first threshold value corresponding to the dynamic range and the first threshold value corresponding to the dynamic range compression gain can be different.
[0323] In some embodiments, the method of determining the target HDR exposure mode according to a preset strategy may also include: detecting whether there is a thermal escape phenomenon in the video shooting; when the thermal escape phenomenon exists, determining that the target HDR exposure mode is the binning mode.
[0324] In some embodiments, when there is a switch between an HDR camera and a non-HDR camera during video shooting, the method further includes: when the switch occurs, the exposure mode of the HDR camera is the first HDR exposure mode, and the exposure mode corresponding to the non-HDR camera is the second HDR exposure mode, then adjusting the first dynamic range gain corresponding to the first HDR exposure mode so that the first dynamic range gain is closest to the second dynamic range gain corresponding to the second HDR exposure mode.
[0325] S1504: When the target HDR exposure mode is different from the default HDR exposure mode, the default HDR exposure mode is switched to the target HDR exposure mode, and the video shooting is continued to obtain a second image.
[0326] In some embodiments, when the target HDR exposure mode is the second HDR exposure mode, the images in the dual-frame mode are fused.
[0327] In some embodiments, when the target HDR exposure mode is the second HDR exposure mode, the image sensor transmits the image in dual-frame mode to the fusion module; the fusion module fuses the image in dual-frame mode; or, when the target HDR exposure mode is the second HDR exposure mode, the image sensor fuses the image in dual-frame mode.
[0328] In some embodiments, when the fusion module fuses the image in the dual-frame mode, the target sensitivity ratio between the dual-frame input image in the DCG mode and the dual-frame input image in the DAG mode is determined according to the sensitivity ratio required by the DXG mode; the DCG mode dual-frame input image and the DAG mode dual-frame input image are superimposed according to the target sensitivity ratio to obtain the superimposed dual-frame input image that meets the sensitivity ratio of the DXG mode.
[0329] In some embodiments, when the image sensor fuses the image in the dual-frame mode, the dual-frame input image in the DCG mode and the dual-frame input image in the DAG mode are superimposed according to a preset sensitivity ratio. Taking iDXG as an example, the sensitivity ratio is, for example, 1:4 or 1:16.
[0330] In some embodiments, for images acquired by different exposure methods, the image parameters corresponding to these exposure methods can be adjusted to be consistent. The specific process may include: presetting target parameters corresponding to the image acquired by the video shooting; adjusting initial parameters corresponding to the first image acquired based on the first HDR exposure method to the target parameters; and / or adjusting initial parameters corresponding to the second image acquired based on the second HDR exposure method to the target parameters.
[0331] In some embodiments, the electronic device supports the first HDR video mode, and the first HDR video mode includes HDR10 or HDR10+.
[0332] According to the video processing method provided in the embodiment of the present application, by seamlessly switching between various types of HDR processing schemes according to changes in factors such as ambient brightness, required dynamic range, and flicker detection, it is possible to use an HDR scheme that is adapted to the actual shooting environment and image quality requirements for image processing, thereby effectively expanding the dynamic range in the recording scene and improving the image quality in the recording scene.
[0333] Based on the same technical concept, an embodiment of the present application also provides an electronic device, including one or more processors; one or more memories; the one or more memories store one or more computer programs, and the one or more computer programs include instructions. When the instructions are executed by the one or more processors, the computer or processor executes one or more steps in any of the above methods.
[0334] Based on the same technical concept, an embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable program instructions. When the computer-executable program instructions are executed on a computer, the computer or processor executes one or more steps in any of the above methods.
[0335] Based on the same technical concept, an embodiment of the present application also provides a computer program product comprising instructions, wherein the computer program product includes computer program code. When the computer program code runs on a computer, the computer or processor executes one or more steps in any of the above methods.
[0336] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)), etc.
[0337] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.
[0338] The above is only a specific implementation of the embodiment of the present application, but the protection scope of the embodiment of the present application is not limited thereto, and any changes or replacements within the technical scope disclosed in the embodiment of the present application should be included in the protection scope of the embodiment of the present application. Therefore, the protection scope of the embodiment of the present application should be based on the protection scope of the claims.
Claims
1. A video processing method, It is characterized in that Applied to electronic equipment, the method comprises: Receiving a first operation input by a user, where the first operation is used to start a recording function of the electronic device; In response to the first operation, acquiring a first image according to a preset default high dynamic range (HDR) exposure mode; Acquire the ambient brightness according to the first image, and determine the target HDR exposure mode according to a preset strategy, wherein the preset strategy includes dynamic range information corresponding to the video shooting, a stroboscopic state, and a correspondence between the ambient brightness and the target HDR exposure mode; When the target HDR exposure mode is different from the default HDR exposure mode, the default HDR exposure mode is switched to the target HDR exposure mode, and the video shooting is continued to obtain a second image.
2. The method according to claim 1, It is characterized in that The target HDR exposure mode is a first HDR exposure mode or a second HDR exposure mode, the first HDR exposure mode is a single-frame mode, and the second HDR exposure mode is a dual-frame mode; When the target HDR exposure mode is the second HDR exposure mode, the image input in the dual-frame mode is fused.
3. The method according to claim 2, It is characterized in that The first HDR exposure mode is a binning mode, and the second HDR exposure mode includes a staggered high dynamic range mode SHDR and DXG, wherein the DXG is a mode in which a dual conversion gain mode DCG and a dual analog gain DAG are superimposed.
4. The method according to claim 2 or 3, It is characterized in that The method further comprises: Presetting target parameters corresponding to the image acquired through the video shooting; adjusting initial parameters corresponding to the first image acquired based on the first HDR exposure method to the target parameters; and / or, Adjusting initial parameters corresponding to the second image acquired based on the second HDR exposure method to the target parameters.
5. The method according to any one of claims 1 to 3, It is characterized in that The electronic device includes an automatic exposure control AEC module, an image sensor, and a sensing module. In response to the first operation, acquiring the first image according to a preset default HDR exposure mode specifically includes: In response to the first operation, the AEC module sends first indication information to the image sensor, where the first indication information is used to indicate that the image is captured using the default HDR exposure mode; In response to the first indication information, the image sensor uses the default HDR exposure mode and acquires the first image.
6. The method according to any one of claims 1 to 3, It is characterized in that The electronic device includes an AEC module, an image sensor, and a perception module. The step of acquiring the ambient brightness according to the first image and determining the target HDR exposure mode according to a preset strategy specifically includes: The image sensor sends the first image to the perception module; The perception module acquires the ambient brightness according to the first image, and indicates the ambient brightness to the AEC module; The AEC module determines the target HDR exposure mode according to the ambient brightness and a preset strategy.
7. The method according to claim 2 or 3, It is characterized in that The electronic device includes an AEC module, an image sensor, a perception module and a fusion module. When the target HDR exposure mode is the second HDR exposure mode, fusing the double-frame input images specifically includes: When the target HDR exposure mode is the second HDR exposure mode, the image sensor transmits the image in the dual-frame mode to the fusion module; The fusion module fuses the images in the dual-frame mode; or, When the target HDR exposure mode is the second HDR exposure mode, the image sensor fuses the image in the dual-frame mode.
8. The method according to claim 7, It is characterized in that When the fusion module fuses the image in the dual-frame mode, the target sensitivity ratio between the dual-frame input image in the DCG mode and the dual-frame input image in the DAG mode is determined according to the sensitivity ratio required by the DXG mode; The DCG mode double-frame input image and the DAG mode double-frame input image are superimposed respectively according to the target sensitivity ratio to obtain a superimposed double-frame input image that meets the DXG mode sensitivity ratio.
9. The method according to claim 7, It is characterized in that When the image sensor fuses the image in the dual-frame mode, the dual-frame input image in the DCG mode and the dual-frame input image in the DAG mode are superimposed according to a preset sensitivity ratio.
10. The method according to any one of claims 1 to 3, It is characterized in that Determining the target HDR exposure mode according to the preset strategy specifically includes: Obtaining a dynamic range corresponding to the video shooting; When the dynamic range information is less than a first threshold, determining that the target HDR exposure mode is the binning mode; When the dynamic range information is greater than or equal to the first threshold, detecting whether there is stroboscopic light; When there is stroboscopic light, determining that the target HDR exposure mode is the binning mode; When there is no stroboscopic light, the target HDR exposure mode is determined according to the ambient brightness; wherein, When the ambient brightness is greater than a second threshold, determining that the target HDR exposure mode is the SHDR mode; When the ambient brightness is less than a third threshold and greater than a fourth threshold, determining that the target HDR exposure mode is the DXG mode, wherein the third threshold is less than the second threshold; When the ambient brightness is less than the fourth threshold, the target HDR exposure mode is determined to be the binning mode.
11. The method according to claim 10, It is characterized in that The method further comprises: Detecting whether there is a thermal escape phenomenon in the video shooting; When the thermal escape phenomenon occurs, the target HDR exposure mode is determined to be the binning mode.
12. The method according to any one of claims 1 to 3, It is characterized in that The electronic device supports a first HDR video mode, and the first HDR video mode includes HDR10 or HDR10+.
13. The method according to claim 3, It is characterized in that When the video is shot, when there is a switch between the HDR camera and the non-HDR camera, the method further includes: When the switch occurs, the exposure mode of the HDR camera is the first HDR exposure mode, and the exposure mode corresponding to the non-HDR camera is the second HDR exposure mode, then the first dynamic range gain corresponding to the first HDR exposure mode is adjusted so that the first dynamic range gain is closest to the second dynamic range gain corresponding to the second HDR exposure mode.
14. The method according to any one of claims 1 to 3, It is characterized in that The method further comprises: A first interface is displayed, wherein the first interface includes a first control, and the first control is used to enable a function of automatically switching the HDR exposure mode.
15. An electronic device, It is characterized in that include: one or more processors; one or more memories; The one or more memories store one or more computer programs, and the one or more computer programs include instructions, which, when executed by the one or more processors, enable the electronic device to perform the method as claimed in any one of claims 1 to 14.
16. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer-executable program instructions, which, when executed on a computer, enable the computer to perform the method according to any one of claims 1 to 14.
Citation Information
Patent Citations
Method and device for intelligently switching to HDR mode of camera
CN103516984A
Method and equipment for realizing automatic HDR
CN112616013A
Photographing method, electronic equipment and storage medium
CN112738414A
Switching method of HDR sensor and non-HDR sensor and electronic equipment
CN114157791A
Method for generating HDR image and electronic equipment
CN114466134A
