Image processing methods
By employing a frame-out strategy that generates three images at adjacent exposure times, the high power consumption problem under the TCG pixel structure is solved, achieving high dynamic range imaging and reducing the power consumption of the image sensor and ISP.
Patent Information
- Application Number
- CN202510631902.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-15
- Publication Date
- 2026-11-17
AI Technical Summary
Existing image sensors consume a lot of power when achieving high dynamic range imaging, mainly because the TCG pixel structure outputs a large number of frames in a single exposure time, which increases the processing burden on the image signal processor.
A frame output strategy is adopted to generate three images in two adjacent exposure times, including an image frame in the first conversion gain mode, an image frame in the second conversion gain mode, and a third image frame with smaller exposure parameters. This reduces the number of processing frames for the image sensor and ISP, and uses the third image frame to supplement missing details in extremely bright light, thereby achieving high dynamic range imaging.
Without compromising image quality, the power consumption of the image sensor and ISP was reduced, and the energy efficiency of the image sensor was improved by optimizing the processing flow by reducing the number of frames.
Smart Images

Figure CN123123731A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image processing technology, and more particularly to an image processing method. Background Technology
[0002] Dynamic range is one of the key performance parameters of an image sensor. It determines the range of light intensity distribution from the darkest shadows to the brightest highlights that the image sensor can capture, thus determining the image quality. Specifically, the wider the dynamic range, the clearer the details in both highlights and shadows, resulting in better image quality. Conversely, a narrower dynamic range makes it easier for highlights to be overexposed and shadows to be underexposed, leading to greater loss of detail in both areas and poorer image quality.
[0003] Currently, to achieve a higher dynamic range, the pixels in image sensors mainly employ... Figure 1 The triple conversion gain (TCG) pixel structure shown can form C by controlling the on and off states of switches SG1 and SG2. par C par +Cs and C par +Cs+C LOFIC Three types of capacitors for charge-to-voltage conversion achieve high, medium, and low conversion gains. For different brightness scenes, the TCG pixel outputs high conversion gain (HCG) frames, low conversion gain (LCG) frames, and lateral overflow integration capacitor (LOFIC) frames in a single exposure time. These frames are used to supplement details in dark, bright, and extremely bright areas, respectively, thereby achieving high dynamic range imaging (HDR).
[0004] However, in the above scheme, the TCG pixel outputs three frames in one exposure time, and the image signal processor (ISP) also needs to process three frames. Therefore, it will result in high power consumption of the image sensor and ISP. Summary of the Invention
[0005] This application provides an image processing method that can reduce the power consumption of image sensors and ISPs without degrading image quality.
[0006] In a first aspect, this application provides an image processing method applied to an electronic device, the electronic device including a camera module. The method includes: the camera module generating a first image frame and a second image frame during a first exposure time in two adjacent exposure times, and generating a third image frame during a second exposure time in two adjacent exposure times, wherein the first image frame is an image frame in a first conversion gain mode, the second image frame is an image frame in a second conversion gain mode, and the third image frame is an image frame in either the first or second conversion gain mode, and the exposure parameters of the third image frame are less than or equal to the exposure parameters of the first and second image frames, the exposure parameters including exposure amount or exposure duration. Subsequently, the camera module outputs a first image or a first video based on the first image frame, the second image frame, and the third image frame.
[0007] It is understood that the image processing method of this application provides a frame output strategy that generates a first image frame, a second image frame, and a third image frame in two adjacent exposure times. Compared to the scheme of an image sensor using a TCG pixel structure generating six image frames (including two HCG frames, two LCG frames, and two LOFIC frames) in two adjacent exposure times, the frame output strategy provided by this application can reduce the number of image frames generated by the image sensor in two adjacent exposure times by half, thereby reducing the number of image frames that the ISP needs to process in the subsequent process by half, and thus reducing the power consumption of the image sensor and the ISP. In addition, when the first conversion gain mode is HCG mode and the second conversion gain mode is LCG mode, the first image frame is an HCG frame, the second image frame is an LCG frame, and the third image frame is an HCG frame or an LCG frame with exposure parameters that are smaller than those of the first and second image frames. The third image frame is used to supplement the extremely bright light details missing in the HCG and LCG frames. In other words, the third image frame can be equivalent to the role of the LOFIC frame in image imaging. Therefore, the image processing method provided by this application can also achieve HDR without degrading image quality. In summary, the image processing method provided in this application can reduce the power consumption of the image sensor and ISP without reducing image quality.
[0008] In one possible implementation, the camera module also generates a fourth and fifth image frame at a third exposure time. The third exposure time is the next exposure time after two adjacent exposure times. The fourth image frame is an image frame in a first conversion gain mode, and the fifth image frame is an image frame in a second conversion gain mode. The exposure parameters of both the fourth and fifth image frames are greater than or equal to the exposure parameters of the third image frame. Subsequently, based on the first, second, third, fourth, and fifth image frames, the camera module outputs a first image or a first video.
[0009] In this embodiment, the image processing method provides a frame output strategy that generates a first image frame, a second image frame, a third image frame, a fourth image frame, and a fifth image frame in three adjacent exposure times. Compared to a scheme where an image sensor using a TCG pixel structure generates nine image frames (including three HCG frames, three LCG frames, and three LOFIC frames) in three adjacent exposure times, the frame output strategy provided in this application reduces the number of image frames generated by the image sensor in three adjacent exposure times by nearly half, thereby reducing the number of image frames that the ISP needs to process by nearly half, and thus reducing the power consumption of the image sensor and the ISP. Furthermore, when the first conversion gain mode is HCG mode and the second conversion gain mode is LCG mode, the first and fourth image frames are both HCG frames, the second and fifth image frames are both LCG frames, and the third image frame is an HCG frame or LCG frame with exposure parameters smaller than those of the first, second, fourth, and fifth image frames. The third image frame is used to supplement the missing extremely bright light details in the HCG and LCG frames. Therefore, the image processing method provided in this embodiment can also achieve HDR without degrading image quality.
[0010] In one possible implementation, the camera module determines one of the first and second image frames as a first reference frame, and fuses the other image frame from the first and second image frames based on the first reference frame to generate a sixth image frame. The camera module then determines a third image frame as a second reference frame, and fuses the sixth image frame based on the second reference frame to generate a seventh image frame. Subsequently, the camera module inserts the seventh image frame into the sixth image frame and outputs the first video, wherein the seventh image frame is the next image frame after the sixth image frame.
[0011] It is understood that in this embodiment, the third image frame is used as the reference frame when fusing the third and sixth image frames. The preceding third image frame in the temporal domain is used to supplement the corresponding dynamic range and noise reduction processing, thus enabling the first video to achieve HDR. Since the seventh image frame is inserted as the next image frame after the sixth image frame, the first video can also achieve a high frame rate.
[0012] In one possible implementation, the exposure of the first reference frame is the same as that of the second reference frame. That is, the brightness of all fused reference frames is kept consistent during the generation of the first video, thereby ensuring the brightness consistency and stability of the first video in the entire video temporal domain.
[0013] In one possible implementation, the resolution of the third image frame is lower than that of the sixth image frame; in other words, the third image frame has a lower resolution, thereby reducing the power consumption of the image sensor.
[0014] In one possible implementation, the camera module determines one of the first and second image frames as a first reference frame, and fuses another image frame from the first and second image frames based on the first reference frame to generate a sixth image frame. The camera module determines one of the fourth and fifth image frames as a third reference frame, and fuses another image frame from the fourth and fifth image frames based on the third reference frame to generate an eighth image frame. The camera module determines the sixth image frame as the fourth reference frame, and fuses the third image frame based on the fourth reference frame to generate a ninth image frame. The camera module determines the eighth image frame as the fifth reference frame, and fuses the third image frame based on the fifth reference frame to generate a tenth image frame. Then, the camera module outputs a first video based on the ninth and tenth image frames.
[0015] It is understood that in this embodiment, the sixth image frame is used as the reference frame for fusing the third and sixth image frames, and the eighth image frame is used as the reference frame for fusing the third and eighth image frames. In other words, the third image frame serves as the dynamic range fill frame for the sixth and eighth image frames, thus enabling the first video to achieve HDR. Furthermore, the third image frame does not constitute an image frame in the first video, therefore, the first video can also achieve a low frame rate. Moreover, the number of third image frames required in generating the first video provided by this embodiment is smaller, thus reducing the power consumption of the image sensor.
[0016] In one possible implementation, the exposure of the third image frame is less than or equal to the exposure of the sixth and eighth image frames. That is, the third image frame is an ultra-short exposure frame, so that the sixth and eighth image frames can use the ultra-short exposure frame to supplement highlight details and fuse them to generate an ultra-high dynamic range frame, thereby enabling the first video to achieve ultra-high dynamic range imaging.
[0017] In one possible implementation, the camera module outputs a first image based on multiple sets of image frames, wherein the multiple sets of image frames include a first image frame group, the first image frame group includes a sixth image frame and a third image frame, and the sixth image frame is an image frame obtained by fusing the first image frame and the second image frame.
[0018] In one possible implementation, the camera module sequentially determines a first target image frame and a second target image frame from multiple sets of image frames. The first target image frame is the sixth image frame with the shortest interval to a first time point, and the second target image frame is the third image frame with the shortest interval to a second time point. The second time point is the generation time of the first target image frame. Based on the state of the camera module or the object being photographed, the camera module determines a reference frame from the first and second target image frames, and fuses the sixth image frame from the multiple sets of image frames based on the sixth reference frame to output a first image.
[0019] In one possible implementation, the multiple image frames further include a second image frame group adjacent to the first image frame group. The exposure parameter of the first image frame group is the exposure duration, and the exposure parameter of the second image frame group is the exposure amount. Specifically, the exposure duration of the third image frame in the first image frame group is less than or equal to the exposure duration of the first and second image frames in the first image frame group, and the exposure amount of the third image frame in the second image frame group is less than or equal to the exposure amount of the first and second image frames in the second image frame group.
[0020] It is understandable that the third image frame of one set of any two adjacent sets of image frames has a short exposure time, which is suitable for capturing scenes and makes the image frames output by the image sensor clearer. The third image frame of the other set of image frames has a small exposure, which can further improve the dynamic range of the final image. Therefore, this implementation method takes into account both the ability to improve clarity and the ability to improve dynamic range.
[0021] In one possible implementation, the second target image frame is a third image frame with the shortest interval to the second time step and an exposure duration less than or equal to the exposure duration of the sixth image frame in the same group of image frames. The camera module further determines the third target image frame from multiple groups of data frames, wherein the third target image frame is the third image frame with the shortest interval to the second time step and an exposure amount less than or equal to the exposure amount of the sixth image frame in the same group of image frames. The camera module fuses the third target image frame and the sixth image frame from the multiple groups of image frames based on the sixth reference frame to output a first image.
[0022] Understandably, when the camera module fuses multiple image frames, it also adds a third target image frame with a small exposure, which can further improve the dynamic range of the final image.
[0023] In one possible implementation, the exposure parameter of each group of image frames in the multiple groups of image frames is the exposure duration. Specifically, the exposure duration of the third image frame in each group of image frames is less than or equal to the exposure duration of the first and second image frames in each group of image frames.
[0024] It is understood that in this embodiment, the exposure time of the third image frame in each group of image frames is short. In other words, the third image frame in each group of image frames is a short exposure time frame (ST frame). The time interval between adjacent ST frames is even shorter. When the camera module or the subject being photographed by the camera module is moving, the ST frame that is closer to the user's shooting time can be selected, and the capture time is more accurate.
[0025] In one possible implementation, the camera module also generates an eleventh image frame at a fourth exposure time, wherein the fourth exposure time is different from both the first and second exposure times. The eleventh image frame is an image frame in either a first conversion gain mode or a second conversion gain mode, and the exposure of the eleventh image frame is less than or equal to the exposure of any sixth image frame in the multiple sets of image frames. The camera module fuses the sixth and eleventh image frames from the multiple sets of image frames based on the sixth reference frame to output a first image.
[0026] Understandably, when the camera module is fusing multiple image frames, it also adds an eleventh image frame with a small exposure, which can further improve the dynamic range of the final image.
[0027] In one possible implementation, the exposure parameter of each image frame in the multiple image frames is the exposure amount. Specifically, the exposure amount of the third image frame in each image frame is less than or equal to the exposure amount of the first and second image frames in each image frame.
[0028] It is understood that in this embodiment, the exposure of the third image frame in each group of image frames is small, which can further improve the dynamic range of the final image. Therefore, the dynamic range of the first image in this embodiment is wider.
[0029] In one possible implementation, the camera module fuses the second target image frame and the sixth image frame from multiple sets of image frames based on the sixth reference frame to output the first image.
[0030] In one possible implementation, the camera module outputs a first image based on multiple sets of image frames, wherein the multiple sets of image frames include a first image frame group, the first image frame group includes a sixth image frame, a third image frame and an eighth image frame, the sixth image frame is an image frame fused from the first image frame and the second image frame, and the eighth image frame is an image frame fused from the fourth image frame and the fifth image frame.
[0031] In one possible implementation, the camera module sequentially determines a first target image frame and a second target image frame from multiple sets of image frames. The first target image frame is either the sixth or eighth image frame with the shortest interval to a first time point, and the second target image frame is the third image frame with the shortest interval to a second time point. The second time point is the generation time of the first target image frame. Based on the state of the camera module or the object being photographed, the camera module determines a sixth reference frame from the first and second target image frames. Based on this sixth reference frame, the sixth and eighth image frames from the multiple sets of image frames are fused to output a first image.
[0032] In one possible implementation, when the camera module or the object being photographed by the camera module is in motion, the exposure parameter of each image frame in the multiple image frames is the exposure duration. Specifically, the exposure duration of the third image frame in each image frame is less than or equal to the exposure duration of the first, second, fourth, and fifth image frames in each image frame.
[0033] It is understood that this embodiment prioritizes capturing images when the camera module or the object being photographed by the camera module is moving. Furthermore, the number of third image frames required in the process of generating the first image provided by this embodiment is smaller. Therefore, not only can the first image be clearer, but the power consumption of the image sensor can also be reduced.
[0034] In one possible implementation, the camera module also generates an eleventh image frame at a fourth exposure time, wherein the fourth exposure time is different from the first, second, and third exposure times. The eleventh image frame is an image frame in either a first conversion gain mode or a second conversion gain mode, and the exposure of the eleventh image frame is less than or equal to the exposure of any sixth image frame and any eighth image frame in the multiple sets of image frames. The camera module fuses the sixth, eighth, and tenth image frames from the multiple sets of image frames based on the sixth reference frame to output a first image.
[0035] Understandably, when the camera module is fusing multiple image frames, it also adds an eleventh image frame with a small exposure, which can further improve the dynamic range of the final image.
[0036] In one possible implementation, when the camera module or the object being photographed by the camera module is stationary, the exposure parameter of each image frame in the multiple image frames is the exposure amount. Specifically, the exposure amount of the third image frame in each image frame is less than or equal to the exposure amounts of the first, second, fourth, and fifth image frames in each image frame.
[0037] It is understood that this embodiment prioritizes capturing dynamic range when the camera module or the subject being photographed by the camera module is stationary. Furthermore, this embodiment requires fewer third image frames to generate the first image, thus not only improving the dynamic range of the captured image but also reducing the power consumption of the image sensor.
[0038] In one possible implementation, the camera module fuses the second target image frame, the sixth image frame and the eighth image frame from multiple sets of image frames based on the sixth reference frame, and outputs the first image.
[0039] In one possible implementation, when the camera module or the object being photographed by the camera module is in a stationary state, the camera module determines the first target image frame as the sixth reference frame; or, when the camera module or the object being photographed by the camera module is in a moving state, the camera module determines the second target image frame as the sixth reference frame.
[0040] In one possible implementation, the camera module includes an image sensor comprising a plurality of pixels arranged in an array. Each pixel includes a photodiode, a charge transfer gate, a floating diffusion node, and a conversion gain mode control circuit. The conversion gain mode control circuit includes at least one capacitor, which includes a first capacitor. The charge transfer gate is used to turn on or off the connection between the cathode of the photodiode and the floating diffusion node. The conversion gain mode control circuit, when in an on or off state, is used to turn on the connection between the floating diffusion node and the first capacitor, or to disconnect the connection between the floating diffusion node and the at least one capacitor. A first conversion gain mode is the conversion gain mode of the image sensor when the conversion gain mode control circuit is off; a second conversion gain mode is the conversion gain mode of the image sensor when the conversion gain mode control circuit is on.
[0041] In one possible implementation, pixel PX is a dual conversion gain (DCG) pixel. At least one capacitor is included (1 in total). The image sensor also includes a readout line. The pixel further includes a reset transistor and a row selector. The conversion gain mode control circuit includes a first switch transistor, wherein the reset transistor and the first switch transistor are connected in series between the floating diffusion node and the power supply, and the connection between the reset transistor and the first switch transistor is connected to a reference ground via a first capacitor. The first end of the row selector is connected to the floating diffusion node, and the second end of the row selector is connected to the readout line.
[0042] The image frame in the first conversion gain mode is determined by a first reset voltage and a first signal voltage. The first reset voltage is the second terminal voltage of the horizontal selector when the horizontal selector is turned on, after both the reset transistor and the first switching transistor are turned off and before the charge transfer gate is turned on. The first signal voltage is the second terminal voltage of the horizontal selector when the horizontal selector is turned on, after the charge transfer gate is turned off.
[0043] The image frame in the second conversion gain mode is determined by the second reset voltage and the second signal voltage. The second reset voltage is the second terminal voltage of the horizontal selector when both the first switch and the horizontal selector are turned on after the reset transistor is turned off and before the charge transfer gate is turned on. The second signal voltage is the second terminal voltage of the horizontal selector when both the first switch and the horizontal selector are turned on after the charge transfer gate is turned off.
[0044] In one possible implementation, the pixel is a TCG pixel. At least one capacitor also includes a second capacitor, the image sensor also includes a readout line, the pixel also includes a reset transistor and a row selector, and the conversion gain mode control circuit also includes a first switch and a second switch, wherein the reset transistor and the first switch are connected in series between the floating diffusion node and the power supply, and the connection between the reset transistor and the first switch is connected to a reference ground through the first capacitor; the second switch and the second capacitor are connected in series between the floating diffusion node and the reference ground; the first end of the row selector is connected to the floating diffusion node, and the second end of the row selector is connected to the readout line.
[0045] The image frame in the first conversion gain mode is determined by a first reset voltage and a first signal voltage. The first reset voltage is the second terminal voltage of the horizontal selector when the horizontal selector is turned on, after both the first and second switches are turned off and before the charge transfer gate is turned on. The first signal voltage is the second terminal voltage of the horizontal selector when the horizontal selector is turned on, after the charge transfer gate is turned off and before the first switch is turned on.
[0046] The image frame in the second conversion gain mode is determined by the second reset voltage and the second signal voltage. The second signal voltage is the second terminal voltage of the horizontal selector when the charge transfer gate is turned off and the reset transistor is turned on, the second switch is turned off and the first switch and the horizontal selector are both turned on. The second reset voltage is the second terminal voltage of the horizontal selector when the reset transistor is turned off, the second switch is turned off and the first switch and the horizontal selector are both turned on. Attached Figure Description
[0047] Figure 1 This is a schematic diagram of the structure of a TCG pixel provided by existing technology;
[0048] Figure 2 This is a schematic diagram illustrating an application scenario of the image processing method provided in this application;
[0049] Figure 3 This is a flowchart illustrating the image processing method provided in this application;
[0050] Figure 4 This is a structural schematic diagram of the camera module provided in this application;
[0051] Figure 5 This is another structural schematic diagram of the camera module provided in this application;
[0052] Figure 6 This is a schematic diagram of the structure of a pixel provided in this application;
[0053] Figure 7 This is another schematic diagram of the pixel structure provided in this application;
[0054] Figure 8This is a control timing diagram of a pixel provided in this application;
[0055] Figure 9 This is another control timing diagram of the pixel provided in this application;
[0056] Figure 10 This is yet another structural schematic diagram of the pixel provided in this application;
[0057] Figure 11 This is another control timing diagram of the pixel provided in this application;
[0058] Figure 12 This is yet another control timing diagram for the pixels provided in this application;
[0059] Figure 13 This is a schematic diagram of the first video generation process provided in this application;
[0060] Figure 14 This is another schematic diagram of the first video generation process provided in this application;
[0061] Figure 15 This is another schematic diagram of the first video generation process provided in this application;
[0062] Figure 16 This is yet another schematic diagram of the first video generation process provided in this application;
[0063] Figure 17 This is a schematic diagram of the first image generation process provided in this application;
[0064] Figure 18 This is another schematic diagram of the first image generation process provided in this application;
[0065] Figure 19 This is another schematic diagram of the first image generation process provided in this application;
[0066] Figure 20 This is another schematic diagram of the first image generation process provided in this application;
[0067] Figure 21 This is yet another schematic diagram of the first image generation process provided in this application. Detailed Implementation
[0068] The image processing method provided in this application can be applied to electronic devices including camera modules. These electronic devices can be mobile phones, tablet personal computers, laptop computers, personal digital assistants (PDAs), cameras, personal computers, laptops, in-vehicle devices, wearable devices, augmented reality (AR) glasses, AR headsets, virtual reality (VR) glasses or VR headsets, or other devices with photo and video recording functions. The following explanation uses the application of the image processing method to a mobile phone as an example.
[0069] See Figure 2 , Figure 2 This is a schematic diagram illustrating an application scenario of the image processing method provided in this application. The image processing method provided in this application is applicable to... Figure 2 The mobile phone 1 shown includes a camera module 11, a display screen 12, and a housing 13, with the display screen 12 connected to the housing 13. The camera module 11 includes a lens 111, an image sensor, and an ISP. The lens 111 can be a front-facing camera or a rear-facing camera. When the camera module 11 begins exposure, light from the outside or reflected light from an object passes through the lens 111 and illuminates the image sensor. The image sensor generates a first image frame and a second image frame during the first exposure time in two adjacent exposure times, and generates a third image frame during the second exposure time in two adjacent exposure times. The first image frame is an image frame under a first conversion gain mode, the second image frame is an image frame under a second conversion gain mode, and the third image frame is an image frame under either the first or second conversion gain mode. Furthermore, the exposure parameters of the third image frame are less than or equal to the exposure parameters of the first and second image frames, and the exposure parameters include exposure amount or exposure duration. Subsequently, the ISP outputs a first image or a first video based on the first, second, and third image frames and displays it on the display screen 12.
[0070] It is understood that the image sensor in this application generates a first image frame, a second image frame, and a third image frame in two adjacent exposure times. Compared to the scheme using an image sensor with a TCG pixel structure that generates six image frames in two adjacent exposure times, the image sensor in this application generates half the number of image frames in two adjacent exposure times, thereby reducing the number of image frames that the ISP needs to process in the subsequent stages by half, and thus reducing the power consumption of the image sensor and the ISP. Furthermore, when the first conversion gain mode is HCG mode and the second conversion gain mode is LCG mode, the first image frame is an HCG frame, the second image frame is an LCG frame, and the third image frame is an HCG frame or LCG frame with exposure parameters smaller than both the first and second image frames. The third image frame is used to supplement the extremely bright light details missing in the HCG and LCG frames. In other words, the third image frame is equivalent to the role of a LOFIC frame in image imaging. Therefore, the image processing method provided in this application can also achieve HDR without degrading image quality. In summary, the image processing method provided in this application can reduce the power consumption of the image sensor and the ISP without reducing image quality.
[0071] The following is combined Figures 3 to 21 The image processing method provided in this application is illustrated with examples. This image processing method is applicable to electronic devices; the following description uses a camera module in an electronic device as the execution subject of the image processing method.
[0072] See Figure 3 , Figure 3 This is a flowchart illustrating the image processing method provided in this application. Figure 3 As shown, the image processing method may include the following steps:
[0073] S101, a first image frame and a second image frame are generated at the first exposure time in two adjacent exposure times, and a third image frame is generated at the second exposure time in two adjacent exposure times.
[0074] Wherein, the first image frame is an image frame in the first conversion gain mode, the second image frame is an image frame in the second conversion gain mode, the third image frame is an image frame in either the first or second conversion gain mode, and the exposure parameters of the third image frame are less than or equal to the exposure parameters of the first and second image frames, the exposure parameters including exposure amount or exposure duration.
[0075] Here, the structure of the camera module is as follows: Figure 4 As shown, it includes a lens, an image sensor, and an ISP. Specifically, the camera module generates a first image frame and a second image frame at the first exposure time using the image sensor, and generates a third image frame at the second exposure time using the image sensor.
[0076] Optionally, the camera module also generates a fourth and fifth image frame at the third exposure time using the image sensor. The third exposure time is the next exposure time between two adjacent exposure times. The fourth image frame is an image frame in the first conversion gain mode, and the fifth image frame is an image frame in the second conversion gain mode. The exposure parameters of both the fourth and fifth image frames are greater than or equal to the exposure parameters of the third image frame. In short, the fourth image frame and the first image frame are of the same type, and the fifth image frame and the second image frame are of the same type.
[0077] The following is combined Figures 5 to 12 The structure of the image sensor or pixel shown is illustrated, with the first conversion gain mode being HCG mode and the second conversion gain mode being LCG mode as an example, to illustrate the generation process of the first image frame, the second image frame and the third image frame.
[0078] The structure of the image sensor in this application is as follows: Figure 5 As shown, it includes multiple pixels (PX) arranged in an array, multiple readout lines (VSL), and an analog-to-digital converter circuit. These multiple pixels (PX) can be arranged in multiple rows and columns, and a column of pixels (PX) can be connected to one or more readout lines (VSL). Each pixel (PX) is as follows... Figure 6As shown, the sensor includes a photodiode (PD), a charge transport gate (TG), a floating diffusion node (FD), and a conversion gain mode control circuit 2. The conversion gain mode control circuit 2 includes at least one capacitor, which includes a first capacitor Cs. The charge transport gate (TG) is used to turn on or off the connection between the cathode of the photodiode (PD) and the FD point. When the conversion gain mode control circuit 2 is in an on or off state, it turns on the connection between the FD point and the first capacitor Cs, or disconnects the connection between the FD point and at least one capacitor. Correspondingly, the first conversion gain mode is the image sensor's conversion gain mode (HCG mode) when the conversion gain mode control circuit 2 is off; the second conversion gain mode is the image sensor's conversion gain mode (LCG mode) when the conversion gain mode control circuit 2 is on. It should be noted that for image sensors, when the incident light intensity is low, the number of photoelectrons generated in the photodiode (PD) is small, and the HCG mode is used. In HCG mode, although the capacitance at the FD point is only the parasitic capacitance Cpar of the FD point, and the capacitance value of Cpar is small, the FD point can completely handle the number of photoelectrons generated by the photodiode (PD). When the incident light intensity is high, the number of photoelectrons generated in the photodiode (PD) is large, and the parasitic capacitance Cpar of the FD point itself cannot completely handle the photoelectrons generated by the photodiode (PD), resulting in the voltage change at the FD point not accurately reflecting the number of photoelectrons generated by the photodiode (PD). Therefore, the LCG mode is used in this case. In LCG mode, the capacitance at the FD point includes not only the parasitic capacitance Cpar but also a first capacitance Cs, which has a large capacitance value, and the FD point can completely handle the number of photoelectrons generated by the photodiode (PD). It is understandable that conversion gain is defined as the voltage change at the FD point caused by each electron, usually expressed in μV / e-, and it is inversely proportional to the capacitance at the FD point. That is, the smaller the capacitance at the FD point, the higher the conversion gain; the larger the capacitance at the FD point, the lower the conversion gain.
[0079] For example, pixel PX is a DCG pixel; please refer to [link / reference]. Figure 7 .like Figure 7As shown, the pixel PX includes a photodiode PD, a charge transfer gate TG, a FD point, and a conversion gain mode control circuit 2. The conversion gain mode control circuit 2 includes a first capacitor Cs and a first switch SG1. The pixel PX also includes a reset transistor RST, a row selector SEL, and a source follower SF. The charge transfer gate TG, reset transistor RST, source follower SF, row selector RS, and first switch SG1 can be any of a MOSFET, IGBT, or GaN transistor. Here, we will use a MOSFET as an example for all switches. The cathode of the photodiode PD is connected to the FD point through the charge transfer gate TG. The reset transistor RST and the first switch SG1 are connected in series between the power supply VDD and the FD point. The first capacitor Cs is connected between the connection point of the reset transistor RST and the first switch SG1 and the FD point. The gate of the source follower SF is connected to the FD point, the drain of the source follower SF is connected to the power supply VDD, the source of the source follower SF is connected to the drain of the row selector SEL, and the source of the row selector SEL is connected to the readout line VSL. Correspondingly, when the first switch SG1 is turned off, the image sensor is in HCG mode; when the first switch SG1 is turned on, the image sensor is in LCG mode.
[0080] Here, in HCG mode, the image frame is determined by a first reset voltage and a first signal voltage. The first reset voltage is the source voltage of the horizontal selector SEL when it is turned on, after both the reset transistor RST and the first switch SG1 are turned off and before the charge transfer gate TG is turned on. The first signal voltage is the source voltage of the horizontal selector SEL when it is turned on, after the charge transfer gate TG is turned off. In LCG mode, the image frame is determined by a second reset voltage and a second signal voltage. The second reset voltage is the source voltage of the horizontal selector SEL when both the first switch SG1 and the horizontal selector SEL are turned on, after the reset transistor RST is turned off and before the charge transfer gate TG is turned on. The second signal voltage is the source voltage of the horizontal selector SEL when both the first switch SG1 and the horizontal selector SEL are turned on, after the charge transfer gate TG is turned off.
[0081] In an optional embodiment, the first image frame and the third image frame are both image frames in HCG mode, and the second image frame is an image frame in LCG mode.
[0082] Specifically, pixel PX can be based on Figure 8 The control timing diagram shown outputs a first reset voltage, a first signal voltage, a second reset voltage, and a second signal voltage. (As follows...) Figure 8As shown, during the time intervals t1 to t2, the reset transistor RST, the first switch SG1, and the charge transfer gate TG are all in the on state, and pixel PX enters the reset phase. Starting from time t2, the charge transfer gate TG, the first switch SG1, and the reset transistor RST are sequentially turned off, pixel PX ends its reset and enters the first exposure time, Exposure1. At time t3, the row selector SEL is turned on, pixel PX enters the readout phase, and then the first switch SG1 and the reset transistor RST are turned on. During the time intervals t4 to t5, both the first switch SG1 and the reset transistor RST are turned on, and the voltage at point FD is reset to the voltage of the power supply VDD. At time t5, the reset transistor RST is turned off, and the voltage reset at point FD ends. During the time intervals t5 to t6, the first switch SG1 is turned on, the image sensor is in LCG mode, and the analog-to-digital converter circuit obtains the source voltage of the selector SEL to obtain the second reset voltage V. LR1 During the time interval t6 to t7, the first switch SG1 is turned off, the image sensor is in HCG mode, and the analog-to-digital converter circuit obtains the source voltage of the selector SEL to obtain the first reset voltage V. HR1 During the time interval t7 to t8, the charge transport gate TG is turned on, and the photogenerated charge of the photodiode PD is transferred to the parasitic capacitance C at point FD. par Transfer. During the period from t8 to t9, both the charge transfer gate TG and the first switching transistor SG1 are turned off, and the image sensor remains in HCG mode. The analog-to-digital converter circuit obtains the source voltage of the selector SEL to obtain the first signal voltage V. HS1 Starting from time t9, the first switch SG1 and the charge transfer gate TG are turned on sequentially. After the charge transfer gate TG is turned on, the photogenerated charge of the photodiode PD is transferred to the parasitic capacitance C at point FD. par The first capacitor Cs is transferred. During the time interval t10 to t11, the charge transfer gate TG is turned off, the first switch SG1 is turned on, the image sensor is in LCG mode, and the analog-to-digital converter circuit obtains the source voltage of the selector SEL to obtain the second signal voltage V. LS1 Starting from time t11, the first switch SG1 and the row selector SEL are turned off sequentially, ending the readout phase for the first exposure time Exposure1.
[0083] During the time interval t12 to t13, the reset transistor RST, the first switch SG1, and the charge transfer gate TG are all in the on state, and pixel PX re-enters the reset stage. Starting from t13, the charge transfer gate TG, the first switch SG1, and the reset transistor RST are sequentially turned off, and pixel PX ends the reset and enters the second exposure time, Exposure2. During the time interval t14 to t15, the first switch SG1 and the reset transistor RST are both turned on, and the voltage at point FD is reset to the power supply VDD. Starting from t15, the reset transistor RST and the first switch SG1 are sequentially turned off. Then, at t16, the horizontal selector SEL is turned on, and pixel PX enters the readout stage. During the time interval t16 to t17, the first switch SG1 is turned off, the image sensor is in HCG mode, and the analog-to-digital converter circuit obtains the source voltage of the selector SEL to obtain the first reset voltage V. HR2 During the period from t17 to t18, the charge transport gate TG is turned on, and the photogenerated charge of the photodiode PD is transferred to the parasitic capacitance C at point FD. par Transfer. During the period from t18 to t19, both the charge transfer gate TG and the first switching transistor SG1 are turned off, and the image sensor remains in HCG mode. The analog-to-digital converter circuit obtains the source voltage of the selector SEL to obtain the first signal voltage V. HS2 At time t19, the row selector SEL is turned off, ending the readout phase for the second exposure time, Exposure2.
[0084] Then, the analog-to-digital converter circuit applies the first reset voltage V. HR1 First signal voltage V HS1 Second reset voltage V LR1 Second signal voltage V LS1 Perform analog-to-digital conversion to obtain the first reset voltage V. HR1 Digital quantization value D HR1 First signal voltage V HS1 Digital quantization value D HS1 Second reset voltage V LR1 Digital quantization value D LR1 Second signal voltage V LS1 Digital quantization value D LS1 The analog-to-digital converter circuit also operates on the first reset voltage V. HR2 and the first signal voltage V HS2 Perform analog-to-digital conversion to obtain the first reset voltage V. HR2 Digital quantization value D HR2 and the first signal voltage V HS2 Digital quantization value D HS2The analog-to-digital converter circuit obtains the first image frame (HCG frame) generated by the image sensor at the first exposure time Exposure1 by subtracting the correlated double sampling (CDS). HS1 -D HR1 The second image frame (LCG frame) generated by Exposure1 at the first exposure time is D. LS1 -D LR1 And, the third image frame (short exposure parameter VS frame) generated at the second exposure time Exposure2 is D HS2 -D HR2 .
[0085] In another alternative embodiment, the first image frame is an image frame in HCG mode, and the second and third image frames are both image frames in LCG mode.
[0086] Specifically, pixel PX can be based on Figure 9 The control timing diagram shown generates the second reset voltage V for the second exposure time Exposure2. LR2 Second signal voltage V LS2 Among them, the second reset voltage V LR2 For analog-to-digital converter circuits in Figure 9 The source voltage of the selector SEL is obtained during the time period t15 to t16 shown; the second signal voltage V LS2 For analog-to-digital converter circuits in Figure 9 During the time period t17 to t18 shown, the source voltage of the selector SEL is obtained. The analog-to-digital converter circuit uses the second reset voltage V. LR2 Second signal voltage V LS2 Perform analog-to-digital conversion to obtain the second reset voltage V. LR2 Digital quantization value D LR2 Second signal voltage V LS2 Digital quantization value D LS2 The analog-to-digital converter circuit uses CDS to calculate the difference, obtaining the third image frame (D) generated by the image sensor at the second exposure time, Exposure2. LS2 -D LR2 For details on the specific implementation of the image sensor generating the first and second image frames for the first exposure time period Exposure1, please refer to [link to relevant documentation]. Figure 8 The descriptions of the corresponding parts in the illustrated embodiments will not be repeated here.
[0087] For example, pixel PX is a TCG pixel; please refer to [link / reference]. Figure 10 .like Figure 10 As shown, with Figure 7Compared to the DCG pixels shown, the only difference between the two is that... Figure 10 The conversion gain mode control circuit shown is 2 times. Figure 7 The conversion gain mode control circuit 2 shown has an additional second capacitor C. LOFIC And the second switch SG2. Wherein, the second capacitor C LOFIC The second switch SG2 is connected in series between the FD point and the reference ground. Correspondingly, when both the first switch SG1 and the second switch SG2 are turned off, the image sensor is in HCG mode; when the first switch SG1 is turned on and the second switch SG2 is turned off, the image sensor is in LCG mode.
[0088] Here, the image frame in the first conversion gain mode is determined by a first reset voltage and a first signal voltage. The first reset voltage is the source voltage of the horizontal selector SEL when it is on, after both the first switch SG1 and the second switch SG2 are off and before the charge transfer gate TG is on. The first signal voltage is the source voltage of the horizontal selector SEL when it is on, after the charge transfer gate TG is off and before the first switch SG1 is on. The image frame in the second conversion gain mode is determined by a second reset voltage and a second signal voltage. The second signal voltage is the source voltage of the horizontal selector SEL when it is off, before the reset transistor RST is on, after the charge transfer gate TG is off, before the reset transistor RST is on, after the second switch SG2 is off, and after both the first switch SG1 and the horizontal selector SEL are on. The second reset voltage is the source voltage of the horizontal selector SEL when both the first switch SG1 and the horizontal selector SEL are on, after the reset transistor RST is off.
[0089] In an optional embodiment, the first image frame and the third image frame are both image frames in HCG mode, and the second image frame is an image frame in LCG mode.
[0090] Specifically, pixel PX can be based on Figure 11 The control timing diagram shown outputs a first reset voltage, a first signal voltage, a second reset voltage, and a second signal voltage. (As follows...) Figure 11As shown, during the time intervals t1 to t2, the reset transistor RST, the first switch SG1, the second switch SG2, and the charge transfer gate TG are all in the on state, and pixel PX enters the reset phase. Starting from time t2, the charge transfer gate TG and the reset transistor RST are sequentially turned off, pixel PX ends the reset process, and enters the first exposure time, Exposure1. At time t3, the row selector SEL is turned on, pixel PX enters the readout phase, and then the first switch SG1 and the second switch SG2 are turned off. During the time intervals t4 to t5, both the first switch SG1 and the second switch SG2 are turned off, the image sensor is in HCG mode, and the analog-to-digital converter circuit obtains the source voltage of the selector SEL to obtain the first reset voltage V. HR1 During the time interval t5 to t6, the charge transport gate TG is turned on, and the photogenerated charge of the photodiode PD is transferred to the parasitic capacitance C at point FD. par Transfer. During the period from t6 to t7, the charge transport gate TG, the first switch SG1, and the second switch SG2 are all turned off. The image sensor remains in HCG mode. The analog-to-digital converter circuit obtains the source voltage of the selector SEL to obtain the first signal voltage V. HS1 During the time interval t7 to t8, the first switch SG1 is turned on and the second switch SG2 is turned off. The image sensor is in LCG mode, and the analog-to-digital converter circuit obtains the source voltage of the selector SEL to obtain the second signal voltage V. LS1 During the period from t8 to t9, both the reset transistor RST and the first switching transistor SG1 are turned on, and the voltage at the connection point of the reset transistor RST and the first switching transistor SG1, as well as the voltage at the FD point, are reset to the voltage of the power supply VDD. At time t9, the reset transistor RST is turned off, and the reset of the voltage at the connection point of the reset transistor RST and the first switching transistor SG1, as well as the voltage at the FD point, is complete. During the period from t9 to t10, the first switching transistor SG1 is turned on and the second switching transistor SG2 is turned off. The image sensor is still in LCG mode, and the analog-to-digital converter circuit obtains the source voltage of the selector SEL to obtain the second reset voltage V. LR1 Starting from time t10, the first switch SG1 and the row selector SEL are turned off sequentially, ending the readout phase for the first exposure time Exposure1.
[0091] During the period from t11 to t12, the reset transistor RST, the first switch SG1, the second switch SG2, and the charge transfer gate TG are all in the on state, and pixel PX re-enters the reset stage. Starting from t12, the charge transfer gate TG and the reset transistor RST are sequentially turned off, pixel PX ends its reset, and enters the second exposure time, Exposure2. At t13, the row selector SEL is turned on, and pixel PX re-enters the readout stage. Afterwards, the first switch SG1 and the second switch SG2 are turned off. During the period from t14 to t15, both the first switch SG1 and the second switch SG2 are turned off, the image sensor is in HCG mode, and the analog-to-digital converter circuit obtains the source voltage of the selector SEL to obtain the first reset voltage V. HR2 During the period from t15 to t16, the charge transport gate TG is turned on, and the photogenerated charge of the photodiode PD is transferred to the parasitic capacitance C at point FD. par Transfer. During the period from t16 to t17, the charge transfer gate TG, the first switch SG1, and the second switch SG2 are all turned off. The image sensor is still in HCG mode. The analog-to-digital converter circuit obtains the source voltage of the selector SEL to obtain the first signal voltage V. HS2 At time t17, the row selector SEL is turned off, ending the readout phase for the second exposure time, Exposure2.
[0092] Subsequently, the analog-to-digital conversion circuit is based on the first reset voltage V HR1 First signal voltage V HS1 Second reset voltage V LR1 Second signal voltage V LS1 Generate a first image frame with a first exposure time of Exposure1, and based on the first reset voltage V. HR2 and the first signal voltage V HS2 Generate a second image frame with a second exposure time of Exposure2. For a detailed description of this part, please refer to [link to relevant documentation]. Figure 8 The descriptions of the corresponding parts in the illustrated embodiments will not be repeated here.
[0093] In another alternative embodiment, the first image frame is an image frame in HCG mode, and the second and third image frames are both image frames in LCG mode.
[0094] Specifically, pixel PX can be based on Figure 12 The control timing diagram shown outputs the second reset voltage V for the second exposure time Exposure2. LR2 Second signal voltage V LS2 Among them, the second signal voltage V LS2 For analog-to-digital converter circuits in Figure 12 The source voltage of the selector SEL is obtained during the time period t14 to t15 shown; the second reset voltage VLR2 For analog-to-digital converter circuits in Figure 12 During the time period t16 to t17 shown, the source voltage of the selector SEL is obtained. The analog-to-digital converter circuit uses the second reset voltage V. LR2 Second signal voltage V LS2 Perform analog-to-digital conversion to obtain the second reset voltage V. LR2 Digital quantization value D LR2 Second signal voltage V LS2 Digital quantization value D LS2 The analog-to-digital converter circuit uses CDS to calculate the difference, obtaining the third image frame (D) generated by the image sensor at the second exposure time, Exposure2. LS2 -D LR2 For details on the specific implementation of the image sensor generating the first and second image frames at the first exposure time (Exposure1), please refer to [link to relevant documentation]. Figure 11 The descriptions of the corresponding parts in the illustrated embodiments will not be repeated here.
[0095] S102, output the first image or the first video based on the first image frame, the second image frame and the third image frame.
[0096] The image processing method provided in this application is applicable to two scenarios: the first is a photography scenario, and the second is a video scenario. Step S102 will be described below in the order of the first and second scenarios.
[0097] Scenario 1:
[0098] In one embodiment, the camera module outputs a first video based on a first image frame, a second image frame, and a third image frame via an ISP.
[0099] Specifically, the camera module uses the ISP to determine one of the first and second image frames as the first reference frame. Based on the first reference frame, it fuses another image frame from the first and second image frames to generate the sixth image frame. The camera module uses the ISP to determine the third image frame as the second reference frame, and uses the second reference frame to fuse the sixth image frame to generate the seventh image frame. Then, the camera module uses the ISP to insert the seventh image frame into the sixth image frame and outputs the first video, where the seventh image frame is the next image frame after the sixth image frame. For the specific implementation of fusing multiple image frames based on reference frames, please refer to the corresponding description in the second scenario section below; it will not be elaborated here. Furthermore, the camera module can also perform fusion processing on the first and second image frames using the image sensor; this application does not limit this. It should be noted that the exposure parameters of the first and second image frames are the same as the exposure parameters of the sixth image frame.
[0100] To facilitate understanding, the following will be combined with... Figure 13 Using VS frames (representing the third image frame), DCG frames (representing the sixth image frame), and VS' frames (representing the seventh image frame) as examples, the generation process of the first video is illustrated. Figure 13 As shown, the image sensor generates two image frame streams: one is a DCG frame stream, composed of all DCG frames output by the image sensor, including DCG1 frame from the first image frame group and DCG2 frame from the second image frame group, with a frame rate of 30fps; the other is a VS frame stream, composed of all VS frames output by the image sensor, including VS1 frame from the first image frame group and VS2 frame from the second image frame group, with a frame rate of 30fps. Here, DCG1, VS1, DCG2, and VS2 frames are four frames generated sequentially by the camera module through the image sensor at four adjacent exposure times.
[0101] Specifically, the camera module uses the ISP to determine the VS1 frame as the second reference frame, and then fuses the DCG1 frame with this second reference frame to generate the VS1' frame. The camera module also uses the ISP to determine the VS2 frame as the second reference frame, and then fuses the DCG2 frame with this second reference frame to generate the VS2' frame. Next, the camera module uses the ISP to insert the VS1' frame as the next image frame after the DCG1 frame, and the VS2' frame as the next image frame after the DCG2 frame, controlling the time interval between any two adjacent frames in the DCG1, VS1', DCG2, and VS2' frames to be 16.6ms, thus obtaining an initial video with a frame rate of 60fps. Afterwards, the camera module uses the ISP to perform image format conversion (e.g., converting .raw format to .YUV format), color correction, tone mapping, sharpening, and other processing on the initial video before outputting the first video.
[0102] Furthermore, the reference frame for fusing the first image frame (HCG frame) and the second image frame (LCG frame) can be either an HCG frame or an LCG frame. In practical applications, to ensure brightness consistency and stability across the entire video temporal domain, the brightness of all fused reference frames needs to be consistent. That is, the brightness of the reference frame when fusing HCG and LCG frames is consistent with the brightness of the reference frame when fusing DCG and VS frames. Because HCG frames have a better signal-to-noise ratio in dark areas in bright scenes and longer exposure times in low-light environments, VS frames can achieve less motion blur and are more advantageous for motion ghosting. Therefore, in bright scenes, when fusing HCG and LCG frames, the HCG frame can be selected as the reference frame, and the exposure of the corresponding VS frame is also consistent with the exposure of the HCG frame. In medium-low brightness scenes, when fusing HCG and LCG frames, the LCG frame can be selected as the reference frame, and the exposure of the corresponding VS frame is consistent with the exposure of the LCG frame.
[0103] It is understandable that in this implementation, the third image frame is used as the reference frame when fusing the third and sixth image frames. The preceding sixth image frame in the temporal domain is used to supplement the corresponding dynamic range and noise reduction processing, thus enabling the first video to achieve HDR. Since the seventh image frame is inserted as the next image frame after the sixth image frame, the first video can also achieve a high frame rate.
[0104] Furthermore, based on this embodiment, the resolution of the third image frame is lower than that of the sixth image frame. In other words, the third image frame has a lower resolution to reduce the power consumption of the image sensor.
[0105] To facilitate understanding, the following will be combined with... Figure 14 Examples are provided for multiple sets of image frames. For example... Figure 14 As shown, with Figure 13 compared to, Figure 14 Each VS frame in the VS frame stream has a small resolution. Specifically, the resolution of DCG1 frame is greater than that of VS1 frame, and the resolution of DCG2 frame is greater than that of VS2 frame.
[0106] In another embodiment, the camera module outputs a first video based on a first image frame, a second image frame, a third image frame, a fourth image frame, and a fifth image frame via an ISP.
[0107] Specifically, the camera module uses the ISP to determine one of the first and second image frames as the first reference frame. Based on the first reference frame, it fuses another image frame from the first and second image frames to generate the sixth image frame. The camera module also uses the ISP to determine one of the fourth and fifth image frames as the third reference frame. Based on the third reference frame, it fuses another image frame from the fourth and fifth image frames to generate the eighth image frame. The camera module uses the ISP to determine the sixth image frame as the fourth reference frame. Based on the fourth reference frame, it fuses the third image frame to generate the ninth image frame. The camera module also uses the ISP to determine the eighth image frame as the fifth reference frame. Based on the fifth reference frame, it fuses the third image frame to generate the tenth image frame. Finally, the camera module uses the ISP to output the first video based on the ninth and tenth image frames.
[0108] To facilitate understanding, the following will be combined with... Figure 15 The generation process of the first video is illustrated using an example, with VS frames representing the third image frame, DCG frames representing the sixth and eighth image frames, and DCG' frames representing the ninth and tenth image frames. Figure 15As shown, the image sensor outputs two image frame streams. One is a DCG frame stream, which consists of all DCG frames output by the image sensor, including the sixth image frame DCG1 and the eighth image frame DCG2. The frame rate of the DCG frame stream is 30fps. The other is a VS frame stream, which consists of all VS frames output by the image sensor, including the third image frame VS1.
[0109] Specifically, the camera module uses the ISP to determine the sixth image frame (DCG1) as the fourth reference frame, and then fuses it with the third image frame (VS1) to generate the ninth image frame (DCG1'). The camera module uses the ISP to determine the eighth image frame (DCG2) as the fifth reference frame, and then fuses it with the third image frame (VS1) to generate the tenth image frame (DCG2'). Next, the camera module uses the ISP to sequentially output the ninth image frame (DCG1') and the tenth image frame (DCG2') at a frame rate of 30fps for the DCG frame stream, resulting in an initial video at 30fps. Afterwards, the ISP performs image format conversion, color correction, tone mapping, and sharpening on the initial video before outputting the first video.
[0110] It is understood that in this embodiment, the DCG frame is used as the reference frame when fusing the VS frame and the DCG frame. In other words, the VS frame serves as the dynamic range fill frame for the DCG frame, thus enabling the first video to achieve HDR. Furthermore, the VS frame does not constitute an image frame in the first video, therefore, the first video can also achieve a low frame rate. In addition, this embodiment requires fewer VS frames to generate the first video, thus reducing the power consumption of the image sensor.
[0111] Furthermore, based on this embodiment, the exposure of the third image frame is less than or equal to the exposure of the sixth and eighth image frames. That is, the third image frame is an ultra-short exposure frame, which allows the sixth and eighth image frames to supplement highlight details using the ultra-short exposure frame and fuse them to generate an ultra-high dynamic range frame, thereby enabling the first video to achieve ultra-high dynamic range imaging.
[0112] To facilitate understanding, the following will be combined with... Figure 16 The generation process of the first video is illustrated with an example. Figure 16 As shown, with Figure 15 compared to, Figure 16 Each VS frame in the VS frame stream is an ultra-short exposure frame. Specifically, the exposure of the third image frame VS1 is less than or equal to the exposure of any one of the sixth image frame DCG1 and the eighth image frame DCG2.
[0113] The second scenario:
[0114] In one embodiment, the camera module outputs a first image based on multiple sets of image frames via an ISP. The multiple sets of image frames include a first image frame group, which includes a sixth image frame and a third image frame. The sixth image frame is an image frame obtained by fusing the first image frame and the second image frame.
[0115] The camera module uses an ISP to sequentially determine a first target image frame and a second target image frame from multiple sets of image frames. The first target image frame is the sixth image frame with the shortest interval to the first time point, and the second target image frame is the third image frame with the shortest interval to the second time point. The first time point is the user's shooting time, which can be a preset shooting time or the moment the user clicks the shutter button. The second time point is the generation time of the first target image frame. Then, the camera module, based on the state of the camera module or the object being photographed, uses the ISP to determine a sixth reference frame from the first and second target image frames. Based on this sixth reference frame, the camera module fuses the sixth image frame from multiple sets of image frames to output the first image.
[0116] In one embodiment, the multiple image frames further include a second image frame group adjacent to the first image frame group. The exposure duration of the third image frame in the first image frame group is less than or equal to the exposure duration of the first and second image frames in the first image frame group, and the third image frame in the first image frame group can be understood as a short exposure duration frame; the exposure amount of the third image frame in the second image frame group is less than or equal to the exposure amount of the first and second image frames in the second image frame group, and the third image frame in the second image frame group can be understood as a short exposure amount frame.
[0117] To facilitate understanding, the following will be combined with... Figure 17 Using DCG frames to represent the sixth image frame, ST frames to represent short exposure duration frames, and ES frames to represent short exposure amount frames as an example, the generation process of the first image is illustrated. Figure 17As shown, the image sensor outputs two image frame streams. One is a DCG frame stream, composed of all DCG frames output by the image sensor, including DCG1, DCG2, DCG3, and DCG4 frames, with a frame rate of 30fps. The other is a VS frame stream, composed of all ST and ES frames output by the image sensor, including the third image frame ST1, ES2, ST3, and ES4 frames. Specifically, the multiple image frames are divided into four groups: the first image frame group, the second image frame group, the third image frame group, and the fourth image frame group. The first image frame group includes DCG1 frame and ST1 frame, where the exposure duration of ST1 frame is less than or equal to the exposure duration of DCG1 frame; the second image frame group includes DCG2 frame and ES2 frame, where the exposure of ES2 frame is less than or equal to the exposure of DCG2 frame; the third image frame group includes DCG3 frame and ST3 frame, where the exposure duration of ST3 frame is less than or equal to the exposure duration of DCG3 frame; the fourth image frame group includes DCG4 frame and ES4 frame, where the exposure of ES4 frame is less than or equal to the exposure of DCG4 frame.
[0118] Specifically, the camera module uses the ISP to sequentially determine the first target image frame (e.g., DCG1 frame), the second target image frame (e.g., ST1 frame), and the third target image frame (e.g., ES2 frame) from the aforementioned four sets of image frames. The first target image frame is the DCG frame with the shortest interval to the first time step, the second target image frame is the ST frame with the shortest interval to the second time step, and the third target image frame is the ES frame with the shortest interval to the second time step. Furthermore, the camera module uses the ISP to determine the first target image frame as the sixth reference frame when the camera module or the subject being photographed is stationary; otherwise, when the camera module or the subject being photographed is in motion, the second target image frame is determined as the sixth reference frame. Next, the ISP fuses the third target image frame and the sixth image frame (DCG frame) from the aforementioned four sets of image frames based on the reference frame, and outputs the first image. The specific implementation is as follows: If the sixth reference frame is the second target image frame (ST1 frame), the ISP calculates the registration matrix between the first target image frame (DCG1 frame) and the sixth reference frame, and maps the first target image frame (DCG1 frame) to the position of the sixth reference frame using this registration matrix, obtaining image a. The ISP calculates the registration matrix between the third target image frame (ES2 frame) and the sixth reference frame, and maps the third target image frame (ES2 frame) to the position of the sixth reference frame using this registration matrix, obtaining image b. The ISP calculates the registration matrices between DCG2 frame, DCG3 frame, DCG4 frame, and the sixth reference frame respectively, and maps DCG2 frame, DCG3 frame, and DCG4 frame to the position of the sixth reference frame using these registration matrices, obtaining images c, d, and e respectively. Next, the ISP inputs the sixth reference frame, images a, b, c, d, and e together into a deep neural network capable of denoising and de-mosaicing. This completes multi-frame image fusion, denoising, and de-mosaicing processing to obtain the RGB image. The ISP then performs color correction, tone mapping, and sharpening on the RGB image to obtain the final image captured by the camera module, i.e., the first image. If the reference frame is the first target image frame DCG1, the ISP skips the step of obtaining image a. For other steps, please refer to the description in the embodiment where the sixth reference frame is the second target image frame ST1.
[0119] It is understood that in this embodiment, the two third image frames generated by four adjacent exposure times are alternating ST frames and ES frames. Since the ST frame has a short exposure time, it is suitable for capturing scenes, making the image frame output by the image sensor clearer. The ES frame has a small exposure amount, which can further improve the dynamic range of the final image. Therefore, this embodiment simultaneously takes into account the ability to improve clarity and improve dynamic range.
[0120] In another embodiment, the exposure duration of the third image frame in each group of image frames is less than or equal to the exposure duration of the sixth image frame in each group of image frames.
[0121] To facilitate understanding, the following will be combined with... Figure 18 Taking the DCG frame as the sixth image frame and the ES frame as the eleventh image frame as an example, the generation process of the first image is illustrated. Figure 18 As shown, with Figure 17 compared to, Figure 18 In the VS frame stream, the third image frame (VS frame) is always an ST frame, including ST1, ST2, ST3, and ST4. Specifically, there are four groups of image frames: the first group, the second group, the third group, and the fourth group. The first group includes DCG1 and ST1, with the exposure duration of ST1 being less than or equal to that of DCG1. The second group includes DCG2 and ST2, with the exposure duration of ST2 being less than or equal to that of DCG2. The third group includes DCG3 and ST3, with the exposure duration of ST3 being less than or equal to that of DCG3. The fourth group includes DCG4 and ST4, with the exposure duration of ST4 being less than or equal to that of DCG4.
[0122] Specifically, the camera module, through the ISP, sequentially determines the first target image frame (e.g., DCG1 frame) and the second target image frame (e.g., ST1 frame) from the aforementioned four sets of image frames. The first target image frame is the DCG frame with the shortest interval to the first time step, and the second target image frame is the ST frame with the shortest interval to the second time step. The ISP also determines the first target image frame as the sixth reference frame when the camera module or the subject being photographed is stationary; otherwise, it determines the second target image frame as the sixth reference frame when the camera module or the subject being photographed is in motion. To improve the dynamic range of the final image, the ISP also controls the image sensor to generate an eleventh image frame (ES frame) at the fourth exposure time after the image sensor generates the aforementioned four sets of image frames. The fourth exposure time is different from the exposure time corresponding to each of the aforementioned four sets of image frames. The eleventh image frame (ES frame) is an image frame in HCG or LCG mode, and the exposure of the eleventh image frame (ES frame) is less than or equal to the exposure of any of the aforementioned four sets of image frames (DCG frames). Next, the ISP fuses the DCG frame and the eleventh image frame (ES frame) from the above four sets of image frames based on the sixth reference frame, and outputs the first image. The specific implementation is as follows: If the sixth reference frame is the second target image frame (ST1 frame), the ISP calculates the registration matrix between the first target image frame (DCG1 frame) and the sixth reference frame, and maps the first target image frame (DCG1 frame) to the position of the sixth reference frame using this registration matrix, obtaining image a. The ISP calculates the registration matrix between the eleventh image frame (ES frame) and the sixth reference frame, and maps the eleventh image frame (ES frame) to the position of the sixth reference frame using this registration matrix, obtaining image b. The ISP calculates the registration matrices between the DCG2 frame, DCG3 frame, DCG4 frame, and the sixth reference frame respectively, and maps the DCG2 frame, DCG3 frame, and DCG4 frame to the position of the sixth reference frame using these registration matrices, obtaining images c, d, and e. Next, the ISP inputs the sixth reference frame, images a, b, c, d, and e together into a deep neural network capable of denoising and de-mosaicing. This completes multi-frame image fusion, denoising, and de-mosaicing processing to obtain the RGB image. The ISP then performs color correction, tone mapping, and sharpening on the RGB image to obtain the final image captured by the camera module, i.e., the first image. If the sixth reference frame is the first target image frame DCG1, the ISP skips the step of obtaining image a. For other steps, please refer to the description in the embodiment where the sixth reference frame is the second target image frame ST1.
[0123] It is understood that this embodiment is a capture-priority scheme. Compared with the previous embodiment, all third image frames in this embodiment are ST frames. The time interval between adjacent ST frames is shorter. When the camera module or the object being photographed by the camera module is moving, the ST frame that is closer to the user's shooting time can be selected, and the capture time is more accurate.
[0124] In another embodiment, the exposure of the third image frame in each group of image frames is less than or equal to the exposure of the sixth image frame in each group of image frames.
[0125] To facilitate understanding, the following will be combined with... Figure 19 The generation process of the first image frame is illustrated with an example. Figure 19 As shown, with Figure 18 compared to, Figure 19 In the VS frame stream, the third image frame (VS frame) is an ES frame, including ES1, ES2, ES3, and ES4. Specifically, there are four image frame groups: the first, second, third, and fourth image frame groups. The first image frame group includes DCG1 and ES1, with the exposure of ES1 being less than or equal to that of DCG1. The second image frame group includes DCG2 and ES2, with the exposure of ES2 being less than or equal to that of DCG2. The third image frame group includes DCG3 and ES3, with the exposure of ES3 being less than or equal to that of DCG3. The fourth image frame group includes DCG4 and ES4, with the exposure of ES4 being less than or equal to that of DCG4.
[0126] Specifically, the camera module uses the ISP to sequentially determine the first target image frame (e.g., DCG1 frame) and the second target image frame (e.g., ES1 frame) from the aforementioned four sets of image frames. The first target image frame is the DCG frame with the shortest interval to the first time step, and the second target image frame is the ES frame with the shortest interval to the second time step. The ISP also determines the first target image frame as the sixth reference frame if the camera module or its subject is stationary; otherwise, it determines the second target image frame as the sixth reference frame if the camera module or its subject is in motion. Then, the ISP fuses the second target frame and the sixth image frame (DCG frame) from the aforementioned four sets of image frames based on the sixth reference frame, outputting the first image. Specifically, if the sixth reference frame is the second target image frame (ES1 frame), the ISP calculates the registration matrix between the first target image frame (DCG1 frame) and the sixth reference frame, and maps the first target image frame (DCG1 frame) to the position of the sixth reference frame using this registration matrix to obtain image a. The ISP calculates the registration matrix between DCG2, DCG3, DCG4, and the sixth reference frame, respectively. Using this matrix, it maps DCG2, DCG3, and DCG4 to the position of the sixth reference frame, resulting in images b, c, and d. Then, the ISP inputs the sixth reference frame, images a, b, c, and d together into a deep neural network capable of denoising and de-mosaicing. This performs multi-frame fusion, denoising, and de-mosaicing to obtain the RGB image. The ISP then performs color correction, tone mapping, and sharpening on the RGB image to obtain the final image captured by the camera module, i.e., the first image. If the sixth reference frame is the first target image frame DCG1, the ISP skips the step of obtaining image a and performs the calculation of the registration matrix between the second target image frame ES1 and the sixth reference frame. The second target image frame ES1 is then mapped to the position of the sixth reference frame using this registration matrix to obtain image e. Image e, along with the sixth reference frame, images a, b, c, and d, is then input into a deep neural network capable of denoising and de-mosaicing. For other steps, please refer to the description in the embodiment where the sixth reference frame is the second target image frame ES1.
[0127] It is understood that this embodiment is a dynamic range-first scheme. All third image frames in this embodiment are ES frames. Since ES frames have a small exposure, they can further improve the dynamic range of the final image. Therefore, the dynamic range of the first image in this embodiment is wider.
[0128] In another embodiment, the camera module outputs a first image based on multiple sets of image frames via an ISP. These multiple sets of image frames include a first image frame group, which comprises a sixth image frame, a third image frame, and an eighth image frame. The sixth image frame is a fused image frame resulting from the first and second image frames, and the eighth image frame is a fused image frame resulting from the fourth and fifth image frames. Here, the sixth and eighth image frames are of the same type, namely, DCG frames.
[0129] The camera module sequentially determines a first target image frame and a second target image frame from multiple sets of image frames via the ISP. The first target image frame is either the sixth or eighth image frame with the shortest interval to the first time step, and the second target image frame is the third image frame with the shortest interval to the second time step. Then, based on the state of the camera module or the object being photographed, the camera module determines a sixth reference frame from the first and second target image frames via the ISP. Based on this sixth reference frame, the sixth and eighth image frames from the multiple sets of image frames are fused to output the first image.
[0130] In one embodiment, when the camera module or the object being photographed by the camera module is in motion, the exposure duration of the third image frame in each group of image frames is less than or equal to the exposure duration of the sixth and eighth image frames in each group of image frames. Here, the third image frame in each group of image frames can be understood as a short exposure duration frame.
[0131] To facilitate understanding, the following will be combined with... Figure 20 Using DCG frames to represent the sixth and eighth image frames, ST frames to represent short-exposure frames, and ES frames to represent the eleventh image frame as examples, the generation process of the first image is illustrated. Figure 20 As shown, the DCG frame stream consists of all DCG frames output by the image sensor, including the sixth image frame DCG1, the eighth image frame DCG2, the sixth image frame DCG3, and the eighth image frame DCG4. The frame rate of the DCG frame stream is 30fps. The VS frame stream consists of all ST frames output by the image sensor, including the third image frame ST1 and the third image frame ST3. The frame rate of the VS frame stream is 15fps. Here, the multiple image frames are specifically divided into two groups: the first image frame group and the second image frame group. The first image frame group includes the sixth image frame DCG1, the third image frame ST1, and the eighth image frame DCG2. The exposure duration of the ST1 frame is less than or equal to the exposure duration of either the DCG1 or DCG2 frame. The second image frame group includes the sixth image frame DCG3, the third image frame ST3, and the eighth image frame DCG4. The exposure duration of the ST3 frame is less than or equal to the exposure duration of either the DCG3 or DCG4 frame.
[0132] Here, the camera module uses the ISP to fuse the sixth and eighth image frames from the aforementioned four sets of image frames based on the sixth reference frame to output the first image. For a detailed implementation of this, please refer to [link to relevant documentation]. Figure 18 The descriptions of the corresponding parts in the illustrated embodiments will not be repeated here.
[0133] It is understood that this embodiment prioritizes snapshot capture when the camera module or the object being photographed by the camera module is moving. In addition, the image processing method provided in this embodiment requires fewer third image frames in the process of generating the first image. Therefore, it can not only make the generated first image clearer, but also reduce the power consumption of the image sensor.
[0134] In another embodiment, when the camera module or the subject being photographed by the camera module is stationary, the exposure of the third image frame in each group of image frames is less than or equal to the exposure of the sixth and eighth image frames in each group of image frames. Here, the third image frame in each group of image frames can be understood as a short exposure frame.
[0135] To facilitate understanding, the following will be combined with... Figure 21 Using DCG frames to represent the sixth and eighth image frames, and ES frames to represent short-exposure frames as examples, the generation process of the first image is illustrated. Figure 21 As shown, with Figure 20 compared to, Figure 20 Each third image frame in the VS frame stream is an ES frame, including ES1 and ES3 frames. Specifically, the multiple image frames are divided into two groups: the first image frame group and the second image frame group. The first image frame group includes the sixth image frame (DCG1), the third image frame (ES1), and the eighth image frame (DCG2). The exposure of ES1 is less than or equal to the exposure of either DCG1 or DCG2. The second image frame group includes the sixth image frame (DCG3), the third image frame (ES3), and the eighth image frame (DCG4). The exposure of ES3 is less than or equal to the exposure of either DCG3 or DCG4.
[0136] Here, the ISP fuses the sixth and eighth image frames from the two sets of image frames mentioned above based on the sixth reference frame to output the first image. For a detailed implementation of this, please refer to [link to relevant documentation]. Figure 19 The descriptions of the corresponding parts in the illustrated embodiments will not be repeated here.
[0137] It is understood that this embodiment prioritizes capturing dynamic range when the camera module or the subject being photographed by the camera module is stationary. Furthermore, the image processing method provided in this embodiment requires fewer third image frames to generate the first image; therefore, it not only improves the dynamic range of the captured image but also reduces the power consumption of the image sensor.
[0138] In addition, in the second scenario, the ISP also performs black level correction, noise removal, interpolation de-mosaic, white balance correction, color gamut correction, tone mapping adjustment, gamma correction, and sharpening on the DCG frame stream in the above embodiments in sequence to obtain the camera preview stream.
[0139] In this application, the image processing method provides a frame output strategy that generates a first image frame, a second image frame, and a third image frame between two adjacent exposure times. Compared to the scheme where an image sensor using a TCG pixel structure generates six image frames between two adjacent exposure times, the frame output strategy provided in this application reduces the number of image frames generated by the image sensor between two adjacent exposure times by half, thereby reducing the number of image frames that the ISP needs to process subsequently by half, and thus reducing the power consumption of the image sensor and the ISP. Furthermore, when the first conversion gain mode is HCG mode and the second conversion gain mode is LCG mode, the first image frame is an HCG frame, the second image frame is an LCG frame, and the third image frame is an HCG frame or an LCG frame with exposure parameters smaller than those of the first and second image frames. The third image frame is used to supplement the extremely bright light details missing in the HCG and LCG frames. In other words, the third image frame can be equivalent to the role of a LOFIC frame in image imaging. Therefore, the image processing method provided in this application can also achieve HDR without degrading image quality. In summary, the image processing method provided in this application can reduce the power consumption of the image sensor and the ISP without reducing image quality.
[0140] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.
Claims
1. An image processing method, characterized by, The method is applied to an electronic device, which includes a camera module; The method includes: A first image frame and a second image frame are generated during the first exposure time in two adjacent exposure times, and a third image frame is generated during the second exposure time in the two adjacent exposure times. The first image frame is an image frame in a first conversion gain mode, the second image frame is an image frame in a second conversion gain mode, and the third image frame is an image frame in either the first conversion gain mode or the second conversion gain mode. Furthermore, the exposure parameters of the third image frame are less than or equal to the exposure parameters of the first image frame and the second image frame, and the exposure parameters include exposure amount or exposure duration. Output a first image or a first video based on the first image frame, the second image frame, and the third image frame.
2. The method of claim 1, wherein, The method further includes: A fourth image frame and a fifth image frame are generated at the third exposure time, wherein the third exposure time is the next exposure time after the two adjacent exposure times, the fourth image frame is an image frame in the first conversion gain mode, the fifth image frame is an image frame in the second conversion gain mode, and the exposure parameters of the fourth image frame and the fifth image frame are both greater than or equal to the exposure parameters of the third image frame. The step of outputting a first image or a first video based on the first image frame, the second image frame, and the third image frame includes: Based on the first image frame, the second image frame, the third image frame, the fourth image frame, and the fifth image frame, output the first image or the first video.
3. The method of claim 1, wherein, The step of outputting a first video based on the first image frame, the second image frame, and the third image frame includes: One of the first image frames and the second image frame is determined as the first reference frame, and the other image frame is fused with the first image frame and the second image frame based on the first reference frame to generate the sixth image frame; The third image frame is determined as the second reference frame, and the sixth image frame is fused based on the second reference frame to generate the seventh image frame; The seventh image frame is inserted into the sixth image frame to output the first video, wherein the seventh image frame is the next image frame after the sixth image frame.
4. The method of claim 3, wherein, The exposure of the first reference frame is the same as that of the second reference frame.
5. The method according to claim 3 or 4, characterized in that, The resolution of the third image frame is smaller than that of the sixth image frame.
6. The method of claim 2, wherein, The step of outputting the first video based on the first image frame, the second image frame, the third image frame, the fourth image frame, and the fifth image frame includes: One of the first image frames and the second image frame is determined as the first reference frame, and the other image frame is fused with the first image frame and the second image frame based on the first reference frame to generate the sixth image frame; One of the fourth and fifth image frames is determined as the third reference frame, and the other image frame is fused with the third reference frame to generate the eighth image frame; The sixth image frame is determined as the fourth reference frame, and the third image frame is fused based on the fourth reference frame to generate the ninth image frame; The eighth image frame is determined as the fifth reference frame, and the third image frame is fused based on the fifth reference frame to generate the tenth image frame; The first video is output based on the ninth and tenth image frames.
7. The method of claim 6, wherein, The exposure of the third image frame is less than or equal to the exposure of the sixth and eighth image frames.
8. The method according to claim 1, characterized in that, The step of outputting a first image based on the first image frame, the second image frame, and the third image frame includes: The first image is output based on multiple sets of image frames, wherein the multiple sets of image frames include a first image frame group, the first image frame group includes a sixth image frame and the third image frame, and the sixth image frame is an image frame after the first image frame and the second image frame are fused.
9. The method according to claim 8, characterized in that, The step of outputting the first image based on multiple sets of image frames includes: The first target image frame and the second target image frame are determined sequentially from the multiple sets of image frames. The first target image frame is the sixth image frame with the shortest interval to the first time point, and the second target image frame is the third image frame with the shortest interval to the second time point. The second time point is the generation time of the first target image frame. Based on the state of the camera module or the object being photographed by the camera module, a sixth reference frame is determined from the first target image frame and the second target image frame; The first image is output by fusing the sixth image frame from the multiple sets of image frames based on the sixth reference frame.
10. The method according to claim 9, characterized in that, The multiple image frames also include a second image frame group adjacent to the first image frame group. The exposure parameter of the first image frame group is the exposure duration, and the exposure parameter of the second image frame group is the exposure amount.
11. The method according to claim 10, characterized in that, The second target image frame is the third image frame that has the shortest interval with the second time and whose exposure time is less than or equal to the exposure time of the sixth image frame in the same group of image frames; The method further includes: A third target image frame is determined from the multiple sets of data frames, wherein the third target image frame is the third image frame with the shortest interval to the second time and whose exposure is less than or equal to the exposure of the sixth image frame in the same set of image frames; The step of fusing the sixth image frame from the multiple sets of image frames based on the sixth reference frame to output the first image includes: The first image is output by fusing the third target image frame and the sixth image frame from the multiple sets of image frames based on the sixth reference frame.
12. The method according to claim 9, characterized in that, The exposure parameter for each of the multiple sets of image frames is the exposure duration.
13. The method according to claim 12, characterized in that, The method further includes: An eleventh image frame is generated at the fourth exposure time, wherein the fourth exposure time is different from both the first and second exposure times, the eleventh image frame is an image frame in the first conversion gain mode or the second conversion gain mode, and the exposure of the eleventh image frame is less than or equal to the exposure of any sixth image frame in the plurality of image frames. The step of fusing the sixth image frame from the multiple sets of image frames based on the sixth reference frame to output the first image includes: The first image is output by fusing the sixth image frame and the eleventh image frame from the multiple sets of image frames based on the sixth reference frame.
14. The method according to claim 9, characterized in that, The exposure parameter for each of the multiple sets of image frames is the exposure amount.
15. The method according to claim 14, characterized in that, The step of fusing the sixth image frame from the multiple sets of image frames based on the sixth reference frame to output the first image includes: The first image is output by fusing the second target image frame and the sixth image frame from the multiple sets of image frames based on the sixth reference frame.
16. The method according to claim 2, characterized in that, The step of outputting the first image based on the first image frame, the second image frame, the third image frame, the fourth image frame, and the fifth image frame includes: The first image is output based on multiple sets of image frames, wherein the multiple sets of image frames include a first image frame group, the first image frame group includes a sixth image frame, the third image frame and an eighth image frame, the sixth image frame is an image frame after the first image frame and the second image frame are fused, and the eighth image frame is an image frame after the fourth image frame and the fifth image frame are fused.
17. The method according to claim 16, characterized in that, The step of outputting the first image based on multiple sets of image frames includes: The first target image frame and the second target image frame are determined sequentially from the multiple sets of image frames. The first target image frame is the sixth or eighth image frame with the shortest interval to the first time point, and the second target image frame is the third image frame with the shortest interval to the second time point. The second time point is the generation time of the first target image frame. Based on the state of the camera module or the object being photographed by the camera module, a sixth reference frame is determined from the first target image frame and the second target image frame; The first image is output by fusing the sixth and eighth image frames from the multiple sets of image frames based on the sixth reference frame.
18. The method according to claim 17, characterized in that, When the camera module or the object being photographed by the camera module is in motion, the exposure parameter of each image frame in the multiple sets of image frames is the exposure duration.
19. The method according to claim 18, characterized in that, The method further includes: An eleventh image frame is generated at the fourth exposure time, wherein the fourth exposure time is different from the first exposure time, the second exposure time and the third exposure time, the eleventh image frame is an image frame in the first conversion gain mode or the second conversion gain mode, and the exposure of the eleventh image frame is less than or equal to the exposure of any sixth image frame and any eighth image frame in the plurality of image frames. The step of fusing the sixth and eighth image frames from the multiple sets of image frames based on the sixth reference frame to output the first image includes: The first image is output by fusing the sixth image frame, the eighth image frame, and the tenth image frame from the multiple sets of image frames based on the sixth reference frame.
20. The method according to claim 17, characterized in that, When the camera module or the object being photographed by the camera module is stationary, the exposure parameter of each image frame in the plurality of image frames is the exposure amount.
21. The method according to claim 20, characterized in that, The step of fusing the sixth and eighth image frames from the multiple sets of image frames based on the sixth reference frame to output the first image includes: The first image is output by fusing the second target image frame, the sixth image frame, and the eighth image frame from the multiple sets of image frames based on the sixth reference frame.
22. The method according to any one of claims 9-15 and 17-21, characterized in that, The step of determining a sixth reference frame from the first target image frame and the second target image frame based on the state of the camera module or the object being photographed by the camera module includes: When the camera module or the object being photographed by the camera module is in a static state, the first target image frame is determined as the sixth reference frame; or, When the camera module or the object being photographed by the camera module is in motion, the second target image frame is determined as the sixth reference frame.
23. The method according to any one of claims 1-22, characterized in that, The camera module includes an image sensor, which includes multiple pixels arranged in an array. Each pixel includes a photodiode, a charge transport gate, a floating diffusion node, and a conversion gain mode control circuit. The conversion gain mode control circuit includes at least one capacitor, which includes a first capacitor. The charge transport gate is used to turn on or off the connection between the cathode of the photodiode and the floating diffusion node. The conversion gain mode control circuit is in an on or off state to turn on the connection between the floating diffusion node and the first capacitor, or to disconnect the connection between the floating diffusion node and the at least one capacitor. The first conversion gain mode is the conversion gain mode of the image sensor when the conversion gain mode control circuit is in the off state; The second conversion gain mode is the conversion gain mode of the image sensor when the conversion gain mode control circuit is in the on state.
24. The method according to claim 23, characterized in that, The number of the at least one capacitor is 1. The image sensor also includes a readout line. The pixel also includes a reset transistor and a row selector. The conversion gain mode control circuit also includes a first switch transistor. The reset transistor and the first switch transistor are connected in series between the floating diffusion node and the power supply. The connection between the reset transistor and the first switch transistor is connected to the reference ground through the first capacitor. The first end of the row selector is connected to the floating diffusion node, and the second end of the row selector is connected to the readout line. The image frame in the first conversion gain mode is determined by a first reset voltage and a first signal voltage, wherein the first reset voltage is the second terminal voltage of the row selector when the row selector is turned on after both the reset transistor and the first switch transistor are turned off and before the charge transfer gate is turned on; the first signal voltage is the second terminal voltage of the row selector when the row selector is turned on after the charge transfer gate is turned off. The image frame in the second conversion gain mode is determined by a second reset voltage and a second signal voltage, wherein the second reset voltage is the second terminal voltage of the horizontal selector when both the first switch and the horizontal selector are turned on after the reset transistor is turned off and before the charge transfer gate is turned on; the second signal voltage is the second terminal voltage of the horizontal selector when both the first switch and the horizontal selector are turned on after the charge transfer gate is turned off.
25. The method according to claim 23, characterized in that, The at least one capacitor further includes a second capacitor; the image sensor further includes a readout line; the pixel further includes a reset transistor and a row selector; the conversion gain mode control circuit further includes a first switch and a second switch, wherein the reset transistor and the first switch are connected in series between the floating diffusion node and the power supply, and the connection between the reset transistor and the first switch is connected to the reference ground through the first capacitor; the second switch and the second capacitor are connected in series between the floating diffusion node and the reference ground; the first end of the row selector is connected to the floating diffusion node, and the second end of the row selector is connected to the readout line; The image frame in the first conversion gain mode is determined by a first reset voltage and a first signal voltage. The first reset voltage is the second terminal voltage of the row selector when the row selector is turned on, after both the first and second switches are turned off and before the charge transfer gate is turned on. The first signal voltage is the second terminal voltage of the row selector when the row selector is turned on, after the charge transfer gate is turned off and before the first switch is turned on. The image frame in the second conversion gain mode is determined by a second reset voltage and a second signal voltage, wherein the second signal voltage is the second terminal voltage of the horizontal selector when the charge transfer gate is turned off and the reset transistor is turned on, the second switch is turned off and both the first switch transistor and the horizontal selector are turned on; the second reset voltage is the second terminal voltage of the horizontal selector when the reset transistor is turned off and the second switch transistor is turned off and both the first switch transistor and the horizontal selector are turned on.
