Image processing method, image sensor, camera, chip system and electronic device

Abstract

The application relates to the technical field of cameras and discloses an image processing method, an image sensor, a camera, a chip system and electronic equipment. The method can be applied to an image sensor of electronic equipment. In the process of reading out an image based on a mode, when a seamless switching instruction of switching modes is detected, the image sensor can judge whether the sum of the default switching delay time and the exposure time of the new mode is less than the vertical blanking time of the current mode. If the judgment is that the sum is less than the vertical blanking time, it is indicated that the current image frame will be reduced in frame time and increased in frame rate when the mode is switched, and then the image sensor automatically adjusts the switching delay time of the new mode, so that the sum of the adjusted switching delay time and the exposure time of the new mode is equal to the vertical blanking time of the current mode, the frame rate of the switching frame in the mode switching process is kept unchanged, seamless switching of the mode is realized, and the continuity and smoothness of the video output by the image sensor are ensured.

Description

Technical Field

[0001] This application relates to the field of camera technology, and in particular to an image processing method, an image sensor, a camera, a chip system, and an electronic device. Background Technology

[0002] When a mobile phone or tablet camera is working, it switches modes based on changes in ambient brightness or the usage scenario to ensure image quality. For example, when switching from a portrait shooting scene to a night scene shooting scene, the mode corresponding to the portrait shooting scene can be switched to the mode corresponding to the night scene shooting scene. Different camera modes have different mode parameters (such as resolution).

[0003] During the process of a camera capturing images and generating a video stream, changes in the frame rate of image frames can cause stuttering or delays in the preview or playback of the video stream. For example, when the camera is capturing and reading the Nth frame in mode B, if a switch command to mode A is received, the camera will switch from mode B to mode A after reading the Nth frame, and then capture and process the (N+1)th frame based on the switched mode A. If the frame rates of the (N-1)th, Nth, and (N+1)th frames are different during the mode switching process, the preview or display of the (N+1)th frame may experience stuttering or delays, resulting in poor video stream smoothness.

[0004] Therefore, to ensure the smoothness of the video stream captured by the camera, it is necessary to maintain a consistent frame rate for each video frame during the camera's mode switching process. Thus, ensuring a constant frame rate during mode switching becomes a pressing issue. Summary of the Invention

[0005] This application provides an image processing method, an image sensor, a camera, a chip system, and an electronic device, which can achieve seamless mode switching and ensure the continuity and smoothness of the video output by the image sensor.

[0006] In a first aspect, embodiments of this application provide an image processing method, the method comprising: acquiring first image data by photoelectric conversion of first image light data captured by an image sensor and lens, wherein the image sensor and lens are located in a camera of an electronic device; processing the first image data based on first mode parameters, the first mode parameters including a first frame rate; detecting a switching command for switching second mode parameters during the reading of the first image data; determining that a second frame rate of the second mode parameters is greater than the first frame rate, wherein the second frame rate is related to a first switching delay time of the second mode parameters; after the first image data is read out, photoelectric conversion is performed on the second image light data captured by the lens after a second switching delay time to obtain second image data, and the second image data is processed based on the second mode parameters, wherein the second switching delay time is greater than the first switching delay time, and the second frame rate corresponding to the second switching delay time is equal to the first frame rate.

[0007] It's understandable that if the second frame rate of the second mode parameter is greater than the first frame rate, it means that during mode switching, the frame time of the current image frame (denoted as the switching frame) will decrease while the frame rate increases. Therefore, the image sensor automatically adjusts the switching delay time of the new second mode parameter so that the frame rate of the second mode parameter corresponding to the adjusted second switching delay time equals the frame rate of the first mode parameter. This ensures that the frame rate of the switching frame remains constant during mode switching, meaning the frame time of the switching frame is consistent with the frame time of the images before and after the mode switch. This achieves seamless mode switching, guarantees the continuity and smoothness of the video output by the image sensor, and avoids visually perceptible changes in the video frame rate for the user.

[0008] Furthermore, when the image sensor automatically detects an increase in the frame rate of the switching frame, it autonomously adjusts the switching delay time of the new mode to ensure that the frame rate of the switching frame containing the first image data remains consistent with the frame rates of the previous and next frames. During this process, the image sensor does not need to interact with the application processor (AP) via signaling, thus improving the efficiency of adjusting the frame rate of the switching frame during mode switching.

[0009] In one possible implementation of the first aspect described above, the mode parameters include at least one of the following: output image size, operating clock, resolution, frame rate, and high dynamic range (HDR) parameters. These mode parameters can be used to configure the camera's output image parameters.

[0010] In one possible implementation of the first aspect above, determining that the second frame rate of the second mode parameter is greater than the first frame rate includes: determining that the second frame rate of the second mode parameter is greater than the first frame rate corresponding to the first vertical blanking time being greater than the second vertical blanking time, wherein the second vertical blanking time is the sum of the first exposure time and the first switching delay time of the second mode parameter, and the first vertical blanking time is the sum of the exposure time and the switching delay time of the first mode parameter.

[0011] The first frame rate can be the reciprocal of the frame time of the first mode parameter, where the frame time includes the first vertical blanking time of the first mode parameter. Similarly, the second frame rate can be the reciprocal of the frame time of the second mode parameter, where the frame time includes the second vertical blanking time of the second mode parameter.

[0012] In one possible implementation of the first aspect described above, the method further includes: corresponding to a first vertical blanking time being greater than a second vertical blanking time, subtracting a first exposure time from the first vertical blanking time to calculate a second switching delay time. It is understood that for an image sensor, the exposure time and vertical blanking time of a mode parameter are usually fixed. Therefore, subtracting the first exposure time from the first vertical blanking time to calculate the second switching delay time ensures that the sum of the second switching delay time and the first exposure time equals the first vertical blanking time. This makes the vertical blanking time of the first image data equal to the vertical blanking time of the second image data, that is, the frame time of the first image data equals the frame time of the second image data.

[0013] In one possible implementation of the first aspect above, the processing of the second image data based on the second mode parameters includes: exposing the second image data according to the first gain and the first exposure time of the second mode parameters, and reading out the second image data according to the first readout time of the second mode parameters.

[0014] In one possible implementation of the first aspect described above, detecting the switching instruction for switching the second mode parameter includes: receiving the switching instruction for switching the second mode parameter from the application processor (AP); wherein the second mode parameter is determined based on the ambient brightness, or the second mode parameter is determined based on the selected shooting mode. That is, as the camera's usage scenario or ambient brightness changes, new mode parameters can be automatically or passively switched to change the new mode.

[0015] In one possible implementation of the first aspect described above, the method further includes: determining that the second frame rate of the second mode parameter is less than the first frame rate; receiving an adjustment instruction from the AP, the adjustment instruction including a second exposure time of the second mode parameter, wherein the second vertical blanking time of the second exposure time is equal to the first vertical blanking time; after the first image data is read out, after a first switching delay time, performing photoelectric conversion on the second image light data acquired by the lens to obtain second image data, and exposing the second image data according to the first gain and the second exposure time of the second mode parameter, and reading out the second image data according to the first readout time of the second mode parameter. Thus, the sum of the first switching delay time and the second exposure time can be made equal to the first vertical blanking time, so that the vertical blanking time of the first image data is equal to the vertical blanking time of the second image data, that is, the frame time of the first image data is equal to the frame time of the second image data.

[0016] In one possible implementation of the first aspect described above, the second exposure time is equal to the first vertical blanking time minus the first switching delay time. For example, the second exposure time can be obtained by the AP by subtracting the first switching delay time from the first vertical blanking time and then sent to the image sensor.

[0017] In one possible implementation of the first aspect described above, both the first image data and the second image data have a first brightness. Furthermore, the method further includes: obtaining a third exposure time and a second gain of the first mode parameters, and obtaining a first exposure time of the second mode parameters, wherein the first brightness is related to the third exposure time and the second gain; determining a first gain based on the third exposure time, the second gain, and the first exposure time, wherein the first brightness is related to the first exposure time and the first gain. In this case, the first gain and the first exposure time can ensure that the brightness of the first image data is consistent with the brightness of the second image data, avoiding visual perception of significant brightness changes in the video by the user.

[0018] In one possible implementation of the first aspect described above, determining the first gain based on the third exposure time, the second gain, and the first exposure time includes: dividing the product of the third exposure time and the second gain by the first exposure time to obtain the first gain. In this case, the product of the third exposure time and the second gain of the first mode parameter is equal to the product of the first exposure time and the first gain of the second mode parameter.

[0019] Secondly, embodiments of this application provide an image sensor for performing the image processing method described in the first aspect and any possible implementation thereof.

[0020] In one possible implementation of the second aspect described above, the image sensor includes a delay unit for adjusting the switching delay time of the mode parameters of the image sensor.

[0021] In one possible implementation of the second aspect described above, the image sensor is a complementary metal-oxide-semiconductor (CMOS) sensor.

[0022] Thirdly, embodiments of this application provide a camera, which includes a lens and an image sensor as described in the second aspect and any possible implementation thereof.

[0023] Fourthly, embodiments of this application provide a chip system, which includes an application processor (AP) and an image sensor; wherein the AP is used to send a switching instruction for the mode parameters of a camera to the image sensor, and the image sensor is used to execute the image processing method described in the first aspect and any possible implementation thereof.

[0024] Fifthly, embodiments of this application provide an electronic device, including: a processor, a memory, and a camera; the memory is used to store instructions executed by one or more processors of the electronic device; the processor is one of the processors of the electronic device and is used to send a mode parameter switching instruction to the camera; the camera is used to execute the image processing method in the first aspect and any possible implementation thereof.

[0025] The beneficial effects of the second, third, fourth, and fifth aspects can be referred to the relevant descriptions in the first aspect, and will not be repeated here. Attached Figure Description

[0026] Figure 1 According to some embodiments of this application, a schematic diagram of the structure of an electronic device is shown;

[0027] Figure 2 According to some embodiments of this application, a schematic diagram of an electronic device outputting an image is shown;

[0028] Figure 3 According to some embodiments of this application, a schematic diagram of the composition of frame time of an image is shown;

[0029] Figure 4 According to some embodiments of this application, a schematic diagram of the frame structure of a MIPI signal in an image output process is shown;

[0030] Figure 5 According to some embodiments of this application, a schematic diagram of an electronic device outputting an image is shown;

[0031] Figure 6According to some embodiments of this application, a schematic diagram of the frame structure of a MIPI signal in an image output process is shown;

[0032] Figure 7 According to some embodiments of this application, a schematic diagram of the exposure structure inside a chip and the frame structure of MIPI during the image output process of an electronic device is shown.

[0033] Figure 8A According to some embodiments of this application, a schematic diagram of the user interface of an electronic device is shown;

[0034] Figure 8B According to some embodiments of this application, a schematic diagram of the user interface of an electronic device is shown;

[0035] Figure 8C According to some embodiments of this application, a schematic diagram of the user interface of an electronic device is shown;

[0036] Figure 8D According to some embodiments of this application, a schematic diagram of the user interface of an electronic device is shown;

[0037] Figure 8E According to some embodiments of this application, a schematic diagram of the user interface of an electronic device is shown;

[0038] Figure 9 According to some embodiments of this application, a schematic flowchart of an image processing method is shown;

[0039] Figure 10 According to some embodiments of this application, a flowchart of an image data processing procedure is shown;

[0040] Figure 11 According to some embodiments of this application, a flowchart of an image data processing procedure is shown;

[0041] Figure 12 According to some embodiments of this application, a schematic flowchart of an image processing method is shown;

[0042] Figure 13 According to some embodiments of this application, a schematic diagram of a layered architecture for an electronic device is shown;

[0043] Figure 14 According to some embodiments of this application, a schematic diagram of a chip system is shown. Detailed Implementation

[0044] The illustrative embodiments of this application include, but are not limited to, image processing methods, image sensors, cameras, electronic devices, and chip systems.

[0045] First, some concepts involved in the embodiments of this application will be briefly explained.

[0046] 1. Initialize parameters

[0047] Initialization parameters are parameters used to configure the camera to start up, such as analog circuit configuration parameters and digital circuit configuration parameters that enable the image sensor to function properly.

[0048] 2. Mode parameters

[0049] Mode parameters refer to the parameters configured according to different usage scenarios of the camera, used to configure the camera's output image parameters. Mode parameters may include, but are not limited to, output image size, clock speed (clk), resolution, frame rate, and specific function (such as high-dynamic range (HDR) function) configuration parameters (such as HDR parameters), which differ from other modes. For example, when the camera is shooting photos, its default output image size is 4:3. When the camera is shooting video, its default output image size is 16:9.

[0050] 3. Effect parameters

[0051] Effect parameters are used to adjust the image quality of the output image, such as horizontal random noise (HRN) and vertical random noise (VRN).

[0052] 4. Readout time

[0053] Readout time refers to the time required for an image sensor to read out one frame of image data. This time depends on the sensor's readout circuit architecture and specific implementation. In complementary metal-oxide-semiconductor (CMOS) image sensors, readout time can range from a few microseconds (μs) to tens of microseconds.

[0054] 5. Exposure time

[0055] Exposure time refers to the time it takes for a pixel in an image sensor to receive light and perform photoelectric conversion. It determines the brightness and dynamic range of an image. The exposure time can be adjusted as needed to adapt to different lighting conditions and shooting requirements. In CMOS image sensors, exposure time is typically measured in milliseconds (ms) or microseconds (μs). For example, when shooting fast-moving objects, a shorter exposure time may be needed to reduce motion blur; while when shooting night scenes or low-light environments, a longer exposure time is needed to increase image brightness.

[0056] 6. Vertical blanking time (V_Blank)

[0057] Vertical blanking time, also known as inter-frame blanking time, refers to the time interval between the completion of reading the last line of data in one frame and the start of reading the first line of data in the next frame. During this time, the sensor does not read image data but performs some internal processing or prepares for the exposure of the next frame.

[0058] 7. Frame Time

[0059] Frame time, also known as frame period, refers to the time required for a camera to generate and output one frame of an image. Specifically, the frame time of one frame can be the time between the start of frame (SOF) of the current frame and the start of frame of the next frame.

[0060] 8. Frame rate

[0061] Frame rate can be defined as the number of image frames generated and output per unit of time (usually per second). Frame rate is inversely proportional to frame time.

[0062] Next, the electronic devices involved in the embodiments of this application will be described.

[0063] like Figure 1 As shown, the electronic device 100 may include a processor 110, a camera 120, a display screen 130, etc.

[0064] Processor 110 may include one or more processing units, such as a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), and an application processor (AP). These different processing units may be independent devices or integrated into one or more processors. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution. Processor 110 may also include memory for storing instructions and data.

[0065] Electronic device 100 implements display functions through a GPU, display screen 130, and application processing unit (AP). The GPU is a microprocessor for image processing, connected to the display screen 130 and the AP. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0066] Electronic device 100 can perform shooting functions through ISP, camera 120, video codec, GPU, display screen 130 and AP, etc.

[0067] The AP is used to send initialization parameters, mode parameters, effect parameters and other parameters to the camera 120. Specifically, it can send switching commands to the camera 120 to trigger mode switching.

[0068] The ISP (Image Signal Processor) is used to process data fed back from the camera 120. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's image sensor. The light signal is converted into an electrical signal and then into a digital image signal. The camera's image sensor then processes this digital image signal. For example, the ISP can perform algorithmic optimization on image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 120.

[0069] The camera 120 is used to capture still images or videos. Specifically, the camera 120 includes a lens and an image sensor.

[0070] A lens is used to transmit light and focus it onto an image sensor. A lens can consist of multiple lens elements or a single lens element; there is no limitation here.

[0071] An image sensor is used to convert light of different colors from a beam of light from a lens into different electrical signals to generate a color image. An image sensor can be called a photosensitive element, specifically a charge-coupled device (CCD) or a CMOS phototransistor. The image sensor converts the light signal into an electrical signal, and then into a digital image signal, which is then passed to an ISP for optimization processing. The ISP outputs the digital image signal to a DSP for further processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or M cameras 120, where M is a positive integer greater than 1.

[0072] DSPs are used to process digital signals. In addition to digital image signals, they can also process other digital signals.

[0073] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0074] The display screen 130 is used to display human-computer interaction interfaces, images, videos, etc., such as displaying the user interface of a camera application, i.e., displaying images captured by the camera. In some embodiments, the electronic device 100 may include one or M display screens 130, where M is a positive integer greater than 1.

[0075] It is understood that the structures illustrated in the embodiments of the present invention do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include... Figure 1 The diagram shows more or fewer components, or combinations of components, or separate components, or different component arrangements. The components shown can be implemented in hardware, software, or a combination of both. For example, camera 120 also includes devices such as a voice coil motor (VCM) and a flexible printed circuit board (FPCB).

[0076] Figure 2 This is a schematic diagram illustrating the process of an electronic device outputting an image. This process is mainly achieved through the interaction between the AP in the processor and the image sensor in the camera. Figure 2 As shown, the electronic device 100 requires hardware power (i.e., ...) before outputting the image. Figure 1The device 100 powers on the image sensor and then uses the AP to send initialization parameters, effect parameters, and mode parameters (such as mode parameter B1 of mode B) sequentially to the image sensor via the inter-integrated circuit (I2C) bus. The image sensor then loads the initialization parameters, effect parameters, and mode parameters. The electronic device 100 can then send a stream command to the image sensor via the AP. Figure 2 The "stream on" command in the context of image sensors instructs the image sensor to output the acquired image. In response, the image sensor can output (i.e., read out) the acquired image via the Mobile Industry Processor Interface (MIPI) line, such as to an ISP or other processing unit.

[0077] In some embodiments, the exposure times for different modes are typically different. Therefore, during mode switching, the camera's exposure time can be adjusted from the exposure time of the previous mode to the exposure time of the next mode.

[0078] However, during mode switching, the new exposure time does not take effect in advance. It only becomes effective after the image acquired in the previous mode has been read out. Specifically, the new exposure time may take effect immediately after the image acquired in the previous mode is read out, or it may take effect after a certain switching delay. For example, some modes may not have a switching delay. Other modes may have a switching delay, such as a fixed default delay or a delay of 0.

[0079] During image output, as the camera's usage scenario or ambient brightness changes, new mode parameters can be automatically or passively switched to change the mode. For example... Figure 2 As shown, during the process of the image sensor outputting (i.e., reading out) the Nth frame image, the AP sends mode parameter A1 of mode A to the image sensor, i.e., sends a seamless switching command for mode A. The image sensor receives and loads mode parameter A1. After the image sensor outputs the Nth frame image, a delay is first applied according to the default switching delay time t0 of mode parameter A, and then the exposure time of mode A is applied to expose the N+1th frame image (i.e., the switching frame). Then, based on mode parameter A1, the N+1th frame image, the N+2th frame image, and so on are output. Here, N is greater than 1, meaning that mode switching usually occurs during the processing of the second frame image or subsequent frame images.

[0080] Reference Figure 3 This explains the composition of the frame time of an image. For example... Figure 3 As shown, the frame time of the Nth frame image ( Figure 3In this context, T_N can be the frame start time of the Nth frame of the image. Figure 3 The frame start time from SOF_N to the (N+1)th frame of the image. Figure 3 The time between SOF_N+1 in the Nth frame. Specifically, the frame time T_N of the Nth frame can be calculated in the following two ways.

[0081] In the first approach, the frame time T_N is the sum of the readout time_N and the vertical blanking time_N of the Nth frame. For example, in a no-handover scene, the frame time T_N of the Nth frame is the sum of the readout time and the vertical blanking time of the current mode.

[0082] In the second approach, the frame time T_N is the sum of the readout time_N of the Nth frame, the delay time_N+1 of the (N+1)th frame, and the exposure time_N+1 of the (N+1)th frame. For example, in a scene with a transition, the frame time of the Nth frame is the sum of the readout time of the mode before the transition, the transition delay time of the mode after the transition, and the exposure time of the mode after the transition.

[0083] Similarly, for Figure 3 The descriptions of the frame time T_N-1 of the (N-1)th frame and the frame time T_N+1 of the (N+1)th frame can be found in the description of the frame time T_N of the Nth frame above. For example, Figure 3 The frame time T_N-1 of the (N-1)th frame is equal to the sum of the readout time_N-1 and the vertical blanking time_N-1 of the (N-1)th frame, which is the sum of the readout time and vertical blanking time of mode B. For example, Figure 3 The frame time T_N+1 of the (N+1)th frame is equal to the sum of the readout time_N+1 and the vertical blanking time_N+1 of the (N+1)th frame, which is the sum of the readout time and the vertical blanking time of mode A.

[0084] It is understandable that within the frame time of an image, after the image is read out, during the vertical blanking time of that image, the next image will begin exposure after a switching delay.

[0085] Combination Figure 2 and Figure 3As shown in the example, the frame time T_N of the Nth frame depends on the exposure time of mode A and the default switching delay time t0. In this case, the frame time T_N of the Nth frame is equal to the sum of the readout time of mode B, the exposure time of mode A, and the switching delay time of mode A. The frame time of the (N-1)th frame before the mode switch is equal to the sum of the readout time and the vertical blanking time of mode B. Since the exposure time, readout time, and vertical blanking time of a mode are usually fixed, if the sum of the exposure time of mode A and the default switching delay time t0 of mode A is not equal to the vertical blanking time of mode B, the frame time of the Nth frame will be different from the frame time of the (N-1)th frame, resulting in a frame rate change between the (N-1)th and Nth frames. Similarly, the frame time of the Nth frame may also be different from the frame time of the (N+1)th frame, resulting in a frame rate change between the Nth and (N+1)th frames.

[0086] In some embodiments, when an image frame is output via a MIPI line, the frame time of the image frame can be characterized by the frame structure of the MIPI signal. The MIPI signal can be a level signal, including the following types of signals: (1) rising edge and falling edge, which refer to the voltage jump during high-low level transitions; (2) the duration of the level. Specifically, the duration of the high level in a MIPI signal frame is the readout time, that is, one frame of image is output within the duration of a high level.

[0087] Referring to example 4, for Figure 2 The frame structure of the MIPI signal during the image output process is illustrated. For example... Figure 4 As shown, the frame time of the (N-1)th frame in mode B is T1, the frame time of the Nth frame when switching to mode A is T2, and the frame time of the (N+1)th frame after switching to mode A is T1. At this point, the frame time of the images before and after switching to mode A is T1, and the frame rate is 1 / T1. The frame time of the image when switching to mode A is T2, and the frame rate is 1 / T2. Clearly, T2 is significantly less than T1, meaning that the frame time decreases and the frame rate increases when switching modes.

[0088] As mentioned earlier, to ensure the smoothness of the video stream captured by the camera, it is necessary to keep the frame rate of each video frame constant during the camera's mode switching process to achieve seamless switching between different modes. Therefore, if the frame rate of the Nth frame increases during mode switching, seamless mode switching will not be possible.

[0089] To address the issue of increased frame rate during mode switching, this application provides an image processing method. This image processing method can be applied to electronic devices including image sensors, such as the aforementioned electronic device 100, and specifically to the image sensor (such as a CMOS image sensor).

[0090] In this application, when a seamless mode switching command is detected during the reading of a frame of image based on a mode, the image sensor can determine whether the sum of the default switching delay time and exposure time of the new mode is less than the vertical blanking time of the current mode. If it is determined to be less, it means that the frame time of the current image frame (denoted as the switching frame) will decrease and the frame rate will increase during the mode switching. The image sensor then automatically adjusts the switching delay time of the new mode so that the sum of the adjusted switching delay time and exposure time equals the vertical blanking time of the current mode. This ensures that the frame rate of the switching frame remains constant during mode switching, meaning the frame time of the switching frame is consistent with the frame time of the images before and after the mode switch, achieving seamless mode switching and guaranteeing the continuity and smoothness of the video output by the image sensor.

[0091] It is understandable that when the image sensor automatically detects an increase in the frame rate of the switching frame, it autonomously adjusts the switching delay time of the new mode to ensure that the frame rate of the switching frame remains consistent with the previous and next frames. During this process, the image sensor does not need to interact with the AP via signaling, thus improving the efficiency of adjusting the frame rate of the switching frame during mode switching.

[0092] Optionally, if it is determined that the sum of the default switching delay time and exposure time of the new mode is greater than the vertical blanking time of the current mode, it indicates that the frame time will increase and the frame rate will decrease when switching modes. In this case, the image sensor can obtain the reduced exposure time of the switched mode from the AP so that the frame rate of the switched frame is consistent with the frame rate of the previous frame.

[0093] In other words, in such Figure 3 When SOF_N receives a switching command for mode A, if the default switching delay time _N+1 and exposure time _N+1 in mode A are less than the vertical blanking time _N in mode B, the switching delay time N+1 is increased. If the switching delay time _N+1 and exposure time _N+1 are greater than the vertical blanking time _N in mode B, the exposure time N+1 is decreased, so that T_N, T_N-1, and T_N+1 are the same.

[0094] Figure 5 This is a schematic diagram illustrating the process of an electronic device outputting an image. Among them, Figure 5 and Figure 2 The similarities will not be elaborated upon. Figure 5 The difference is that, Figure 5When switching to mode A, the adjusted switching delay time t1 is applied for a delay before the (N+1)th frame of the image is exposed. At this point, the adjusted switching delay time t1 is greater than the default switching delay time t0 for mode A. In other words, in this application, the image sensor can autonomously decide to increase the switching delay time for the switching frame when switching to mode A, ensuring that the frame time of the switching frame remains consistent with the frame times of other frames.

[0095] Referring to Figure 6, for Figure 5 This is a schematic diagram illustrating the MIPI frame structure during the image output process. (Example:) Figure 6 As shown in the MIPI frame structure, based on the adjusted switching delay time for mode A, the frame time of the (N-1)th frame, the Nth frame, and the N+1th frame are all T1, and the frame rate is 1 / T1. This means that the frame rate remains constant during the switch from mode B to mode A. Thus, seamless switching with a consistent frame rate during mode transitions can be achieved.

[0096] In some embodiments, the image sensor can be a system-on-a-chip (SoC), also known as a camera chip. Specifically, the image sensor in this application may include a delay unit for adjusting the mode switching delay time of the camera. The delay unit can be implemented in software, hardware, or a combination of both, and this application does not impose any specific limitations.

[0097] It's understandable that different image sensors may support multiple exposure modes, such as global shutter and rolling shutter. In global shutter mode, all pixels start and stop exposure simultaneously, suitable for applications requiring high-speed capture of dynamic scenes. In rolling shutter mode, pixels are exposed sequentially in row or column order, suitable for capturing static or slow-moving objects. The exposure mode affects the exposure structure of the image frame, that is, the exposure structure within the chip housing the image sensor.

[0098] Reference Figure 7 This diagram illustrates the exposure structure inside the chip and the frame structure of MIPI during the image output process of an electronic device. Figure 7 The exposure structure inside the chip shown is the exposure structure in the rolling shutter exposure mode. In this case, the start-of-frame (SOF) time of an image frame is the time when the exposure of the first row of pixels in the image frame ends, and the image frame is read out line by line starting from this SOF time.

[0099] first, Figure 7 The image above shows a scene without switching from frame N to frame N+1, in which each frame is processed in mode B.

[0100] exist Figure 7In the exposure structure inside the chip for the scene without switching shown, the Nth frame image starts from the SOF time (i.e. Figure 7 The reading begins at SOF_N and continues for the duration of the readout in mode B (i.e., ...). Figure 7 The readout (shown as readout_B) is completed. The vertical blanking time of the Nth frame image (i.e., Figure 7 Within the shown VB_B), the N+1th frame image is exposed according to the exposure time of mode B (i.e. Figure 7 The exposure is performed line by line (expo_B) as shown, and the SOF time is when the first row of pixels finishes exposure (i.e. Figure 7 Reading begins at SOF_N+1, allowing the N+1th frame of the image to be read out after the readout time of mode B.

[0101] exist Figure 7 In the MIPI frame structure of the handover-free scene shown, the MIPI signal generates a rising edge at SOF_N of the Nth frame, remains high during the readout_B of the Nth frame, and generates a falling edge at the end of the readout_B. Subsequently, the MIPI signal remains low during the vertical blanking time VB_B of the Nth frame, generates a rising edge at SOF_N+1 of the N+1th frame, remains high during the readout_B of the N+1th frame, and generates a falling edge at the end of the readout_B.

[0102] Understandable. Figure 7 The frame time of the Nth frame of the image (i.e. Figure 7 The T shown represents the time between SOF_N and SOF_N+1. Specifically, in Figure 7 In a scene without transitions, all frames have the same frame time and frame rate. At this time, the frame time of the Nth frame (i.e.,...) Figure 7 The T_B shown can be the sum of the readout time of the Nth frame (readout_B) and the vertical blanking time (VB_B) of the Nth frame image, i.e., T_B = readout_B + VB_B.

[0103] Secondly Figure 7 The following shows a switching scenario from frame N to frame N+1, where frame N is processed in mode B and frame N+1 is processed during the switching from mode B to mode A.

[0104] exist Figure 7 In the exposure structure inside the chip shown, which has scene switching, the Nth frame image starts from the SOF time (i.e., Figure 7The reading begins at SOF_B and is completed after the readout time (readout_B) of mode B. Furthermore, within the readout time (readout_B) of the Nth frame image, the chip (i.e., the image sensor) detects a seamless switching command to mode A. This chip can adjust the switching delay time for mode A to obtain an adjusted delay time t1. Then, within the vertical blanking time (VB_B) of the Nth frame image, the adjusted switching delay time (i.e., SOF_B) is first passed... Figure 7 The delay shown is t1), and then the N+1th frame image is exposed according to the exposure time of mode A (i.e., Figure 7 The exposure is performed line by line (expo_A) as shown, starting from the SOF time when the first row of pixels ends (i.e. Figure 7 The SOF_A shown is started to read out, so that the N+1th frame of the image is read out after the readout time of mode A.

[0105] exist Figure 7 In the MIPI frame structure of the no-handover scene shown, the MIPI signal generates a rising edge at SOF_N of the Nth frame, remains high during the readout_B of the Nth frame, and generates a falling edge at the end of the readout_B time. Subsequently, the MIPI signal remains low during the vertical blanking time VB_B of the Nth frame, generates a rising edge at SOF_N+1 of the N+1th frame, remains high during the readout time (denoted as readout_A) of the corresponding mode A in the N+1th frame, and generates a falling edge at the end of the readout_A time.

[0106] Specifically, in Figure 7 The frame time of the Nth frame image in the scene shown is (i.e., Figure 7 The T_A shown can be the sum of the readout time of the Nth frame (readout_B), the switching delay time of the N+1th frame (dealy, the switching delay time of mode A), and the exposure time of the N+1th frame (expo_A), i.e., T_A = readout_B + dealy + expo_A.

[0107] Therefore, to keep the frame time of the Nth frame constant during the switching to mode A, T_B must equal T_A, i.e., readout_B + VB_B = readout_B + dealy + expo_A. Since readout_B and expo_A are fixed, VB_B = T_A = dealy + expo_A. In other words, to ensure the frame time of the Nth frame remains constant, the adjusted switching delay time dealy in mode A can be controlled to satisfy VB_B = dealy + expo_A.

[0108] The following is combined Figures 8A to 8E This paper introduces the application scenarios of the image processing method provided in the embodiments of this application. Specifically, Figures 8A to 8E These are schematic diagrams of a set of user interfaces provided in the embodiments of this application.

[0109] Figure 8A This is a user interface (homepage) on an electronic device 100 that displays icons of installed applications. Figure 8A As shown, the main page displays multiple application icons, such as the "Camera" application icon, which is icon 111. A camera app is an image-taking application used on smartphones, tablets, and other electronic devices.

[0110] like Figure 8A As shown, the electronic device 100 can detect user actions performed on icon 111, such as clicking or long-pressing. In response to these actions, the electronic device 100 can open the camera application, load software parameters such as mode parameters, and display... Figure 8B The user interface shown is Figure 8B The user interface shown can be the default photo-taking interface provided by the camera application.

[0111] It is understood that the electronic device 100 can also open the camera application in other ways. For example, the electronic device 100 can detect the user's voice command "open camera" and trigger the opening of the camera application. This application embodiment does not limit the way of opening the camera application.

[0112] Figure 8B This is the main interface of the "camera" when the "camera" is first run on an electronic device 100. For example... Figure 8B As shown, the user interface may include window 121, camera mode options 122, album shortcut controls 123, shutter controls 124, and camera flip controls 125. Window 121 can be used to display a preview of the image captured in real-time by the camera. The electronic device 100 can refresh the displayed content in real time so that the user can preview the image currently captured by the camera.

[0113] Camera mode option 122 can display one or more shooting mode options, which may include a large aperture mode, night scene mode, still photo mode, portrait mode, video mode, professional shooting mode, and more. When a user operation is detected on a shooting mode option, the electronic device 100 can activate the user-selected shooting mode. Specifically, when a user operation is detected on a more advanced mode option, the electronic device 100 can further display additional shooting mode options, such as a slow-motion shooting mode, thus showcasing richer shooting functions to the user. (Not limited to...) Figure 8B As shown, camera mode option 122 may not display more mode options. Electronic device 100 can detect the user's left / right swipe operation in camera mode option 122 and display other shooting mode options. By default, when electronic device 100 enters the camera application, it can automatically activate the photo mode option. In this case, in response to the user's operation on the camera application icon 111, when electronic device 100 opens the camera application, it can load the mode parameters corresponding to the default photo mode to process the image data captured by the camera based on these mode parameters.

[0114] The photo album shortcut control 123 can be used to open the gallery application. In response to a user action on the photo album shortcut control 123, such as a click, the electronic device 100 can open the gallery application. This allows users to conveniently view captured pictures and videos without having to exit the camera application first and then open the gallery application. The gallery application supports various operations on the pictures and videos stored on the electronic device 100, such as browsing, editing, deleting, and selecting. Additionally, the electronic device 100 can also display thumbnails of saved images in the photo album shortcut control 123.

[0115] The shutter control 124 can be used to detect user actions that trigger shooting. In response to such actions, the electronic device 100 can save the image in window 121 as a picture in a gallery application or start recording video.

[0116] The camera flip control 125 can be used to detect user actions that trigger the camera flipping. In response to this action, the electronic device 100 can flip the camera, for example, switching the rear camera to the front camera. By default, when the electronic device 100 enters the camera application, the electronic device 100 can activate the rear camera and display the image captured by the camera in window 121.

[0117] After the camera application is launched, the electronic device 100 can detect user operation on the shutter control 124, and in response to the operation, the electronic device 100 can display as follows: Figure 8B The user interface shown. At this point, Figure 8B It is the user interface displayed by the electronic device 100 after taking a photo using the rear camera based on the mode parameters of the photo mode.

[0118] In one embodiment of this application, as the ambient brightness changes, the electronic device 100 can automatically adjust its mode parameters, such as adjusting the mode parameters corresponding to the shooting mode, so that the brightness of each frame of continuously acquired images remains consistent, thereby ensuring the image quality of the image output by the electronic device 100. It can be understood that the electronic device 100 displays... Figure 8B Following the image shown in the user interface, the ambient brightness changes as the subject changes and the outdoor light dims. For example, when the electronic device 100 displays... Figure 8B Following the image displayed in the user interface, the electronic device 100 detects a user operation on the shutter control 124. In response to this operation, the electronic device 100 can adjust the mode parameters based on the changed ambient brightness using the image sensor. Simultaneously, it automatically increases the switching delay time as the frame rate increases to maintain a constant frame rate during mode switching, allowing the electronic device 100 to display... Figure 8C The image shown is from the user interface. At this point, Figure 8C It is the user interface displayed after the electronic device 100 takes a photo using the rear camera with adjusted mode parameters based on the photo mode. Figure 8B The images shown and Figure 8C The images shown have consistent brightness and frame rate.

[0119] In another embodiment of this application, as the selected shooting mode changes, the electronic device 100 can adjust the mode parameters through the image sensor to ensure the image quality of the image output by the electronic device 100. For example, as Figure 8D As shown, the electronic device 100 detects a user operation applied to the portrait mode option 122. In response to this operation, the electronic device 100 can switch to the mode parameters corresponding to portrait mode via the image sensor. Simultaneously, it automatically increases the switching delay time as the frame rate increases, so that the frame rate remains constant during the mode switching process. Furthermore, the electronic device 100 can detect a user operation applied to the shutter control 124. In response to this operation, the electronic device 100 can display... Figure 8E The user interface shown. At this point, Figure 8E It is the user interface displayed after the electronic device 100 takes a photo based on the portrait mode parameters using its rear camera, and Figure 8D The images shown and Figure 8E The images shown have consistent brightness and frame rate.

[0120] Next, combined Figures 8A to 8E Example user interface, see Figure 9The image processing method provided in the embodiments of this application will be described. The execution subject of this method can be the image sensor in the camera, and the method includes the following steps:

[0121] S901: Processes the first image data acquired by the camera based on the first mode parameters.

[0122] Specifically, the image sensor can perform photoelectric conversion on the first image light data captured by the lens to obtain the first image data. That is, the aforementioned first image data is the image data generated by the light beam from the lens of the camera, which is then collected by the image sensor and converted into an electrical signal. For example, the first image data is... Figure 7 The image data of the Nth frame of the example.

[0123] It is understood that processing the first image data based on the first mode parameters can enable the first image data to be output according to the output parameters (such as resolution or HDR parameters) indicated by the first mode parameters.

[0124] The processing of the first image data may include: exposing the first image data and reading out (i.e., outputting) the first image data.

[0125] In some embodiments, the first mode parameter can be the mode parameter loaded by the image sensor when the camera is started, or it can be the mode parameter after mode switching after startup. During the image acquisition process, the electronic device can display the acquired image through a user interface.

[0126] For example, refer to Figure 8A As shown in the main interface, the user's action of opening the camera application icon 111 triggers the image sensor in the electronic device 100 to load first mode parameters. Specifically, the first mode parameters can be the mode parameters corresponding to the default shooting mode, and the AP in the processor 110 of the electronic device 100 can detect the opening operation and, in response to the opening operation, send the first mode parameters corresponding to the shooting mode to the image sensor, causing the electronic device 100 to display as shown. Figure 8B The image shown is from the user interface.

[0127] In some embodiments, the camera's mode parameters include at least one of the following: resolution, frame rate, and HDR parameters. Of course, the mode parameters in this application include, but are not limited to, the examples above, and may also include other parameters that affect image quality.

[0128] Specifically, the first mode parameter includes the first frame rate. It can be understood that the frame rate of the previous frame of image data in the video stream containing the first image number, processed based on the first mode parameter, is the first frame rate.

[0129] In some embodiments, the shooting modes corresponding to the camera's mode parameters include at least one of the following: still mode, portrait mode, night mode, large aperture mode, and professional mode. Of course, the shooting modes in this application include, but are not limited to, the examples above, and may also be... Figure 8A , Figure 8E Other modes are shown.

[0130] S902: During the reading of the first image data, a switching instruction to switch the second mode parameters is detected. At this time, the switching instruction can be a seamless switching instruction, that is, the frame rate of the image must remain unchanged during the switching between the first mode parameters and the second mode.

[0131] In some embodiments, the image sensor reads out the first image data, which may be output to an image processing unit such as an ISP or DSP.

[0132] In some embodiments, the image sensor can receive the aforementioned switching command from the AP.

[0133] The image sensor can detect the switching command for the second mode parameter at the SOF time of the first image data. For example, in Figure 6 As shown in the MIPI frame structure, the image sensor can detect the switching command to switch to the second mode parameter at the rising edge of the MIPI signal corresponding to the Nth frame image.

[0134] In some embodiments, the switching instruction can be generated based on changes in ambient brightness, in which case the second mode parameter is determined based on the ambient brightness.

[0135] As an example, an image sensor can detect the brightness of a captured image in real time to reflect the ambient brightness. When the image sensor detects a change in the brightness of the captured image, it can autonomously adjust its mode parameters to optimize the image quality.

[0136] Taking the first image data as the image data of the Nth frame captured by the camera as an example, when the image sensor captures the (N-1)th frame image, it can analyze the pixels of the image data of the (N-1)th frame image to determine the brightness of the image data (denoted as brightness 1). Then, during the acquisition of the Nth frame image, the image sensor can analyze the pixels in the image data of the Nth frame image to determine the brightness of the image data (i.e., brightness 2). If brightness 2 is different from brightness 1, a second mode parameter corresponding to brightness 2 is determined to detect a switching command to switch to the second mode parameter.

[0137] As another example, the AP can detect ambient brightness and, upon detecting a change in ambient brightness, generate a switching command, such as a switching command to a second mode parameter, and send this switching command to the image sensor. After receiving the switching command from the AP, the image sensor can passively initiate the mode switching process. For example, the camera may include an ambient light sensor, which the AP can use to detect whether the ambient brightness has changed.

[0138] In some embodiments, the switching instruction can be generated based on changes in the selected shooting mode. In this case, the second mode parameter is determined based on the selected shooting mode. It can be understood that the selected shooting mode is the one selected by the user through the camera application's user interface.

[0139] For example, refer to Figure 8D As shown in the user interface, the user's selection of the portrait mode option triggers the image sensor in the electronic device 100 to load a second mode parameter, switching the first mode parameter to the second mode parameter. Specifically, the second mode parameter can be the mode parameter corresponding to the portrait mode, and the AP in the processor 110 of the electronic device 100 can detect the selection and, in response to the selection operation, send the second mode parameter corresponding to the shooting mode to the image sensor, causing the electronic device 100 to display as shown. Figure 8E The image shown is from the user interface.

[0140] S903: Determine the difference between the second frame rate and the first frame rate of the second mode parameters.

[0141] At this point, the second frame rate can be any frame rate included in the second mode parameters.

[0142] If it is determined that the second frame rate is greater than the first frame rate, it means that the frame rate of the first image data increases and the frame time decreases during the switching of the second mode parameters, and the process proceeds to S904. If it is determined that the second frame rate is less than the first frame rate, it means that the frame rate of the first image data decreases and the frame time increases during the switching of the second mode parameters, and the process proceeds to S906. Furthermore, if it is determined that the second frame rate is equal to the first frame rate, it means that the frame rate and frame time of the first image data remain unchanged during the switching of the second mode parameters, and the process proceeds to S909.

[0143] The second frame rate is related to the first delay time, which can be the default switching delay time of the second mode parameters. In essence, the second frame rate is the reciprocal of the time frame of the first image data when the second mode parameters are switched. Specifically, when the second mode parameters are switched, the frame time of the first image data is equal to the sum of the readout time of the first mode parameters, the first delay switching time, and the exposure time of the second mode parameters.

[0144] In some embodiments, the image sensor can determine that the first vertical blanking time of the first mode parameter is greater than the second vertical blanking time of the second mode parameter, thereby determining that the second frame rate of the second mode parameter is greater than the first frame rate. In this case, the second vertical blanking time is the sum of the first exposure time and the first switching delay time of the second mode parameter, and the first vertical blanking time is the sum of the exposure time (or third exposure time) and the switching delay time of the first mode parameter.

[0145] In some embodiments, determining the magnitude of the second frame rate and the first frame rate of the second mode parameter in this application can specifically be as follows: determining whether the first vertical blanking time of the first mode parameter is greater than the sum of the first exposure time and the first switching delay time of the second mode parameter. If it is determined to be greater than, then the second frame rate is greater than the first frame rate. If it is determined to be less than, then the second frame rate is less than the first frame rate. If it is determined to be equal to, then the second frame rate is equal to the first frame rate. It can be understood that the first frame rate is related to the vertical blanking time of the first mode parameter, and the second frame rate is related to the sum of the exposure time and the first switching delay time of the second mode parameter.

[0146] S904: Increase the first switching delay time to the second switching delay time so that the second frame rate is equal to the first frame rate.

[0147] In some embodiments, if the first vertical blanking time corresponding to the first mode parameter is greater than the second vertical blanking time of the second mode parameter, the first vertical blanking time is subtracted from the first exposure time of the second mode parameter to calculate the second switching delay time, thereby increasing the first switching delay time to the second switching delay time.

[0148] As an example, refer to Figure 7 For example, assuming the first mode parameter is the mode parameter of mode B and the second mode parameter is the mode parameter of mode A, then the adjusted second delay time is... Figure 7 The adjusted switching delay time is `delay`. At this point, `T_B = T_A = readout_B + VB_B = readout_B + late + expo_A`, that is, `VB_B = late + expo_A`. In other words, the vertical blanking time of the first mode parameter (e.g., `VB_B`) is equal to the adjusted second delay time of the second mode parameter (e.g., `delay`, i.e., the switching delay time `t1`) and the default exposure time (e.g., `expo_A`).

[0149] S905: After the first image data is read out, and after a second switching delay time, the second image data acquired by the camera is processed based on the second mode parameters.

[0150] Specifically, the image sensor can perform photoelectric conversion on the second image light data captured by the lens to obtain the second image data.

[0151] It is understood that processing the second image data based on the second mode parameters can enable the second image data to be output according to the output parameters (such as resolution or HDR parameters) indicated by the second mode parameters.

[0152] The processing of the second image data may include: exposing the second image data and reading out (i.e., outputting) the second image data.

[0153] The second image data can be the data of the next frame of the first image data. For example, referring to the example above, when the first image data is the Nth frame, the second image data is the (N+1)th frame.

[0154] S906: Receives adjustment instructions from the application processing unit (AP).

[0155] Among them, the second frame rate of the second mode parameter is related to the first exposure time and the first switching delay time of the second mode parameter.

[0156] In some embodiments, the above adjustment instructions may include a second exposure time for the second mode parameter. For example, combined with Figure 7 In the example, the first exposure time is expo_A, and the second exposure time is the exposure time adjusted from expo_A.

[0157] In some embodiments, the AP can subtract the first switching delay time from the first vertical blanking time to obtain the second exposure time, so that the adjusted second vertical blanking time is equal to the first vertical blanking time, thereby making the adjusted second frame rate equal to the first frame rate.

[0158] S907: In response to an adjustment command, acquire a second exposure time.

[0159] At this point, the image sensor can reset the first exposure time, that is, reset the exposure time of the image sensor to 0, and then apply the second exposure time.

[0160] S908: After the first image data is read out, and after a first switching delay time, the second image data acquired by the camera is processed based on the second mode parameters.

[0161] At this time, the present application can use a second exposure time to expose the second image data so that the second image data can be output according to the output parameters (such as resolution or HDR parameters) indicated by the second mode parameters.

[0162] S909: After the first image data is read out, and after a first switching delay time, the second image data acquired by the camera is processed based on the second mode parameters.

[0163] At this time, the image sensor can expose the second image data based on the default first exposure time of the second mode parameters, so as to process the second image data based on the second mode parameters, so that the frame rate of the first image data and the second image data remains unchanged during the switching of the second mode parameters.

[0164] It is understood that the above-mentioned S901 to S909 can be executed by the electronic device controlling the interaction between the camera's image sensor and the AP in the processor.

[0165] Thus, during the process of reading the first image data based on the first mode parameters, this application receives a switching instruction for the second switching mode. When it is determined that the second frame rate of the second mode parameters is greater than the first frame rate of the first mode parameters, the first switching delay time related to the second frame rate is adjusted to the second switching delay time, so that the adjusted second frame rate is equal to the first frame rate. Furthermore, after the first image data is read out and the second delay time has elapsed, the second image data captured by the camera can be processed based on the second mode parameters, thereby ensuring that the frame rates of the first image data, the second image data, and the previous frame of the first image data remain consistent.

[0166] In some embodiments, the image sensor in this application processes images based on a mode parameter, specifically based on the gain, exposure time, and readout time of that mode parameter. For example... Figure 10 As shown above, Figure 9 The illustrated S901 includes the following steps:

[0167] S901a: Controls the camera to expose the first image data according to the gain and exposure time of the first mode parameters.

[0168] That is, the image sensor can perform photoelectric conversion on the first image light data captured by the lens to obtain the first image data, and expose the first image data with the gain of the first mode parameter (or the second gain) and the exposure time (or the third exposure time).

[0169] S901b: Read out the first image data according to the readout time of the first mode parameter, and stop reading out the data according to the vertical blanking time of the first mode parameter.

[0170] Combination Figure 7 In the example shown, the readout time of the first mode parameter can be readout_B, and the vertical blanking time can be VB_B. Furthermore, the first image data can be exposed line by line in a rolling shutter exposure mode, and then read out line by line.

[0171] Thus, in this application, the image sensor processes the first image data based on the gain, exposure time, and readout time of the first mode parameters, so that the preview image or the image effect of the captured photo of the first image data conforms to the first mode parameters.

[0172] Furthermore, in some implementations, when an image sensor processes an image based on a mode parameter, the gain of that mode parameter can be determined based on the image's brightness. For example... Figure 11 As shown above, Figure 9 The S905 shown includes the following steps:

[0173] S905a: Acquire the first brightness of the first image data.

[0174] The image sensor can receive the brightness of the first image data from the AP, or it can detect the brightness of the first image data autonomously.

[0175] In some embodiments, the image sensor can obtain a third exposure time and a second gain of a first mode parameter from the AP, and obtain a first exposure time of a second mode parameter, wherein the first brightness is related to the third exposure time and the second gain.

[0176] As an example, this application can characterize the relationship between brightness, gain, and exposure time using the formula AE = shutter * gain. Here, AE represents brightness, shutter represents exposure time, and gain represents gain. For example, the first brightness of the first image is AE1 = shutter1 * gain1, where shutter1 is the exposure time of the first mode parameter (or the third exposure time), and gain1 is the gain of the first mode parameter (or the second gain).

[0177] S905b: Determine the first gain of the second mode parameter based on the first exposure time of the first brightness and the second mode parameter.

[0178] In some embodiments, the image sensor determines a first gain based on a third exposure time, a second gain, and a first exposure time, wherein the first brightness is related to the first exposure time and the first gain. For example, the first gain can be obtained by dividing the product of the third exposure time and the second gain by the first exposure time.

[0179] As an example, this application can calculate the second gain of the second mode parameter using AE = shutter * gain. For instance, assuming the first brightness of the second image data is AE2, and the first exposure time of the second mode parameter is shutter2 (e.g., expo_A), then let AE2 = AE1. The first gain of the second mode parameter, gain2, can be obtained by calculating shutter1 * gain1 = shutter2 * gain2. Here, shutter2 and gain2 are the exposure time and gain that keep the brightness of the second image data consistent with the brightness of the first image data.

[0180] S905c: After the first image data is read out, and after a second switching delay time, the camera is controlled to expose the second image data according to the first gain and first exposure time of the second mode parameters, so that the second image data has a first brightness.

[0181] At this time, the camera's exposure time can be the default exposure time of the second mode parameter, such as the first exposure time.

[0182] Combination Figure 7 In the example shown, the second switching delay time of the second mode parameter can be the adjusted switching delay time delay, i.e., the adjusted switching delay time t1, and the exposure time can be expo_A. Furthermore, the second image data can be exposed line by line according to the rolling shutter exposure mode, and then read out line by line.

[0183] It is understandable that during the vertical blanking time of the first image data, the image sensor can load the second mode parameters and determine the first exposure time, first gain, and second switching delay time of the second mode parameters.

[0184] S905d: Read out the second image data according to the first readout time of the second mode parameters, and stop reading out the data according to the second vertical blanking time of the second mode parameters.

[0185] Thus, when switching modes, the image sensor can calculate the exposure time and gain of the new mode, ensuring that the exposure time and gain maintain the consistency of image brightness before and after the mode switch, thereby guaranteeing the display effect of the video stream containing the image captured by the camera on the display screen.

[0186] Similarly, the above Figure 9The steps in S908 shown are similar to those in S905, and the similarities will not be repeated. The difference is that in S908, after the first image data is read out, the image sensor delays the second image data based on the default first delay time of the second mode parameters, and then exposes the second image data according to the adjusted second exposure time. In this way, the frame rate of each frame can be kept constant during the switching of the second mode parameters by adjusting the exposure time.

[0187] In addition, the above Figure 9 The steps in S909 shown are similar to those in S905, and the similarities will not be repeated. The difference is that in S909, after the first image data is read out, the image sensor delays the second image data based on the default first delay time of the second mode parameters, and then exposes the second image data according to the default first exposure time. In this way, the frame rate of each frame can be kept constant during the switching of the second mode parameters by adjusting the exposure time.

[0188] In some embodiments, combined with Figure 7 Examples, combined Figure 12 The image processing method provided in the embodiments of this application will be described, and the subject executing the method can be an electronic device. Specifically, as shown in the following example... Figure 12 As shown, the method includes the following steps:

[0189] S1201: During the processing of the Nth frame image based on mode B, a seamless handover instruction to switch to mode A is received from the AP. This seamless handover instruction is also called a seamless handover command.

[0190] S1201 can be executed by an electronic device controlling the image sensor, that is, the electronic device controls the image sensor to receive the seamless switching command for mode A from the AP.

[0191] S1202: Obtain information about mode B before switching modes, including the exposure time expo_B, gain_B, readout time readout_B, and vertical blanking time VB_B of mode B.

[0192] In some embodiments, in mode B, the image sensor can expose the Nth frame image according to the exposure time expo_B and the gain gain_B, then read out the Nth frame image according to the readout time readout_B, and stop data reading for a vertical blanking time VB_B.

[0193] S1202 can be executed by an electronic device controlling an image sensor, such as acquiring information about mode B from the AP through the image sensor.

[0194] S1203: Obtain information about mode A after switching modes, including the exposure time expo_A, gain_A, readout time readout_A, and vertical blanking time VB_A of mode A.

[0195] S1203 can be executed by an electronic device controlling the image sensor, such as by controlling the image sensor to obtain information about mode A from the AP.

[0196] S1204: Determine whether VB_B > delay + expo_A is true.

[0197] S1204 can be executed by an electronic device controlling the image sensor, such as determining whether VB_B > delay + expo_A is true through the delay unit in the image sensor.

[0198] S1205: Adjust the delay time so that VB_B = delay + expo_A.

[0199] S1205 can be executed by an electronic device controlling the image sensor, such as by adjusting the delay time through a delay unit in the image sensor.

[0200] When VB_B > delay + expo_A, the image sensor can increase the delay so that VB_B = delay + expo_A.

[0201] Furthermore, this application can calculate gain_A by expo_B*gain_B = expo_A*gain_A. In this case, expo_B, gain_B, and expo_A can be known parameters, and the unknown parameter gain_A can be calculated using these known parameters.

[0202] In some embodiments, S1001 to S1005 described above can be performed during the reading of the Nth frame image data.

[0203] S1206: Switch to mode A in the N+1 frame image, process the N+1 frame image based on mode A, so that the frame rate from the Nth frame image to the N+1 frame image remains unchanged.

[0204] S1205 can be executed by an electronic device controlling the image sensor, such as delaying the exposure of the N+1th frame image by the image sensor according to the adjusted delay time.

[0205] At this point, the image sensor processes the (N+1)th frame image based on mode A, according to the adjusted delay and the default expo_A.

[0206] In some embodiments, in mode A, the image sensor can expose the N+1th frame image according to the exposure time expo_A and the gain gain_A, then read out the Nth frame image according to the readout time readout_B, and stop data reading for a vertical blanking time VB_B.

[0207] Specifically, S1001 to S1006 can be executed by the image sensor during the reading of the Nth frame of image data.

[0208] S1207: Process information of mode A through AP.

[0209] S1206 can be executed by the AP controlled by an electronic device, such as by the AP interacting with the image sensor to reduce the exposure time of the N+1th frame image.

[0210] For example, when VB_B < delay + expo_A, the AP can reduce expo_A so that VB_B = delay + expo_A. Then, the adjusted expo_A is sent to the image sensor so that the image sensor processes the (N+1)th frame image based on the default delay and the adjusted expo_A.

[0211] Furthermore, when it is determined that VB_B equals delay + expo_A, the image sensor processes the N+1th frame image based on mode A according to the default delay and expo_A.

[0212] In some embodiments, when the image sensor determines that VB_B < delay + expo_A, it can return feedback information to the AP, indicating that the exposure time and switching delay time of mode A may experience a decrease in frame rate and an increase in frame time. Subsequently, the AP is triggered to reduce the exposure time of mode A and then send the adjusted exposure time back to the image sensor.

[0213] Thus, in this application, the image sensor can automatically adjust the delay of mode A when VB_B > delay + expo_A, so as to ensure that the frame rate of the Nth frame image after switching modes remains unchanged, thereby improving the smoothness of the video stream.

[0214] The following describes the layered architecture involved in this application, that is, the layered architecture of the electronic device in which the image sensor is located.

[0215] In this embodiment, the aforementioned electronic device is an electronic device capable of running an operating system and installing applications. Optionally, the operating system running on the electronic device can be a system, a system, a system, etc. For example, the software system of the aforementioned electronic device 100 can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. This embodiment of the invention uses the layered architecture Android system as an example to exemplify the software structure of the electronic device 100.

[0216] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through interfaces. In some embodiments, the Android system may include an application layer, an application framework layer, an Android runtime and system libraries, a hardware abstraction layer (HAL), and a kernel layer. It should be noted that this application uses the Android system as an example; however, the solution can also be implemented in other operating systems (such as iOS), as long as the functions implemented by each module are similar to those in the embodiments of this application.

[0217] The application layer can include a series of application packages.

[0218] like Figure 13 As shown, the application package may include applications such as camera app, gallery, calendar, calling, maps, navigation, wireless local area networks (WLAN), Bluetooth, music, video, SMS, settings, etc.

[0219] Of course, the application layer may also include other application packages, such as payment applications, shopping applications, banking applications, or chat applications, etc., which are not limited in this application.

[0220] In this embodiment, an application with shooting capabilities, such as a camera application, can be installed in the application layer. The camera application has the functions of taking photos and videos. Of course, other applications can also call the camera application to perform shooting when they need to use the shooting function.

[0221] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. These may include, for example, an activity manager, a window manager, a content provider, a view system, a resource manager, a notification manager, and a camera service, etc., though this application embodiment does not impose any limitations on these.

[0222] The Camera Service can start during the power-on phase of an electronic device. During operation, the Camera Service can interact with the Camera HAL in the Hardware Abstraction Layer (HAL).

[0223] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (such as OpenGL ES), 2D graphics engines (such as SGL), etc.

[0224] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0225] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0226] OpenGL ES is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0227] SGL is a 2D graphics engine.

[0228] The Android Runtime comprises the core libraries and the virtual machine. The Android Runtime is responsible for scheduling and managing the Android system. The core libraries consist of two parts: one part contains the functionalities that Java calls, and the other part contains the core Android libraries. The application layer and application framework layer run in the virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0229] The HAL layer is a wrapper around Linux kernel drivers, providing interfaces to the upper layers and shielding them from the implementation details of the underlying hardware.

[0230] The HAL layer can include Wi-Fi HAL, audio HAL, camera HAL, decision module, etc.

[0231] The Camera HAL is the core software framework for the camera, responsible for interacting with the hardware devices (such as cameras) that enable shooting in electronic devices. The Camera HAL hides the implementation details of the relevant hardware devices (such as specific image processing algorithms) while providing the Android system with interfaces to access these hardware devices.

[0232] The kernel layer is the layer between hardware and software. The kernel layer may include display drivers, camera drivers, audio drivers, sensor drivers, etc. The camera driver is the driver layer for the camera device, primarily responsible for interaction with the hardware. The hardware layer includes displays, cameras, etc. A camera may include an image sensor. In some embodiments, the image sensor may include a software- or hardware-implemented delay unit. Specifically, in this application, the image sensor is used to adaptively adjust the switching delay time of the new mode when switching the current camera mode, to ensure that the output frame rate of the camera remains unchanged before and after the mode switch.

[0233] In some embodiments, the camera may include, for example, a wide-angle camera (also referred to as a wide-angle camera module or wide-angle lens), a telephoto camera (also referred to as a telephoto camera module or telephoto lens), an ultra-wide-angle camera (also referred to as an ultra-wide-angle camera module or ultra-wide-angle lens), a monochrome camera (also referred to as a monochrome camera module or monochrome lens), a macro camera (also referred to as a macro camera module or macro lens), a computer vision (CV) camera (also referred to as a CV camera module or CV lens), etc.

[0234] It should be noted that this application's embodiments use taking a photograph as an example to describe the image processing method provided in this application. It is understood that the image processing method provided in this application's embodiments is also applicable to the video recording function; the specific implementation can be found in the photographing function, and will not be described in detail in this application's embodiments.

[0235] Some embodiments of this application provide an image sensor for performing various functions or steps performed by the image sensor in the above-described method embodiments, such as performing the above-described... Figures 9 to 11 The method in the middle.

[0236] For example, the image sensor described above includes a delay unit used to adjust the switching delay time of the image sensor's mode parameters. The image sensor can be a CMOS sensor.

[0237] Some embodiments of this application provide a camera that includes the image sensor described above.

[0238] like Figure 14As shown, this application also provides a chip system 1400, which may include an AP 1401 and an image sensor 1402. The AP 1401 is used to send software parameters such as camera mode parameters to the image sensor, and to send a switching command to the image sensor to switch the camera mode parameters. The image sensor 1402 is used to execute the above... Figures 9 to 11 The method described above ensures that the frame rate remains unchanged before and after the mode switching. Of course, this chip system may also include other discrete components, but this application does not specifically limit this.

[0239] This application provides an electronic device in some embodiments, which may include a memory, multiple cameras, and one or more processors. The cameras, memory, and processors are coupled. The memory stores computer program code, which includes computer instructions. When the processor executes the computer instructions, the electronic device can perform various functions or steps performed by the electronic device in the above method embodiments. The structure of the electronic device can be referred to... Figure 1 The structure of the electronic device 100 shown.

[0240] The various embodiments of the mechanisms disclosed in this application can be implemented in hardware, software, firmware, or a combination of these implementation methods. Embodiments of this application can be implemented as computer programs or program code executable on a programmable system, the programmable system including at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0241] Program code can be applied to input instructions to execute the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, the processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application-specific integrated circuit (ASIC), or a microprocessor.

[0242] The program code can be implemented using a high-level procedural language or an object-oriented programming language to communicate with the processing system. Assembly language or machine language can also be used when needed. In fact, the mechanisms described in this application are not limited to any particular programming language. In either case, the language can be a compiled language or an interpreted language.

[0243] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored thereon on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, the instructions may be distributed via a network or through other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine-readable (e.g., computer-readable) form, including but not limited to floppy disks, optical disks, CD-ROMs, magneto-optical disks, read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic cards or optical cards, flash memory, or tangible machine-readable storage for transmitting information (e.g., carrier waves, infrared signals, digital signals, etc.) using the Internet in the form of electrical, optical, acoustic, or other propagation signals. Therefore, machine-readable media include any type of machine-readable medium suitable for storing or transmitting electronic instructions or information in a machine-readable (e.g., computer-readable) form.

[0244] In the accompanying drawings, some structural or methodological features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be necessary. Rather, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. Furthermore, the inclusion of structural or methodological features in a particular figure does not imply that such features are required in all embodiments, and in some embodiments, these features may be omitted or may be combined with other features.

[0245] It should be noted that all units / modules mentioned in the device embodiments of this application are logical units / modules. Physically, a logical unit / module can be a physical unit / module, a part of a physical unit / module, or a combination of multiple physical units / modules. The physical implementation of these logical units / modules themselves is not the most important factor; the combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed in this application. Furthermore, to highlight the innovative aspects of this application, the above-described device embodiments of this application have not introduced units / modules that are not closely related to solving the technical problems proposed in this application. This does not mean that the above-described device embodiments do not contain other units / modules.

[0246] It should be noted that in the examples and description of this patent, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0247] Although this application has been illustrated and described with reference to certain preferred embodiments thereof, those skilled in the art should understand that various changes in form and detail may be made thereto without departing from the spirit and scope of this application.

Claims

1. An image processing method, characterized in that, The method includes: The first image data is obtained by photoelectric conversion of the first image light data captured by the lens by the image sensor. The image sensor and the lens are located in the camera of the electronic device. The first image data is processed based on a first mode parameter, the first mode parameter including a first frame rate; During the reading of the first image data, a switching command to switch the second mode parameters was detected; Corresponding to the first vertical blanking time being greater than the second vertical blanking time, it is determined that the second frame rate of the second mode parameter is greater than the first frame rate. The second frame rate is related to the first switching delay time of the second mode parameter. The second vertical blanking time is the sum of the first exposure time and the first switching delay time of the second mode parameter. The first vertical blanking time is the sum of the exposure time and the switching delay time of the first mode parameter. Since the first vertical blanking time is greater than the second vertical blanking time, the first vertical blanking time is subtracted from the first exposure time to calculate the second switching delay time. After the first image data is read out and after the second switching delay time, the second image light data acquired by the lens is photoelectrically converted to obtain the second image data. The second image data is processed based on the second mode parameters. Wherein, the second switching delay time is greater than the first switching delay time, and the second frame rate corresponding to the second switching delay time is equal to the first frame rate.

2. The method according to claim 1, characterized in that, The mode parameters include at least one of the following: output size, working clock, resolution, frame rate, and high dynamic range (HDR) parameters.

3. The method according to claim 1, characterized in that, The processing of the second image data based on the second mode parameters includes: The second image data is exposed according to the first gain and the first exposure time of the second mode parameters, and the second image data is read out according to the first readout time of the second mode parameters.

4. The method according to claim 1, characterized in that, The detected switching instruction for switching the second mode parameters includes: Receive the switching instruction from the application processor AP to switch the second mode parameter; The second mode parameter is determined based on the ambient brightness. Alternatively, the second mode parameter is determined based on the selected shooting mode.

5. The method according to claim 1, characterized in that, The method further includes: If it is determined that the second frame rate of the second mode parameter is less than the first frame rate, an adjustment instruction is received from the AP. The adjustment instruction includes the second exposure time of the second mode parameter, and the second vertical blanking time of the second exposure time is equal to the first vertical blanking time. After the first image data is read out and the first switching delay time has elapsed, the second image light data acquired by the lens is photoelectrically converted to obtain the second image data. The second image data is exposed according to the first gain and the second exposure time of the second mode parameters, and the second image data is read out according to the first readout time of the second mode parameters.

6. The method according to claim 5, characterized in that, The second exposure time is equal to the first vertical blanking time minus the first switching delay time.

7. The method according to claim 3, characterized in that, Both the first image data and the second image data have a first brightness, and, The method further includes: Obtain the third exposure time and the second gain of the first mode parameters, and obtain the first exposure time of the second mode parameters, wherein the first brightness is related to the third exposure time and the second gain; The first gain is determined based on the third exposure time, the second gain, and the first exposure time, and the first brightness is related to the first exposure time and the first gain.

8. The method according to claim 7, characterized in that, Determining the first gain based on the third exposure time, the second gain, and the first exposure time includes: The first gain is obtained by dividing the product of the third exposure time and the second gain by the first exposure time.

9. An image sensor, characterized in that, The image sensor is used to perform photoelectric conversion on the image light data captured by the lens to obtain image data. The image sensor and the lens are located in the camera of the electronic device. The image sensor includes a delay unit, which is used to adjust the switching delay time of the mode parameters of the image sensor. The image sensor is specifically used to perform the image processing method as described in any one of claims 1 to 8 on the converted image data.

10. The image sensor according to claim 9, characterized in that, The image sensor is a complementary metal-oxide-semiconductor (CMOS) sensor.

11. A camera, characterized in that, The camera includes a lens and an image sensor as described in claim 9 or 10.

12. A chip system, characterized in that, The chip system includes an application processor (AP) and an image sensor; The AP is used to send a switching instruction for the camera's mode parameters to the image sensor, and the image sensor is used to execute the image processing method as described in any one of claims 1 to 8.

13. An electronic device, characterized in that, include: Processor, memory, and camera; The memory is used to store instructions executed by one or more processors of the electronic device; The processor is one of the processors of the electronic device, and is used to send mode parameter switching instructions to the camera; The camera is used to perform the image processing method according to any one of claims 1 to 8.

Images