Image processing method, hybrid image sensor, electronic device, and storage medium

Abstract

Embodiments of the present application disclose an image processing method, a hybrid image sensor, an electronic device and a computer readable storage medium, which are used for introducing an event camera into an electronic device in the form of a hybrid image sensor, improving the image capability of the electronic device through event data output by the event camera, and improving the image experience of a user. The electronic device includes a main chip, a first camera and a second camera, the main chip is in communication connection with the first camera and the second camera respectively; the second camera includes a hybrid image sensor, and the hybrid image sensor includes a target pixel part and an event pixel part. The main chip is used for acquiring first image data output by the first camera, acquiring second image data and event data output by the second camera; and performing image display and / or image storage according to at least one of the first image data, the second image data and the event data.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image technology, and in particular to an image processing method, a hybrid image sensor, an electronic device, and a computer readable storage medium. BACKGROUND

[0002] With the continuous development of image technology and the wide application of electronic devices, more and more users take pictures and videos through the image system of electronic devices.

[0003] At present, the camera of an electronic device is usually a traditional camera (for example, an RGB camera). However, the image capability of the traditional camera in some scenes (for example, a motion scene) is poor, thereby causing poor image experience of the user. SUMMARY

[0004] The embodiments of the present application provide an image processing method, a hybrid image sensor, an electronic device, and a computer readable storage medium, which can improve the image capability of the electronic device and improve the image experience of the user.

[0005] In a first aspect, the embodiments of the present application provide an electronic device, which includes a main chip, a first camera, and a second camera. The second camera includes a hybrid image sensor, and the hybrid image sensor includes a target pixel part and an event pixel part.

[0006] The first camera is configured to output first image data. The second camera is configured to output second image data through the target pixel part and output event data through the event pixel part.

[0007] The main chip is configured to acquire the first image data, the second image data, and the event data, and perform image display and / or image storage according to at least one of the first image data, the second image data, and the event data.

[0008] As can be seen from the above, the embodiments of the present application indirectly introduce an event camera into an electronic device in the form of a hybrid image sensor, add event data output by the event camera to the electronic device, thereby improving the image capability of the electronic device and improving the image experience of the user. In addition, compared with directly introducing an event camera into an electronic device, introducing an event camera in the form of a hybrid image sensor has less impact on the existing scheme and is easier to implement, because the second image data output by the hybrid sensor can be used for related calculations such as focusing, electronic anti-shake, and optical anti-shake, and then the lens is moved, without the need for additional calculations on the event data such as focusing and anti-shake.

[0009] In a possible implementation of the first aspect, the first camera is a front main camera of the electronic device, and the second camera is a front auxiliary main camera of the electronic device; or the first camera is a rear main camera of the electronic device, and the second camera is a rear auxiliary main camera, a wide-angle camera or a long-focus camera of the electronic device.

[0010] Further, the implementation can overcome or reduce the limitations of the Hybrid sensor by using the main camera and the Hybrid sensor in combination. For example, the Hybrid sensor is used as an auxiliary main camera, a wide-angle camera or a long-focus camera in combination with the main camera, which does not significantly reduce the image imaging quality of the electronic device; the Hybrid sensor has a low resolution requirement, thereby reducing the manufacturing cost; the amount of event data is small, which can be realized under the bandwidth constraint of the camera of the electronic device, and the implementation difficulty is low.

[0011] In a possible implementation of the first aspect, the main chip is specifically configured to: perform binocular registration on the first image data and the second image data to obtain a registration matrix; perform registration on the first image data and the event data according to the registration matrix to obtain registered first image data and registered event data; and perform image display and / or image storage according to the registered first image data and the registered event data. In this way, the first image data and the event data are registered in the two-stage "bridging" registration manner, so that the registration deviation is smaller.

[0012] In a possible implementation of the first aspect, the first camera is a rear main camera, and the second camera is a wide-angle camera; and if the electronic device works in a main camera focal length range, the main chip is specifically configured to:

[0013] According to the field of view angle of the first image data, the field of view angle of the second image data is cropped to obtain cropped second image data; the event data is processed by using a preset wide-angle image processing algorithm to obtain first processed event data; according to the field of view angle of the first image data, the field of view angle of the first processed event data is cropped to obtain cropped event data; the cropped second image data and the first image data are binocularly registered to obtain a registration matrix; and the cropped event data and the first image data are registered according to the registration matrix to obtain registered first image data and registered event data.

[0014] In a possible implementation of the first aspect, if the electronic device works in a wide-angle focal length range, the main chip is further configured to: perform wide-angle distortion correction on the second image data to obtain corrected second image data; process the event data by using a preset wide-angle image processing algorithm to obtain second processed event data; and perform image display and / or image storage according to the second processed event data and the corrected second image data.

[0015] In a possible implementation manner of the first aspect, the first camera is a front main camera, and the second camera is a front auxiliary main camera; if the electronic device works in a wide-angle focal length range, the main chip is specifically configured to:

[0016] perform wide-angle distortion correction on the first image data to obtain processed first image data; perform wide-angle distortion correction on the second image data to obtain processed second image data; process the event data by using a preset wide-angle image processing algorithm to obtain third processed event data; perform binocular registration on the processed second image data and the processed first image data to obtain a registration matrix; and perform registration on the third processed event data and the processed first image data according to the registration matrix to obtain registered first image data and registered event data.

[0017] In a possible implementation manner of the first aspect, the main chip is specifically configured to: generate a first stylized video or a first stylized picture according to the registered first image data and the registered event data, and display the first stylized video or the first stylized picture. In this implementation manner, the event data is superimposed on the first image data to form a stylized picture or video, which provides more camera play for users and provides a better image experience for users.

[0018] In a possible implementation manner of the first aspect, the main chip is specifically configured to: generate a second stylized video or a second stylized picture according to the second image data and the event data, and display the second stylized video or the second stylized picture; or generate a third stylized video or a third stylized picture according to the event data, and display the third stylized video or the third stylized picture.

[0019] In this implementation manner, not only can the event data be superimposed on the second image data to form a stylized picture or video, but also the stylized video or picture can be directly formed according to the event data, which provides more camera play for users and provides a better image experience for users.

[0020] In a possible implementation manner of the first aspect, the target pixel part includes a target pixel array and a target pixel readout circuit; and the event pixel part includes an event pixel array and an event pixel readout circuit.

[0021] The target pixel readout circuit is configured to perform exposure operation and readout operation on the target pixel array to obtain the second image data; and the event pixel readout circuit is configured to perform readout operation on the event pixel array to obtain the event data.

[0022] In a possible implementation manner of the first aspect, the hybrid image sensor further comprises a pull-up circuit, the target pixel readout circuit comprises a frame readout controller, and the event pixel readout circuit comprises a time stamping circuit.

[0023] The frame readout controller is connected with the pull-up circuit through a hardware connection line, and the pull-up circuit is connected with the time stamping circuit through the hardware connection line.

[0024] The frame readout controller is further configured to output a frame synchronization signal.

[0025] The pull-up circuit is configured to receive the frame synchronization signal, perform voltage pull-up on the frame synchronization signal, and output a hard synchronization signal.

[0026] The time stamping circuit is further configured to receive the hard synchronization signal, and print an additional time stamp and output the additional time stamp under the triggering of the hard synchronization signal.

[0027] In the implementation manner, the frame readout controller is used to output the frame synchronization signal as a standard time point of the hard synchronization, so that the hard synchronization alignment of the second image data and the event data is realized in the hybrid sensor, which can further reduce the image registration deviation, and the stability of the peripheral circuit of the hybrid sensor is better, and the time synchronization effect is better and the time synchronization rate is faster because the same sensor clock signal is used in the target pixel part and the event pixel part.

[0028] In a possible implementation manner of the first aspect, the event pixel readout circuit is further configured to: embed the additional time stamp in an asynchronous event data stream in time sequence, or embed the additional time stamp in a synchronous event frame data stream; and the event data is the asynchronous event data stream or the synchronous event frame data stream. In this way, the additional time stamp can be transmitted to the main chip along with the event data stream, so that the additional time stamp can be transmitted to the main chip more timely, and the waiting time of the main chip is reduced.

[0029] In a possible implementation manner of the first aspect, the main chip is specifically configured to: receive the asynchronous event data stream or the synchronous event frame data stream; and perform time alignment on the event data and the second image data according to the additional time stamp in the asynchronous event data stream or the synchronous event frame data stream.

[0030] In a possible implementation manner of the first aspect, if the exposure mode of the target pixel array is global exposure and the readout mode of the event pixel array is asynchronous readout, the frame readout controller is specifically configured to: output one frame synchronization signal at the exposure start time and the exposure end time respectively for each frame of second image data.

[0031] If the exposure mode of the target pixel array is local exposure and the readout mode of the event pixel array is asynchronous readout, the frame readout controller is specifically configured to: output one frame synchronization signal at the exposure end time of the first row of target pixels for each frame of second image data.

[0032] In a possible implementation of the first aspect, the exposure mode of the target pixel array is global exposure, and the readout mode of the event pixel array is synchronous readout; the frame readout controller is specifically configured to: output one frame synchronization signal at the exposure start time and the exposure end time for each frame of second image data.

[0033] In a possible implementation of the first aspect, the exposure mode of the target pixel array is global exposure, and the readout mode of the event pixel array is synchronous readout;

[0034] The frame readout controller is specifically configured to: output one frame synchronization signal at the exposure start time and the exposure end time for each frame period, or output one frame synchronization signal at the exposure start time, the frame period being a time period from the exposure start time of the current frame to the exposure start time of the next frame.

[0035] The event pixel readout circuit is further configured to: resynchronize the first frame event frame in the frame period with the hard synchronization signal corresponding to the exposure start time, and embed the additional time stamp corresponding to the exposure start time into the time stamp of the first frame event frame in the frame period.

[0036] In a possible implementation of the first aspect, the exposure mode of the target pixel array is local exposure, and the readout mode of the event pixel array is synchronous readout; the frame readout controller is specifically configured to: output one frame synchronization signal at a preset time for each frame of second image data, the preset time being the exposure end time of the first row of target pixels or the start readout time of the first row of target pixels.

[0037] In a possible implementation of the first aspect, the exposure mode of the target pixel array is local exposure, and the readout mode of the event pixel array is synchronous readout;

[0038] The frame readout controller is specifically configured to: output one frame synchronization signal at the exposure end time of the first row of target pixels for each frame period, the frame period being a time period from the exposure end time of the first row of target pixels of the current frame to the exposure end time of the first row of target pixels of the next frame.

[0039] The event pixel readout circuit is further configured to: resynchronize the first frame event frame in the frame period with the hard synchronization signal corresponding to the exposure end time of the first row of target pixels, and embed the additional time stamp corresponding to the exposure end time of the first row of target pixels into the time stamp of the first frame event frame in the frame period.

[0040] In a possible implementation of the first aspect, if there are at least two frames of second image data in one frame period, the frame readout controller is specifically configured to: for each frame of second image data, output one frame synchronization signal at the exposure start time and the exposure end time respectively, or output one frame synchronization signal at the exposure end time of the first row of target pixels.

[0041] In a possible implementation of the first aspect, the host chip is further configured to: issue a control signal to the hybrid image sensor, the control signal comprising the first readout parameter and the second readout parameter, the first readout parameter comprising the exposure mode and the exposure time, and the second readout parameter comprising the event readout mode.

[0042] The hybrid image sensor is further configured to: receive the control signal, parse the control signal to obtain the first readout parameter and the second readout parameter; control the target pixel readout circuit to perform the exposure operation and the readout operation according to the first readout parameter, to obtain the second image data, and control the frame readout controller to output the frame synchronization signal; and control the event pixel readout circuit to perform the readout operation according to the second readout parameter, to obtain the event data and the additional time stamp.

[0043] In this implementation, the host chip can further control the working process and output result of the Hybrid sensor by issuing sensor setting parameters. For example, by issuing sensor setting parameters, the host chip controls the frame readout controller to issue the frame synchronization signal at corresponding time under different exposure modes and different readout modes, so as to realize the hard synchronization of the second image data and the event data in the Hybrid sensor under various conditions.

[0044] In a possible implementation of the first aspect, the host chip is further configured to: calculate the sensor setting parameters to be issued according to the second image data and the event data, and generate the control signal according to the sensor setting parameters to be issued.

[0045] In a possible implementation of the first aspect, if the exposure mode of the target pixel array is local exposure and the readout mode of the event pixel array is synchronous readout.

[0046] The host chip is further configured to: if the switching operation is detected, generate a switching control signal in response to the switching operation, and issue the switching control signal to the hybrid image sensor, the switching operation being used to indicate switching from the video mode to the photographing mode, and the switching control signal comprising the sensor setting parameters in the photographing mode.

[0047] The hybrid image sensor is also configured to receive the switching control signal, analyze the switching control signal, and obtain sensor setting parameters in the photographing mode; according to the sensor setting parameters, output a frame synchronization signal at an exposure start time of a first row of target pixels of each frame of second image data and at an exposure end time of a last row of target pixels, to control the time stamping circuit to print an extra time stamp at the exposure start time of the first row of target pixels and at the exposure end time of the last row of target pixels in response to the hard synchronization signal, and embed the extra time stamp into a next frame of event frames.

[0048] In this implementation, the internal hard synchronization mode of the Hybrid sensor in different application modes is different. For example, in the video mode and the photographing mode, when the application mode is switched, the sensor setting parameters can be issued to control the mode of the internal hard synchronization of the Hybrid sensor, so that the hard synchronization effect is better, and the user image experience is better.

[0049] In a possible implementation of the first aspect, the switching control signal is issued within an exposure time of the nth frame of second image data and takes effect within an exposure time of the (n+2)th frame of second image data.

[0050] In a second aspect, an embodiment of the present application provides a hybrid image sensor, including a target pixel array, a target pixel readout circuit, an event pixel array, an event pixel readout circuit, and a pull-up circuit.

[0051] The target pixel readout circuit includes a frame readout controller, and the event pixel readout circuit includes a time stamping circuit.

[0052] The frame readout controller is connected to the pull-up circuit through a hardware connection line, and the pull-up circuit is connected to the time stamping circuit through the hardware connection line.

[0053] The target pixel readout circuit is configured to perform exposure and readout operations on the target pixel array and output second image data.

[0054] The event pixel readout circuit is configured to perform readout operations on the event pixel array and output event data.

[0055] The frame readout controller is configured to output a frame synchronization signal.

[0056] The pull-up circuit is configured to receive the frame synchronization signal, perform voltage pull-up on the frame synchronization signal, and output a hard synchronization signal.

[0057] The time stamping circuit is also configured to receive the hard synchronization signal, print an extra time stamp under the triggering of the hard synchronization signal, and output the extra time stamp.

[0058] It can be seen from the above that, by implementing the hard synchronization alignment of the second image data and the event data inside the Hybrid sensor, the embodiment of the application can not only further reduce the image registration deviation, but also make the peripheral circuit stability of the Hybrid sensor better, and since the target pixel part and the event pixel part use the same sensor clock signal, the time synchronization effect is better and the time synchronization rate is faster.

[0059] In a possible implementation of the second aspect, the event pixel readout circuit is further configured to: embed the additional time stamp in a time sequence in an asynchronous event data stream, or embed the additional time stamp in a synchronous event frame data stream; and the event data is the asynchronous event data stream or the synchronous event frame data stream. In this way, the additional time can be transmitted to the host chip along with the event data stream, so that the additional time stamp can be transmitted to the host chip more timely, reducing the waiting time of the host chip.

[0060] In a possible implementation of the second aspect, if the exposure mode of the target pixel array is global exposure and the readout mode of the event pixel array is asynchronous readout, the frame readout controller is specifically configured to: for each frame of second image data, output one frame synchronization signal at the exposure start time and the exposure end time, respectively.

[0061] If the exposure mode of the target pixel array is local exposure and the readout mode of the event pixel array is asynchronous readout, the frame readout controller is specifically configured to: for each frame of second image data, output one frame synchronization signal at the exposure end time of the first row of target pixels.

[0062] In a possible implementation of the second aspect, the exposure mode of the target pixel array is global exposure and the readout mode of the event pixel array is synchronous readout; and the frame readout controller is specifically configured to: for each frame of second image data, output one frame synchronization signal at the exposure start time and the exposure end time, respectively.

[0063] In a possible implementation of the second aspect, the exposure mode of the target pixel array is global exposure and the readout mode of the event pixel array is synchronous readout.

[0064] The frame readout controller is specifically configured to: for each frame period, output one frame synchronization signal at the exposure start time and the exposure end time, respectively, or output one frame synchronization signal at the exposure start time, and the frame period is a time period from the exposure start time of the current frame to the exposure start time of the next frame.

[0065] The event pixel readout circuit is further configured to: resynchronize the first frame event frame in the frame period with the hard synchronization signal corresponding to the exposure start time, and embed the additional time stamp corresponding to the exposure start time into the first frame event frame in the frame period.

[0066] In a possible implementation manner of the second aspect, the exposure mode of the target pixel array is local exposure, and the readout mode of the event pixel array is synchronous readout; the frame readout controller is specifically configured to: output a frame synchronization signal at the exposure end time of the first row of target pixels for each frame of second image data.

[0067] In a possible implementation manner of the second aspect, the exposure mode of the target pixel array is local exposure, and the readout mode of the event pixel array is synchronous readout;

[0068] The frame readout controller is specifically configured to: output a frame synchronization signal at the exposure end time of the first row of target pixels for each frame period, the frame period being a time period between the exposure end time of the first row of target pixels of the current frame and the exposure end time of the first row of target pixels of the next frame;

[0069] The event pixel readout circuit is further configured to: resynchronize the first frame of event frames in the frame period with the hard synchronization signal corresponding to the exposure end time of the first row of target pixels, and embed the additional time stamp corresponding to the exposure end time of the first row of target pixels into the time stamp of the first frame of event frames in the frame period.

[0070] In a possible implementation manner of the second aspect, if there are at least two frames of second image data in a frame period, the frame readout controller is specifically configured to: output a frame synchronization signal at the exposure start time and the exposure end time for each frame of second image data, or output a frame synchronization signal at the exposure end time of the first row of target pixels.

[0071] In a possible implementation manner of the second aspect, the hybrid image sensor is further configured to: receive a control signal from the main chip, analyze the control signal to obtain the first readout parameter and the second readout parameter; control the target pixel readout circuit to perform the exposure operation and the readout operation according to the first readout parameter, to obtain the second image data, and control the frame readout controller to output the frame synchronization signal; and control the event pixel readout circuit to perform the readout operation according to the second readout parameter, to obtain the event data and the additional time stamp; wherein the first readout parameter includes the exposure mode and the exposure time, and the second readout parameter includes the event readout mode.

[0072] In a possible implementation manner of the second aspect, the exposure mode of the target pixel array is local exposure, and the readout mode of the event pixel array is synchronous readout; the hybrid image sensor is further configured to:

[0073] receive a switching control signal from the main chip, the switching control signal being a signal generated by the main chip after detecting a switching operation, the switching operation being used to indicate switching from a video mode to a photographing mode, and the switching control signal including a sensor setting parameter in the photographing mode;

[0074] The switching control signal is parsed to obtain sensor setting parameters in the photographing mode;

[0075] According to the sensor setting parameters, a frame synchronization signal is output at the exposure start time of the first row of target pixels of each frame of second image data and at the exposure end time of the last row of target pixels by controlling the frame readout controller, to control the time stamping circuit to print an extra time stamp at the exposure start time of the first row of target pixels and at the exposure end time of the last row of target pixels in response to the hard synchronization signal, and embed the extra time stamp into the next frame of event frame.

[0076] In a possible implementation of the second aspect, the switching control signal is issued within the exposure time of the nth frame of second image data and takes effect within the exposure time of the n+2th frame of second image data.

[0077] In a third aspect, an embodiment of the present application provides an image processing method, an electronic device includes a first camera and a second camera, the second camera includes a hybrid image sensor, the hybrid image sensor includes a target pixel part and an event pixel part; the method includes:

[0078] Obtaining first image data output by the first camera; obtaining second image data output by the target pixel part and event data output by the event pixel part of the second camera; registering the first image data and the second image data to obtain a registration matrix; and registering the first image data and the event data according to the registration matrix to obtain registered first image data and registered event data.

[0079] As can be seen from the above, the embodiment of the present application can make the registration deviation smaller by performing image registration on the first image data and the event data in a two-section "bridging" registration manner.

[0080] In a possible implementation of the third aspect, the first camera is a front main camera of the electronic device, and the second camera is a front auxiliary main camera of the electronic device; or the first camera is a rear main camera of the electronic device, and the second camera is a rear auxiliary main camera, a wide-angle camera or a long-focus camera of the electronic device.

[0081] Further, the implementation mode can overcome or reduce the limitations of the Hybrid sensor by using the main camera and the Hybrid sensor in combination. For example, the Hybrid sensor is used as a secondary main camera, a wide-angle camera, or a long-focus camera in combination with the main camera, which does not significantly reduce the image imaging quality of the electronic device; the resolution requirement of the Hybrid sensor is not high, thereby making the manufacturing cost lower; the amount of event data is small, which can be realized under the camera bandwidth constraint of the electronic device, and the implementation difficulty is relatively small.

[0082] In a possible implementation mode of the third aspect, the first camera is a rear main camera, and the second camera is a wide-angle camera.

[0083] If the electronic device works in the main camera focal length range, the first image data and the second image data are registered to obtain a registration matrix, and the first image data and the event data are registered according to the registration matrix to obtain registered first image data and registered event data, including:

[0084] The field of view angle of the first image data is used to perform field of view angle cropping on the second image data to obtain cropped second image data; the event data is processed by using a preset wide-angle image processing algorithm to obtain first processed event data; the field of view angle of the first image data is used to perform field of view angle cropping on the first processed event data to obtain cropped event data; the cropped second image data and the first image data are binocularly registered to obtain a registration matrix; and the cropped event data and the first image data are registered according to the registration matrix to obtain the registered first image data and the registered event data.

[0085] In a possible implementation mode of the third aspect, the first camera is a front main camera, and the second camera is a front secondary main camera.

[0086] If the electronic device is in the wide-angle focal length range, the first image data and the second image data are registered to obtain a registration matrix, and the first image data and the event data are registered according to the registration matrix to obtain registered first image data and registered event data, including:

[0087] The first image data is subjected to wide-angle distortion correction to obtain processed first image data; the second image data is subjected to wide-angle distortion correction to obtain processed second image data; the event data is processed by using a preset wide-angle image processing algorithm to obtain third processed event data; the processed second image data and the processed first image data are binocularly registered to obtain a registration matrix; and the third processed event data and the processed first image data are registered according to the registration matrix to obtain the registered first image data and the registered event data.

[0088] In a possible implementation form of the third aspect, the method further comprises: displaying the first stylized video or the first stylized picture according to the registered first image data and the registered event data.

[0089] In a fourth aspect, the embodiments of the present application provide an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method of any one of the third aspect when executing the computer program.

[0090] In a fifth aspect, the embodiments of the present application provide a computer readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the method of any one of the third aspect.

[0091] In a sixth aspect, the embodiments of the present application provide a chip system, which comprises a processor coupled with a memory, and the processor executes a computer program stored in the memory to implement the method of any one of the third aspect. The chip system can be a single chip or a chip module composed of multiple chips.

[0092] In a seventh aspect, the embodiments of the present application provide a computer program product, which, when running on an electronic device, causes the electronic device to execute the method of any one of the third aspect.

[0093] It can be understood that the beneficial effects of the second aspect to the seventh aspect can be referred to the related description of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0094] Figure 1 An event pixel readout schematic diagram provided by the embodiments of the present application;

[0095] Figure 2 An internal frame schematic diagram of a Hybrid sensor provided by the embodiments of the present application;

[0096] Figure 3A A pixel arrangement schematic diagram provided by the embodiments of the present application;

[0097] Figure 3B Another pixel arrangement schematic diagram provided by the embodiments of the present application;

[0098] Figure 3C An overall circuit frame schematic diagram of a Hybrid sensor provided by the embodiments of the present application;

[0099] Figure 4A A pixel arrangement schematic diagram provided by the embodiments of the present application;

[0100] Figure 4B Another overall circuit framework schematic of the Hybrid sensor provided for embodiments of the present application;

[0101] Figure 5 A structural schematic block diagram of the electronic device 500 provided for embodiments of the present application;

[0102] Figure 6A A rear camera module arrangement schematic provided for embodiments of the present application;

[0103] Figure 6B Another rear camera module arrangement schematic provided for embodiments of the present application;

[0104] Figure 6C Yet another rear camera module arrangement schematic provided for embodiments of the present application;

[0105] Figure 6D A front camera module arrangement schematic provided for embodiments of the present application;

[0106] Figure 7A An event video mode schematic provided for embodiments of the present application;

[0107] Figure 7B A stylized image schematic provided for embodiments of the present application;

[0108] Figure 7C A stylized image schematic provided for embodiments of the present application;

[0109] Figure 7D A stylized image schematic provided for embodiments of the present application;

[0110] Figure 7E A stylized image schematic provided for embodiments of the present application;

[0111] Figure 8 A schematic of the bridge registration method provided for embodiments of the present application;

[0112] Figure 9A A registration process schematic provided for embodiments of the present application;

[0113] Figure 9B Another registration process schematic provided for embodiments of the present application;

[0114] Figure 9C Yet another registration process schematic provided for embodiments of the present application;

[0115] Figure 10A Another internal framework schematic of the Hybrid sensor provided for embodiments of the present application;

[0116] Figure 10B A readout circuit schematic diagram provided for an embodiment of the present application;

[0117] Figure 10C Another readout circuit schematic diagram provided for an embodiment of the present application;

[0118] Figure 10D A framework schematic diagram of an image system of the electronic device 500 provided for an embodiment of the present application;

[0119] Figure 11 Another framework schematic diagram of an image system of the electronic device 500 provided for an embodiment of the present application;

[0120] Figure 12 A schematic diagram of a hard synchronization delay in the Hybrid sensor provided for an embodiment of the present application;

[0121] Figure 13 A schematic diagram of a hard synchronization in the Hybrid sensor provided for an embodiment of the present application;

[0122] Figure 14 Another schematic diagram of a hard synchronization in the Hybrid sensor provided for an embodiment of the present application;

[0123] Figure 15 Still another schematic diagram of a hard synchronization in the Hybrid sensor provided for an embodiment of the present application;

[0124] Figure 16 Still another schematic diagram of a hard synchronization in the Hybrid sensor provided for an embodiment of the present application;

[0125] Figure 17A Still another schematic diagram of a hard synchronization in the Hybrid sensor provided for an embodiment of the present application;

[0126] Figure 17B A switching timing schematic diagram provided for an embodiment of the present application;

[0127] Figure 18 Another schematic block diagram of the electronic device 500 provided for an embodiment of the present application;

[0128] Figure 19 A schematic block diagram of an image processing method provided for an embodiment of the present application. DETAILED DESCRIPTION

[0129] In the following description, for the sake of explanation, specific details are set forth in order to provide a thorough understanding of embodiments of the application. However, it is apparent that the application can be practiced without specific details or with an equivalent amount of detail.

[0130] Generally, the imaging process of a traditional camera can be as follows: during an exposure time, light is transmitted to a photosensitive element through a lens; the photosensitive element generates an electrical signal by photosensing a light signal, and transmits the electrical signal to an image signal processor (ISP); and the ISP converts the electrical signal into image data and outputs the image data at a fixed frame rate.

[0131] In the above imaging process, the traditional camera needs to be exposed for a period of time, so that the photosensitive element accumulates a certain number of photons. During the exposure time, if the object being photographed moves at a high speed relative to the camera, motion blur will occur. In addition, even if the frame rate of a high-performance traditional camera is high, there will still be a high latency of milliseconds. In poor light or high brightness, the information obtained by the traditional camera is limited, resulting in a low dynamic range.

[0132] That is, due to the imaging mechanism and processing mechanism of the traditional camera, there are usually problems such as motion blur, high latency (milliseconds), and low dynamic range, which reduces the image capability of the electronic device and the user's image experience. The electronic device includes a traditional camera.

[0133] In order to improve the user's image experience, an event camera can be introduced into the electronic device to utilize the high dynamic range, low latency (microsecond level), and no motion blur of the event camera, and to improve the image capability of the traditional camera in some scenes (such as a motion scene) by using event data output by the event camera.

[0134] The event camera (Event camera) can also be referred to as a dynamic vision sensor (dynamic vision sensor, DVS) or a neuromorphic vision sensor. The imaging mechanism, data format, and processing mechanism of the event camera are completely different from those of a complementary metal oxide semiconductor (Complementary Metal Oxide Semiconductor, CMOS) RGB camera and a charge coupled device (charge coupled device, CCD) RGB camera.

[0135] Unlike traditional cameras that capture the light intensity information of all pixels synchronously at a fixed frequency, the event camera captures the brightness change of each pixel asynchronously, i.e., each pixel of the event camera is independent of each other and outputs events independently. The event data output by the event camera has high resolution, which can reach 1 microsecond (us) accuracy, and low latency, so it can capture high-speed moving objects, and because there is no concept of exposure time, the event camera has no motion blur. In addition, the event camera can still work normally in very bright or very dark environments, and has a high dynamic range.

[0136] The event camera includes an event pixel array, and each event pixel in the event pixel array can output a corresponding event signal according to a brightness change. Specifically, for each event pixel, when the brightness on the event pixel changes and the change amount reaches a certain threshold, an "event" is output. The "event" has three elements: a timestamp, a pixel coordinate, and a polarity. The timestamp is used to represent the time when the brightness of the event pixel changes, the pixel coordinate is used to represent the position of the event pixel, and the polarity is used to represent whether the brightness on the event pixel increases or decreases. That is, the "event" can represent "at what time, which event pixel, and whether the brightness increases or decreases".

[0137] According to whether the brightness on the event pixel increases or decreases, the "event" can be divided into positive polarity events and negative polarity events. When the brightness on the event pixel increases and the brightness increase value reaches a first threshold, a positive event signal, i.e., a positive polarity event, is output; when the brightness on the event pixel decreases and the brightness decrease value reaches a second threshold, a negative event signal, i.e., a negative polarity event, is output. The first threshold and the second threshold can be set according to actual needs.

[0138] When the "events" of each event pixel of the event camera are read out, an event data stream can be obtained. The readout mode of the event camera can be divided into asynchronous readout and synchronous readout.

[0139] In the asynchronous readout mode, when the brightness value of the event pixel changes and the change value exceeds a set threshold, an event signal is output, then the event signal is marked with address information and timestamp information, and an event data packet is output, and then an asynchronous event data stream is obtained.

[0140] Exemplarily, referring to Figure 1 the event readout schematic diagram shown in the figure, Figure 1Only one event pixel is shown, and the light sensing element of the event pixel is exemplarily a photodiode (PD). The readout circuit of the event pixel includes, but is not limited to, a log circuit, an AD circuit, an ON comparison circuit, an OFF comparison circuit, a RESET circuit, a time stamping circuit, and an address stamping circuit, and the like logical units.

[0141] After the light is transmitted to the PD through the lens, the PD senses the light signal and outputs a photoelectric current. The photoelectric current is log-transformed to obtain a log voltage. After the log voltage is amplified by an AD, the amplified voltage value is input to the ON comparison circuit and the OFF comparison circuit, respectively. The amplified voltage value is compared with the set threshold value of the ON comparison circuit and the set threshold value of the OFF comparison circuit, respectively. When the amplified voltage value reaches the set threshold value of the ON comparison circuit, it is considered that the brightness on the PD increases, and the brightness increase value reaches the first threshold value, and a positive event signal is output. When the amplified voltage value reaches the set threshold value of the OFF comparison circuit, it is considered that the brightness on the PD decreases, and the brightness decrease value reaches the second threshold value, and a negative event signal is output.

[0142] The ON comparison circuit (or positive polarity comparison circuit) and the OFF comparison circuit (or negative polarity comparison circuit) are independent of each other. If the event pixel triggers a positive polarity event or a negative polarity event, the RESET circuit is triggered to reset the related capacitance on the event pixel.

[0143] After the positive event signal or the negative event signal is output, the time stamping circuit stamps the positive event signal or the negative event signal with time stamp information, and the address stamping circuit also stamps the positive event signal or the negative event signal with address information, i.e., the x and y coordinates of the event signal, and finally outputs a four-dimensional event data packet of (x, y, p, t).

[0144] The x and y in the four-dimensional event data packet represent the horizontal coordinate and the vertical coordinate of the event pixel, respectively. The p represents the polarity value of the event pixel, i.e., whether the brightness on the event pixel increases or decreases. The t represents the time stamp, i.e., the time when the brightness of the event pixel changes.

[0145] It can be understood that each event pixel of the event camera can output a four-dimensional event data packet through a readout circuit and a corresponding readout process as shown in FIG. 1B. The four-dimensional event data packets of a plurality of event pixels can form an asynchronous event data stream. Figure 1

[0146] ​In the synchronous readout mode, when the luminance value of an event pixel changes and the change value exceeds the set threshold value, an event signal is output, then the event signal is time-stamped, and the event data is read out in the form of an event frame to obtain a synchronous event frame data stream. The frame rate of the event frame is very high, which can reach 10,000 frames per second or higher. Each event frame can share one or several time stamps.

[0147] Unlike the asynchronous readout mode, the synchronous readout mode reads out the event data in the form of an event frame. When each event frame is read out, if the luminance change value on a certain event pixel reaches the set threshold value of the ON comparison circuit, the value of the pixel in the event frame is +1; if the luminance change value on a certain pixel reaches the set threshold value of the OFF comparison circuit, the value of the pixel in the event frame is -1; if the luminance change value on a pixel does not reach the set threshold value of the ON comparison circuit nor the set threshold value of the OFF comparison circuit, the value of the pixel in the event frame is 0. Therefore, the value of each event pixel in the event frame read out in the synchronous mode is +1, -1 or 0.

[0148] Similar to the asynchronous readout mode, in the synchronous readout mode, after the photosensitive element of each event pixel photosensitizes the light signal to generate an electrical signal, the electrical signal is amplified by an AD amplifier to obtain an amplified voltage value; the amplified voltage value is transmitted to the ON comparison circuit and the OFF comparison circuit, and the value of the pixel is output according to the comparison result between the amplified voltage value and the set threshold value. However, since the entire event frame is read out, it is no longer necessary to time-stamp each valid event signal, and the readout circuit of the event pixel can not include a time-stamping circuit. That is, the time-stamping circuit in the readout circuit of the event pixel is optional. Figure 1 If the readout mode is the asynchronous readout, the time-stamping circuit is needed to time-stamp each event signal; if the readout mode is the synchronous readout, the time-stamping circuit is not needed to time-stamp each event signal.

[0149] In order to take advantage of the characteristics of the event camera, such as no motion blur and low latency, to improve the image capability of the electronic device in some scenarios, the event camera can be directly introduced into the electronic device. At this time, the electronic device can include the event camera and a traditional camera.

[0150] However, the inventors have found in the research process that if the event camera is directly introduced into the electronic device, a focusing and anti-shake calculation process needs to be performed on the event data output by the event camera, which has a great impact on the existing scheme and is difficult to implement.

[0151] Therefore, in order to better introduce the event camera into the electronic device and reduce the impact on the existing scheme, the embodiments of the present application indirectly introduce the event camera into the electronic device in the form of a hybrid sensor.

[0152] The Hybrid sensor is an image sensor that mixes event pixels and target pixels. The event data output by the event pixels in the Hybrid sensor improves the image capability of a traditional camera in a motion scene, thereby indirectly introducing an event camera into an electronic device.

[0153] The Hybrid sensor includes an event pixel array and a target pixel array. The event pixel array can output event data, and the target pixel array can output second image data. The target pixels can be, but are not limited to, RGB pixels, RYB pixels, black-and-white pixels, infrared (IR) pixels, or near-infrared (NIR) pixels. The type of target pixel is not limited herein.

[0154] Taking the target pixels as RGB pixels as an example, refer to Figure 2 An internal framework of the Hybrid sensor provided in an embodiment of the present application is shown in FIG. 2. The Hybrid sensor can include, but is not limited to, an event pixel array 21, an RGB pixel array 22, an event pixel readout circuit 23, and an RGB pixel readout circuit 24.

[0155] The event pixel array 21 includes at least one event pixel, and the RGB pixel array 22 includes at least one RGB pixel. The event pixel readout circuit 23 is configured to read out the current obtained by the event pixel after being exposed to light and output event data. The event readout mode can be asynchronous readout or synchronous readout. The RGB pixel readout circuit 24 is configured to read out the current obtained by the RGB pixel after being exposed to light and output RGB data.

[0156] The RGB pixel array 22 can be, for example, but is not limited to, a Bayer pattern or a quad pattern. The design of the RGB pixel part can be substantially the same as that of an RGB image sensor, or can be different.

[0157] The event pixel is added to the RGB pixel array 22. The way of adding the event pixel to the RGB pixel array 22 can include, but is not limited to, spatial mixing and customizing a special pixel.

[0158] Spatial mixing refers to removing part of the RGB pixels in the RGB pixel array 22 and replacing them with event pixels.

[0159] For example, refer to Figure 3A A pixel arrangement is shown in FIG. 3, which includes R pixels, G pixels, B pixels, and event pixels.

[0160] If it is a traditional RGB pixel array, Figure 3A The event pixel in the array is a G pixel. By replacing the G pixel in the traditional RGGB pixel arrangement with an event pixel, the event pixel and the RGB pixel are fused in one image sensor.

[0161] It should be noted that, Figure 3A The front of the RGB pixel in the array is increased by a visible light band filter, and the front of the event pixel is not increased by a visible light band filter. The RGB pixel and the event pixel can have an infrared band filter thereon. Among them, the front of the R pixel is increased by a red light filter, the front of the G pixel is increased by a green light filter, and the front of the B pixel is increased by a blue light filter.

[0162] Replacing part of the RGB pixels in the RGB pixel array 22 with event pixels can embed event pixels in the RGB pixel array 22, but will reduce the number of RGB pixels. The reduction of the number of RGB pixels will reduce the RGB imaging quality. Generally, the level of RGB imaging quality is generally proportional to the number of RGB pixels, so the higher the proportion of the number of RGB pixels in the Hybrid sensor, the higher the RGB imaging quality, and vice versa. The lower the proportion of the number of RGB pixels, the lower the RGB imaging quality.

[0163] When the Hybrid sensor is implemented in a spatial mixing manner, in order not to greatly reduce the RGB imaging quality, the number of event pixels can be much smaller than the number of RGB pixels, so that the Hybrid sensor can output RGB data with high resolution and event data with low resolution. Exemplarily, the ratio of the number of RGB pixels to the number of event pixels in the Hybrid sensor is 8:1.

[0164] Of course, if the RGB data output by the Hybrid sensor is not used for direct imaging, or high-resolution event data is needed, the number of event pixels in the Hybrid sensor can be appropriately increased, and the number of RGB pixels can be reduced. That is, the number of event pixels and the number of RGB pixels in the Hybrid sensor can be set as needed, which is not limited herein.

[0165] In some embodiments of this application, the hybrid sensor can be used as a secondary main camera in conjunction with the main camera. The RGB data of the secondary main camera is not directly used for imaging, but only for the image registration process. In this case, the resolution and image quality requirements of the RGB data output by the hybrid sensor are not high, so the number of event pixels in the hybrid sensor can be appropriately increased, or even the RGB pixels in the hybrid sensor can be replaced with black and white pixels.

[0166] By replacing the RGB pixels in a hybrid sensor with monochrome pixels, the hybrid sensor then includes monochrome pixels and event pixels. For example, see... Figure 3B Another pixel arrangement diagram is shown, which includes black and white pixel 1, black and white pixel 2, black and white pixel 3, and event pixel. Figure 3B The black and white pixels may not have a visible light filter in front of them, but both black and white pixels and event pixels can have infrared filters.

[0167] In the process of replacing RGB pixels or black and white pixels with event pixels, in addition to setting the number of RGB pixels (or black and white pixels) and the number of event pixels as needed, the target area of ​​the event pixels and the target area of ​​the RGB pixels (or black and white pixels) can also be set as needed. For example, the target area of ​​an event pixel can be the same as the target area of ​​an RGB pixel (or black and white pixel), or it can be several times the target area of ​​an RGB pixel (or black and white pixel).

[0168] Figure 3A and Figure 3B This is merely a schematic diagram of pixel arrangement; in practical applications, it can be based on... Figure 3A and Figure 3B The pixel arrangement shown periodically replaces a portion of the RGB pixels or black and white pixels in the image sensor with event pixels, in order to fuse RGB pixels (or black and white pixels) and event pixels into a single image sensor.

[0169] It's worth noting that using spatial blending to implement a hybrid sensor can both avoid significantly reducing the image quality of RGB and embed event pixels. Furthermore, RGB pixels (or monochrome pixels) and event pixels are naturally aligned, and since the hybrid sensor shares a single lens, the RGB data stream and event data stream output by the hybrid sensor are already registered, eliminating the need for additional registration algorithms.

[0170] When the Hybrid sensor is implemented in a spatial mixing manner, the event pixel and the RGB pixel (or the black and white pixel) each have a corresponding photosensitive element, i.e., the event pixel and the RGB pixel (or the black and white pixel) do not share one photosensitive element. At this time, for the RGB pixel (or the black and white pixel), the RGB data can be read out by the RGB pixel readout circuit 24; and for the event pixel, the event data can be read out by the event pixel readout circuit 23.

[0171] Exemplarily, referring to Figure 3C FIG. 1 shows a schematic diagram of an overall circuit framework of the Hybrid sensor, which only shows one RGB pixel (or black and white pixel) and one event pixel. The photosensitive element of the RGB pixel (or black and white pixel) is exemplarily PD1, and the photosensitive element of the event pixel is exemplarily PD2.

[0172] The PD1 can be read out according to the readout circuit of a normal RGB CMOS (or black and white CMOS). As shown in FIG. 2, after the light passes through the lens and is transmitted to the PD1, the PD1 performs photosensitive output of a photosensitive current. The photosensitive current is amplified by AD1 and then output to the RGB readout circuit. The RGB readout circuit outputs an RGB data stream according to the photosensitive current. Figure 3C

[0173] The readout circuit of the PD2 can be basically the same as the principle of the event pixel readout circuit of a normal event camera. As shown in FIG. 3, after the light passes through the lens and is transmitted to the PD1, the PD1 performs photosensitive output of a photosensitive current. The photosensitive current is log-transformed to obtain a log voltage. The log voltage is amplified by AD2 to obtain an amplified voltage, and the amplified voltage is output to the ON comparison circuit and the OFF comparison circuit, respectively. When the amplified voltage is greater than the set threshold of the ON comparison circuit, it is considered that the brightness sensed by the PD2 increases, and the brightness increase value reaches a certain threshold. The ON comparison circuit outputs an effective event signal, which represents a +1 event. When the amplified voltage is greater than the set threshold of the OFF comparison circuit, it is considered that the brightness sensed by the PD2 decreases, and the brightness decrease value reaches a certain threshold. The OFF comparison circuit outputs an effective event signal, which represents a -1 event. After the ON comparison circuit or the OFF comparison circuit triggers the effective event signal, the Reset circuit can be triggered to take effect and reset the register circuit of the PD2, and then the exposure is restarted. Figure 3C The effective event signal triggered by the ON comparison circuit or the OFF comparison circuit is transmitted to the time stamping circuit and the address circuit, and the coordinate information and the time stamp information of the event signal are printed to output an event data stream.

[0174]

[0175] ​​It can be understood that the reading mode of the event data can be synchronous reading or asynchronous reading. If the reading mode is asynchronous reading, an address circuit is needed, and if the reading mode is synchronous reading, the address circuit can not be needed.

[0176] In Figure 3C , the event pixel reading circuit can include logic units such as current log removal, AD2, ON comparison circuit, OFF comparison circuit, Reset circuit, time stamping circuit and address circuit, wherein the logic units such as current log removal, AD2, ON comparison circuit, OFF comparison circuit and Reset circuit exist for each event pixel, and the time stamping circuit and the address circuit are logic units shared by multiple event pixels.

[0177] It should be noted that Figure 3C The overall circuit framework of the Hybrid sensor shown only shows the logic units, and there can be register circuits and resistors and other electronic devices between different logic units. In addition, in addition to Figure 3C the logic units shown, there can be other logic units, for example, column readout control units and row readout control units can also be included to control the exposure and readout operation of the event pixel rows and columns.

[0178] As described above, in addition to the Hybrid sensor that can be implemented in a spatially mixed manner, the Hybrid sensor can also be implemented in a manner of customizing special pixels.

[0179] Customizing special pixels means that a special pixel is set, which shares a photosensitive element and integrates two different readout circuits, one of which is used to read out the current change of the photosensitive element to form normal RGB pixel values (or black and white pixel values), and the other of which converts the current change of the photosensitive element into an event data stream with a time interval of microsecond level through the event pixel reading circuit.

[0180] Exemplarily, referring to Figure 4A A pixel arrangement diagram is shown, which includes R pixels, G pixels, G pixels and B pixels, and each RGB pixel can also be an event pixel. For example, for the R pixel in Figure 4A , it not only serves as an R pixel, but also as an event pixel. At this time, a pixel can serve as both an event pixel and an RGB pixel, so Figure 2 the RGB pixel array 21 and the event pixel array 22 in may be the same pixel array. And a pixel integrates an event pixel reading circuit 23 and an RGB reading circuit 24, respectively used for reading out event data and RGB data.

[0181] It should be noted that, Figure 4A The RGB pixel in the RGB pixel can also be replaced by a black and white pixel. At this time, the special pixel not only acts as a black and white pixel, but also as an event pixel. For example, see Figure 4B Another overall circuit framework diagram of the Hybrid sensor is shown, which only shows one pixel, and the light sensing element of the pixel is exemplified as PD1. The pixel not only acts as an RGB pixel (or a black and white pixel), but also as an event pixel.

[0182] As shown in Figure 4B , an additional readout circuit of the event pixel is added below the PD1 of the RGB pixel, and the RGB pixel and the event pixel share the PD1, but the readout circuits are different, that is, each pixel has two sets of readout circuits, one of which reads out the RGB pixel value, and the other of which reads out the high-frame-rate event pixel value (i.e. positive and negative polarity values or zero value). In this case, since the readout circuit is more, 3D stacking can be used on the image sensor to increase more electronic components.

[0183] It can be understood that, Figure 4B The visible light filter on each pixel in the RGB pixel is adapted to the RGB pixel. Of course, if Figure 4B The RGB pixel in the RGB pixel is replaced by a black and white pixel, and an infrared waveband filter can be provided on all pixels.

[0184] As shown in Figure 4B , after the light passes through the lens and reaches the PD1, the PD1 senses the light to obtain a light sensing current, which is transmitted to the RGB readout circuit and the event pixel readout circuit, respectively, and the two readouts are read out respectively, and the RGB data stream and the event data stream are output. The RGB readout circuit and the event pixel readout circuit do not affect each other, and there will also be an anti-crosstalk design.

[0185] For the event pixel readout circuit, the light sensing current of the PD1 is log-transformed to obtain a log voltage, and after being amplified by the AD2, an amplified voltage is obtained; the amplified voltage is transmitted to the ON comparison circuit and the OFF comparison circuit to trigger the ON comparison circuit or the OFF comparison circuit to output an event signal; the event signal is further timestamped by the timestamping circuit and the address circuit, and after being stamped with address information and timestamp information, an event data stream is obtained.

[0186] For the RGB readout circuit, the light sensing current of the PD1 is amplified by the AD1 and transmitted to the RGB readout circuit, and the RGB readout circuit outputs the RGB data stream according to the amplified current.

[0187] It should be noted that in the case of sharing the light sensing element, since the readout frequency of the event pixel is much higher than that of the RGB pixel (or the black and white pixel), the Figure 4BAfter triggering an ON comparison circuit or OFF comparison circuit output event signal, the Reset circuit is triggered to take effect, but the Reset circuit only resets the capacitor in the event pixel readout circuit and does not interfere with the continuous exposure of PD1, thereby avoiding affecting the readout circuit of the RGB pixel (or black and white pixel). For example, an additional register capacitor can be added before the current log circuit of the event pixel readout circuit to store the photocurrent generated by the brightness change of PD1 in a very short time, and the Reset circuit only resets the register capacitor, thereby not affecting the readout circuit of the RGB pixel (or black and white pixel). The above-mentioned very short time can be changed by the relevant parameters of the Hybrid sensor issued by the upper layer.

[0188] The way of customizing special pixels is to share the photosensitive element, so the RGB data and event data output by the Hybrid sensor are naturally aligned and do not need additional registration. Moreover, the readout circuit of the event pixel and the readout circuit of the RGB pixel in the Hybrid sensor are independent of each other, so the event pixel can adopt a synchronous readout mode to output a high-frame-rate synchronous event frame, or an asynchronous readout mode to output an asynchronous event data stream.

[0189] It should be noted that, Figure 4B The overall circuit framework of the Hybrid sensor shown only shows logical units, and there can be register capacitors and resistors and other electronic devices between different logical units.

[0190] Whether it is a spatial mixing method or a special pixel customization method, there can be two mobile industry processor interface (MIPI) transmissions in the Hybrid sensor, respectively transmitting the RGB data stream and the event data stream. The event data stream can also have necessary compression algorithms before MIPI output.

[0191] It should be noted that the above describes the spatial mixing and special pixel customization methods of the two pixel mixing methods, as well as the overall circuit framework of the Hybrid sensor, taking RGB pixels and black and white pixels as examples. When the target pixel is an RYYB pixel, an IR pixel or an NIR pixel, the spatial mixing, special pixel customization and overall circuit framework of the RGB pixel are similar. For example, when the target pixel is an RYYB pixel, Figure 3A The G pixel in the RYYB pixel can be adaptively changed to a Y pixel. For example, when the target pixel is an IR pixel or an NIR pixel, Figure 3A The RGB pixel in the RYYB pixel can be replaced by an IR pixel or an NIR pixel.

[0192] Compared with directly introducing an event camera into an electronic device, indirectly introducing an event camera into an electronic device in the form of a Hybrid sensor can use second image data output by the Hybrid sensor to perform related calculations such as focusing, electronic image stabilization, and optical image stabilization, and then drive lens movement, without the need to calculate focusing and image stabilization and other processes on event data, and has less impact on existing solutions and is easier to implement.

[0193] However, the inventors have also found in the research process that, in the process of introducing a Hybrid sensor into an electronic device, due to many limitations of the Hybrid sensor, the Hybrid sensor cannot be well introduced into the electronic device.

[0194] Taking a target pixel as an RGB pixel as an example, whether it is an event camera or a Hybrid sensor, the readout circuit of an event pixel is more complex than that of an RGB pixel. Therefore, the area of the readout circuit of an event pixel is much larger than that of an RGB pixel (or a black-and-white pixel).

[0195] Due to the larger area of the readout circuit of an event pixel, if the same number of layers are stacked, the placement area of the readout circuit of an event pixel is larger than that of an RGB pixel. Based on this, if a side-by-side placement method is used in the Hybrid sensor, an event pixel will occupy the positions of several R pixels, G pixels, or B pixels, which will seriously affect the imaging quality of the RGB pixels.

[0196] Based on this, if the Hybrid sensor is directly introduced into an electronic device as a main camera, due to the much larger placement area of the event pixel than that of the RGB pixel, the imaging quality of the RGB data output by the Hybrid sensor is low, which further affects the RGB imaging quality on electronic devices such as mobile phones and tablets.

[0197] In addition, when the resolutions of the RGB data and the event data are the same, whether the readout method of the event pixel is synchronous readout or asynchronous readout, the data amount of the event pixel will be significantly higher than that of the RGB pixel. Due to the bandwidth limitation of the camera of the electronic device, the event resolution in the Hybrid sensor cannot be very high. In this way, the event resolution of the Hybrid sensor on electronic devices such as mobile phones and tablets is much lower than the resolution of the RGB camera, for example, the event resolution in the Hybrid sensor is less than 8M.

[0198] In the embodiments of the present application, in order to better introduce the Hybrid sensor into the electronic device, the main camera and the Hybrid sensor are used in combination to overcome or reduce the many limitations of the Hybrid sensor.

[0199] After the event camera is indirectly introduced into the mobile phone and the tablet through the form of the Hybrid sensor, the event data generated by the event pixels can be added to the mobile phone and the tablet, and the event data can improve the image capability of the traditional camera of the electronic device for the motion scene.

[0200] The Hybrid sensor provided by the embodiments of the present application can be applied to an electronic device, which can be a smart mobile terminal such as a smart phone and a tablet computer, or other types of electronic devices, which are not limited here.

[0201] Exemplarily, Figure 5 A structural schematic block diagram of an electronic device 500 provided by the embodiments of the present application is shown, which can include a processor 510, a memory 520, a first camera 530, a display screen 540 and a second camera 550.

[0202] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 500. In other embodiments of the present application, the electronic device 500 can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements. The components shown can be implemented in hardware, software or a combination of software and hardware.

[0203] For example, when the electronic device 500 is a smart mobile terminal such as a mobile phone or a tablet computer, the electronic device 500 can further include at least one of a universal serial bus (USB) interface, a charging management module, a power management module, a battery, an antenna, a mobile communication module, a wireless communication module, an audio module, a speaker, a receiver, a microphone, a headset interface, a sensor module, a key, a motor, an indicator, and a subscriber identification module (SIM) card interface, etc. The sensor module can include a pressure sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a distance sensor, a proximity light sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, and a bone conduction sensor, etc.

[0204] The processor 510 can include one or more processing units, for example: the processor 510 can include an application processor (AP), a graphics processing unit (GPU), an ISP, a controller, a video codec, a digital signal processor (DSP), and the like. Among them, different processing units can be independent devices, or can be integrated in one or more processors. The controller can generate operation control signals according to instruction operation codes and timing signals, complete the control of fetching instructions and executing instructions.

[0205] In some embodiments, the processor 510 can include one or more interfaces. The interface can include an inter-integrated circuit (I2C) interface, a mobile industry processor interface (MIPI), and a general-purpose input / output (GPIO) interface, and the like.

[0206] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 510 can contain multiple groups of I2C buses. The processor 510 can be coupled with the touch sensor through the I2C interface, so that the processor 510 and the touch sensor communicate through the I2C bus interface, and realize the touch function of the electronic device 500.

[0207] The MIPI interface can be used to connect the processor 510 and the display screen 540, the first camera 530 and the second camera 550, and other peripheral devices. The MIPI interface includes a camera serial interface (CSI), a display screen serial interface (DSI), and the like. In some embodiments, the processor 510 and the first camera 530 and / or the second camera 550 communicate through the CSI interface, and realize the shooting function of the electronic device 500. The processor 510 and the display screen 540 communicate through the DSI interface, and realize the display function of the electronic device 500.

[0208] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 510 to the first camera 530, the second camera 550, the display screen 540, and the like. The GPIO interface can also be configured as an I2C interface, an MIPI interface, and the like.

[0209] It can be understood that the interface connection relationship between the modules shown in the embodiments of the present application is only illustrative and does not constitute a structural limitation of the electronic device 500. In some other embodiments of the present application, the electronic device 500 can also use different interface connection modes or a combination of multiple interface connection modes in the above embodiments.

[0210] The electronic device 500 implements a display function through a GPU, a display screen 540, and an application processor, and the like. The GPU is a microprocessor for image processing, connected to the display screen 540 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 510 can include one or more GPUs that execute program instructions to generate or change display information.

[0211] The display screen 540 is used to display images and videos, and the like. The display screen 540 includes a display panel. The display panel can use a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light-emitting diode (QLED), and the like. In some embodiments, the electronic device 500 can include one or N display screens 540, and N is a positive integer greater than 1.

[0212] The electronic device 500 can implement a shooting function through an ISP, a first camera 530, a second camera 550, a video codec, a GPU, a display screen 540, and an application processor, and the like.

[0213] ISP is used to process the data fed back by the first camera 530 or the second camera 550. For example, when taking a photo, the shutter is opened, the light is transmitted to the camera photosensitive element through the lens, the light signal is converted into an electric signal, and the camera photosensitive element transmits the electric signal to the ISP for processing and conversion into an image visible to the naked eye. The ISP can also optimize the algorithm of the noise, brightness, and skin color of the image. The ISP can also optimize the exposure, color temperature, and other parameters of the shooting scene. In some embodiments, the ISP can be arranged in the first camera 530 or the second camera 550.

[0214] The first camera 530 is used to capture a still image or a video. An object generates an optical image through a lens and projects the optical image onto a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electric signal, and then transmits the electric signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV, or the like format.

[0215] The first camera 530 is usually a traditional camera, such as an RGB camera, and the type of the first camera 530 is not limited herein. The second camera 550 can include, but is not limited to, a lens and a hybrid sensor, the lens is composed of one or more lenses, and the hybrid sensor includes event pixels and target pixels. The related content of the hybrid sensor can be referred to the corresponding content described above, and will not be repeated here.

[0216] In the embodiments of the present application, when the first camera 530 is used as the main camera of the electronic device 500, the first camera 530 can be used in combination with the second camera 550, and at this time, the second camera 550 can be used as the auxiliary main camera, the wide-angle camera, or the long-focus camera of the electronic device 500.

[0217] It can be understood that the electronic device 500 can further include other cameras in addition to the first camera 530 and the second camera 550, for example, the electronic device 500 can further include at least one of an RGB wide-angle camera, an RGB long-focus camera, and an auxiliary camera.

[0218] The digital signal processor is used to process digital signals, and in addition to processing digital image signals, it can also process other digital signals. For example, when the electronic device 500 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, and the like.

[0219] A video codec is used to compress or decompress digital video. The electronic device 500 can support one or more video codecs. In this way, the electronic device 500 can play or record videos in a variety of encoding formats, such as moving picture experts group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, and the like.

[0220] The memory 520 can be used to store computer-executable program code including instructions. The memory 520 can include a program storage area and a data storage area. The program storage area can store an operating system, application programs (such as a sound play application, an image play application, and the like) required by at least one function, and the like. The data storage area can store data (such as audio data, a phone book, and the like) created during use of the electronic device 500, and the like. In addition, the memory 520 can include a high-speed random access memory, and can further include a non-volatile memory such as at least one of a magnetic disk storage device, a flash memory device, a universal flash storage (UFS), and the like. The processor 510 executes various function applications and data processing of the electronic device 500 by running instructions stored in the internal memory 520 and / or instructions stored in a memory disposed in the processor.

[0221] As described above, the electronic device 500 can include multiple cameras. In the embodiments of the present application, the electronic device 500 can include a first camera 530 and a second camera 550. Optionally, the electronic device 500 can further include other cameras, which include but are not limited to at least one of the following: an RGB wide-angle camera, an RGB telephoto camera, and an auxiliary camera. The auxiliary camera can exemplarily be a time of flight (ToF) camera, and the like.

[0222] In specific applications, the Hybrid sensor can be used as the main camera of the electronic device 500, or as the auxiliary main camera, the telephoto camera, or the wide-angle camera of the electronic device 500. In addition, the Hybrid sensor can be a front camera or a rear camera.

[0223] The following exemplarily shows the camera module arrangement that can be involved in the embodiments of the present application.

[0224] Referring to Figure 6AThe diagram shows a rear camera module layout for a mobile phone 61. The rear camera module includes a main camera 611, a secondary main camera 612, a wide-angle camera 613, a telephoto camera 614, a telephoto camera 615, and an auxiliary camera 616. The auxiliary camera 616 can be a ToF type auxiliary camera. The main camera 611, wide-angle camera 613, telephoto camera 614, and telephoto camera 615 are all RGB cameras.

[0225] Among them, the secondary main camera 612 is a camera that includes a hybrid sensor, which includes event pixels and RGB pixels (or black and white pixels). That is, the hybrid sensor is used as the secondary main camera in the rear camera module of the phone 61.

[0226] The RGB and event pixels of the secondary main camera 612 can be significantly lower than those of the main camera 611. For example, the resolution of the main camera 611 is greater than 50M, while the resolution of the RGB pixels of the secondary main camera 612 is less than or equal to 12M, and the event pixels can be less than or equal to 3M.

[0227] The field of view (FOV) of the secondary main camera 612 is close to that of the main camera 611. Furthermore, the main camera 611 and the secondary main camera 612 can share a camera mount. Since the main camera 611 has optical image stabilization (OIS), the secondary main camera will also have an OIS device. The main camera 611 is an automatic focus (AF) lens, while the secondary main camera 612 can be either an AF lens or a fixed focus (FF) lens. The main camera 611 has a variable aperture, while the secondary main camera 612 can have either a variable aperture or a fixed aperture.

[0228] It should be noted that the pixel count, FOV, OIS, AF, FF, and aperture parameters of the main camera 611 and the secondary main camera 612 mentioned above are only examples. In specific applications, the parameters of the main camera 611 and the secondary main camera 612 can be set as needed.

[0229] like Figure 6A As shown, an additional camera module is added to the rear camera module, and a hybrid sensor is placed in the position of this additional camera module, with the hybrid sensor serving as the secondary main camera.

[0230] In addition to serving as an additional secondary main camera, the hybrid sensor can also replace the RGB wide-angle camera, acting as a wide-angle camera. For example, see... Figure 6BThe diagram shows another rear camera module arrangement. The rear camera module of the phone 62 may include a main camera 621, a wide-angle camera 622, a telephoto camera 623, a telephoto camera 624, and an auxiliary camera 625. The main camera 621, the telephoto camera 623, and the telephoto camera 624 can all be RGB cameras.

[0231] Among them, the wide-angle camera 622 is a camera that includes a hybrid sensor, which includes event pixels and RGB pixels (or black and white pixels). That is, the RGB wide-angle camera is replaced by a hybrid sensor, and the hybrid sensor is used as the wide-angle camera in the rear camera module.

[0232] It should be noted that the parameters such as pixel count, FOV, OIS, AF, FF, and aperture of the main camera 621 and the wide-angle camera 622 can be set as needed, and are not limited here.

[0233] In addition to serving as a secondary main rear camera and a wide-angle rear camera, a hybrid sensor can also function as a telephoto rear camera. For example, see... Figure 6C The diagram shows another possible rear camera module arrangement. The rear camera module of the phone 63 may include a main camera 631, a wide-angle camera 63, a telephoto camera 633, and an auxiliary camera 634. Both the main camera 631 and the wide-angle camera 632 are RGB cameras.

[0234] Among them, the telephoto camera 633 is a camera that includes a hybrid sensor, which includes event pixels and RGB pixels (or black and white pixels). That is, the RGB telephoto camera is replaced by a hybrid sensor, and the hybrid sensor is used as the telephoto camera in the rear camera module.

[0235] It should be noted that the parameters such as pixel count, FOV, OIS, AF, FF, and aperture of the main camera 631 and the telephoto camera 633 can be set as needed, and are not limited here.

[0236] In addition to serving as a rear camera, a hybrid sensor can also function as a front-facing camera. For example, see... Figure 6D The diagram shows a front-facing camera module arrangement. The front-facing camera module of the mobile phone 64 may include a main camera 641, a secondary main camera 642, and an auxiliary camera 643. The main camera 641 is an RGB camera.

[0237] The secondary main camera 642 is a camera including a hybrid sensor, which includes event pixels and RGB pixels (or black-and-white pixels), that is, the hybrid sensor is used as a secondary main camera in a front camera module.

[0238] It should be noted that the pixels, FOV, OIS, AF, FF, and aperture of the main camera 641 and the secondary main camera 642 can be set as needed, and are not limited herein.

[0239] In the Figures 6A to 6D , the electronic device 500 can be specifically a Figures 6A to 6D phone in the , the first camera 530 can be specifically a main camera of the phone, and the second camera 550 can be specifically a rear secondary main camera, a rear wide-angle camera, a rear long-focus camera, or a front secondary main camera of the phone.

[0240] That is, the hybrid sensor can be used as a front camera of the electronic device 500, or as a rear camera of the electronic device 500. In addition, the hybrid sensor can be used as a secondary main camera, a wide-angle camera, or a long-focus camera of the electronic device 500. Of course, the hybrid sensor can also be used as a main camera of the electronic device 500.

[0241] However, compared with directly using the hybrid sensor as a main camera, using the hybrid sensor as a secondary main camera, a wide-angle camera, or a long-focus camera in combination with a main camera can make the RGB imaging quality of the electronic device 500 better, the cost lower, and the implementation easier.

[0242] Specifically, if the hybrid sensor is directly used as a main camera, whether the hybrid sensor uses a spatial mixing method or a special pixel customization method, the RGB image quality of the hybrid sensor will be greatly degraded, and thus the RGB imaging quality of the electronic device 500 is poor.

[0243] In other words, because the area of the event pixels is larger than the area of the RGB pixels (or black-and-white pixels), using a spatial mixing method to replace a part of the RGB pixels with event pixels will cause a great degradation of the RGB image quality. Even if the hybrid sensor is implemented by using a special pixel customization method, there will be a relatively large crosstalk, which affects the imaging quality of the RGB pixels and the event pixels.

[0244] In addition, the manufacturing cost of the Hybrid sensor is high, and the more the pixels, the higher the cost. The resolution requirement of the electronic device 500 is usually high, so a high-resolution Hybrid sensor is usually required. The Hybrid sensor with high resolution has more pixels, and the cost is very high.

[0245] The more event pixels in the Hybrid sensor, the larger the data volume of the event data output by the Hybrid sensor, and the higher the required camera bandwidth. Therefore, the size of the event pixels of the Hybrid sensor is also limited. Under the camera bandwidth constraint (such as MIPI) of the electronic device, it is difficult to achieve a very high resolution of the event pixels of the Hybrid sensor.

[0246] However, when the Hybrid sensor is used as a secondary main camera, a wide-angle camera, or a long-focus camera, and is used in combination with the main camera, the RGB data of the Hybrid sensor will not be directly used for imaging. Even if the imaging quality of the Hybrid sensor is poor, it will not significantly reduce the RGB imaging quality of the electronic device 500.

[0247] In addition, the resolution of the Hybrid sensor as a secondary main camera, a wide-angle camera, or a long-focus camera does not need to be very high. The manufacturing cost of the Hybrid sensor with low resolution is lower, and the data volume of the event pixels is smaller, which can be realized under the camera bandwidth constraint of the electronic device, and the difficulty of implementation is smaller.

[0248] In addition, by introducing the event camera into the electronic device 500 in the form of the Hybrid sensor, power consumption can be saved. Specifically, the power consumption of the event pixels in the Hybrid sensor is lower than that of the RGB pixels, so the power consumption of the Hybrid sensor with event pixels is lower than that of the RGB image sensor with the same resolution. In addition, after adding the event pixels, the high-resolution RGB camera can output a high-resolution low-frame-rate RGB video, and then the low-power event pixels can insert the high-resolution low-frame-rate RGB video into a high-frame-rate video (such as 15fps to 60fps). Compared with the high-resolution RGB image sensor directly outputting a high-frame-rate video (such as 60fps directly), the overall power consumption is lower.

[0249] In addition to the exemplary introduction of the hardware structure of the electronic device 500 and the possible camera module arrangement and distribution of the electronic device 500, the process of using the first camera 530 and the second camera 550 in combination will be introduced and described below taking the electronic device 500 as an example.

[0250] The electronic device 500 can output first image data by the first camera 530 and output second image data and event data by the second camera 550 after starting the first camera 530 and the second camera 550. Specifically, the second camera 550 includes an event pixel part and a target pixel part, and the event data can be output based on the event pixel part and the second image data can be output based on the target pixel part.

[0251] The electronic device 500 can perform corresponding processing on at least one of the first image data, the second image data and the event data after obtaining the first image data, the second image data and the event data, and then perform image display and / or image storage.

[0252] In some embodiments, the electronic device 500 can use the event data output by the second camera 550 to improve the image capability of the first camera for a motion scene. At this time, the second image data and the event data output by the second camera 550 are generally not directly displayed. For example, the electronic device 500 uses the event data output by the second camera 550 to perform image deblurring or video interpolation on the first image data output by the first camera 530, obtains a display image or a display video stream, and displays the display or display video stream on the display screen 540 of the electronic device 500.

[0253] In other embodiments, in addition to using the event data output by the second camera 550 to improve the image capability of the first camera 530 for a motion scene, the event data and / or the second image data output by the second camera 550 can also be used to provide more image modes for the user to improve the user's image experience.

[0254] At this time, the electronic device 500 can display the data output by the second camera 550. Specifically, the electronic device 500 can display the event data alone, can display the second image data alone, or can display the event data and the first image data superimposed, or can display the event data and the second image data superimposed.

[0255] In some embodiments of the present application, the electronic device 500 can add a "event video" shooting mode, and after entering the mode, the electronic device 500 can display stylized pictures or stylized videos based on the second image data and / or the event data output by the second camera 550, provide more playability for the user, and improve the user experience.

[0256] Exemplarily, referring to Figure 7AThe illustrated diagram shows the event video mode. The main interface 711 of the phone 71 includes applications such as Smart Life, Settings, Calendar, Clock, and Camera 712. Upon detecting a click on the camera 712, the phone 71 responds by entering the camera mode and displaying the camera interface 713. The camera interface 713 displays control buttons for shooting modes such as Night Scene, Portrait, Video Recording, and Event Video 714. Upon detecting a click on Event Video 714, the phone 71 enters the "Event Video" mode.

[0257] Normally, when the electronic device 500 turns on the camera 712, it activates the first camera 530 to take a picture or record a video, and simultaneously activates the second camera 550. Of course, when the electronic device 500 turns on the camera 712, it only activates the first camera 530, and only activates the second camera 550 when a click operation is detected on the event video 714.

[0258] In "Event Video" mode, the phone 71 can compress all event data output by the second camera 550 within a fixed time period into frames, obtain a display image, and display that display image. For example, as Figure 7B As shown, in "Event Video" mode, mobile phone 71 compresses raw event data within a fixed time period into frames to obtain image 716, which is then displayed on interface 715. In image 716, white represents a +1 event for an event pixel, black represents a -1 event for an event pixel, and gray represents no event.

[0259] In "Event Video" mode, the phone 71 can synthesize partial event data output by the second camera 550 within a fixed time period to obtain a display image, and then display that display image. For example, as Figure 7C As shown, in "Event Video" mode, mobile phone 71 synthesizes a portion of the event data within the RGB exposure time to obtain image 718, which is then displayed on interface 717. In image 718, white represents a +1 event for an event pixel, and black represents a -1 event for an event pixel. At this time, the fixed time is the RGB exposure time of either the first camera 530 or the second camera 550.

[0260] In "Event Video" mode, the phone 71 can also synthesize all event data output by the second camera 550 within a fixed time period to obtain a composite image. This composite image is then overlaid on an RGB image to obtain a display image, which is then displayed. For example, as... Figure 7DAs shown, in "Event Video" mode, mobile phone 71 synthesizes all event data within the full RGB exposure time to obtain a composite image. This composite image is then overlaid on the RGB image to obtain image 720, which is displayed on interface 719. In image 720, white represents an event pixel's +1 event, and black represents an event pixel's -1 event. The RGB image overlaid on the composite image can be output by either the first camera 530 or the second camera 550; that is, event data can be overlaid on either the RGB image output by the first camera 530 or the RGB image output by the second camera 550.

[0261] In "Event Video" mode, the phone 71 can also overlay event data output from the second camera 550 onto an edge map to obtain a display image, which is then displayed. This edge map is generated based on an RGB image. For example, as... Figure 7E As shown, in "Event Video" mode, mobile phone 71 overlays event image data onto an edge map to obtain image 722, which is then displayed on interface 721. In image 722, red represents a +1 event for the event pixel, blue represents a -1 event for the event pixel, and the black and white image represents the edge map generated from the RGB image. The RGB image at this time can be output by either the first camera 530 or the second camera 550.

[0262] like Figure 7E As shown in image 722, the black and white edge map generated from the RGB image serves as the background and the edges of stationary objects in the image, while the event data is represented in blue and red and serves as moving objects in the image, forming a stylized video similar to a sketch.

[0263] Stylized videos can be further refined into comic book or sketch styles, specifically as follows: Figure 7C As shown in image 718; stylized video can also predict the trajectory of some moving objects, specifically as follows: Figure 7D As shown in image 720; stylized video can also create motion trailing effects, similar to "light trails," and the color style of moving areas can be modified, such as... Figure 7E Image 722 is shown.

[0264] It should be noted that, Figures 7B to 7E In this system, mobile phone 71 can obtain a video stream based on multiple consecutive frames of transmitted images, enabling it to output stylized video in "event video" mode. Alternatively, mobile phone 71 can also be set to "event image" mode and output stylized images in that mode.

[0265] In addition, the colors of the positive and negative polarities of the event data on the video can be customized by the user. For example, the default colors of the positive and negative polarities of the event data are red and blue, and the user can adjust the default colors to black and white or any other two contrasting colors as needed. In this way, the user's video experience can be further improved.

[0266] By introducing the event camera into the electronic device 500 in the form of a hybrid sensor, since the event camera records the complete motion of an object, the equivalent frame rate can reach tens of thousands of frames, and since the event data only contains the contour of the object, the electronic device 500 can output the event data to the user as a special video or superimpose it on the image video (e.g., an RGB video) to form many stylized videos. For example, the contour map of the scene currently seen by the user can be extracted in real time to form a video in the style of a cartoon or sketch; or the continuous motion recorded by the event camera can be superimposed on the motion video recorded by the RGB camera to form a special high-frame-rate video. For another example, the event data can display the motion trajectory of an object and can continuously display the trajectory without frame skipping, and can present a cartoon or sketch style video based on real motion. In this way, the user can explore new and interesting camera play methods, and the user experience is better.

[0267] Generally, when displaying a stylized video or a stylized picture based on the first image data output by the first camera 530 and the event data output by the second camera 550, the first image data and the event data need to be registered. Similarly, when displaying a stylized video or a stylized picture based on the second image data and the event data, the second image data and the event data also need to be registered.

[0268] The second image data and the event data output by the second camera 550 are registered, so there is no need to add an additional image registration algorithm.

[0269] However, the first image data output by the first camera 530 and the event data output by the second camera 550 are not registered, so an additional image registration algorithm is usually needed to register the first image data and the event data, and then the registered first image data and the registered event data are used for image display and / or image storage.

[0270] In some embodiments, the electronic device 500 can directly register the first image data output by the first camera 530 and the event data output by the second camera 550 to obtain registered first image data and registered event data. For example, the first image data and the event data are registered by an artificial intelligence (AI) registration method.

[0271] In some other embodiments, the electronic device 500 can also register the first image data and the event data of the first camera in a two-stage "bridging" registration manner.

[0272] The two-stage "bridging" registration manner includes two registrations. The first registration is between the first image data output by the first camera 530 and the second image data output by the second camera 550. The second registration is between the first image data output by the first camera 530 and the event data output by the second camera 550.

[0273] Specifically, the electronic device 500 first performs binocular registration on the first image data and the second image data to obtain a registration matrix H, and then performs registration on the first image data and the event data according to the registration matrix H to obtain registered first image data and registered event data.

[0274] Taking the first camera 530 as the main camera and the first image data as RGB image data as an example. Referring to Figure 8 FIG. 7 shows a schematic diagram of the bridging registration manner. The main camera outputs a first RGB image data stream, and the hybrid sensor of the second camera 550 outputs a second RGB image data stream and an event data stream. After the first RGB image data stream is processed by the first ISP and the second RGB image data stream is processed by the second ISP, the first RGB image data stream processed by the first ISP and the second RGB image data stream processed by the second ISP are subjected to binocular RGB registration.

[0275] In the binocular RGB registration, the electronic device 500 can first obtain preset parameters and then perform binocular RGB registration according to the preset parameters.

[0276] The preset parameters can include, but are not limited to, the following parameters: internal and external parameter calibration data and other calibration parameters obtained by offline calibration of the main camera and the hybrid sensor; data fed back by a gyroscope sensor or an inertial measurement unit (IMU) in real time; real-time OIS data of the main camera and the hybrid sensor; focus motor stroke data of the main camera and the hybrid sensor; phase detection (PD) focus data of the main camera and the hybrid sensor; current focus depth data obtained by a TOF and a laser depth device.

[0277] The other calibration data can be, for example, brightness and noise related calibration parameters. Through the other calibration data, the brightness, noise, and color differences of the images of the main camera and the hybrid sensor at the same time after binocular RGB registration can be within a certain range.

[0278] The electronic device 500 can include a gyroscope sensor or an IMU. Based on the data fed back by the gyroscope sensor or the IMU, the displacement and the rotation angle of each axis of the electronic device 500 can be calculated. Based on the real-time OIS data of the main camera and the hybrid sensor, the electronic device 500 can calculate the frame difference between the main camera and the hybrid sensor. Based on the focus motor stroke data of the main camera and the hybrid sensor, the PD focus data, and the focus depth data obtained by the depth device such as TOF, the electronic device 500 can determine the depth information required for binocular RGB registration.

[0279] After the electronic device 500 performs binocular RGB registration on the first RGB image data stream and the second RGB image data stream, the registration matrix H can be obtained. Then, the registration matrix H is used for the second registration. Specifically, after the event data stream is processed by the event signal processor (ESP), the event data stream processed by the ESP and the first RGB image data stream processed by the first ISP are registered according to the registration matrix H.

[0280] In the second registration process, the following parameters are also required: the internal and external parameter calibration data and other calibration parameters of the offline calibration of the main camera and the hybrid sensor; the data fed back by the real-time gyroscope sensor or the IMU; the real-time OIS data of the main camera and the hybrid sensor; the focus motor stroke data of the main camera and the hybrid sensor; the PD focus data of the main camera and the hybrid sensor; and the current focus depth data obtained by the depth device such as TOF and laser.

[0281] After the second registration, the registered high-resolution RGB image data stream and the low-resolution event data stream can be obtained.

[0282] It should be noted that, compared with the direct registration mode, the two-stage "bridging" registration mode has smaller registration deviation. The direct registration refers to directly registering the first image data stream output by the first camera 530 with the event data stream output by the second camera 550.

[0283] As described above, the second camera 550 can serve as a front sub-primary camera, a rear sub-primary camera, a wide-angle camera or a telephoto camera of the electronic device 500. Depending on the role played by the second camera 550, the two-stage "bridging" registration process can be different accordingly. For example, when the second camera 550 serves as a wide-angle camera, wide-angle distortion correction needs to be performed on the ISP and ESP processed images to adapt to the wide-angle lens, and then FOV cropping is performed to ensure that the formats of the first camera 530 and the second camera 550 are consistent.

[0284] Below, the corresponding two-stage "bridging" registration process will be exemplarily introduced when the second camera 550 serves as a front sub-primary camera, a rear sub-primary camera, a wide-angle camera and a telephoto camera of the electronic device 500, respectively.

[0285] 1. The first camera 530 serves as the primary camera of the electronic device 500, and the second camera 550 serves as the sub-primary camera of the electronic device 500. The primary camera and the sub-primary camera can be front cameras or rear cameras.

[0286] Exemplarily, referring to Figure 9A a registration process diagram is shown, which is similar to Figure 8 After the data output by the primary camera and the data output by the hybrid sensor are subjected to the two-stage "bridging" registration, high-resolution RGB image data stream and low-resolution event data stream are obtained. For details, please refer to the related content of Figure 8 , which will not be described here again.

[0287] After the electronic device 500 obtains the high-resolution RGB image data stream and the low-resolution event data stream, it can perform image cropping and image warp according to the focal length selected by the user and the result of the electric image stabilization (EIS) algorithm, etc., to obtain the to-be-processed image data. The to-be-processed image data can include to-be-processed RGB image data and to-be-processed event data.

[0288] The to-be-processed image data can be directly displayed, i.e., the electronic device 500 directly displays the to-be-processed image data in the viewfinder. For example, the electronic device 500 displays the to-be-processed RGB image data as one preview video stream and the to-be-processed event data as another preview video stream, and displays the two preview video streams on one interface to realize dual-channel preview. For another example, the electronic device 500 can display the to-be-processed RGB image data or the to-be-processed event data as a recording video stream, or generate a stylized video stream according to the to-be-processed RGB image data and the to-be-processed event data, and display the recording video stream or the stylized video stream in the recording viewfinder.

[0289] Electronic device 500 can form, as described above, based on the RGB image data to be processed and the event data to be processed. Figures 7B to 7E The stylized video or image shown can be used as an example; alternatively, the image data to be processed can be transmitted to the application algorithm, which will then process the image data before displaying and storing it. The application algorithm can include, for example, deblur algorithms, video frame interpolation algorithms, face recognition, gesture recognition, and eye tracking algorithms. In this case, face recognition, gesture recognition, and eye tracking algorithms are implemented based on RGB data and event data.

[0290] It should be pointed out that, Figure 9A The processing within the dashed box can be hardcoded onto the system-on-a-chip (SOC) of the electronic device 500 to reduce power consumption and processing latency. The processing within the dashed box can include... Figure 9A The following parts are included: First ISP, Second ISP, ESP, First Registration, Second Registration, and Image Cropping.

[0291] exist Figure 9A In this configuration, when the main camera is a rear main camera and the hybrid sensor is a rear secondary main camera, the focal length selected by the user can be, for example, 0.8X to 3.5X. In this case, the application algorithms can include deblur algorithms and video frame interpolation algorithms, etc. When the main camera is a front main camera and the hybrid sensor is a front secondary main camera, the application algorithms can include, in addition to deblur algorithms and video frame interpolation algorithms, algorithms such as face recognition, gesture recognition, and eye tracking based on RGB data and event data.

[0292] exist Figure 9A In this configuration, when the main camera is the front-facing main camera and the hybrid sensor is the front-facing secondary main camera, if the front-facing main camera operates at a non-wide-angle focal length (e.g., 0.8X to 2X), it is not necessary to add algorithms for adapting to wide-angle lenses in the second ISP and ESP. If the front-facing main camera operates at a wide-angle focal length (e.g., 0.6X to 0.8X) and is a wide-angle camera, algorithms such as RGB wide-angle distortion correction can be adaptively added in the first and second ISPs. In the ESP, algorithms such as event wide-angle distortion correction, lens shading correction (LSC), and time-stamp correction (TSC) can be added to adapt to wide-angle lenses.

[0293] 2. The first camera 530 serves as the main camera of the electronic device 500, and the second camera 550 serves as the wide-angle camera of the electronic device 500.

[0294] At this time, the electronic device 500 works in the main focus segment and the wide-angle focus segment, and the processing process is different accordingly.

[0295] Exemplarily, referring to Figure 9B Another registration process schematic diagram is shown. After the data output by the main camera and the data output by the hybrid sensor are subjected to two-segment "bridging" registration, high-resolution RGB image data stream and low-resolution event data stream can be obtained. For details, refer to Figure 8 , which will not be described here again.

[0296] In Figure 9B , the user selects the focus segment of 0.8X-3.5X, that is, in the main focus segment. At this time, after the second RGB image data stream is processed by the second ISP, a wide-angle FOV clipping operation is performed to clip the redundant picture content in the wide-angle RGB image. In addition, a wide-angle distortion correction algorithm is added in the second ISP to adapt to the wide-angle lens. In addition, a wide-angle distortion correction algorithm, an LSC algorithm, and a TSC algorithm for event data are added in the ESP to adapt to the wide-angle lens. In addition, after the event data stream is processed by the ESP, event wide-angle FOV clipping is performed to clip the picture beyond the main camera FOV. Based on this, after the two-segment "bridging" registration process, the same frame high-resolution RGB image data stream and low-resolution event data stream can be obtained. The process after obtaining the same frame RGB image data stream and event data stream is similar to that of Figure 9A , which will not be described here again.

[0297] Exemplarily, referring to Figure 9C Another registration process schematic diagram is shown. The user selects the focus segment of 0.6X-0.8X, that is, in the wide-angle focus segment. In the wide-angle focus segment, the electronic device 500 can only start the hybrid sensor, and after processing the second RGB image data stream and the event data stream output by the hybrid sensor, according to the user-selected focus segment and the EIS algorithm result, image anti-shake clipping and warp and other operations are performed to obtain the high-resolution RGB image data stream and the event data stream registered in the wide-angle focus segment.

[0298] At this time, in order to adapt to the wide-angle lens, the RGB image wide-angle distortion correction algorithm can be added in the second ISP, and the wide-angle distortion correction, LSC, and TSC algorithms for event data can be added in the ESP. It can be understood that the second RGB image data and the event data stream output by the hybrid sensor are registered, and therefore, no additional two-segment "bridging" registration process is needed for image registration.

[0299] 3. The first camera 530 is the main camera of the electronic device 500, and the second camera 550 is the telephoto camera of the electronic device 500.

[0300] When the Hybrid sensor works as the telephoto camera, the process similar to Figure 9B and Figure 9C may be used to first perform two-stage "bridging" registration of the high-resolution RGB data and the low-resolution event data, and then apply the algorithm, which will not be described here.

[0301] For example, when working in the main camera focal length, similar to Figure 9B , the first RGB image data stream output by the main camera and the event data output by the Hybrid sensor are registered through a two-stage "bridging" registration process to obtain a registered high-resolution RGB image data stream and a low-resolution event data stream; and since the Hybrid sensor works as the telephoto camera, the second ISP and ESP can increase the related algorithm to adapt to the telephoto lens to perform FOV cropping and other operations on the first RGB image data stream.

[0302] When working in the telephoto focal length, similar to Figure 9C , under the telephoto focal length, only the Hybrid sensor can be started, and the second RGB image data stream and the event data stream output by the Hybrid sensor are processed, and then according to the focal length selected by the user and the EIS algorithm result, image anti-shake cropping and warp and other operations are performed to obtain a registered high-resolution RGB image data stream and an event data stream under the telephoto focal length. At this time, in order to adapt to the telephoto lens, the corresponding algorithm can be added in the second ISP and ESP. The second RGB image data and the event data stream output by the Hybrid sensor are registered, so there is no need to perform image registration through the two-stage "bridging" registration process.

[0303] For example, when operating at a lower focal length of a telephoto lens (e.g., 3.5-10X), the electronic device 500 can simultaneously activate the first camera 530 and the second camera 550 to acquire first RGB image data, second RGB image data, and event data. Based on the telephoto focal length, the first RGB image data is cropped to obtain cropped first image data. The cropped first RGB image data and the second RGB image data are then registered using binocular RGB matching to obtain a registration matrix. Next, based on the registration matrix, the cropped first RGB image data and the event data are registered to obtain registered event data and registered first RGB image data. Finally, the registered first RGB image data, registered event data, and second RGB image data are fused to obtain fused image data. This fused image data shows significant improvements in color and noise clarity, and eliminates motion blur. In other words, by fusing event data, RGB image data can be deblurred. This application's embodiments employ a two-stage "bridging" registration method, which not only reduces registration deviation but also incorporates the high temporal resolution and motion blur-free characteristics of event data into related algorithms such as deblur and video frame interpolation. It also addresses the issues of poor registration between event data and high-resolution RGB data, as well as the difficulty of autofocusing with a standalone event camera. Furthermore, using a hybrid sensor as a front-facing secondary main camera allows the characteristics of event data to be utilized in algorithms such as face recognition, gesture recognition, and eye tracking, improving the user experience.

[0304] As mentioned above, the electronic device 500 can reduce registration deviation through two-stage "bridging" registration. However, if the second image data and event data output by the hybrid sensor are not synchronized in time, and the first camera 530 and the second camera 550 are not synchronized in time, the image data after two-stage "bridging" registration will still have a certain registration deviation.

[0305] To further reduce registration deviation, embodiments of this application can synchronize the second image data and event data output by the Hybrid sensor in time, and synchronize the first camera 530 and the second camera 550 in time.

[0306] Time synchronization methods can be divided into soft synchronization and hard synchronization. This application embodiment uses hard synchronization to synchronize the second image data and event data.

[0307] In some embodiments, hard synchronization of the second image data and event data can be performed externally to the hybrid sensor.

[0308] The Hybrid sensor includes a target pixel part and an event pixel part. The target pixel part can include, but is not limited to, a target pixel array, a target pixel readout circuit, and the like. The event pixel part can include, but is not limited to, an event pixel array and an event pixel array readout circuit.

[0309] Specifically, an interface (for example, a GPIO interface) can be opened in the target pixel part. Based on the interface, the target pixel part and the event pixel part are connected through an external circuit to achieve hard synchronization of the second image data and the event data. The external circuit is arranged outside the Hybrid sensor.

[0310] At this time, the target pixel part and the event pixel part can be regarded as two independent image sensors. The target pixel part and the event pixel part are connected through the external circuit, which is equivalent to connecting the two independent image sensors in series. In addition, the clock signals of the target pixel part and the event pixel part are different. In a specific implementation, hard synchronization can be achieved through a master-slave synchronization mode, that is, the target pixel part is a master device, and the event pixel part is a slave device.

[0311] In other embodiments, the second image data and the event data can also be hard synchronized inside the Hybrid sensor. At this time, a frame readout controller inside the Hybrid sensor can give a frame synchronization signal according to the exposure time of the target pixel and the readout mode of the event data, and transmit the frame synchronization signal to a time stamping circuit in the event pixel readout circuit through a hardware connection line, so as to trigger the time stamping circuit to stamp an additional time stamp. At this time, the frame synchronization signal serves as a standard time point for hardware synchronization.

[0312] Exemplarily, referring to Figure 10A Another internal framework diagram of the Hybrid sensor is shown. The Hybrid sensor 100 can include, but is not limited to, a target pixel array 101, a column readout controller 102, a row readout controller 103, a target pixel readout circuit 104, a frame readout controller 105, a pull-up circuit 106, an event pixel readout circuit 107, an event pixel array 108, a column readout controller 109, and a row readout controller 110.

[0313] The frame readout controller 105 is connected to the pull-up circuit 106 through a hardware connection line, and the pull-up circuit 106 is connected to the event pixel readout circuit 107 through a hardware connection line.

[0314] The frame readout controller 105 outputs a frame synchronization signal according to the exposure time of the target pixel; the frame synchronization signal is transmitted to the pull-up circuit 106 through a hardware connection line; the pull-up circuit 106 outputs a hard synchronization signal after pulling up the voltage of the frame synchronization signal; the hard synchronization signal is transmitted to the time stamping circuit of the event pixel readout circuit 107 through a hardware connection line, and the time stamping circuit prints a time stamp under the triggering of the hard synchronization signal and marks it as a special trigger (External trigger) time stamp and an additional time stamp, that is, the time stamping circuit can print an additional time stamp in response to the hard synchronization signal.

[0315] Generally, the frame synchronization signal voltage given by the frame readout controller 105 is lower than the high level of the valid hard synchronization signal, so the pull-up circuit 106 is needed to pull up the voltage of the frame synchronization signal. Of course, in some cases, if the voltage of the frame synchronization signal is equal to or higher than the high level of the valid hard synchronization signal, the pull-up circuit 106 can not be needed to pull up the voltage of the frame synchronization signal.

[0316] It is worth pointing out that, in the embodiment of the present application, the frame readout controller leads out the frame synchronization signal, takes the frame synchronization signal as the standard moment of hardware synchronization, and transmits the frame synchronization signal to the time stamping circuit of the event pixel through a hardware connection line to trigger the time stamping circuit to print an additional time stamp, so as to realize the hard synchronization inside the Hybrid sensor. Compared with realizing the hard synchronization outside the Hybrid sensor through an external circuit, the embodiment of the present application realizes the hard synchronization inside the Hybrid sensor, which can make the stability of the peripheral circuit of the Hybrid sensor better, and because the target pixel part and the event pixel part use the same sensor clock signal, the time synchronization effect is better and the time synchronization rate is faster.

[0317] It can be understood that the target pixel array 101 includes at least one target pixel. The target pixel is exemplarily an RGB pixel.

[0318] The column readout controller 102 and the row readout controller 103 are used to realize the exposure operation and the readout operation of the target pixel. For example, the column readout controller 102 and the row readout controller 103 can respectively control the exposure and readout of the pixels in each column and each row by controlling the closing and opening of the column switch and the row switch according to the sensor internal clock signal and the exposure time.

[0319] The target pixel readout circuit 104 is configured to read out the second image data after the target pixel is exposed to the light. For example, the column readout controller 102 and the row readout controller 103 control the column switch and the row switch to be closed, and the target pixel is exposed to the light to generate a photo current. The photo current is read out by the target pixel readout circuit 104 to obtain the second image data.

[0320] The frame readout controller 105 can be configured to read out the image frame according to the frame rate of the target pixel, in addition to sending the frame synchronization signal according to the exposure time of the target pixel to realize the hard synchronization.

[0321] The event pixel array 108 includes at least one event pixel. The column readout controller 109 and the row readout controller 110 are configured to realize the readout operation of the event pixel, and the event pixel readout circuit 107 is configured to read out the event of the event pixel to obtain an event data stream. For example, the column readout controller 109 is configured to control the opening and closing of the column switch, and the row readout controller 110 is configured to control the closing and opening of the row switch. When the column switch and the row switch are closed, the event pixel is exposed to the light to generate a photo current. The photo current is read out by the event pixel readout circuit 107 to obtain the event data stream. The readout mode can be asynchronous readout or synchronous readout, which will not be described here.

[0322] It can be known from Figure 3A , Figure 3B and Figure 4A that the Hybrid sensor can be realized by spatial mixing, or can be realized by customizing special pixels. When the Hybrid sensor is realized by customizing special pixels, one pixel serves as both an event pixel and a target pixel. Therefore, the target pixel array 101 and the event pixel array 108 can be the same at this time.

[0323] It should be noted that the target pixel readout circuit in a broad sense can include the column readout controller 102, the row readout controller 103, the frame readout controller 105, and the target pixel readout circuit 104, Figure 10A and the division is only for the convenience of description. Similarly, the event pixel readout circuit in a broad sense can include the column readout controller 109, the row readout controller 110, and the event pixel readout circuit 107, Figure 10A and the division is only for the convenience of description.

[0324] It can be known from Figure 3C and Figure 4BIt can be seen that the event pixel readout circuit 107 may include, but is not limited to: an ON comparison circuit, an OFF comparison circuit, a Reset circuit, a timestamp circuit, and an address circuit. When hard synchronization is implemented inside the hybrid sensor, the timestamp circuit, in addition to timestamping the event signal, also timestamps an additional time based on the hard synchronization signal and outputs the additional time stamp.

[0325] For example, taking the target pixel as RGB pixels and the second image data as RGB data as an example, see [link to example]. Figure 10B The diagram shown is a readout circuit schematic. Figure 3C Based on this, the RGB frame synchronization signal from the frame readout controller is transmitted to the pull-up circuit via a hardware connection. The pull-up circuit boosts the voltage of the RGB frame synchronization signal and outputs a hard synchronization signal. The hard synchronization signal is then transmitted to the timestamping circuit via a hardware connection. Triggered by the hardware synchronization signal, the timestamping circuit adds additional timestamp information and outputs the additional timestamp. At this point, not only is the event data stream output, but also the additional timestamp.

[0326] For example, taking the target pixel as RGB pixels and the second image data as RGB data, see [reference needed]. Figure 10C Another readout circuit diagram is shown, in Figure 4B Based on this, the timestamp circuit responds to the hard synchronization signal by adding additional timestamp information and outputs the additional timestamp.

[0327] It should be pointed out that, Figure 10B and Figure 3C The similarities, and Figure 10C and Figure 4B For similarities, please refer to the above text. Figure 3C and Figure 4B The relevant content will not be repeated here.

[0328] It is understandable that the ON comparator circuit, OFF comparator circuit, and Reset circuit are circuits present in each event pixel, while the timestamp circuit, address circuit, and pull-up circuit are circuits shared by multiple pixels. The address circuit is optional; when the event readout method is asynchronous, the address circuit is needed to assign address information to each event signal; when the event readout method is synchronous, the address circuit is not needed to assign address information to each event signal.

[0329] Generally, the time stamping circuit can time stamp the positive event signal or the negative event signal with time stamp information, for example, t in the four-dimensional event data packet (x, y, p, t). When the hard synchronization is implemented internally in the Hybrid sensor, the time stamping circuit additionally time stamps a special trigger time stamp as an extra time stamp when receiving the hard synchronization signal. It can be understood that the extra time stamp is different from t in the four-dimensional event data packet (x, y, p, t).

[0330] The extra time stamp can include but is not limited to extra time stamp marker information and time stamp information. The extra time stamp marker information can represent that the data is an extra time stamp, that is, through which information it can be determined which data is an extra time stamp.

[0331] The extra time stamp can be used to time-synchronize and align the second image data and the event data. Generally, after the SOC of the electronic device 500 receives the second image data, the event data and the extra time stamp, the SOC can time-synchronize and align the event data and the second image data according to the extra time stamp.

[0332] The extra time stamp can be embedded in the event data stream and transmitted to the SOC of the electronic device 500 along with the event data stream. For example, when the event readout mode is asynchronous readout, the extra time stamp can be inserted into the asynchronous event data stream in chronological order and transmitted to the SOC along with the asynchronous event data stream; when the event readout mode is synchronous readout, the extra time stamp can be inserted into the event frame and transmitted to the SOC along with the synchronous event frame data stream. Specifically, the extra time stamp can be inserted into the frame header or the frame tail of the event frame, and the position of the extra time stamp in the event frame is not limited herein.

[0333] Of course, the extra time stamp can also not be embedded in the event data stream and transmitted to the SOC along with the event data stream, but transmitted to the SOC through the interface between the Hybrid sensor and the SOC, at which time the interface used to transmit the extra time stamp is different from the interface used to transmit the event data stream. For example, the event data stream is transmitted through the MIPI interface, and the extra time stamp is transmitted through the I2C interface.

[0334] Compared with transmitting the extra time stamp to the SOC independently of the event data stream, inserting the extra time stamp into the asynchronous event data stream or the synchronous event frame so that the extra time stamp is transmitted to the SOC along with the event data stream can make the extra time stamp transmitted to the SOC more timely and the SOC waiting time less.

[0335] Specifically, if the Hybrid sensor transmits the additional timestamp to the SOC through an additional interface (for example, an I2C interface), because the resolution of the event data is high and the transmission rate of the I2C interface is lower than that of the event data, after the SOC obtains the event data, the SOC needs to wait for the additional timestamp transmitted through the I2C interface. If the additional timestamp is inserted into the event data stream, when the SOC receives the event data, the SOC also receives the additional timestamp, and the waiting time is less.

[0336] In other words, the equivalent frame rate of the event data is very high, but the data transmission frequency of the I2C interface and the like is much lower than the frame rate of the event data. If the additional timestamp is not embedded in the event data stream, after the backend algorithm receives a period of event data and second image data, the backend algorithm needs to wait for the additional timestamp transmitted by the I2C interface and the like before aligning the second image data and the event data. In this way, a large time delay exists in many application algorithms, and real-time processing cannot be performed.

[0337] As shown in Figure 10A , the frame readout controller 105 can issue a frame synchronization signal according to an exposure time, and perform image frame readout according to a frame rate. The exposure time and the frame rate and the like are setting parameters of the Hybrid sensor, and the setting parameters can be pre-set. When the Hybrid sensor is working, the Hybrid sensor can work according to the setting parameters of the Hybrid sensor, and output second image data, event data, and an additional timestamp and the like.

[0338] The setting parameters of the Hybrid sensor can be set through a Hybrid sensor setting, and the Hybrid sensor setting is issued by the SOC of the electronic device 500. At this time, the SOC can issue different Hybrid sensor settings as needed to adjust the setting parameters of the Hybrid sensor, and further adjust the working process and output result of the Hybrid sensor.

[0339] Exemplarily, referring to Figure 10D , a schematic diagram of a framework of an image system of the electronic device 500 is shown, and the image system includes the Hybrid sensor 100 and the SOC 200. The related introduction and description of the Hybrid sensor 100 can be referred to the related content of Figure 10A , and will not be described herein again. The SOC 200 can be a main control chip system of the electronic device 500, and can be integrated with a central processing unit (CPU), an ISP, and an ESP and the like. In addition, the SOC 200 can be integrated with at least one of a storage unit, an interface unit, a system clock, and a power supply unit and the like.

[0340] As shown in Figure 10D , the SOC 200 issues a control signal to the Hybrid sensor 100, which can include a Hybrid sensor setting. The Hybrid sensor setting includes but is not limited to the following control parameters: exposure time and exposure mode of the target pixels, and readout mode of the event pixels. The exposure mode includes global shutter (GS) and rolling shutter (RS), and the exposure time includes exposure start time and exposure end time. The readout mode includes asynchronous readout and synchronous readout.

[0341] After receiving the control signal issued by the SOC 200, the Hybrid sensor 100 parses the control instruction, obtains the corresponding control parameters, and controls the Hybrid sensor to work according to the corresponding control parameters, outputting second image data, event data, and additional time stamps, etc.

[0342] Exemplarily, after parsing the control signal, the exposure time, exposure mode, and readout mode are obtained. According to the exposure time and exposure mode, the exposure of the target pixel array 101 is controlled, and the target pixel readout circuit 104 is controlled to perform readout operation to obtain the second image data; according to the readout mode, the event pixel readout circuit 107 is controlled to perform event readout to obtain the event data. In addition, the frame readout controller 105 also outputs a frame synchronization signal according to the exposure time and exposure mode, etc. to trigger the time stamping circuit in the event pixel readout circuit 107 to stamp the additional time stamp.

[0343] In order to better receive the scheme of implementing hard synchronization inside the Hybrid sensor, the following will be exemplarily introduced and explained in combination with Figure 11 . At this time, the target pixels are RGB pixels, and the second image data is RGB data.

[0344] Referring to Figure 11 another frame schematic diagram of the image system of the electronic device 500 is shown, which can include a Hybrid sensor 1101 and a SOC 1102. The Hybrid sensor 1101 and the SOC 1102 transmit RGB data and event data through two MIPI interfaces respectively, and the SOC 1102 transmits a control signal to the Hybrid sensor 1101 through an I2C interface.

[0345] Among them, the Hybrid sensor 1101 includes an RGB pixel part and an event pixel part.

[0346] The RGB pixel section includes, but is not limited to: RGB pixel array 1103, column readout controller 1104, row readout controller 1105, frame readout controller 1106, AMP amplifier 1107, ADC 1108, on-chip ISP 1109, and MIPI transmitter 1110. The pixel array and readout circuitry of the RGB pixel section can be the same as or similar to the pixel array and readout circuitry of an RGB image sensor. The RGB pixel readout circuitry includes AMP amplifier 1107 and ADC 110.

[0347] The event pixel section includes, but is not limited to: event pixel array 1113, column readout controller 1114, row readout controller 1115, event pixel readout circuit 1112, on-chip ESP 1116, and MIPI transmitter 1117.

[0348] Both the RGB pixel portion and the event pixel portion are controlled by the sensor clock inside the hybrid sensor, but the event pixel portion may multiply the sensor clock, meaning that the clocks for the RGB pixel portion and the event pixel portion are the same.

[0349] The RGB pixel portion and the event pixel portion are connected via a pull-up circuit 1111. Specifically, the frame readout controller 1106 is connected to the pull-up circuit 1111 via a hardware connection line, and the pull-up circuit 1111 is connected to the event pixel readout circuit 1112 via a hardware connection line, specifically, it can be connected to the timestamping circuit in the event pixel readout circuit 1112. For a detailed description of the event pixel readout circuit 1112, please refer to the preceding text. Figure 10B and Figure 10C This will not be elaborated upon here.

[0350] SOC 1102 may include a MIPI receiver 1118, a hardened ESP 1119, a MIPI receiver 1120, a hardened ISP 1121, a software algorithm 1122, and an application algorithm 1123.

[0351] The control command register 1124 of the hybrid sensor 1101 can receive control signals from the SOC 1102 and, based on these signals, control the operation of each unit within the hybrid sensor to output RGB data, event data, and additional timestamps. The control signals can be transmitted from the SOC 1102 to the control command register 1124 via the I2C interface.

[0352] Exemplarily, the control signal issued by the SCO 1102 can include at least one of the following control parameters: Hybrid sensor setting, on-chip ISP algorithm parameter or on-chip ESP algorithm parameter, MIPI transmission parameter of RGB data, MIPI transmission parameter of event data, and One Time Programmable (OTP) information, etc. The Hybrid sensor setting can include but is not limited to: RGB readout parameter and event readout parameter.

[0353] The RGB readout parameter can include: RGB exposure mode, exposure time, RGB gain, and bit number, etc. The exposure time can include exposure start time and exposure end time.

[0354] The event readout parameter can include: event readout mode, filtering parameter of each event pixel analog end, comparison threshold, refractory period of Reset circuit, and hard synchronization signal trigger mode and trigger time, etc.

[0355] After receiving the control signal, the Hybrid sensor 1101 can parse the control signal, obtain the instruction code and instruction address corresponding to each control parameter, and transmit the instruction code corresponding to each control parameter to each unit according to the instruction address, so as to control each unit to work according to the control parameter.

[0356] For example, if the control signal includes RGB gain, the control instruction register 1124 transmits the instruction code corresponding to the RGB gain to the AMP amplifier 1107 according to the instruction address, and the AMP amplifier 1107 outputs the result according to the issued RGB gain in response to the instruction code, so that the RGB gain of the AMP amplifier 1107 can be controlled according to the control signal.

[0357] Similarly, if the control signal includes bit number, the control instruction register 1124 transmits the instruction code corresponding to the bit number to the ADC 1108 according to the instruction address, and the ADC 1108 performs analog-to-digital conversion according to the issued bit number in response to the instruction code, so that the bit number of the ADC 1108 can be controlled according to the control signal.

[0358] If the control signal includes the on-chip ISP algorithm parameters and the MIPI transmission parameters of the RGB data, the control instruction register 1124 transmits the instruction code corresponding to the on-chip ISP algorithm parameters to the on-chip ISP 1109 and the instruction code corresponding to the MIPI transmission parameters of the RGB data to the MIPI sending end 1110 according to the instruction address; the on-chip ISP 1109 runs the corresponding algorithm according to the on-chip ISP algorithm parameters in response to the instruction code; and the MIPI sending end 1110 controls the MIPI packaging mode, MIPI transmission mode and transmission rate of the RGB data stream according to the MIPI transmission parameters in response to the instruction code.

[0359] If the control signal includes the event readout parameters, the control instruction register 1124 transmits the instruction code corresponding to the event readout parameters to the event pixel readout circuit 1112 according to the instruction address; the event pixel readout circuit 1112 controls the set threshold of the ON comparison circuit and the OFF comparison circuit in the event pixel readout circuit 1112, controls the sleep time (i.e. reset time) of the Reset circuit in the event pixel readout circuit 1112, and controls the filtering parameter of each event pixel at the analog end, which is exemplified as the upper and lower cutoff frequencies of the band-pass filter, in response to the instruction code.

[0360] In addition, the time stamping circuit in the event pixel readout circuit 1112 also prints an additional time stamp under the triggering of the hard synchronization signal. The hard synchronization signal is transmitted to the time stamping circuit by the pull-up circuit 1111 through the hardware connection line, and the hard synchronization signal is a signal output by the pull-up circuit after voltage pulling of the frame synchronization signal introduced by the frame readout controller 1106.

[0361] If the control signal includes the on-chip ISP algorithm parameters and the MIPI transmission parameters of the RGB data, the control instruction register 1124 transmits the instruction code corresponding to the on-chip ISP algorithm parameters to the on-chip ISP 1109 and the instruction code corresponding to the MIPI transmission parameters of the RGB data to the MIPI sending end 1110 according to the instruction address; the on-chip ISP 1109 runs the corresponding algorithm according to the on-chip ISP algorithm parameters in response to the instruction code; and the MIPI sending end 1110 controls the MIPI packaging mode, MIPI transmission mode and transmission rate of the RGB data stream according to the MIPI transmission parameters in response to the instruction code.

[0362] The on-chip ISP1109 can integrate on-chip ISP algorithm pipes. For example, the on-chip ISP1109 integrates on-chip dead pixel correction (DPC) algorithm, remoscia algorithm or demoscia algorithm, and automatic white balance (AWB) algorithm.

[0363] In this embodiment, the DPC algorithm on the on-chip ISP 1109 may differ from the DPC algorithm of a traditional RGB image sensor. Specifically, in the hybrid sensor, the on-chip ISP 1109 runs both a DPC algorithm for event pixel occupancy and a DPC algorithm for traditional RGB pixels. The DPC algorithm for event pixel occupancy and the DPC algorithm for traditional RGB pixels can be implemented sequentially or in parallel.

[0364] It should be noted that if the RGB pixels and event pixels of the hybrid sensor are arranged in parallel, meaning that the event pixels periodically occupy RGB pixels in space, the on-chip ISP algorithm will add provisions to address the missing RGB pixels caused by event pixel occupation. In this case, each event pixel is considered a dead pixel for the RGB pixels, therefore, a DPC algorithm will be implemented within the on-chip ISP to address the occupation of event pixels.

[0365] The on-chip ESP1116 can integrate an on-chip ESP algorithm pipe. In this case, the on-chip ESP algorithm can, for example, include algorithms such as event pixel DPC (dead / hot pixel correction), event rate control, and event compression. That is, different modes and related algorithm parameters in the on-chip ESP algorithm pipe within the hybrid sensor can be controlled via control signals sent through I2C.

[0366] like Figure 11 As shown, the RGB data and event data output by the Hybrid sensor 1101 are transmitted to the SOC 1102 via MIPI. The SOC 1102 receives the event data sent by the MIPI transmitter 1117 via the MIPI receiver 1118. This event data can be a synchronous event frame data stream or an asynchronous event data stream. At this time, the event data includes an additional timestamp, which is inserted into the asynchronous event data stream or the synchronous event frame. The SOC 1102 also receives the RGB data stream sent by the MIPI transmitter 1110 via the MIPI receiver 1120.

[0367] After the MIPI receiver 1118 receives the event data, the event data is transmitted to the hardened ESP 1119 for data processing. Similarly, after the MIPI receiver 1120 receives the RGB event, the RGB event is transmitted to the hardened ISP 1121 for data processing.

[0368] In the embodiments of the present application, the hardened ESP 1119 and the hardened ISP 1121 can perform denoising and flicker removal on the RGB data and the event data, respectively, to improve the signal quality of the RGB data and the event data. In addition, the functions of the hardened ESP 1119 and the hardened ISP 1121 can also be implemented by the software ESP and the software ISP.

[0369] In the embodiments of the present application, the hardened ESP 1119 and the hardened ISP 1121 can also transmit relevant data to each other, which can be used as an algorithm input of the ISP algorithm or the ESP algorithm. The relevant data can include, for example, the statistical results of the RGB image and the statistical results of the event data.

[0370] After the software algorithm 1122 receives the RGB data processed by the hardened ISP 1121 and the event data processed by the hardened ESP 1119, the software algorithm 1122 performs 3A algorithm on the processed RGB data and the processed event data to calculate the Hybrid sensor setting, and transmits the Hybrid sensor setting to the Hybrid sensor 1101 through the I2C interface according to a certain setting timing. The Hybrid sensor 1101 responds to the Hybrid sensor setting and outputs the RGB data and the event data. The Hybrid sensor setting can include RGB exposure parameters and event camera parameters. The RGB exposure parameters can include, for example, exposure mode and exposure time. The event camera parameters can include, for example, event readout mode, readout rate, set threshold values of ON comparison circuit and OFF comparison circuit, filter parameters, time of hard synchronization trigger, and reset time of reset circuit.

[0371] The 3A algorithm can include, for example, Auto-exposure, Auto-focus, and Auto white balance algorithms.

[0372] It can be understood that the Hybrid sensor 1101 has a pre-set default Hybrid sensor setting. Generally, the Hybrid sensor 1101 first outputs the extra timestamp, the RGB data and the event data according to the default Hybrid sensor setting, and then the SOC 1102 calculates a new Hybrid sensor setting according to the RGB data and the event data, and sends the new Hybrid sensor setting to the Hybrid sensor 1101 to adjust the working process and the output result of the Hybrid sensor 1101, so as to obtain new extra timestamp, RGB data and event data.

[0373] The event data output by the Hybrid sensor 1101 includes the extra timestamp, and the software algorithm 1122 receives the extra timestamp when receiving the event data, which reduces the waiting time of the SOC. After receiving the extra timestamp, the software algorithm 1122 can determine the event data corresponding to each frame of RGB data according to the extra timestamp.

[0374] In the embodiment of the present application, the software algorithm 1122 can be used to calculate and control the related parameters of the hybrid sensor, so as to adjust the output quality of the RGB pixels and the event pixels, and obtain the RGB image data and the event image data that can be accepted by the application algorithm 1123.

[0375] The event data and the RGB data processed by the software algorithm 1122 are transmitted to the application algorithm 1123, and the application algorithm 1123 processes the event data and the RGB data correspondingly, and finally obtains the display image data and / or the stored image data. The application algorithm 1123 can be deblur, video interpolation, and face recognition, gesture recognition and eye tracking based on RGB and event data.

[0376] It can be understood that the software algorithm 1122 and the application algorithm 1123 can be applied to the processor of the SOC 1102, for example, running on the CPU and the AP processor. In addition, different algorithm modules in the hard ISP 1120, the hard ESP 1118, the software algorithm 1122 and the application algorithm 1123 may need to store a large amount of data in the calculation process, so the double data rate (DDR) memory in the SOC can be called online or offline for data storage or data transfer during the algorithm processing process. The SOC 1102 can also adaptively include power supply and system clock units, and the Hybrid sensor 1101 can also include power supply units.

[0377] It should be noted that the hard synchronization alignment of the RGB data and the event data in the Hybrid sensor can not only reduce the image registration deviation, but also enable the electronic device 500 to reduce the opening of one camera, without the need for re-opening the mold, thereby reducing the cost. Meanwhile, the target pixels and the event pixels in the Hybrid sensor can share the AF and OIS devices.

[0378] In a specific application, the Hybrid sensor generally outputs the second image data at a fixed frame rate. At this time, the frame readout controller can read out the second image data at the pre-set frame rate. However, the exposure mode of the target pixels can be global exposure (Global Shutter, GS) or local exposure (Rolling Shutter, RS). In addition, the event data readout mode of the Hybrid sensor can be synchronous readout or asynchronous readout. Among them, there is no concept of frame in asynchronous readout, while the synchronous readout is to read out the entire event frame.

[0379] The time and number of the frame synchronization signals issued by the frame readout controller can also be different accordingly due to different exposure modes and different event readout modes. Generally, there are the following four cases: global exposure and synchronous readout, global exposure and asynchronous readout, local exposure and synchronous readout, and local exposure and asynchronous readout. That is, the hard synchronization strategies in different cases can be different, and the above four different cases can have four different hard synchronization strategies.

[0380] The circuits and transmission lines required by the four different hard synchronization strategies are implemented in the Hybrid sensor. The overall circuit framework in the Hybrid sensor can be different by using the spatial mixing method and using the customized special pixel method. For details, please refer to the Figure 10B and Figure 10C , which will not be repeated here.

[0381] The hard synchronization strategies in the four different cases are described below.

[0382] Case 1, global exposure and asynchronous readout.

[0383] At this time, the target pixel array adopts the global exposure mode, so the exposure start time and the exposure end time are the same for each target pixel. The frame readout controller can issue one frame synchronization signal at the exposure start time and one frame synchronization signal at the exposure end time for each frame of second image data, that is, two frame synchronization signals are issued for each frame of second image data, which represent the exposure start time and the exposure end time, respectively.

[0384] For each frame synchronization signal, the frame synchronization signal is transmitted to the pull-up circuit through the hardware connection line, the pull-up circuit outputs the hard synchronization signal after pulling up the frame synchronization signal, and the hard synchronization signal is transmitted to the time stamping circuit through the hardware connection line; the time stamping circuit prints an extra time stamp under the triggering of the hard synchronization signal. The extra time stamp can be inserted into the asynchronous event data stream in time sequence.

[0385] That is, for each frame of second image data, the time stamping circuit receives two hard synchronization signals and prints two extra time stamps in response to the two frame synchronization signals.

[0386] It should be noted that the frame synchronization signal is transmitted to the time stamping circuit through the hardware connection line, so there is a signal transmission delay and a pull-up circuit delay between the frame synchronization signal being given and the time stamping circuit receiving the hard synchronization signal. The signal transmission delay = transmission line length / light speed, for example, the transmission line length is 2 cm, and the signal transmission delay is about 2 cm / 3 * 10^8 m / s = 0.067 ns (nanosecond).

[0387] In addition to the signal transmission delay and the pull-up circuit delay, there is also a time stamping delay. The time stamping delay refers to the processing delay of the time stamping circuit in stamping the extra time stamp after receiving the hard synchronization signal.

[0388] In specific applications, transmitting the frame synchronization signal and the hard synchronization signal through the hardware connection line can ensure that the total delay between the frame synchronization signal and the hard synchronization signal is in the nanosecond (ns) level.

[0389] Exemplarily, referring to Figure 12 The hard synchronization delay in the Hybrid sensor is shown in the figure, at this time, the total hard synchronization delay = pull-up circuit delay + signal transmission delay + time stamping delay. The RGB frame synchronization signal is xvs signal, and the hard synchronization signal is sync signal.

[0390] Since the voltage of the RGB frame synchronization signal is lower than the effective high level of the hard synchronization signal, a pull-up circuit needs to be added in the middle, so there is a pull-up circuit delay. The time stamping delay is the processing delay of the time stamping circuit in stamping the extra time stamp on the event signal after receiving the sync signal.

[0391] After the time stamping circuit prints the extra time stamp under the triggering of the hard synchronization signal, since the event readout mode is asynchronous readout, the extra time stamp can be inserted into the asynchronous event data stream in time sequence and transmitted to the SOC along with the asynchronous event data stream.

[0392] Exemplarily, referring to Figure 13A schematic diagram of the hard synchronization in the Hybrid sensor, the black dots represent the event data packets (x, y, p, t) in the asynchronous event data stream output by the Hybrid sensor, and the white dots represent the extra timestamps which are the extra timestamp signals output by the timestamping circuit according to the hard synchronization signals.

[0393] The extra timestamps can be inserted into the asynchronous event data and transmitted to the SOC of the electronic device 500, or can be transmitted to the SOC of the electronic device 500 as extra metadata data packets and event data streams.

[0394] As shown in the figure, the RGB pixels in the Hybrid sensor are globally exposed, so the exposure start time and the exposure end time of all the RGB pixels are the same. And each frame of RGB data under global exposure will give two RGB frame synchronization signals, respectively representing the exposure start time and the exposure end time. Figure 13 That is, at the exposure start time of each frame of RGB data, the frame readout controller will give an RGB frame synchronization signal, which is transmitted to the pull-up circuit through the hardware connection line; the pull-up circuit outputs a hard synchronization signal after pulling up the frame synchronization signal, and transmits the hard synchronization signal to the timestamping circuit through the hardware connection line; the timestamping circuit prints an extra timestamp under the trigger of the hard synchronization signal, and then inserts the extra timestamp into the asynchronous event data stream in time sequence.

[0395] Similarly, at the exposure end time, the frame readout controller will also give an RGB frame synchronization signal, which is transmitted to the pull-up circuit through the hardware connection line; the pull-up circuit outputs a hard synchronization signal after pulling up the frame synchronization signal, and transmits the hard synchronization signal to the timestamping circuit through the hardware connection line; the timestamping circuit prints a timestamp according to the hard synchronization signal, and then inserts the extra timestamp into the asynchronous event data stream in time sequence.

[0396] In this way, each frame of RGB data under global exposure will correspond to two extra timestamps. As shown in the figure, the exposure start time and the exposure end time of the nth frame of RGB data correspond to extra timestamp 1 and extra timestamp 2 respectively; the exposure start time and the exposure end time of the n+1th frame of RGB data correspond to extra timestamp 3 and extra timestamp 4 respectively; the exposure start time and the exposure end time of the n+2th frame of RGB data correspond to extra timestamp 5 and extra timestamp 6 respectively.

[0397] Figure 13

[0398] ​​After the SOC of the electronic device 500 receives the additional time stamp, RGB data and asynchronous event data stream of the Hybrid sensor output, the SOC time synchronously aligns the RGB data and asynchronous event data according to the additional time stamp, and then can determine the event data corresponding to each frame of RGB image. For example, the event data corresponding to the nthframe of RGB image is the four-dimensional event data packet whose time is located between additional time stamp 1 and additional time stamp 2; the event data corresponding to the nth+1frame of RGB image is the four-dimensional event data packet whose time is located between additional time stamp 3 and additional time stamp 4; and the event data corresponding to the nth+2frame of RGB image is the four-dimensional event data packet whose time is located between additional time stamp 5 and additional time stamp 6.

[0399] Case 2, partial exposure and asynchronous readout.

[0400] At this time, the target pixel array adopts a partial exposure mode, and the front several rows of pixels of the target pixel may have started the exposure of the next frame when the last several rows of pixels of the target pixel have not ended the exposure. In this case, the hard synchronization strategy can be that a frame synchronization signal is issued for each frame of second image data, and the issuance time of the frame synchronization signal can be any time within the visible light image of the frame. Exemplarily, the frame readout controller can issue a frame synchronization signal when the exposure of the first row of pixels of each frame of second image data image ends.

[0401] Similar to case 1 described above, the frame synchronization signal is transmitted to the pull-up circuit through the hardware connection line, and the pull-up circuit transmits the hard synchronization signal to the time stamping circuit through the hardware connection line. The time stamping circuit prints an additional time stamp under the trigger of the hard synchronization signal. The additional time stamp can be inserted into the asynchronous event data stream or transmitted to the SOC as additional metadata data packet. In addition, the total time delay between the frame synchronization signal and the hard synchronization signal includes the signal transmission time delay, the pull-up time delay and the time stamping time delay.

[0402] Exemplarily, referring to Figure 14 Another schematic diagram of the hard synchronization in the Hybrid sensor is shown in FIG. 6. The RGB pixels of the Hybrid sensor are partially exposed, so that the front several rows of pixels of the RGB pixels may have started the exposure of the next frame when the last several rows of pixels of the RGB pixels have not ended the exposure. At this time, the frame readout controller issues an RGB frame synchronization signal when the exposure of the first row of RGB pixels ends.

[0403] That is, according to the internal clock signal of the Hybrid sensor, the frame readout controller issues an RGB frame synchronization signal when the exposure of the first row of RGB pixels in each frame of RGB image ends. The RGB frame synchronization signal is transmitted to the pull-up circuit through the hardware connection line, and the pull-up circuit outputs a hard synchronization signal after pulling up the RGB frame synchronization signal, and transmits the hard synchronization signal to the time stamping circuit of the event pixel through the hardware connection line; the time stamping circuit gives an additional time stamp to the event signal according to the hard synchronization signal. The additional time stamp can be inserted into the asynchronous event data stream.

[0404] In this case, one frame synchronization signal is issued for each frame of RGB image, so each frame of RGB image corresponds to one additional time stamp. As shown in Figure 14 , the exposure end time of the first row of the nth frame of RGB data corresponds to the additional time stamp 1, the exposure end time of the first row of the n+1th frame of RGB data corresponds to the additional time stamp 2, and the exposure end time of the first row of the n+2th frame of RGB data corresponds to the additional time stamp 3.

[0405] By comparing Figure 14 and Figure 13 , it can be seen that Figure 14 each frame of RGB image only gives one hard synchronization signal, while Figure 13 each frame of RGB image gives two hard synchronization signals.

[0406] After the SOC of the electronic device 500 receives the additional time stamp, the RGB data and the asynchronous event data stream output by the Hybrid sensor, it can calculate the exposure start time and the exposure end time of each row of RGB according to the additional time stamp, the exposure time of the RGB image frame, and the exposure interval of the adjacent two rows of RGB, and then can time-synchronize and align the RGB data and the asynchronous event data to determine the event data corresponding to each frame of RGB image. In addition, the total time delay between the RGB frame synchronization signal and the hard synchronization signal can also be considered in this calculation process (for example, as shown in Figure 12 , in addition, if the RGB image is cropped, the time delay difference of each RGB pixel and time pixel can be calculated according to the cropping ratio.

[0407] Case 3, global exposure and synchronous readout.

[0408] Unlike asynchronous readout, synchronous readout reads event frames at a higher frame rate. And each frame of event frame is a three-value graph containing +1, -1 and 0, and the event frame no longer records the coordinate information of the event; at the same time, each frame of event frame shares one or several time stamps. For example, each frame of event frame only records one time stamp as the time stamp of the whole frame of event frame, or records the time stamp representing all positive events in the event frame and the time stamp of all negative events.

[0409] In the global exposure mode, the frame readout controller can issue two frame synchronization signals in each frame of second image data, specifically, one frame synchronization signal can be issued at the exposure start time and one frame synchronization signal can be issued at the exposure end time; or one frame synchronization signal or two frame synchronization signals can be issued in a frame period, which can refer to the time period between the exposure start time of the current frame and the exposure start time of the next frame.

[0410] When one frame synchronization signal is issued in a frame period, the issuing time of the frame synchronization signal can be the exposure start time of the current frame, that is, the frame readout controller issues one frame synchronization signal at the exposure start time of each frame of second image data. At this time, the event pixel readout circuit not only prints an additional time stamp according to the hard synchronization signal, but also re-synchronizes the first frame event frame in the frame period with the hard synchronization signal. Therefore, the additional time stamp corresponding to the exposure start time is aligned with the exposure start time.

[0411] When two frame synchronization signals are issued in a frame period, the issuing time of the frame synchronization signal can be the exposure start time and the exposure end time of the current frame, that is, the frame readout controller issues one frame synchronization signal at the exposure start time of each frame of second image data and issues one frame synchronization signal at the exposure end time. At this time, the event pixel readout circuit not only prints an additional time stamp according to the hard synchronization signal, but also re-synchronizes the first frame event frame in the frame period with the hard synchronization signal corresponding to the exposure start time. Therefore, the additional time stamp corresponding to the exposure start time is aligned with the exposure start time, but the additional time stamp corresponding to the exposure end time is not necessarily aligned with the exposure end time.

[0412] Exemplarily, referring to Figure 15 Another schematic diagram of the hard synchronization in the Hybrid sensor is shown, the RGB pixels in the Hybrid sensor are globally exposed, so the exposure start time and the exposure end time of all the RGB pixels are the same.

[0413] In an implementation mode, two RGB frame synchronization signals are given for each frame of RGB data in the global exposure mode, which respectively represent the exposure start time and the exposure end time, and an additional time stamp is printed according to the two RGB frame synchronization signals.

[0414] As shown in Figure 15 For each frame of RGB data, the frame readout controller issues one RGB frame synchronization signal at the RGB exposure start time and one RGB frame synchronization signal at the RGB exposure end time according to the internal clock signal of the Hybrid sensor. Each RGB frame synchronization signal is transmitted to the pull-up circuit through the hardware connection line, the pull-up circuit outputs the hard synchronization signal after pulling up the RGB frame synchronization signal, and the hard synchronization signal is transmitted to the time stamping circuit through the hardware connection line.

[0415] The timestamping circuit can print an extra timestamp in real time in response to the hard synchronization signal, and register the extra timestamp, which is in a state of waiting to be read out. In the next frame event frame, the registered extra timestamp is inserted into the event frame.

[0416] Of course, the timestamping circuit can also not print an extra timestamp in real time, but wait for the next frame event frame after receiving the hard synchronization signal; in response to the hard synchronization signal when the next frame event frame arrives, print an extra timestamp, and insert the extra event timestamp into the next frame event frame.

[0417] It should be noted that in the process of inserting the extra timestamp into the event frame, if there is currently an extra timestamp that has not been read out, and the current event frame is being read out, the extra timestamp can be inserted at the end of the current event frame; if there is currently an extra timestamp that has not been read out, and the current event frame has been read out, the extra timestamp can be added to the beginning of the next frame event frame, so that the extra timestamp is transmitted to the SOC along with the synchronous event frame data stream.

[0418] As shown in FIG. 6, the synchronous event frame data stream includes a plurality of event frames, wherein each large rectangle represents an event frame, and each large rectangle includes four small rectangles. An RGB frame synchronization signal is issued at the exposure start time and the exposure end time of the nth frame of RGB data, the nth+1 frame of RGB data, and the nth+2 frame of RGB data. The timestamping circuit prints an extra timestamp according to the hard synchronization signal, and adds the extra timestamp to the beginning of the next event frame, as shown in extra timestamp 1 to extra timestamp 6. Figure 15 After the SOC of the electronic device 500 receives the extra timestamp, the RGB data and the synchronous event frame data stream output by the hybrid sensor, the RGB data and the synchronous event frame are time-synchronized and aligned according to the extra timestamp, and then the event data corresponding to each frame of RGB image is determined. For example, the event data corresponding to the nth frame of RGB image is the event frame whose time is between extra timestamp 1 and extra timestamp 2; the event data corresponding to the nth+1 frame of RGB image is the event frame whose time is between extra timestamp 3 and extra timestamp 4; and the event data corresponding to the nth+2 frame of RGB image is the event frame whose time is between extra timestamp 5 and extra timestamp 6.

[0419]

[0420] ​In another implementation, for each frame of RGB data, a RGB frame period is calculated from the exposure start time of the frame to the exposure start time of the next frame. In this case, a RGB frame sync signal is sent at a fixed time, and then the event camera of the Hybrid sensor is synchronized to output a fixed number of frames within the RGB frame period, which can be controlled by the Hybrid sensor setting and can be changed by the backend algorithm.

[0421] The fixed time can exemplarily be the exposure start time, or the exposure start time and the exposure end time.

[0422] That is, in this case, a RGB frame sync signal can be sent only at the exposure start time, or a RGB frame sync signal can be sent at the exposure start time and the exposure end time respectively. Therefore, there can be only one extra timestamp in a RGB frame period, or there can be two extra timestamps in a RGB frame period.

[0423] When a RGB frame sync signal is sent only at the exposure start time, there can be only one extra timestamp in a RGB frame period. As shown in FIG. 6, Figure 15 in a RGB frame period, a RGB frame sync signal is sent at the exposure start time of the nth frame of RGB image, and no RGB frame sync signal is sent at the exposure end time of the nth frame of RGB image, so that only the extra timestamp 1 is included in the frame period. In the next RGB frame period, a RGB frame sync signal is sent at the exposure start time of the (n+1)th frame of RGB image, and no RGB frame sync signal is sent at the exposure start time of the (n+1)th frame of RGB image, so that only the extra timestamp 3 is included in the frame period. Similarly, a RGB frame sync signal is sent only at the exposure start time of the (n+2)th frame of RGB image, so that only the extra timestamp 5 is included in the frame period.

[0424] When a RGB frame sync signal is sent at the exposure start time and the exposure end time respectively, there can be two extra timestamps in a RGB frame period. As shown in FIG. 7, Figure 15 in a RGB frame period, a RGB frame sync signal is sent at the exposure start time and the exposure end time of the nth frame of RGB image respectively, so that the extra timestamp 1 and the extra timestamp 2 are included in the frame period. In the next RGB frame period, a RGB frame sync signal is sent at the exposure start time and the exposure end time of the (n+1)th frame of RGB image respectively, so that the extra timestamp 3 and the extra timestamp 4 are included in the frame period. Similarly, a RGB frame sync signal is sent at the exposure start time and the exposure end time of the (n+2)th frame of RGB image respectively, so that the extra timestamp 5 and the extra timestamp 6 are included in the frame period.

[0425] No matter whether one RGB frame synchronization signal or two RGB frame synchronization signals are sent in one RGB frame period, the first frame event frame in the frame period needs to be re-synchronized with the hard synchronization signal corresponding to the exposure start time. Figure 15 As shown in FIG. 8, the additional time stamp 1, the additional time stamp 3, and the additional time stamp 5 are aligned with the exposure start time of the RGB, while the additional time stamp 2, the additional time stamp 4, and the additional time stamp 6 are not necessarily aligned with the exposure end time of the RGB.

[0426] The re-synchronization refers to that if there is no event frame output at the moment corresponding to the hard synchronization signal, the event pixel readout circuit is refreshed once to ensure that there is event frame output at the moment corresponding to the hard synchronization signal, so that the additional time stamp printed according to the hard synchronization signal can be aligned with the first frame event frame in the frame period.

[0427] The process of the re-synchronization can be as follows: the comparison circuit of the event pixel is turned on in hardware, at this time, if the luminance change on part of the event pixels meets the trigger threshold of the comparison circuit, an event trigger will occur; if there is an event trigger, the event pixel readout circuit will print the time stamp of the event trigger. Even if no event pixel on the current event frame triggers, a frame time stamp will be output, for example, if there is an encoding algorithm on the sensor, the event data of the event frame will be skipped, only the frame header and the frame tail are retained, and the frame time stamp information is placed in the frame header or the frame tail, if there is no encoding algorithm on the sensor, an event image with all pixel values being 0 will be output.

[0428] It should be noted that if the frame rate of the event frame is not an integer multiple of the frame rate of the RGB frame, the period of the event frame will be unstable, that is, Figure 15 the interval time of two continuous event frames will be lengthened, and a blank waiting time will occur, as shown in the dashed box in FIG. 9. For example, if the frame rate of the RGB frame is 30 FPS and the frame rate of the event frame is 4000 FPS, the waiting time can be calculated as 4K / 30=133.33, (133.33 mod 1=0.33)*1 / 4000=0.083 ms. At this time, the image sensor can mark the waiting time and transmit it to the back-end processing algorithm.

[0429] Case 4, partial exposure and synchronous readout.

[0430] Similar to case 3, there are two hard synchronization trigger modes, the first mode is that one frame synchronization signal is output for each second image data, specifically, one frame synchronization signal can be output at the exposure end time of the first row of target pixels, or the time when the first row of target pixels starts to be read out, or a time interval from the above two times. The second mode is that one frame synchronization signal is output in the RGB frame period, and the first frame event frame in the frame period is re-synchronized with the hard synchronization signal.

[0431] Exemplarily, referring to Figure 16 FIG. 6, in another implementation, the frame readout controller sends a RGB frame sync signal at the end of the first row of pixels exposure, and the event camera of the Hybrid sensor synchronizes a fixed number of frames in the RGB frame period. The fixed number can be controlled by the Hybrid sensor setting and can be changed by the back-end algorithm. The RGB frame period is the time period between the end of the first row of pixels exposure of the current frame and the end of the first row of pixels exposure of the next frame.

[0432] In another implementation, the frame readout controller sends a RGB frame sync signal at the end of the first row of pixels exposure, and the event camera of the Hybrid sensor synchronizes a fixed number of frames in the RGB frame period. The fixed number can be controlled by the Hybrid sensor setting and can be changed by the back-end algorithm. The RGB frame period is the time period between the end of the first row of pixels exposure of the current frame and the end of the first row of pixels exposure of the next frame.

[0433] At this time, the first frame event frame in each RGB frame period is resynchronized with the hard sync signal of the RGB. As shown in Figure 16 FIG. 7, the n-th frame of the RGB image corresponds to the extra timestamp 1, the n+1-th frame of the RGB image corresponds to the extra timestamp 2, and the n+2-th frame of the RGB image corresponds to the extra timestamp 3. The extra timestamp 1, the extra timestamp 2, and the extra timestamp 3 are all aligned with the end of the first row of pixels exposure of the RGB.

[0434] In addition, there is a waiting time between the last frame event frame in the RGB frame period and the first frame event frame in the next RGB frame period, which can be shown by the dashed box at the extra timestamp 3 in Figure 16 FIG. 8.

[0435] If the frame rate of the event frame is not an integer multiple of the frame rate of the RGB frame, the interval of the waiting time can be calculated. The specific calculation method can be referred to the related content in the global exposure and synchronous readout above.

[0436] However, the extra requirement of the hard sync strategy of the global exposure and synchronous readout is that the waiting time is the same as the normal frame interval time. That is, not only in the Hybrid sensor, but also in the SOC, the frame rate of the event frame should be an integer multiple of the frame rate of the RGB frame. Figure 16 In addition, it is required that the received Hybrid sensor setting must satisfy that the frame rate of the event frame to be sent out is an integer multiple of the frame rate of the RGB frame.

[0437] It should be noted that in the above four cases of hard synchronization strategy, only one frame of image data is output in a frame period. However, if there are multiple frames in a frame period, such as staggered frame output. At this time, the calculation method of the frame period can be changed to a complete frame period containing multiple frames.

[0438] At this time, the multiple frames in a frame period can be regarded as different frames, and a frame synchronization signal is given at the exposure start time and the exposure end time of each frame, or at the exposure end time of the first row of each frame. For example, there are 2 frames in a frame period, which are long frame and short frame. At this time, the long frame and the short frame are regarded as different frames. If the exposure mode is global exposure, a frame synchronization signal can be given at the exposure start time and the exposure end time of the long frame, and a frame synchronization signal can be given at the exposure start time and the exposure end time of the short frame. At this time, there are 4 hard synchronization signals in a frame period. If the exposure mode is local exposure, a frame synchronization signal can be given at the exposure end time of the first row of pixels of the long frame, and a frame synchronization signal can be given at the exposure end time of the first row of pixels of the short frame. At this time, there are 2 hard synchronization signals in a frame period.

[0439] It should be noted that the electronic device 500 has different application modes, such as a photographing mode and a video mode (or a recording mode). In different application modes, the hard synchronization trigger mode can be different. For example, when the electronic device 500 is in the video mode (or the recording mode), the hard synchronization process shown in FIG. 6 can be used to realize the hard synchronization of the RGB data and the event data in the Hybrid sensor. Figure 16

[0440] In the photographing mode, especially in the snapshot mode, the Hybrid sensor can also have a special hard synchronization trigger mode. For example, referring to FIG. 7, which shows another schematic diagram of the hard synchronization in the Hybrid sensor, which is different from that shown in FIG. 6. Figure 17A Figure 16 At this time, two frame synchronization signals are sent out during the local exposure of each frame of RGB image. Specifically, an RGB frame synchronization signal is sent out at the exposure start time of the first row of RGB pixels, and an RGB frame synchronization signal is also sent out at the exposure end time of the last row of RGB pixels.

[0441] For example, referring to FIG. 8, which shows another schematic diagram of the hard synchronization in the Hybrid sensor, which is different from that shown in FIG. 6. Figure 17A ​​As shown, for the nth frame of RGB image, the frame reading controller issues an RGB frame synchronization signal at the exposure start time of the first pixel in the first row of the nth frame of RGB image, and also issues an RGB frame synchronization signal at the exposure end time of the last pixel in the last row of the nth frame of RGB image. Similarly, the nth+1 frame of RGB image and the nth+2 frame of RGB image also issue two RGB frame synchronization signals.

[0442] Similarly to the above cases, after the frame reading controller outputs the RGB frame synchronization signal, the RGB frame synchronization signal is transmitted to the pull-up circuit through the hardware connection line, the pull-up circuit outputs the hard synchronization signal after pulling up the RGB frame synchronization signal; the hard synchronization signal is transmitted to the time stamping circuit through the hardware connection, and the time stamping circuit prints an extra time stamp under the trigger of the hard synchronization signal. At this time, since two RGB frame synchronization signals are issued for each frame of RGB image, two extra time stamps correspond to each frame of RGB image.

[0443] As shown in Figure 17A , the two RGB frame synchronization signals of the nth frame of RGB image correspond to extra time stamp 1 and extra time stamp 3 respectively, the two RGB frame synchronization signals of the nth+1 frame of RGB image correspond to extra time stamp 2 and extra time stamp 5 respectively, and the two RGB frame synchronization signals of the nth+2 frame of RGB image correspond to extra time stamp 4 and extra time stamp 6 respectively.

[0444] Extra time stamp 1 to extra time stamp 6 can be inserted into the frame header data of the event frame, and transmitted to the SOC of the electronic device 500 along with the synchronous event frame data stream. In addition, one RGB frame period also corresponds to two extra time stamps. For example, the RGB frame period between the exposure start time of the nth frame of RGB image and the exposure start time of the nth+1 frame of RGB image corresponds to extra time stamp 1 and extra time stamp 2.

[0445] However, since it is partial exposure, one RGB frame period is not the complete exposure time of one frame of RGB image. At this time, the RGB frame period is the time period between the exposure start time of the first row of pixels of the current frame and the exposure start time of the first row of pixels of the next frame. For example, Figure 17A , the complete exposure time of the nth frame of RGB image is the time period between extra time stamp 1 and extra time stamp 3, but the RGB frame period is the time stamp between extra time stamp 1 and extra time stamp 2.

[0446] In the photographing mode, if the back-end algorithm on the SOC needs to process a certain frame of RGB image, it needs all the event frames in the complete exposure time of the frame image, rather than all the event frames in the RGB frame period.

[0447] For example, for the nth frame of the RGB image, the complete exposure time is the time period between the extra timestamp 1 and the extra timestamp 3, so all the event frames between the extra timestamp 1 and the extra timestamp 3 are needed, instead of all the event frames between the extra timestamp 1 and the extra timestamp 2.

[0448] The hard synchronization signal triggering mode will also change accordingly under different application modes. The hard synchronization signal triggering mode can also be controlled by the SOC. At this time, the SOC can issue different Hybrid sensor settings to the Hybrid sensor according to different application modes, and then the Hybrid sensor switches accordingly after receiving the Hybrid sensor settings. For example, the electronic device 500 is in a video shooting mode at a certain moment, at which time the Hybrid sensor can perform hard synchronization in the manner shown in Figure 16 to output RGB data and event data, that is, an RGB frame synchronization signal is issued within each frame of the RGB image; at the next moment, the user needs to switch from the video shooting mode to the shooting mode, which can be achieved by clicking the corresponding control. For example, referring to the scene shown in Figure 7A the user can click the "shooting" mode control in the interface 713 to switch from the video shooting mode to the shooting mode.

[0449] After detecting the switching instruction, the electronic device 500 generates a corresponding Hybrid sensor setting and issues the Hybrid sensor setting to the Hybrid sensor. The Hybrid sensor responds to the issued Hybrid sensor setting and performs hard synchronization triggering in the manner shown in Figure 17A to output RGB data and event data.

[0450] Generally, the Hybrid sensor setting issued in the nth frame of the RGB image takes effect at the n+2th frame of the RGB image.

[0451] For example, referring to the switching timing diagram shown in Figure 17B at this time, if the user switches from the video mode to the shooting mode before the nth frame of the RGB image starts exposure, the hard synchronization triggering mode also changes. Based on this, the SOC generates a corresponding Hybrid sensor setting in the manner shown in Figure 17A and issues the Hybrid sensor setting to the Hybrid sensor through I2C.

[0452] After the Hybrid sensor acquires the Hybrid sensor setting, the Hybrid sensor issues an instruction to the RGB pixel part and the event pixel part within the nth frame RGB exposure time. The instruction can be issued within the complete exposure time of the nth frame RGB, generally within the time interval from the end of the exposure of the first row to the end of the exposure of the last row, or after a fixed time interval after the exposure of the first pixel of the first row. At this time, since the first row of the n+1th frame RGB can have started exposure, the n+1th frame RGB will not respond to the setting within the frame period, but will switch the setting within the frame period of the n+2th frame RGB. That is, the time of the arrow in the figure.

[0453] It should be noted that in different application modes, the SOC can issue different Hybrid sensor settings to achieve the hard synchronization trigger mode of the Hybrid sensor in the current application mode. For example, Figure 17A When switching from the video mode to the shooting mode, the SOC can issue the corresponding Hybrid sensor setting to switch the hard synchronization trigger mode from Figure 16 to Figure 17A . Of course, if switching from the shooting mode to the video mode, the SOC can also issue the corresponding Hybrid sensor setting to switch the hard synchronization trigger mode from Figure 17A to Figure 16 .

[0454] It should be noted that the frequency of the Hybrid sensor internal clock is very high, and the frame synchronization signal and the hard synchronization signal need to be aligned with the rising edge or the falling edge of the clock signal, Figures 12-17B The alignment of the signals in the above is only for example to illustrate the hard synchronization trigger in different cases, and does not cause special limitation.

[0455] That is, Figures 12 to 17B In the timing diagram shown in the above, the frequency of the clock signal is very high, and each signal can be triggered according to the rising edge or the falling edge of the clock signal. However, Figures 12 to 17B The rising edge and the falling edge of the clock signal corresponding to the dashed line in the above do not represent the actual edge direction, and each signal can be triggered according to the rising edge or the falling edge. Some signals can be triggered according to the rising edge, and some signals can be triggered according to the falling edge.

[0456] In addition to the hard synchronization mode of the Hybrid sensor to realize the hard synchronization of the RGB data and the event data, the first camera 530 and the second camera 550 can also be time-aligned in the hard synchronization mode to further reduce the registration deviation.

[0457] At this time, the second image data and the event data in the Hybrid sensor have been internally hard-synchronized, so the first image data of the first camera 530 only needs to be hard-synchronized with the second image data of the Hybrid sensor.

[0458] In some embodiments of the present application, the FOVs of the first camera 530 and the Hybrid sensor are not very different, but the resolutions are quite different. Specifically, the resolution of the first image data of the first camera 530 is high, while the resolution of the second image data of the Hybrid sensor is low. In addition, the clocks of the first camera 530 and the Hybrid sensor are also not consistent.

[0459] Based on this, when the first camera 530 and the Hybrid sensor are synchronously started, the frame rate can be ensured to be the same, and according to the actual resolution, lens FOV, internal clock difference, readout time of each row of the first camera 530 and the Hybrid sensor, and double target calibration deviation, the exposure time difference of each row of pixels in the same region of interest (ROI) captured by the first camera 530 and the Hybrid sensor is controlled to be less than X milliseconds (ms), and an error of at most Y milliseconds (ms) is allowed. For example, X is 3 ms, and Y is 6 ms.

[0460] Referring to Figure 18 Another schematic block diagram of the electronic device 500 is shown, which can include the first camera 530, the second camera 550, and the main chip 560. The second camera 550 includes a Hybrid image sensor, which includes a target pixel part and an event pixel part.

[0461] The first camera 530 is configured to output first image data; the second camera 550 is configured to output second image data through the target pixel part and output event data through the event pixel part;

[0462] The main chip 560 is configured to acquire the first image data, the second image data, and the event data; and perform image display and / or image storage according to at least one of the first image data, the second image data, and the event data.

[0463] It can be understood that the main chip 560 can be a SOC of the electronic device 500, which can be integrated with one or more processors 510 and memories, etc. For example, the main chip 560 can be integrated with processors such as CPUs, ISPs, and DSPs.

[0464] The first camera 530 can be, for example, an RGB camera, and the target pixel portion can be, for example, an RGB pixel portion. In this case, the first image data and the second image data are both RGB image data.

[0465] In an embodiment of the present application, the main chip 560 can start at least one of the first camera 530 and the second camera 550 as needed.

[0466] The main chip 560 can perform image display and / or image storage only according to the first image data or the second image data. For example, the main chip 560 can directly display after performing image stabilization and image cropping on the first image data or the second image data according to the focal length selected by the user and the EIS algorithm result, or can send the image data to an application algorithm for processing before display and storage.

[0467] The main chip 560 can perform image display and / or image storage only according to the event data. For example, the main chip 560 can obtain a stylized video or a stylized picture by frame compression of the event data, and perform display and storage of the stylized video or the stylized picture.

[0468] The main chip 560 can perform image display and / or image storage according to the second image data and the event data. For example, the main chip 560 can obtain a stylized video or a stylized picture by superimposing the event data on the second image data, and perform display and storage of the stylized video or the stylized picture.

[0469] The main chip 560 can perform image display and / or image storage according to the first image data, the second image data, and the event data. For example, the main chip 560 can first perform a two-section "bridging" registration process according to the first image data, the second image data, and the event data to obtain registered first image data and registered event data, and then perform image cropping according to the registered first image data and the registered event data to obtain a display image or send to an application algorithm for processing.

[0470] In a possible implementation, if the first camera 530 is a rear main camera and the second camera 550 is a wide-angle camera, if the electronic device works in the main camera focal length, additional corresponding operations for adapting the wide-angle camera need to be performed.

[0471] Exemplarily, the first camera 530 and the second camera 550 can be arranged in a manner as shown in FIG. 6A, and the main chip 560 can be arranged in a manner as shown in FIG. 6B. Figure 9BThe process is similar, and the algorithms and operations suitable for the wide-angle camera are added in the ISP and ESP of the main chip 560. At this time, the main chip 560 first performs field of view angle cropping on the second image data according to the field of view angle of the first image data to obtain cropped second image data. Generally, the FOV of the wide-angle camera is larger than the FOV of the main camera, and the redundant pictures in the second image data can be cropped through FOV cropping. The FOV of the cropped second image data is the same as or slightly different from the FOV of the first image data. Then, the event data is processed through a preset wide-angle image processing algorithm to obtain first processed event data. The preset wide-angle image processing algorithm can include wide-angle distortion correction, LSC, and TSC, etc. Then, the first processed event data is cropped according to the field of view angle of the first image data to obtain cropped event data, and the FOV of the cropped event data is slightly different from or the same as the FOV of the first image data. The cropped second image data and the first image data are binocularly registered to obtain a registration matrix. The cropped event data and the first image data are registered according to the registration matrix to obtain registered first image data and registered event data.

[0472] In a possible implementation, if the first camera 530 is a rear main camera and the second camera 550 is a wide-angle camera, when the electronic device works in a wide-angle focal length, the main chip 560 is specifically configured to: perform wide-angle distortion correction on the first image data to obtain processed first image data. The processed first image data is image data after additional wide-angle distortion correction; perform wide-angle distortion correction on the second image data to obtain processed second image data; process the event data through a preset wide-angle image processing algorithm to obtain third processed event data; perform binocular registration on the processed second image data and the processed first image data to obtain a registration matrix; and register the third processed event data and the processed first image data according to the registration matrix to obtain registered first image data and registered event data. Figure 9C The process is similar, and the algorithms and operations suitable for the wide-angle camera are added in the ISP and ESP of the main chip 560. At this time, the main chip 560 first performs field of view angle cropping on the second image data according to the field of view angle of the first image data to obtain cropped second image data. Generally, the FOV of the wide-angle camera is larger than the FOV of the main camera, and the redundant pictures in the second image data can be cropped through FOV cropping. The FOV of the cropped second image data is the same as or slightly different from the FOV of the first image data. Then, the event data is processed through a preset wide-angle image processing algorithm to obtain first processed event data. The preset wide-angle image processing algorithm can include wide-angle distortion correction, LSC, and TSC, etc. Then, the first processed event data is cropped according to the field of view angle of the first image data to obtain cropped event data, and the FOV of the cropped event data is slightly different from or the same as the FOV of the first image data. The cropped second image data and the first image data are binocularly registered to obtain a registration matrix. The cropped event data and the first image data are registered according to the registration matrix to obtain registered first image data and registered event data.

[0473] In a possible implementation, the first camera is a front main camera, and the second camera is a front auxiliary main camera. When the electronic device works in a wide-angle focal length, the main chip 560 is specifically configured to: perform wide-angle distortion correction on the first image data to obtain processed first image data. The processed first image data is image data after additional wide-angle distortion correction; perform wide-angle distortion correction on the second image data to obtain processed second image data; process the event data through a preset wide-angle image processing algorithm to obtain third processed event data; perform binocular registration on the processed second image data and the processed first image data to obtain a registration matrix; and register the third processed event data and the processed first image data according to the registration matrix to obtain registered first image data and registered event data.

[0474] The first processed event data, the second processed event data, and the third processed event data refer to event data processed by a preset wide-angle image processing algorithm.

[0475] In a possible implementation, the main chip 560 is specifically configured to generate a first stylized video or a first stylized picture according to the registered first image data and the registered event data, and display the first stylized video or the first stylized picture. Alternatively, a second stylized video or a second stylized picture is generated according to the second image data and the event data, and the second stylized video or the second stylized picture is displayed. Alternatively, a third stylized video or a third stylized picture is generated according to the event data, and the third stylized video or the third stylized picture is displayed.

[0476] It can be understood that the first, second, and third in the first stylized video or the first stylized picture, the second stylized video or the second stylized picture, and the third stylized video or the third stylized picture are only used to distinguish the stylized videos or stylized pictures in different cases. The process of generating the stylized video or the stylized picture in various cases can be referred to the above, and will not be described here.

[0477] In a possible implementation, the main chip 560 can also control the hard synchronization triggering mode inside the Hybrid sensor by issuing a control signal to the second camera 550. When switching the application mode, a switching control signal can also be issued to the second camera 550 to switch the hard synchronization triggering mode inside the Hybrid sensor. The specific process can be referred to the above, and will not be described here. Of course, the hard synchronization triggering mode inside the Hybrid sensor can also be pre-fixed, and the main chip 560 cannot change the hard synchronization triggering mode inside the Hybrid sensor through the control signal.

[0478] Referring to Figure 19 A schematic block diagram of an image processing method is shown, the method can be applied to the electronic device 500, and the method includes the following steps:

[0479] In step S1901, the electronic device 500 acquires first image data output by the first camera 530.

[0480] In step S1902, the electronic device 500 acquires second image data output by the target pixel part and event data output by the event pixel part of the second camera 550.

[0481] In step S1903, the electronic device 500 registers the first image data and the second image data to obtain a registration matrix.

[0482] In step S1904, the electronic device 500 performs registration on the first image data and the event data according to the registration matrix to obtain registered first image data and registered event data.

[0483] It can be understood that in the registration process of step S1903 and step S1904, preset parameters need to be used. The preset parameters can be referred to the above, and will not be described here. In addition, due to the different roles played by the first camera 530 and the second camera 550, the two-stage "bridge" registration process can also be different. For example, when a wide-angle lens is involved, an operation of adapting the wide-angle lens needs to be additionally performed, which can include wide-angle distortion correction, FOV cropping, TSC, and LSC, etc. The specific process can be referred to the above, and will not be described here.

[0484] The electronic device provided by the embodiments of the present application can include a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the method of any one of the above method embodiments.

[0485] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps in the above various method embodiments.

[0486] The embodiments of the present application provide a computer program product, which, when running on an electronic device, causes the electronic device to execute the steps in the above various method embodiments.

[0487] The embodiments of the present application also provide a chip system, which includes a processor, the processor is coupled with a memory, and the processor executes a computer program stored in the memory to implement the method described in the above various method embodiments. The chip system can be a single chip or a chip module composed of multiple chips.

[0488] In the above embodiments, the description of each embodiment is focused on, and the part not described or recorded in a certain embodiment can be referred to the relevant description of other embodiments. It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application. In addition, in the description of the present application and the appended claims, the terms "first", "second", "third" and the like are only used to distinguish the description and cannot be understood as indicating or implying relative importance. In the present application, the reference "one embodiment" or "some embodiments" means that in one or more embodiments of the present application, the specific features, structures or characteristics described in connection with the embodiment are included. Therefore, the statements "in one embodiment", "in some embodiments", "in other some embodiments", "in other some embodiments" and the like appearing in different places in the specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized. Finally, it should be pointed out that: the above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any change or replacement within the technical scope disclosed in the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An electronic device, characterized in that, It includes a main chip, a first camera and a second camera, the second camera including a hybrid image sensor, the hybrid image sensor including a target pixel portion and an event pixel portion; The first camera is used to output first image data; The second camera is used to output second image data through the target pixel portion and to output event data through the event pixel portion; The main chip is used to acquire the first image data, the second image data, and the event data; Based on at least one of the first image data, the second image data, and the event data, perform image display and / or image storage; The main chip is specifically used for: Perform binocular registration on the first image data and the second image data to obtain a registration matrix; According to the registration matrix, the first image data and the event data are registered to obtain the registered first image data and the registered event data; Based on the registered first image data and the registered event data, perform image display and / or image storage.

2. The electronic device according to claim 1, characterized in that, The first camera is the front-facing main camera of the electronic device, and the second camera is the front-facing secondary main camera of the electronic device; Alternatively, the first camera may be the main rear camera of the electronic device, and the second camera may be the secondary main rear camera, wide-angle camera, or telephoto camera of the electronic device.

3. The electronic device according to claim 2, characterized in that, The first camera is a rear main camera, and the second camera is a wide-angle camera; If the electronic device operates at the main camera focal length, the main chip is specifically used for: Based on the field of view of the first image data, the second image data is cropped to obtain the cropped second image data; The event data is processed by a preset wide-angle image processing algorithm to obtain the first processed event data; Based on the field of view of the first image data, the first processed event data is cropped by the field of view to obtain cropped event data. The cropped second image data and the first image data are binocularly registered to obtain the registration matrix; Based on the registration matrix, the cropped event data and the first image data are registered to obtain the registered first image data and the registered event data.

4. The electronic device according to claim 3, characterized in that, If the electronic device operates in the wide-angle focal length range, the main chip is also used for: Wide-angle distortion correction is performed on the second image data to obtain the corrected second image data; The event data is processed by a preset wide-angle image processing algorithm to obtain second processed event data; Based on the second processed event data and the corrected second image data, perform image display and / or image storage.

5. The electronic device according to claim 1, characterized in that, The first camera is the front-facing main camera, and the second camera is the front-facing secondary main camera; If the electronic device operates in the wide-angle focal length range, the main chip is specifically used for: Wide-angle distortion correction is performed on the first image data to obtain the processed first image data; Wide-angle distortion correction is performed on the second image data to obtain the processed second image data; The event data is processed by a preset wide-angle image processing algorithm to obtain third-processed event data; The processed second image data and the processed first image data are then registered using binoculars to obtain the registration matrix; According to the registration matrix, the third processed event data and the processed first image data are registered to obtain the registered first image data and the registered event data.

6. The electronic device according to claim 1, characterized in that, The main chip is specifically used for: Based on the registered first image data and the registered event data, a first stylized video or a first stylized image is generated, and the first stylized video or the first stylized image is sent for display.

7. The electronic device according to claim 1, characterized in that, The main chip is specifically used for: Based on the second image data and the event data, a second stylized video or a second stylized image is generated, and the second stylized video or the second stylized image is sent for display. Alternatively, based on the event data, a third-stylized video or a third-stylized image can be generated and then displayed.

8. The electronic device according to any one of claims 1 to 7, characterized in that, The target pixel portion includes a target pixel array and a target pixel readout circuit; the event pixel portion includes an event pixel array and an event pixel readout circuit. The target pixel readout circuit is used to perform exposure and readout operations on the target pixel array to obtain the second image data; The event pixel readout circuit is used to read out the event pixel array to obtain the event data.

9. The electronic device according to claim 8, characterized in that, The hybrid image sensor also includes a lifting circuit, the target pixel readout circuit includes a frame readout controller, and the event pixel readout circuit includes a timestamp circuit. The frame readout controller is connected to the lifting circuit via a hardware connection line, and the lifting circuit is connected to the timestamping circuit via a hardware connection line; The frame readout controller is used to output a frame synchronization signal; The pull-up circuit is used to receive the frame synchronization signal, and after voltage boosting the frame synchronization signal, output a hard synchronization signal. The timestamp circuit is also used to receive the hard synchronization signal, and under the trigger of the hard synchronization signal, print an additional timestamp and output the additional timestamp.

10. The electronic device according to claim 9, characterized in that, The event pixel readout circuit is also used to: embed the additional timestamp in the asynchronous event data stream in chronological order, or embed the additional timestamp in the synchronous event frame data stream; The event data is either the asynchronous event data stream or the synchronous event frame data stream.

11. The electronic device according to claim 10, characterized in that, The main chip is specifically used to: receive the asynchronous event data stream or the synchronous event frame data stream; and time-align the event data and the second image data according to the additional timestamp in the asynchronous event data stream or the synchronous event frame data stream.

12. The electronic device according to claim 9, characterized in that, If the exposure mode of the target pixel array is global exposure and the readout mode of the event pixel array is asynchronous readout, the frame readout controller is specifically used to: output a frame synchronization signal at the exposure start time and the exposure end time for each frame of the second image data; If the exposure mode of the target pixel array is partial exposure and the readout mode of the event pixel array is asynchronous readout, the frame readout controller is specifically used to: output a frame synchronization signal at the end of the exposure time of the first row of target pixels for each frame of the second image data.

13. The electronic device according to claim 9, characterized in that, The target pixel array is exposed using global exposure, and the event pixel array is read out using synchronous readout. The frame readout controller is specifically used to: for each frame of the second image data, output a frame synchronization signal at the exposure start time and the exposure end time, respectively.

14. The electronic device according to claim 9, characterized in that, The target pixel array is exposed using global exposure, and the event pixel array is read out using synchronous readout. The frame readout controller is specifically used to: for each frame period, output a frame synchronization signal at the exposure start time and the exposure end time, or output a frame synchronization signal at the exposure start time, wherein the frame period is the time period from the exposure start time of the current frame to the exposure start time of the next frame; The event pixel readout circuit is further configured to: resynchronize the first event frame within the frame period with the hard synchronization signal corresponding to the exposure start time, and embed the additional timestamp corresponding to the exposure start time into the first event frame within the frame period.

15. The electronic device according to claim 9, characterized in that, The target pixel array is exposed using partial exposure, and the event pixel array is read out using synchronous readout. The frame readout controller is specifically used to: output a frame synchronization signal at a preset time for each frame of the second image data, wherein the preset time is the end time of exposure of the first row of target pixels or the start time of readout of the first row of target pixels.

16. The electronic device according to claim 9, characterized in that, The target pixel array is exposed using partial exposure, and the event pixel array is read out using synchronous readout. The frame readout controller is specifically used to: for each frame period, at the end of the exposure of the first row of target pixels, output a frame synchronization signal, wherein the frame period is the time period from the end of the exposure of the first row of target pixels in the current frame to the end of the exposure of the first row of target pixels in the next frame; The event pixel readout circuit is further configured to: resynchronize the first event frame within the frame period with the hard synchronization signal corresponding to the exposure end time of the first row of target pixels, and embed the additional timestamp corresponding to the exposure end time of the first row of target pixels into the timestamp of the first event frame within the frame period.

17. The electronic device according to claim 14 or 16, characterized in that, If at least two frames of second image data are generated within one frame period, the frame readout controller is specifically configured to: output a frame synchronization signal at the exposure start time and exposure end time for each frame of second image data, or output a frame synchronization signal at the exposure end time of the first row of target pixels.

18. The electronic device according to any one of claims 9 to 17, characterized in that, The main chip is also used to: send control signals to the hybrid image sensor, the control signals including a first readout parameter and a second readout parameter, the first readout parameter including an exposure mode and an exposure time, and the second readout parameter including an event readout mode; The hybrid image sensor is further configured to: receive the control signal, parse the control signal to obtain the first readout parameter and the second readout parameter; control the target pixel readout circuit to perform exposure and readout operations according to the first readout parameter to obtain the second image data, and control the frame readout controller to output the frame synchronization signal; and control the event pixel readout circuit to perform readout operations according to the second readout parameter to obtain the event data and the additional timestamp.

19. The electronic device according to claim 18, characterized in that, The main chip is also used to: calculate the sensor setting parameters to be sent based on the second image data and the event data, and generate the control signal based on the sensor setting parameters to be sent.

20. The electronic device according to claim 19, characterized in that, The target pixel array is exposed using partial exposure, and the event pixel array is read out using synchronous readout. The main chip is also configured to: if a switching operation is detected, generate a switching control signal in response to the switching operation, and send the switching control signal to the hybrid image sensor, wherein the switching operation is used to indicate switching from video mode to photo mode, and the switching control signal includes sensor setting parameters in the photo mode; The hybrid image sensor is further configured to: receive the switching control signal, parse the switching control signal to obtain sensor setting parameters in the shooting mode; and, based on the sensor setting parameters, control the frame readout controller to output a frame synchronization signal at the start time of exposure of the first row of target pixels in each frame of the second image data, and output a frame synchronization signal at the end time of exposure of the last row of target pixels, so as to control the timestamp circuit to respond to the hard synchronization signal, print an additional timestamp at the start time of exposure of the first row of target pixels, print an additional timestamp at the end time of exposure of the last row of target pixels, and embed the additional timestamp into the next event frame.

21. The electronic device according to claim 20, characterized in that, The switching control signal is issued during the exposure time of the second image data in the nth frame and takes effect during the exposure time of the second image data in the (n+2)th frame.

22. An image processing method, characterized in that, The invention is applied to an electronic device, which includes a first camera and a second camera, the second camera including a hybrid image sensor, the hybrid image sensor including a target pixel portion and an event pixel portion; The method includes: Acquire the first image data output by the first camera; Acquire the second image data output by the second camera through the target pixel portion and the event data output through the event pixel portion; The first image data and the second image data are registered to obtain a registration matrix; Based on the registration matrix, the first image data and the event data are registered to obtain the registered first image data and the registered event data.

23. The method according to claim 22, characterized in that, The first camera is the front-facing main camera of the electronic device, and the second camera is the front-facing secondary main camera of the electronic device; Alternatively, the first camera may be the main rear camera of the electronic device, and the second camera may be the secondary main rear camera, wide-angle camera, or telephoto camera of the electronic device.

24. The method according to claim 23, characterized in that, The first camera is the rear main camera, and the second camera is the wide-angle camera; If the electronic device operates at the main camera focal length, the first image data and the second image data are registered to obtain a registration matrix. Based on the registration matrix, the first image data and the event data are registered to obtain registered first image data and registered event data, including: Based on the field of view of the first image data, the second image data is cropped to obtain the cropped second image data; The event data is processed by a preset wide-angle image processing algorithm to obtain the first processed event data; Based on the field of view of the first image data, the first processed event data is cropped by the field of view to obtain cropped event data. The cropped second image data and the first image data are binocularly registered to obtain the registration matrix; Based on the registration matrix, the cropped event data and the first image data are registered to obtain the registered first image data and the registered event data.

25. The method according to claim 23, characterized in that, The first camera is the front-facing main camera, and the second camera is the front-facing secondary main camera; If the electronic device is in a wide-angle focal length, the first image data and the second image data are registered to obtain a registration matrix. Based on the registration matrix, the first image data and the event data are registered to obtain registered first image data and registered event data, including: Wide-angle distortion correction is performed on the first image data to obtain the processed first image data; Wide-angle distortion correction is performed on the second image data to obtain the processed second image data; The event data is processed by a preset wide-angle image processing algorithm to obtain third-processed event data; The processed second image data and the processed first image data are then registered using binoculars to obtain the registration matrix; According to the registration matrix, the third processed event data and the processed first image data are registered to obtain the registered first image data and the registered event data.

26. The method according to claim 22, characterized in that, The method further includes: Based on the registered first image data and the registered event data, display the first stylized video or the first stylized image.

27. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 22 to 26.

28. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 22 to 26.

Images