Complementary metal oxide semiconductor (CMOS) image sensor of mixed event and frame pixel and reading method of CMOS image sensor

Abstract

The invention belongs to the technical field of image sensors, and provides a hybrid event and frame pixel CMOS image sensor and a reading method thereof. Comprising a CMOS pixel array and a reading circuit, the CMOS pixel array is composed of event pixels and frame pixels F of four channels R, G, B and W, the event pixels and the frame pixels are arranged in a staggered mode, and event pixel signals and frame pixel signals can be output at the same time; the reading circuit comprises a first reading circuit and a second reading circuit, and the first reading circuit is located below the CMOS pixel array and controls and outputs event pixel signals of four channels R, G, B and W; and the second reading circuit is positioned below the first reading circuit and is used for controlling and outputting a frame pixel signal. The reading method comprises two modes, event pixels are read asynchronously or synchronously, and frame pixel signals are read synchronously.

Description

Technical Field

[0001] The present invention relates to the technical field of image sensors, and in particular to a CMOS image sensor of mixed event and frame pixels and a reading method thereof. Background Art

[0002] An event camera, also known as a neuromorphic camera, silicon retina, or dynamic vision sensor, is an image sensor that responds to local changes in brightness. An event camera does not use a shutter to capture images like a traditional (frame) camera. Instead, each pixel in an event camera operates independently and asynchronously, reporting brightness changes when they occur and remaining silent otherwise.

[0003] The design of the event camera is inspired by the dynamic perception mechanism of the retina. The core idea is on-demand sampling, which simulates the sensitivity of the retina to fast-moving scenes, which is similar to the sparse output and high temporal resolution mechanism of the retina to detect dynamic scenes. However, the human eye can obtain dynamic and static information at the same time, obtain fine detail textures, perceive static scenes and construct complete images, while the event camera cannot directly process static surfaces and needs to be combined with a frame camera to complete. In addition, the event camera lacks color perception, that is, the event camera lacks the color perception ability like the cone cells of the human eye. Therefore, there is a technical route to embed RGB pixels into the event camera.

[0004] Foreign companies engaged in this field mainly include iniVation of Switzerland, Prophesee of France, Samsung of South Korea, Sony of Japan, etc. At present, among foreign companies, only IniVation provides brain-like imaging sensors based on shared pixel RGB frames and event streams. This is because the solution of shared pixel circuit affects the quality of event output, and the 100,000-resolution RGB frame is difficult to meet complex application requirements. Therefore, the solution of shared pixel circuit is gradually abandoned.

[0005] The latest technical solution uses back-illuminated and stacked technology. The event pixel circuit independently forms a layer of pixel array, and the RGB pixel circuit also independently forms a layer of pixel array. This solution is compatible with high-resolution RGB pixel circuits and low-resolution DVS pixel circuits. In 2024, Sony Corporation of Japan and Prophesee of France cooperated to use front-end circuit sharing technology to realize 2.97μm pixel pitch back-illuminated stacked EVS, and designed a camera with a resolution of 640×640 and a maximum event rate of 1412Meps that can output events and frames simultaneously. This method enables high-resolution RGB imaging and high temporal resolution, lower resolution event imaging in a single chip. However, this technical solution is difficult, and the back-illuminated and multi-layer stacking solutions lead to a huge increase in the noise level of the event pixel circuit. Summary of the invention

[0006] In view of the shortcomings of the prior art, the present invention proposes a CMOS image sensor with mixed event and frame pixels and a reading method thereof. The CMOS image sensor adopts a design in which event pixels and frame pixels are arranged alternately on the same layer, and can output event pixel signals and frame pixel signals at the same time, and the event pixel signals have two output modes: asynchronous and synchronous.

[0007] The technical solution of the present invention is as follows: A CMOS image sensor with mixed event and frame pixels, comprising a CMOS pixel array and a readout circuit; The CMOS pixel array is composed of event pixels of four channels, R, G, B, and W, and frame pixels F, which are arranged alternately; R, G, B, and W are respectively for sensing the changes in the relative intensity of red, green, blue, and white light, and F is the RGB pixel of the frame camera.

[0008] Preferably, the event pixels and frame pixels are arranged alternately in a column direction of the CMOS pixel array.

[0009] Preferably, the event pixels and frame pixels are arranged alternately in both the row direction and the column direction of the CMOS pixel array.

[0010] Preferably, a combined pixel group is formed by two frame pixels and two event pixels, and a 32-bit encoding method is adopted, wherein R, G, B, and W are each encoded using a 2-bit width, and F is encoded using an 8-bit width.

[0011] Preferably, R, G, B, and W use 2-bit bit width encoding, 00 indicates no event output, 01 indicates positive event output, that is, brightness increases, and 10 indicates negative event output, that is, brightness decreases; F uses 8-bit bit width encoding to represent absolute brightness.

[0012] Preferably, the reading circuit includes a first reading circuit and a second reading circuit, the first reading circuit is located below the CMOS pixel array, and controls the output of event pixel signals of four channels R, G, B, and W; the second reading circuit is located below the first reading circuit, and controls the output of frame pixel signals.

[0013] A method for reading a CMOS image sensor with mixed event and frame pixels includes two modes: Mode 1: Event pixels are read asynchronously, and frame pixels are read synchronously. The event pixel is connected to the row-column arbiter and output in a row-first or column-first manner. The event pixels of the four channels R, G, B, and W are a whole, and the event trigger status is read at the same time; the frame pixel adopts a synchronous pipeline column scanning reading method to achieve a frame rate of 50fps to output absolute light intensity signals; Mode 2: Event pixels adopt synchronous reading mode, and frame pixels adopt synchronous reading mode; The event pixel adopts the synchronous pipeline column scanning and reading method to achieve the output of relative light intensity change signal at a frame rate of 2000fps; the frame pixel adopts the synchronous pipeline column scanning and reading method to achieve the output of absolute light intensity signal at a frame rate of 50fps.

[0014] Preferably, in the mode 1, event pixel reading adopts 64-bit encoding, including 32-bit timestamp, 8-bit row address, 8-bit column address, 4-bit event information encoding, and the rest are valid signal flag bits.

[0015] Preferably, in Mode 2, event pixels are read using a 4-bit encoding method.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. Advantages and effects of the CMOS image sensor designed by the present invention: a) Process cost advantage: The present invention adopts a design in which event pixels and frame pixels are arranged alternately on the same layer, avoiding the requirement of traditional stacking process for multi-layer process, thereby reducing manufacturing cost. Compared with image sensors that require complex 3D stacking process, the present invention achieves performance comparable to stacking process through planar layout, significantly reducing process complexity.

[0017] b) Multifunctional integration: The CMOS image sensor designed in the present invention supports both event perception and frame information acquisition functions, and is suitable for both dynamic scene detection (event triggering) and traditional static image acquisition, thus expanding the application scenarios of the sensor.

[0018] c) Efficient space utilization: Each combined pixel group consists of two event pixels and two frame pixels. The compact layout improves space utilization while reducing interference between photoelectric conversion devices.

[0019] 2. Advantages and effects of the reading method of the present invention: a) Model flexibility: Provides two modes: asynchronous event reading and synchronous event reading to meet the needs of different scenarios: Mode 1: The asynchronous reading method is highly sensitive to dynamic events and is suitable for capturing sudden or high-speed changing scenes.

[0020] Mode 2: The synchronous reading method is suitable for high frame rate real-time scenarios to ensure the coordinated collection of event pixel and frame pixel information.

[0021] b) Efficient data reading: Asynchronous reading uses row and column arbiters to output according to priority, reducing event-triggered reading delays and improving event detection efficiency. Frame pixels use synchronous pipeline column scanning to ensure the output of absolute light intensity information at a fixed frame rate, and the data output is stable and reliable.

[0022] c) High frame rate output support: In Mode 2, event pixel reading achieves a light intensity change signal output of up to 2000fps, greatly improving the ability to capture dynamic scenes. Even when reading high frame rate events, the frame pixel still maintains a stable output of 50fps, taking into account the needs of both dynamic and static scenes.

[0023] d) Coding optimization: In mode 1, the 64-bit encoding method contains detailed timestamp, address and event information, providing sufficient time and space information for subsequent data processing.

[0024] In mode 2, event pixels are read using a streamlined 4-bit encoding method, which greatly reduces the data bandwidth requirement and is suitable for application scenarios with high real-time requirements.

[0025] e) Strong adaptability: High dynamic range, high sensitivity event detection and high-quality frame image acquisition can be achieved through different reading modes. It is suitable for various application scenarios such as unmanned driving, robot vision, security monitoring, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. The features and advantages of the present invention can be more clearly understood by referring to the drawings. The drawings are schematic and should not be understood as limiting the present invention in any way. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 This is the first manner in which event pixels and frame pixels are arranged alternately in the CMOS image sensor of the present invention that mixes event pixels and frame pixels.

[0028] Figure 2 This is the second manner in which event pixels and frame pixels are arranged alternately in the CMOS image sensor of the present invention that mixes event pixels and frame pixels.

[0029] Figure 3 The present invention is a schematic diagram of a readout circuit in a CMOS image sensor that mixes event and frame pixels.

[0030] Figure 4This is the event pixel encoding method of mode 1 in the method for reading a CMOS image sensor that mixes event and frame pixels of the present invention. DETAILED DESCRIPTION

[0031] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.

[0032] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0033] The present invention proposes a CMOS image sensor with mixed event and frame pixels, including a CMOS pixel array and a reading circuit. The CMOS pixel array is composed of event pixels and frame pixels F of four channels R, G, B, and W, and the event pixels and frame pixels are arranged alternately, such as Figure 1 and Figure 2 As shown, there are two specific arrangements: one is that the event pixels and the frame pixels are staggered only in the column direction of the CMOS pixel array, and the other is that the event pixels and the frame pixels are staggered in both the row direction and the column direction of the CMOS pixel array.

[0034] R, G, B, and W are the changes in the relative intensity of red, green, blue, and white light, respectively, and F is the RGB pixel of the frame camera. A combined pixel group consists of two frame pixels and two event pixels, that is, two frames and two RGBW are output. R, G, B, and W use 2-bit bit width encoding, 00 means no event output, 01 means positive event output, that is, brightness increases, and 10 means negative event output, that is, brightness decreases; F uses 8-bit bit width encoding to represent absolute brightness.

[0035] The readout circuit includes a first readout circuit and a second readout circuit. The first readout circuit is located at the event layer, i.e., the next layer of the CMOS pixel array, and controls the output of event pixel signals of four channels, R, G, B, and W. The second readout circuit is located at the frame layer, i.e., the next layer of the first readout circuit, and controls the output of frame pixel signals. Figure 3 shown.

[0036] The present invention also proposes a method for reading a CMOS image sensor based on combining events and frame pixels. Specifically, two reading methods can be selected: The first reading method: event pixels are read asynchronously, and frame pixels are read synchronously.

[0037] The event pixel is connected to the row and column arbiter and output in a row-first or column-first manner. The event pixels of the four channels R, G, B, and W are a whole, and the event trigger status is read at the same time. The event pixel reading adopts a 64-bit encoding method, including a 32-bit timestamp, an 8-bit row address, an 8-bit column address, a 4-bit event information encoding, and the rest are valid signal flags, such as Figure 4 The frame pixels are read by synchronous pipeline column scanning to achieve a frame rate of 50fps to output absolute light intensity signals.

[0038] The second reading method: event pixels adopt synchronous reading method, and frame pixels adopt synchronous reading method.

[0039] The event pixel adopts the synchronous pipeline column scanning reading method to achieve the output of the relative change signal of light intensity at a frame rate of 2000fps, and the event pixel reading adopts the 4-bit encoding method. The frame pixel adopts the synchronous pipeline column scanning reading method to achieve the output of the absolute light intensity signal at a frame rate of 50fps.

[0040] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A CMOS image sensor with mixed event and frame pixels, characterized in that: including a CMOS pixel array and a readout circuit; The CMOS pixel array is composed of event pixels of four channels, R, G, B, and W, and frame pixels F, which are arranged alternately; R, G, B, and W are respectively for sensing the changes in the relative intensity of red, green, blue, and white light, and F is the RGB pixel of the frame camera.

2. The CMOS image sensor according to claim 1, characterized in that: A combined pixel group is composed of two frame pixels and two event pixels, and adopts a 32-bit encoding method, in which R, G, B, and W are each encoded with a 2-bit width, and F is encoded with an 8-bit width.

3. The CMOS image sensor according to claim 1, wherein: R, G, B, and W use 2-bit bit width encoding, 00 means no event output, 01 means positive event output, that is, brightness increases, and 10 means negative event output, that is, brightness decreases; F uses 8-bit bit width encoding to represent absolute brightness.

4. The CMOS image sensor according to claim 1, characterized in that: The reading circuit includes a first reading circuit and a second reading circuit. The first reading circuit is located below the CMOS pixel array and controls the output of event pixel signals of four channels R, G, B, and W; the second reading circuit is located below the first reading circuit and controls the output of frame pixel signals.

5. The CMOS image sensor according to any one of claims 1 to 4, characterized in that: The event pixels and frame pixels are arranged alternately in the column direction of the CMOS pixel array.

6. The CMOS image sensor according to any one of claims 1 to 4, characterized in that: The event pixels and frame pixels are arranged alternately in both the row direction and the column direction of the CMOS pixel array.

7. A method for reading a CMOS image sensor according to any one of claims 1 to 6, characterized in that: There are two modes: Mode 1: Event pixels are read asynchronously, and frame pixels are read synchronously. The event pixel is connected to the row-column arbiter and output in a row-first or column-first manner. The event pixels of the four channels R, G, B, and W are a whole, and the event trigger status is read at the same time; the frame pixel adopts a synchronous pipeline column scanning reading method to achieve a frame rate of 50fps to output absolute light intensity signals; Mode 2: Event pixels adopt synchronous reading mode, and frame pixels adopt synchronous reading mode; The event pixel adopts a synchronous pipeline column scanning and reading method to achieve a frame rate of 2000fps to output the relative light intensity change signal; the frame pixel adopts a synchronous pipeline column scanning and reading method to achieve a frame rate of 50fps to output the absolute light intensity signal.

8. The reading method according to claim 7, characterized in that: In the mode 1, event pixel reading adopts 64-bit encoding, including 32-bit timestamp, 8-bit row address, 8-bit column address, 4-bit event information encoding, and the rest are valid signal flag bits.

9. The reading method according to claim 7, characterized in that: In the mode 2, event pixels are read using a 4-bit encoding method.

Images