A fused CMOS image sensor with event-driven frame exposure

Abstract

The present invention relates to a fusion CMOS image sensor with event-driven frame exposure, which belongs to the field of image sensor technology and solves the problems of large data volume and motion blur in the existing technology. The fusion CMOS image sensor of the present invention includes a CMOS pixel array, an event generation circuit, a signal holding circuit, a pulse selector, an exposure pulse drive circuit EPDU, a single pixel exposure unit SPU, a modal judgment circuit, an arbitration circuit, a reading circuit and a mode switching switch. The fusion CMOS image sensor of the present invention adopts a method of controlling the asynchronous exposure of RGB pixels by a continuous event triggering monitoring circuit, so that the output of RGB pixels is highly synchronized with movement and light intensity changes, breaking through the disadvantage of the synchronous exposure method that is unrelated to the movement and change of the scene or carrier, so that the RGB information has the same advantage of high temporal resolution as the event information. Complete retinal visual simulation is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of image sensors, and in particular to a fused CMOS image sensor with event-driven frame exposure, and more particularly to a fused CMOS image sensor with event-driven frame exposure in which an RGB pixel exposure mode is asynchronously controlled by a pulse signal. Background Art

[0002] The human retina is capable of simultaneously perceiving dynamic changes and static information such as color details and textures. To mimic the working mode of the human retina, neuromorphic vision sensors (also known as event cameras, brain-inspired vision sensors, or silicon retinas) have been developed. Existing neuromorphic vision sensor CMOS modes include event pixels only, pixels containing both event pixel circuits and RGB pixel circuits, and independent event pixels and RGB pixels in the same CMOS. Event pixels capture the logarithmic relative change in light intensity. When the light intensity change exceeds a positive or negative threshold, an event with a polarity of 1 or -1 is output, indicating the direction of the light intensity change. Events are represented as tuples, achieving high temporal resolution, high dynamic range, and low power consumption for dynamic changes. RGB pixels are exposed synchronously to output RGB frames.

[0003] However, existing neuromorphic vision sensor solutions fail to achieve full human vision, which requires ultra-low data volumes yet rich and complete dynamic and RGB information. When motion is high, RGB pixels based on synchronized exposure inevitably experience motion blur. While shortening exposure time (increasing frame rate) can mitigate this phenomenon to some extent, it results in a significant increase in data volume, undermining the goal of data lightweighting. Summary of the Invention

[0004] In view of the above problems, the present invention provides a fusion CMOS image sensor with event-driven frame exposure, which solves the problems of large data volume and motion blur in the existing technology.

[0005] The present invention provides a fusion CMOS image sensor for event-driven frame exposure, comprising a CMOS pixel array, an event generation circuit, a signal holding circuit, a pulse selector MUX, an exposure pulse drive circuit EPDU, a single pixel exposure unit SPU, a mode judgment circuit, an arbitration circuit, a reading circuit, and a mode switching switch;

[0006] The CMOS pixel array includes an event pixel circuit and an RGB pixel circuit, which are connected to an event generation circuit, a modal judgment circuit, an arbitration circuit, and a mode switching switch, and is used to detect light intensity changes in a scene and generate asynchronous event signals and capture color image information;

[0007] The event generation circuit includes a photodiode PD, an operational amplifier Ab, a capacitor C1, a capacitor C2, a comparator, and a switch. The event generation circuit is connected to the signal holding circuit and is used to monitor light intensity changes in real time and generate a trigger signal. It is used to detect light intensity change signals in the scene and generate asynchronous event signals.

[0008] The signal holding circuit includes three D flip-flops, which are connected to the pulse selector MUX through a combination circuit of an AND gate and an OR gate, and are used to temporarily store asynchronous event signals and synchronize the output RGB signals of the RGB pixel circuit;

[0009] The pulse selector MUX is connected to the signal holding circuit and is used to select the execution of single or multiple exposure logic;

[0010] The exposure pulse driving circuit EPDU is connected to the single pixel exposure unit SPU and is used to control the exposure mode of the single pixel exposure unit SPU;

[0011] The single pixel exposure unit SPU is controlled by the exposure pulse drive circuit EPDU to perform the exposure task;

[0012] The exposure pulse drive circuit EPDU outputs a control signal, and the single-pixel exposure unit SPU performs exposure after receiving the control signal;

[0013] The reading circuit is used to output the RGB value obtained by the final exposure.

[0014] Optionally, a light intensity change signal is input, and an operating mode is selected by a signal holding circuit, a pulse selector MUX, a modal judgment circuit, an arbitration circuit, and a mode switching switch. The operating modes include mode 1 and mode 2. In mode 1, the event pixel circuit and the RGB pixel circuit are connected to a photodiode PD, and the exposure of the RGB pixel circuit is controlled by an event pixel circuit trigger signal module. In mode 2, the event pixel circuit and the RGB pixel circuit are independent, and the event pixel circuit and multiple RGB pixel circuits within its neighborhood form a control group, and the exposure of the RGB pixel circuit is controlled by the event pixel circuit trigger signal module.

[0015] Optionally, in mode 1 and mode 2, the event signal output circuits of the event pixel circuits are respectively connected to the signal holding circuits; two AND gates AND are used to monitor whether there are two or more consecutive positive events or negative events in the three D flip-flops in the signal holding circuits; and exposure is performed based on the monitoring results.

[0016] Optionally, when two consecutive positive events or negative events are monitored, the two AND gates control the RGB pixel circuit to perform single exposure; when multiple consecutive positive events or negative events are monitored, the two AND gates control the RGB pixel circuit to perform high-frequency continuous exposure.

[0017] Alternatively, the expression for the D flip-flop state is:

[0018]

[0019] in, Indicates the next moment Output status; Indicates the input status at time t, and RST is the reset signal.

[0020] Optionally, the signal holding circuit state is obtained according to the three D flip-flop states, and the expression is:

[0021]

[0022] Among them, CLK is the clock signal; Indicates the input state at time t; represents the state of the first D flip-flop DFF1 at time t; represents the state of the second D flip-flop DFF2 at time t; Indicates that the first D flip-flop DFF1 will Status; Indicates that the second D flip-flop DFF2 will Status; Indicates that the D third flip-flop DFF3 will be status.

[0023] Optionally, two continuous signal trigger modes in three D flip-flops are monitored respectively through two AND gates, and the two continuous signal trigger modes are trigger mode 1 and trigger mode 2, wherein trigger mode 1 is to monitor two positive events or negative events that are continuously triggered; trigger mode 2 is to monitor three or more positive events or negative events that are continuously triggered.

[0024] Optionally, the judgment conditions of trigger mode 1 and trigger mode 2 are obtained; and a selection signal is obtained according to the judgment conditions.

[0025] Optionally, based on the selection signal, the exposure pulse driving circuit EPDU controls the single-pixel exposure unit SPU to select single or multiple continuous exposures, and outputs the RGB value obtained by the final exposure.

[0026] Compared with the prior art, the present invention has at least the following beneficial effects:

[0027] (1) The fusion CMOS image sensor of the present invention uses a continuous event-triggered monitoring circuit to control the asynchronous exposure of RGB pixels, making the output of RGB pixels highly synchronized with motion and light intensity changes. This overcomes the disadvantage of synchronous exposure methods that are independent of scene or carrier motion and changes, and enables RGB information to have the same high temporal resolution as event information. This achieves complete retinal vision simulation.

[0028] (2) The fusion CMOS image sensor of the present invention supports two RGB pixel control modes, flexibly meeting the requirements of different application scenarios. The shared photodiode (PD) solution is suitable for application scenarios that have low resolution requirements but strict response speed requirements, such as visual SLAM (Simultaneous Localization and Mapping). The independent event pixel and RGB pixel solution is suitable for application scenarios that require high scene texture details and high dynamic requirements, such as monitoring. The mode flexibility is good.

[0029] (3) The data readout method of the fusion CMOS image sensor of the present invention breaks through the existing synchronous exposure readout method of RGB frames. It adopts a method of dynamically maintaining the RGB matrix to output RGB frames on demand, without RGB data redundancy, which greatly reduces the data volume. It achieves an ultra-efficient data processing mode close to that of the human brain. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The drawings are only for purposes of illustrating particular embodiments and are not to be considered limiting of the invention.

[0031] Figure 1 Schematic diagram of mode 1 of the fused CMOS image sensor with event-driven frame exposure according to the present invention.

[0032] Figure 2 2 is a schematic diagram of mode 2 of the fused CMOS image sensor with event-driven frame exposure according to the present invention.

[0033] Figure 3 This is a first typical design schematic diagram of a CMOS image sensor pixel control group of the present invention that asynchronously controls RGB pixel exposure modes using pulse signals to fuse events and frames.

[0034] Figure 4 This is a second typical design schematic diagram of a CMOS image sensor pixel control group of the present invention that asynchronously controls RGB pixel exposure modes using pulse signals to fuse events and frames.

[0035] Figure 5 This is a schematic diagram of the first RGB matrix of the present invention, in which the fusion event of the RGB pixel exposure mode is asynchronously controlled by a pulse signal and the CMOS image sensor of the frame is dynamically maintained.

[0036] Figure 6 This is a second RGB matrix schematic diagram of the present invention, in which the pulse signal asynchronously controls the fusion event of the RGB pixel exposure mode and the dynamic maintenance of the CMOS image sensor of the frame. DETAILED DESCRIPTION

[0037] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. In addition, the present invention can also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited by the specific embodiments disclosed below.

[0038] A specific embodiment of the present invention, as Figures 1-6 , discloses a fusion CMOS image sensor with event-driven frame exposure, including a CMOS pixel array, an event generation circuit, a signal holding circuit, a pulse selector, an exposure pulse drive circuit EPDU, a single pixel exposure unit SPU, a mode judgment circuit, an arbitration circuit, a reading circuit and a mode switching switch;

[0039] The CMOS pixel array includes an event pixel circuit and an RGB pixel circuit, which are connected to an event generation circuit, a modal judgment circuit, an arbitration circuit and a mode switching switch, and is used to detect changes in light intensity in the scene and generate asynchronous event signals and capture color image information.

[0040] The asynchronous event signal drives the RGB pixel exposure through the signal holding circuit, pulse selector, exposure pulse drive circuit EPDU, and single pixel exposure unit SPU, and the color image information forms the final RGB value obtained by exposure;

[0041] Furthermore, the event pixel circuit is used to detect light intensity changes and generate asynchronous event signals, and the RGB pixel circuit is used to capture color image information;

[0042] The event generation circuit includes a photodiode, an operational amplifier Ab, a capacitor C1, a capacitor C2, a comparator, and a switch. The event generation circuit is connected to the signal holding circuit and is used to monitor light intensity changes in real time and generate a trigger signal. It is used to detect light intensity change signals in the scene and generate asynchronous event signals.

[0043] The signal holding circuit includes three D flip-flops, which are connected to the pulse selector through a combination circuit of an AND gate and an OR gate. They are used to temporarily store the asynchronous event signal and synchronize it with the RGB signal output by the RGB pixel circuit to achieve high-time-resolution fusion of the asynchronous event signal and the RGB signal.

[0044] The pulse selector is connected to the signal holding circuit and is used for selecting to execute single or multiple exposure logic;

[0045] The exposure pulse driving circuit is connected to the single-pixel exposure unit and is used to control the exposure mode of the single-pixel exposure unit;

[0046] The single-pixel exposure unit is controlled by an exposure pulse driving circuit to perform exposure tasks;

[0047] The exposure pulse driving circuit outputs a control signal, and the single-pixel exposure unit performs exposure after receiving the control signal;

[0048] The reading circuit is used to output the RGB value obtained by the final exposure.

[0049] During operation, the fusion CMOS image sensor receives a light intensity variation signal as input, and the signal holding circuit, pulse selector, modal judgment circuit, arbitration circuit, and mode switch select an operating mode to control sensor operation. The operating modes include Mode 1 and Mode 2. The light intensity variation signal is generated by the event generation circuit through photodiode detection of scene light intensity variations. The signal holding circuit temporarily stores the asynchronous event signal generated by the light intensity variation signal using three D-type flip-flops. This signal is then logically processed using a combination of AND and OR gates to generate a control signal that triggers RGB pixel exposure. The modal judgment circuit analyzes the trigger conditions for consecutive events, such as two, three, or more consecutive positive or negative events, and outputs a logic signal. The arbitration circuit prioritizes trigger logic for three or more consecutive events, and the pulse selector selects the trigger pulse for single or multiple exposures based on the logic output of the modal judgment circuit. The mode switch determines whether the operating mode is Mode 1 or Mode 2 based on the processing results of the light intensity variation signal. In shared photodiode mode (Mode 1), the event pixel circuit and the RGB pixel circuit are connected to the same photodiode. The exposure pulse drive circuit (EPDU) drives the single-pixel exposure unit (SPU) for single or high-frequency continuous exposure based on the trigger pulse of the pulse selector. The readout circuit dynamically maintains the RGB matrix and outputs low-redundancy RGB frames, suitable for high-response speed scenarios. In independent pixel mode (Mode 2), the event pixel circuit and the RGB pixel circuit are independent, with one event pixel circuit controlling multiple RGB pixel circuits. The EPDU drives the SPU for single or multiple continuous exposures based on the trigger pulse of the pulse selector. The readout circuit outputs high-resolution RGB frames, suitable for scenes with high texture detail.

[0050] See also Figure 1Mode 1: The event pixel circuit and the RGB pixel circuit are connected to the same photodiode PD. The RGB pixel circuit does not use a synchronization signal for control; instead, the event pixel circuit trigger signal module controls exposure. The event pixel circuit's event signal output circuits are connected to a signal holding circuit consisting of three D-type flip-flops. Two AND gate logic circuits monitor the three D-type flip-flops in the signal holding circuit for two or more (three or more) consecutive positive or negative events. When two consecutive positive or negative events are detected, the two AND gate logic circuits control the RGB pixel circuit for a single exposure. When multiple consecutive positive or negative events are detected, the two AND gate logic circuits control the RGB pixel circuit for high-frequency continuous exposure. When multiple consecutive positive or negative events are triggered, the control timing takes precedence over the triggering of two consecutive positive or negative events. After exposure, the RGB pixel circuit is reset.

[0051] Preferably, the exposure of the RGB pixel circuit is controlled by an event pixel circuit.

[0052] Preferably, the RGB pixel exposure control circuit is composed of a multi-modal continuous event trigger monitoring circuit, including a signal holding circuit, an exposure pulse driving circuit EPDU and a mode judgment circuit.

[0053] Preferably, the exposure modes (single exposure and high-frequency continuous exposure) of the RGB pixel circuit are controlled by an event-triggered mode, that is, a single exposure is performed when two consecutive positive events or negative events are detected, and a high-frequency continuous exposure is performed when multiple consecutive positive events or negative events are continuously detected.

[0054] Preferably, the RGB pixel circuit asynchronously outputs the RGB value of the last exposure, ie, the output value of the CMOS image sensor.

[0055] See also Figure 2 Mode 2: The event pixel circuit and the RGB pixel circuits are independent. One event pixel circuit and multiple RGB pixel circuits within its neighborhood form a control group. The RGB pixel circuits are not controlled by synchronization signals; instead, the event pixel circuit trigger signal module controls exposure. The event signal output circuits of the event pixel circuits are connected to a signal holding circuit consisting of three D flip-flops. Two AND gate logic circuits monitor the three D flip-flops in the signal holding circuit for two or more (three or more) consecutive positive or negative events. When two consecutive positive or negative events are detected, the two AND gates control the RGB pixel circuits in the neighborhood control group for a single exposure. When three consecutive positive or negative events are detected, the two AND gates control the RGB pixel circuits in the neighborhood control group for high-frequency continuous exposure. When three consecutive positive or negative events are triggered, the control timing takes precedence over that of two consecutive positive or negative events. After exposure, the RGB pixel circuits are reset.

[0056] Preferably, the event pixel circuit and a plurality of RGB pixel circuits in its neighborhood form a control group, and within the control group, the event pixel circuit is connected to the RGB pixel circuit.

[0057] Preferably, the multimodal continuous events in the control group trigger the monitoring circuit to control the exposure of all RGB pixels in the control group.

[0058] Furthermore, the positive event channel and the negative event channel of the event trigger circuit in the event pixel circuit are respectively connected to a signal holding circuit formed by three D flip-flops.

[0059] Furthermore, the expression of the D flip-flop state is:

[0060]

[0061] in, Indicates the next moment Output status; Indicates the input status at time t; RST is the reset signal.

[0062] Furthermore, the signal holding circuit state is obtained according to the three D flip-flop states, and the expression is:

[0063]

[0064] Among them, CLK is the clock signal; Indicates the input state at time t; represents the state of the first D flip-flop DFF1 at time t; represents the state of the second D flip-flop DFF2 at time t; Indicates that the first D flip-flop DFF1 will Status; Indicates that the second D flip-flop DFF2 will Status; Indicates that the D third flip-flop DFF3 will be status.

[0065] It can be understood that, for the three D flip-flops DFF1, DFF2, and DFF3, the flip-flops receiving negative events are DFF1-OFF, DFF2-OFF, and DFF3-OFF, and the flip-flops receiving positive events are DFF1-ON, DFF2-ON, and DFF3-ON.

[0066] Furthermore, two continuous signal triggering modes in the three D flip-flops in the signal holding circuit are monitored respectively through two AND gate logic circuits, namely trigger mode 1: monitoring of two continuously triggered positive events or negative events; trigger mode 2: monitoring of three or more continuously triggered positive events or negative events.

[0067] For trigger mode 1, the AND gate and NAND gate of two positive or negative events are obtained. The expression is:

[0068]

[0069] in, AND gate representing two positive or negative events; The state DFF2-ON indicates that the second D flip-flop DFF2 receives a positive event; The state DFF3-ON indicates that the third D flip-flop DFF3 receives a positive event; NAND gate that represents two positive events or negative events; The state of DFF2-OFF indicating that the second D flip-flop DFF2 receives a negative event is not; The state of DFF3-OFF indicating that the third D flip-flop DFF3 receives a negative event is negated.

[0070] Furthermore, the judgment condition of trigger mode 1 is obtained, and the expression is:

[0071]

[0072] in, Indicates the judgment condition for trigger mode 1.

[0073] Furthermore, if =1, that is, the trigger condition is , the single pulse generator SPG in the arbitration circuit triggers the RGB pixel circuit for single exposure:

[0074]

[0075] in, Indicates time t The exposure drive signal; is the initial moment, is the pulse width; is a rectangular function.

[0076] For trigger mode 2, obtain the AND gate logic and NAND gate logic of three or more positive events or negative events. The expression is:

[0077]

[0078] in, AND gate logic representing three or more positive or negative events; NAND gate logic that represents three or more positive or negative events; Indicates the state of DFF1-ON when the first D flip-flop DFF1 receives a positive event; The state of DFF1-OFF indicating that the first D flip-flop DFF1 receives a negative event is not; The state of DFF2-OFF indicating that the second D flip-flop DFF2 receives a negative event is not; The state of DFF3-OFF indicating that the third D flip-flop DFF3 receives a negative event is negated.

[0079] Furthermore, the judgment condition of trigger mode 2 is obtained for:

[0080]

[0081] Furthermore, if =1, multiple high-frequency continuous exposures are triggered by the multi-pulse generator MPG in the arbitration circuit, and the expression is:

[0082]

[0083] in, Indicates time t The exposure drive signal; T is the pulse period, D is the duty cycle, N is the number of pulses, is the initial moment; i A count representing the number of exposures.

[0084] Furthermore, monitoring of three or more positive events or negative events that are triggered continuously has a higher priority than monitoring of two positive events or negative events that are triggered continuously. The pulse selector MUX performs arbitration and outputs the selection signal, and the expression is:

[0085]

[0086]

[0087] in, Indicates time t selection signal.

[0088] Furthermore, the selection signal is transmitted to the exposure pulse driving circuit EPDU, which drives the RGB pixel circuit to perform single or high-frequency continuous exposure, realizes high time resolution synchronization of the event signal and the RGB pixel, and outputs the RGB value obtained by the last exposure.

[0089] like Figure 2 As shown, for mode 2 in which the event pixel circuit and the RGB pixel circuit are independent, a CMOS image sensor that fuses events and frames and asynchronously controls the exposure mode of RGB pixels by a pulse signal is used. Figure 1 Based on the above, the event pixel circuit and the RGB pixel circuit are separated. The event pixel circuit only contains the event trigger circuit and the continuous event monitoring control circuit (signal holding circuit), and the pulse selector MUX outputs the selection signal.

[0090] Furthermore, the exposure pulse driving circuit EPDU controls all RGB pixel units (RGB0~RGBn) and exposure units (SPU1~SPUn) in the control group to perform single or multiple continuous exposures, and outputs the RGB values ​​obtained by the last exposure of each RGB pixel circuit in the control group.

[0091] Furthermore, one event pixel circuit is surrounded by multiple RGB pixel circuits, that is, one event pixel circuit controls N RGB pixel circuits.

[0092] For example, Figure 3 As shown in Figure 2, when the CMOS image sensor is in Mode 2, the event pixel circuit Event in the control group is independent of the RGB pixel circuits. Event pixel circuit Event0 and the eight RGB pixel circuits RGB0-1 through RGB0-8 form a control group. Event pixel circuit Event0 simultaneously controls the exposure time and exposure mode of the eight RGB pixel circuits RGB0-1 through RGB0-8.

[0093] For example, Figure 4 As shown, when the CMOS image sensor is in Mode 2, the control group's event pixel circuit Event0 and the twelve RGB pixel circuits RGB0-1 through RGB0-12 form a control group Con0. Event pixel circuit Event0 simultaneously controls the exposure time and exposure mode of the twelve RGB pixel circuits RGB0-1 through RGB0-12. Event pixel circuit Event1 and the four RGB pixel circuits RGB1-1 through RGB1-4 form a control group Con1. Event pixel circuit Event1 also controls the exposure time and exposure mode of the four RGB pixel circuits RGB1-1 through RGB1-4. Con0 and Con1 are completely independent and operate independently of each other.

[0094] like Figure 5As shown, a structure for dynamically controlling the RGB pixel matrix of a CMOS image sensor that integrates events and frames, using pulse signals to asynchronously control the exposure mode of RGB pixels. The RGB pixel array is part of the CMOS pixel array. Based on the light intensity change signal (generated by the event pixel circuit through photodiode detection) as an input parameter, it is processed by the event generation circuit, signal holding circuit, pulse selector, and exposure pulse drive circuit EPDU to output the asynchronous event signal and the RGB pixel matrix (composed of RGB pixel values). Specifically:

[0095] RGB pixel matrix C The dimension is , H For CMOS pixel matrix high, W is the width of the CMOS pixel matrix, including The dimensions of the RGB pixel matrix are determined by the size of the CMOS pixel array. The RGB pixel matrix stores the R, G, and B three-channel pixel values. The expression for the pixel value at is:

[0096]

[0097] in, Represents the pixel coordinates in the RGB array The RGB value at x Represents the horizontal coordinate in the RGB pixel array, y Represents the vertical coordinate in the RGB pixel array; Indicates time t The red value of the pixel's asynchronous exposure; Indicates time t Green value of the pixel for asynchronous exposure; Indicates time t The blue value of the pixel's asynchronous exposure.

[0098] When the coordinates of the RGB pixel array are When the RGB pixels are asynchronously exposed, the output RGB values ​​are used to replace the previous RGB pixel array coordinates. For the RGB pixel array coordinates There is no RGB pixel selected for asynchronous exposure, and the pixel at this coordinate retains the original RGB value Unchanged, realize RGB matrix C Finally, by actively reading out the signal, the RGB matrix obtained from the last exposure is output. F .

[0099] Specifically, at the initial moment when the RGB pixel circuit in the CMOS pixel circuit works, a global exposure is performed to obtain the initial value of the RGB matrix After that, the RGB matrix is ​​selected according to the signal ,renew The value of the RGB pixel matrix at , the expression is:

[0100] .

[0101] Furthermore, an active readout signal is introduced S When the active readout signal is valid, the image frame consisting of the matrix of RGB values ​​obtained from the last exposure is obtained. F , the expression is:

[0102]

[0103] in, F An image frame consisting of a matrix of RGB values ​​obtained from the last exposure; C Represents the RGB matrix C The value of is an indicator function, which is 1 if the condition is met and 0 otherwise.

[0104] For example, Figure 5 The RGB matrix in is , in each cell t Indicates t The RGB value is updated constantly. is the initial moment, at which a global pixel exposure is performed so that the RGB matrix C has an initial value. ~ Indicates the moment of exposure of RGB pixels triggered by asynchronous control of pulse signals under monotonically increasing moments. The actual cell stores the updated RGB value obtained by exposure at that moment. For intuitive representation, only the cell is marked .

[0105] For example, Figure 6 As shown, a structure for dynamically maintaining an RGB matrix of a CMOS image sensor that asynchronously controls RGB pixel exposure modes using pulse signals to fuse events and frames is disclosed.

[0106] Figure 6 This paper demonstrates an RGB matrix paradigm constructed with multiple, multi-mode control groups. There are 12 control groups, Con0 through Con12. Each control group is controlled by event pixel circuits, Event0 through Event12. Different control groups determine the exposure conditions for the RGB pixels within the group, and all RGB pixels within the group have the same exposure time and exposure mode.

[0107] Although the specific embodiments of the present invention depict various actions or steps in a specific order, this should be understood as requiring such actions or steps to be performed in the specific order shown or in a sequential order, or requiring that all illustrated actions or steps should be performed to obtain the desired result. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although a number of specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present invention. Certain features described in the context of separate embodiments can also be implemented in a single implementation in combination. Conversely, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination. The above is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or replacements that can be easily thought of by any technician in this field within the technical scope disclosed by the present invention should be included within the scope of protection of the present invention.

[0108] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A fusion CMOS image sensor with event-driven frame exposure, characterized in that: It includes a CMOS pixel array, an event generation circuit, a signal holding circuit, a pulse selector MUX, an exposure pulse drive circuit EPDU, a single pixel exposure unit SPU, a mode judgment circuit, an arbitration circuit, a reading circuit and a mode switching switch; The CMOS pixel array includes an event pixel circuit and an RGB pixel circuit, which are connected to an event generation circuit, a modal judgment circuit, an arbitration circuit, and a mode switching switch, and is used to detect light intensity changes in a scene and generate asynchronous event signals and capture color image information; The event generation circuit includes a photodiode PD, an operational amplifier Ab, a capacitor C1, a capacitor C2, a comparator, and a switch. The event generation circuit is connected to the signal holding circuit and is used to monitor light intensity changes in real time and generate a trigger signal. It is used to detect light intensity change signals in the scene and generate asynchronous event signals. The signal holding circuit includes three D flip-flops, which are connected to the pulse selector MUX through a combination circuit of an AND gate and an OR gate, and are used to temporarily store asynchronous event signals and synchronize the output RGB signals of the RGB pixel circuit; The pulse selector MUX is connected to the signal holding circuit and is used to select the execution of single or multiple exposure logic; The exposure pulse driving circuit EPDU is connected to the single pixel exposure unit SPU and is used to control the exposure mode of the single pixel exposure unit SPU; The single pixel exposure unit SPU is controlled by the exposure pulse drive circuit EPDU to perform the exposure task; The exposure pulse drive circuit EPDU outputs a control signal, and the single-pixel exposure unit SPU performs exposure after receiving the control signal; Reading circuit, used to output the RGB value obtained by the final exposure; The light intensity change signal is input, and the signal holding circuit, pulse selector MUX, mode judgment circuit, arbitration circuit and mode switching switch select the operating mode. The operating modes include Mode 1 and Mode 2. In Mode 1, the event pixel circuit and the RGB pixel circuit are connected to the photodiode PD, and the RGB pixel circuit is controlled by the event pixel circuit triggering the signal module to control the exposure. In mode 2, the event pixel circuit and the RGB pixel circuit are independent. The event pixel circuit and multiple RGB pixel circuits in its neighborhood form a control group. The exposure of the RGB pixel circuit is controlled by the event pixel circuit trigger signal module.

2. The fused CMOS image sensor according to claim 1, wherein: In Mode 1 and Mode 2, the event signal output circuits of the event pixel circuits are connected to the signal holding circuits respectively. Two AND gates are used to monitor whether there are two or more consecutive positive events or negative events in the three D flip-flops in the signal holding circuits respectively. Exposure is performed based on the monitoring results.

3. The fused CMOS image sensor according to claim 2, wherein: When two consecutive positive events or negative events are detected, the two AND gates control the RGB pixel circuit to perform single exposure; when multiple consecutive positive events or negative events are detected, the two AND gates control the RGB pixel circuit to perform high-frequency continuous exposure.

4. The fused CMOS image sensor according to any one of claims 1 to 3, wherein: The expression of the D flip-flop state is: in, Indicates the next moment Output status; Indicates the input status at time t; RST is the reset signal.

5. The fused CMOS image sensor according to claim 4, wherein: The signal holding circuit state is obtained according to the three D flip-flop states. The expression is: Among them, CLK is the clock signal; Indicates the input state at time t; represents the state of the first D flip-flop DFF1 at time t; represents the state of the second D flip-flop DFF2 at time t; Indicates that the first D flip-flop DFF1 is Status; Indicates that the second D flip-flop DFF2 will Status; Indicates that the D third flip-flop DFF3 will be status.

6. The fused CMOS image sensor according to claim 5, wherein: Two continuous signal trigger modes in the three D flip-flops are monitored respectively through two AND gates. The two continuous signal trigger modes are trigger mode 1 and trigger mode 2. Among them, trigger mode 1 is to monitor two positive events or negative events that are continuously triggered; trigger mode 2 is to monitor three or more positive events or negative events that are continuously triggered.

7. The fused CMOS image sensor according to claim 6, wherein: The judgment conditions of trigger mode 1 and trigger mode 2 are obtained; and a selection signal is obtained according to the judgment conditions.

8. The fused CMOS image sensor according to claim 7, wherein: Based on the selection signal, the exposure pulse driving circuit EPDU controls the single-pixel exposure unit SPU to select single or multiple continuous exposures, and outputs the RGB value obtained by the final exposure.

Images