Sensor mode switching method in event-based sensors and imaging cameras for low power applications

Abstract

An image sensor includes a plurality of pixels. A pixel is capable of detecting a change in an amount of light intensity, and the pixel is capable of detecting an amount of light intensity. In a first mode, the sensor outputs data from one or more first pixels of the pixels. In a second mode, the sensor outputs data from one or more second pixels of the pixels. Based on at least one of a result of processing of the data output in the first mode and a result of processing of the data output in the second mode, a processor selects at least one of the first mode and the second mode.

Description

Technical Field

[0001] This disclosure relates to cameras having both event-based sensor capabilities and image sensor capabilities, and to dynamic switching between event-based sensors and RGB sensors for low-power applications. Background Technology

[0002] In related technologies, RGB sensors are used to generate image data to capture details of a scene. RGB sensors provide imagery in many applications, ranging from security to sporting events. However, RGB sensors may require high power consumption rates, making their use impractical or undesirable in many applications. For example, in some applications, it may be desirable for a camera to record video and / or still image data related to an event. If the event is infrequent, or if the timing of the event is unknown or difficult to predict, the RGB camera may need to operate continuously at a high frame rate, which can make the use of an RGB camera or sensor impractical due to its high power consumption rate. As a result, recording image data for such events using conventional methods would require a significant amount of power.

[0003] What is needed is an image capture system capable of capturing image data related to the event without excessive power consumption. Summary of the Invention

[0004] Technical issues

[0005] Cameras that combine event-based sensor (“EBS”) capabilities with RGB sensing capabilities and trigger imaging or RGB functionality in response to event detection can overcome some limitations of using conventional imaging devices or RGB sensors alone, providing efficient operation while reliably detecting events. By intelligently using EBS data collected in EBS operating mode to detect the occurrence of events or scenes where imaging data is required, the high-power RGB mode can be activated or deactivated only when needed.

[0006] EBS sensors offer the benefits of low-power operation. They enable high-speed event detection while operating at low power. RGB sensors, on the other hand, provide high-precision color images and / or video while operating at relatively higher power compared to EBS sensors. As disclosed herein, an RGB sensor can be activated when a trigger event is recorded based on information received from an EBS sensor. Switching to RGB mode is possible when the number of events in EBS mode exceeds a predefined spatiotemporal density or when the number of events falls below a predefined spatiotemporal density.

[0007] In some implementations, the triggering event may include high event density detection from the EBS sensor. In some implementations, the triggering event may include low event density detection from the EBS sensor. For example, RGB mode may be triggered when the event density in the EBS, either throughout the frame or in a predefined region of the frame, exceeds a threshold or falls below a threshold.

[0008] In some implementations, the triggering event may be based on data analyzed by a neural network from an EBS sensor. In some implementations, the triggering event may be based on EBS events tracked by a recurrent neural network. For example, EBS data can be fed into a neural network, and the RGB mode can be triggered when a desired group of object categories is identified and / or detected, or when a meaningful group of events is detected.

[0009] In some implementations, the triggering event can be the EBS movement direction. For example, after detecting an event sequence and detecting an object moving at a specific speed or in a specific direction, the system can switch to RGB.

[0010] Therefore, this disclosure provides cameras, sensor systems, apparatuses, and methods that have improved image sensor efficiency and performance compared to other configurations and are capable of providing both imaging and event detection functions.

[0011] Technical solutions to the problem

[0012] According to embodiments and aspects of this disclosure, a camera or sensor system is provided that has EBS and image sensor (e.g., red, green, and blue (“RGB”) image sensor) capabilities or functions. The EBS and image sensor capabilities can be provided by separate EBS and image sensor devices. The EBS and image sensing capabilities can also be implemented by a sensor device having combined EBS and image sensing capabilities. A sensor device having combined EBS and image sensing capabilities may include a sensor device with a pixel array comprising EBS pixels and image sensing pixels. Furthermore, the combined EBS and image sensor may include a photoelectric conversion region disposed as part of the pixels performing the EBS and image sensing functions. For ease of illustration, the discussion herein will refer to the EBS and image sensor functions provided by separate EBS and image sensors; however, unless otherwise specifically stated, it should be understood that EBS and image sensors can be integrated into a single sensor device providing both EBS and image sensor functions.

[0013] As disclosed herein, an event detected by an EBS sensor or by a pixel with EBS capability can trigger activation of an image sensor or a pixel capable of image sensing, such as an RGB pixel. Furthermore, according to at least some embodiments of this disclosure, activation of the image sensor includes activating the image sensor at a specific frame rate. According to at least some embodiments of this disclosure, the event may include motion or no motion, or the event may include an intensity change or no intensity change, or event detection may include detecting a specific object or an object moving in a specific direction.

[0014] According to at least some embodiments and aspects of this disclosure, event detection can be performed by a processor communicating with an EBS sensor or a sensor with EBS capability. The processor can implement neural networks or other types of analysis algorithms. For example, the processor can analyze EBS sensor data and detect events corresponding to objects moving relative to the EBS sensor, and in response, can activate or deactivate the RGB capability of an RGB sensor or certain pixels. According to other embodiments and aspects of this disclosure, detection of a relatively fast-moving object may result in the activation of an image sensor at a relatively high frame rate. The frame rate of the activated image sensor can also vary based on the characteristics of the object detected by the processor analyzing the EBS sensor data. For example, the image sensor may operate at a relatively low frame rate if the detected object is moving slowly. The image sensor may operate at a relatively high frame rate if the detected object is moving quickly.

[0015] According to at least some embodiments of this disclosure, the characteristics of objects detected by an EBS sensor can be analyzed in conjunction with determining the operating parameters of the image sensor. For example, a neural network or other decision-making facility can determine whether a detected event has been triggered by an object within a desired object category. If the desired object category has been detected, the frame rate of the image sensor can be adjusted based on the characteristics of the object. The amount of time the image sensor is activated can also depend on the detection characteristics of the detected object. For example, data from the EBS sensor can be analyzed by, for example, a neural network or other decision-making facility to perform object recognition, object classification, object detection, or gesture recognition, etc.

[0016] Typically, it is desirable to interrupt the operation of the image sensor and return to EBS sensor operation after certain conditions are met to save power. Embodiments and aspects of this disclosure can interrupt the operation of the image sensor and return the system to EBS mode when certain conditions are met. These can include after determining that nothing of interest has occurred. For example, after a previously moving object stops, imaging of the object can be interrupted, and the image sensor can return to sleep mode. Image sensor operation can also be interrupted after an object is identified and it is determined that continued imaging of the identified object is unnecessary or undesirable. As another example, image sensor operation can be interrupted after the object has moved out of the imaging scene. As yet another example, image sensor operation can be interrupted after a predetermined period of time. According to embodiments of this disclosure, EBS sensor operation can remain continuously active regardless of whether image sensor operation is active.

[0017] This disclosure can provide a camera, system, or device with event-based sensing and imaging capabilities that can improve power consumption, data transmission, and data processing efficiency. Attached Figure Description

[0018] Figure 1 This is a block diagram illustrating a schematic example of the construction of an imaging apparatus according to an embodiment of the present disclosure.

[0019] Figure 2 This is a view illustrating an example of a stacked structure of an image sensor according to an embodiment of the present disclosure.

[0020] Figure 3 This is a block diagram illustrating an example of the functional construction of an image sensor according to an embodiment of the present disclosure.

[0021] Figure 4 The illustration shows an example of a unit pixel array according to an embodiment of the present disclosure in the case of using a Bayer array in a color filter array.

[0022] Figure 5A An aspect of an image sensor according to an embodiment of the present disclosure is illustrated.

[0023] Figure 5B The illustration shows aspects of an image sensor according to other embodiments of the present disclosure.

[0024] Figure 5C The illustration shows aspects of an image sensor according to other embodiments of the present disclosure.

[0025] Figure 5D The illustration shows aspects of an image sensor according to other embodiments of the present disclosure.

[0026] Figure 5E The illustration shows aspects of an image sensor according to other embodiments of the present disclosure.

[0027] Figure 5F The illustration shows aspects of an image sensor according to other embodiments of the present disclosure.

[0028] Figure 6A This is a circuit diagram illustrating a schematic example of the construction of a unit pixel having a combined event detection function and an image sensor function according to an embodiment of the present disclosure.

[0029] Figure 6B This is a circuit diagram illustrating a schematic example of the construction of an image sensing pixel group according to an embodiment of the present disclosure.

[0030] Figure 6C This is a circuit diagram illustrating a schematic example of the construction of an event detection pixel according to an embodiment of the present disclosure.

[0031] Figure 7 This is a block diagram illustrating a schematic construction example of an address event detection unit according to an embodiment of the present disclosure.

[0032] Figure 8 This is a circuit diagram illustrating a schematic example of the construction of a subtractor and a quantizer according to an embodiment of the present disclosure.

[0033] Figure 9 This is a block diagram illustrating a schematic example of the construction of a column ADC according to an embodiment of the present disclosure.

[0034] Figure 10A This is a timing diagram illustrating an example of the operation of an image sensor according to an embodiment of the present disclosure.

[0035] Figure 10B This is a timing diagram illustrating an example of the operation of an image sensor according to other embodiments of the present disclosure.

[0036] Figure 11 This is a flowchart illustrating an operational example of an imaging apparatus according to an embodiment of the present disclosure.

[0037] Figure 12 This is a block diagram of a switching circuit according to an embodiment of this disclosure.

[0038] Figure 13A An exemplary EBS frame that can be analyzed by an imaging apparatus according to an embodiment of this disclosure is depicted.

[0039] Figure 13B An exemplary EBS frame that can be analyzed by an imaging apparatus according to an embodiment of this disclosure is depicted.

[0040] Figure 13C An exemplary EBS frame that can be analyzed by an imaging apparatus according to an embodiment of this disclosure is depicted.

[0041] Figure 14A Convolutional neural networks according to other embodiments of this disclosure are described.

[0042] Figure 14B Recurrent neural networks according to other embodiments of this disclosure are depicted.

[0043] Figure 15A This is a diagram of RGB data according to other embodiments of this disclosure.

[0044] Figure 15B This is a diagram of EBS data according to other embodiments of this disclosure. Detailed Implementation

[0045] In the following, embodiments of the present disclosure will be described in detail based on the accompanying drawings. Furthermore, in the following embodiments, the same reference numerals are assigned to the same or equivalent parts or elements, and repeated descriptions thereof are omitted.

[0046] Typical event-based sensors (EBS) employ a so-called event-driven approach, in which the presence or absence of an address event ignition is detected for each unit pixel, and the pixel signal is read out from the unit pixel from which the address event ignition and ignition timing information are detected.

[0047] Furthermore, in this specification, a unit pixel refers to a pixel comprising a photoelectric conversion element (also known as a "photoreceiving element") or the smallest unit of a unit pixel, and as an example, may correspond to each point in the image data read from an image sensor. Additionally, an address event refers to an event occurring for each address that can be assigned to each of a plurality of unit pixels arranged in a two-dimensional lattice shape. The event detection sensor responds to intensity changes without being limited by the intra-frame integration time constraints of conventional image sensors. Intensity changes are associated with changes in photocurrent, and if such a change exceeds a constant threshold, it can be detected as an event.

[0048] Figure 1 This is a block diagram illustrating a schematic construction example of an imaging apparatus according to at least some embodiments of the present disclosure. Figure 1 As shown, for example, imaging device 100 includes an imaging lens 110, a solid-state imaging device or image sensor 200, a recording unit 120, a communication interface 124, and a processor system or control system 130. Various components of imaging device 100 can be interconnected with each other via a communication bus 128 or signal lines. As an example, imaging device 100 may be configured as a camera mounted in an industrial robot, a vehicle-mounted camera, or part thereof, or as part of or connected to other devices or instruments.

[0049] Imaging lens 110 may include an optical system for collecting light from within field of view 114. The collected or incident light is guided (e.g., converged) onto the light-receiving surface of image sensor 200. Specifically, imaging lens 110 can collect light from a selected area of ​​the scene by guiding the field of view 114 to include a selected area of ​​the scene. The light-receiving surface is the surface of a substrate on which photoelectric conversion elements of pixels 310 included in image sensor 200 are arranged. Image sensor 200 performs photoelectric conversion on incident light to generate image data. As discussed herein, image sensor 200 may include different groups of photoelectric conversion elements arranged on the same or different substrates. Furthermore, image sensor 200 may include photoelectric conversion elements performing one or more functions. These functions may include event detection and imaging functions. Additionally, image sensor 200 may perform predetermined signal processing on the generated image data, such as noise removal and white balance adjustment. The results obtained through signal processing and detection signals indicating the presence or absence of address event bursts and burst time information may be output by image sensor 200 to processor system 130. The method for generating a detection signal indicating the presence or absence of an address event will be explained later.

[0050] For example, the recording unit 120 is composed of flash memory, dynamic random access memory (DRAM) or static random access memory (SRAM), and records data provided by the image sensor 200.

[0051] For example, processor system 130 may be composed of a central processing unit (CPU) or the like. For example, processor system 130 may include one or more general-purpose processors, controllers, field-programmable gate arrays (FPGAs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), or combinations thereof. Furthermore, processor system 130 may implement application programming or routines as software or firmware stored in memory or data memory included in or interconnected with processor system 130 to perform various functions and methods as described herein. For example, processor system 130 may process data output from image sensor 200. For example, as described herein, processor system 130 may process event detection signals output by the EBS sensor function or portion of image sensor 200, and may control the imaging sensor function or portion of the solid-state imaging device, at least in part, in response to the event detection signals. In addition to image sensor 200, processor system 130 may also control other components of imaging device 100, such as the operation of recording unit 120, communication interface 124, and focusing and shutter operations that imaging lens 110 may support. As described herein, according to other embodiments of this disclosure, processor system 130 can implement advanced processing capabilities, including but not limited to neural network and artificial intelligence capabilities and functions.

[0052] Next, an example of the construction of the image sensor 200 will be described in detail with reference to the accompanying drawings.

[0053] Figure 2 This is a view illustrating an example of a stacked structure of an image sensor 200 according to at least some embodiments of the present disclosure. Figure 2 As shown, the image sensor 200 can have a structure in which a light-receiving chip 201 and a logic chip 202 are vertically stacked. The side of the light-receiving chip 201 opposite to the logic chip 202 is a light-receiving surface 204. In the bonding of the light-receiving chip 201 and the logic chip 202, for example, a so-called direct bonding can be used, in which the bonding surfaces of the chips are planarized and the chips are stacked by inter-electro-forces. However, it is not limited to this; for example, a so-called Cu-Cu bonding or bump bonding, in which copper (Cu) electrode pads formed on the bonding surfaces are bonded, can also be used.

[0054] Additionally, for example, the optical receiver chip 201 and the logic chip 202 are electrically connected to each other via a connection such as a through-silicon via (TSV) through a semiconductor substrate. In TSV connections, for example, a so-called dual TSV method can be employed: in which two TSVs, including a TSV formed in the optical receiver chip 201 and a TSV formed from the optical receiver chip 201 to the logic chip 202, are connected to each other on the outer surface of the chip; and a so-called shared TSV method: in which the optical receiver chip 201 and the logic chip 202 are connected via a TSV penetrating both chips, and so on.

[0055] However, when Cu-Cu bonding or bump bonding is used in the bonding of the optical receiver chip 201 and the logic chip 202, the optical receiver chip 201 and the logic chip 202 are electrically connected to each other through Cu-Cu connectors or bump connectors.

[0056] Those skilled in the art will understand upon considering this disclosure that the imaging device 200, implemented as a connection between the light receiving chip 201 and the logic chip 202, may include an image sensor 200 component arranged as part of the light receiving chip 201, and some or all of the processor system 130 components arranged as part of the logic chip 202. Other components, such as the recording unit 120 component and the communication interface component, may be distributed within one or both of the chips 201 and 202. According to other embodiments, a data memory or other chip may be stacked and electrically connected to the light receiving chip 201 and the logic chip 202. Furthermore, for example, in the case where the image sensor 200 includes multiple sensor devices, the light receiving chip may include multiple substrates bonded to respective logic chips 202 or a common logic chip 202.

[0057] Figure 3This is a block diagram illustrating an example of the functional construction of an image sensor 200 according to at least some embodiments of the present disclosure. Figure 3 As shown, the image sensor 200 may include a driving circuit 211, a signal processing unit 212, an arbitrator 213, a column ADC 220, and a pixel array unit 300. Some or all of the components may be fully or partially integrated into the processor system 130 or implemented by the processor system 130.

[0058] Multiple unit cells or unit pixels 310 (hereinafter also simply referred to as pixels 310) are arranged in pixel array unit 300. Details of the unit pixels 310 will be described later. For example, each unit pixel 310 includes a photoelectric conversion element such as a photodiode and a pixel signal (hereinafter referred to as pixel circuitry) that generates a voltage value corresponding to the amount of charge generated in the photoelectric conversion element. Furthermore, as discussed in more detail elsewhere in this document, the pixel circuitry may include either or both of a first or imaging signal generation circuitry and a second or address event detection readout circuitry. Each photoelectric conversion element may be associated with a corresponding pixel circuitry, or multiple photoelectric conversion elements may be associated with a shared pixel circuitry.

[0059] In this example, multiple unit pixels 310 are arranged in a two-dimensional grid shape in pixel array unit 300. The multiple unit pixels 310 can be grouped into multiple pixel blocks or groups, each pixel block or group including a predetermined number of unit pixels. In the following text, the set of unit pixels arranged in the horizontal direction is called a "row", and the set of unit pixels arranged in a direction orthogonal to the row is called a "column".

[0060] Each unit pixel 310 generates a charge corresponding to the amount of light received at the corresponding photoelectric conversion element. Furthermore, at least some of the unit pixels 310 can be operated to detect the presence or absence of an address event burst based on whether the value or change in the current (hereinafter referred to as photocurrent) generated by the charge produced in the photoelectric conversion element exceeds a predetermined threshold. When an address event bursts, a signal is output to the arbitrator 213.

[0061] Arbitrator 213 arbitrates requests received from unit pixel 310 performing the event detection function and transmits a predetermined response to the requesting unit pixel 310 based on the arbitration result. Unit pixel 310 that receives the response supplies a detection signal (hereinafter referred to as "address event detection signal") indicating the presence or absence of an address event burst to drive circuit 211 and signal processing unit 212.

[0062] The driving circuit 211 drives each unit pixel 310 and allows each unit pixel 310 to output a pixel signal to the column ADC 220.

[0063] For each unit pixel of 310 columns, the column ADC 220 converts the analog pixel signal from that column into a digital signal. Additionally, the column ADC 220 supplies the resulting digital signal to the signal processing unit 212.

[0064] The signal processing unit 212 performs predetermined signal processing on the digital signal transmitted from the column ADC 220, such as correlated double sampling (CDS) processing (noise removal) and white balance adjustment. In addition, the signal processing unit 212 supplies the signal processing results and address event detection signals to the recording unit 120 via signal line 209.

[0065] The unit pixels 310 within the pixel array unit 300 can be arranged in the pixel group 314. For example, in Figure 3 In the illustrated configuration, the pixel array unit 300 comprises a pixel group 314 including a set of unit pixels 310, which receive the wavelength components required for reconstructing color information from the scene. For example, in the case of color reconstruction based on RGB three primary colors, in the pixel array unit 300, optical color filter material can be deposited on the pixels according to a predetermined color filter array to control the arrival of light of the desired wavelength on the pixel surface. Specifically, unit pixels 310 that receive red (R) light, unit pixels 310 that receive green (G) light, and unit pixels 310 that receive blue (B) light are arranged in group 314 according to a predetermined color filter array.

[0066] Examples of color filter array constructions include various arrays or pixel groups, such as: a 2×2 pixel Bayer array; a 3×3 pixel color filter array used in X-Trans (trademarked) CMOS sensors (hereinafter also referred to as an "X-Trans (trademarked) type array"); a 4×4 pixel quad Bayer array (also referred to as a "Quadra array"); and a 4×4 pixel color filter array in which white RGB color filters are combined into the Bayer array (hereinafter also referred to as a "white RGB array"), and so on. Additionally, and as discussed in more detail elsewhere in this document, event detection pixels can be distributed or included within pixel array unit 300. Similarly, as discussed in more detail elsewhere in this document, event detection pixels can be configured as dedicated event detection pixels that perform only event detection functions, or as combined event detection and image sensing pixels that perform both event detection and image sensor functions.

[0067] Figure 4 This is a schematic diagram illustrating an example array of unit pixels 310 using pixel group 314, which has unit pixels 310 and associated color filters arranged in an array of color filters configured to form a plurality of Bayer arrays 310A. Figure 4 As shown, when using a Bayer array as the color filter array structure, in the pixel array unit 300, a basic pattern 310A comprising a total of four unit pixels 310 of 2×2 pixels is repeatedly arranged in the column and row directions. For example, the basic pattern 310A is composed of unit pixels 310R including a red (R) color filter 401, unit pixels 310Gr including a green (Gr) color filter 401, unit pixels 310Gb including a green (Gb) color filter 401, and unit pixels 310B including a blue (B) color filter 401.

[0068] Figures 5A-5D Various construction examples of the imaging apparatus 100 according to embodiments of the present disclosure are depicted, particularly the arrangement of pixels in the solid-state imaging device or image sensor 200. More specifically, Figure 5A An image sensor 200 is depicted having a first or EBS sensor 530 and a second or imaging sensor 540. The first or EBS sensor 530 includes an array of pixels 310 in the form of address event detection pixels 503 disposed on a first light receiving chip or substrate 201a, and the second or imaging sensor 540 includes an array of pixels 310 in the form of image sensing pixels 502 disposed on a second light receiving chip or substrate 201b. Those skilled in the art will understand upon considering this disclosure that an imaging device 100 including separate EBS sensors 530 and imaging sensors 540 can be configured to have separate lens assemblies 110 that collect light from the same or similar fields of view, or it can be configured to have a shared lens assembly 110 that directs light to sensors 530 and 540 via a beam splitter. According to embodiments of this disclosure, the number of address event detection pixels 503 included in the EBS sensor 530 may be equal to the number of image sensing pixels 502 included in the imaging sensor 540. Furthermore, the area of ​​each address event detection pixel 503 can be the same as the area of ​​each image sensing pixel 502. Alternatively, the EBS sensor 530 and the imaging sensor 540 can have different numbers of pixels 310. For example, the image sensor 200 may include an EBS sensor 530 with a relatively low number of event detection pixels 503, thereby providing a relatively low resolution, and an imaging sensor 540 with a relatively high number of image sensing pixels 502, thereby providing a relatively high resolution. According to at least some embodiments of this disclosure, event detection operations and image sensing operations can be performed simultaneously.

[0069] Figure 5BAn image sensor 200 with pixels 310 is depicted, which are configured as combined or shared event detection and image sensing pixels 501 arranged on a single light-receiving chip or substrate 201. Those skilled in the art will understand upon considering this disclosure that the shared event detection and image sensing pixels 501 can operate selectively in either an event detection mode or an image sensing mode. Furthermore, according to at least some embodiments of this disclosure, event detection operations and image sensing operations can be performed simultaneously using some pixels operating in event detection mode and some pixels operating in image sensing mode.

[0070] Figure 5C An image sensor 200 with an array of unit pixels 310 is depicted, the array including a plurality of event detection pixels 503 and a plurality of image sensing pixels 502 formed on the same light receiving chip or substrate 201. In the illustrated example, most of the unit pixels are in the form of image sensing pixels 502, with a small number of event detection pixels 503 arranged between the image sensing pixels 502. However, an image sensor 200 having both event detection pixels 503 and image sensing pixels 502 arranged on the same light receiving chip or substrate 201 may include the same number of pixels 502 and 503, or may have more event detection pixels 503 than image sensing pixels 502. According to at least some embodiments of the present disclosure, event detection operations and image sensing operations can be performed simultaneously.

[0071] Figure 5D An image sensor 200 with an array of unit pixels 310 is depicted, the array comprising a group of shared event detection and image sensing pixels 501 and a group of image sensing pixels 502 formed on the same light receiving chip or substrate 201. The respective groups can be configured as Bayer arrays alternating between the group of shared event detection and image sensing pixels 501 and the group of image sensing pixels 502. Therefore, Figure 5D This is an example of an image sensor 200 in which different shared event detection and image sensing pixels 501 can respond to light in different wavelength ranges. For example, the shared event detection and image sensing pixels 501 may be associated with a color filter. Alternatively, the shared pixels 501 may all receive light in the same wavelength range. Although the same number of groups containing the same number of corresponding pixels 310 are depicted in the figures, other configurations are possible. Those skilled in the art will understand upon considering this disclosure that the shared event detection and image sensing pixels 501 may be operated selectively in either event detection mode or image sensing mode. Furthermore, according to at least some embodiments of this disclosure, event detection operation and image sensing operation may be performed simultaneously.

[0072] Figure 5EAn image sensor 200 with an array of unit pixels 310 is depicted, the array comprising groups of shared event detection and image sensing pixels 501 and groups of event detection pixels 503 formed on the same light receiving chip or substrate 201. The respective groups of shared event detection and image sensing pixels can be configured as Bayer arrays alternating with the groups of event detection pixels 503. Although the same number of groups containing the same number of corresponding pixels 310 are depicted in the figures, other configurations are possible. Those skilled in the art will understand upon considering this disclosure that the shared event detection and image sensing pixels 501 can be operated selectively in either event detection mode or image sensing mode. Furthermore, according to at least some embodiments of this disclosure, event detection operation and image sensing operation can be performed simultaneously.

[0073] Figure 5F An image sensor 200 with an array of unit pixels 310 is depicted, the array comprising a group of shared event detection and image sensing pixels 501, a group of image sensing pixels 502, and a group of event detection pixels 503 all formed on the same light receiving chip or substrate 201. Some or all of the pixel groups may be configured as Bayer arrays. For example, in at least one exemplary configuration, the group of shared event detection and image sensing pixels 501 and the group of image sensing pixels may be configured as Bayer arrays, and each event detection pixel 503 may be configured to receive light from the same wavelength range. For example, the shared event detection and image sensing pixels 501 and the image sensing pixels may be associated with a color filter, and the event detection pixel 503 may not have a color filter. Although the arrangement of pixels 310 is half shared event detection and image sensing pixels 501, one-quarter of pixels 310 is image sensing pixels 502, and one-quarter of pixels 310 is event detection pixels 503 is possible, other configurations are also possible. Those skilled in the art will understand upon considering this disclosure that the shared event detection and image sensing pixel 501 can be selectively operated in either event detection mode or image sensing mode. Furthermore, according to at least some embodiments of this disclosure, event detection operations and image sensing operations can be performed simultaneously.

[0074] Next, we will explain an example of constructing a unit pixel of 310. Figure 6A This is a circuit diagram illustrating a schematic construction example of a unit pixel 310 according to at least some embodiments of the present disclosure, particularly according to an embodiment comprising an event detection (EBS) and image sensing (IS) pixel 501 configured to perform a combination or shared function of both event detection and image sensor functions. Figure 6AAs shown, for example, a unit pixel 310 includes a pixel imaging signal generation unit (or readout circuit) 320, a light receiving unit 330, and an address event detection unit (or readout circuit) 400. According to at least one exemplary embodiment, the event detection readout circuit 400 can trigger the operation of the imaging signal generation readout circuit 320 based on the charge generated by the photoelectric conversion element (or photoelectric conversion region) 333 and based on the operation of the logic circuit 210. For example, Figure 6A The logic circuit 210 in the middle includes Figure 3 The logic circuitry of the driving circuit 211, signal processing unit 212, and arbitrator 213 is described. According to at least some embodiments of this disclosure, the logic circuitry can be implemented in the processor system 130. As described in more detail elsewhere herein, the logic circuitry 210 can determine whether to trigger the operation of the imaging signal generation readout circuit 320 or the operation of the pixel imaging signal generation unit 320 associated with other unit pixels 310 based on the output of the event detection readout circuit 400 or the output of another event detection readout circuit 400.

[0075] For example, the light receiving unit 330 includes a first or imaging transmission transistor or gate (first transistor) 331, a second or address event detection transmission transistor or gate (second transistor) 332, and a photoelectric conversion element 333. A first transmission or control signal TG1 transmitted from the driving circuit 211 is selectively supplied to the gate of the first transmission transistor 331 of the light receiving unit 330, and a second transmission or control signal TG2 transmitted from the driving circuit 211 is selectively supplied to the gate of the second transmission transistor 332. The output of the first transmission transistor 331 of the light receiving unit 330 is connected to the pixel imaging signal generation unit 320, and the output of the second transmission transistor 332 is connected to the address event detection unit 400.

[0076] The pixel imaging signal generation unit 320 may include a reset transistor (third transistor) 321, an amplification transistor (fourth transistor) 322, a selection transistor (fifth transistor) 323, and a floating diffusion layer (FD) 324.

[0077] According to at least some embodiments of the present invention, for example, the first transmission transistor 331 and the second transmission transistor 332 of the light receiving unit 330 are constructed using N-type metal-oxide-semiconductor (MOS) transistors (hereinafter simply referred to as "NMOS transistors"). Similarly, for example, the reset transistor 321, the amplification transistor 322, and the selection transistor 323 of the pixel imaging signal generation unit 320 are each constructed using NMOS transistors.

[0078] The address event detection unit 400 may include a current-to-voltage conversion unit 410 and a subtractor 430. The address event detection unit 400 may also include a buffer, a quantizer, and a transmission unit. These will be discussed in conjunction with the following description. Figure 7 Details of the address event detection unit 400 are explained.

[0079] In the illustrated configuration, the photoelectric conversion element 333 of the light receiving unit 330 performs photoelectric conversion on the incident light to generate charge. The first transmission transistor 331, according to the first control signal TG1, transmits the charge generated in the photoelectric conversion element 333 to the floating diffusion layer 324 of the imaging signal generation readout circuit 320. The second transmission transistor 332, according to the second control signal TG2, supplies an electrical signal (photocurrent) based on the charge generated in the photoelectric conversion element 333 to the address event detection unit 400.

[0080] When the processor system 130 issues an instruction for image sensing, the drive circuit 211 in the logic circuit 210 outputs a control signal TG1 to set the first transmission transistor 331 of the light receiving unit 330 of the selected unit pixel 310 in the pixel array unit 300 to the ON state. Through this arrangement, the photocurrent generated in the photoelectric conversion element 333 of the light receiving unit 330 is supplied to the pixel imaging signal generation readout circuit 320 via the first transmission transistor 331. More specifically, the floating diffusion layer 324 accumulates the charge transferred from the photoelectric conversion element 333 via the first transmission transistor 331. The reset transistor 321 discharges (initializes) the charge accumulated in the floating diffusion layer 324 according to the reset signal transmitted from the drive circuit 211. The amplifying transistor 322 allows a pixel signal with a voltage value corresponding to the amount of charge accumulated in the floating diffusion layer 324 to appear in the vertical signal line VSL. The selection transistor 323 switches the connection between the amplifying transistor 322 and the vertical signal line VSL according to the selection signal SEL transmitted from the drive circuit 211. In addition, the analog pixel signals appearing in the vertical signal line VSL are read out by the column ADC 220 and converted into digital pixel signals.

[0081] When the processor system 130 issues an instruction to initiate address event detection, the drive circuit 211 in the logic circuit 210 outputs a control signal to set the second transmission transistor 332 of the light receiving unit 330 in the pixel array unit 300 to the on state. Through this arrangement, the photocurrent generated in the photoelectric conversion element 333 of the light receiving unit 330 is supplied to the address event detection unit 400 of each unit pixel 310 via the second transmission transistor 332.

[0082] When an address event burst is detected based on the photocurrent from the light receiving unit 330, the address event detection unit 400 of each unit pixel 310 outputs a request to the arbitrator 213. The arbitrator 213 arbitrates the requests transmitted from each unit pixel 310 and transmits a predetermined response to the unit pixel 310 that issued the request based on the arbitration result. The unit pixel 310 that receives the response supplies a detection signal (hereinafter referred to as the "address event detection signal") indicating the presence or absence of the address event burst to the drive circuit 211 and signal processing unit 212 in the logic circuit 210.

[0083] The driving circuit 211 can also set the second transmission transistor 332 in the unit pixel 310, which serves as the supply source for the address event detection signal, to an off state. This arrangement stops the supply of photocurrent from the light receiving unit 330 to the address event detection unit 400 in the unit pixel 310.

[0084] Next, the driving circuit 211 sets the first transmission transistor 331 in the light receiving unit 330 of the unit pixel 310 to the on state via the transmission signal TG1. Through this arrangement, the charge generated in the photoelectric conversion element 333 of the light receiving unit 330 is transmitted to the floating diffusion layer 324 via the first transmission transistor 331. Furthermore, a pixel signal with a voltage value corresponding to the amount of charge accumulated in the floating diffusion layer 324 appears in the vertical signal line VSL connected to the selection transistor 323 of the pixel imaging signal generation unit 320.

[0085] As described above, in the image sensor 200, a pixel signal SIG is output from a unit pixel 310 where an address event burst is detected to the column ADC 220. According to other embodiments of this disclosure, a pixel signal is output from a group or subarray of unit pixels 310 associated with the address of the unit pixel 310 for which an address event detection signal has been provided.

[0086] Furthermore, for example, two logarithmic (LG) transistors (sixth and seventh transistors) 411 and 414 and two amplifying transistors (eighth and ninth transistors) 412 and 413 in the current-to-voltage conversion unit 410 of the light receiving unit 330, the pixel imaging signal generation unit 320, and the address event detection unit 400 are arranged, for example, in... Figure 2 In the light receiver chip 201 shown, other components may be arranged, for example, in a logic chip 202 that is bonded to the light receiver chip 201 via a Cu-Cu bonding. Therefore, in the following description, the structure arranged in the light receiver chip 201 in a unit pixel 310 is referred to as the "upper-layer circuit".

[0087] Figure 6BThe present disclosure describes an example of the construction of a group of unit pixels 310 of image sensing pixels 502 configured to have a shared pixel imaging signal generation readout circuit 320, according to at least some embodiments of the present disclosure. In this example, each photoelectric conversion element 333 is selectively connected to a floating diffusion layer 324 via a corresponding transmission gate 331. Furthermore, the photoelectric conversion elements 333 share components of the pixel imaging signal readout circuit 320. In this example, four photoelectric conversion elements 333a-333d and four corresponding transmission gates 331a-331d are shown. However, the shared pixel imaging signal readout circuit 320 may include any number of photoelectric conversion elements 333 and corresponding transmission gates 331.

[0088] Figure 6C The figure depicts a construction example of a unit pixel 310 configured as a single-function address event detection pixel 503 and an associated address event detection readout circuit 400 element. As shown, this example includes a single photoelectric conversion element 333, which is selectively connected to components of the address event detection readout circuit 400 via a transfer gate 332. An event scan control block 415 controls the operation of the address event detection readout circuit 400.

[0089] Figure 7 This is a block diagram illustrating a schematic construction example of an address event detection unit 400 according to at least some embodiments of the present disclosure. Figure 7 As shown, the address event detection unit 400 includes a current-to-voltage conversion unit 410, a buffer 420, a subtractor 430, a quantizer 440, and a transmission unit 450. The current-to-voltage conversion unit 410 converts the photocurrent from the optical receiving unit 330 into a voltage signal and supplies the converted voltage signal to the buffer 420. The buffer 420 corrects the voltage signal transmitted from the current-to-voltage conversion unit 410 and outputs the corrected voltage signal to the subtractor 430. The subtractor 430 reduces the voltage level of the voltage signal transmitted from the buffer 420 according to the row drive signal transmitted from the driving circuit 211 and supplies the reduced voltage signal to the quantizer 440. The quantizer 440 quantizes the voltage signal transmitted from the subtractor 430 into a digital signal and outputs the quantized digital signal as a detection signal to the transmission unit 450. The transmission unit 450 transmits the detection signal from the quantizer 440 to the signal processing unit 212, etc. For example, when an address event burst is detected, the transmission unit 450 supplies the arbitrator 213 with a request to transmit an address event detection signal from the transmission unit 450 to the drive circuit 211 and the signal processing unit 212. Additionally, when a response to this request is received from the arbitrator 213, the transmission unit 450 supplies the detection signal to the drive circuit 211 and the signal processing unit 212.

[0090] Figure 7 The current-to-voltage conversion unit 410 in the illustrated configuration may include, for example: Figure 6A The diagram shows two LG transistors 411 and 414, two amplifying transistors 412 and 413, and a constant current circuit 415. For example, the source of LG transistor 411 and the gate of amplifying transistor 413 are connected to the drain of the second transmission transistor 332 of the light receiving unit 330. Additionally, for example, the drain of LG transistor 411 is connected to the source of LG transistor 414 and the gate of amplifying transistor 412. For example, the drain of LG transistor 414 is connected to the power supply terminal VDD. Additionally, for example, the source of amplifying transistor 413 is grounded, and its drain is connected to the gate of LG transistor 411 and the source of amplifying transistor 412. For example, the drain of amplifying transistor 412 is connected to the power supply terminal VDD via the constant current circuit 415. For example, the constant current circuit 415 is composed of a load MOS transistor such as a P-type MOS transistor. In this connection relationship, a ring source follower circuit is formed. Through this arrangement, the photocurrent from the light receiving unit 330 is converted into a logarithmic voltage signal corresponding to its charge amount. Furthermore, for example, LG transistors 411 and 414, as well as amplifying transistors 412 and 413, can be made of NMOS transistors, respectively.

[0091] Figure 8 This is a circuit diagram illustrating a schematic construction example of a subtractor 430 and a quantizer 440 according to at least some embodiments of the present disclosure. Figure 8 As shown, subtractor 430 includes capacitors 431 and 433, inverter 432, and switch 434. Additionally, quantizer 440 includes comparator 441. One end of capacitor 431 is connected to the output terminal of buffer 420, and the other end is connected to the input terminal of inverter 432. Capacitor 433 is connected in parallel to inverter 432. Switch 434 opens or closes the path connecting the two ends of capacitor 433 according to the row drive signal. Inverter 432 inverts the voltage signal input through capacitor 431. Inverter 432 outputs the inverted signal to the non-inverting input terminal (+) of comparator 441. When switch 434 is on, the voltage signal V... init The input is given to the buffer 420 side of capacitor 431. The opposite side becomes a virtual ground terminal. For convenience, the potential of the virtual ground terminal is set to zero. At this time, when the capacitance of capacitor 431 is set to C1, the potential Q accumulated in capacitor 431... init It is represented by the following expression (1). On the other hand, the two ends of capacitor 433 are short-circuited, so its accumulated charge is zero.

[0092] Q init =C1×V init (1)

[0093] Next, considering that switch 434 is open and the voltage change of capacitor 431 on the buffer 420 side reaches V... after At that time, the charge Q accumulated in capacitor 431 after It is represented by the following expression (2).

[0094] Q after =C1×V after (2)

[0095] On the other hand, when the output voltage is set to V out At that time, the charge Q2 accumulated in capacitor 433 is represented by the following expression (3).

[0096] Q2=-C2×V out (3)

[0097] At this time, the total charge of capacitors 431 and 433 remains unchanged, so the following expression (4) holds true.

[0098] Q init =Q after +Q2 (4)

[0099] When expressions (1) to (3) are substituted into expression (4), the following expression (5) is obtained.

[0100] V out =-(C1 / C2)×(V) after -V init (5)

[0101] Expression (5) represents the subtraction operation of the voltage signal, and the gain of the subtraction result is C1 / C2. Generally, it is desirable to maximize (or increase) the gain; therefore, it is preferable to design the signal so that C1 is large and C2 is small. On the other hand, when C2 is too small, kTC noise increases, thus raising concerns about noise characteristic degradation. Therefore, the reduction in the capacity of C2 is limited to a range that allows for noise reduction. Furthermore, since an address event detection unit 400 including a subtractor 430 is installed for each unit pixel 310, there are area limitations on the capacities C1 and C2. The values ​​of capacities C1 and C2 are determined taking these limitations into account.

[0102] Comparator 441 compares the voltage signal transmitted from subtractor 430 with a predetermined threshold voltage Vth applied to the inverting input terminal (-). Comparator 441 outputs a signal indicating the comparison result as a detection signal to transmission unit 450. Additionally, when the conversion gain of current-to-voltage conversion unit 410 is set to CG... logWhen the gain of buffer 420 is set to "1", the overall gain A of address event detection unit 400 is represented by the following expression (6).

[0103] [Mathematical Expression 1]

[0104]

[0105] In expression (6), i photo _n represents the photocurrent of the nth unit pixel 310, and its unit is, for example, ampere (A). N represents the number of unit pixels 310 in the pixel block, and is "1" in this embodiment.

[0106] Figure 9 This is a block diagram illustrating a schematic construction example of a column ADC according to at least some embodiments of the present disclosure. The column ADC 220 includes a plurality of ADCs 230 configured for each column of unit pixels 310. Each ADC 230 converts an analog pixel signal appearing in the vertical signal line VSL into a digital signal. For example, the pixel signal is converted into a digital signal with a bit length greater than the bit length of the detection signal. For example, when the detection signal is set to two bits, the pixel signal is converted into a digital signal of three bits or more (16 bits, etc.). The ADC 230 supplies the generated digital signal to the signal processing unit 212.

[0107] Next, the operation of the image sensor 200 according to at least one embodiment of the present disclosure will be described in detail with reference to the accompanying drawings.

[0108] First, an operational example of the image sensor 200 will be illustrated using a timing diagram. Figure 10A This is a timing diagram illustrating an example of the operation of an image sensor according to an embodiment of the present disclosure.

[0109] like Figure 10A As shown, at timing T0, when the processor system 130 issues an instruction to initiate address event detection, the drive circuit 211 raises the control signal TG2 applied to the gates of the second transmission transistors 332 of all light receiving units 330 in the pixel array unit 300 to a high level. With this arrangement, the second transmission transistors 332 of all light receiving units 330 enter a conducting state, and photocurrent based on the charge generated in the photoelectric conversion element 333 of each light receiving unit 330 is supplied from each light receiving unit 330 to each of the plurality of address event detection units 400.

[0110] Furthermore, during the period when the control signal TG2 is at a high level, all transmission signals TG1 applied to the gate of the first transmission transistor 331 in each optical receiving unit 330 remain at a low level. Therefore, during this period, the multiple transmission transistors 331 in all optical receiving units 330 are in the off state.

[0111] Next, it will be assumed that the address event detection unit 400, configured to perform event detection on any unit pixel 310, detects an address event burst during the period when the control signal TG2 is high. In this case, the address event detection unit 400 that detects the address event burst transmits a request to the arbitrator 213. The arbitrator 213 arbitrates the request and returns a response to the request to the address event detection unit 400 that issued the request.

[0112] For example, during the time period from timing T1 to timing T2, the address event detection unit 400, upon receiving a response, raises the detection signal input to the drive circuit 211 and the signal processing unit 212 to a high level. Furthermore, in this description, it is assumed that the detection signal is a one-bit signal.

[0113] In timing T1, the drive circuit 211, which inputs a high-level detection signal from the address event detection unit 400, subsequently lowers all control signals TG2 to a low level in timing T2. This arrangement stops the supply of photocurrent from all light receiving units 330 of the pixel array unit 300 to the address event detection unit 400.

[0114] According to an embodiment of this disclosure, when the processor system 130 determines that the pixel imaging signal generation circuit 320 should be enabled, at timing T2, the drive circuit 211 raises the selection signal SEL applied to the gate of the selection transistor 323 in the pixel imaging signal generation unit 320, in which the unit pixel 310 (hereinafter referred to as the "readout target unit pixel") where an address event burst is detected, to a high level, and raises the reset signal RST applied to the gate of the reset transistor 321 in the same pixel imaging signal generation unit 320 to a high level for a constant pulse period, thereby draining (initializing) the charge accumulated in the floating diffusion layer 324 of the pixel imaging signal generation unit 320. In this way, the voltage appearing in the vertical signal line VSL in the initialized state of the floating diffusion layer 324 is read out by the ADC 230 connected to the vertical signal line VSL in the column ADC 220 as a reset level pixel signal (hereinafter simply referred to as the "reset level"), and is converted into a digital signal.

[0115] Next, at timing T3 after the readout reset level, the drive circuit 211 applies a transmission signal TG1 of a constant pulse period to the gate of the first transmission transistor 331 of the light receiving unit 330 in the readout target pixel 310. Through this arrangement, the charge generated in the photoelectric conversion element 333 of the light receiving unit 330 is transferred to the floating diffusion layer 324 in the pixel imaging signal generation unit 320, and a voltage corresponding to the charge accumulated in the floating diffusion layer 324 appears in the vertical signal line VSL. In this way, the voltage appearing in the vertical signal line VSL is read out by the ADC 230 connected to the vertical signal line VSL in the column ADC 220 as a signal level pixel signal (hereinafter simply referred to as "signal level") of the light receiving unit 330 and converted into a digital value.

[0116] The signal processing unit 212 performs CDS processing, wherein the difference between the reset level and the signal level read out as described above is obtained as a net pixel signal corresponding to the amount of light received by the photoelectric conversion element 333.

[0117] Next, at timing T4, the drive circuit 211 lowers the selection signal SEL of the gate of the selection transistor 323 in the readout circuit 320, which generates the pixel imaging signal applied to the readout target unit pixel 310, to a low level, and raises the control signal TG2 of the gate of the second transmission transistor 332 applied to all light receiving units 330 in the pixel array unit 300 to a high level. This arrangement restarts the address event burst detection in all light receiving units 330 in the pixel array unit 300.

[0118] Figure 10B This is a timing diagram illustrating an operational example of an image sensor according to another embodiment of the present disclosure. At timing T0, when the processor system 130 issues an instruction to initiate address event detection, the drive circuit 211 raises the control signal TG2 applied to the gate of the transfer transistor 332 associated with the photoelectric conversion element 333 of the selectively activated address event detection unit 400. More specifically, some or all of the address event detection units 400 can be activated.

[0119] Additionally, the transmission signal TG1 applied to the gate of the first transmission transistor 331 remains at a low level. Therefore, the associated transmission transistor 331 is in the off state.

[0120] In this example, while the control signal TG2 is high and the associated transmission transistor 332 is on, the arbitrary address event detection unit 400 detects an address event burst at timing T1. In response to the event trigger, image frame capture begins. Image frame capture can be a full-frame image capture involving all image sensing pixels 502 included in the pixel array unit 300. Alternatively, event detection by a specific event detection unit 400 can act as a trigger to capture images from a group of image sensing pixels 502 near or otherwise associated with the event detection unit 400. The readout of signals obtained from the image sensing pixels can then be performed. Furthermore, as discussed elsewhere herein, the processor system 130 can operate to control the frame rate of the enabled image sensing pixels 502 or circuitry 320.

[0121] Figure 11 The illustration shows aspects of the operation of an imaging apparatus 100 according to at least some embodiments of the present disclosure. Initially, the imaging apparatus 100 can monitor a scene in EBS mode (step 1100). In at least some operating scenarios, monitoring the scene in EBS mode includes one or more pixels outputting EBS data to a processor that communicates with the imaging apparatus.

[0122] When the imaging device 100 monitors a scene, the EBS data output by the pixels can be analyzed by a processor (step 1104). The processor can be configured to analyze the EBS data to detect changes in light intensity within the scene. Those skilled in the art will understand upon considering this disclosure that the shared event detection and image sensing pixel 501 or the address event detection pixel 503 can be operated to detect events in the form of changes in light intensity within the scene. Furthermore, according to at least some embodiments of this disclosure, the imaging device 100 can be operated to continuously detect events.

[0123] The detection operation can be performed by the driving circuit 211 and / or by application programming implemented by the processor system 130. Those skilled in the art will understand upon considering this disclosure that events are typically indicated by signals output from one or more event detection pixels 501, 503 within the pixel array unit 300.

[0124] When analyzing EBS data, the processor can detect triggering events. Triggering events can be detected by the processor by identifying one or more patterns among multiple possible patterns in the EBS data, or by other information associated with the event. For example, triggering events can be detected by monitoring the event density in the EBS data and determining whether the event density exceeds or falls below a threshold across the entire scene or in a predefined area of ​​the scene. In some implementations, EBS data can be used as input to a neural network that can output a determination of whether a triggering event has occurred. For example, the neural network can be trained to identify or detect desired object category groups or meaningful event groups in the input EBS data. In some implementations, EBS data can be analyzed by a processor capable of identifying triggering events based on the direction of motion in the EBS data.

[0125] If a trigger event is detected in step 1108, the processor can generate a signal in step 1112 to switch the sensor to RGB mode. If no trigger event is detected in step 1108, the method can return to step 1104, where the EBS data is analyzed.

[0126] After a trigger event is detected in step 1108, the RGB mode can be activated in step 1112. In some embodiments, after a trigger event has been detected, a determination can be made regarding the parameters that should be applied when collecting image data in RGB mode. For example, the imaging device 100 can be operated to collect image data at a specific frame rate.

[0127] Then, it can be determined whether to terminate the image sensing operation (step 1116). According to embodiments of this disclosure, image information acquisition can continue for a predetermined time period or until a predetermined number of image data frames have been acquired. Therefore, image information acquisition can be terminated after an initial image or image group has been acquired. According to other embodiments of this disclosure, image information acquisition can continue as long as the detected object remains within the field of view 114 of the imaging device 100. After determining that the object has left the field of view of the imaging device 100, the acquisition of image information associated with the object can be terminated. Alternatively, the acquisition of image information associated with the object can continue until sufficient image information is acquired to allow the application programming implemented by the processor system 130 of the imaging device 100 of the associated system to perform object recognition and determine that the image acquisition operation associated with the object can be terminated.

[0128] After determining that the image sensing operation can be stopped, the next step is to determine whether the operation of the imaging device 100 should be stopped (step 1120). If the operation continues, the process may include switching from RGB mode back to EBS mode in step 1124 before returning to step 1104. Otherwise, the operation may be terminated in step 1128.

[0129] Figure 12 A block diagram illustrating a system for switching between EBS pixel signals and RGB pixel signals is provided. (See above regarding...) Figures 5A-5D The imaging device 100 can be implemented in various configurations, as discussed. For example, such as Figure 5A As shown, the image sensor 200 may have a first or EBS sensor 530 and a second or imaging sensor 540. For example... Figure 5B As shown, the image sensor 200 may have a pixel 310, which is configured as a combination or shared event detection and image sensing pixel 501 that can selectively operate in either an event detection mode or an image sensing mode. Figure 5C As shown, the image sensor 200 may have an array of unit pixels 310 including a plurality of event detection pixels 503 and a plurality of image sensing pixels 502. Regardless of the type of image sensor 200 used, it can utilize, as shown... Figure 12 The switching system shown enables switching between event detection or EBS mode and image sensing or RGB mode.

[0130] As in Figure 12 As can be understood, EBS pixel data 1200 and RGB pixel data 1204 can be derived from the above information. Figures 5A-5F The image sensor 200 outputs EBS pixel data 1200 and RGB pixel data 1204. Depending on the implementation, EBS pixel data 1200 and RGB pixel data 1204 can be output simultaneously or separately. The EBS pixel data can be input to an EBS event analysis system 1212, such as a processor communicating with the image sensor 200. The EBS event analysis system 1212 can implement neural networks or some other type of analysis algorithm. The EBS event analysis system 1212 can control a switch 1208. For example, switch 1208 can be a transistor. Switch 1208 can control the data flow from the EBS pixels and RGB pixels to the output circuit 1216. In this way, the EBS event analysis system 1212 can analyze the data from the EBS pixel data 1200 and can control whether to output EBS pixel data 1200 or RGB pixel data 1204 from the imaging device 100 based on the analysis of the EBS pixel data 1200.

[0131] The switching logic can be used to switch the sensor from EBS data mode to RGB data mode and vice versa. In some implementations, the switching logic can be used to turn RGB data mode on and off only. In some implementations, EBS data can be analyzed by a computer system capable of controlling the switch to toggle the EBS / RGB switchable sensor between EBS and RGB modes. Analysis can be performed using neural networks or other data analysis methods. Depending on the decision logic, the output circuit can output either EBS data or RGB data from the sensor.

[0132] For example, the processor can be configured to process outputs from sensors operating in EBS mode and / or sensors operating in RGB mode. The processor can be configured to output event signals based on EBS data and / or image signals based on RGB data. The processor can also be configured to select between EBS mode and RGB mode based on the processing of EBS data and / or RGB data.

[0133] Whether the event needs to be switched from EBS to RGB depends on the application. Many switching methods that support low-power designs can be used, and some implementations can be as described in this article.

[0134] For example, depending on the application, one or more of the following methods may be used to determine when and whether to switch from EBS mode to RGB mode: detection of high EBS event density, detection of low EBS event density, EBS data analysis via neural networks, EBS data analysis via recurrent neural networks, and detection of EBS motion in a specific direction. It should be noted that these methods should not be considered the only possible approaches to determining when and whether to switch from EBS mode to RGB mode.

[0135] Data collected via EBS mode can also be used to determine the speed of objects and can be used to switch to a higher frame rate.

[0136] In one implementation, the sensor can switch from EBS mode to RGB mode when the EBS event density exceeds a threshold amount across the entire scene or a predefined area of ​​the scene. This implementation can be useful for capturing motion. For example, a sensor configured to switch from EBS mode to RGB mode based on EBS event density exceeding a threshold amount can be used to identify vehicles entering a scene or people entering a room.

[0137] In some implementations, the processor system 130 is capable of using event detection data to determine the frame rate applied to the RGB mode. The determined frame rate for the RGB mode can be based on factors such as object identification, the relative speed of the object, or the degree of interest in the identified object, as determined from the event detection data. For example, a relatively high frame rate can be applied to a car, a medium frame rate to a cyclist, and a relatively low frame rate to a pedestrian. A higher frame rate can be applied to objects moving at a faster apparent speed compared to stationary objects or objects moving at a lower apparent speed.

[0138] Various operations performed by the processor system 130 on the event detection data and / or image data may include applying one or more neural networks to analyze the collected information.

[0139] The embodiments disclosed herein allow for the continued operation of event detection pixels 501 and 503 while image sensing pixels 501 and 502 are operating. As described elsewhere herein, event detection pixels 501 and 503 typically operate asynchronously. By continuing to operate event detection pixels 501 and 503, event detection functions can be performed continuously without loss or reduction in the temporal event detection performance of the imaging device 100.

[0140] Therefore, embodiments of this disclosure provide an imaging apparatus 100 having one or more pixel array units 300 capable of performing both event detection and imaging operations. Furthermore, the event detection pixels can operate continuously, while the image sensing pixels can operate selectively. Additionally, the frame rate applied to the operation of the image sensing pixels can be selected based on the characteristics or identifiers of one or more detected events. After a selected time period, after the imaged event no longer exists, or after certain other criteria have been met, the operation of the image sensing pixels can be interrupted, while the operation of the event detection pixels continues. Thus, continuous monitoring of events is provided in conjunction with the selected imaging operation, thereby providing relevant image data while saving power, data transmission, and data processing resources.

[0141] EBS sensors, or sensors including EBS pixels, can generate data frames indicating changes in light intensity. For example, in a frame, a positive change in light intensity can be reflected by pixels with values ​​such as +1 or specific colors such as red. Similarly, in a frame, a negative change in light intensity can be reflected by pixels with specific values ​​such as -1 or other colors such as blue. If the EBS pixel does not detect a change in light intensity, a zero value or a color such as white can be used.

[0142] In some implementations, the EBS sensor, or a sensor including EBS pixels, can indicate the amount of change in light intensity. For example, a relatively high change in light intensity can be reflected by a pixel with a value of +1.00, while a relatively low but positive change in light intensity can be reflected by a value of +0.01. The values ​​+1.00 and +0.01 can be represented by 8-bit numerical values ​​of 255 and 1, respectively. Similarly, a range of colors can be used to indicate the amount of change.

[0143] However, EBS cameras only provide change and time information. For example, data from the EBS sensor corresponding to a pixel event may correspond to three states: -1 for negative change, +1 for positive change, and 0 for no change. Information about the timing of the change can also be provided. EBS cameras themselves do not directly provide color or grayscale information. Therefore, EBS cameras are not general-purpose cameras used for capturing image or video information. The EBS pixels mentioned above that are associated with color pixels should not be interpreted as EBS pixels associated with colors from the scene, but rather as simply using color as a visualization of changes in light intensity.

[0144] Data frames generated by an EBS sensor or a sensor including EBS pixels can be like... Figures 13A-13C As shown in charts 1300, 1304, and 1308. Figures 13A-13C In charts 1300, 1304, and 1308, pixels with non-zero values ​​are represented by simple dots. Darker dots represent pixels with positive polarity data, and lighter dots represent pixels with negative polarity data. The darker and lighter dots are chosen for visualization purposes, and it should be understood that pixel values ​​generated by EBS pixels can be represented by any type of indicator. Furthermore, each pixel can be associated with a value that can be stored in a register or other type of memory location.

[0145] When operating in EBS mode, the EBS / RGB switchable sensor can operate with relatively low power consumption. When operating in RGB mode, the EBS / RGB switchable sensor can operate with relatively high power consumption. Therefore, EBS mode can be used for lower power consumption and can be activated or switched to RGB mode only when needed.

[0146] The switching logic can be used to switch the sensor from EBS data mode to RGB data mode and vice versa. In some implementations, the switching logic can be used to turn RGB data on and off only. In some implementations, the EBS data can be analyzed by a computer system capable of controlling the switch to toggle the EBS / RGB switchable sensor between EBS and RGB modes. Analysis can be performed using neural networks or other data analysis methods. Depending on the decision logic, the output circuit can output either EBS data or RGB data from the sensor.

[0147] For example, the processor can be configured to process outputs from sensors operating in EBS mode and / or sensors operating in RGB mode. The processor can be configured to output event signals based on EBS data and / or image signals based on RGB data. The processor can also be configured to select between EBS mode and RGB mode based on the processing of EBS data and / or RGB data.

[0148] Whether the event needs to be switched from EBS to RGB depends on the application. Many switching methods that support low-power designs can be used, and some implementations can be as described in this article.

[0149] For example, depending on the application, one or more of the following methods may be used to determine when and whether to switch from EBS mode to RGB mode: detection of high EBS event density, detection of low EBS event density, EBS data analysis via neural networks, EBS data analysis via recurrent neural networks, and detection of EBS motion in a specific direction. It should be noted that these methods should not be considered the only possible approaches to determining when and whether to switch from EBS mode to RGB mode.

[0150] In some implementations, the sensor can switch from EBS mode to RGB mode when the EBS event density exceeds a threshold amount across the entire scene or a predefined area of ​​the scene. This implementation can be useful for capturing moving objects. For example, a sensor configured to switch from EBS mode to RGB mode based on EBS event density exceeding a threshold amount can be used to identify vehicles entering a scene or people entering a room.

[0151] Switching from EBS mode to RGB mode based on EBS event density exceeding a threshold can include switching to RGB mode when the number of events exceeds a predefined spatiotemporal density. RGB mode may be triggered when a high event density is detected in EBS mode. RGB mode may not be triggered when sparse events are detected in EBS mode. EBS frames can be analyzed within a set integration time (e.g., 1 millisecond).

[0152] For example, such as Figure 13A As shown in EBS frame 1300, when a relatively large number of EBS pixels within a specific region detect changes in light intensity over a short period of time, the number of events may exceed a predefined spatiotemporal density. This scenario can be illustrated as follows: Figure 13A EBS frame 1300 is shown in the image, where a cluster of pixels recording data appears in frame 1300.

[0153] Or or otherwise, such as Figure 13B As shown in EBS frame 1304, when a relatively large number of EBS pixels detect changes in light intensity throughout the frame, the number of events may exceed the predefined spatiotemporal density. This scenario can be illustrated as follows: Figure 13B EBS frame 1304 is shown in the image, in which many pixels with recorded data appear throughout frame 1304.

[0154] In another implementation, the sensor can switch from EBS mode to RGB mode when the EBS event density drops below a threshold amount. This implementation can be useful for capturing static scenes when motion has stopped or is not required. For example, a sensor configured to switch from EBS mode to RGB mode based on the EBS event density dropping below a threshold amount can be used to identify machines or production lines that have stopped operating, thereby allowing for troubleshooting or other appropriate actions.

[0155] This scenario can be like... Figure 13C EBS frame 1308 in the image is shown. When a relatively small number of EBS pixels detect changes in light intensity throughout the entire frame, the number of events may drop below a threshold. This scenario can be illustrated as follows: Figure 13C EBS frame 1308 is shown in the image, where pixel clusters appear.

[0156] Switching from EBS mode to RGB mode based on EBS event density falling below a threshold can include switching to RGB mode when the number of events drops below a predefined spatiotemporal density. RGB mode may be triggered when low event density is detected in EBS mode, and may not be triggered when high event density is detected in EBS mode. EBS frames can be analyzed within a set integration time (e.g., 1 millisecond).

[0157] In some implementations, switching between EBS and RGB modes can be triggered based on the processing of EBS frames by a convolutional neural network (“CNN”). In such implementations, EBS frames can be fed into CNNs such as LeNet, VGG16, and YOLO. If the probability of identifying or otherwise detecting a specific object, such as a person or vehicle, is high, RGB mode may be triggered to capture a color image of the object for further analysis.

[0158] If the neural network determines that the probability of a certain type of object, such as a person or a car, exceeds a predefined threshold, it may trigger the RGB mode.

[0159] For example, such as Figure 14A As shown, one or more EBS frames can be used as input 1400 to CNN 1404, which can output a trigger decision 1408. In some embodiments, a single EBS frame can be used as input to generate a trigger decision. A single EBS frame can be a collection of EBS signals collected within a specific time frame, such as 1 millisecond. In some embodiments, multiple EBS frames can be used as input. For example, a series of EBS frames captured within a given time period, such as 1 second, can be used.

[0160] CNN 1404 may include multiple layers and can be trained to detect one or more types of EBS-related events. For example, CNN 1404 may include multiple convolutional layers (e.g., conv1, conv2, conv3, conv4, conv5, etc.) and one or more max pooling layers. CNN 1404 can be trained by inputting EBS frames that display known events. In some implementations, CNN 1404 can be trained to output a trigger decision 1408 upon detecting an event in EBS data that displays a specific event occurring. The trigger decision 1408 can be as simple as +1 for yes and 0 for no. In some implementations, the trigger decision 1408 may be more complex, for example, identifying the event type of the detected event. For example, CNN 1404 may detect input with EBS data that shows a large number of events exceeding a predefined spatiotemporal density or that is identified by CNN 1404 as indicating the presence of a specific object such as a person or vehicle. The trigger decision 1408 may include information about the objects detected and / or identified by CNN 1404.

[0161] This implementation scheme can be useful in many applications, such as in factory automation to detect one or more faulty objects on a production line, intrusion detection by detecting moving people in a restricted area, traffic safety by detecting cars in a bicycle line, and so on.

[0162] In some implementations, the switching between EBS mode and RGB mode can be triggered based on the processing of EBS frames by a recurrent neural network (“RNN”). In this implementation, EBS frames can be fed into the RNN, and RGB mode may be triggered if the probability of detecting a meaningful sequence of events is high. The EBS frames can be fed into the RNN along with timestamp data and / or polarity data. The RNN is able to output a triggering decision that can be used to determine when and whether the sensor should switch from EBS mode to RGB mode.

[0163] For example, such as Figure 14B As shown, one or more EBS frames can be used as input 1412 to RNN 1416, which can output a trigger decision 1420. In some embodiments, a single EBS frame can be used as input to generate the trigger decision. A single EBS frame can be a collection of EBS signals collected within a specific time frame, such as 1 millisecond. In some embodiments, multiple EBS frames can be used as input. For example, a series of EBS frames captured within a given time period, such as 1 second, can be used.

[0164] An RNN can be described as a neural network in which connections between nodes form a directed graph along a time series. In some implementations, an RNN can be of the finite-spur or infinite-spur type. Both finite-spur and infinite-spur recurrent networks can have additional storage states, and the storage can be directly controlled by the RNN. This controlled state can be called a gated state or a gated memory, and can be part of one or more Long Short-Term Memory (LSTM) networks and gated recurrent units. This is also known as a feedback neural network.

[0165] The RNN 1416 can include multiple layers and can be trained to detect one or more types of EBS-related events. For example, the RNN 1416 can include an input layer, one or more hidden layers, and an output layer. The RNN 1416 can be trained by inputting EBS frames that display known events.

[0166] The input layer of RNN 1416 may include data associated with the input EBS data. In some implementations, the EBS data frame may be input along with other information such as timestamps and polarity. The hidden layers of RNN 1416 may be one or more intermediate layers computed by RNN 1416. The output layer of RNN 1416 may be data from which the trigger decision 1420 is directly derived. RNN 1416 may include one or more context nodes that can be connected to nodes in one or more hidden layers.

[0167] In some implementations, the RNN 1416 can be trained to output a trigger decision 1420 upon detecting an event in EBS data that indicates a specific event has occurred. The trigger decision 1420 can be as simple as +1 for yes and 0 for no. In some implementations, the trigger decision 1420 may be more complex, for example, identifying the event type of the detected event. For example, the RNN 1416 can detect inputs with EBS data that shows a large number of events exceeding a predefined spatiotemporal density or that the RNN 1416 identifies as indicating the presence of a specific object such as a person or vehicle. The trigger decision 1420 may include information about the object detected and / or identified by the RNN 1416.

[0168] This implementation scheme can be useful in many applications, such as tracking moving objects like cars or airplanes, or tracking sports events such as detecting when a ball hits a racket or when a moving ball approaches a goal. When such an event is detected, the sensor can switch from EBS mode to RGB mode.

[0169] In some implementations, the switch from EBS to RGB can be triggered based on the motion direction detected in the EBS data. For example, predefined objects identified by convolutional neural networks or other object detection methods can be monitored to detect the motion direction of the objects. Depending on the motion direction of the detected objects, the sensor can switch to RGB mode or high frames per second (“FPS”) RGB mode.

[0170] Object movement can be detected in both RGB and EBS frames. For example, Figure 15A and Figure 15B As shown, a single frame can indicate motion in a specific direction. Figure 15A An RGB image of ball 1500 is shown. The analysis method described herein is capable of examining one or more RGB frames and determining the direction and / or velocity of movement of ball 1500 or any other type of detected object. Similarly, as Figure 15B As shown, one or more frames of EBS data 1504 can be used to identify the orientation and / or velocity of a detected object. Since positive EBS data indicates a positive change in light intensity, and negative EBS data indicates a negative change in light intensity, any system and method for analyzing EBS data as described herein can examine EBS frames and identify the orientation and / or velocity of an object.

[0171] This implementation scheme could be useful in many applications. For example, a camera could be fixed to observe a soccer goal. A high FPS mode could be triggered whenever a soccer ball is detected moving towards the goal from the nearby area. This implementation scheme could also be used for collision warnings. For example, when a first object is detected approaching a second object within a certain distance, the camera could switch to high FPS mode. Such applications may require additional data collection methods, such as distance measurements based on stereo vision, time-of-flight, structured light, etc.

[0172] The sensor's switching from EBS mode to RGB mode can last for a predetermined amount of time. For example, after switching to RGB mode, the sensor may switch back to EBS mode after several seconds. In some implementations, the RGB mode can be analyzed to determine when the event ends, at which point the sensor can switch back to EBS mode.

[0173] The embodiments of this disclosure have been described above; however, the technical scope of this disclosure is not limited to the above embodiments, and various modifications can be made without departing from the spirit of this disclosure. Furthermore, the constituent elements of other embodiments and modifications can be appropriately combined.

[0174] This paper discloses a combined EBS and RGB camera that can utilize the advantages of both EBS and RGB modes. As described herein, the sensor typically operates in EBS mode and switches to RGB mode when an event requires it. As used herein, RGB can refer to data related to a quantity of light intensity. An RGB sensor, or a sensor operating in RGB mode, can be or is configured to detect a quantity of light intensity.

[0175] As described herein, EBS / RGB switchable sensors can take various forms. For example, in some implementations, separate EBS and RGB sensors can be used. In such implementations, the separate EBS and RGB sensors can each comprise multiple pixels. The separate EBS and RGB sensors can be physically connected and can share a single lens.

[0176] In some implementations, a single sensor with a mosaic of RGB pixels and EBS pixels can be used. For example, a single sensor may include a pixel grid. The pixel grid can be a variety of non-switchable RGB pixels and EBS pixels. The pixel pattern can be randomly arranged or can be a specific pattern. In some implementations, EBS pixels can be in a small portion of the pixel grid or can be uniformly distributed throughout the grid.

[0177] In some implementations, a single sensor with switchable RGB and EBS pixels can be used. For example, the sensor may include a pixel grid. Each pixel is capable of detecting both EBS and color. For example, a first pixel may switch between collecting EBS data and red data, a second pixel may switch between collecting EBS data and green data, and a third pixel may switch between collecting EBS data and blue data.

[0178] Additional implementations may include other combinations of switchable and non-switchable pixel and / or other color mosaic patterns.

[0179] As described in this article, the EBS / RGB switchable sensor can be used in either EBS or RGB mode, or both. The advantage of the EBS sensor is its ability to capture event data at a high rate. The EBS sensor also consumes relatively less power than the RGB sensor.

[0180] Furthermore, the effects described in the embodiments in this specification are merely illustrative, and other effects may exist without limitation.

[0181] In addition, this technology may include the following constructions: (1)

[0183] A system comprising:

[0184] The first sensor is configured to detect changes in light intensity.

[0185] The second sensor is configured to detect the amount of light intensity; and

[0186] The processor is constructed as follows:

[0187] In the first mode, the output from the first sensor is processed and an event signal is output.

[0188] In the second mode, the output from the second sensor is processed and an image signal is output, and

[0189] The first mode and the second mode are selected based on at least one of the processing results in the first mode and the processing results in the second mode. (2)

[0191] The system as described in (1) includes processing the output from the first sensor by analyzing the data output from the first sensor using a neural network. (3)

[0193] The system as described in (2) wherein the neural network is trained to detect objects. (4)

[0195] The system as described in (3) wherein the second mode is selected when the neural network detects the object in the data output from the first sensor. (5)

[0197] The system as described in (4) wherein the first sensor is an EBS sensor and the second sensor is an RGB sensor. (6)

[0199] The system as described in (1) is wherein the second mode is selected based on the detection that the event density in the data output from the first sensor in the first mode exceeds a threshold. (7)

[0201] The system as described in (1) is wherein the second mode is selected based on the detection that the event density in the data output from the first sensor in the first mode drops below a threshold. (8)

[0203] The system as described in (1) wherein the frame rate is adjusted based on the processing result in the first mode. (9)

[0205] An imaging system comprising:

[0206] The first sensor is configured to detect changes in light intensity; and

[0207] The second sensor is configured to detect a quantity of light intensity, wherein,

[0208] In the first mode, the processor processes the output from the first sensor.

[0209] In the second mode, the processor processes the output from the second sensor, and

[0210] Based on at least one of the processing results in the first mode and the processing results in the second mode, the processor selects at least one of the first mode and the second mode. (10)

[0212] The imaging system as described in (9) includes processing the output from the first sensor by analyzing the data output from the first sensor using a neural network. (11)

[0214] The imaging system as described in (10) wherein the neural network is trained to detect objects. (12)

[0216] The imaging system as described in (11) wherein the second mode is selected when the neural network detects the object in the data output from the first sensor. (13)

[0218] The imaging system as described in (12) wherein the first sensor is an EBS sensor and the second sensor is an RGB sensor. (14)

[0220] The imaging system as described in (9) is wherein the second mode is selected based on the detection that the event density in the data output from the first sensor in the first mode exceeds a threshold. (15)

[0222] The imaging system as described in (9) is wherein the second mode is selected based on the detection that the event density in the data output from the first sensor in the first mode drops below a threshold. (16)

[0224] The imaging system as described in (9) wherein the frame rate is adjusted based on the processing result in the first mode. (17)

[0226] An image sensor comprising multiple pixels, wherein,

[0227] One or more first pixels among the pixels are capable of detecting changes in light intensity.

[0228] One or more second pixels among the pixels are capable of detecting the amount of light intensity.

[0229] In the first mode, the sensor outputs data from one or more first pixels among the pixels.

[0230] In the second mode, the sensor outputs data from one or more second pixels among the pixels, and

[0231] Based on at least one of the processing results of the data output in the first mode and the processing results of the data output in the second mode, the processor selects at least one of the first mode and the second mode. (18)

[0233] The image sensor as described in (17) wherein at least one of the plurality of pixels exists in the form of one or more first pixels and one or more second pixels. (19)

[0235] The image sensor as described in (17) wherein one or more of the plurality of pixels are switchable EBS / RGB pixels. (20)

[0237] The image sensor as described in (17) wherein each of the one or more first pixels in the pixel is different from each of the one or more second pixels in the pixel.

Claims

1. An imaging system comprising: The first sensor is configured to detect changes in light intensity. The second sensor is configured to detect the amount of light intensity; and The processor is constructed as follows: In the first mode, the output from the first sensor is processed and an event signal is output. In the second mode, the output from the second sensor is processed and an image signal is output, and The second mode is selected based on the detection that the event density of a specific event in the data output from the first sensor in the first mode exceeds one threshold or falls below another threshold. Wherein, the event density is the event density of a specific event in a predefined region of the scene, and After the second mode has been in effect for a predetermined amount of time, the system switches back to the first mode.

2. The imaging system as claimed in claim 1, wherein, Processing the output from the first sensor includes analyzing the data output from the first sensor using a neural network.

3. The imaging system as described in claim 2, wherein, The neural network is trained to detect objects.

4. The imaging system as claimed in claim 1, wherein, The first sensor is an EBS sensor, and the second sensor is an RGB sensor.

5. The imaging system as claimed in claim 1, wherein, The frame rate is adjusted based on the event signal output in the first mode.

6. The imaging system as claimed in any one of claims 1 to 5, wherein, The processor is configured to output the event signal along with time information in the first mode.

7. An imaging system comprising: The first sensor is configured to detect changes in light intensity. and The second sensor is configured to detect a quantity of light intensity, wherein, In the first mode, the processor processes the output from the first sensor. In the second mode, the processor processes the output from the second sensor, and Based on the detection that the event density of a specific event in the data output from the first sensor in the first mode exceeds a threshold or falls below another threshold, the processor selects the second mode, and The event density is the event density of a specific event in a predefined region of the scene, and After the second mode has been in effect for a predetermined amount of time, the system switches back to the first mode.

8. The imaging system of claim 7, wherein, Processing the output from the first sensor includes analyzing the data output from the first sensor using a neural network.

9. The imaging system of claim 8, wherein, The neural network is trained to detect at least one object.

10. The imaging system of claim 7, wherein, The first sensor is an EBS sensor, and the second sensor is an RGB sensor.

11. The imaging system according to any one of claims 7 to 10, wherein, The frame rate is adjusted based on the event signal output in the first mode.

12. An image sensor comprising multiple pixels, wherein, One or more first pixels among the pixels are capable of detecting changes in light intensity. One or more second pixels among the pixels are capable of detecting the amount of light intensity. In the first mode, the sensor outputs data from one or more first pixels among the pixels. In the second mode, the sensor outputs data from one or more second pixels among the pixels. Based on the detection that the event density of a specific event in the data output in the first mode exceeds a threshold or falls below another threshold, the processor selects the second mode, and The event density is the event density of a specific event in a predefined region of the scene, and After the second mode has been in effect for a predetermined amount of time, the system switches back to the first mode.

13. The image sensor of claim 12, wherein, At least one pixel exists in two forms: one or more first pixels and one or more second pixels.

Images