Visual sensor, image processing device, and method of operating a visual sensor

Abstract

A vision sensor includes: a pixel array comprising pixels arranged in a matrix; event detection circuitry; an event rate controller; and interface circuitry. Each pixel is configured to generate an electrical signal in response to detecting a change in the intensity of incident light. The event detection circuitry, based on processing the electrical signals received from one or more pixels, detects whether a change in the intensity of incident light has occurred at any pixel, and generates one or more event signals corresponding to the one or more pixels where the intensity of the incident light has been determined to have changed. The event rate controller selects one or more event signals corresponding to a region of interest on the pixel array as one or more output event signals. The interface circuitry communicates with an external processor and transmits one or more output event signals to the external processor.

Description

[0001] This application is a divisional application of Chinese invention patent application filed on June 10, 2020, with application number 202010526059.X.

[0002] Cross-reference to related applications

[0003] This application claims the benefit of Korean Patent Application No. 10-2019-0076347, filed on June 26, 2019, with the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0004] The present invention relates to a visual sensor, and more specifically, to a visual sensor configured to adjust the amount of transmitted event data, an image processing apparatus including the visual sensor, and a method of operating the visual sensor. Background Technology

[0005] When an event (e.g., a change in the intensity of light incident on a vision sensor) occurs, the vision sensor (e.g., a motion vision sensor) generates information about the event (e.g., associated with the event), i.e., an event signal, and provides the event signal to the processor. Since the change in light intensity (e.g., incident light) primarily occurs along the outline of the object being detected by the vision sensor, the vision sensor does not generate unnecessary information about the background, and the amount of data to be processed by the processor can be rapidly reduced. Furthermore, when a large number of events (e.g., quantity, count, etc.) occur over a long period and the amount of event data (e.g., event signals indicating the occurrence of said events) to be transmitted to the processor exceeds the system's transmission limits, data loss may occur during the transmission of event data. Summary of the Invention

[0006] Some exemplary embodiments of the present invention provide one or more visual sensors, one or more image processing devices including one or more visual sensors, and one or more operating methods of one or more visual sensors, wherein the one or more visual sensors are configured to reduce or prevent data loss during the transmission of event data through the one or more visual sensors.

[0007] According to some example embodiments, a vision sensor may include a pixel array comprising a plurality of pixels arranged in a matrix, each pixel being configured to generate a separate electrical signal in response to detecting a change in the intensity of incident light. The vision sensor may include event detection circuitry configured to detect whether a change in the intensity of incident light has occurred at any pixel in the plurality of pixels based on processing electrical signals received from one or more pixels of the plurality of pixels, and to generate one or more event signals corresponding to the one or more pixels of the plurality of pixels where the change in the intensity of incident light is determined to have occurred. The vision sensor may include an event rate controller configured to select one or more event signals from the one or more event signals that correspond to a region of interest on the pixel array, as one or more output event signals. The vision sensor may include interface circuitry configured to communicate with an external processor and transmit the one or more output event signals to the external processor.

[0008] According to some example embodiments, an image processing device may include a vision sensor configured to: capture an image of an object; generate a plurality of event signals corresponding to a plurality of pixels included in a pixel array, each event signal being generated based on movement of the object relative to the vision sensor; and, in response to the vision sensor operating in a selected operating mode of a first operating mode or a second operating mode, selectively transmit all or a limited selection of the plurality of event signals as one or more output event signals, the limited selection of event signals corresponding to one or more pixels included in a region of interest in the pixel array. The image processing device may further include a processor configured to process the one or more output event signals to detect movement of the object.

[0009] According to some example embodiments, a method of operating a vision sensor may include defining at least one region of interest on a pixel array comprising a plurality of pixels, each of the plurality of pixels being configured to generate an event signal in response to a change in the intensity of light received by that pixel. The method may include detecting the amount of event signals generated during a particular first time period by counting a plurality of event signals generated during that particular first time period. The method may include selectively controlling the vision sensor to operate in a selected operating mode based on determining that the amount of event signals generated during the particular first time period is equal to or greater than a threshold. The method may include, based on the vision sensor operating in the selected operating mode, transmitting a limited selection of event signals from a plurality of event signals to an external processor, the limited selection of event signals corresponding to at least one region of interest. Attached Figure Description

[0010] Example embodiments of the inventive concept will be more clearly understood from the following description taken in conjunction with the accompanying drawings, in which:

[0011] Figure 1 This is a block diagram illustrating an image processing apparatus according to an exemplary embodiment of the concept of the present invention;

[0012] Figure 2 It is a graph showing the situation where signal loss occurs during an event;

[0013] Figure 3 This is a block diagram illustrating some exemplary embodiments of a vision sensor according to a concept conceived in this invention;

[0014] Figure 4A and Figure 4B The formats of event data output from a visual sensor are shown in some example embodiments of the present invention.

[0015] Figure 5A and Figure 5B These are all figures used to describe a method for outputting an event signal corresponding to a region of interest from a visual sensor according to some exemplary embodiments of the present invention;

[0016] Figure 6 This is a block diagram illustrating an example embodiment of a vision sensor according to a concept of the present invention;

[0017] Figure 7 These are circuit diagrams illustrating some example embodiments of pixels;

[0018] Figure 8A , Figure 8B , Figure 8C and Figure 8D These are the instructions shown separately. Figure 7 A circuit diagram illustrating an example embodiment of the connection relationship between the photoelectric conversion element and the amplifier;

[0019] Figure 9 This is a flowchart illustrating an operation method of a vision sensor according to some exemplary embodiments of the present invention;

[0020] Figure 10 This is a flowchart illustrating an operation method of a vision sensor according to some exemplary embodiments of the present invention;

[0021] Figure 11 This is a diagram illustrating a method for determining the operating mode of a vision sensor according to some exemplary embodiments of the present invention;

[0022] Figure 12A and Figure 12B It is shown Figure 3 A block diagram of an embodiment of the event rate controller shown;

[0023] Figure 13 Examples of methods for setting the region of interest of a visual sensor according to some exemplary embodiments of the present invention are shown;

[0024] Figure 14 Methods for estimating the location of a region of interest in a visual sensor are illustrated according to some exemplary embodiments of the present invention.

[0025] Figure 15A , Figure 15B and Figure 15C A method for adjusting the rate of event data output from a vision sensor is shown according to some example embodiments;

[0026] Figure 16 This is a block diagram illustrating some example embodiments of an image processing apparatus according to a concept of the present invention;

[0027] Figure 17 This is a flowchart illustrating a method of operating a vision sensor according to some example embodiments;

[0028] Figure 18 This is a flowchart illustrating a method of operating a vision sensor according to some example embodiments; and

[0029] Figure 19 This is a block diagram illustrating examples of electronic devices employing vision sensors according to some exemplary embodiments of the present invention. Detailed Implementation

[0030] Figure 1 This is a block diagram illustrating some example embodiments of an image processing apparatus according to a concept of the present invention.

[0031] Image processing device 10, according to some exemplary embodiments of the present invention, can be installed (e.g., may be included) in electronic devices having (e.g., configured to perform) image sensing and / or light sensing functions. For example, image processing device 10 can be installed in electronic devices such as cameras, smartphones, Internet of Things (IoT) devices, tablet PCs, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation devices, drones, and advanced driver assistance systems (ADAS). Additionally, image processing device 10 can be provided as a component in vehicles, furniture, manufacturing equipment, doors, various measuring devices, etc.

[0032] refer to Figure 1 The image processing device 10 may include a vision sensor 100 and a processor 200.

[0033] A vision sensor 100 (which may be interchangeably referred to as an image sensor and / or a light sensor) can generate an electrical signal in response to detecting and / or absorbing at least a portion of light incident on at least a portion of the vision sensor 100. The vision sensor 100 may include an array of pixels (e.g., a pixel array), wherein each pixel in the array is configured to individually receive and sense (e.g., detect, absorb, etc.) incident light. To sense the incident light, the pixels may be configured to output (e.g., generate, transmit, etc.) an electrical signal in response to receiving, absorbing, etc., the incident light. In some example embodiments, the electrical signals output by the pixels of the pixel array of the vision sensor 100 may be processed to generate an image of an external scene, including images of objects within the scene, wherein the electrical signal output from each pixel of the pixel array of the vision sensor 100 is used to generate the corresponding pixel of the generated image. The processing for generating the image may be performed by the vision sensor 100, by a processor 200 based on the signals generated by the pixels of the vision sensor 100 being transmitted to the processor 200, or by any combination thereof. In this way, the scene and / or objects in the scene can be "imaged" by the vision sensor 100 and / or the image processing device 10.

[0034] In some example embodiments, the image processing device 10 and / or electronic equipment including the image processing device 10 may include a separate vision sensor 100 for generating one or more images of a scene and / or objects separately from the vision sensor 100 operating as a dynamic vision sensor. For example, Figure 1 The visual sensor 100 of the image processing device 10 may transmit only the event signal EVS to the processor 200, while the separate visual sensor 101 may include a separate pixel array and may generate and / or transmit signals that can be processed to generate images of external scenes and / or objects.

[0035] The vision sensor 100 can sense changes in the intensity of incident light and can output (e.g., generate, transmit, etc.) one or more event signals (EVS) in response to sensing said changes. The vision sensor 100 may include a dynamic sensor that outputs (e.g., generates, transmits, etc.) event signals EVS corresponding to pixels of the vision sensor 100 that detect changes in incident light thereon (i.e., pixels where an event occurs, e.g., pixels in the pixel array of the vision sensor 100 where the intensity of incident light on them has changed). Changes in the intensity of incident light at a pixel can be caused by movement of an object captured (e.g., imaged) by the pixel array of the vision sensor 100, movement of the vision sensor 100, and / or movement of the image processing device 10. Again, such changes can be caused by movement of the imaged object relative to the vision sensor 100. The vision sensor 100 can transmit the event signals EVS to the processor 200 periodically or non-periodically. The vision sensor 100 can transmit the event signals EVS to the processor 200 in group units or frame units. For example, the pixel array of the vision sensor 100 can capture an image of an object (e.g., image the object) based on pixels PX of the pixel array, wherein pixels PX generate one or more electrical signals in response to receiving and / or absorbing incident light at pixel PX. When the imaged object moves relative to the vision sensor 100, the intensity of the incident light at one or more pixels PX may change due to the movement, and the one or more pixels PX may generate one or more electrical signals in response to the change, which may cause the vision sensor 100 to generate one or more event signals EVS corresponding to the pixel PX whose incident light intensity has changed. Therefore, the vision sensor 100 can generate multiple event signals corresponding to pixels PX whose incident light intensity has changed due to the movement of the imaged object, and thus the event signals EVS can be generated based on the movement of the imaged object relative to the vision sensor 100.

[0036] The vision sensor 100 can selectively transmit the event signal EVS to the processor 200. In conjunction with the pixel array of the vision sensor 100 (… Figure 3In the event signals EVS corresponding to the region of interest (PX) set on the pixel array 110, the vision sensor 100 can transmit to the processor 200 the event signals EVS generated according to a limited selection of pixels PX corresponding to the region of interest (PX) set on the pixel array 110. Therefore, the vision sensor 100 can transmit a limited selection of event signals EVS to the processor 200. To reiterate, the vision sensor 100 can transmit any or all of the generated event signals EVS to the processor 200 when operating in certain operating modes, and when operating in one or more other separate operating modes, it can transmit only some (or one) of the generated event signals EVS to the processor 200, while avoiding transmission (e.g., excluding from transmission) of the remaining one or more event signals EVS.

[0037] In some example embodiments, when an excessive number of events (e.g., quantity, count, etc.) occur (e.g., within a specific time period), or when an excessive number of events are expected to occur (e.g., within a specific time period), the vision sensor 100 may selectively transmit event signals EVS to the processor 200 (e.g., only some of the generated event signals EVS are transmitted to the processor 200, excluding the transmission of others). Additionally, the vision sensor 100 may adjust event occurrence conditions and / or detection conditions (e.g., thresholds) to reduce the amount of event signals EVS generated and / or transmitted at any given point in time (e.g., quantity, count, etc.). In some example embodiments, the vision sensor 100 may set event occurrence conditions or detection conditions relative to a region of interest in the pixel array of the vision sensor 100 different from event occurrence conditions or detection conditions set relative to other regions of the pixel array of the vision sensor 100. In some example embodiments, the vision sensor 100 may set event occurrence conditions or detection conditions differently relative to multiple regions of interest in the pixel array of the vision sensor 100.

[0038] Processor 200 can process event signals EVS received from vision sensor 100 and, based on the processing, detect movement of an object being imaged by vision sensor 100 (or, for example, movement of an object in an image (e.g., an image generated, captured, etc.) recognized by image processing device 10 based on signals generated by vision sensor 100 and / or vision sensor 101). Processor 200 may include application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), dedicated microprocessors, microprocessors, general-purpose processors, etc. In some example embodiments, processor 200 may include an application processor or an image processor.

[0039] Some or all of the vision sensor 100 and / or processor 200 may be included in one or more instances of processing circuitry (e.g., hardware including logic circuitry; hardware / software combinations, such as a processor executing software; or combinations thereof), and may include one or more instances of processing circuitry, and / or may be implemented by one or more instances of processing circuitry. For example, the processing circuitry may more specifically include, but is not limited to, a central processing unit (CPU), an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field-programmable gate array (FPGA), a system-on-a-chip (SoC), a programmable logic unit, a microprocessor, an application-specific integrated circuit (ASIC), etc. In some example embodiments, the processing circuitry may include: a non-transitory computer-readable storage device (e.g., a memory), such as a solid-state drive (SSD), storing instruction programs; and a processor configured to execute instruction programs to implement some or all of the functions of the vision sensor 100 and / or processor 200.

[0040] Alternatively, the vision sensor 100 and processor 200 can each be implemented as an integrated circuit (IC). For example, the vision sensor 100 and processor 200 may include separate semiconductor chips. Alternatively, the vision sensor 100 and processor 200 may be implemented on a single chip. For example, the vision sensor 100 and processor 200 may be implemented as a single system-on-a-chip (SoC).

[0041] Figure 2 This is a graph showing the data loss situation when the event signal EVS occurs.

[0042] refer to Figure 2 The horizontal axis represents elapsed time, and the vertical axis represents the amount (e.g., quantity, count, etc.) of event signals EVS transmitted from vision sensor 100 to processor 200 at any given point in time. During the transmission of event signals EVS to processor 200, loss of event signals EVS, i.e., data loss, may occur. Data loss may occur when a large number of events occur over a long period (e.g., changes in the intensity of incident light on one or more pixels of the pixel array of the vision sensor) and the amount of event signals EVS transmitted from vision sensor 100 to processor 200 (i.e., the amount of data transmitted over that period, data transmission rate, etc.) exceeds system limits (also known as the threshold amount of data that can be transmitted from the vision sensor to the processor at any given time) (e.g., the transmission speed limit of the communication channel between vision sensor 100 and processor 200). Therefore, the performance of image processing device 10 (e.g., the accuracy with which image processing device 10 detects the movement of an object) may be reduced.

[0043] However, in the image processing apparatus 10 according to some example embodiments, as referenced above... Figure 1 The vision sensor 100 can selectively transmit one or more event signals EVS generated at the vision sensor 100 to the processor 200. This may include selectively transmitting a limited selection (e.g., only some or one) of the event signals EVS generated at the vision sensor 100 during one or more specific time periods, and the amount of event signals EVS transmitted can be reduced compared to the amount of event signals EVS transmitted if all event signals EVS generated during one or more specific time periods were transmitted from the vision sensor 100 to the processor 200. Therefore, data loss during the transmission of event signals EVS between the vision sensor 100 and the processor 200 can be reduced or prevented. Consequently, the performance of the image processing device 10 can be improved, for example, the accuracy of detecting object movement by the image processing device 10.

[0044] Figure 3 This is a block diagram illustrating a vision sensor 100 according to an exemplary embodiment of the present invention.

[0045] refer to Figure 3 The vision sensor 100 may include a pixel array 110, an event detection circuit 120, an event rate controller 130, and an interface circuit 140. The pixel array 110, event detection circuit 120, event rate controller 130, and interface circuit 140 may each be implemented as an integrated circuit (IC). For example, the pixel array 110, event detection circuit 120, event rate controller 130, and interface circuit 140 may include separate semiconductor chips. Alternatively, two or more of the pixel array 110, event detection circuit 120, event rate controller 130, or interface circuit 140 may be implemented on a single chip. For example, the event detection circuit 120, event rate controller 130, and interface circuit 140 may be implemented as a single system-on-a-chip (SoC).

[0046] Some or all of the pixel array 110, event detection circuitry 120, event rate controller 130, and interface circuitry 140 may be included in one or more instances of processing circuitry (e.g., hardware including logic circuitry; hardware / software combinations, such as a processor executing software; or combinations thereof), and may include one or more instances of processing circuitry, and / or may be implemented by one or more instances of processing circuitry. For example, the processing circuitry may more specifically include, but is not limited to, a central processing unit (CPU), an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field-programmable gate array (FPGA), a system-on-a-chip (SoC), a programmable logic unit, a microprocessor, an application-specific integrated circuit (ASIC), etc. In some example embodiments, the processing circuitry may include: a non-transitory computer-readable storage device (e.g., a memory), such as a solid-state drive (SSD), storing instruction programs; and a processor configured to execute instruction programs to implement some or all of the functions of the pixel array 110, event detection circuitry 120, event rate controller 130, and interface circuitry 140.

[0047] Pixel array 110 may include a plurality of pixels arranged in a matrix. Each of the plurality of pixels PX can sense incident light. In some example embodiments, each of the plurality of pixels PX can generate one or more electrical signals in response to incident light received at each pixel PX, and an image of an external scene, including an image of an object, can be generated based on (e.g., at processor 200) processing the signals generated by the pixels PX in response to the received incident light. The image may be generated by a separate pixel array 110, which may be included in a separate visual sensor (e.g., in…). Figure 1 In the visual sensor 101 shown, multiple pixels PX can sense events in which the intensity of received light (e.g., the intensity of incident light on each pixel PX) increases or decreases. For example, each of the multiple pixels PX can be connected to the event detection circuit 120 via column lines extending in the column direction and row lines extending in the row direction. An electrical signal for notifying (e.g., indicating) the occurrence of an event and information about the polarity of the event (i.e., whether the event is an on event with increasing light intensity or an off event with decreasing light intensity) can be transmitted from the pixel PX that sends the event (e.g., generated at the pixel PX and transmitted from the pixel PX) to the event detection circuit 120 in response to the occurrence of the event (e.g., in response to a change in the intensity of incident light received and / or absorbed at the pixel PX).

[0048] Event detection circuit 120 can read and process events from pixel array 110. For example, a pixel PX of pixel array 110 can output an electrical signal in response to a change in the intensity of incident light at pixel PX, and event detection circuit 120 can receive and process the electrical signal output by pixel PX to determine the occurrence of an event at pixel PX. In response to such determination, event detection circuit 120 can generate an event signal EVS corresponding to pixel PX (also referred to as an event signal EVS corresponding to the occurrence of an event at pixel PX and / or the event itself), wherein event signal EVS includes (e.g., indicating) the polarity of the event that occurred (e.g., whether the incident light intensity increased or decreased at pixel PX), the address of the pixel PX where the event occurred, and a timestamp indicating when the event occurred. To reiterate, the event detection circuit 120 can be configured to detect whether a change in the intensity of incident light has occurred at any pixel PX in the plurality of pixels PX based on processing one or more electrical signals received from one or more pixels PX among the plurality of pixels PX, and the event detection circuit 120 can generate one or more event signals EVS corresponding to one or more pixels PX among the plurality of pixels PX for which a change in the intensity of incident light has been determined to have occurred.

[0049] The event detection circuit 120 can process events occurring in the pixel array 110 in the form of pixel units, pixel group units including multiple pixels, column units, or frame units (e.g., processing electrical signals output from each pixel PX of the pixel array 110).

[0050] The event rate controller 130 can adjust the amount of event signals EVS (generated at the event detection circuit 120) to be transmitted to the processor 200. In some example embodiments, the event rate controller 130 may select only the event signals EVS generated during a given time period from all event signals EVS generated at the event detection circuit 120 during that given time period (all event signals EVS generated during that time period that correspond to any pixel PX included in the entire region 110-E of the pixel array 110), based on events occurring at a specific pixel PX included in the ROI 110-R of the pixel array 110 (e.g., only event signals EVS corresponding to a specific pixel PX included in the ROI 110-R of the pixel array 110). For example, the event rate controller 130 can select a limited selection of event signals EVS, which includes only event signals EVS generated during a given time period that correspond to a limited selection of pixel PXs of the pixel array 110 included in the ROI 110-R of the pixel array 110. The limited selection of event signals EVS generated during a given time period may be less than all event signals EVS generated during that time period, so at least one event signal EVS may not be selected (e.g., it may be excluded from the limited selection of event signals EVS). The event rate controller 130 may output the selected event signal (hereinafter referred to as the output event signal EVS_O) to the interface circuit 140, or control the event detection circuit 120 to output the output event signal EVS_O to the interface circuit 140, thereby avoiding the output of any unselected event signals EVS (e.g., excluding the unselected event signals EVS from being transmitted to the interface circuit 140). To reiterate, the event rate controller 130 can be configured to select from one or more event signals EVS generated at the event detection circuit 120 that correspond to a region of interest (ROI) on the pixel array, as one or more output event signals EVS_O. The one or more output event signals EVS_O may include a limited selection of event signals EVS, thus excluding one or more of the event signals EVS generated at the event detection circuit 120.

[0051] In some example embodiments, the event rate controller 130 may reduce the amount of event signals EVS generated at the event detection circuit 120. The event rate controller 130 may adjust event occurrence conditions or event detection conditions (e.g., event occurrence threshold or event detection threshold), and thus may be able to adjust the thresholds upon which the event detection circuit 120 selectively generates one or more event signals in response to receiving one or more electrical signals from the pixel array 110. For example, event occurrence conditions may include the sensitivity of the pixel PX, and event detection conditions may include event detection period, denoising conditions (e.g., denoising threshold), etc.

[0052] In some example embodiments, when an excessive number of events occurs or is expected to occur within a given time period (e.g., a time interval), the event rate controller 130 may reduce the amount of event signals EVS to be transmitted or reduce the amount of event signals EVS generated at the event detection circuit 120. For example, the event rate controller 130 may count the number (amount) of event signals EVS in a specific (or alternatively, predetermined) time unit (a time interval, time period, etc.), and when the counted amount of event signals EVS is equal to or greater than a specific (or alternatively, predetermined) reference value (also referred to herein as a threshold or threshold amount), the event rate controller 130 may responsively determine that an excessive number of events has occurred during the specific time unit. For example, the event rate controller 130 may base its determination on the brightness of the captured object, the illuminance around the object, and the pixel array 110 (see [link to relevant documentation]). Figure 1 The event rate controller 130 determines whether a condition for an expected excessive event has occurred by measuring the amount of light received (e.g., intensity). Therefore, the event rate controller 130 can control the generation and / or transmission of one or more event signals EVS such that the amount of the output event signal EVS_O does not meet or exceed a threshold amount (e.g., as...). Figure 2 (System limitations shown).

[0053] Interface circuit 140 can receive the output event signal EVS_O transmitted by event rate controller 130, and transmit the output event signal EVS_O to processor 200 according to a specific (or, alternatively, predetermined) protocol (see...). Figure 1Therefore, interface circuitry 140 can be configured to communicate with an external processor (e.g., processor 200) relative to vision sensor 100 and transmit one or more output event signals EVS_O to the external processor. Interface circuitry 140 can package the output event signals EVS_O into separate signal units, packet units, or frame units to generate event data EDT according to a specific (or alternatively, predetermined) protocol, and transmit the event data EDT to processor 200. For example, interface circuitry 140 may include one of an AER interface, a Mobile Industrial Processor Interface (MIPI), and a parallel interface. Processor 200 can process the transmitted output event signals EVS_O or the event signals EVS transmitted by interface circuitry 140 to detect movement of an object imaged by the pixel array 110 of vision sensor 100.

[0054] In the following description, according to the present invention, the event signal EVS or the output event signal EVS_O is output, which indicates that the event signal EVS or the output event signal EVS_O is converted into event data EDT through the interface circuit 140, and the event data EDT is transmitted to the processor 200.

[0055] Figure 4A and Figure 4B Each illustrates the format of event data output from the vision sensor 100 according to some example embodiments of the present invention.

[0056] refer to Figure 4A The output of a packet PK, including at least one event signal EVS, can be used as event data (EDT). The packet PK may include the timestamp TS of the event signal EVS, the column address C_ADDR, the row address R_ADDR, and the polarity information POL, and the above arrangement order is not limited to... Figure 4A The order shown. The header H indicating the start of a packet PK and the tail T indicating the end of a packet PK can be added to the front and back of the packet PK, respectively. Although Figure 4A The diagram shows that the group PK includes only the event signal EVS, but the group PK is not limited to this and may include multiple event signals EVS.

[0057] A timestamp TS can include information about when an event occurred. For example, a timestamp TS can be 32 bits, but is not limited to this.

[0058] The column address C_ADDR and row address R_ADDR can each include multiple bits, such as 8 bits. In this case, a vision sensor comprising multiple pixels arranged in up to eight columns and eight rows can be provided. However, this is only an example, and the bit rate of the column address C_ADDR and row address R_ADDR can vary depending on the number of pixels (PX).

[0059] The polarity information (POL) can include information related to on and off events. For example, the polarity information (POL) can include: a bit containing information about whether an on event has occurred; and a bit containing information about whether a off event has occurred. For example, the bit indicating the on event and the bit indicating the off event do not necessarily both have to be "1", but they can both have to be "0".

[0060] refer to Figure 4B It can output frame data comprising multiple groups PK (e.g., first group PK1 to m-th group PKm) (m is an integer equal to or greater than 2) as event data. In some example embodiments, the frame data may include event data when pixel array 110 (see...) Figure 3 The event signal EVS is generated when a frame is scanned once, therefore, the number of packet PKs can vary for each frame of data. In some example embodiments, the frame data may include a fixed number of packet PKs.

[0061] Figure 5A and Figure 5B These are all figures used to describe a method for outputting an event signal EVS corresponding to the ROI from a vision sensor 100 according to some example embodiments of the present invention.

[0062] refer to Figure 5A At least one ROI can be set (e.g., established, defined, etc.) relative to the pixel array 110, such as a first ROI ROI1 and a second ROI ROI2. Figure 5A The diagram shows the setting of two ROIs (e.g., first ROI ROI1 and second ROI ROI2), however, the number of ROIs is not limited to this, and one or more ROIs can be set.

[0063] Among the multiple regions set relative to the pixel array 110, the region where multiple events occur can be set as the ROI, or any region corresponding to a pixel PX that experiences multiple events during a specific (or, alternatively, predetermined) time period can be set as the ROI. Alternatively, the user can arbitrarily set the ROI, or any region corresponding to a pixel PX that senses a specific object can be set as the ROI. However, the ROI is not limited to this and can be set according to various methods.

[0064] An event signal EVS is generated based on events occurring in the pixel array 110. Since the event signal EVS includes an address, a virtual event graph EVM can be constructed based on the position of the corresponding pixel PX on the pixel array 110. For example... Figure 5A As shown, the event signal EVS can be classified into event signal EVS_I corresponding to the ROI and event signal EVS_NI corresponding to the region outside the ROI. The event signal EVS_I corresponding to the ROI can be generated as event data EDT and transmitted to processor 200 (see [link]). Figure 1 The event signal EVS_NI can be selectively excluded from the event data EDT, and thus excluded from being transmitted to the processor 200.

[0065] refer to Figure 3 The event rate controller 130 can receive event signals EVS from the event detection circuit 120, and can select only the event signal EVS_I corresponding to the ROI from the received event signals EVS as the output event signal EVS_O. The interface circuit 140 can output the output event signal EVS_O, which includes the event signal EVS_I corresponding to the ROI and excludes the event signal EVS_NI corresponding to the region outside the ROI, as event data EDT to the processor 200 (see [link]). Figure 6 ).

[0066] refer to Figure 5B An event can occur in some of the multiple pixels PX in the pixel array 110. From the pixel PX where the event occurs (e.g., a change in incident light intensity), one or more event signals EVS corresponding to the pixel PX_I corresponding to the ROI can be generated, and event signals EVS_NI corresponding to the pixel PX_NI corresponding to the region outside the ROI can be ignored (e.g., selectively excluded from being transmitted to the processor 200).

[0067] refer to Figure 3The event rate controller 130 can provide the event detection circuit 120 with information related to the ROI, i.e., the address related to the ROI, and the event detection circuit 120 can selectively sense the pixel PX_I corresponding to the ROI among the plurality of pixels PX in the pixel array 110 based on the information related to the ROI. In doing so, events of the pixel PX_I corresponding to the ROI can be read out (e.g., electrical signals received from the pixel PX_I at the event detection circuit 120), and the event detection circuit 120 can selectively generate event signals EVS based on the read events (e.g., generating event signals EVS only in response to electrical signals received from the pixel PX_I, and selectively avoiding generating event signals EVS in response to electrical signals received from the pixel PX_I). The event signals EVS generated based on events occurring at the pixel PX_I corresponding to the ROI (e.g., event signals EVS corresponding to the pixel PX_I) can be provided to the interface circuit 140. The interface circuit 140 can output (e.g., transmit) the received event signals EVS to the processor 200 (see...). Figure 1 This is used as event data (EDT).

[0068] Figure 6 This is a block diagram illustrating some example embodiments of a vision sensor 100 according to the present invention.

[0069] refer to Figure 6 The vision sensor 100 includes a pixel array 110, an event detection circuit 120, an event rate controller 130, and an interface circuit 140. The event detection circuit 120 may include a row address event representation (AER) 121, a column AER 122, a voltage generator 123, and an event signal processing (ESP) unit.

[0070] Due to reference Figure 3 The pixel array 110, the event rate controller 130, and the interface circuit 140 are described, so repeated descriptions will be omitted.

[0071] Line AER 121 can receive an electrical signal (i.e., a line request) notifying (e.g., indicating) the occurrence of an event from one or more pixels PX in the pixel array 110, and generate the line address R_ADDR of the pixel PX where the event occurred.

[0072] Column AER 122 can receive an electrical signal (i.e., a column request) notifying (e.g., indicating) the occurrence of an event from one or more pixels PX of the pixel array 110, and generate the column address C_ADDR of the pixel PX where the event occurred.

[0073] In some example embodiments, the pixel array 110 can be scanned column by column, and when column AER 122 receives a request from a specific column (e.g., the first column), column AER 122 can transmit a response signal to the first column. A pixel PX that receives the response signal (where an event has occurred) can transmit polarity information POL (e.g., a signal indicating that an event has been enabled or disabled) to row AER 121. When polarity information POL is received, row AER 121 can transmit a reset signal to the pixel PX where the event occurred. The pixel PX where the event occurred can be reset in response to the reset signal. Row AER 121 can control the period at which the reset signal is generated. Row AER 121 can generate information related to the time of the event, i.e., a timestamp TS.

[0074] In some exemplary embodiments, as referenced above Figure 3 As described above, when adjusting the amount of the generated event signal, line AER 121 can extend the period during which the reset signal is generated. When the period during which the reset signal occurs is extended, the time period during which the pixel array 110 is scanned (i.e., the time interval) increases, thus reducing the amount of events occurring within a unit time interval. Therefore, the performance of the image processing device 10, including the vision sensor 100, can be improved, for example, by increasing the accuracy of detecting object movement using the image processing device 10.

[0075] Assuming the pixel array 110 is scanned column-by-column, the operations of row AER 121 and column AER 122 are described. However, the operations of row AER 121 and column AER 122 are not limited to this, and row AER 121 and column AER 122 can read the occurrence and polarity information POL of the event from the pixel PX where the event occurred in various ways. For example, the pixel array 110 can be scanned row-by-row, and the operations of row AER 121 and column AER 122 can be changed; that is, column AER 122 can receive the polarity information POL and transmit a reset signal to the pixel array 110. Additionally, row AER 121 and column AER 122 can also access the pixel PX where the event occurred independently.

[0076] Voltage generator 123 can generate voltages supplied to pixel array 110. For example, voltage generator 123 can generate threshold voltages or bias voltages for detecting on and off events from pixel PX. In some example embodiments, as referenced above... Figure 3 As stated above, when adjusting the sensitivity of pixel PX, voltage generator 123 can change the level (or bias voltage) of the reference voltage line under the control of event rate controller 130. Voltage generator 123 can change the voltage level of the threshold voltage supplied to pixel PX in ROI, and can change the voltage level of the threshold voltage in a variety of ways relative to multiple ROIs.

[0077] ESP unit 124 can generate an event signal EVS based on the row address R_ADDR, column address C_ADDR, polarity signal POL, and timestamp TS received from row AER 121 and column AER 122. In some example embodiments, ESP unit 124 can remove noisy events and generate event signals EVS for valid events. For example, when the amount of events occurring over a period of time (e.g., the amount of signals generated by one or more pixels PX and received at event detection circuit 120) is less than a specific (or alternatively, a predetermined) threshold (e.g., a denoising threshold), ESP unit 124 can identify the event as noise and, in response to the noisy event, not generate (e.g., avoid generating) an event signal EVS, which can be referred to as "removing" and / or "excluding" the noisy event. To reiterate, event detection circuit 120 can determine that one or more electrical signals generated by one or more pixels PX are noisy events based on the denoising threshold and can responsively avoid generating event signals EVS corresponding to the noisy events.

[0078] In some example embodiments, such as reference Figure 3 When adjusting the amount of the event signal EVS, the event rate controller 130 can increase the threshold voltage used to determine noise, i.e., the denoising threshold (e.g., via the signal CON to the event detection circuit 120), and the ESP unit 124 can identify noisy events based on the increased threshold. For example, when the threshold is set to 10 and the amount of events occurring in a unit time period (e.g., during the unit time period) is 12, the ESP unit 124 can determine these events as valid events and generate an event signal EVS for the event (e.g., corresponding to the event). When the threshold is increased to 20 and the amount of events occurring in the unit time period is 12, the ESP unit 124 can determine the event as a noisy event, ignore the event, and not generate (e.g., avoid generating) an event signal EVS for the event.

[0079] Figure 7 This is a circuit diagram illustrating some example embodiments of pixel PX.

[0080] refer to Figure 7 The pixel PX may include a photoelectric conversion element PD, an amplifier 111, a first comparator 112, a second comparator 113, an on event hold 114, an off event hold 115, and a reset switch SW. Additionally, the pixel PX may also include: a capacitor for removing noise present in the pixel or entering from the outside; various switches; and feedback circuitry.

[0081] A photoelectric conversion device (PD) can convert incident light (i.e., an optical signal) into an electrical signal (e.g., an electric current). A PD can include, for example, a photodiode, a phototransistor, a photogate, a pinned photodiode, etc. When the intensity of the incident light increases, the PD can generate an electrical signal with a high level.

[0082] Amplifier 111 can convert the received current into voltage and amplify the voltage level. The output voltage of amplifier 111 can be provided to a first comparator 112 and a second comparator 113. In some example embodiments, a feedback circuit can be connected between the input and output terminals of amplifier 111.

[0083] The first comparator 112 compares the output voltage Vout of the amplifier 111 with the turn-on threshold voltage TH1 and generates an turn-on signal E_ON based on the comparison result. The second comparator 113 compares the output voltage of the amplifier 111 with the turn-off threshold voltage TH2 and generates a turn-off signal E_OFF based on the comparison result. When the change in the amount of light received by the photoelectric conversion device PD is equal to or greater than a specific change level, the first comparator 112 and the second comparator 113 can generate either the turn-on signal E_ON or the turn-off signal E_OFF.

[0084] For example, when the amount of light received by the photoelectric conversion device PD increases to a certain level or higher, the turn-on signal E_ON is high, and when the amount of light received by the photoelectric conversion device decreases to a certain level or lower, the turn-off signal E_OFF can be high. The turn-on event hold unit 114 and the turn-off event hold unit 115 can hold and output the turn-on signal E_ON and the turn-off signal E_OFF, respectively. When the pixel PX is scanned, the turn-on signal E_ON and the turn-off signal E_OFF can be output. Furthermore, as described above, when adjusting the sensitivity, the levels of the turn-on threshold voltage TH1 and the turn-off threshold voltage TH2 can be changed. That is, the sensitivity can be reduced. Therefore, the level of the turn-on threshold voltage TH1 can be increased, and the level of the turn-off threshold voltage TH2 can be decreased. Therefore, when the change in the amount of light received by the photoelectric conversion device PD is greater than before (i.e., before changing the levels of the turn-on threshold voltage TH1 and the turn-off threshold voltage TH2), the first comparator 112 and the second comparator 113 can generate the turn-on signal E_ON or the turn-off signal E_OFF.

[0085] Figure 8A , Figure 8B , Figure 8C and Figure 8D These are the instructions shown separately. Figure 7 A circuit diagram of an example embodiment showing the connection relationship between the photoelectric conversion device PD and the amplifier 111.

[0086] refer to Figure 8A Amplifier 111a may include output circuit 12 and feedback circuit 11a, and feedback circuit 11a may include converter circuit 21 and boost circuit 22a. Amplifier 111a may also include current source 13 that provides bias current to output circuit 12.

[0087] One end of the output circuit 12 can be connected to one end of the photoelectric conversion device PD and one end of the feedback circuit 11a. The other end of the output circuit 12 can be connected to the other end of the feedback circuit 11a and the output node, through which the output voltage Vout is output.

[0088] Output circuit 12 can generate an output voltage Vout, i.e., an output signal, based on the input signal received from the photoelectric conversion device PD and the feedback signal received from the feedback circuit 11a. As shown in the figure, output circuit 12 may include an amplifying transistor M. AMP The amplifying transistor M AMP This includes a gate node in contact with one end of the feedback circuit 11a, a source node to which a ground voltage is applied, and a drain node that generates the output voltage Vout. The output circuit 12 can generate the output voltage Vout by amplifying the feedback signal using a set gain (positive gain or negative gain).

[0089] Feedback circuit 11a can feed back the output voltage Vout to the front end of output circuit 12, that is, to amplifying transistor M. AMP The gate node. For example, feedback circuit 11a can generate a feedback signal that is exponentially proportional to the change in output voltage Vout and feed this feedback signal back to the front end of output circuit 12.

[0090] The converter circuit 21 can generate a conversion signal based on the output voltage Vout. In some example embodiments, the conversion signal can be a current that is exponentially proportional to the change in the output voltage Vout. LOGN The transistor M LOGN It includes a gate node for receiving the output voltage Vout and a source node for outputting a conversion signal based on the output voltage Vout. Figure 8A Transistor M is shown LOGN This is the case with NMOS transistors, but transistor M... LOGN It is not limited to this, and can also be a PMOS transistor.

[0091] The boost circuit 22a can output a boost voltage that is proportional to the natural logarithm of the input current. The boost circuit 22a can include a transistor M1, wherein the drain node and the gate node are in contact with each other. The transistor M1 included in the boost circuit 22a can operate in the subthreshold region and generate a boost voltage based on the input current. The boost voltage can indicate the voltage applied to the gate node and source node of the transistor M1. Figure 8A A boost circuit 22a including an NMOS transistor is shown, but the boost circuit 22a is not limited thereto and may include a PMOS transistor.

[0092] In some example embodiments, Figure 8A The change in output voltage Vout over time, ΔVout, can be expressed as 2 × β × ln(Iin2 / Iin1). In this case, Iin1 is the current supplied from the photoelectric conversion device PD at the first time point, and Iin2 is the current supplied from the photoelectric conversion device PD at the second time point after the first time point. β is a coefficient determined based on the characteristics of the transistor or its operating temperature.

[0093] Figures 8B to 8D The diagram shows the connection between the photoelectric conversion device (PD) and the amplifier, the components of the amplifier, and the operation of the amplifier. Figure 8A The connection between the photoelectric conversion device PD and the amplifier 111, the components of the amplifier 111a, and the operation of the amplifier are similar. Therefore, the differences will be mainly described below.

[0094] refer to Figure 8B In amplifier 111b, one end of output circuit 12 can be connected to both the photoelectric conversion device PD and one end of feedback circuit 11b. The other end of output circuit 12 can be connected to the other end of feedback circuit 11b. Figure 8B In the circuit, the feedback circuit 11b includes three terminals, and the last terminal of the three terminals can be connected to the output node from which the output voltage Vout is output.

[0095] The boost circuit 22b included in the feedback circuit 11b may include a first transistor T1 and a second transistor M2. The first transistor M1 may include a gate node in contact with the source node of the second transistor M2 and a drain node in contact with the gate node of the second transistor M2. The second transistor M2 may include a source node in contact with the gate node of the first transistor M1 and a gate node in contact with the drain node of the first transistor M1. The second transistor M2 may operate based on a bias current supplied from the current source 13, and the first transistor M1 may operate based on a received input current. Although in Figure 8BThe first transistor M1 and the second transistor M2 are shown as NMOS transistors, but the first transistor M1 and the second transistor M2 are not limited to this and can also be PMOS transistors.

[0096] refer to Figure 8C The amplifier 111c may also include a current source 14 that synchronizes the bias current from the feedback circuit 11c.

[0097] Components and operation of feedback circuit 11c Figure 8B The components and operation of the feedback circuit 11b are similar. The boost circuit 22c included in the feedback circuit 11c may include a first transistor M1 and a second transistor M2. (The last sentence appears to be incomplete and possibly refers to a different circuit.) Figure 8C Unlike other transistors, the drain node of the second transistor M2 can receive the power supply voltage, and the source node can be connected to the current source 14 and the gate node of the first transistor M1.

[0098] refer to Figure 8D Amplifier 111d may include output circuit 12d and feedback circuit 11d, and feedback circuit 11d may include converter circuit 21d and boost circuit 22d. Amplifier 111d may also include: current source 13, which provides bias current to output circuit 12d; and current source 15, which synchronizes the bias current from output circuit 12d.

[0099] One end of the output circuit 12d can be connected to the photoelectric conversion device PD and one end of the feedback circuit 11d. The other end of the output circuit 12d can be connected to the other end of the feedback circuit 11d. The output circuit 12d can generate an output voltage Vout, i.e., an output signal, based on the input signal received from the photoelectric conversion device PD and the feedback signal received from the feedback circuit 11d. The output circuit 12d may include: an amplifying transistor M. AMP It receives input current and includes a gate node corresponding to one end; and an output transistor M. SF It includes a gate node corresponding to the other end and a source node that generates the output voltage Vout. Here, the input current can be generated by a photoelectric conversion device PD.

[0100] The feedback circuit 11d may include a circuit structure in which a boost circuit 22d, in which the source follower is cascaded to the converter circuit 21d. The converter circuit 21d may include an NMOS transistor M. LOGN and PMOS transistor M LOGP The NMOS transistor M included in the converter circuit 21d LOGNThis may include a source node that contacts the drain node of the first transistor M1 and the gate node of the second transistor M2 in the boost circuit 22d, and a gate node that contacts the other end of the output circuit 12d. This includes a PMOS transistor M1 in the converter circuit 21d. LOGP This may include a source node that contacts the source node of the first transistor M1 in the boost circuit 22d, and a drain node that contacts one end of the output circuit 12d and one end of the photoelectric conversion device PD. Additionally, the bias power V can be... BIAS Applied to PMOS transistor M LOGP The gate node.

[0101] Figure 9 A flowchart illustrating the operation method of a vision sensor 100 according to some example embodiments of the present invention is shown. Figure 9 The operation method can be found in Figure 3 This is executed within the vision sensor 100, and more specifically, within the event rate controller 130. Therefore, the description relating to the vision sensor 100 and its operation can be applied to... Figure 9 The example embodiment is shown. In the following, reference will be made to... Figure 3 To provide a description Figure 9 .

[0102] refer to Figures 3 to 9 The vision sensor 100 can set (e.g., define) the ROI (S110). See reference... Figure 5A As described, the event rate controller 130 can set the ROI based on the amount of events that occur. Alternatively, the event rate controller 130 can set the ROI according to user settings, or set the pixels PX of the sensed object as the ROI.

[0103] The visual sensor 100 can measure the amount of event signals that occur (e.g., the amount of event signals EVS generated in response to changes in incident light intensity by one or more pixels PX within a specific time period) (S120). For example, the amount of event signals that occur can be measured when the event rate controller 130 counts the event signals EVS generated during a specific (or, alternatively, predetermined) time period unit. In some example embodiments, operation S110 may be performed simultaneously with operation S120 or after operation S120.

[0104] The vision sensor 100 can compare the quantity (e.g., number, count, etc.) of event signals EVS that occur (e.g., generated during a specific time period) with a reference value (S130). When the quantity of the event signals EVS is less than the reference value, the vision sensor 100 can output event signals EVS corresponding to the entire area (e.g., it can output any event signals EVS generated during the specific time period) (S140). When the quantity of the event signals is equal to or greater than the reference value, the vision sensor 100 can selectively output event signals EVS corresponding to the ROI (e.g., it can output only event signals EVS corresponding to the ROI generated during the specific time period, and exclude any or all other event signals EVS generated during that specific time period) (S150). For example, the event rate controller 130 can compare the quantity of the event signals EVS that occur with a reference value. When the quantity of the event signals EVS is less than the reference value, the event rate controller 130 can output event signals EVS corresponding to the entire area and can also output event signals EVS corresponding to the ROI.

[0105] In other words, in Figure 9 In the method shown, the vision sensor 100 can determine an operating mode (also referred to herein as the operating mode of the vision sensor 100) based on the amount of event signals EVS occurring within a given time period, and thus selectively operate the vision sensor 100 in a determined (e.g., selected) specific operating mode. When (e.g., in response to determining) the number of event signals occurring is less than a reference value, the vision sensor 100 can responsively and selectively operate in a first operating mode that outputs event signals EVS corresponding to the entire area, such that the event rate controller 130 can select one or more event signals EVS corresponding to the entire area of ​​the pixel array 110 (e.g., such as...) in response to the vision sensor 100 operating in the first operating mode. Figure 3 As shown, the event signal EVS corresponding to the signal output by any pixel PX in region 110-E, including all pixels PX in pixel array 110, is output as event signal EVS_O. When (e.g., in response to determination) the amount of the event signal EVS that occurs is equal to or greater than a reference value, the vision sensor 100 can responsively and selectively operate in a second operating mode to output the event signal EVS corresponding to the ROI, such that the event rate controller 130 can select the event signal EVS corresponding to the ROI (e.g., only the event signal EVS corresponding to the signal output by any pixel PX in region 110-R, such as...) in response to the vision sensor 100 operating in the second operating mode. Figure 3As shown, region 110-R is the ROI including a finite portion of pixels PX in pixel array 110, and serves as the output event signal EVS_O.

[0106] Therefore, it will be understood that the event rate controller 130 may: 1) measure the amount (e.g., quantity) of event signals EVS generated at the event detection circuit 120 based on counting event signals EVS generated at the event detection circuit 120 over a specific (e.g., predetermined) time period; and 2) selectively set the operating mode (also referred to herein as the operating mode) of the vision sensor 100 to one of a first operating mode or a second operating mode based on whether the measured amount of event signals EVS generated within the specific time period is less than a reference value.

[0107] Therefore, the vision sensor 100 can selectively transmit all event signals EVS generated based on events occurring at any pixel PX in the pixel array 110 based on the movement of the imaged object, or a limited selection of event signals EVS, as one or more output event signals EVS_O, in response to the vision sensor 100 operating in a first operating mode or a second operating mode. The limited selection of event signals corresponds to one or more pixel PXs among a plurality of pixels included in the region of interest (ROI) of the pixel array 110. The vision sensor 100 can selectively operate in a selected operating mode of the first operating mode or the second operating mode based on a comparison of the amount of event signals EVS to be generated during a specific time period with a threshold.

[0108] In some example embodiments, the event rate controller 130 may operate in a second operating mode in response to the vision sensor 100, for example, by controlling the voltage generator 123 to change (e.g., increase) the voltage level of the threshold voltage supplied to the pixel PX in the ROI, setting the ROI to have low sensitivity (e.g., the sensitivity of the pixel PX in the ROI may be adjusted to have reduced sensitivity), and changing (e.g., increasing) the voltage level of the threshold voltage differently relative to multiple ROIs. In some example embodiments, in response to the vision sensor 100 operating in the second operating mode, the event rate controller 130 may increase the denoising threshold of one or more pixels PX in one or more ROIs, causing the event detection circuitry to generate a reduced event signal EVS during a specific time period in response to signals received from one or more pixels PX. Therefore, when operating in the second operating mode, the vision sensor 100 can set multiple ROIs and set different sensitivities for pixels in different ROIs (for example, pixels PX in the first ROI can be set to all have the same first sensitivity, and pixels PX in the second ROI can be set to all have the same second sensitivity that is different from the first sensitivity), and can set different denoising thresholds for pixels in different ROIs (for example, pixels PX in the first ROI can be set to all be associated with the same first denoising threshold, and pixels PX in the second ROI can be set to all be associated with the same second denoising threshold that is different from the first denoising threshold).

[0109] Figure 10 A flowchart illustrating the operation method of a vision sensor 100 according to some example embodiments of the present invention is shown. Figure 10 The operation method can be found in Figure 3 This is executed within the vision sensor 100, and more specifically, within the event rate controller 130. Therefore, the description relating to the vision sensor 100 and its operation can be applied to... Figure 10 The example embodiment is shown. In the following, reference will be made to... Figure 3 To describe Figure 10 .

[0110] refer to Figure 3 and Figure 10 The visual sensor 100 can set a first ROI ROI1 and a second ROI ROI2 (S210). In other words, the visual sensor 100 can set multiple ROIs, such that the pixel array 110 includes multiple ROIs. The first ROI ROI1 can be set to have a lower importance (or priority) than the second ROI ROI2.

[0111] The vision sensor 100 can measure (e.g., count) the amount of event signals EVS that occur (e.g., generated at the event detection circuit 120 during a specific time period) (S220), and compare the measured amount of event signals EVS with a first reference value (S230). When the amount of event signals EVS is less than the first reference value, the vision sensor 100 can output an event signal EVS corresponding to the entire area of ​​the pixel array 110 (S240).

[0112] When the magnitude of the event signal EVS is equal to or greater than the first reference value, the vision sensor 100 can compare the magnitude of the event signal EVS with the second reference value (S250). The second reference value may be greater than the first reference value.

[0113] When the amount of the event signal EVS is less than the second reference value, the vision sensor 100 can output the event signal EVS corresponding to the first ROI ROI1 and the second ROI ROI2 (S260). In other words, the vision sensor 100 can output the event signal EVS corresponding to any or all ROIs. Reiterating, the event rate controller 130 can, in response to the measured amount of event signals generated during a specific time period being equal to or greater than the first reference value and less than the second reference value, set the operating mode to the second operating mode and select one or more event signals EVS corresponding to any ROI as one or more output event signals EVS_O. When the amount of the event signal EVS is equal to or greater than the second reference value, the vision sensor 100 can output the event signal EVS corresponding to the second ROI ROI2 (S270). In other words, based on the amount of the event signal EVS, the vision sensor 100 can output the event signal EVS corresponding to the area determined to be a relatively important area. Each individual ROI can be associated with a separate importance value, and the importance value associated with one of the ROIs (e.g., ROI1) can be greater than the importance value associated with the remaining ROIs (e.g., ROI2). Therefore, the event rate controller 130 can set the operating mode to a third operating mode in response to the measured amount of event signals generated during a specific time period being equal to or greater than a second reference value, and select one or more event signals EVS corresponding to at least one ROI as one or more output event signals EVS_O, wherein the importance value associated with the at least one ROI is greater than the importance value associated with the remaining ROIs. While no event signals EVS relating to the entire region are transmitted, event signals EVS corresponding to regions with high importance are selectively transmitted to the processor 200 (see [link to processor 200]). Figure 1Therefore, the performance degradation of the vision sensor 100 can be reduced or prevented, thereby improving the performance of electronic devices (e.g., computing devices) including the image processing device 10, and improving the performance of the vision sensor 100, especially in terms of the accuracy of tracking the movement of objects.

[0114] Furthermore, the operating mode of the vision sensor 100 is determined based on the amount of the event signal EVS, and the operating mode can be changed in specific (or, alternatively, predetermined) unit time period units. (See reference...) Figure 11 The description determines (e.g., selectively sets) the operating mode of the vision sensor 100.

[0115] Figure 11 This is a diagram illustrating a method for determining the operating mode of a vision sensor 100 according to some exemplary embodiments of the present invention.

[0116] refer to Figure 11 For each defined unit time period, such as each of the first time period TP1 to the fourth time period TP4, the amount of an event signal EVS that occurs (e.g., generated in response to an electrical signal received from one or more pixels PX in the pixel array 110 of the vision sensor 100) can be measured, and the operating mode of the vision sensor 100 can be determined based on the amount of the event signal EVS that occurs.

[0117] First, the vision sensor 100 can operate in a default mode (i.e., first mode MD1) during the initial time period TP1. The vision sensor 100 can output all event signals EVS generated during the first time period TP1. When the output of the event signals EVS is delayed by a specific (or, alternatively, predetermined) unit time period or longer, the vision sensor 100 can output all event signals EVS generated before the first time period TP1.

[0118] When the amount of event signals EVS counted during the first time period TP1 (i.e., the amount of event signals EVS that have occurred) is equal to or greater than the first reference value, the vision sensor 100 can be set to a second operating mode MD2, and therefore can operate in the second operating mode MD2 during the second time period TP2. The vision sensor 100 can selectively output event signals EVS corresponding to the ROI from the event signals EVS generated during the second time period TP2 or the first time period TP1. In some example embodiments, the occurrence conditions or detection conditions of the event can be adjusted to reduce the amount of event signals EVS that have occurred.

[0119] When the amount of event signals EVS counted during the second time period TP2 (i.e., the amount of event signals EVS that occurred) is less than the first reference value, the vision sensor 100 can be reset to the first operating mode MD1, and therefore can operate in the first operating mode MD1 during the third time period TP3. The vision sensor 100 can output all event signals EVS generated during the third time period TP3 or the second time period TP2. Additionally, the occurrence conditions or detection conditions of events can be initialized during the third time period TP3. For example, values ​​can be set according to internally configured registers or by the processor 200 (see [link to relevant documentation]). Figure 1 The register settings are used to initialize the conditions for the occurrence or detection of events.

[0120] When the amount of event signals EVS counted during the third time period TP3 (i.e., the amount of event signals EVS that occurred) is equal to or greater than the second reference value, the vision sensor 100 can be set to a third operating mode MD3, and therefore can operate in the third operating mode MD3 during the fourth time period TP4. The vision sensor 100 can selectively output event signals corresponding to ROIs of high importance from the event signals EVS generated during the fourth time period TP4 or the third time period TP3. In some example embodiments, the occurrence or detection conditions of the event can be adjusted to reduce the amount of event signals EVS that occurred. In this case, the occurrence or detection conditions of the event can be adjusted to be higher than the second time period TP2 (i.e., in a direction where the amount of events or event signals EVS that occurred is further reduced).

[0121] As described, the operating mode of the vision sensor 100 can be dynamically changed based on the amount of event signals EVS that occur. For example, the event rate controller 130 can measure the amount of event signals EVS generated at the event detection circuit 120 during each specific time period in a series of specific time periods, and dynamically set the operating mode for each specific time period based on the amount of event signals counted at the event rate controller during the specific time period.

[0122] Figure 12A and Figure 12B It is shown Figure 3 A block diagram of an embodiment of the event rate controller 130 shown.

[0123] refer to Figure 12AThe event rate controller 130 may include an event counter 131, an analog parameter controller 132, a noise reduction (DNS) controller 133, and an area of ​​origin (ROI) controller 134. The configuration of the event rate controller 130 (i.e., the event counter 131, analog parameter controller 132, DNS controller 133, and ROI controller 134) can be implemented in hardware or software. For example, the configuration of the event rate controller 130 can be implemented as logic circuitry. Alternatively, the configuration of the event rate controller 130 can be implemented as instructions programmed to perform functions and a processor for executing those instructions.

[0124] Event counter 131 can count the event signals EVS output from ESP unit 124 and output the amount of the counted event signals, that is, the amount of the event signals EVS that have occurred.

[0125] When the magnitude of the event signal EVS is equal to or greater than a reference value, the analog parameter controller 132 can adjust at least one setting parameter from the analog circuitry (i.e., row AER 121, column AER 122, voltage generator 123). For example, the analog parameter controller 132 can control the voltage generator 123 (see...). Figure 6 ), so that it is provided to the comparator (e.g., the first comparator 112 and the second comparator 113) (see Figure 7 The voltage levels of the first threshold voltage TH1 and the second threshold voltage TH2 are used to detect on and off events from pixel PX. Analog parameter controller 132 can change the voltage levels of the voltage generator 123 (see [link to analog parameter controller 132]) in relation to the first threshold voltage TH1 and the second threshold voltage TH2. Figure 6 The register value set in the register of ) causes the first threshold voltage TH1 to increase and the second threshold voltage TH2 to decrease.

[0126] In some example embodiments, the first threshold voltage TH1 and the second threshold voltage TH2 can be targeted at the ROI and the area outside the ROI on the pixel array 110 (see [reference]). Figure 3 The first threshold voltage TH1 and the second threshold voltage TH2 can be set differently for each ROI.

[0127] When the amount of the event signal EVS is equal to or greater than a reference value, the DNS controller 133 can increase the denoising threshold. Therefore, no event signal EVS is generated for events determined to be noise. Consequently, the amount of the event signal EVS can be reduced.

[0128] In some example embodiments, noise determination can be performed at the pixel block unit level, and the ROI controller 134 can target both the ROI and the area outside the ROI on the pixel array 110 (see [link]). Figure 3The ROI controller 134 can set different denoising thresholds. In addition, when multiple ROIs are set, the ROI controller 134 can set the first threshold TH1 and the second threshold TH2 differently among the ROIs.

[0129] When the magnitude of the event signal EVS is equal to or greater than a reference value, the ROI controller 134 can output the event signal EVS corresponding to the ROI as an output event signal EVS_O from the event signals EVS received from the ESP unit 124. In some example embodiments, multiple ROIs are set, and when the magnitude of the event signal EVS is equal to or greater than a first reference value and less than a second reference value, the ROI controller 134 can output event signals EVS corresponding to multiple ROIs. When the magnitude of the event signal is equal to or greater than the second reference value, the ROI controller 134 can output the event signal EVS corresponding to the smaller area, that is, the event signal EVS corresponding to the ROI with higher importance.

[0130] refer to Figure 12B The event rate controller 130a may include a first event counter 131a, a second event counter 132a, a first filter 133a, a second filter 134a, an analog parameter controller 135a, a DNS controller 136a, and an ROI controller 137a.

[0131] The first event counter 131a can count events before the use of the ESP unit 124, that is, count the event signal EVS before denoising, and output a first value of the event signal EVS that occurred. The second event counter 132a can count events after the use of the ESP unit 124, that is, count the event signal EVS after denoising, and output a second value of the event signal EVS that occurred.

[0132] The first filter 133a can average or calculate a first quantity of event signals EVS that occur (e.g., generated at event detection circuit 120) within a specific (or, alternatively, predetermined) time period unit (e.g., during which time). For example, the first filter 133a can average or calculate the first quantity of event signals EVS calculated at time point t1 and the first quantity of event signals EVS calculated at earlier time points (e.g., time points t-1, t-2, t-3, etc.). For example, weights can be multiplied by each of the first quantities of event signals EVS calculated at each time point, and the average of the weighted values ​​can be output as the filtered quantity of the event signal (e.g., generated at event detection circuit 120 during the specific time period unit). In this case, a smaller weight can be set when calculating the quantity of event signals EVS at a time point farther from the present. Therefore, rapid changes in the first quantity of the event signals EVS can be removed.

[0133] Therefore, it will be understood that the event rate controller 130 can apply (e.g., multiply) weights assigned according to the time point at which the event signal is generated at the event detection circuit 120 to the amount of the event signal EVS generated and measured within a specific time period unit. The event rate controller 130 can then generate the amount of filtered event signal generated during the specific time period unit by averaging the weighted values. The event rate controller 130 can then set the operating mode of the vision sensor 100 based on the resulting amount of filtered event signal generated at the event detection circuit 120 during the specific time period unit.

[0134] When the first quantity of the filtered event signal output from the first filter 133a is equal to or greater than the reference value, the analog parameter controller 135a can adjust at least one setting parameter from the analog circuit (i.e., row AER 121, column AER 122, and voltage generator 123).

[0135] The second filter 134a averages and calculates the amount of the event signal received from the ESP unit 124 (i.e., the amount of the first event signal) and the amount of the event signal indicating the occurrence of events before the use of the ESP unit 124 (i.e., events before noise removal) to generate a third amount of the event signal EVS. Rapid changes in the third amount of the event signal EVS can be removed by averaging or calculating the third amount of the event signal occurring in a specific (or, alternatively, predetermined) time period unit.

[0136] When the third value of the event signal output from the second filter 134a is equal to or greater than the reference value, the DNS controller 136a can increase the noise reduction threshold, and the ROI controller 137a can output the event signal EVS corresponding to the ROI from the event signals EVS received from the ESP unit 124 as the output event signal EVS_O. Alternatively, when the third value of the event signal is equal to or greater than the first reference value and less than the second reference value, the ROI controller 137a can output event signals EVS corresponding to multiple ROIs, and when the third value of the event signal is equal to or greater than the second reference value, the ROI controller 137a can output event signals EVS corresponding to smaller areas, i.e., event signals EVS corresponding to ROIs with high importance.

[0137] As described above, since the amount of the event signal that occurs, calculated in the event rate controller 130a, is filtered based on time, even when the amount of the event signal increases or decreases, the change in the amount of the event signal can be applied in a delayed manner when adjusting the output of the event signal and the amount of the event signal.

[0138] Figure 13 Examples of methods for setting the ROI of a vision sensor 100 according to some exemplary embodiments of the present invention are shown.

[0139] refer to Figure 13 ROI controller 137a (see Figure 12B ) can be used to create a pixel array 110 (see Figure 3 The ROI is categorized into multiple patches, each containing a separate set of one or more pixels from a plurality of pixels. The ROI controller 137a counts the event signals EVS corresponding to each patch (e.g., for each patch, counting the event signals EVS generated based on signals output from any pixel PX in that patch (e.g., events occurring at any pixel PX in that patch) as the amount of event signals EVS corresponding to that patch). When multiple events occurring in a particular patch (e.g., occurring at any pixel PX in that patch) at least satisfy a threshold, the ROI controller 137a can designate that patch or the region including that patch as an ROI.

[0140] For example, the first block P1 can be designated as an ROI when the difference between the event signal O(t) of the first block P1 counted at time point t and the event signal O1(t-1) of the first block P1 counted at a previous time point (e.g., time point t-1) is equal to or greater than a threshold set for distinguishing ROIs. The first block P1 can not be designated as an ROI when the difference between the event signal O(t) of the first block P1 counted at time point t and the event signal O(t-1) of the first block P1 counted at a previous time point (e.g., time point t-1) is less than the threshold set for distinguishing ROIs.

[0141] When the difference between the magnitude O(t) of the event signal O2 of the second block P2 counted at time point t and the magnitude O2(t-1) of the event signal of the second block counted at time point t-1, which is the previous time point, is equal to or greater than the threshold set for distinguishing ROIs, the second block P2 or a specific (or alternatively, predetermined) region that includes the second block P2 can be set as an ROI.

[0142] Therefore, the event rate controller 130 can count each individual quantity of the event signal EVS corresponding to each of the multiple blocks, and the event rate controller 130 can also identify the block corresponding to the quantity of the event signal EVS whose change over a period of time (e.g., the value of the count) is equal to or greater than a threshold as ROI.

[0143] Figure 14 A method for estimating the location of the ROI of a visual sensor 100 according to some example embodiments of the present invention is shown.

[0144] refer to Figure 14 The location of the ROI can be changed according to time (i.e., frames). Each frame can be an image comprising an array of pixels corresponding to an array of pixels PX in pixel array 110, wherein the signal (e.g., shadow) of each pixel in the frame corresponds to the signal (or no signal) output by the corresponding pixel PX of pixel array 110 during the time period corresponding to that frame. Visual sensor 100 (see...) Figure 3Motion estimation can be performed to determine the movement of the ROI, i.e., the movement of the ROI in the X and / or Y directions. For example, the vision sensor 100 can perform motion estimation for estimating the ROI ROI_F2 in the second frame based on the ROI ROI_F1 in the first frame. In some example embodiments, the vision sensor 100 can perform motion estimation based on the generated event occurrence signal. Therefore, the event rate controller 130 can perform motion estimation on the first frame based on the position of the ROI in the first frame ROI_F1, and can determine the movement of the ROI's position on the second frame ROI_F2 based on the result of the motion estimation. As another example, in an image processing device 10 (see image processing device 10), the motion estimation can be performed on the second frame ROI_F2. Figure 1 In electronic products, another sensor (e.g., a gyroscope sensor) installed adjacent to the vision sensor 100 can transmit position sensing signals to the vision sensor 100, and the vision sensor 100 can perform motion estimation based on the position sensing signals.

[0145] When the ROI moves within a specific (or alternatively, predetermined) range or further, the vision sensor 100 can reset the ROI. For example, the range RG can be set based on the position of the ROI_F1 in the first frame, and the vision sensor 100 can reset the ROI when it is determined that the estimated ROI_F2' in the second frame exceeds the range RG.

[0146] For example, the vision sensor 100 can notify the user that the ROI has moved out of a specific (or, alternatively, predetermined) range, allowing the user to reset the ROI. In some example embodiments, as described above, the vision sensor 100 can reset the ROI based on the amount of events that have occurred.

[0147] Figure 15A , Figure 15B and Figure 15C A method for adjusting the rate of event data output from the vision sensor 100 is shown.

[0148] refer to Figure 15A and Figure 15BThe vision sensor 100 outputs multiple frames FRM1 to FRM4 as event data to the processor 200, and blank sections BLK1 to BLK3 can be transmitted between the sections transmitting the multiple frames FRM1 to FRM4. When the amount of event signals EVS is excessive, i.e., when the measured amount of event signals EVS is equal to or greater than a reference value, the vision sensor 100 can increase the blank sections to reduce the amount of event signals transmitted. Therefore, the vision sensor 100 can transmit multiple frames FRM1 to FRM4 to the processor 200, and in response to operating in a second operating mode (where a limited selection of event signals in the multiple event signals EVS is output as one or more output event signals EVS_O, wherein the limited selection of event signals corresponds to one or more pixels PX included in the ROI of the pixel array 110), the vision sensor 100 can increase the length of the blank sections between the sections transmitting the multiple frames FRM1 to FRM4 to be longer than the length of the blank sections between the sections transmitting the multiple frames FRM1 to FRM4 when the vision sensor operates in the first operating mode. In other words, when the vision sensor 100 operates in the second operating mode, the vision sensor 100 can reduce the rate at which event data is transmitted (e.g., reduce the data transmission rate in the processor 200).

[0149] refer to Figure 15A When the amount of event signal EVS is less than a reference value, i.e., before the third frame FRM3, the first blank segment BLK1 and the second blank segment BLK2 can be set to T1. When it is determined that the amount of event signal EVS occurring in the third frame FRM3 is too large, the vision sensor 100 can set the third blank segment BLK3 to T2, which is longer than T1. Therefore, the amount of event signal EVS can be reduced.

[0150] refer to Figure 15B Blank segments can be classified into two types, for example, the first blank type BLK_1 and the second blank type BLK_2. Processor 200 (see...) Figure 1 Multiple frames received from the vision sensor 100 can be used in an overlapping manner. For example, the vision sensor 100 can merge multiple frames. The blanking segments between frames used in an overlapping manner can be set as a first blanking type BLK_1, and the blanking segments between non-overlapping frames can be set as a second blanking type BLK_2.

[0151] When the magnitude of the event signal EVS is less than the reference value, the time period of the second blank type BLK_2 can be set to be the same as or similar to the time period of the first blank type BLK_1. When the magnitude of the event signal EVS is equal to or greater than the reference value, the time period of the second blank type BLK_2 can be set to be relatively longer than the time period of the first blank type BLK1.

[0152] For example, frames FRM1 to FRM3 can overlap, and frames FRM4 to FRM6 can be used in an overlapping manner. The visual sensor 100 can set the blank areas between frames FRM1 to FRM3 and between frames FRM4 and FRM6 as a first blank type BLK_1; the visual sensor 100 can set the blank areas between frames FRM3 and FRM4 and between frames FRM6 and FRM7 as a second blank type BLK_2. The blank area between frames FRM3 and FRM4 is set as T1, and T1 can be the same as or similar to the time period of the first blank type BLK_1. Next, when it is determined that the amount of the event signal EVS is equal to or greater than a reference value, the visual sensor 100 can increase the time period of the second blank type BLK_2 without changing the time period of the first blank type BLK_1. As shown in the figure, the visual sensor 100 can set the second blank type BLK_2 (i.e., the time period of the blank segment between the sixth frame FRM6 and the seventh frame FRM7) to be relatively longer than T1, which is T2.

[0153] At the same time, it can be based on pixel array 110 ( Figure 6 Set the resolution accordingly. Figure 15A and Figure 15B The maximum length of blank segments in a transmission frame. For example, the maximum length of blank segments can be set to be between approximately 10% and 25% of the segments in the transmitted frame.

[0154] refer to Figure 15C Data can be transmitted in packet units, and when the amount of event signals EVS is excessive, the vision sensor 100 can reduce the amount of packet data transmitted during a specific (or, alternatively, predetermined) time period. For example, when the amount of event signals EVS is not excessive, the vision sensor 100 does not limit the amount of packets transmitted, as in the first time period T1 and the second time period T2. First packets PKD1 to third packets PKD3 can be transmitted during the first time period T1, and fourth packets PKD4 to seventh packets PKD7 can be transmitted during the second time period T2.

[0155] However, when it is determined that too many packets are transmitted during a time period (e.g., the third time period T3), the number of packets transmitted (or the number of packets that are wasted and not transmitted) can be adjusted. The vision sensor 100 can select the generated packets and output a limited number of packets. For example, the number of packets transmitted during the third time period T3 can be limited to three, and therefore, packets PKD8 through PKD10 can be transmitted during the third time period T3.

[0156] When the amount of packets transmitted is limited, the maximum amount of packets wasted can be set based on the resolution of the pixel array 110, for example, by setting it to a range of about 10% to about 25% of the amount of packet data generated during a specific (or, alternatively, predetermined) time period.

[0157] It will be understood that when the terms “about” or “substantially” are used in conjunction with numerical values ​​in this specification, it means that the associated numerical value includes a tolerance of ±10% around the stated value. When a range is specified, the range includes all values ​​within that range, such as increments of 0.1%.

[0158] Figure 16 This is a block diagram illustrating an image processing apparatus 10a according to an exemplary embodiment of the present invention.

[0159] refer to Figure 16 The image processing device 10a may include a first sensor (i.e., a vision sensor 100), a processor 200, and a second sensor 300. For example, when the image processing device 10a is operating, the second sensor 300 may detect the external environment, such as the illumination around an object or the brightness of the object, or the position of the image processing device 10a. For example, the second sensor 300 may be a gyroscope sensor or an illumination sensor.

[0160] The second sensor 300 can provide the sensed signal to the event rate controller 130 of the vision sensor 100 either through the processor 200 or directly. The vision sensor 100 can determine its operating mode based on the signal received from the second sensor 300. For example, the second sensor 300 can be an illuminance sensor, and illuminance information generated by the second sensor 300 can be provided to the event rate controller 130, the illuminance information being illuminance information indicating the illuminance associated with an object imaged by the vision sensor 100 or a region including the object. The event rate controller 130 can determine, based on the illuminance information, whether conditions indicating an expected excessive event have occurred. For example, when the received illuminance information indicates an illuminance equal to or greater than a reference illuminance, the event rate controller 130 can anticipate that the ambient light level is high and an excessive event may occur.

[0161] As described above, when an excessive number of events is anticipated, the event rate controller 130 can output an event signal EVS corresponding to the region of interest (ROI) to reduce the amount of event signal output. Additionally, the event rate controller 130 can change the event occurrence or detection conditions to reduce the number of events generated. In some example embodiments, the event rate controller 130 can set the event occurrence or detection conditions for the ROI to be different from those set for other regions.

[0162] The event rate controller 130 can change its operating mode differently based on the illuminance indicated by the illuminance information. For example, when the illuminance information indicates an illuminance equal to or greater than a reference illuminance, the event rate controller 130 can change the operating mode from a first operating mode (which is the default operating mode) to a second operating mode. An event signal EVS corresponding to the ROI can be output in the second operating mode. Furthermore, the event rate controller 130 can change the event occurrence conditions or detection conditions in the second operating mode to reduce the number of events occurring. Therefore, the vision sensor 100 can selectively operate in either the first or second operating mode based on whether the illuminance indicated by the illuminance information received from the illuminance sensor is equal to or greater than a reference illuminance.

[0163] Simultaneously, when the second sensor 300 is a gyroscope sensor, the event rate controller 130 can perform motion estimation based on the position signal provided from the gyroscope sensor, such as by reference. Figure 14 As described.

[0164] Figure 17 This is a flowchart illustrating the operation method of a vision sensor 100 according to some example embodiments of the present invention. Figure 17 The operation method can be found in Figure 3 This is executed in the vision sensor 100, and more specifically, in the event rate controller 130.

[0165] refer to Figure 3 and Figure 17 The visual sensor 100 can set the ROI (S310). The visual sensor 100 can receive information about the sensing environment (e.g., the surrounding environment) (S320). For example, as referenced... Figure 16 As described, the visual sensor 100 can receive ambient illuminance information from the illuminance sensor.

[0166] The vision sensor 100 can determine whether the sensing environment corresponds to the reference conditions based on the received sensing environment information (S330). The vision sensor 100 can determine whether the sensing environment is a situation where an excessive event is expected. For example, when the received illuminance information is equal to or greater than the reference illuminance, the vision sensor 100 can expect that the amount of light around the vision sensor 100 is high and an excessive event may occur.

[0167] When the sensed environment does not correspond to the reference conditions, for example, when the illuminance information is less than the reference illuminance, the vision sensor 100 can output an event signal EVS corresponding to the entire area (S340). When the sensed environment corresponds to the reference conditions, the vision sensor 100 can output an event signal EVS corresponding to the ROI (S350).

[0168] The vision sensor 100 can set an operating mode based on the sensing environment. For example, when the sensing environment does not correspond to the reference conditions, the vision sensor 100 can operate in a first operating mode, in which an event signal EVS corresponding to the entire area is output; and when the sensing environment corresponds to the reference conditions, the vision sensor 100 can operate in a second operating mode, in which an event signal corresponding to the ROI in the entire area can be output.

[0169] Figure 18 A flowchart illustrating the operation method of a vision sensor 100 according to some example embodiments of the present invention is shown. Figure 18 The operation method can be found in Figure 3 This is executed in the vision sensor 100, and more specifically, in the event rate controller 130.

[0170] refer to Figure 3 and Figure 18 The vision sensor 100 can set the ROI (S410). The vision sensor 100 can receive information about the sensing environment (e.g., the surrounding environment) (S420). The vision sensor 100 can determine whether the sensing environment corresponds to a reference condition based on the received sensing environment information (S430). For example, the reference condition is a condition in which an excessive event may occur. When the sensing environment does not correspond to the reference condition, the vision sensor 100 can output an event signal EVS corresponding to the entire area (S440).

[0171] When the sensing environment corresponds to the reference conditions, the vision sensor 100 can measure the amount of the event signal EVS that occurs (S450). For example, the vision sensor 100 compares the amount of the event signal EVS that occurs with a reference value (S460). When the amount of the event signal EVS that occurs is less than the reference value, the vision sensor 100 outputs the event signal EVS corresponding to the entire area (S440). When the amount of the event signal that occurs is equal to or greater than the reference value, the vision sensor 100 can output the event signal EVS corresponding to the ROI (S470).

[0172] As described above, when an excessive event signal EVS is anticipated, for example, when the ambient illumination is high or the object's brightness is high, the vision sensor 100 can measure the amount of the event signal EVS that has occurred, and when an event signal EVS equal to or greater than a reference value occurs, the vision sensor 100 outputs an event signal EVS corresponding to the ROI to reduce the amount of event data transmitted to the processor 200 (see [reference]). Figure 1 ).

[0173] Figure 19 This is a block diagram illustrating an example of an electronic device employing a vision sensor 100, based on some exemplary embodiments of the concept according to the present invention.

[0174] refer to Figure 19 The electronic device 1000 may include a vision sensor 1100, a main processor 1200, a working memory 1300, a storage device 1400, a display device 1500, a communication unit 1600, and a user interface 1700.

[0175] References can be used Figures 1 to 18 The described vision sensor 100 is referred to as vision sensor 1100. Vision sensor 1100 can sense an object to generate an event signal EVS and transmit the generated event signal EVS to a main processor 1200. When the sensed environment (e.g., the environment surrounding the object) corresponds to conditions where an excessive amount of event signal is expected to occur, or when the measured amount of the generated event signal EVS is equal to or greater than a reference value, vision sensor 1100 can selectively transmit the event signal corresponding to the ROI from the generated event signal to the main processor 1200. Therefore, the amount of transmitted event signal (i.e., the amount of event data) can be reduced or maintained at or below a certain level. Thus, data loss during the transmission of the event signal EVS can be reduced or prevented.

[0176] Additionally, when the amount of the event signal EVS is determined to be equal to or greater than a reference value, the event occurrence conditions and / or detection conditions can be adjusted to reduce the amount of the generated event signal EVS. In some example embodiments, the event occurrence conditions and / or detection conditions can be set differently in the ROI and in regions other than the ROI, and can also be set differently in multiple ROIs.

[0177] The main processor 1200 can control all operations of the electronic device 1000 and process event data (i.e. event signals EVS) received from the vision sensor 1100 to detect the movement of objects.

[0178] The working memory 1300 can store data used for the operation of the electronic device 1000. For example, the working memory 1300 can temporarily store packets or frames processed by the main processor 1200. For example, the working memory 1300 may include volatile memory (e.g., dynamic RAM (DRAM), synchronous RAM (SDRAM)) and non-volatile memory (e.g., phase-change RAM (PRAM), magnetoresistive RAM (MRAM), resistive RAM (ReRAM), and ferroelectric RAM (FRAM)).

[0179] Storage device 1400 can store data requested from main processor 1200 or other configurations. Storage device 1400 may include flash memory and non-volatile memory, such as PRAM, MRAM, ReRAM, and FRAM.

[0180] Display device 1500 may include a display panel, display driving circuitry, and a display serial interface (DSI). For example, the display panel may be implemented as various devices, such as a liquid crystal display (LCD), a light-emitting diode (LED) display, an organic LED (OLED) device, and an active-matrix OLED device. The display driving circuitry may include a timing controller, source drivers, etc., for driving the display panel. The DSI host embedded in the main processor 1200 can perform serial communication with the display panel via the DSI.

[0181] Communication unit 1600 can exchange signals with external devices / systems via antenna 1630. Transceiver 1610 and modem 1620 (modulator / demodulator) can process the signals exchanged between communication unit 1600 and external devices / systems according to wireless communication protocols, such as LTE, WiMAX (Microwave Access Global Interoperability), GSM (Global System for Mobile Communications), CDMA (Code Division Multiple Access), Bluetooth, NFC (Near Field Communication), Wi-Fi (Wireless Fidelity), and RFID (Radio Frequency Identification).

[0182] User interface 1700 may include at least one of the following input interfaces: keyboard, mouse, keypad, button, touch panel, touch screen, touchpad, touch ball, gyroscope sensor, vibration sensor, accelerometer, etc.

[0183] Components of the electronic device 1000 (e.g., vision sensor 1100, main processor 1200, working memory 1300, storage device 1400, display device 1500, communication unit 1600, and user interface 1700) can exchange data based on one or more of various interface protocols such as: Universal Serial Bus (USB), Small Computer System Interface (SCSI), MIPI, I2C, Peripheral Component Interconnect Express (PCIe), Mobile PCIe (M-PCIe), Advanced Technology Attachment (ATA), Parallel ATA (PATA), Serial ATA (SATA), Serial Attached SCSI (SAS), Integrated Drive Electronics (IDE), Enhanced IDE (EIDE), Non-Volatile Memory Express (NVMe), Universal Flash Memory (UFS), etc.

[0184] Although the inventive concept has been specifically shown and described with reference to embodiments thereof, it will be understood that various changes in form and detail may be made therein without departing from the spirit and scope of the appended claims.

Claims

1. A dynamic vision sensor, comprising: A pixel array comprising a plurality of pixels arranged in a matrix, each pixel being configured to generate one or more electrical signals in response to a change in the intensity of incident light; An event detection circuit is configured to receive electrical signals from the pixel array and generate a first event signal; An event rate controller is configured to transmit a portion of the first event signal when the magnitude of the first event signal is equal to or greater than a reference value; as well as The interface circuit is configured to receive a portion of the first event signal and transmit that portion of the first event signal as a packet unit or frame unit to an external device. The event rate controller or the event detection circuit is configured to transmit a second event signal corresponding to a first limited selection of pixels in the pixel array as part of the first event signal, wherein the first limited selection of pixels is included in a first region of interest in the pixel array.

2. The dynamic vision sensor according to claim 1, wherein, The grouping unit or the frame unit includes polarity information of at least one event signal in a portion of the first event signal, and the row address of at least one pixel in the pixel array that generates the electrical signal.

3. The dynamic vision sensor according to claim 1, wherein, The event rate controller or the event detection circuit is configured to transmit a third event signal corresponding to a second finitely selected pixel of the pixel array as part of the first event signal, wherein the second finitely selected pixel is included in a second region of interest of the pixel array.

4. The dynamic vision sensor according to claim 2, wherein, The first region of interest of the pixel array includes M rows and N columns of the pixel array, and Where M and N are integers.

5. The dynamic vision sensor according to claim 2, wherein, The dynamic vision sensor is configured to adjust event occurrence conditions or event detection conditions.

6. The dynamic vision sensor according to claim 5, wherein, The event detection conditions include noise reduction conditions.

7. The dynamic vision sensor according to claim 2, wherein, The event rate controller is configured to transmit a portion of the first event signal when the amount of the first event signal generated within a predetermined time period is equal to or greater than the reference value.

8. The dynamic vision sensor according to claim 7, wherein, The row address of at least one pixel in the pixel array comprises 8 bits.

9. The dynamic vision sensor according to claim 2, wherein, Each of the plurality of pixels includes a photoelectric conversion element, a boost circuit configured to be connected to the photoelectric conversion element, and an amplifying transistor. The gate of the amplifying transistor is connected to the photoelectric conversion element.

10. The dynamic vision sensor according to claim 9, wherein, The first node of the amplifying transistor is configured to receive ground voltage.

11. The dynamic vision sensor according to claim 9, wherein, The boost circuit includes a first transistor and a second transistor, and The first node of the first transistor is configured to connect to the photoelectric conversion element.

12. The dynamic vision sensor according to claim 11, wherein, Each of the plurality of pixels further includes a third transistor configured to be connected to the boost circuit, and The gate of the third transistor is configured to receive the output voltage.

13. A dynamic vision sensor, comprising: A pixel array comprising a plurality of pixels arranged in a matrix, each pixel being configured to generate one or more electrical signals in response to a change in the intensity of incident light; An event detection circuit is configured to receive electrical signals from the pixel array and generate a first event signal; as well as An event rate controller is configured to transmit a portion of the first event signal when the magnitude of the first event signal is equal to or greater than a reference value; The event rate controller or the event detection circuit is configured to transmit a second event signal corresponding to a first limited selection of pixels in the pixel array as part of the first event signal, wherein the first limited selection of pixels is included in a first region of interest in the pixel array.

14. The dynamic vision sensor according to claim 13, wherein, The event detection circuit includes: A row address event indicates that a row request indicative of an event has occurred is received from at least one pixel in the pixel array that generates the electrical signal, and a row address is generated for the at least one pixel; and A column address event indicates that a column request indicating the occurrence of the event is received from at least one pixel in the pixel array, wherein the electrical signal is generated, and a column address of the at least one pixel is generated.

15. The dynamic vision sensor according to claim 14, wherein, The event detection circuit also includes: An event signal processing unit is configured to receive the electrical signal and generate the first event signal.

16. The dynamic vision sensor according to claim 15, wherein, The event signal processing unit is configured to remove noise events and generate the first event signal.

17. The dynamic vision sensor according to claim 15, wherein, The dynamic vision sensor is configured to adjust event occurrence conditions or event detection conditions.

18. The dynamic vision sensor according to claim 15, wherein, The event detection conditions include noise reduction conditions.

19. The dynamic vision sensor according to claim 18, wherein, Each of the plurality of pixels includes a photoelectric conversion element, a boost circuit configured to be connected to the photoelectric conversion element, and an amplifying transistor. The gate of the amplifying transistor is connected to the photoelectric conversion element.

20. The dynamic vision sensor according to claim 19, wherein, The boost circuit includes multiple transistors.

Images