Visual sensor, image processing device, and method of operating a visual sensor

Abstract

Provided is a vision sensor, comprising: a pixel array including a plurality of pixels arranged in a matrix form; an event detection circuit configured to detect whether an event has occurred in the plurality of pixels and generate an event signal corresponding to the pixels in which the event has occurred among the plurality of pixels; a graph data processor configured to generate a timestamp graph based on the event signal; and an interface circuit configured to send vision sensor data including at least one of the event signal and the timestamp graph to an external processor, wherein the timestamp graph includes timestamp information indicating the polarity information, address information, and event occurrence time of the pixel included in the event signal corresponding to the pixel.

Description

[0001] Cross - reference to related applications

[0002] This application is based on and claims priority to Korean Patent Application No. 10 - 2020 - 0165943, filed with the Korean Intellectual Property Office on December 1, 2020, the disclosure of which is incorporated herein by reference in its entirety. Technical field

[0003] Example embodiments of the present disclosure relate to a vision sensor, and more particularly, to a vision sensor capable of generating and transmitting a timestamp map, an image processing apparatus including the vision sensor, and an operation method of the vision sensor. Background art

[0004] A vision sensor (e.g., a dynamic vision sensor) generates information related to an event (i.e., an event signal), such as a change in the intensity of light, when an event occurs, and transmits the event signal to a processor.

[0005] According to related art, a dynamic vision sensor only transmits event data and timestamps, and mainly uses a processor located outside the dynamic vision sensor to perform data processing (such as writing a timestamp map by using the data). The operation of writing a timestamp map by using raw data consumes resources of the host, and thus, a method for reducing the burden on the host needs to be studied. Summary of the invention

[0006] One or more example embodiments provide a vision sensor that generates a timestamp map and an optical flow map based on event data and timestamps, an image processing apparatus including the vision sensor, and an operation method of the vision sensor.

[0007] According to an aspect of an example embodiment, there is provided a vision sensor including: a pixel array including a plurality of pixels arranged in a matrix form; an event detection circuit configured to detect whether an event has occurred in the plurality of pixels and generate an event signal corresponding to the pixels in which the event has occurred among the plurality of pixels; a map data processor configured to generate a timestamp map based on the event signal; and an interface circuit configured to transmit vision sensor data including at least one of the event signal and the timestamp map to an external processor, wherein the timestamp map includes timestamp information indicating polarity information, address information, and event occurrence time of a pixel included in the event signal corresponding to the pixel.

[0008] According to another aspect of the exemplary embodiment, there is provided an image processing apparatus including: a vision sensor configured to generate a plurality of event signals corresponding to pixels among the pixels included in a pixel array in which an event has occurred based on the movement of an object, generate a timestamp map based on polarity information, address information, and event occurrence time information of the pixels included in the plurality of event signals, and output vision sensor data including the plurality of event signals and the timestamp map; and a processor configured to detect the movement of the object by processing the vision sensor data output from the vision sensor.

[0009] According to one aspect of the exemplary embodiment, there is provided an operation method of a vision sensor, the operation method including: detecting whether an event has occurred in a plurality of pixels and generating event signals corresponding to the pixels in which the event has occurred among the plurality of pixels; generating vision sensor data including a timestamp map based on the event signals; and sending the vision sensor data to an external processor, wherein the timestamp map includes timestamp information indicating polarity information, address information, and event occurrence time of the pixels included in the event signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and / or other aspects, features, and advantages of the exemplary embodiment will be more clearly understood through the following detailed description in conjunction with the accompanying drawings, in which:

[0011] Figure 1 is a block diagram showing an image processing apparatus according to an exemplary embodiment;

[0012] Figure 2 is a block diagram showing a vision sensor according to an exemplary embodiment;

[0013] Figure 3 is a block diagram showing an event detection circuit according to an exemplary embodiment;

[0014] Figure 4A and Figure 4B is a block diagram showing a graph data module according to an exemplary embodiment;

[0015] Figure 5 is a flowchart of an operation method of a vision sensor according to an exemplary embodiment;

[0016] Figure 6 is a circuit diagram showing an implementation example of a pixel;

[0017] Figure 7A and Figure 7B is a diagram showing a method of sending data by a vision sensor according to an exemplary embodiment via an interface circuit;

[0018] Figure 8A 、 Figure 8B andFigure 8C is a diagram showing event merging of a vision sensor according to an exemplary embodiment;

[0019] Figure 9 is a diagram for describing a method of generating a plurality of timestamp maps by using a vision sensor according to an exemplary embodiment;

[0020] Figure 10 is a diagram for describing a timestamp map generator in a vision sensor according to an exemplary embodiment;

[0021] Figure 11 is a diagram for describing a method of reducing data of a timestamp map by a vision sensor according to an exemplary embodiment;

[0022] Figure 12 is a diagram for describing the time of generating a timestamp map and the time of generating an optical flow map in a vision sensor according to an exemplary embodiment;

[0023] Figure 13A and Figure 13B is a diagram for describing generating an optical flow map in a vision sensor according to an exemplary embodiment; and

[0024] Figure 14 is a block diagram showing an example of an electronic device to which a vision sensor according to an exemplary embodiment is applied. Detailed Description

[0025] Hereinafter, various exemplary embodiments will be described with reference to the accompanying drawings.

[0026] Figure 1 is a block diagram showing an image processing device according to an exemplary embodiment.

[0027] An image processing device 10 according to an exemplary embodiment may be installed in an electronic device having an image or light sensing function. For example, the image processing device 10 may be installed in an electronic device such as: a camera, a smart phone, a wearable device, the Internet of Things (IoT), a tablet personal computer (PC), a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, a drone, an advanced driver assistance system (ADAS), etc. Moreover, the image processing device 10 may be included as a component of a vehicle, furniture, manufacturing equipment, a door, various measuring instruments, etc.

[0028] Referring Figure 1 , the image processing device 10 may include a vision sensor 100 and a processor 200. The vision sensor 100 detects a change in the intensity of incident light and sends vision sensor data VSD including at least one of an event signal EVS, a timestamp map TSM, and an optical flow map OFM to the processor 200.

[0029] The vision sensor 100 can detect changes in the intensity of incident light and output an event signal. The vision sensor 100 can be a dynamic vision sensor that outputs an event signal EVS for pixels in which a change in light intensity is detected (e.g., pixels for which an event has occurred). The change in light intensity may be due to the movement of an object whose image is captured using the vision sensor 100, or due to the movement of the vision sensor 100 or the image processing device 10. The vision sensor 100 can send the event signal EVS to the processor 200 periodically or aperiodically. The vision sensor 100 can send not only the event signal EVS including an address, polarity, and timestamp to the processor 200, but also a timestamp map TSM or an optical flow map OFM generated based on the event signal EVS to the processor 200.

[0030] The vision sensor 100 can selectively send the event signal EVS to the processor 200. The vision sensor 100 can send these event signals EVS that are generated from pixels PX corresponding to a region of interest (ROI) set in the pixel array among the event signals generated by the pixel array 110.

[0031] According to an exemplary embodiment, the vision sensor 100 can apply cropping or event merging to the event signal EVS. Moreover, the vision sensor 100 can generate a timestamp map TSM or an optical flow map OFM based on the event signal EVS. According to an exemplary embodiment, the vision sensor 100 can selectively send the event signal EVS, the timestamp map TSM, or the optical flow map OFM to the outside according to a sending mode.

[0032] The processor 200 can process the event signal EVS received from the vision sensor 100, detect the movement of an object recognized by the image processing device 10 or the movement of an object on an image, and can use an algorithm such as simultaneous localization and mapping (SLAM). The processor 200 can include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), an application-specific microprocessor, a microprocessor, a general-purpose processor, etc. According to an exemplary embodiment, the processor 200 can include an application processor or an image processor.

[0033] In addition, the vision sensor 100 and the processor 200 can each be implemented as an integrated circuit (IC). For example, the vision sensor 100 and the processor 200 can be implemented as separate semiconductor chips, or can be implemented as a single chip. For example, the vision sensor 100 and the processor 200 can be implemented as a system-on-chip (SoC).

[0034] Figure 2 is a block diagram showing a vision sensor according to an exemplary embodiment.

[0035] refer to Figure 2 The visual sensor 100 may include a pixel array 110 , an event detection circuit 120 , a map data module 130 , and an interface circuit 140 .

[0036] The pixel array 110 may include a plurality of pixels PX arranged in a matrix form. Each of the pixels PX may detect an event, such as an increase or decrease in the intensity of received light. For example, each of the pixels PX may be connected to the event detection circuit 120 via a column line extending in a column direction and a row line extending in a row direction. A signal indicating that an event has occurred and polarity information of the event (e.g., whether the event is a stronger event when the light intensity increases or a weaker event when the light intensity decreases) may be output from the pixel PX where the event has occurred to the event detection circuit 120.

[0037] The event detection circuit 120 can read events from the pixel array 110 and process the events. The event detection circuit 120 can generate an event signal EVS, which includes polarity information of the event that has occurred, the address of the pixel where the event has occurred, and a timestamp. The event signal EVS is generated in various formats, for example, an address-event representation (AER) format including address information of the pixel where the event has occurred, timestamp information, and polarity information, or a raw format including event occurrence information of all pixels.

[0038] The event detection circuit 120 may process events occurring in the pixel array 110 in units of pixels, in units of pixel groups including a plurality of pixels, in units of columns, or in units of frames.

[0039] The graph data module 130 may adjust the number of event signals EVS by post-processing such as cropping, which refers to setting an ROI in a pixel array or event merging for setting a data amount relative to the event signal EVS, and adjusting the size of the generated timestamp map TSM and optical flow map OFM. According to an example embodiment, the graph data module 130 may generate graph data MDT including a timestamp map or an optical flow map, and output the graph data MDT to the interface circuit 140 or output the event signal EVS received from the event detection circuit 120 to the interface circuit 140.

[0040] The interface circuit 140 can receive the event signal EVS and each graph data MDT according to the set protocol, and send the event signal EVS and each graph data MDT to the processor 200 ( Figure 1)Send visual sensor data VSD. The interface circuit 140 can pack the event signal EVS and each graph data MDT into respective signal units, packet units, or frame units according to the set protocol to generate the visual sensor data VSD, and send the visual sensor data VSD to the processor 200. For example, the interface circuit 140 can include a Mobile Industry Processor Interface (MIPI).

[0041] Hereinafter, the event signal EVS or each graph data MDT output via the interface circuit 140 can be the event signal EVS or each graph data MDT that has been converted into the visual sensor data VSD, and the visual sensor data VSD is sent to the processor 200.

[0042] Figure 3 is a block diagram showing an event detection circuit according to an exemplary embodiment.

[0043] Reference Figure 3 , the event detection circuit 120 can include a voltage generator 121, a digital timing / AER generator (DTAG) 122, and an event signal processing (ESP) unit 123. The DTAG 122 can include a column AER generator and a row AER generator.

[0044] The voltage generator 121 can generate a voltage supplied to the pixel array 110. For example, the voltage generator 121 can generate a threshold voltage for detecting a strengthening event or a weakening event from a pixel PX ( Figure 2 ), or a bias voltage. The voltage generator 121 can change the voltage level of the threshold voltage supplied to the pixels of the ROI, and can change the voltage level of the threshold voltage differently for a plurality of ROIs.

[0045] The DTAG 122 can receive a signal indicating that an event has occurred from a pixel PX where an event has occurred, and generate a timestamp TS including information related to the time of the event that has occurred in the pixel PX and an address ADDR including a column address, a row address, or a group address.

[0046] For example, the column AER generator can receive a signal indicating that an event has occurred (e.g., a column request) from a pixel PX where an event has occurred, and generate a column address C_ADDR of the pixel PX where the event has occurred.

[0047] The row AER generator can receive a signal indicating that an event has occurred (e.g., a row request) from a pixel PX where an event has occurred, and generate a row address R_ADDR of the pixel PX where the event has occurred. Instead of the row AER generator that generates a row address, a group address G_ADDR can also be generated in units of preset groups.

[0048] According to an example embodiment, the pixel array 110 may be scanned column by column, and when a request is received from a specific column (e.g., the first column), the column AER generator may send a response signal to the first column. The pixel PX in which an event has occurred and has received the response signal may send the polarity information Pol (e.g., a signal indicating that a strengthening event or a weakening event has occurred) to the row AER generator. When receiving the polarity information Pol, the row AER generator may send a reset signal to the pixel PX in which an event has occurred. In response to the reset signal, the pixel PX in which an event has occurred may be reset. The row AER generator may control the period during which the reset signal is generated. The row AER generator may generate information related to the time when an event has occurred (i.e., the timestamp TS).

[0049] The operations of the row AER generator and the column AER generator have been described above by assuming that the pixel array 110 is scanned column by column. However, the operations of the row AER generator and the column AER generator are not limited thereto, and the row AER generator and the column AER generator may read whether an event has occurred and the polarity information Pol from the pixel PX in which an event has occurred in various ways. For example, the pixel array 110 may be scanned row by row, and the operations of the row AER generator and the column AER generator may be exchanged with each other, i.e., the column AER generator may receive the polarity information Pol and send a reset signal to the pixel array 110. Moreover, the row AER generator and the column AER generator may individually access the pixel PX in which an event has occurred.

[0050] The ESP unit 123 may generate an event signal EVS based on the column address C_ADDR, row address R_ADDR, group address G_ADDR, polarity information Pol, and timestamp TS received from the DTAG 122. According to an example embodiment, the ESP unit 123 may remove noise events and generate an event signal EVS regarding valid events. For example, when the number of events occurring within a specific time period is less than a threshold, the ESP unit 123 may determine the event as noise and may not generate an event signal EVS regarding the noise event.

[0051] Figure 4A and Figure 4B is a block diagram showing a graph data module according to an example embodiment.

[0052] Reference Figure 4A , the graph data module 130 may include a cropping unit 131, a merging unit 132, a timestamp graph generator 133, and a graph data buffer 135.

[0053] The cropping unit 131 may set at least one ROI on a pixel array including a plurality of pixels according to user settings or a preset algorithm. For example, the cropping unit 131 may set, as the ROI, an area where a plurality of events occur among the plurality of areas set for the pixel array 110, or may set, as the ROI, any area corresponding to pixels where a plurality of events occur within a specific time period. According to another exemplary embodiment, an area arbitrarily set by the user or any area corresponding to the pixel PX from which a specific object is sensed may be set as the ROI. However, the ROI is not limited thereto, and may be set in various ways.

[0054] The merging unit 132 may adjust the amount of data used when generating a graph according to user settings or a preset algorithm. For example, the merging unit 132 may perform a merging operation on the ROI by grouping the pixels included in the set ROI in specific units.

[0055] The timestamp graph generator 133 may generate a timestamp graph TSM based on the event signal EVS adjusted using the cropping unit 131 and the merging unit 132. For example, the timestamp graph generator 133 may generate only the timestamp graph TSM for the ROI specified by cropping. Moreover, the timestamp graph generator 133 may generate a timestamp graph TSM having a size adjusted by merging.

[0056] The graph data buffer 135 may store the timestamp graph TSM generated using the timestamp graph generator 133, and send the stored timestamp graph TSM to the interface circuit 140 in the vision sensor 100 as graph data MDT.

[0057] Reference Figure 4B , the graph data module 130 may include a cropping unit 131, a merging unit 132, a timestamp graph generator 133, an optical flow graph generator 134, and a graph data buffer 135. The optical flow graph generator 134 may generate an optical flow graph OFM based on the timestamp graph TSM stored in the graph data buffer 135. The optical flow graph OFM may be stored in the graph data buffer 135 that stores the timestamp graph TSM.

[0058] Optical flow may be a pattern of apparent movement of objects, surfaces, and edges in a visual scene caused by the relative movement between an observer and the scene, and may be a distribution of the apparent speed of movement of the luminance pattern in an image. According to an exemplary embodiment, optical flow may be information related to the direction and speed of pixel changes for which an event signal has been generated.

[0059] Figure 5 is a flowchart of an operation method of a vision sensor according to an exemplary embodiment.

[0060] Reference Figure 2 andFigure 5 The event detection circuit 120 can detect whether an event occurs in each of the multiple pixels and generate an event signal EVS corresponding to the pixels in which the event has occurred. The event detection circuit 120 can generate the event signal EVS in the AER format including the address information, timestamp information, and polarity information of the pixels in which the event has occurred, or in the raw format including the event occurrence information of all pixels.

[0061] The map data module 130 can receive the event signal EVS from the event detection circuit 120 (S110).

[0062] The map data module 130 can perform cropping of the event signal generated in the ROI of the pixel array selected from the event signals according to the user's settings or a preset algorithm (S120).

[0063] The map data module 130 can perform event merging (S130). When the number of events occurring in the pixels of the preset N*N size is greater than the preset threshold, the map data module 130 can determine that an event has occurred and store the event signal EVS corresponding to the pixels of the N*N size on the array for storing the timestamp map.

[0064] The map data module 130 can generate a timestamp map TSM (S140). The map data module 130 can generate multiple timestamp maps TSM with different reference frames from each other in the event signal frame period generated periodically.

[0065] For example, the map data module 130 can generate a first timestamp reference and a first timestamp map based on a preset number of event signal frames counted from the first timestamp reference periodically, and generate a second timestamp reference and a second timestamp map based on a preset number of event signal frames counted from the second timestamp reference periodically. For example, the first timestamp map can be generated by setting the first timestamp reference frame as the first event signal frame and accumulating to the eleventh event signal frame, and the second timestamp map can be generated by setting the second timestamp reference frame as the fifth event signal frame and accumulating from the fifth event signal frame to the fifteenth event signal frame. Three or more timestamp maps TSM can also be generated according to the occurrence of the event signal EVS or the user's settings.

[0066] The graph data module 130 may store the offset value of the timestamp based on a preset timestamp reference of the timestamp graph TSM, and set the timestamp reference of the timestamp graph TSM based on the event occurrence information. When later sending the timestamp graph TSM to another device, the graph data module 130 may send the original data by adding the timestamp reference value corresponding to the timestamp graph TSM. For example, when the event generation time is r + k, the timestamp reference may be set to r, and the offset value of the timestamp may be stored as k. When sending the timestamp graph to another device after the timestamp graph is generated, the value of r + k obtained by adding the offset value k to the timestamp reference r may be sent.

[0067] The graph data module 130 may generate an optical flow map OFM by estimating an optical flow including information related to the direction and speed of pixel changes that generated an event signal based on the timestamp graph TSM according to a preset algorithm. The graph data module 130 may store the optical flow map OFM in the position in the graph data buffer 135 where the timestamp graph TSM is stored, instead of the timestamp graph TSM. Therefore, the graph data buffer size can be controlled more effectively.

[0068] The interface circuit 140 may send visual sensor data VSD (S150) including the event signal EVS, the timestamp graph TSM, or the optical flow map OFM to an external processor. The interface circuit 140 may selectively send each data according to the output mode.

[0069] For example, in the first output mode, the interface circuit 140 may output only the event signal included in the visual sensor data VSD to the external processor. In the second output mode, the interface circuit 140 may generate at least one virtual channel, send the event signal to the external processor via the first virtual channel, and send the timestamp graph data to the external processor via the second virtual channel. In the third output mode, the interface circuit 140 may generate at least one virtual channel, send the event signal to the external processor via the first virtual channel, and send the optical flow map data to the external processor via the second virtual channel.

[0070] Figure 6 is a circuit diagram showing an implementation example of a pixel.

[0071] Reference Figure 6 , the pixel PX may include a photoelectric conversion device PD, an amplifier 111, a first comparator 112, a second comparator 113, a strengthening event holder 114, a weakening event holder 115, and a reset switch SW. The pixel PX may further include a capacitor for removing noise generated in the pixel PX or from the outside or each switch.

[0072] The photoelectric conversion device PD can convert incident light (i.e., optical signal) into an electrical signal (e.g., current). The photoelectric conversion device PD can include, for example, a photodiode, a phototransistor, a port gate, a pinned photodiode, etc. The photoelectric conversion device PD can generate an electrical signal with a higher level as the intensity of the incident light increases.

[0073] The amplifier 111 may convert the received current into a voltage and amplify the voltage level. The output voltage of the amplifier 111 may be provided to the first comparator 112 and the second comparator 113.

[0074] The first comparator 112 may compare the output voltage Vout of the amplifier 111 with the change-intensity threshold voltage TH1, and generate a change-intensity signal E_ON based on the comparison result. The second comparator 113 may compare the output voltage Vout of the amplifier 111 with the change-intensity threshold voltage TH2, and generate a change-intensity signal E_OFF based on the comparison result. When the amount of change in light received by the photoelectric conversion device PD is equal to or greater than a specific change level, the first comparator 112 and the second comparator 113 may generate the change-intensity signal E_ON or the change-intensity signal E_OFF.

[0075] For example, when the amount of light received by the photoelectric conversion device PD increases to a specific level or more, the intensity signal E_ON may be at a high level; and when the amount of light received by the photoelectric conversion device PD decreases to a specific level or less, the weakening signal E_OFF may be at a high level. The intensity event holder 114 and the weakening event holder 115 may respectively hold the intensity signal E_ON and the weakening signal E_OFF, and then output them. When scanning the pixel PX, the intensity signal E_ON and the weakening signal E_OFF may be output. As described above, when adjusting the light sensitivity, the level of the intensity threshold voltage TH1 and the weakening threshold voltage TH2 may be modified. For example, the light sensitivity may be reduced. Therefore, the level of the intensity threshold voltage TH1 may be increased, and the level of the weakening threshold voltage TH2 may be reduced. Therefore, when the change in light received by the photoelectric conversion device PD is greater than before modifying the level of the intensity threshold voltage TH1 and the weakening threshold voltage TH2, the first comparator 112 and the second comparator 113 may generate the intensity signal E_ON or the weakening signal E_OFF.

[0076] Figure 7A and Figure 7B is a diagram illustrating a method of transmitting data via an interface circuit by a vision sensor according to example embodiments.

[0077] refer to Figure 7A, event data can be generated based on, for example, an event signal EVS generated based on 2 bits / pixel, 1000 fps, and a pixel array size (a×b). A packet including at least one event signal EVS can be output as event data from the interface circuit 140. The packet can include a timestamp, column address, row address, and polarity information of the event signal, and the order of arrangement thereof is not limited. A header indicating the start of the packet can be added to the front end of the packet, and a tail indicating the end of the packet can be added to the back end of the packet. The packet can include at least one event signal.

[0078] The timestamp can include information related to the time when the event has occurred. For example, the timestamp can include 32 bits, but is not limited thereto.

[0079] The column address and the row address can each include a plurality of bits (e.g., 8 bits). In this case, a vision sensor including a plurality of pixels arranged in up to eight rows and up to eight columns can be supported. However, this is an example, and the number of bits of the column address and the row address can be changed according to the number of pixels.

[0080] The polarity information can include information related to a strengthening event and a weakening event. For example, the polarity information can include: 1 bit including information related to whether a strengthening event has occurred, and 1 bit including information related to whether a weakening event has occurred. For example, the bit indicating the strengthening event and the bit indicating the weakening event may not both be '1', but may both be '0'.

[0081] The frame data MDT includes more information in each frame than the event signal EVS. Therefore, the size and generation period of the frame data MDT can be adjusted according to the user's settings. For example, the frame data MDT can be generated based on 16 bits / pixel and 50 fps. The size (c×d) of the frame data MDT can be adjusted by cropping and event merging settings.

[0082] Reference Figure 7B , according to an exemplary embodiment, the interface circuit 140 can be implemented as a MIPI interface, and a D-PHY interface, for example, used as an interface between a camera and a display, can be used. Here, by using a plurality of virtual channels VC, the frame data MDT can be transmitted simultaneously with the periodic transmission of the event data EDT. For example, as Figure 7B shown, while the event data EDT is periodically transmitted via the first virtual channel VC1, the frame data MDT can be transmitted via the second virtual channel VC2. By appropriately setting the size and transmission period of the frame data MDT, the frame data MDT can also be additionally transmitted without affecting the transmission of the event data EDT.

[0083] Figures 8A to 8CIt is a diagram showing event merging of a vision sensor according to an exemplary embodiment.

[0084] Merging refers to a data preprocessing technique that divides the original data into smaller parts (bins) and replaces the value of that part with, for example, the median. In the present disclosure, according to event merging, when the number of event signals generated on a pixel array of a preset N×N size is greater than a threshold, it is determined that an event has occurred. Refer to Figure 8A , each 4×4 size period of the pixel array can be set as a merging region, and when two or more events have occurred in the merging region (con), it is determined that an event has occurred in the merging region.

[0085] Refer to Figure 8B , it shows the events that have occurred in an 8×8 size pixel array. n represents the pixels where a weakening event has occurred, and p represents the pixels where a strengthening event has occurred. For example, it can be determined that a weakening event has occurred in the pixels in the first merging region b1, a strengthening event has occurred in four pixels in the second merging region b2, a weakening event has occurred in three pixels in the third merging region b3, and a strengthening event has occurred in one pixel in the fourth merging region b4.

[0086] Refer to Figure 8C , the data obtained after applying event merging may include information indicating that no event has occurred in the first merging region b1, a strengthening event has occurred in the second merging region b2, a weakening event has occurred in the third merging region b3, and no event has occurred in the fourth merging region b4. By applying event merging to the event signals generated in an 8×8 size pixel array in 4×4 size, data of 2×2 size can be obtained.

[0087] Figure 9 It is a diagram for describing a method of generating multiple timestamp maps by using a vision sensor according to an exemplary embodiment.

[0088] When the vision sensor 100 generates a timestamp map TSM, the number of event signal frames or the time period can be set. Refer to Figure 9 , timestamp map data is generated for every twelve event signal frames. The event signal here can be in AER format.

[0089] The vision sensor 100 can generate multiple timestamp maps TSM, for example, the first timestamp map to the third timestamp map. In this case, the periods of the frames used to generate the first timestamp map to the third timestamp map can overlap with each other. Multiple timestamp maps TSM can be generated based on the same event signal frames, but can be different in the timestamp reference which is the timestamp generation period.

[0090] For example, when generating three timestamp maps TSM, the first timestamp map can be generated as follows: Timestamp map data 1 TSM1 is generated by accumulating the nth to (n + 11)th event signal frames (a total of 12 event signal frames with respect to the first timestamp reference frame TSR1) included in the first map accumulation time window (Map1 accumulation time window). The second timestamp map can be generated as follows: Timestamp map data 2 TSM2 is generated by accumulating a total of 12 event signal frames with respect to the second timestamp reference frame TSR2 starting from the (n + 4)th event signal frame in the second map accumulation time window (Map2 accumulation time window). The third timestamp map can be generated as follows: Timestamp map data 3 TSM3 is generated by accumulating a total of 12 event signal frames with respect to the third timestamp reference frame TSR3 starting from the (n + 8)th event signal frame included in the third map accumulation time window (Map3 accumulation time window). In the timestamp maps, the timestamp references used as references are different, but the event signal frames used when generating the timestamp maps can overlap with each other.

[0091] Figure 10 is a diagram for describing a timestamp map generator in a vision sensor according to an exemplary embodiment.

[0092] Reference Figure 10 , the timestamp map generator 133 may include an address generator 133a and an offset calculator 133b. When an event is sensed in the pixel array 110 and an event signal EVS is generated, the address generator 133a may determine an address WA for recording information including the event signal EVS and the timestamp TS on the timestamp map TSM based on the column address C_ADDR, the group address G_ADDR, the positive event PE, and / or the negative event NE. The offset calculator 133b generates the timestamp map TSM by using a frame counter FC indicating the order of the timestamp TS, the event signal frames that have occurred, and the timestamp reference TSR. Reference Figure 11 The operation of the offset calculator 133b will be described in detail.

[0093] Figure 11 is a diagram for describing a method of reducing data of a timestamp map by a vision sensor according to an exemplary embodiment.

[0094] Reference Figure 10 and Figure 11, the offset calculator 133b can generate a timestamp map TSM including an offset value, which is the difference between the reference time Ref and the timestamp. For example, when the reference time Ref is set, only the offset value with respect to the reference time Ref of the generated event signal EVS is stored in the timestamp map, and then when the timestamp map data is sent, the reference time Ref can be added to the offset value for transmission. Therefore, the storage efficiency of the map data buffer 135, which is a limited storage space, can be improved. The offset calculator 133b can set the reference time Ref based on the information related to the generation of the event signal EVS.

[0095] Figure 12 is a diagram for describing the time for generating a timestamp map and the time for generating an optical flow map in a vision sensor according to an exemplary embodiment.

[0096] When the map generation period is set, a timestamp map or an optical flow map can be generated for each time signal frame in a preset number of event signal frames. For example, referring to Figure 12 , a timestamp map can be generated immediately after eight event signal frames are generated, and an optical flow map can be generated based on the timestamp map after the timestamp map is generated. For example, an optical flow map can be generated immediately after the timestamp map is generated and before the ninth signal frame is generated. The timestamp map and the optical flow map can be periodically generated every time eight event signal frames are generated. The event signal here can be in AER format.

[0097] Figure 13A and Figure 13B are diagrams for describing the generation of an optical flow map in a vision sensor according to an exemplary embodiment.

[0098] For example, in order to generate an optical flow map, a 5x5 size digital filter (z) with appropriate filter coefficients regarding the timestamp map (x) can be used. Referring to Figure 13A , virtual data (y) can be located at the edge of the timestamp map (x) to apply the digital filter (z). The user can set the digital filter coefficients by considering the relationship between the kernel size and the filter coefficients. Various forms of filters can calculate the optical flow OF. However, the exemplary embodiment is not limited thereto. For example, various algorithms other than the method of applying a filter can be used.

[0099] When the timestamp map TSM is completed, the optical flow regarding the timestamp map TSM can be calculated. Referring to Figure 13B, the optical flow map OFM can be calculated and stored at the storage location of the timestamp map TSM. For example, when generating the timestamp map TSM for specific data for which cropping or merging is performed and storing the timestamp map TSM at a specific location in the graph data buffer, the optical flow map OFM can be stored at that specific location instead of the timestamp map TSM. The size of the memory (e.g., synchronous static random access memory (SRAM)) in the graph data buffer 135 can be managed more efficiently in this way.

[0100] Figure 14 is a block diagram showing an example of an electronic device to which a vision sensor according to an example embodiment is applied.

[0101] Reference Figure 14 , the electronic device 1000 may include a vision sensor 1100, an image sensor 1200, a main processor 1300, a working memory 1400, a storage device 1500, a display device 1600, a user interface 1700, and a communicator 1800.

[0102] Reference Figures 1 to 13B The described vision sensor 100 can be applied as the vision sensor 1100. The vision sensor 1100 can sense an object to generate an event signal and generate a timestamp map (e.g., Figure 1 TSM in Figure 1 ) or an optical flow map (e.g., Figure 2 OFM in

[0103] ), and send at least one of the event signal (e.g., Figure 2 EVS in

[0103] ), the timestamp map TSM, or the optical flow map OFM to the main processor 1300.

[0104] The image sensor 1200 can generate image data (such as raw image data) based on the received optical signal and provide the image data to the main processor 1300. Figure 1

[0105] The main processor 1300 can control the overall operation of the electronic device 1000 and can detect the movement of an object by processing event data (i.e., the event signal received from the vision sensor 1100). Moreover, an image frame can be received from the image sensor 1200, and image processing can be performed based on preset information. Similar to Figure 1 the processor 200 shown in

[0105] , the main processor 1300 may include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), an application-specific microprocessor, a microprocessor, a general-purpose processor, etc. According to an example embodiment, the main processor 1300 may include an application processor or an image processor.The working memory 1400 can store data for the operation of the electronic device 1000. For example, the working memory 1400 can temporarily store packets or frames processed by the main processor 1300. For example, the working memory 1400 can include volatile memories (such as dynamic random access memory (DRAM) and SRAM) and / or non-volatile memories (such as phase change RAM (PRAM), magnetoresistive RAM (MRAM), resistive RAM (ReRAM), and ferroelectric RAM (FRAM)).

[0106] The storage device 1500 can store data requested to be stored from the main processor 1300 or other components. The storage device 1500 can include non-volatile memories such as flash memory, PRAM, NRAM, ReRAM, and FRAM.

[0107] The display device 1600 can include a display panel, a display driving circuit, and a display serial interface (DSI). For example, the display panel can be implemented as various devices such as a liquid crystal display (LCD) device, a light emitting diode (LED) display, an organic LED (OLED) display, and an active matrix OLED (AMOLED) display. The display driving circuit can include a timing controller, a source driver, etc. required to drive the display panel. The DSI host embedded in the main processor 1300 can perform serial communication with the display panel through the DSI.

[0108] The user interface 1700 can include at least one of the following input interfaces, such as a keyboard, a mouse, a keypad, a button, a touch panel, a touch screen, a touchpad, a touch ball, a gyro sensor, a vibration sensor, and an acceleration sensor.

[0109] The communicator 1800 can exchange signals with an external device / system through the antenna 1830. The transceiver 1810 and the modulator / demodulator (MODEM) 1820 of the communicator 1800 can process the signals exchanged with the external device / system according to the following wireless communication protocols, such as Long Term Evolution (LTE), Worldwide Interoperability for Microwave Access (WIMAX), Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), Bluetooth, Near Field Communication (NFC), Wireless Fidelity (Wi-Fi), or Radio Frequency Identification (RFID).

[0110] Components of the electronic device 1000, such as the vision sensor 1100, the image sensor 1200, the main processor 1300, the working memory 1400, the storage device 1500, the display device 1600, the user interface 1700, and the communicator 1800, may exchange data according to at least one of the following various interface protocols, such as Universal Serial Bus (USB), Small Computer System Interface (SCSI), MIPI, Inter-Integrated Circuit (I2C), Peripheral Component Interconnect Express (PCIe), Mobile PCIe (M-PCIe), Advanced Technology Attachment (ATA), Parallel ATA (PATA), Serial ATA (SATA), Serial Attached SCSI (SAS), Integrated Drive Electronics (IDE), Enhanced IDE (EIDE), Non-Volatile Memory Express (NVMe), or Universal Flash Storage (UFS).

[0111] According to an exemplary embodiment, at least one of the components, elements, modules, or units (collectively referred to as "units" in this paragraph) represented by the blocks in the accompanying drawings including Figure 3 and Figure 4A can be embodied as various numbers of hardware, software, and / or firmware structures that perform the respective functions described above. For example, at least one of these units may use a direct circuit structure, such as a memory, a processor, a logic circuit, a look-up table, etc., which can perform the respective functions under the control of one or more microprocessors or other control devices. Moreover, at least one of these units may be specifically embodied by a module, a program, or a portion of code that includes one or more executable instructions for performing specific logical functions and is executed by one or more microprocessors or other control devices. In addition, at least one of these units may include a processor, such as a central processing unit (CPU), a microprocessor, etc., that performs the respective functions, or at least one of these units may be implemented by a processor, such as a central processing unit (CPU), a microprocessor, etc., that performs the respective functions. Two or more of these units may be combined into a single unit that performs all the operations or functions of the combined two or more units. Moreover, at least a part of the function of at least one of these units may be performed by another one of these units. In addition, although a bus is not shown in the above block diagram, communication between the units may be performed via a bus. The functional aspects of the above example embodiments may be implemented as algorithms executed on one or more processors. In addition, the units, or processing steps, represented by the blocks may employ any number of related technologies for electronic configuration, signal processing and / or control, data processing, etc.

[0112] Although example embodiments have been shown and described above, it will be apparent to those skilled in the art that modifications and changes can be made without departing from the scope defined by the appended claims and their equivalents.

Claims

1. A sensor, comprising: A pixel array including a plurality of pixels arranged in a matrix; A data buffer; An optical flow map generator configured to generate a first optical flow map based on an optical flow of a pattern representing the movement of an object and a timestamp map stored in the data buffer, the optical flow map generator being configured to store the first optical flow map in the data buffer; And An interface circuit configured to receive the first optical flow map stored in the data buffer and send the first optical flow map from the sensor to an external processor implemented as a separate semiconductor chip, Wherein the optical flow is generated from the plurality of pixels based on the movement of the object, and Wherein the optical flow map generator and the interface circuit are implemented in the sensor.

2. The sensor according to claim 1, wherein, The interface circuit is configured to periodically send the first optical flow map to the external processor.

3. The sensor according to claim 1, wherein, The sensor is configured to adjust the size of the first optical flow map and send the first optical flow map to the external processor.

4. The sensor according to claim 2, wherein The interface circuit is configured to receive a second optical flow map generated by the optical flow map generator for each of a preset number of event signal frames and send the second optical flow map to the external processor, the second optical flow map being generated based on pixels corresponding to a plurality of regions of interest.

5. The sensor according to claim 2, wherein, The interface circuit is configured to receive a second optical flow map generated by the optical flow map generator for each of a preset number of event signal frames and send the second optical flow map to the external processor, the second optical flow map being generated based on pixels corresponding to a region of interest.

6. The sensor according to claim 5, wherein, The second optical flow map includes information related to the moving direction of the object.

7. The sensor according to claim 6, wherein The interface circuit is configured to send the first optical flow map to the external processor in a first mode; and Wherein the interface circuit is configured not to send the first optical flow map to the external processor in a second mode different from the first mode.

8. The sensor according to claim 7, wherein, The first optical flow map includes information related to the speed of the object.

9. The sensor according to claim 1, wherein The optical flow map generator is configured to store the first optical flow map in the data buffer, replacing the timestamp map.

10. A sensor, comprising: A pixel array including a plurality of pixels arranged in a matrix; A data buffer; An optical flow map generator configured to generate a first optical flow map based on an optical flow of a pattern representing the movement of an object and a timestamp map stored in the data buffer, the optical flow map generator being configured to store the first optical flow map in the data buffer; And An interface circuit configured to receive the first optical flow map stored in the data buffer and send the first optical flow map from the sensor to an external processor implemented as a separate semiconductor chip, Wherein the optical flow is generated from the plurality of pixels based on the movement of the sensor, and Wherein the optical flow map generator and the interface circuit are implemented in the sensor.

11. The sensor according to claim 10, wherein, The sensor is configured to adjust the size of the first optical flow map and send the first optical flow map to the external processor.

12. The sensor according to claim 10, wherein, The interface circuit is configured to periodically send the first optical flow map to the external processor.

13. The sensor according to claim 12, wherein, The interface circuit is configured to receive a second optical flow map generated by the optical flow map generator for each of a preset number of event signal frames and send the second optical flow map to the external processor, the second optical flow map being generated based on pixels corresponding to the region of interest.

14. The sensor according to claim 13, wherein, The interface circuit is configured to receive a third optical flow map generated by the optical flow map generator for each of the preset number of event signal frames and send the third optical flow map to the external processor, the third optical flow map being generated based on pixels corresponding to a plurality of regions of interest.

15. The sensor according to claim 14, wherein, The interface circuit is configured to send the second optical flow map to the external processor in a first mode; and wherein the interface circuit is configured not to send the second optical flow map to the external processor in a second mode different from the first mode.

16. The sensor according to claim 15, wherein, The first optical flow map includes information related to the speed of the object.

17. The sensor according to claim 16, wherein, The optical flow map generator is configured to store the first optical flow map in the data buffer instead of the timestamp map.

18. A sensor, comprising: a pixel array including a plurality of pixels arranged in a matrix; a data buffer; an optical flow map generator configured to generate a first optical flow map based on an optical flow representing a pattern of movement of an object and a timestamp map stored in the data buffer, the optical flow map generator being configured to store the first optical flow map in the data buffer; and an interface circuit configured to receive the first optical flow map stored in the data buffer and send the first optical flow map from the sensor to an external processor implemented as a separate semiconductor chip, wherein the optical flow is generated from the plurality of pixels based on a relative movement between an observer and a scene, and wherein the optical flow map generator and the interface circuit are implemented in the sensor.

19. The sensor according to claim 18, wherein, The interface circuit is configured to periodically send the first optical flow map to the external processor; and wherein the interface circuit is configured to receive a second optical flow map generated by the optical flow map generator for each of a preset number of event signal frames and send the second optical flow map to the external processor, the second optical flow map being generated based on pixels corresponding to the region of interest.

20. The sensor according to claim 19, wherein, The interface circuit is configured to send the first optical flow map to the external processor in a first mode; and wherein the interface circuit is configured not to send the first optical flow map to the external processor in a second mode different from the first mode.

Images