Signal processing device, electronic device, signal processing method and program
By designing a signal processing device in an event-driven vision sensor, using the forced readout mechanism of the buffer memory and the readout control unit, the processing delay problem caused by asynchronous signal generation is solved, and more efficient data processing is achieved.
Patent Information
- Application Number
- CN201980078651.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2018-12-05
- Filing Date
- 2019-12-03
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2039-12-03
AI Technical Summary
The signal in the event-driven visual sensor is generated asynchronously in time, resulting in data processing delay problems, especially because the burst size of data packets in the AER format is not fully filled, resulting in the buffer being unable to be effectively utilized, resulting in processing delays.
A signal processing device is designed, including a communication unit receiving a data packet of an event-driven vision sensor and temporarily storing it in a buffer memory, and forcibly reading the data packets under a predetermined condition through the readout control unit to avoid delays.
By forcibly reading out the data packets while meeting predetermined conditions, the data processing delay of the event-driven vision sensor output can be effectively suppressed, and processing efficiency can be ensured, especially when the sensor array detects an event signal, the capacity of the buffer memory can be utilized in time.
Smart Images

Figure CN113170059B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a signal processing device, an electronic device, a signal processing method and a program. Background Art
[0002] An event-driven visual sensor is known in which pixels that detect changes in incident light intensity generate signals asynchronously in time. The advantage of the event-driven visual sensor is that the sensor can operate at low power and high speed compared to a frame-type visual sensor (specifically, an image sensor such as a CCD (charge coupled device) or a CMOS (complementary metal oxide semiconductor)) that scans all pixels in a predetermined cycle. For example, technology regarding such an event-driven visual sensor is described in PTL 1 and PTL 2.
[0003] As a method of transmitting event signals generated by such event-driven visual sensors, an AER (Address Event Representation) format is known. For example, in the AER format, four data are transmitted: the x-coordinate and y-coordinate of the pixel where the event occurred, the on / off (polarity) of the event, and a timestamp.
[0004] [Citation list]
[0005] [Patent Document]
[0006] [PTL 1]
[0007] JP 2014-535098T
[0008] [PTL 2]
[0009] JP 2018-85725A Summary of the invention
[0010] [Technical issues]
[0011] However, since signals are generated asynchronously in time in an event-driven vision sensor, the amount of data transmitted each time in the AER format varies, and therefore there is a problem that the burst size of the buffer is not filled and a delay occurs in data processing.
[0012] Therefore, an object of the present invention is to provide a signal processing device, an electronic device, a signal processing method, and a program capable of eliminating a processing delay of data output from an event-driven vision sensor.
[0013] [Solution to the problem]
[0014] According to one aspect of the present invention, there is provided a signal processing device, comprising: a communication unit, which receives a data packet sent from an event-driven visual sensor, the event-driven visual sensor comprising a sensor array, the sensor array comprising a sensor that generates an event signal when a change in incident light intensity is detected; a buffer memory, the data packet is temporarily stored in the buffer memory; and a read-out control unit, the read-out control unit forcibly reads the data packet from the buffer memory when a predetermined condition is met.
[0015] According to another aspect of the present invention, an electronic device comprising the above signal processing device is provided.
[0016] According to another aspect of the present invention, there is provided a signal processing method, comprising: a step of receiving a data packet sent from an event-driven visual sensor, the event-driven visual sensor comprising a sensor array, the sensor array comprising a sensor that generates an event signal when a change in incident light intensity is detected; a step of temporarily storing the data packet in a buffer memory; and a step of forcibly reading out the data packet from the buffer memory when a predetermined condition is met.
[0017] According to another aspect of the present invention, a program is provided that allows a processing circuit of a signal processing device to execute: a step of temporarily storing a data packet sent from an event-driven visual sensor in a buffer memory, the event-driven visual sensor comprising a sensor array, the sensor array comprising a sensor that generates an event signal when a change in incident light intensity is detected; and a step of forcibly reading out the data packet from the buffer memory when a predetermined condition is met.
[0018] According to the above configuration, since a data packet is forcibly read out from the buffer memory when a predetermined condition is satisfied, it is possible to suppress a processing delay of data output from the event-driven vision sensor. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 is a block diagram showing a schematic configuration of an electronic device according to a first embodiment of the present invention.
[0020] Figure 2A 1 is a diagram showing a transmission end packet for conceptually describing the first embodiment of the present invention.
[0021] Figure 2B 1 is a diagram showing a transmission end packet for conceptually describing the first embodiment of the present invention.
[0022] Figure 2C 1 is a diagram showing a transmission end packet for conceptually describing the first embodiment of the present invention.
[0023] Figure 3 is a flowchart showing an example of processing by the processing circuit of the vision sensor in the first embodiment of the present invention.
[0024] Figure 4 is a flowchart showing an example of the processing of the readout control unit in the first embodiment of the present invention.
[0025] Figure 5A : is a diagram showing a transmission start packet for conceptually describing the second embodiment of the present invention.
[0026] Figure 5B : is a diagram showing a transmission start packet for conceptually describing the second embodiment of the present invention.
[0027] Figure 6 is a flowchart showing an example of processing of a readout control unit in the third embodiment of the present invention. DETAILED DESCRIPTION
[0028] Hereinafter, several embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that in the specification and the drawings, constituent elements having substantially the same functional configuration are denoted by the same reference symbols, and repeated description thereof is omitted.
[0029] (First embodiment)
[0030] Figure 1 1 is a block diagram showing a schematic configuration of an electronic device according to a first embodiment of the present invention. Figure 1 As shown, the electronic device 10 includes an event-driven visual sensor 100 and a control unit 200 .
[0031] The visual sensor 100 includes a sensor array 110 including sensors 110A, 110B, etc. corresponding to image pixels, and a processing circuit 120 connected to the sensor array 110. The sensors 110A, 110B, etc. include light receiving elements and generate event signals when a change in the intensity of incident light (more specifically, a change in brightness) is detected. The event signal is output from the processing circuit 120 as information indicating, for example, a timestamp, identification information of the sensor (for example, the position of the pixel), and the polarity (increase or decrease) of the brightness change. As will be described later, the processing circuit 120 adds a transmission end data packet to the data packet of the event signal. When a subject moves within the field of view of the sensor array 110, the intensity of light reflected or scattered by the subject changes, and therefore, the movement of the subject can be detected in a time series by, for example, an event signal corresponding to the edge of the subject generated by the sensors 110A, 110B, etc.
[0032] The control unit 200 includes a communication interface 210, a processing circuit 220, and a memory 230. The communication interface 210 receives a data packet transmitted from the processing circuit 120 of the visual sensor 100. The data packet is temporarily stored in the buffer memory 231. The processing circuit 220 operates according to a program stored in the memory 230, for example, and processes the data packet read out from the buffer memory 231. For example, the processing circuit 220 generates an image in which positions where brightness changes are mapped in a time series based on an event signal included in the data packet; and temporarily or continuously stores the image in the memory 230 or further transmits the image to another device via the communication interface 210. Here, the processing circuit 220 includes a readout control unit 221, which forcibly reads out a data packet from the buffer memory 231 when a predetermined condition is satisfied.
[0033] FIG. 2A to FIG. 2C 1 and 2 are diagrams conceptually describing the transmission end data packet in the first embodiment of the present invention. FIG. 2A to FIG. 2C In the example shown, the minimum unit (one packet) of the event signal output from the vision sensor 100 is 4 bytes, and the size of the buffer memory 231 in the control unit 200 is 64 bytes (corresponding to 16 packets).
[0034] exist Figure 2A In the example shown, the readout control unit 221 is configured to read out the data packet 121 from the buffer memory 231 when the capacity of the buffer memory 231 becomes full. Here, in the case where the number of events detected by the sensor array 110 is small and the visual sensor 100 outputs only event signals corresponding to 5 data packets (i.e., 20 bytes), the capacity of the buffer memory 231 is still available. Therefore, the readout control unit 221 does not read out the data packet 121 from the buffer memory 231. This situation continues until the capacity of the buffer memory 231 becomes full due to an event signal received thereafter, or until the data packet 121 is forcibly read out from the buffer memory 231 due to a timeout. Therefore, a delay may occur between the generation of the event signal by the visual sensor 100 and the processing of the event signal by the processing circuit 220.
[0035] The situation described above occurs because the event signal in the event-driven visual sensor 100 is generated asynchronously in time. Among the sensors 110A, 110B, etc. configuring the sensor array 110 in the visual sensor 100, only the sensor that detects the brightness change generates the event signal, and the sensor that does not detect the brightness change does not generate the event signal, so low power and high-speed operation can be performed compared to the frame type visual sensor. On the contrary, in the visual sensor 100, unlike the frame type visual sensor that transmits data corresponding to all pixels at the time of scanning, the amount of data transmitted each time is variable, so there is a possibility that the delay described above occurs.
[0036] Therefore, if Figure 2B and Figure 2C As shown in the illustrated example, the processing circuit 120 in the visual sensor 100 adds a transmission end packet 122 to the packet 121 of the event signal in the embodiment. More specifically, when the scanning of the sensor 110A, 110B, etc. is completed, the processing circuit 120 adds the transmission end packet 122 to the packet 121 of the event signal. In this case, when the transmission end packet 122 is stored in the buffer memory 231, the readout control unit 221 forcibly reads out the packet 121 from the buffer memory 231 regardless of the available capacity of the buffer memory 231. Figure 2B In the example shown, when the transmission end data packet 122 is found, the read control unit 221 stores the padding data 123 in the free capacity of the buffer memory 231 to fill the capacity of the buffer memory 231, and the data packet 121 is sent to the processing circuit 220 accordingly. Figure 2C In the example shown, when finding the transmission end data packet 122 , the read control unit 221 may also be configured to forcibly read the data packet 121 from the buffer memory 231 without filling and send the data packet to the processing in the processing circuit 220 .
[0037] Figure 3 1 is a flowchart showing an example of the processing of the processing circuit of the visual sensor in the first embodiment of the present invention. In the example shown, the processing circuit 120 performs scanning of the sensor array 110 (step S101). Here, the scanning of the sensor array 110 is performed by scanning the sensors 110A, 110B, etc. in a predetermined order, and acquiring event signals only from sensors that detect brightness changes. For example, although scanning is performed in the same manner even in a frame-type visual sensor, the scanning of the event-driven visual sensor 100 is performed at a higher speed because the processing for the sensor that does not detect brightness changes is skipped. When the scanning in step S101 is completed, the processing circuit 120 adds a transmission end data packet to the data packet of the event signal (step S102). The processing circuit 120 repeats these processes (step S103).
[0038] Figure 41 is a flowchart showing an example of the processing of the readout control unit in the first embodiment of the present invention. In the example shown, the readout control unit 221 stores the data packet of the event signal received from the visual sensor 100 in the buffer memory 231 (step S121). The readout control unit 221 monitors the data packet of the event signal stored in the buffer memory 231, and in the case where the transmission end data packet 122 is received or stored (step S122), the data packet 121 is read out from the buffer memory 231 (step S123). Even in the case where the capacity of the buffer memory 231 becomes full (step S124), the readout control unit 221 reads out the data packet 121 from the buffer memory 231 (step S123). The readout control unit 221 repeats these processes (step S125).
[0039] In the above example, in the case where the number of events detected by the sensor array 110 is large and the data packets 121 to be transmitted exceed the capacity of the buffer memory 231, the data packets 121 are sequentially read out from the buffer memory 231 by the processing in the above step S105. If the transmission efficiency in the case where the transmission amount is large as described above is considered, it is preferable that the capacity of the buffer memory 231 is large to some extent. In the present embodiment, in the case where the number of event signals output from the visual sensor 100 is small, as described above, forced reading is performed by the transmission end data packet 122, so that the processing delay of the event signal in the control unit 200 can be suppressed to a time corresponding to one scan of the sensor array 110 at most while ensuring the capacity of the buffer memory 231.
[0040] (Second embodiment)
[0041] Next, a second embodiment of the present invention will be described. In the second embodiment, the processing circuit 120 of the visual sensor 100 adds a transmission start packet to the packet of the event signal in the same configuration of the electronic device as the first embodiment described above. The processing of the processing circuit 120 in this embodiment may be, for example, by using Figure 3 , in the example described above, the order of step S101 and step S102 is exchanged, and the process of replacing the transmission end packet with the transmission start packet is performed. In other aspects, the configuration of this embodiment is the same as that of the first embodiment described above, and therefore, repetition and detailed description thereof are omitted.
[0042] Figure 5A and Figure 5B are diagrams conceptually describing a transmission start data packet in the second embodiment of the present invention. Figure 5A and Figure 5B In the example shown above, Figure 2A to Figure 2CThe buffer memory 231 is configured in the same manner as in the example described above. Note that the case where no transmission start packet is received is the same as that described in the example described above. Figure 2A The examples described are the same and therefore omitted.
[0043] exist Figure 5A In the example shown, when a transmission start packet 124 is newly received, the read control unit 221 forcibly reads out the packet 121 of the event signal stored in the buffer memory 121 at that time, and sends the packet 121 to the processing in the processing circuit 220. The read control unit 221 can perform the following operations: Figure 2B In this case, the processing delay of the event signal in the control unit 200 is suppressed to a time corresponding to one scan of the sensor array 110 at most.
[0044] At the same time, Figure 5B In the example shown, when the number of packets 121 of the event signal stored in the buffer memory 231 reaches a predetermined number n counted from the transmission start packet 124, the readout control unit 221 forcibly reads out the packets 121 of the event signal stored in the buffer memory 231 at that time, and sends the packets to the processing in the processing circuit 220. For example, the predetermined number n may be statistically determined as an average value or a maximum value of the number of event signals output in one scan of the sensor array 110.
[0045] In this embodiment as well, similar to the first embodiment described above, while ensuring the capacity of the buffer memory 231, it is possible to suppress the processing delay of the event signal in the control unit 200. In addition, the present embodiment can also be applied to a case where the end of scanning is not specified in the processing circuit 120, for example, because the event signal generated by the visual sensor 100 is asynchronous in time.
[0046] In the above-mentioned first and second embodiments of the present invention, the processing circuit 120 of the visual sensor 100 adds a delimiter packet to a packet corresponding to an event signal, such as the transmission end packet 122 or the transmission start packet 124, thereby avoiding a processing delay of the event signal. Note that the processing circuit 120 does not have to perform the scanning of the sensor array 110 as described above. In this case, for example, the processing circuit 120 sends a delimiter packet to the control unit 200 in each predetermined time cycle, and the readout control unit 221 processes the delimiter packet in the same manner as the transmission end packet 122 and the transmission start packet 124 in the above-mentioned example, thereby avoiding a processing delay of the event signal.
[0047] (Third Embodiment)
[0048] Next, a third embodiment of the present invention will be described. In the third embodiment, in the configuration of the electronic device that is the same as the first embodiment described above, the processing circuit 120 of the visual sensor 100 does not add a delimiter packet to the packet of the event signal, and the readout control unit 221 of the control unit 200 forcibly reads out the packet 121 of the event signal stored in the buffer memory 231 according to a predetermined condition, thereby avoiding a delay in the processing of the event signal. In other respects, the configuration of this embodiment is the same as that of the first embodiment described above, and therefore, repetition and detailed description thereof are omitted.
[0049] Figure 6 is a flowchart showing an example of the processing of the readout control unit in the third embodiment of the present invention. In the example shown, the readout control unit 221 stores the data packet of the event signal received from the visual sensor 100 in the buffer memory 231 (step S301). The readout control unit 221 monitors the data packet of the event signal stored in the buffer memory 231, and in the case where the identification information of the sensor indicated by the event signal (for example, the position of the pixel) is circulated (step S302), the data packet 121 is read out from the buffer memory 231 (step S303). Even in the case where the capacity of the buffer memory 231 becomes full (step S304), the readout control unit 221 reads out the data packet 121 from the buffer memory 231 (step S303). The readout control unit 221 repeats these processes (step S305).
[0050] In the above example, under the condition that the identification information of the sensor indicated by the event signal is circulated, the readout control unit 221 forcibly reads the data packet 121 from the buffer memory 231. Specifically, for example, in the case where the identification information is the coordinates (x, y) indicating the position of the pixel corresponding to the sensor, if the time stamp t 1 The associated coordinates (x 1 ,y 1 ) and timestamp t 2 (t 2 >t 1 ) associated with the coordinate (x 2 , y 2 ) between x 2 <x 1 or x 2 =x 1 And y 2 <y 1 , then the read control unit 221 may determine that the identification information is cyclic. Alternatively, in the case of receiving an event signal indicating the same identification information associated with different timestamps, the read control unit 221 may determine that the identification information is cyclic.
[0051] Even in this embodiment, similar to the above-described first and second embodiments, while ensuring the capacity of the buffer memory 231, the processing delay of the event signal in the control unit 200 can be suppressed. In addition, this embodiment can also be applied to a case where, for example, it is not easy to implement the function of adding a delimiter packet to the event signal in the processing circuit 120.
[0052] Although several embodiments of the present invention have been described in detail above with reference to the accompanying drawings, the present invention is not limited to these examples. Obviously, a person skilled in the art in the art to which the present invention belongs can think of various variations and modifications within the scope of the technical concept described in the claims, and it should be understood that these variations and modifications obviously belong to the technical scope of the present invention.
[0053] [Reference Symbol List]
[0054] 10: Electronic devices
[0055] 100: Vision Sensor
[0056] 110: Sensor Array
[0057] 110A, 110B: Sensor
[0058] 120: Processing circuit
[0059] 121: Data packet
[0060] 122: Transmission end data packet
[0061] 123: Fill data
[0062] 124: Transmission start data packet
[0063] 200: Control unit
[0064] 210: Communication interface
[0065] 220: Processing circuit
[0066] 221: Read out control unit
[0067] 230: Memory
[0068] 231: Buffer memory.
Claims
1. A signal processing device, comprising: a communication unit that receives a data packet sent from an event-driven vision sensor, the event-driven vision sensor comprising a sensor array including sensors that generate event signals when a change in incident light intensity is detected; a buffer memory in which the data packets are temporarily stored; as well as a readout control unit, wherein if the communication unit receives a delimiter data packet or the delimiter data packet is stored in the buffer memory, the readout control unit forcibly reads the data packet from the buffer memory, The data packet includes a data packet corresponding to the event signal and the delimiter data packet added to the data packet corresponding to the event signal in the vision sensor.
2. The signal processing device according to claim 1, in, The delimiter packet is a transmission start packet, and the readout control unit forcibly reads out the packet from the buffer memory if the delimiter packet is stored in the buffer memory and the number of packets counted from the delimiter packet reaches a predetermined number.
3. An electronic device, comprising: The signal processing device according to claim 1 or 2.
4. A signal processing method, comprising: The step of receiving a data packet transmitted from an event-driven vision sensor, the event-driven vision sensor comprising a sensor array, the sensor array comprising sensors that generate event signals when a change in incident light intensity is detected; The step of temporarily storing the data packet in a buffer memory; as well as If a delimiter data packet is received or the delimiter data packet is stored in the buffer memory, the step of forcibly reading the data packet from the buffer memory, The data packet includes a data packet corresponding to the event signal and the delimiter data packet added to the data packet corresponding to the event signal in the vision sensor.
5. A non-transitory computer-readable storage medium having a program stored thereon, wherein the program enables a processing circuit of a signal processing device to perform: the step of temporarily storing in a buffer memory a data packet transmitted from an event-driven vision sensor, the event-driven vision sensor comprising a sensor array including sensors that generate event signals when a change in incident light intensity is detected; and If a delimiter data packet is received or the delimiter data packet is stored in the buffer memory, the step of forcibly reading the data packet from the buffer memory, in, The data packet includes a data packet corresponding to the event signal, and the delimiter data packet added to the data packet corresponding to the event signal in the vision sensor.
