Signal processing device, sensor device, signal processing method, and program

Abstract

Provided is a signal processing device comprising a transmission determination unit that determines whether or not to transmit an event signal that is output from an event-driven vision sensor including multiple sensors constituting a sensor array, the determination being made on the basis of position information for each of the sensors in the sensor array.

Description

Technical Field

[0001] The present invention relates to a signal processing device, a sensor device, a signal processing method, and a program.

Background Art

[0002] An event-driven vision sensor is known in which pixels that detect changes in the intensity of incident light generate signals asynchronously in time. The event-driven vision sensor is advantageous in that it can operate at low power and high speed compared to a frame-type vision sensor that scans all pixels at a predetermined period, specifically an image sensor such as a CCD or a CMOS. Techniques related to such event-driven vision sensors are described in, for example, Patent Document 1 and Patent Document 2.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] As described above, since the event-driven vision sensor has high resolution both temporally and spatially, it outputs a large number of event signals including those due to noise. Depending on the type of processing based on the event signals, the load associated with the transmission and calculation of such a large number of signals may become excessive, but no solution has been proposed for such cases.

[0005] Therefore, an object of the present invention is to provide a signal processing device, a sensor device, a signal processing method, and a program capable of reducing the load associated with the transmission and calculation of event signals.

Means for Solving the Problems

[0006] According to an aspect of the present invention, there is provided a signal processing apparatus including a transmission determination unit that determines whether to transmit an event signal output from an event-driven vision sensor including a plurality of sensors constituting a sensor array based on position information of each sensor in the sensor array.

[0007] According to another aspect of the present invention, there is provided a sensor device including an event-driven vision sensor including a plurality of sensors constituting a sensor array, an image processing unit that performs image processing based on an event signal output from the vision sensor, and a region designating unit that designates a region based on a result of the image processing, wherein the vision sensor is configured to output an event signal limited to the designated region.

[0008] According to still another aspect of the present invention, there is provided a signal processing method including a step of determining whether to transmit an event signal output from an event-driven vision sensor including a plurality of sensors constituting a sensor array based on position information of each sensor in the sensor array.

[0009] According to still another aspect of the present invention, there is provided a program for causing a computer to realize a function of determining whether to transmit an event signal output from an event-driven vision sensor including a plurality of sensors constituting a sensor array based on position information of each sensor in the sensor array.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Mode for Carrying Out the Invention

[0011] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, and redundant explanations are omitted.

[0012] (First Embodiment) FIG. 1 is a diagram showing a schematic configuration of a system according to the first embodiment of the present invention. In the illustrated example, the system 10 includes an EDS (Event Driven Sensor) 100, which is an event-driven vision sensor, and a signal processing device 200.

[0013] The EDS100 includes a sensor array 120 composed of a plurality of sensors 110 and a sensor control unit 130 connected to the sensor array 120. The sensor 110 includes a light receiving element and generates an event signal when detecting a change in the intensity of incident light, more specifically, a change in luminance. In an image obtained by mapping the event signal based on the address of each sensor 110 within the sensor array 120, each sensor 110 corresponds to a pixel. In the following description, the region corresponding to the entire sensor array 120 in the above image is also referred to as a pixel region, and the region corresponding to each individual sensor 110 is also referred to as a pixel. However, when the event signal is read from the sensor 110 according to the address generated by the address generator included in the sensor control unit 130, the reading from the sensor 110 that did not detect an event is not executed, so the event signal output from the EDS100 is asynchronous in time.

[0014] The signal processing device 200 includes a communication interface 210, a buffer memory 220, an arithmetic unit 230, and a storage unit 240. The communication interface 210 receives an event signal from the sensor control unit 130 of the EDS100. The received event signal is temporarily stored in the buffer memory 220. The arithmetic unit 230 is implemented by a processor that operates according to, for example, a program stored in the storage unit 240 and processes the event signal read from the buffer memory 220.

[0015] In this embodiment, the arithmetic unit 230 includes a transmission determination unit 231 and an image processing unit 232 as functional parts realized by operating according to a program. The transmission determination unit 231 determines whether to transmit an event signal to the image processing unit 232, for example, as in the example described later. That is, not all of the event signals temporarily stored in the buffer memory 220 are transmitted to the image processing unit 232. The image processing unit 232 executes various image processes based on the event signals transmitted from the buffer memory 220 according to the determination of the transmission determination unit 231. For example, the image processing unit 232 may generate, in time series, an image mapping the positions where luminance changes occur, and execute processes such as tracking a subject or calculating an optical flow on the image. The storage unit 240 stores a transmission determination rule 241 referred to by the transmission determination unit 231.

[0016] FIG. 2 is a diagram for explaining an example of transmission determination in the first embodiment of the present invention. In the illustrated example, the transmission determination unit 231 calculates a score for each segment (SEG) including a plurality of pixels obtained by dividing a pixel region into a predetermined number based on the event signal output from the EDS 100. As described above, the pixel region corresponds to the sensor array 120, and since the sensor 110 corresponds to a pixel, the segment is defined within the sensor array 120 and includes a plurality of sensors 110. For example, when the pixels are arranged in two orthogonal directions, a rectangular region of 6 pixels × 6 pixels may be used as one segment. The transmission determination unit 231 calculates a score by integrating the number of event signals for each segment, and when the score exceeds a threshold value, transmits the event signals output in the segment and adjacent segments to the image processing unit 232. Note that the score of the segment for which transmission has been executed is reset.

[0017] In the illustrated example, event signals are transmitted when the number of event signals exceeds half the number of pixels in the segment (6×6÷2 = 18). In segments that overlap or are in the vicinity of a subject (OBJ) present in the pixel region, the output of event signals is relatively higher compared to other regions. On the other hand, event signals are also output in segments not related to the subject due to the influence of noise or the like, but the number is relatively small. The transmission determination unit 231 identifies the event signals in each of such subject portions and other portions based on the position information of each sensor 110 in the sensor array 120, and determines whether to transmit each event signal. Therefore, with the configuration as described above, it is possible to suppress the transmission of event signals in segments not related to the subject, and reduce the load on the transmission of event signals and the image processing operations in the image processing unit.

[0018] In the above example, the transmission determination unit 231 may attenuate or reset the score of each segment at predetermined time intervals. For example, if the number of event signals in each segment is integrated without time limit, the number of event signals due to noise or the like may reach the threshold over a long time even in segments not related to the subject. By attenuating or resetting the score for segments where the number of event signals has not reached the threshold over a long time, it is possible to further reduce the load on the transmission of event signals generated due to the influence of noise and the image processing operations.

[0019] In the process of the transmission determination unit 231 as described above, the shape and number of segments for dividing the pixel region, as well as the calculation method and threshold value of the score for each segment, are stored in the storage unit 240 as the transmission determination rule 241. Here, in the transmission determination rule 241, for example, a plurality of different criteria may be defined according to conditions related to the image processing executed by the image processing unit 232. For example, when it is required to reduce the data transmission amount in the process in the image processing unit 232 or in the process after transmitting the processing result to another device, the threshold value for determination in the transmission determination unit 231 may be increased. Conversely, when it is required to improve the responsiveness rather than reducing the data transmission amount, the threshold value for determination in the transmission determination unit 231 may be decreased. Also, when the sizes of a plurality of segments are defined, the threshold value may be set according to the size of the segment, that is, the number of pixels included in the segment. Specifically, a larger threshold value may be set for a segment with a larger size, and a smaller threshold value may be set for a segment with a smaller size.

[0020] (Second Embodiment) FIG. 3 is a diagram showing a schematic configuration of a system according to the second embodiment of the present invention. As a difference from the above-described first embodiment, in this embodiment, the arithmetic unit 230 of the signal processing apparatus 200 includes a region designating unit 233 in addition to the transmission determination unit 231 and the image processing unit 232 as functional parts realized by operating according to a program. The region designating unit 233 designates a region to be the target of transmission of the event signal based on the result of the image processing in the image processing unit 232 as in the example described below. Since the other configurations are the same as those of the above-described first embodiment, redundant detailed descriptions are omitted.

[0021] FIG. 4 is a diagram for explaining an example of transmission determination in the second embodiment of the present invention. In the illustrated example, the transmission determination unit 231 calculates a score for each segment (SEG) obtained by dividing the pixel region in the same manner as in the example of the first embodiment described above. On the other hand, the region designating unit 233 designates a region (R) that is the target of transmission of the event signal based on the result of the image processing executed by the image processing unit 232. The region (R) may be, for example, a region that at least partially overlaps the subject (OBJ) or includes the subject. Also, the region (R) may be, for example, a region corresponding to a region of interest (ROI) in tracking or optical flow calculation. The transmission determination unit 231 calculates a score by integrating the number of event signals output for each segment. When the score exceeds a threshold value and the segment is included in the designated region (R), the transmission determination unit 231 transmits the event signals output in the segment and adjacent segments to the image processing unit 232.

[0022] Note that in the illustrated example, the transmission determination unit 231 calculates scores for all segments and then determines whether to transmit the event signal depending on whether the segment is included in the designated region (R). However, in other examples, the transmission determination unit 231 may calculate scores only for segments included in the designated region (R). In any case, the transmission determination unit 231 determines whether to transmit the event signal output from the sensor 110 having position information in the designated region (R).

[0023] In the illustrated example, there are two subjects (OBJ) within the pixel region, and the output of the event signal is relatively higher in the segments overlapping these subjects compared to other regions. In the first embodiment described above, in such a case, the event signals respectively generated for both subjects are transmitted from the transmission determination unit 231 to the image processing unit 232. On the other hand, in this embodiment, since the region (R) required for the image processing in the image processing unit 232 is specified in advance by the region specifying unit 233, the event signals generated for the non-target subject among the two subjects are not transmitted from the transmission determination unit 231 to the image processing unit 232. When the region (R) required for the image processing in the image processing unit 232 can be specified in advance, the transmission of the event signal and the load on the image processing operation in the image processing unit can be further reduced by the configuration as described above.

[0024] FIG. 5 is a diagram showing an example of a data path in the second embodiment of the present invention. In this embodiment, the transmission of the event signal from the transmission determination unit 231 to the image processing unit 232 and the transmission of the information for specifying the region from the region specifying unit 233 to the transmission determination unit 231 can be implemented by a queue using the ring buffers RB1 and RB2 as shown in the figure. Such a data path can be used for the transmission of the event signal and the transmission of the information for specifying the region in other embodiments.

[0025] (Third Embodiment) FIG. 6 is a diagram showing a schematic configuration of a sensor device according to a third embodiment of the present invention. In the present embodiment, a sensor array 120 including a sensor 110 which is a component of the EDS and a sensor control unit 130 are incorporated in the sensor device 300 (in the following description, these parts may be referred to as the EDS 100 for convenience). Therefore, the output of the event signal from the sensor control unit 130 and the input of the information for designating a region to the sensor control unit 130 described later are executed by the communication within the device via a bus interface or the like, rather than the communication between devices via a communication interface. Note that since the other configurations are the same as those in the second embodiment described above, redundant detailed descriptions are omitted.

[0026] FIG. 7 is a diagram for explaining an example of transmission determination in the third embodiment of the present invention. In the present embodiment, the region designating unit 233 inputs information designating a region (R) that is the target of event signal transmission to the EDS 100, and the EDS 100 outputs an event signal limited to the designated region (R). Specifically, the sensor control unit 130 executes reading of the event signal limited to the designated region (R). The transmission determination unit 231 calculates a score for each segment (SEG) obtained by dividing the pixel region in the same manner as in the first embodiment described above. However, since no event signal is output outside the region (R) designated by the region designating unit 233, the score is calculated limited to the segments in the designated region (R) as a result. When the score of a segment exceeds the threshold value, the event signals output in the segment and the adjacent segments are transmitted to the image processing unit 232. The region designating unit 233 designates a region (R) that is the target of event signal transmission based on the result of the image processing executed by the image processing unit 232 in the same manner as in the second embodiment described above. The region (R) may be, for example, a region that at least partially overlaps with the subject (OBJ) or a region that includes the subject. Further, the region (R) may be, for example, a region corresponding to the region of interest (ROI) in tracking or optical flow calculation.

[0027] Even in the illustrated example, similar to the example described above with reference to FIG. 4, there are two subjects (OBJ) within the pixel region. In the example of FIG. 4, event signals generated for both subjects were output from the EDS100, but in this embodiment, since the region designating unit 233 designates the region (R) in which the EDS100 outputs an event signal, event signals generated for subjects that are not the target among the two subjects are not output from the EDS100 in the first place. Thus, similar to the second embodiment described above, the region (R) required for image processing in the image processing unit 232 can be specified in advance, and when the sensor device 300 can interlock the arithmetic unit 230 and the sensor control unit 130, the transmission of event signals and the load related to image processing operations in the image processing unit can be further reduced by the configuration as described above. Specifically, not only the load related to the transmission from the transmission determination unit 231 to the image processing unit 232, but also the load related to the output of event signals from the sensor control unit 130 and the transmission to the transmission determination unit 231 can be reduced.

[0028] (Fourth Embodiment) FIG. 8 is a diagram showing a schematic configuration of a sensor device according to a fourth embodiment of the present invention. In this embodiment, an RGB sensor 410 is further incorporated into the same sensor device 400 as in the third embodiment described above, and the image processing unit 232 executes image processing using an image signal output from the RGB sensor 410 in addition to image processing using an event signal output from the EDS100. The region designating unit 233 mainly designates a region to be the target of event signal transmission based on the result of image processing using an image signal. Since other configurations are the same as those in the third embodiment described above, redundant detailed descriptions are omitted.

[0029] FIG. 9 is a diagram for explaining an example of transmission determination in the fourth embodiment of the present invention. In this embodiment, based on the result of image processing such as detection of a subject executed by the image processing unit 232 based on the image signal input from the RGB sensor 410, the region designating unit 233 designates a region (R) that is the target of transmission of the event signal. The region designating unit 233 inputs information for designating the target region (R) to the EDS 100, and the EDS 100 outputs an event signal limited to the designated region (R). As a result, similar to the example of FIG. 7 above, among the two subjects (OBJ) existing in the pixel region, the event signal generated for the non-target subject is not output from the EDS 100, and the event signal generated for the target subject within the region (R) is transmitted from the EDS 100 to the image processing unit 232. Note that when the event signal transmitted to the image processing unit 232 can be selected by controlling the EDS 100 as in this embodiment, the processes of score calculation and transmission determination in the transmission determination unit 231 do not necessarily have to be executed.

[0030] (Example of segment shape) FIG. 10 is a diagram showing an example of a segment shape applicable to each of the embodiments of the present invention described above. In the above, an example in which a rectangular area of 6 pixels × 6 pixels is taken as one segment when pixels are arranged in two orthogonal directions has been described. However, the shape of the segment is not limited to the above example. For example, the number of pixels constituting one segment may be different for each direction in which the pixels are arranged. Specifically, a rectangular area of 10 pixels × 5 pixels may be taken as one segment. Further, the segment does not necessarily have to be a rectangular area. For example, it may be a triangular area like SEG1 in the example shown in FIG. 10. In this example, for a spherical subject (OBJ), the triangular segment SEG1 can capture the subject with a smaller number of segments than the rectangular segments SEG2 divided into the same number per area (SEG1 is 8 out of 16 segments, SEG2 is 12 out of 16 segments). In this way, by combining boundary lines parallel to the direction in which the pixels are arranged and boundary lines obliquely intersecting those directions to define the shape of the segment, various-shaped subjects can be captured with a smaller number of segments.

[0031] As described above, the preferred embodiments of the present invention have been described in detail with reference to the accompanying drawings. However, the present invention is not limited to such examples. It is obvious that those having ordinary knowledge in the technical field to which the present invention pertains can conceive of various modifications or corrections within the scope of the technical idea described in the claims, and these are of course understood to belong to the technical scope of the present invention.

Explanation of Reference Numerals

[0032] 10... System, 110... Sensor, 120... Sensor array, 130... Sensor control unit, 200... Signal processing device, 210... Communication interface, 220... Buffer memory, 230... Arithmetic unit, 231... Transmission determination unit, 232... Image processing unit, 233... Region designation unit, 240... Storage unit, 241... Transmission determination rule, 300... Sensor device, 400... Sensor device, 410... RGB sensor.

Claims

1. Output from an event-driven vision sensor including a plurality of sensors constituting a sensor array, for event signals generated at each of the subject part and the part other than the subject part, determines whether to transmit based on the position information of each sensor in the sensor array A transmission determination unit to be performed; An image processing unit that executes image processing based on the transmitted event signal, and The transmission determination unit is a signal processing device that determines whether to transmit the event signal to the image processing unit based on different criteria according to the required degree of reduction in data transmission amount in the processing by the image processing unit.

2. The transmission determination unit determines whether to transmit the event signal based on a score calculated by integrating the number of the event signals for each segment defined in the sensor array and including the plurality of sensors, according to claim 1. The signal processing device described.

3. The transmission determination unit attenuates or resets the score at predetermined time intervals, according to claim 2. The signal processing device described.

4. The segment is defined by combining a boundary line parallel to the direction in which the sensors are arranged and a boundary line obliquely intersecting the direction, according to claim 2 or claim 3. The signal processing device described.

5. Further includes a region designating unit that designates a region based on the result of the image processing, The transmission determination unit determines whether to transmit the event signal output from the sensor having the position information in the designated region, according to any one of claims 1 to 4. The signal processing device described.

6. The region designating unit designates a region that at least partially overlaps the subject or includes the subject, according to claim 5. The signal processing device described.

7. The image processing unit executes tracking of the subject or calculation of optical flow, The region designating unit designates a region corresponding to a target region (ROI) in the tracking or the optical flow calculation, according to claim 5 or claim 6. The signal processing device described.

8. An event-driven vision sensor including a plurality of sensors constituting a sensor array, An image processing unit that executes image processing based on an event signal output from the vision sensor and an image signal output from an RGB sensor different from the vision sensor, A region designating unit that designates a region based on the result of the image processing executed based on the image signal comprising The vision sensor is a sensor device configured to output the event signal limited to the designated region.

9. The sensor device according to claim 8, wherein the region designating unit designates a region that at least partially overlaps with the subject or includes the subject.

10. The image processing unit executes tracking of a subject or calculation of an optical flow, The sensor device according to claim 8 or claim 9, wherein the region designating unit designates a region corresponding to a target region (ROI) in the tracking or the optical flow calculation.

11. The sensor device according to any one of claims 8 to 10, further comprising a transmission determination unit that determines whether to transmit the event signal to the image processing unit based on the position information of each sensor in the sensor array for the event signal generated in each of the subject portion and the portion other than the subject portion.

12. The transmission determination unit according to claim 11 determines whether to transmit the event signal based on a score calculated by integrating the number of event signals for each segment including a plurality of the sensors defined in the sensor array.

13. A signal processing method including a step of determining whether to transmit to the image processing unit based on the position information of each sensor in the sensor array according to different criteria according to the degree of requirement for reduction of data transmission amount in the image processing based on the event signal in the image processing unit for the event signals generated in each of the subject portion and the portion other than the subject portion, which are output from an event-driven vision sensor including a plurality of sensors constituting a sensor array.

14. A program for causing a computer to realize a function of determining whether to transmit to the image processing unit based on the position information of each sensor in the sensor array according to different criteria according to the degree of requirement for reduction of data transmission amount in the image processing based on the event signal in the image processing unit for the event signals generated in each of the subject portion and the portion other than the subject portion, which are output from an event-driven vision sensor including a plurality of sensors constituting a sensor array.

Images