Data processing device, data processing method and program

Abstract

The present technology relates to an image data processing device, an image data processing method, and a program capable of obtaining appropriate frame data. A frame data generation unit generates: first frame data, which is based on event data, the event data indicates a change in an electrical signal of a pixel that performs photoelectric conversion and generates an electrical signal, and the event data occurs within a first cumulative time from the start time of the first frame generation to the end time of the first frame generation; and second frame data, which is based on event data occurring within a second cumulative time from the start time of the second frame generation to the end time of the second frame generation. A first frame period is set and provided to the frame data generation unit, the first frame period being a period from the start of the first frame generation to the start of the second frame generation. For example, when frame data is generated based on event data output from a dynamic vision sensor (DVS: Dynamic Vision Sensor), the present technology can be applied.

Description

Technical Field

[0001] The present invention relates to a data processing device, a data processing method and a program. In particular, the present technology relates to a data processing device, a data processing method and a program that can obtain appropriate frame data when generating frame data (for example, the frame data is image data in a frame format) based on event data of the occurrence of an event indicating a change in an electrical signal of a pixel. Background Art

[0002] People are developing image sensors that output event data indicating the occurrence of an event when an event occurs, which assumes a change in pixel brightness as an event. In this regard, an image sensor that captures an image in synchronization with a vertical synchronization signal and outputs frame data as image data in a frame format may be referred to as a synchronous image sensor. On the other hand, an image sensor that outputs event data does not output the event data in synchronization with a vertical synchronization signal. Therefore, it may be referred to as an asynchronous image sensor. An asynchronous image sensor is referred to as, for example, a dynamic vision sensor (DVS: dynamic vision sensor).

[0003] Since it is difficult to use the event data output by the DVS as it is for existing image processing, the event data is converted into frame data for image processing. As a method of converting event data into frame data, that is, a method of generating frame data from event data, a method of aggregating event data within a specific time period into frames or aggregating a specific amount of event data into frames has been developed (for example, see Patent Document 1).

[0004] Citation List

[0005] Patent Literature

[0006] Patent Document 1: Japanese Unexamined Patent Application Publication No. 2017-530467 Summary of the invention

[0007] Problems to be solved by the present invention

[0008] When frame data as image data is generated in a frame format, it is desirable to obtain appropriate frame data according to event data and use it for subsequent image processing or the like.

[0009] The present invention has been made in view of such circumstances, and aims to enable acquisition of appropriate frame data.

[0010] Solution to the problem

[0011] According to the first data processing device of the present invention, the first data processing device includes: a frame data generating unit, which is constructed to generate first frame data and second frame data, the first frame data is based on event data indicating changes in the electrical signal of a pixel that generates an electrical signal by performing photoelectric conversion and occurs within a first cumulative time from the start time of the first frame generation to the end time of the first frame generation, and the second frame data is based on event data occurring within a second cumulative time from the start time of the second frame generation to the end time of the second frame generation; and a frame period setting unit, which is constructed to set a first frame period and provide the first frame period to the frame data generating unit, the first frame period being a period from the start time of the first frame generation to the start time of the second frame generation.

[0012] The first data processing device according to the present invention generates first frame data based on event data indicating a change in an electrical signal of a pixel that generates an electrical signal by performing photoelectric conversion and occurs within a first accumulation time from the first frame generation start time to the first frame generation end time, and second frame data based on event data occurring within a second accumulation time from the second frame generation start time to the second frame generation end time. In addition, a first frame period is set and provided to the frame data generation unit, the first frame period being a period from the first frame generation start time to the second frame generation start time.

[0013] The second data processing device or the first program according to the present invention is a data processing device or a program that causes a computer to function as such a data processing device, wherein the data processing device includes: a frame data generating unit, which is constructed to generate frame data based on event data indicating a change in an electrical signal of a pixel that generates an electrical signal by performing photoelectric conversion, the event data occurring within a cumulative time from a start time of frame generation to an end time of frame generation; and a frame period setting unit, which is constructed to set a frame period as an interval between a start time of frame data generation of a first frame data and a start time of frame data generation of a second frame data, and to provide the frame period to the frame data generating unit, the second frame data being generated continuously with the first frame data.

[0014] The first data processing method according to the present invention is the following data processing method, which includes: a frame data generating unit generates frame data based on event data indicating a change in an electrical signal of a pixel that generates an electrical signal by performing photoelectric conversion, and the event data occurs within a cumulative time from a start time of frame generation to an end time of frame generation; and a frame period is set as an interval between a start time of frame data generation of a first frame data and a start time of frame data generation of a second frame data, and the frame period is provided to the frame data generating unit, and the second frame data is generated continuously with the first frame data.

[0015] According to the second data processing device, the first data processing method, and the first program of the present invention, frame data is generated based on event data indicating a change in an electrical signal of a pixel that generates an electrical signal by performing photoelectric conversion, the event data occurring within a cumulative time from a frame generation start time to a frame generation end time. In addition, a frame period that is an interval between a frame data generation start time of the first frame data and a frame data generation start time of the second frame data is set, and the frame period is provided to the frame data generation unit, and the second frame data is generated continuously with the first frame data.

[0016] The third data processing device or the second program according to the present invention is a data processing device or a program that enables a computer to function as such a data processing device, wherein the data processing device includes: a frame data generating unit, which is constructed to generate first frame data and second frame data in a manner that overlaps the first accumulation time and the second accumulation time, the first frame data being based on event data indicating a change in the electrical signal of a pixel that generates an electrical signal by performing photoelectric conversion and occurring within the first accumulation time from the start time of the first frame generation to the end time of the first frame generation, and the second frame data being based on the event data occurring within the second accumulation time from the start time of the second frame generation to the end time of the second frame generation.

[0017] The second data processing method according to the present invention is a data processing method comprising: generating first frame data and second frame data in a manner that overlaps a first accumulation time and a second accumulation time, the first frame data being based on event data indicating a change in an electrical signal of a pixel that generates an electrical signal by performing photoelectric conversion and occurring within the first accumulation time from a start time of first frame generation to an end time of first frame generation, and the second frame data being based on the event data occurring within the second accumulation time from a start time of second frame generation to an end time of second frame generation.

[0018] According to the third data processing device, the second data processing method and the second program of the present invention, first frame data and second frame data are generated in a manner that the first accumulation time and the second accumulation time overlap, the first frame data being based on event data indicating changes in the electrical signal of a pixel that generates an electrical signal by performing photoelectric conversion and occurring within the first accumulation time from the start time of first frame generation to the end time of first frame generation, and the second frame data being based on the event data occurring within the second accumulation time from the start time of second frame generation to the end time of second frame generation.

[0019] The fourth data processing device according to the present invention is a data processing device as follows: the data processing device generates frame data based on event data indicating changes in the electrical signal of a pixel that generates an electrical signal by performing photoelectric conversion, and the event data occurs within the cumulative time from the start time of frame generation to the end time of frame generation, wherein the number of frame data generated per unit time varies according to the number of event data per unit time.

[0020] According to a fourth data processing device of the present invention, frame data is generated based on event data indicating a change in an electrical signal of a pixel that generates an electrical signal by performing photoelectric conversion, the event data occurring within a cumulative time from a frame generation start time to a frame generation end time. The number of generated frame data per unit time varies according to the number of event data per unit time.

[0021] The fifth data processing device according to the present invention is a data processing device comprising a data processing unit, the data processing unit comprising: an event generating unit having pixels that generate electrical signals by performing photoelectric conversion, and being configured to generate event data indicating the occurrence of an event, the event being a change in the electrical signal of the pixel; a frame data generating unit, being configured to generate frame data based on the event data occurring within a cumulative time from a start moment of frame generation to an end moment of frame generation; and a frame period setting unit, being configured to set a frame period as an interval between a start moment of frame data generation of a first frame data and a start moment of frame data generation of a second frame data, a frame amount and provide the frame period to the frame data generating unit, the second frame data being generated continuously with the first frame data.

[0022] The third data processing method according to the present invention is the following data processing method, which includes: generating event data representing the occurrence of an event, wherein the event is a change in the electrical signal of a pixel that generates an electrical signal by performing photoelectric conversion; generating frame data based on the event data occurring within the cumulative time from the start time of frame generation to the end time of frame generation; and setting a frame period as the interval between the start time of frame data generation of the first frame data and the start time of frame data generation of the second frame data, and providing the frame period to the frame data generation unit, and the second frame data is generated continuously with the first frame data.

[0023] According to the fifth data processing device and the third data processing method of the present invention, event data indicating the occurrence of an event is generated by a frame data generating unit, the event being a change in the electrical signal of a pixel that generates an electrical signal by performing photoelectric conversion, and the frame data is generated based on the event data occurring within the cumulative time from the frame generation start time to the frame generation end time. In addition, a frame period is set as an interval between the frame data generation start time of the first frame data and the frame data generation start time of the second frame data, and the frame period is provided to the frame data generating unit, and the second frame data is generated continuously with the first frame data.

[0024] The data processing device may be an independent unit or an internal block included in a device or a single unit. In addition, the data processing device may be constructed as a single-chip semiconductor chip or a module.

[0025] The program can be distributed by transmitting the program via a transmission medium or by recording the program on a recording medium. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a diagram showing a structural example of an embodiment of a data processing chip to which the present invention is applied.

[0027] Figure 2 is a flow chart for explaining the operation of the data processing chip.

[0028] Figure 3 2 is a block diagram showing a configuration example of the event generating unit 21 .

[0029] Figure 4 : is a block diagram showing a configuration example of the pixel array section 31 .

[0030] Figure 5 is a circuit diagram showing a structural example of the pixel 51.

[0031] Figure 6 is a block diagram showing a configuration example of the event detector 52 .

[0032] Figure 7 : is a circuit diagram showing a configuration example of the current-voltage converter 71 .

[0033] Figure 8 2 is a circuit diagram showing a configuration example of the subtractor 73 and the quantizer 74 .

[0034] Fig. 9 2 is a block diagram showing a configuration example of the data processing unit 22 .

[0035] Fig.10 This is a diagram for explaining the outline of event data output by the event generating unit 21 .

[0036] Fig.11 This is a diagram for explaining an overview of a method of generating frame data from event data.

[0037] Fig.12 FIG. 1 is a diagram showing a first method of setting a frame width and a frame interval.

[0038] Fig.13 FIG. 1 is a diagram showing a second method of setting the frame width and the frame interval.

[0039] Fig.14 It is a diagram showing a third method of setting the frame width and the frame interval.

[0040] Fig.15 : is a block diagram showing a configuration example of the data generating unit 113 .

[0041] Fig.16 1 is a diagram for explaining an example of control of the memory 121 by the memory control unit 122 .

[0042] Fig.17 is a flowchart for explaining an example of processing performed by the data processing unit 22 .

[0043] Fig.18 : is a flowchart for explaining an example of data generation processing performed by the data generation unit 113 .

[0044] Fig.19 It is a diagram showing a fourth method of setting the frame width and the frame interval.

[0045] Fig. 20 It is a diagram showing a fifth method of setting the frame width and the frame interval.

[0046] Fig.21 It is a diagram showing a sixth method of setting the frame width and the frame interval.

[0047] Fig. 22 This is a block diagram showing a configuration example of an embodiment of a computer to which the present invention is applied.

[0048] Fig.23 is a block diagram showing a schematic configuration example of a vehicle control system.

[0049] Fig.24 It is an explanatory diagram showing an example of the installation positions of the vehicle exterior information detection unit and the imaging unit.

[0050] Fig.25 This is a diagram showing a list of setting methods for setting the frame interval and the frame width.

[0051] Fig.26 : is a diagram showing an example of frame data generated when one frame width is set for the entire frame of the frame data.

[0052] Fig. 27 is a diagram showing an example of how a frame is divided into segmented areas.

[0053] Fig.28 : is a diagram showing an example of frame data generated when the frame width is set for each divided region.

[0054] Fig.29 : is a flowchart showing an example of a setting method in the case where the number of event data is set as a frame width for each divided area.

[0055] Fig.30 is a diagram showing an example of the evaluation value F.

[0056] Fig.31 is a diagram showing another configuration example of the event detector 52 .

[0057] Fig.32 : is a block diagram showing a structural example of a scanning type imaging device. DETAILED DESCRIPTION

[0058] <Embodiment of a data processing chip to which the present invention is applicable>

[0059] Figure 1 This is a diagram showing a configuration example of an embodiment of a data processing chip to which the present technology is applied.

[0060] The data processing chip is a single-chip semiconductor chip and includes a plurality of dies (substrates), namely, a sensor die (substrate) 11 and a logic die 12 stacked on each other. In addition, the data processing chip may include one die or three or more dies stacked on each other.

[0061] exist Figure 1 In the data processing chip shown, the sensor die 11 includes (circuitry of) an event generation unit 21, and the logic die 12 includes a data processing unit 22. In addition, a portion of the event generation unit 21 may be included in the logic die 12. A portion of the data processing unit 22 may also be included in the sensor die 11.

[0062] The event generation unit 21 has pixels that generate electrical signals by photoelectrically converting incident light, and generates event data representing the occurrence of an event as a change in the electrical signal of the pixel. The event generation unit 21 provides the event data to the data processing unit 22. In other words, for example, similar to a synchronous image sensor, the event generation unit 21 performs imaging that generates electrical signals by photoelectrically converting incident light of the pixel. However, the event generation unit 21 does not generate image data in a frame format, or, while generating image data in a frame format, also generates event data representing the occurrence of an event as a change in the electrical signal of the pixel.

[0063] The data processing unit 22 performs data processing according to the event data supplied from the event generating unit 21 , and outputs a data processing result as a result of the data processing.

[0064] Figure 2 It is used to illustrate Figure 1 Flowchart of the operation of the data processing chip in FIG.

[0065] In step S1 , if a change in the electrical signal of a pixel as an event occurs, the event generation unit 21 generates event data and supplies it to the data processing unit 22 .

[0066] In step S2, the data processing unit 22 performs data processing for converting event data into frame data, that is, data processing such as generation of frame data, etc., based on the event data supplied from the event generating unit 21. Then, the data processing unit 22 outputs the result obtained by the data processing.

[0067] <Configuration Example of Event Generating Unit 21>

[0068] Figure 3 It is shown Figure 1 A block diagram of a construction example of the event generation unit 21 in FIG.

[0069] The event generation unit 21 includes a pixel array section 31 , a driver 32 , an arbiter 33 , and an output unit 34 .

[0070] The pixel array section 31 includes a plurality of pixels arranged in a two-dimensional grid therein. In addition, the pixel array section 31 includes a plurality of divided pixel blocks each having a predetermined number of pixels. A group of pixels or pixel blocks arranged in the horizontal direction is called a "row", and a group of pixels or pixel blocks arranged in a direction perpendicular to the row is called a "column".

[0071] When a change in photocurrent exceeding a predetermined threshold is used as an electrical signal generated by photoelectric conversion of the pixel, the pixel block detects the change in photocurrent as an event. Upon detecting an event, the pixel block outputs a request to the arbiter 33 to output event data indicating the occurrence of the event.

[0072] The driver 32 supplies a control signal to the pixel array section 31 to drive the pixel array section 31 .

[0073] The arbitrator 33 arbitrates requests from the pixel blocks included in the pixel array section 31 and returns a response indicating whether the output of event data is permitted or not to the pixel blocks. Upon receiving a response indicating that the output of event data is permitted, the pixel blocks output the event data to the output unit 34.

[0074] The output unit 34 performs necessary processing on the event data output by the pixel blocks included in the pixel array section 31, and supplies the result to the data processing unit 22 ( Figure 1 ).

[0075] In this specification, the change of the photocurrent as the electrical signal of the pixel can be regarded as the change of the brightness of the pixel. Therefore, the event can also be referred to as the change of the brightness of the pixel (the change of the brightness exceeding the threshold).

[0076] The event data indicating the occurrence of an event includes position information (such as coordinates, etc.) indicating at least the position of a pixel block where a brightness change as an event occurs. In addition, the event data may include the polarity (positive or negative) of the brightness change.

[0077] For a series of event data output from a pixel block when an event occurs, as long as the intervals between the event data items are kept as when the event occurs, it can be said that the event data implicitly includes time information indicating the time (relative time) when the event occurs. However, if the intervals between the event data items are not kept as when the event occurs because the event data is stored in a memory, etc., the time information implicitly included in the event data will be lost. Therefore, the output unit 34 includes time information such as a timestamp in the event data, which indicates the time (relative time) of the event before the intervals between the event data items are not kept as when the event occurs. Before the time information implicitly included in the event data is lost, the processing of including the time information in the event data can be performed in any block other than the output unit 34.

[0078] <Structural Example of Pixel Array Section 31>

[0079] Figure 4 It is shown Figure 3 A block diagram of a structural example of the pixel array unit 31 in FIG.

[0080] The pixel array section 31 has a plurality of pixel blocks 41. The pixel block 41 includes a plurality of (I×J) pixels 51 arranged in I rows and J columns (where I and J are integers) and an event detector 52. The plurality of pixels 51 in the pixel block 41 share the event detector 52.

[0081] The pixel 51 receives incident light from the subject and performs photoelectric conversion to generate a photocurrent as an electrical signal. The pixel 51 supplies the photocurrent to the event detector 52 under the control of the driver 32 .

[0082] The event detector 52 detects a change in the photocurrent from each pixel 51 that exceeds a predetermined threshold as an event. When an event is detected, the event detector 52 provides a request for outputting event data indicating the occurrence of the event to the arbitrator 33. Then, when a response allowing the output of the event data is received from the arbitrator 33 as a response to the request, the event detector 52 outputs the event data to the output unit 34.

[0083] <Structural Example of Pixel 51>

[0084] Figure 5 It is shown Figure 4 A circuit diagram of a structural example of pixel 51 in FIG.

[0085] The pixel 51 includes a photoelectric transducer 61 and a transfer transistor 62 .

[0086] For example, the photoelectric converter 61 includes a photodiode (PD) and receives incident light and performs photoelectric conversion to generate electric charges.

[0087] For example, the transfer transistor 62 includes an N (Negative) type metal-oxide-semiconductor (MOS) field-effect transistor (FET). The transfer transistor 62 included in the n-th pixel 51 of the pixels 51 of I rows×J columns forming the pixel block 41 is driven by the driver 32 ( Figure 3 ) is turned on / off by a control signal TRGn supplied from the photoelectric converter 61. By turning on the transfer transistor 62, the charge generated by the photoelectric converter 61 is transferred (supplied) to the event detector 52 as a photocurrent.

[0088] The pixels 51 forming the I rows×J columns of the pixel block 41 are connected to the event detector 52 included in the pixel block 41 via the node 60. Therefore, the photocurrent generated by the pixel 51 (the photoelectric transducer 61 thereof) is supplied to the event detector 52 via the node 60. Therefore, the event detector 52 is supplied with the sum of the photocurrents of all the pixels 51 in the pixel block 41.

[0089] The event detector 52 detects the event according to the Figure 3 ) as a control signal, and detects a change in the photocurrent (total) supplied from the pixel 51 as an event. When a change in the photocurrent is detected as an event, that is, a brightness change, the event detector 52 outputs event data indicating the occurrence of the event.

[0090] In the present specification, when an event is detected, the event detector 52 is capable of outputting a voltage corresponding to a photocurrent provided from the pixel 51 as a pixel signal corresponding to the incident light incident on the pixel 51 separately from the event data. By simultaneously turning on the transfer transistors 62 of a plurality of pixels 51, the sum of the pixel signals of the plurality of pixels 51 can be output as an output of the pixel signal. In addition, by sequentially turning on the transfer transistors 62 of the pixels 51 included in the pixel block 41, it is possible to output a pixel signal for each pixel 51. In the case of outputting the pixel signal of the pixel 51, an AD converter configured to perform analog-to-digital conversion on the pixel signal is provided in the event generation unit 21.

[0091] like Figure 4 As shown, the pixel block 41 having a plurality of pixels 51 sharing the event detector 52 enables a reduction in the number of event detectors 52, compared with a case where one event detector 52 is provided for one pixel 51. Therefore, the size of the pixel array section 31 can be reduced.

[0092] Furthermore, the pixel block 41 may include one pixel 51 instead of a plurality of pixels 51 .

[0093] <Configuration Example of Event Detector 52>

[0094] Figure 6 It is shown Figure 4 A block diagram of a construction example of the event detector 52 in FIG.

[0095] The event detector 52 includes a current-voltage converter 71 , a buffer 72 , a subtractor 73 , a quantizer 74 , and a transmission unit 75 .

[0096] The current-voltage converter 71 converts the photocurrent (sum) from the pixel 51 into a voltage corresponding to the logarithm of the photocurrent (hereinafter, also referred to as photovoltage), and supplies it to the buffer 72 .

[0097] The buffer 72 buffers (corrects) the photovoltage from the current-voltage converter 71 , and supplies it to the subtractor 73 .

[0098] The subtractor 73 calculates the difference between the current photovoltage and the photovoltage at a timing slightly different from the current time, and supplies a difference signal corresponding to the difference to the quantizer 74. This calculation is performed at a timing according to the row drive signal from the driver 32 serving as a control signal.

[0099] The quantizer 74 quantizes the difference signal from the subtractor 73 into a digital signal, and supplies the quantized value of the difference signal to the transmission unit 75 .

[0100] The transmission unit 75 transmits (outputs) the event data to the output unit 34 according to the quantization value of the difference signal from the quantizer 74. Specifically, the transmission unit 75 provides a request for outputting the event data to the arbitrator 33. Then, when a response permitting outputting the event data is received from the arbitrator 33 as a response to the request, the transmission unit 75 outputs the event data to the output unit 34.

[0101] <Structural Example of Current-Voltage Converter 71>

[0102] Figure 7 It is shown Figure 6 A circuit diagram of a construction example of the current-to-voltage converter 71 in FIG.

[0103] The current-voltage converter 71 includes transistors 81 to 83. As examples of the transistors 81 and 83, N-type MOS FETs may be employed, and as an example of the transistor 82, P-type MOS FETs may be employed.

[0104] The source of transistor 81 is connected to the gate of transistor 83, and the photocurrent from pixel 51 is supplied to the connection point between the source of transistor 81 and the gate of transistor 83. The drain of transistor 81 is connected to power supply VDD, and its gate is connected to the drain of transistor 83.

[0105] The source of transistor 82 is connected to the power supply VDD, and the drain thereof is connected to a connection point between the gate of transistor 81 and the drain of transistor 83. A predetermined bias voltage Vbias is applied to the gate of transistor 82. Transistor 82 supplies transistor 83 with a constant current.

[0106] The source of the transistor 83 is grounded.

[0107] In the current-voltage converter 71, the transistor 81 is connected to the power supply VDD and functions as a source follower. The photocurrent from the pixel 51 is converted into a photovoltage corresponding to the logarithm of the photocurrent by the transistor 81 used as a source follower and the transistor 83 having a gate connected to the source of the transistor 81.

[0108] The photovoltage is supplied from the connection point between the gate of transistor 81 and the drain of transistor 83 to buffer 72 ( Figure 6 ).

[0109] <Configuration Example of Subtractor 73 and Quantizer 74>

[0110] Figure 8 It is shown Figure 6 A circuit diagram of a construction example of the subtractor 73 and the quantizer 74 in FIG.

[0111] The subtractor 73 includes a capacitor 91, an operational amplifier 92, a capacitor 93, and a switch 94. The quantizer 74 includes a comparator 101.

[0112] One end of the capacitor 91 is connected to the buffer 72 ( Figure 6 ) output terminal, and the other end is connected to the input terminal of the operational amplifier 92.

[0113] An output terminal of the operational amplifier 92 is connected to a non-inverting input terminal (+) of the comparator 101 .

[0114] One end of the capacitor 93 is connected to the input terminal of the operational amplifier 92 , and the other end is connected to the output terminal of the operational amplifier 92 .

[0115] The switch 94 is connected to the capacitor 93 to turn on / off the connection with both ends of the capacitor 93. The switch 94 turns on / off the connection with both ends of the capacitor 93 by turning on / off according to the row drive signal from the driver 32 serving as a control signal.

[0116] The connection point between the capacitor 91 and the operational amplifier 92 is virtually grounded, and the potential of the virtual ground is set to zero for convenience.

[0117] Assume that when switch 94 is turned on, the buffer 72 ( Figure 6 ) side is Vinit and the capacitance of capacitor 91 is C1, then the charge Qinit accumulated in capacitor 91 when switch 94 is turned on is expressed by the following equation. On the other hand, when switch 94 is turned on, both ends of capacitor 93 are short-circuited, so the charge accumulated in capacitor 93 is zero.

[0118] Qinit=C1×Vinit…(1)

[0119] Then, assume that switch 94 is open and at this time the buffer 72 ( Figure 6 ) side is Vafter, then the charge Qafter accumulated in capacitor 91 is expressed by the following formula.

[0120] Qafter=C1×Vafter…(2)

[0121] On the other hand, assuming that the capacitance of the capacitor 93 is C2 and the output voltage of the operational amplifier 92 is Vout, the charge Q2 stored in the capacitor 93 is expressed by the following equation.

[0122] Q2=C2×Vout…(3)

[0123] The total charge amount which is the sum of the charge of the capacitor 91 and the charge of the capacitor 93 does not change before and after the switch 94 is turned off, whereby the following equation is obtained.

[0124] Qinit=Qafter+Q2…(4)

[0125] By substituting equations (1) to (3) into equation (4) and transforming it, the following equation can be obtained.

[0126] Vout=-(C1 / C2)×(Vafter-Vinit)…(5)

[0127] Formula (5) shows that in the subtractor 73, subtraction is performed between the photovoltage Vafter and Vinit, specifically, a difference signal (Vout) corresponding to the difference Vafter-Vinit between the photovoltage Vafter and Vinit is calculated. According to formula (5), the subtraction gain of the subtractor 73 is C1 / C2. It is generally desired to maximize the gain, so it is preferred to design C1 to be larger and C2 to be smaller. On the other hand, if C2 is too small, it is likely to increase kT / C noise, resulting in degradation of noise characteristics. Therefore, the capacity reduction of C2 is limited to an allowable noise range. In addition, since an event detector 52 with a subtractor 73 is provided for each pixel block 41, the capacities C1 and C2 have area limitations. These factors are taken into consideration to determine the values ​​of the capacities C1 and C2.

[0128] The comparator 101 compares the difference signal from the subtractor 73 with a predetermined threshold voltage Vth applied to the inverting input terminal (-). For example, the comparator 101 outputs an H (High) level or an L (Low) level as a quantized value of the difference signal to the transmission unit 75, wherein the H (High) level or the L (Low) level indicates a result obtained by comparing the difference signal and the threshold voltage Vth.

[0129] When a brightness change as an event is identified based on the quantized value of the difference signal from the quantizer 74 , that is, when the difference signal (Vout) is greater than the threshold voltage Vth, the transmission unit 75 outputs event data indicating the occurrence of the event to the output unit 34 .

[0130] The output unit 34 outputs the event data obtained from the transmission unit 75 together with the contained position information, time information, and, if necessary, the polarity of the brightness change as the event. The position information indicates the position of the pixel 51 (the pixel block 41 having the pixel) where the event indicated by the event data occurs, and the time information indicates the time when the event occurs.

[0131] For example, event data including position information of the pixel 51 where an event occurs, time information indicating the time when the event occurs, and polarity of brightness change as the event may adopt a data format called Address Event Representation (AER).

[0132] In addition, the gain A of the entire event detector 52 is expressed by the following equation, where the gain of the current-to-voltage converter 71 is CG log , and the gain of buffer 72 is "1".

[0133] A=CG log C1 / C2(∑i photo _n)…(6)

[0134] In the above formula, i photo _n represents the photocurrent of the n-th pixel 51 among the pixels 51 of I rows×J columns forming the pixel block 41. "Σ" in the formula (6) represents the sum to n among integers from 1 to I×J.

[0135] In addition, for example, by providing a color filter such as a color filter that transmits predetermined light, the pixel 51 is able to receive any light as incident light. For example, in the case where the pixel 51 receives visible light as incident light, the event data indicates the occurrence of a change in the pixel value of the image mapped with a visually identifiable subject. In addition, for example, in the case where the pixel 51 receives infrared rays or millimeter waves, etc. for distance measurement as incident light, the event data indicates the occurrence of a change in the distance to the subject. In addition, for example, in the case where the pixel 51 receives infrared rays for measuring temperature as incident light, the event data indicates the occurrence of a change in the temperature of the subject. It is assumed that in the present embodiment, the pixel 51 receives visible light as incident light.

[0136] <Configuration Example of Data Processing Unit 22>

[0137] Fig. 9 It is shown Figure 1 A block diagram of a configuration example of the data processing unit 22 in FIG.

[0138] exist Fig. 9 In the embodiment, the data processing unit 22 includes a frame interval setting unit 111 , a frame width setting unit 112 , a data generating unit 113 , and an image processing unit 114 .

[0139] For example, the frame interval setting unit 111 sets the frame interval according to a user operation or the like, and provides it to the data generating unit 113. The frame interval represents the interval between frames of the frame data generated according to the event data, and the frame interval can be specified based on time or the number of event data, and thus set. In this specification, the frame interval set by the frame interval setting unit 111 is also referred to as a set frame interval.

[0140] For example, the frame width setting unit 112 sets the frame width according to a user operation or the like, and provides it to the data generating unit 113. The frame width represents the width of time of event data used to generate frame data of one frame, and similarly to the frame interval, the frame width can be specified based on time or the number of event data, and thus the frame width is set. In this specification, the frame width set by the frame width setting unit 112 is also referred to as a set frame width.

[0141] The data generating unit 113 converts the event data into frame data by generating frame data as image data in a frame format according to the event data output from the event generating unit 21 to the data processing unit 22. The data generating unit 113 supplies the frame data to the image processing unit 114.

[0142] The data generating unit 113 generates frame data according to the set frame interval supplied from the frame interval setting unit 111 and the set frame width supplied from the frame width setting unit 112 .

[0143] In other words, the data generating unit 113 generates frame data for each set frame interval according to event data within a set frame width from the start of the set frame interval.

[0144] The image processing unit 114 performs image processing using the frame data supplied from the data generating unit 113, for example, performs recognition processing etc. on the frame data. The image processing unit 114 outputs a result obtained by performing the image processing (image processing result) as a data processing result.

[0145] The image processing unit 114 may employ any device capable of processing frame data. For example, the image processing unit 114 may employ a device configured to perform image recognition using a neural network or an image display device including a display and software as a viewer.

[0146] In addition, a predetermined value may be adopted as the frame width. In this case, the data processing unit 22 may be configured not to include the frame width setting unit 112.

[0147] In addition, if Fig. 9As shown, in the case where the recognition processing of the frame data is performed and the reliability of the recognition result of the recognition processing can be obtained, the image processing unit 114 can provide (feed back) the reliability of the recognition result of the recognition processing from the image processing unit 114 to the frame interval setting unit 111 and the frame width setting unit 112. In this case, the frame interval setting unit 111 and the frame width setting unit 112 can set the frame interval and the frame width according to the reliability of the recognition result.

[0148] In other words, the frame interval setting unit 111 and the frame width setting unit 112 may first set two default values ​​of the frame interval and the frame width. Then, the image processing unit 114 obtains two reliabilities of the recognition result by performing recognition processing on the frame data generated according to the two default values ​​of the frame interval and the frame width. According to the change of such reliability, the frame interval setting unit 111 and the frame width setting unit 112 may respectively use a search algorithm such as a gradient method to set a new frame interval and frame width to improve the reliability of the recognition result.

[0149] According to the change in the reliability of the recognition results obtained for the new frame interval and frame width, the frame interval setting unit 111 and the frame width setting unit 112 can repeatedly set additional new frame intervals and frame widths, respectively, to improve the reliability of the recognition results until the reliability of the recognition results is maximized (maximum).

[0150] As described above, the frame interval setting unit 111 and the frame width setting unit 112 can set the frame interval and the frame width, respectively, according to the result (image processing result) obtained by the image processing performed by the image processing unit 114 .

[0151] In addition, it can be considered that Fig. 9 , the data generating unit 113 functions as a frame data generating unit configured to generate first frame data and second frame data, the first frame data being based on event data indicating a change in an electrical signal of a pixel generating an electrical signal by performing photoelectric conversion, the event data occurring within a first accumulation time from a first frame generation start time to a first frame generation end time, and the second frame data being based on event data occurring within a second accumulation time from a second frame generation start time to a second frame generation end time. The first accumulation time and the second accumulation time are frame widths.

[0152] It can be considered that the frame interval setting unit 111 functions as a frame period setting unit that sets a first frame period from the first frame generation start time to the second frame generation start time and provides the first frame period to the frame data generation unit. The first frame period is a frame interval.

[0153] The data generating unit 113 used as the frame data generating unit can further generate the third frame data based on the event data within the third accumulation time from the third frame generation start time to the third frame generation end time. The frame interval setting unit 111 used as the frame period setting unit can set the first frame period and the second frame period to be different. The second frame period ranges from the second frame generation start time to the third frame generation start time. The second frame period is the frame interval.

[0154] The frame width setting unit 112 functions as an accumulation time setting unit that sets a first accumulation time and a second accumulation time and supplies both to the data generating unit 113 that functions as a frame data generating unit.

[0155] The frame width setting unit 112 serving as an accumulation time setting unit is capable of setting the first accumulation time and the second accumulation time to be different from each other.

[0156] It can be considered that the data generating unit 113 functions as a frame data generating unit that generates frame data based on event data indicating a change in an electrical signal of a pixel that undergoes photoelectric conversion to generate an electrical signal. The event data occurs within a cumulative time from the frame generation start time to the frame generation end time. The cumulative time is the frame width.

[0157] The frame interval setting unit 111 functions as a frame period setting unit configured to set a frame period indicating an interval between a frame data generation start time of the first frame data and a frame data generation start time of the second frame data, the second frame data being generated continuously with the first frame data, and provide the frame period to the frame data generating unit. The frame period is a frame interval.

[0158] The frame interval setting unit 111 as a frame period setting unit can set the frame period in terms of time.

[0159] When the frame cycle is set by time, the frame width setting unit 112 as the accumulation time setting unit can set the accumulation time by time or according to the number of event data. In addition, it is possible to switch between setting the accumulation time by time and setting the accumulation time according to the number of event data.

[0160] The frame interval setting unit 111 as a frame period setting unit can set the frame period according to the number of event data.

[0161] When the frame cycle is set according to the number of event data, the frame width setting unit 112 as the accumulation time setting unit can set the accumulation time by time or by the number of event data. In addition, it is possible to switch between setting the accumulation time by time and setting the accumulation time by the number of event data.

[0162] It can be considered that the data generating unit 113 functions as a frame data generating unit that generates the first frame data and the second frame data in such a manner that the first accumulation time and the second accumulation time can overlap. The first frame data is event data based on a change in an electrical signal of a pixel that performs photoelectric conversion to generate an electrical signal, and the first frame data is generated within a first accumulation time from the start time of the first frame generation to the end time of the first frame generation. The second frame data is based on event data generated within a second accumulation time from the start time of the second frame generation to the end time of the second frame generation. The overlap of the first accumulation time and the second accumulation time means that an overlapping area described later will be generated.

[0163] In the case where the frame period is set according to the number of event data, the number of frame data generated per unit time (the number of frame data generated by the data generating unit 113) varies according to the number of event data per unit time (e.g., 1 second). The greater the number of event data per unit time, the greater the number of frame data generated per unit time.

[0164] It can be considered that the data generating unit 113 functions as a frame data generating unit configured to generate frame data based on event data occurring within the cumulative time from the frame generation start time to the frame generation end time. It can be considered that the frame interval setting unit 111 functions as a frame cycle setting unit configured to set a frame cycle indicating an interval between the frame data generation start time of the first frame data and the frame data generation start time of the second frame data and provide the frame cycle to the frame data generating unit, and the second frame data is generated continuously with the first frame data.

[0165] As will be described later, the frame width setting unit 112 as the accumulation time setting unit can set the accumulation time in units of segmented areas in which a frame of frame data is divided into a plurality of areas. For example, the accumulation time setting unit can set the accumulation time based on an evaluation value indicating appropriateness of the accumulation time. More specifically, the accumulation time setting unit can set the accumulation time based on an evaluation value obtained using frame data generated based on event data within the accumulation time.

[0166] <Generation of Frame Data Based on Event Data>

[0167] Fig.10 Is used to illustrate the Figure 1 A diagram showing an overview of event data output by the event generation unit 21 in FIG.

[0168] As reference Fig. 9 As described above, the event generation unit 21 (the output unit 34) outputs, for example, a signal including time information t i , coordinates(x i ,y i) and event data of the polarity pi of the brightness change of the event. Time information t i Indicates the time point when an event occurs (hereinafter, also referred to as the event moment). i ,y i ) is used as the position information of pixel 51 where the event occurs.

[0169] Event data is output whenever an event occurs, which is different from image data (frame data) in a frame format that is output in frame cycles in synchronization with a vertical synchronization signal. Therefore, event data cannot be displayed as is on a display device such as a projector or input to a recognizer (classifier) ​​to be used for image processing. Event data needs to be converted into frame data.

[0170] In this manual, Fig.10 In the listed event data, the time t of an event i and the time t of the event adjacent to this event i+1 The interval between them does not have to be constant. In other words, the event time t i and event time t i+1 can be the same time or different times. However, suppose the event time t i and event time t i+1 With the formula t i <=t i+1 The relationship represented.

[0171] Fig.11 This is a diagram for explaining an overview of a method of generating frame data from event data.

[0172] exist Fig.11 In , in a three-dimensional space having an x-axis, a y-axis, and a time axis t, points indicating event data are plotted at a time t and a position (coordinates) (x, y) of an event included in the event data.

[0173] In other words, assuming that the position (x, y, t) in the three-dimensional space represented by the time t and position (x, y) of the event included in the event data is the spatiotemporal position of the event, then Fig.11 In , event data is plotted as points at the spatiotemporal location (x, y, t) of the event.

[0174] For example, the data generating unit 113 uses a predetermined time point as a generation start time to start the generation of frame data and starts generating frame data according to the event data. Examples of the predetermined time point include a time point when the generation of frame data is instructed by a user operation or a time point when the power of the data processing chip is turned on.

[0175] In the following, it is assumed that a rectangular parallelepiped of a set frame width in the direction of the time axis t for each set frame interval from the start of generation is referred to as a frame volume. For example, the size of the frame volume in the x-axis direction and the y-axis direction is equal to the number of pixel blocks 41 or pixels 51 in the x-axis direction and the y-axis direction.

[0176] The data generating unit 113 generates frame data of one frame for each set frame interval, based on the event data in the frame volume of the set frame width from the start of the set frame interval.

[0177] For example, pixels (pixel values) of the frame at the event position (x, y) included in the event data are set to white and pixels at other positions of the frame are set to a predetermined color such as gray, and thus frame data is generated.

[0178] In addition, the frame data may be generated taking into account the polarity included in the event data. For example, if the polarity is positive, the pixel may be set to white, and if the polarity is negative, the pixel may be set to black. Figure 5 As described above, in the case where the event detector 52 outputs event data and a pixel signal of the pixel 51, frame data can be generated based on the event data using the pixel signal of the pixel 51. In other words, the frame data can be generated by setting the pixel of the frame at the event position (x, y) included in the event data to the pixel signal of the pixel 51 at the position (x, y) and setting the pixels at other positions of the frame to a predetermined color such as gray.

[0179] In addition, a plurality of event data having different event times t but the same event position (x, y) may exist in the frame volume. In this case, for example, event data having the latest or oldest event time t can be given priority. In addition, in the case where the event data has polarity, the polarities of a plurality of event data having different event times t but the same event position (x, y) may be added, and a pixel value corresponding to the added value obtained by the addition may be set to the pixel at the event position (x, y).

[0180] In this specification, when the frame width and the frame interval are the same, the frame volumes are in contact with each other without a gap. In addition, when the frame interval is larger than the frame width, the frame volumes are arranged with a gap. When the frame width is larger than the frame interval, the frame volumes are arranged in a partially overlapping state.

[0181] Fig.12 FIG. 1 is a diagram showing a first method of setting a frame width and a frame interval.

[0182] exist Fig.12 In , event data is plotted with the horizontal axis being the event time t and the vertical axis being the event position (x, y).

[0183] In the first setting method, the frame width and the frame interval are set to have the same time as each other and a relatively short cycle by specifying the frame width and the frame interval based on time.

[0184] In the first setting method, the frame interval is short, so frame data with a larger frame rate can be obtained. If the frame rate of the frame data is greater than the frame rate of playback (for example, such as 30 frames per second (fps), etc.), the playback of the image associated with the frame data is slow playback.

[0185] In the first setting method, the frame width is the same as the frame interval of a short time, so the frame volume is relatively thin, and the amount of event data included in the frame volume tends to become small. Then, the image associated with the frame data generated by such a small amount of event data tends to become an image in which the subject cannot be clearly presented (difficult to see), and image recognition performed on such an image has difficulty in obtaining sufficient recognition performance.

[0186] Fig.13 FIG. 1 is a diagram showing a second method of setting the frame width and the frame interval.

[0187] exist Fig.13 In, similar to Fig.12 , the event data is plotted with the horizontal axis being the event time t and the vertical axis being the event position (x, y).

[0188] In the second setting method, the frame width and the frame interval are set to have the same time by specifying the frame width and the frame interval based on time. However, in the second setting method, the frame width and the frame interval are set to have a longer time than in the first setting method.

[0189] In the second setting method, the frame interval is longer than that of the first setting method, so the frame rate of the frame data generated based on the event data is lower than that of the first setting method. If the frame rate of the frame data is lower than the frame rate to be reproduced, the reproduction of the image related to the frame data is fast forward reproduction.

[0190] In the second setting method, the frame width is the same as the long frame interval, so the frame volume is relatively thick, and the amount of event data included in the frame volume tends to be easily increased to a certain extent. Then, the image associated with the frame data generated based on such a large amount of event data is an image that clearly presents the subject.

[0191] However, in the second setting method, for a subject that moves violently, the image associated with the frame data is often an image with obvious motion blur, making it difficult to visually recognize the fine movement of the subject. In addition, in the second setting method, the frame interval is long and the frame rate of the frame data is low. Therefore, the movement of the subject displayed in the image associated with the frame data is likely to become sluggish. In addition, image recognition of the image associated with the frame data is likely to obtain insufficient recognition performance of a moving object (moving subject).

[0192] Fig.14 It is a diagram showing a third method of setting the frame width and the frame interval.

[0193] exist Fig.14 In, similar to Fig.12 , the event data is plotted with the horizontal axis being the event time t and the vertical axis being the event position (x, y).

[0194] In this manual, Fig.14 The rectangles indicating the frame volumes are shown tilted to a certain extent to avoid overlapping of the rectangles as the frame volumes and making them difficult to visually identify. Fig.14 The reason for tilting the rectangle indicating the frame volume in FIG. 1 is to avoid making the drawing difficult to read. The frame volume is still not tilted. Fig.14 In FIG. 1 , although the rectangle as the frame volume includes a thick-line rectangle and a thin-line rectangle, this is also intended to avoid making the figure difficult to observe. There is no difference between the frame volumes represented by the thick-line rectangle and the thin-line rectangle.

[0195] In the third setting method, the frame width and the frame interval are set to have separate times, respectively, by specifying the frame width and the frame interval based on time.

[0196] exist Fig.14 , part A shows an example of setting the frame interval and the frame width in the case where the frame width is set to be longer in time than the frame interval. Fig.14 , part B shows an example of setting the frame interval and the frame width in the case where the frame width is set to be shorter in time than the frame interval.

[0197] exist Fig.14 In part A, the frame interval is set to a short time as in the first setting method, and the frame width is set to a long time as in the second setting method.

[0198] exist Fig.14 In part A of , since the frame width is set to a long time as in the second setting method, the amount of event data included in the frame volume can be easily increased to a certain extent. Then, the image associated with the frame data generated based on such a large amount of event data makes the subject appear clearly and has high visibility.

[0199] In addition, Fig.14 In part A, since the frame interval is short similar to the first setting method, the frame rate of the frame data can be increased to a certain extent. Therefore, as an image related to the frame data, an image that smoothly presents the motion of a fast-moving subject (moving object) can be obtained.

[0200] As described above, in the third setting method, appropriate frame data can be obtained by independently setting the frame width and the frame interval at different time intervals. In other words, for example, frame data that generates an image with high visibility and smooth motion can be obtained. Therefore, image recognition performed on an image associated with such frame data can obtain sufficient recognition performance.

[0201] In addition, Fig.14 In part A, the frame width is set to be longer in time than the frame interval, so the frame volumes of one frame (a frame for generating frame data) and the next frame have a partially overlapping area.

[0202] In this case, in the data generating unit 113 ( Fig. 9 ), frame data is generated for each of two adjacent frames based on event data in the overlapping region of the frame volumes of the two frames. Therefore, it can be considered that frame data sharing the same event data in the overlapping region is generated for the two adjacent frames.

[0203] In addition, this can also be understood that, for the next f+1 frame of a certain f-th frame, frame data is generated by reusing the event data used to generate the frame data of the f-th frame before the f+1-th frame in the overlapping area.

[0204] exist Fig.14 In part B, the frame interval is set to be longer in time than the frame width (i.e., the frame width is set to be shorter in time than the frame interval).

[0205] For example, when the frequency of occurrence of an event is low, it is effective to use a frame interval set longer than the frame width. In other words, setting the frame interval longer than the frame width can prevent or suppress the generation of frame data in which frames showing similar scenes continue for a long time.

[0206] <Configuration Example of Data Generating Unit 113>

[0207] Fig.15 It is shown Fig. 9 A block diagram of a construction example of the data generating unit 113 in FIG.

[0208] exist Fig.15 In the embodiment, the data generating unit 113 has a memory 121, a memory control unit 122, a frame data generating unit 123 and an acquiring unit 124.

[0209] The memory 121 is provided with event data from the event generating unit 21. The memory 121 stores the event data from the event generating unit 21. In addition, the memory 121 stores the event data in sequence in the storage area. If the storage area is full of event data, the memory 121 stores the latest event data in the form of overwriting the oldest event data with the latest event data.

[0210] The memory control unit 122 controls reading of event data from the memory 121, etc. For example, the memory control unit 122 controls the memory 121 according to the set frame interval and the set frame width provided from the acquisition unit 124, and thereby reads event data in a frame volume from the memory 121, and provides the event data to the frame data generation unit 123.

[0211] The frame data generation unit 123 generates frame data of one frame based on the event data in the frame volume supplied from the memory control unit 122, and supplies the frame data to the image processing unit 114 ( Fig. 9 ).

[0212] The acquisition unit 124 acquires the set frame interval from the frame interval setting unit 111, and acquires the set frame width from the frame width setting unit 112. Then, the acquisition unit 124 supplies the set frame interval and the set frame width to the memory control unit 122.

[0213] Fig.16 1 is a diagram for explaining an example of control of the memory 121 by the memory control unit 122 .

[0214] The memory 121 sequentially (in time series) stores the event data supplied from the event generating unit 21 .

[0215] For example, the memory control unit 122 reads event data including event time t within a time set as a frame width from event time t0 of event data initially stored in the memory 121. In this case, the event data is read as event data in a frame volume of the first frame from the memory 121.

[0216] exist Fig.16 In the example, the event data from the event time t0 to the event time t8 are read from the memory 121 as the event data in the frame volume of the first frame. The frame data is generated based on the read event data.

[0217] The memory control unit 122 reads the event data from the event time t0 to the event time t8 from the memory 121 as the event data in the frame volume of the first frame. Then, the memory control unit 122 releases the storage area including the event data at the time t of the event in the storage area of ​​the memory 121. The release is performed within the time from the beginning of the frame volume of the first frame (the beginning of the set frame interval) to the time immediately before the elapse of the set frame interval.

[0218] exist Fig.16 In the memory 121, the storage area from the event data at the event time t0 to the event data at the event time t5 is released. New event data can be stored in the released storage area in the memory 121. Fig.16 From the event time t 10 At event time t 13 The event data at the event time t0 is stored by overwriting the storage area storing the event data from the event time t0 to the event time t3.

[0219] Then, the memory control unit 122 sets the time after the set frame interval from the start time of the frame volume whose event data was read from the memory 121 immediately before as the start of the second frame volume. Then, the memory control unit 122 reads the event data of the time t including the event within the time of the set frame width from the memory 121 as the event data in the frame volume of the second frame from the start.

[0220] exist Fig.16 In the example, the data from event time t6 to event time t is read from memory 121. 10 The event data is used as the event data in the frame volume of the second frame. Frame data is generated according to the read event data.

[0221] Then, the third event data and subsequent event data are similarly read.

[0222] <Processing by Data Processing Unit 22>

[0223] Fig.17 Is used to illustrate the Fig. 9 Flowchart of an example of processing performed by the data processing unit 22.

[0224] In step S11, the frame interval setting unit 111 sets the frame interval and supplies the frame interval to the data generating unit 113. The frame width setting unit 112 sets the frame width and supplies the frame width to the data generating unit 113. The process proceeds to step S12.

[0225] In step S12, the data generation unit 113 performs data generation processing for generating frame data based on the frame interval (set frame interval) from the frame interval setting unit 111, the frame width (set frame width) from the frame width setting unit 112, and the event data from the event generation unit 21( Figure 1 ). The process proceeds to step S13.

[0226] In step S13, the image processing unit 114 performs image processing using the frame data obtained in the data generation processing executed in step S12.

[0227] Fig.18 is a flowchart for explaining Fig.17 an example of the data generation processing executed by the data generation unit 113 in step S12.

[0228] In the data generation unit 113( Fig.15 ), in step S21, the memory 121 sequentially stores the event data from the event generation unit 21.

[0229] On the other hand, in the data generation unit 113, in step S31, the acquisition unit 124 acquires the set frame interval St set by the frame interval setting unit 111 and the set frame width It set by the frame width setting unit 112, and provides both to the memory control unit 122. The process proceeds to step S32.

[0230] In step S32, the memory control unit 122 sets the time (variable representing time) T at the start of the frame volume (frame interval) to zero. The process proceeds to step S33.

[0231] In step S33, the memory control unit 122 releases the storage area (memory area) of the memory 121 that stores the event data whose event time t satisfies the equation t < T. The process proceeds to step S34. The event data obtained in step S21 can be stored in the released storage area of the memory 121.

[0232] In step S34, the memory control unit 122 reads the event data whose event time t satisfies the equation T <= t < T + It from the memory 121 as the event data in the frame volume of one frame, and provides the event data to the frame data generation unit 123.

[0233] The frame data generation unit 123 generates frame data based on the event data in the frame volume from the memory control unit 122, and provides the frame data to the image processing unit 114. The process proceeds from step S34 to step S35.

[0234] In step S35, the memory control unit 122 updates the time T to T+St. Then, the process returns from step S35 to step S33, and similar processes are repeated.

[0235] <Specify the frame interval and frame width using the number of event data>

[0236] Fig.19 It is a diagram showing a fourth method of setting the frame width and the frame interval.

[0237] exist Fig.19 In, similar to Fig.12 , the event data is plotted with the horizontal axis being the event time t and the vertical axis being the event position (x, y).

[0238] In the fourth setting method, the frame interval is set by specifying the frame interval based on time, and the frame width is set by specifying the frame width based on the number of event data.

[0239] In the fourth setting method, since the frame interval is specified based on time, the frame interval is a constant time. Therefore, the frame rate of the frame data is constant.

[0240] However, in the fourth setting method, the frame width is specified based on the number of event data, and the frame width is an indefinite time depending on the frequency of occurrence of the event. In other words, in the case where the frame width is specified based on the number of event data, when the number of event data in the frame volume reaches the specified number, the width of the frame volume in the direction of the time axis t is regarded as the set frame width.

[0241] Therefore, in the fourth setting method, when the occurrence frequency of event data is low, the time as the frame width is large. On the other hand, when the occurrence frequency of event data is high, the time as the frame width is small.

[0242] In the fourth setting method, generation of frame data waits until the number of event data in the frame volume reaches a specified number, and generates frame data for one frame based on the specified number of event data.

[0243] Therefore, for example, in a scene where a moving object moves from one side of the left and right sides of the screen to the other side, the movement of the moving object causes a large number of events in a short time while the moving object moves on the screen. Frame data in which the moving object appears in the image is generated.

[0244] On the other hand, in the case where there is no moving object on the screen, few events occur, so the number of event data in the frame volume takes a long time to reach the specified number. For example, (most of) the events that occur in the case where there is no moving object on the screen are events caused by slight movement of the background. Therefore, in the case where there is no moving object on the screen, frame data is generated after waiting for the specified number of events to occur due to the long-term background movement.

[0245] In this case, a slightly moving background is displayed on the image associated with the frame data.

[0246] Therefore, the fourth setting method can realize an automatic exposure (AE) function that adaptively adjusts the exposure time according to the frequency of occurrence of events. Examples of the AE function that adaptively adjusts the exposure time according to the frequency of occurrence of events include capturing an image with a short exposure time during the movement of a moving object on the screen or capturing an image with a long exposure time when there is no moving object on the screen.

[0247] In addition, as in the third setting method and the fourth setting method, the frame interval can be set by specifying the frame interval based on time. In this case, it is possible to appropriately switch between setting the frame width by specifying the frame width based on time as in the third setting method and setting the frame width by specifying the frame width based on the number of event data as in the fourth setting method.

[0248] In other words, for example, setting the frame width by specifying the frame width based on time and setting the frame width by specifying the frame width based on the number of event data can be switched in response to a user operation.

[0249] Furthermore, for example, setting the frame width by specifying the frame width based on time or setting the frame width by specifying the frame width based on the number of event data may be switched according to the occurrence frequency of events within a predetermined fixed time (ie, the number of event data).

[0250] In the present specification, in the case where the frame width is set by specifying the frame width based on the number of event data, frame data is not generated until the number of event data reaches the specified number. Therefore, if the frequency of occurrence of events continues to be very low, a situation where frame data is not generated for a long time may occur. Therefore, in the case where the frequency of occurrence of events within a predetermined fixed time is lower than a preset threshold, the setting of the frame width can be switched to a time-based setting. On the other hand, in the case where the frequency of occurrence of events within a predetermined fixed time is equal to or higher than a preset threshold, the setting of the frame width based on time can be switched to a frame width setting based on the number of event data.

[0251] Fig. 20It is a diagram showing a fifth method of setting the frame width and the frame interval.

[0252] exist Fig. 20 In, similar to Fig.12 , the event data is plotted with the horizontal axis being the event time t and the vertical axis being the event position (x, y).

[0253] In the fifth setting method, the frame width is set by specifying the frame width based on time, and the frame interval is set by specifying the frame interval based on the number of event data.

[0254] In the fifth setting method, the frame width is specified based on time, so the frame width is a fixed time.

[0255] However, in the fifth setting method, the frame interval is specified based on the number of event data, so the frame interval is an indefinite time according to the frequency of occurrence of the event. In other words, in the case where the frame interval is specified based on the number of event data, when the number of event data from the beginning of the frame volume to the beginning of the next frame volume reaches the specified number, the time from the beginning of the frame volume to the beginning of the next frame volume is regarded as the set frame interval.

[0256] Therefore, in the fifth setting method, when the frequency of occurrence of event data is relatively low, the time between frames is large. On the other hand, when the frequency of occurrence of event data is relatively high, the time between frames is small.

[0257] In the fifth setting method, the timing when the number of event data from the start of the immediately preceding frame volume reaches a specified number is set as the start of the latest frame volume, and frame data of one frame is generated from the event data in the latest frame volume.

[0258] Therefore, for example, in a scene where a moving object moves from one side of the left and right sides of the screen to the other side, while the moving object moves on the screen, the movement of the moving object causes a large number of events in a short period of time. This makes the frequency of frame data generation larger. As a result, the frame rate of the frame data is larger.

[0259] On the other hand, when there is no moving object on the screen, few events occur, so it takes a long time for the number of event data in the frame volume to reach the specified number. Therefore, when there is no moving object on the screen, the frequency of frame data generation is low, and as a result, the frame rate of the frame data is low.

[0260] Therefore, the fifth setting method can realize the function of adaptively adjusting the frame rate of frame data according to the frequency of occurrence of events. Examples of the function of adaptively adjusting the frame rate of frame data include increasing the frame rate of frame data when a moving object moves on the screen and reducing the frame rate of frame data when there is no moving object on the screen.

[0261] Such adaptive adjustment of the frame rate of the frame data enables continuous shooting, since the frame rate of the frame data is large even when there is a moving object on the screen.

[0262] In addition, it is assumed that the frame data obtained using the fifth setting method is reproduced at a constant frame rate. In a scene where a moving object moves on the screen, the reproduction of the image associated with the frame data is slow reproduction. On the other hand, in a scene where there is no moving object on the screen, the reproduction of the image associated with the frame data is fast forward reproduction.

[0263] Fig.21 It is a diagram showing a sixth method of setting the frame width and the frame interval.

[0264] exist Fig.21 In, similar to Fig.12 , the event data is plotted with the horizontal axis being the event time t and the vertical axis being the event position (x, y).

[0265] In the sixth setting method, the frame interval and the frame width are set by specifying the frame interval and the frame width based on the number of event data.

[0266] In the sixth setting method, the frame interval and the frame width are specified based on the number of event data, and therefore, similar to the fifth setting method and the fourth setting method, the frame interval and the frame width are respectively indefinite times according to the occurrence frequency of the event. In other words, a higher occurrence frequency of the event makes the time as the frame interval and the frame width shorter, while a lower occurrence frequency of the event makes the time as the frame interval and the frame width longer.

[0267] Therefore, similar to the fifth setting method, the sixth setting method can realize the function of adaptively adjusting the frame rate of frame data according to the frequency of occurrence of events. Examples of the function of adaptively adjusting the frame rate of frame data include increasing the frame rate of frame data when a moving object moves on the screen and reducing the frame rate of frame data when there is no moving object on the screen.

[0268] In addition, similar to the fourth setting method, the sixth setting method can implement an AE function that adaptively adjusts the exposure time according to the frequency of occurrence of an event. Examples of the AE function that adaptively adjusts the exposure time according to the frequency of occurrence of an event include capturing an image with a short exposure time during movement of a moving object on the screen or capturing an image with a long exposure time when there is no moving object on the screen.

[0269] In addition, as in the fifth setting method and the sixth setting method, the frame interval can be set by specifying the frame interval based on the number of event data. In this case, it is possible to adaptively switch between setting the frame width by specifying the frame width based on time as in the fifth setting method and setting the frame width by specifying the frame width based on the number of event data as in the sixth setting method.

[0270] In other words, as reference Fig. 20 As described above, the frame width can be set by specifying the frame width based on time or by specifying the frame width based on the number of event data, for example, according to a user operation or the frequency of occurrence of an event within a predetermined fixed time.

[0271] Furthermore, it is possible to adaptively switch between setting the frame interval by specifying the frame interval based on time as in the third and fourth setting methods, or setting the frame interval by specifying the frame interval based on the number of event data as in the fifth and sixth setting methods.

[0272] In other words, for example, setting the frame interval by specifying the frame interval based on time and setting the frame interval by specifying the frame interval based on the number of event data can be switched in response to a user operation.

[0273] Furthermore, setting the frame interval by specifying the frame interval based on time or setting the frame interval by specifying the frame interval based on the number of event data may be switched according to, for example, the frame rate of frame data or power consumption of the data processing unit 22 .

[0274] In this specification, when the frame interval is set by specifying the frame interval based on the number of event data, the frame rate of the frame data varies. Therefore, the data processing unit 22 ( Figure 1 ), the storage capacity required for storing frame data, or the image processing unit 114 ( Fig. 9 )The load of image processing performed on frame data varies.

[0275] Therefore, in the case where the power consumption obtained by measuring the power consumption of the data processing unit 22 exceeds the threshold value, the setting of the frame interval can be switched to setting by designation based on time. On the other hand, in the case where the power consumption of the data processing unit 22 is equal to or less than the threshold value, the setting of the frame interval can be switched to setting by designation based on the number of event data. This can prevent or reduce the increase in the power consumption of the data processing unit 22.

[0276] In addition, it is possible to switch between setting the frame interval by specifying the frame interval based on time or by specifying the frame interval based on the number of event data. This switching depends on the storage capacity required to store frame data or the load of image processing performed on frame data by the image processing unit 114, etc.

[0277] <Description of Computer to Which the Present Invention is Applicable>

[0278] Next, a series of processes by the data processing unit 22 can be executed by hardware or software. In the case where a series of processes is executed by software, a program constituting the software is installed in a computer such as a microcomputer.

[0279] Fig. 22 This is a block diagram showing a configuration example of an embodiment of a computer in which a program for executing the above-described series of processes is installed.

[0280] The program can be recorded in advance on the ROM 903 serving as a recording medium provided with the computer.

[0281] Alternatively, the program may be stored (recorded) on a removable recording medium. Such a removable recording medium may be provided in the form of so-called packaged software. In this regard, examples of the removable recording medium include a floppy disk, a compact disc read-only memory (CD-ROM), a magneto-optical (MO) disk, a digital versatile disc (DVD), a magnetic disk, or a semiconductor memory, etc.

[0282] Furthermore, the program may be installed in a computer from a removable recording medium as described above, or may be downloaded and installed in a computer via a communication network or a broadcasting network.

[0283] The computer has a built-in central processing unit (CPU) 902 , and a ROM 903 and a RAM 904 are connected to the CPU 902 via a bus 901 .

[0284] The CPU 902 executes a program stored in a read only memory (ROM) 903 .

[0285] This enables the CPU 902 to execute the processing according to the above-described flowchart or the processing executed using the configuration of the above-described block diagram.

[0286] In this regard, in this specification, a computer does not necessarily execute processing according to a program in the order shown in chronological order in a flowchart. That is, the processing executed by a computer according to a program also includes processing executed in parallel or individually (for example, parallel processing or object-oriented processing).

[0287] In addition, the program may be processed by a single computer (processor) or by a plurality of computers in a dispersed manner. In addition, the program may be transferred to a remote computer for execution.

[0288] <Application example to mobile objects>

[0289] The technology according to the embodiments of the present invention (the present technology) is applicable to a variety of products. For example, the technology according to the embodiments of the present invention is implemented as a device installed on any type of mobile body such as a car, an electric car, a hybrid car, a motorcycle, a bicycle, a personal mobile device, an airplane, a drone, a ship, or a robot.

[0290] Fig.23 : is a block diagram showing a schematic configuration example of a vehicle control system as an example of a moving body control system to which the technology according to the embodiment of the present invention is applied.

[0291] The vehicle control system 12000 includes a plurality of electronic control units connected to each other via a communication network 12001. Fig.23 In the example shown, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an external information detection unit 12030, an internal information detection unit 12040, and an integrated control unit 12050. In addition, as a functional configuration of the integrated control unit 12050, a microcomputer 12051, a sound / image output unit 12052, and a vehicle-mounted network interface (I / F: vehicle-mounted network interface) 12053 are shown.

[0292] The drive system control unit 12010 controls the operation of devices related to the drive system of the vehicle according to various programs. For example, the drive system control unit 12010 serves as a control device for the following devices: a drive force generating device such as an internal combustion engine, a drive motor, etc. for generating a drive force for the vehicle; a drive force transmitting mechanism for transmitting the drive force to the wheels; a steering mechanism for adjusting the steering angle of the vehicle; and a braking device for generating a braking force for the vehicle, etc.

[0293] The body system control unit 12020 controls the operation of various devices provided in the vehicle body according to various programs. For example, the body system control unit 12020 is used as a control device for a keyless entry system, a smart key system, a power window device, or various lights such as a headlight, a taillight, a brake light, a turn signal light, or a fog light. In this case, a radio wave transmitted from a mobile device instead of a key or a signal of various switches may be input to the body system control unit 12020. The body system control unit 12020 receives these input radio waves or signals, and controls a door lock device, a power window device, or a light, etc. of the vehicle.

[0294] The vehicle exterior information detection unit 12030 detects information about the exterior of the vehicle including the vehicle control system 12000. For example, the vehicle exterior information detection unit 12030 is connected to the camera unit 12031. The vehicle exterior information detection unit 12030 causes the camera unit 12031 to capture an image of the exterior of the vehicle and receives the captured image. Based on the received image, the vehicle exterior information detection unit 12030 can perform detection processing on objects such as people, vehicles, obstacles, signs, or text on the road surface, or distance detection processing to these objects.

[0295] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. The imaging unit 12031 may output the electrical signal as an image, or may output the electrical signal as information about the measured distance. In addition, the light received by the imaging unit 12031 may be visible light, or may be invisible light such as infrared rays.

[0296] The in-vehicle information detection unit 12040 detects information about the interior of the vehicle. For example, the in-vehicle information detection unit 12040 is connected to a driver state detection unit 12041 for detecting the driver's state. For example, the driver state detection unit 12041 includes a camera for photographing the driver. Based on the detection information input from the driver state detection unit 12041, the in-vehicle information detection unit 12040 can calculate the driver's fatigue level or the driver's concentration level, or can determine whether the driver is dozing off.

[0297] Based on the information about the inside and outside of the vehicle obtained by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040, the microcomputer 12051 can calculate the control target value of the driving force generating device, the steering mechanism or the braking device, and output a control instruction to the driving system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of the Advanced Driver Assistance System (ADAS), which includes: collision avoidance or collision mitigation of the vehicle, following driving based on the following distance, vehicle speed maintenance driving, vehicle collision warning or vehicle lane departure warning, etc.

[0298] In addition, based on the information about the vehicle's surroundings obtained by the external information detection unit 12030 or the internal information detection unit 12040, the microcomputer 12051 can perform collaborative control aimed at automatic driving, which enables the vehicle to drive autonomously by controlling the driving force generating device, steering mechanism or braking device, etc. without relying on the driver's operation.

[0299] In addition, the microcomputer 12051 can output a control instruction to the body system control unit 12020 based on the information about the exterior of the vehicle obtained by the exterior information detection unit 12030. For example, the microcomputer 12051 can perform cooperative control aimed at preventing glare, for example, by controlling the headlights to change from high beam to low beam based on the position of the preceding vehicle or the oncoming vehicle detected by the exterior information detection unit 12030.

[0300] The sound / image output unit 12052 transmits an output signal of at least one of sound and image to an output device capable of visually or auditorily notifying the vehicle's passengers or the outside of the vehicle of the information. Fig.23 In the example of FIG. 1 , an audio speaker 12061, a display portion 12062, and an instrument panel 12063 are illustrated as output devices. The display portion 12062 may include, for example, at least one of an in-vehicle display and a head-up display.

[0301] Fig.24 This is a diagram showing an example of the installation position of the camera unit 12031.

[0302] exist Fig.24 In the figure, the vehicle 12100 includes camera units 12101, 12102, 12103, 12104 and 12105 as the camera unit 12031.

[0303] For example, the camera units 12101, 12102, 12103, 12104, and 12105 are located at the front nose, side mirrors, rear bumper, and rear door, and the upper portion of the windshield of the cabin of the vehicle 12100. The camera unit 12101 disposed at the front nose and the camera unit 12105 disposed at the upper portion of the windshield in the vehicle mainly obtain images in front of the vehicle 12100. The camera units 12102 and 12103 attached to the side mirrors mainly obtain images of the side areas of the vehicle 12100. The camera unit 12104 disposed at the rear bumper or rear door mainly obtains images of the rear of the vehicle 12100. The images of the front of the vehicle obtained by the camera units 12101 and 12105 are mainly used to detect the front vehicle, pedestrians, obstacles, signal lights, traffic signs, lanes, etc.

[0304] By the way, Fig.24 Examples of the photographing ranges of the camera units 12101 to 12104 are shown. The photographing range 12111 indicates the photographing range of the camera unit 12101 provided at the front nose. The photographing ranges 12112 and 12113 respectively indicate the photographing ranges of the camera units 12102 and 12103 provided at the side mirrors. The photographing range 12114 indicates the photographing range of the camera unit 12104 provided at the rear bumper or the rear door. For example, a bird's-eye view image of the vehicle 12100 viewed from above can be obtained by superimposing the image data photographed by the camera units 12101 to 12104.

[0305] At least one of the imaging units 12101 to 12104 may have a function of obtaining distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera composed of a plurality of imaging elements, or may be an imaging element having pixels for phase difference detection.

[0306] For example, the microcomputer 12051 can determine the distance to each three-dimensional object within the photographic range 12111 to 12114 and the time domain change of the distance (relative speed relative to the vehicle 12100) based on the distance information obtained from the camera units 12101 to 12104, thereby extracting the nearest three-dimensional object as the front vehicle, in particular, the three-dimensional object exists on the driving path of the vehicle 12100 and travels in the same direction as the vehicle 12100 at a predetermined speed (e.g., equal to or greater than 0 km / h). In addition, the microcomputer 12051 can pre-set the front distance to be followed by the front of the front vehicle, and perform automatic braking control (including follow-up stop control) or automatic acceleration control (including follow-up start control), etc. Therefore, cooperative control of automatic driving aimed at making the vehicle travel autonomously without relying on the operation of the driver, etc. can be performed.

[0307] For example, the microcomputer 12051 can classify the three-dimensional object data about the three-dimensional object into three-dimensional object data of two-wheeled vehicles, standard-sized vehicles, large vehicles, pedestrians, and other three-dimensional objects such as electric poles based on the distance information obtained from the camera units 12101 to 12104, extract the classified three-dimensional object data, and use the extracted three-dimensional object data to automatically avoid obstacles. For example, the microcomputer 12051 identifies obstacles around the vehicle 12100 as obstacles that the driver of the vehicle 12100 can visually identify and obstacles that are difficult for the driver of the vehicle 12100 to visually identify. Then, the microcomputer 12051 determines a collision risk indicating the risk of collision with each obstacle. When the collision risk is equal to or higher than a set value and therefore there is a possibility of collision, the microcomputer 12051 outputs a warning to the driver via the audio speaker 12061 or the display unit 12062, and performs forced deceleration or evasive steering via the drive system control unit 12010. The microcomputer 12051 can thereby assist driving to avoid collisions.

[0308] At least one of the camera units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 may identify a pedestrian by determining whether there is a pedestrian in the captured images of the camera units 12101 to 12104. For example, such identification of pedestrians is performed by extracting feature points in the captured images of the camera units 12101 to 12104 as infrared cameras and determining whether it is a pedestrian by performing pattern matching processing on a series of feature points representing the contour of the object. When the microcomputer 12051 determines that there is a pedestrian in the captured images of the camera units 12101 to 12104 and thus identifies the pedestrian, the sound / image output unit 12052 controls the display unit 12062 to display a square contour line for emphasis in a manner superimposed on the identified pedestrian. In addition, the sound / image output unit 12052 may also control the display unit 12062 to display an icon representing a pedestrian, etc., at a desired position.

[0309] An exemplary vehicle control system to which the technology according to the present invention can be applied has been described above. For example, the technology according to the present invention can be applied to the camera unit 12031 in the above-mentioned configuration. Specifically, Figure 1 The data processing chip shown can be applied to the camera section 12031. The camera section 12031 to which the technology according to the present invention is applied enables appropriate frame data to be obtained, thereby providing appropriate driving support by performing image processing on the obtained frame data.

[0310] <List of methods for setting frame interval and frame width>

[0311] Fig.25 This is a diagram showing a list of setting methods for setting the frame interval and the frame width.

[0312] Reference Fig.25 The method of setting the frame interval and frame width of the present invention will be described again.

[0313] In the third setting method ( Fig.14 ), the frame interval and frame width are set based on time (by specifying based on time). In the fourth setting method ( Fig.19 ), the frame interval is set based on time, and the frame width is set based on the number of event data (by specifying based on the number of event data). In the fifth setting method ( Fig. 20 ), the frame interval is set based on the number of event data, and the frame width is set based on time. In the sixth setting method ( Fig.21 ), the frame interval and frame width are set based on the amount of event data.

[0314] <Example of Frame Data Generated When One Frame Width is Set for the Entire Frame of Frame Data>

[0315] Fig.26 : is a diagram showing an example of frame data generated when one frame width is set for the entire frame of the frame data.

[0316] exist Fig.26 In the image, there are subjects with high event rates (e.g., subjects moving at high speed) and subjects with low event rates (e.g., subjects at very low speed or nearly stationary), where the event rate indicates the frequency of occurrence of events. There is a case where one frame width is set for the entire frame, and there are subjects with high event rates and subjects with low event rates mixed. In this case, the frame volume includes a large amount of event data obtained from the subject with the high event rate and a small amount of event data obtained from the subject with the low event rate.

[0317] For a subject including too much event data in the frame volume, in some cases, the outline of the subject appears blurred in an image associated with the frame data. On the other hand, for a subject including too little event data in the frame volume, the outline of the subject cannot be clearly displayed in an image associated with the frame data, and the visibility of the subject is deteriorated in some cases. In addition, image recognition performed on such an image may result in insufficient recognition performance.

[0318] Therefore, the frame width can be set in units of divided regions that divide a frame of frame data into a plurality of regions.

[0319] <Setting the frame width in units of divided areas>

[0320] Fig. 27 is a diagram showing an example of how a frame is divided into segmented areas.

[0321] For example, the frame may be divided into four equal segmented regions in the horizontal direction. In addition, the frame may be equally divided in the horizontal and vertical directions in a manner that the frame is equally divided into 4×4 segmented regions in the horizontal×vertical directions. Other methods may be used to divide the frame into segmented regions. In addition, the frame may be divided into a plurality of segmented regions having different sizes.

[0322] The frame width can be set for each segment area.

[0323] Fig.28 : is a diagram showing an example of frame data generated when the frame width is set for each divided region.

[0324] exist Fig.28 In, similar to Fig.26 , there are subjects with high event rates and subjects with low event rates. Then, Fig.28For example, a time shorter than a predetermined time is set as the frame width of a segmented region showing a subject with a high event rate. Also, for example, a time longer than a predetermined time is set as the frame width in a segmented region showing a subject with a low event rate.

[0325] As a result, Fig.28 In the present invention, event data of a subject with a high event rate and event data of a subject with a low event rate are included in the frame volume in the same amount. As a result, for two subjects with a high event rate and a low event rate, the contours of the subjects are clearly displayed in the image associated with the frame data.

[0326] Therefore, the visibility of the subject displayed in the image associated with the frame data can be improved. In addition, image recognition performed on the image associated with the frame data can obtain sufficient recognition performance. In addition, in the case of setting the frame width based on the number of event data, for example, for segmented areas displaying subjects with high event rates and low event rates, the same and appropriate number of event data can be set as the frame width. In other words, for example, the number of event data that clearly displays the outline of the subject can be set as the frame width. In this case, the time of the frame width of the segmented area that displays the subject with a high event rate is shorter than the time of the frame width of the segmented area that displays the subject with a low event rate. The time of the frame width of the segmented area that displays the subject with a low event rate is longer than the time of the frame width of the segmented area that displays the subject with a high event rate.

[0327] Fig.29 : is a flowchart showing an example of a setting method in the case where the number of event data is set as a frame width for each divided area.

[0328] In this specification, a divided area in which the number of event data is set as a frame width is also referred to as a region of interest. Fig.29 is a flowchart showing a method of setting a frame width of a region of interest.

[0329] The frame width of the region of interest may be set according to an evaluation value F indicating appropriateness of the frame width. For the region of interest, the evaluation value F may be obtained according to frame data (corresponding image) in the region of interest generated in response to event data in a frame volume of the frame width of the region of interest.

[0330] When setting the frame width of the target region, in step S101, the frame width setting unit 112 ( Fig. 9 ) The variables p, q and r are set to zero as initial values, and the processing enters step S102.

[0331] In this specification, p is a variable representing a candidate for the number of event data as a frame width, and q is a variable representing the number of event data as a frame width. In addition, r is an evaluation value F representing the appropriateness of the frame width represented by the variable p (variable representing the candidate), that is, the variable r is a variable representing a threshold of the evaluation value F, which represents the appropriateness of using the candidate p of the number of event data as the frame width (the number of event data as the frame width).

[0332] In step S102, the frame width setting unit 112 determines whether the variable p is greater than a threshold value p_max, which is the maximum value of the number of event data as a predetermined frame width. If it is determined in step S102 that the variable p is less than or equal to the threshold value p_max, the process proceeds to step S103.

[0333] In step S103, the frame width setting unit 112 sets the variable p (i.e., the candidate p for the number of event data as the frame width) to the temporary frame width p. In addition, the frame width setting unit 112 obtains the evaluation value F using the frame data I (x, y) generated based on the event data in the frame volume of the temporary frame width p as the pixel value of each pixel in the region of interest. Then, the process proceeds to step S104. Here, I (x, y) represents the pixel value of the pixel at the position (x, y) in the region of interest.

[0334] In step S104, the frame width setting unit 112 determines whether the evaluation value F is greater than a threshold value (a variable representing a threshold value) r. If it is determined in step S104 that the evaluation value F is less than or equal to the threshold value r, the process skips step S105 and proceeds to step S106.

[0335] Furthermore, if it is determined in step S104 that the evaluation value F is greater than the threshold r, the process proceeds to step S105. In step S105, the frame width setting unit 112 updates the variable q to the temporary frame width p and the threshold r to the evaluation value F, and the process proceeds to step S106.

[0336] In step S106, the frame width setting unit 112 increases the variable p by a predetermined value δp. Then, the process returns from step S106 to step S102, and the same process is repeated.

[0337] Then, if it is determined in step S102 that the variable p is greater than the threshold value p_max, the process proceeds to step S 107. In step S107, the frame width setting unit 112 sets the variable q to the frame width of the region of interest, and the process ends.

[0338] Furthermore, in the above description, although Fig.29A method of setting a frame width specified based on the number of event data for each segment area is given, but Fig.29 The setting method is applicable to the case where the frame width specified based on time is set for each segmented area. Fig.29 In FIG. 1 , the variable p represents a candidate time as a frame width, and the variable q represents a time as a frame width. In addition, p_max represents a maximum value of a time as a predetermined frame width.

[0339] Fig.30 is a diagram showing an example of the evaluation value F.

[0340] exist Fig.30 , the horizontal axis represents time as a frame width or the amount of event data, and the vertical axis represents the evaluation value F.

[0341] like Fig.30 As shown, an example of the evaluation value F may adopt a function having a large value when the time as the frame width of the attention area or the number of event data is an appropriate value.

[0342] The situation where the number of event data as the frame width of the focus area is appropriate means that the outline of the subject is clearly displayed. Specifically, this means that in the focus area, the outline of the subject is not in a state where the outline of the subject is insufficiently displayed due to too little event data, and the outline of the subject is not in a state where the outline of the subject is blurred due to too much event data.

[0343] An example of the evaluation value F may employ a value (function) that increases when a line having a width of one or more pixels exists in the region of interest. Specifically, the following values ​​may be used as the evaluation value F.

[0344] Matrix A X and A Y It is expressed by formula (7).

[0345] [Mathematical formula 1]

[0346]

[0347]

[0348] In addition, f X (x, y) and f Y (x, y) is expressed by formula (8).

[0349] [Mathematical formula 2]

[0350]

[0351]

[0352] In the case of a line having a width of one pixel in the vertical direction and the horizontal direction, f X (x, y) and f Y The absolute value of (x, y) is large. For example, the evaluation value F can be expressed by equation (9).

[0353] [Mathematical formula 3]

[0354]

[0355] For the pixels in the region of interest, the sum of equation (9) is used.

[0356] In addition, the matrix A X and A Y The value of each element of is not limited to formula (7). In addition, the matrix A X and A Y The number of elements and the structure (size) of are not limited to the elements of 3×3 horizontal lines×vertical lines. In addition, the evaluation value F is not limited to the value obtained by calculating the f shown in formula (9) X (x, y) and f Y The value obtained by adding the square root of the sum of the squares of each of (x, y). An example of the evaluation value F can be adopted by calculating f as shown in Formula (10): X (x, y) and f Y The value obtained by adding the square sum of each of (x, y). In addition, for example, it can be obtained by converting f X (x, y) and f Y The evaluation value F is obtained by summing the values ​​obtained by adding (x, y) itself, or by selecting f as shown in formula (12) X (x, y) and f Y The evaluation value F is obtained by summing the values ​​obtained by taking the larger of (x, y). In addition, the evaluation value F may be designed to take the maximum value (or minimum value) when a desired image (such as line width) is obtained as an image related to the frame data generated from the event data.

[0357] [Formula 4]

[0358]

[0359] [Mathematical formula 5]

[0360]

[0361] [Mathematical formula 6]

[0362]

[0363] <Another Exemplary Configuration of Event Detector 52>

[0364] Fig.31 is a diagram showing another configuration example of the event detector 52 .

[0365] exist Fig.31 , the event detector 52 has a subtractor 430, a quantizer 440, a memory 451, and a controller 452. The subtractor 430 and the quantizer 440 correspond to the subtractor 73 and the quantizer 74, respectively.

[0366] In addition, Fig.31 In FIG. 5 , the event detector 52 also has blocks corresponding to the current-to-voltage converter 71 and the buffer 72, but in Fig.31 Their illustration is omitted.

[0367] The subtractor 430 includes a capacitor 431, an operational amplifier 432, a capacitor 433, and a switch 434. The capacitor 431 to the switch 434 correspond to the capacitor 91 to the switch 94, respectively.

[0368] The quantizer 440 has a comparator 441 . The comparator 441 corresponds to the comparator 101 .

[0369] The comparator 441 compares the voltage signal (difference signal) from the subtractor 430 with a predetermined threshold voltage Vth applied to the inverting input terminal (-). The comparator 441 outputs a signal indicating the comparison result as a detection signal (quantized value).

[0370] The voltage signal from the subtractor 430 may be input to an input terminal (−) of the comparator 441 , and a predetermined threshold voltage Vth may be input to an input terminal (+) of the comparator 441 .

[0371] The controller 452 provides a predetermined threshold voltage Vth to be applied to the inverting input terminal (-) of the comparator 441. The threshold voltage Vth to be provided may have different voltage values ​​in time division. For example, the controller 452 provides a threshold voltage Vth1 corresponding to an ON event (e.g., a positive change in the photocurrent) and a threshold voltage Vth2 corresponding to an OFF event (e.g., a negative change in the photocurrent) at different timings, thereby being able to detect multiple types of address events (events) using one comparator.

[0372] The memory 451 accumulates the output of the comparator 441 based on a sampling signal (sample signal) provided from the controller 452. The memory 451 may be a sampling circuit such as a switch, a parasitic capacitor, or may be a digital storage circuit such as a latch or a trigger. For example, the memory 451 may use the threshold voltage Vth1 corresponding to the ON event to maintain the comparison result of the comparator 441 during the period when the threshold voltage Vth2 corresponding to the OFF event is provided to the inverting input terminal (-) of the comparator 441. In addition, the memory 451 is not necessarily provided, and the memory 451 may also be provided in the pixel (pixel block 41), or may be provided outside the pixel.

[0373] <Scanning method>

[0374] Figure 1 The data processing chip is an asynchronous camera device that reads events by an asynchronous readout method. However, the event readout method is not limited to the asynchronous readout method and may be a synchronous readout method. The camera device to which the synchronous readout method is applicable is a scanning camera device, which is the same as a general camera device that performs imaging at a predetermined frame rate.

[0375] Fig.32 : is a block diagram showing a structural example of a scanning type imaging device.

[0376] like Fig.32 As shown, the imaging device 510 includes a pixel array section 521 , a driver 522 , a signal processing unit 525 , a readout region selection unit 527 , and a signal generation unit 528 .

[0377] The pixel array section 521 includes a plurality of pixels 530. The plurality of pixels 530 outputs output signals in response to the selection signal of the readout region selection unit 527. For example, Fig.31 As shown, each of the plurality of pixels 530 may be provided with a quantizer in the pixel. The plurality of pixels 530 output an output signal corresponding to the amount of change in light intensity. Fig.32 As shown, the plurality of pixels 530 may be two-dimensionally arranged in a matrix.

[0378] The driver 522 drives each of the plurality of pixels 530 so that the signal processing unit 525 outputs a pixel signal generated in each pixel 530 through the output line 514. In addition, the driver 522 and the signal processing unit 525 are circuit units configured to obtain grayscale information. Therefore, in the case of obtaining only event information (event data), the driver 522 and the signal processing unit 525 can be omitted.

[0379] The readout area selection unit 527 selects a part of the plurality of pixels 530 included in the pixel array section 521. For example, the readout area selection unit 527 selects any one or more rows of the rows included in the two-dimensional matrix structure corresponding to the pixel array section 521. The readout area selection unit 527 sequentially selects one or more rows according to a preset cycle. In addition, the readout area selection unit 527 can determine the selection area in response to a request from each pixel 530 of the pixel array section 521.

[0380] The signal generating unit 528 generates an event signal corresponding to an active pixel in which an event is detected in the selected pixel 530, based on the output signal of the pixel 530 selected by the readout area selecting unit 527. Here, the event is an event in which the light intensity changes. An active pixel is a pixel 530 in which the amount of change in the light intensity corresponding to the output signal exceeds or falls below a preset threshold. For example, the signal generating unit 528 compares the output signal of the pixel 530 with a reference signal, detects an active pixel that outputs the output signal when the output signal is greater than or less than the reference signal, and generates an event signal (event data) corresponding to the active pixel.

[0381] For example, the signal generating unit 528 may have a configuration including a column selection circuit that arbitrates signals entering the signal generating unit 528. In addition, the signal generating unit 528 can output not only information of valid pixels detecting an event but also information of invalid pixels not detecting an event.

[0382] The signal generating unit 528 outputs the address information and time stamp information (e.g., (X, Y, T)) of the valid pixel where the event is detected through the output line 515. However, the data output from the signal generating unit 528 may be not only the address information and the time stamp information but also the frame format information (e.g., (0, 0, 1, 0, ...)).

[0383] In addition, the embodiments of the present invention are not limited to the above-described embodiments, and various changes and modifications may be made without departing from the scope of protection of the present technology.

[0384] Furthermore, each step described by the above-mentioned flowchart may be performed by one device or by allocating a plurality of devices.

[0385] In addition, in the case where a plurality of processes are included in one step, the plurality of processes included in the step may be executed by one device or by sharing a plurality of devices.

[0386] Furthermore, the effects described in this specification are merely exemplary and non-limiting, and other effects may be obtained.

[0387] Additionally, the present invention can be constructed as follows.

[0388] <1>

[0389] A data processing device, comprising:

[0390] a frame data generating unit configured to generate first frame data based on event data indicating a change in an electrical signal of a pixel that generates an electrical signal by performing photoelectric conversion and occurring within a first accumulation time from a first frame generation start time to a first frame generation end time, and second frame data based on event data occurring within a second accumulation time from a second frame generation start time to a second frame generation end time; and

[0391] A frame cycle setting unit is configured to set a first frame cycle and provide the first frame cycle to the frame data generating unit, wherein the first frame cycle is a cycle from the first frame generation start time to the second frame generation start time.

[0392] <2>

[0393] according to <1> The data processing device, wherein:

[0394] The frame data generating unit further generates third frame data based on event data occurring within a third accumulation time from a third frame generation start time to a third frame generation end time,

[0395] The frame period setting unit sets the first frame period and the second frame period to be different from each other, the second frame period being a period from the second frame generation start time to the third frame generation start time.

[0396] <3>

[0397] according to <1> or <2> The data processing device further comprises:

[0398] An accumulation time setting unit is configured to set the first accumulation time and the second accumulation time and provide the first accumulation time and the second accumulation time to the frame data generating unit.

[0399] <4>

[0400] according to <3> The data processing device, wherein:

[0401] The accumulation time setting unit sets the first accumulation time and the second accumulation time to be different from each other.

[0402] <5>

[0403] A data processing device, comprising:

[0404] a frame data generating unit configured to generate frame data based on event data indicating a change in an electrical signal of a pixel generating the electrical signal by performing photoelectric conversion, the event data occurring within a cumulative time from a frame generation start timing to a frame generation end timing; and

[0405] A frame period setting unit is configured to set a frame period as an interval between a frame data generation start time of a first frame data and a frame data generation start time of a second frame data, and provide the frame period to the frame data generating unit, wherein the second frame data is generated continuously with the first frame data.

[0406] <6>

[0407] according to <5> The data processing device further comprises:

[0408] An accumulation time setting unit is configured to set the accumulation time and provide the accumulation time to the frame data generating unit.

[0409] <7>

[0410] according to <5> or <6> The data processing device, wherein:

[0411] The frame period setting unit sets the frame period based on time.

[0412] <8>

[0413] according to <6> The data processing device, wherein:

[0414] The cumulative time setting unit sets the cumulative time based on time.

[0415] <9>

[0416] according to <6> The data processing device, wherein:

[0417] The accumulation time setting unit sets the accumulation time according to the amount of the event data.

[0418] <10>

[0419] according to <6> The data processing device, wherein:

[0420] The accumulation time setting unit switches between setting the accumulation time based on time and setting the accumulation time according to the number of event data.

[0421] <11>

[0422] according to <5> The data processing device, wherein:

[0423] The frame period setting unit sets the frame period according to the amount of the event data.

[0424] <12>

[0425] according to <11> The data processing device further comprises:

[0426] an accumulation time setting unit configured to set the accumulation time and provide the accumulation time to the frame data generating unit,

[0427] The cumulative time setting unit sets the cumulative time based on time.

[0428] <13>

[0429] according to <11> The data processing device further comprises:

[0430] an accumulation time setting unit configured to set the accumulation time and provide the accumulation time to the frame data generating unit,

[0431] The accumulation time setting unit sets the accumulation time according to the amount of the event data.

[0432] <14>

[0433] according to <11> The data processing device further comprises:

[0434] an accumulation time setting unit configured to set the accumulation time and provide the accumulation time to the frame data generating unit;

[0435] The accumulation time setting unit switches between setting the accumulation time based on time and setting the accumulation time according to the number of event data.

[0436] <15>

[0437] according to <5> or <6> The data processing device, wherein:

[0438] The frame cycle setting unit switches between setting the frame cycle based on time and setting the frame cycle according to the number of event data.

[0439] <16>

[0440] A data processing method, comprising:

[0441] a frame data generating unit generating frame data based on event data indicating a change in an electrical signal of a pixel generating the electrical signal by performing photoelectric conversion, the event data occurring within a cumulative time from a frame generation start timing to a frame generation end timing; and

[0442] A frame period is set as an interval between the frame data generation start time of the first frame data and the frame data generation start time of the second frame data, and the frame period is provided to the frame data generation unit, and the second frame data is generated continuously with the first frame data.

[0443] <17>

[0444] A program that causes a computer to function as:

[0445] a frame data generating unit configured to generate frame data based on event data indicating a change in an electrical signal of a pixel generating the electrical signal by performing photoelectric conversion, the event data occurring within a cumulative time from a frame generation start timing to a frame generation end timing; and

[0446] A frame period setting unit is configured to set a frame period as an interval between a frame data generation start time of a first frame data and a frame data generation start time of a second frame data, and provide the frame period to the frame data generating unit, wherein the second frame data is generated continuously with the first frame data.

[0447] <18>

[0448] A data processing device, comprising:

[0449] a frame data generating unit configured to generate first frame data and second frame data in such a manner that the first accumulation time and the second accumulation time overlap,

[0450] The first frame data is based on event data indicating a change in the electrical signal of a pixel that generates an electrical signal by performing photoelectric conversion and occurring within the first cumulative time from the start time of the first frame generation to the end time of the first frame generation, and the second frame data is based on the event data occurring within the second cumulative time from the start time of the second frame generation to the end time of the second frame generation.

[0451] <19>

[0452] A data processing method, comprising:

[0453] generating first frame data and second frame data in such a manner that the first accumulation time and the second accumulation time overlap,

[0454] The first frame data is based on event data indicating a change in the electrical signal of a pixel that generates an electrical signal by performing photoelectric conversion and occurring within the first cumulative time from the start time of the first frame generation to the end time of the first frame generation, and the second frame data is based on the event data occurring within the second cumulative time from the start time of the second frame generation to the end time of the second frame generation.

[0455] <20>

[0456] A program that causes a computer to function as:

[0457] a frame data generating unit configured to generate first frame data and second frame data in such a manner that the first accumulation time and the second accumulation time overlap,

[0458] The first frame data is based on event data indicating a change in the electrical signal of a pixel that generates an electrical signal by performing photoelectric conversion and occurring within the first cumulative time from the start time of the first frame generation to the end time of the first frame generation, and the second frame data is based on the event data occurring within the second cumulative time from the start time of the second frame generation to the end time of the second frame generation.

[0459] <21>

[0460] A data processing device that generates frame data based on event data indicating changes in an electrical signal of a pixel that generates an electrical signal by performing photoelectric conversion,

[0461] The event data occurs within the cumulative time from the start time of frame generation to the end time of frame generation,

[0462] The number of frame data generated per unit time varies according to the number of event data generated per unit time.

[0463] <22>

[0464] according to <21> The data processing device, wherein:

[0465] The larger the number of the event data per unit time, the larger the number of frame data generated per unit time.

[0466] <23>

[0467] A data processing device, comprising:

[0468] A data processing unit, the data processing unit comprising:

[0469] an event generating unit having a pixel generating an electrical signal by performing photoelectric conversion and configured to generate event data indicating occurrence of an event, the event being a change in the electrical signal of the pixel;

[0470] a frame data generating unit configured to generate frame data based on the event data occurring within a cumulative time from a frame generation start time to a frame generation end time, and

[0471] A frame period setting unit is configured to set a frame period as an interval between a frame data generation start time of a first frame data and a frame data generation start time of a second frame data, and provide the frame period to the frame data generating unit, wherein the second frame data is generated continuously with the first frame data.

[0472] <24>

[0473] according to <23> The data processing device further comprises:

[0474] An image processing unit is configured to perform image processing using the frame data.

[0475] <25>

[0476] according to <24> The data processing device, wherein:

[0477] The image processing unit performs recognition processing on the frame data.

[0478] <26>

[0479] according to <23> to <25> The data processing device as described in any one of the preceding claims, wherein:

[0480] The data processing device includes a semiconductor chip having a plurality of dies stacked on each other.

[0481] <27>

[0482] A data processing method, comprising:

[0483] generating event data indicating occurrence of an event, the event being a change in an electrical signal of a pixel that generates an electrical signal by performing photoelectric conversion;

[0484] The frame data generating unit generates frame data based on the event data occurring within a cumulative time from a frame generation start time to a frame generation end time; and

[0485] A frame period is set as an interval between the frame data generation start time of the first frame data and the frame data generation start time of the second frame data, and the frame period is provided to the frame data generation unit, and the second frame data is generated continuously with the first frame data.

[0486] <28>

[0487] according to <6> , <8> , <9> , <12> and <13> The data processing device as described in any one of the preceding claims, wherein:

[0488] The accumulation time setting unit sets the accumulation time in units of divided regions, the divided regions being obtained by dividing a frame of the frame data into a plurality of regions.

[0489] <29>

[0490] according to <28> The data processing device, wherein:

[0491] The accumulation time setting unit sets the accumulation time according to an evaluation value indicating appropriateness of the accumulation time.

[0492] <30>

[0493] according to <29> The data processing device, wherein:

[0494] The accumulation time setting unit sets the accumulation time according to the evaluation value obtained using the frame data generated from the event data within the accumulation time.

[0495] <a1>

[0496] A data processing device, comprising:

[0497] a frame interval setting unit configured to set a frame interval indicating an interval of frames; and

[0498] A data generating unit is configured to generate frame data as image data in a frame format of the frame interval based on event data indicating occurrence of an event as a change in an electrical signal of a pixel generating an electrical signal by performing photoelectric conversion.

[0499] <a2>

[0500] according to <a1>The data processing device further comprises:

[0501] a frame width setting unit configured to set a frame width representing a time width of the event for generating the frame data,

[0502] The data generating unit generates the frame data from the beginning of the frame interval according to the event data within the frame width.

[0503] <a3>

[0504] according to <a2>The data processing device, wherein:

[0505] The frame interval setting unit sets the frame interval based on time.

[0506] <a4>

[0507] according to <a2>or <a3>The data processing device, wherein:

[0508] The frame width setting unit sets the frame width based on time.

[0509] <a5>

[0510] according to <a2>or <a3>The data processing device, wherein:

[0511] The frame width setting unit sets the frame width according to the amount of the event data.

[0512] <a6>

[0513] according to <a3>The data processing device, wherein:

[0514] The frame width setting unit switches between setting the frame width based on time and setting the frame width according to the number of event data.

[0515] <a7>

[0516] according to <a2>The data processing device, wherein the frame interval setting unit sets the frame interval according to the amount of the event data.

[0517] <a8>

[0518] according to <a2>or <a7>The data processing device, wherein:

[0519] The frame width setting unit sets the frame width based on time.

[0520] <a9>

[0521] according to <a2>or <a7>The data processing device, wherein:

[0522] The frame width setting unit sets the frame width according to the amount of the event data.

[0523] <a10>

[0524] according to <a7>The data processing device, wherein:

[0525] The frame width setting unit switches between setting the frame width based on time and setting the frame width according to the number of event data.

[0526] <a11>

[0527] according to <a1>or <a2>The data processing device, wherein:

[0528] The frame interval setting unit switches between setting the frame interval based on time and setting the frame interval according to the number of event data.

[0529] <a12>

[0530] A data processing method, comprising:

[0531] setting a frame interval indicating an interval of frames; and

[0532] Frame data as image data is generated in the frame format of the frame interval based on event data indicating occurrence of an event as a change in an electrical signal of a pixel that generates an electrical signal by performing photoelectric conversion.

[0533] <a13>

[0534] A program that causes a computer to function as:

[0535] a frame interval setting unit configured to set a frame interval indicating an interval of frames; and

[0536] A data generating unit is configured to generate frame data as image data in a frame format of the frame interval based on event data indicating occurrence of an event as a change in an electrical signal of a pixel generating an electrical signal by performing photoelectric conversion.

[0537] <a14>

[0538] A data processing device, comprising:

[0539] A data generating unit is configured to generate frame data as image data in a frame format based on event data representing the occurrence of an event as a change in an electrical signal of a pixel that generates an electrical signal by performing photoelectric conversion, and the frame data shares the same event data between adjacent frames.

[0540] <a15>

[0541] A data processing method, comprising:

[0542] Frame data as image data is generated in a frame format based on event data indicating occurrence of an event as a change in an electrical signal of a pixel generating an electrical signal by performing photoelectric conversion, the frame data sharing the same event data between adjacent frames.

[0543] <a16>

[0544] A program that causes a computer to function as:

[0545] A data generating unit is configured to generate frame data as image data in a frame format based on event data representing the occurrence of an event as a change in an electrical signal of a pixel that generates an electrical signal by performing photoelectric conversion, and the frame data shares the same event data between adjacent frames.

[0546] <a17>

[0547] A data processing device, comprising:

[0548] an event generating unit having a pixel generating an electrical signal by performing photoelectric conversion and configured to generate event data indicating the occurrence of an event as a change in the electrical signal of the pixel;

[0549] a frame interval setting unit configured to set a frame interval indicating an interval of frames; and

[0550] A data generating unit is configured to generate frame data as image data in a frame format at the frame interval based on the event data.

[0551] <a18>

[0552] according to <a17>The data processing device further comprises:

[0553] An image processing unit is configured to perform image processing using the frame data.

[0554] <a19>

[0555] according to <a18>The data processing device, wherein:

[0556] The image processing unit performs recognition processing on the frame data.

[0557] <a20>

[0558] according to <a17>to <a19>The data processing device as described in any one of the preceding claims, wherein:

[0559] The data processing device includes a semiconductor chip having a plurality of dies stacked on each other.

[0560] <a21>

[0561] A data processing method, comprising:

[0562] generating event data indicating occurrence of an event as a change in an electrical signal of a pixel that generates an electrical signal by performing photoelectric conversion;

[0563] setting a frame interval indicating an interval of frames; and

[0564] Frame data as image data is generated in a frame format at the frame interval based on the event data.

[0565] <a22>

[0566] according to <a4> 、 <a5> 、 <a8>and <a9>The data processing device as described in any one of the preceding claims, wherein:

[0567] The frame width setting unit sets the frame width in units of divided regions, the divided regions being obtained by dividing a frame of the frame data into a plurality of regions.

[0568] <a23>

[0569] according to <a22>The data processing device, wherein:

[0570] The frame width setting unit sets the frame width according to an evaluation value indicating appropriateness of the frame width.

[0571] <a24>

[0572] according to <a23>The data processing apparatus described above, wherein the frame width setting unit sets the frame width based on the evaluation value obtained by using the frame data generated based on the event data within the frame width.

[0573] Reference numerals list

[0574] 11 Sensor die

[0575] 12 Logic Die

[0576] 21 Event Generator Unit

[0577] 22 Data processing unit

[0578] 31 Pixel array unit

[0579] 32 Drivers

[0580] 33 Arbitrator

[0581] 34 Output Units

[0582] 41 pixel block

[0583] 51 pixels

[0584] 52 Event Detector

[0585] 60 nodes

[0586] 61 Photoelectric converter

[0587] 62 Pass transistor

[0588] 71 Current-to-voltage converter

[0589] 72 Buffer

[0590] 73 Subtractor

[0591] 74 Quantizer

[0592] 75 Transmission Unit

[0593] 81~83 Transistor

[0594] 91 Capacitor

[0595] 92 Operational Amplifier

[0596] 93 Capacitor

[0597] 94 Switch

[0598] 101 Comparator

[0599] 111 Frame interval setting unit

[0600] 112 Frame width setting unit

[0601] 113 Data generation unit

[0602] 114 Image Processing Unit

[0603] 121 Memory

[0604] 122 Memory control unit

[0605] 123 frame data generation unit

[0606] 124 Get Unit

[0607] 901 Bus

[0608] 902 CPU

[0609] 903 ROM

[0610] 904 RAM < / a5> < / a4>

Claims

1. A data processing device, comprising: a frame data generating unit configured to generate first frame data based on event data indicating a change of an electrical signal of a pixel generating an electrical signal by performing photoelectric conversion and occurring within a first accumulation time from a first frame generation start time to a first frame generation end time, and second frame data based on event data occurring within a second accumulation time from a second frame generation start time to a second frame generation end time; a frame period setting unit configured to set a first frame period and provide the first frame period to the frame data generating unit, the first frame period being a period from a first frame generation start time to a second frame generation start time; and an accumulation time setting unit configured to set the first accumulation time and the second accumulation time and provide the first accumulation time and the second accumulation time to the frame data generating unit, wherein the accumulation time setting unit is further configured to switch between setting the first accumulation time and the second accumulation time based on time and setting the first accumulation time and the second accumulation time depending on the number of event data in response to a user operation, The accumulation time setting unit sets the first accumulation time and the second accumulation time in units of divided regions, wherein the divided regions are obtained by dividing the frame into a plurality of regions based on a plurality of objects corresponding to the frame of each of the first frame data and the second frame data.

2. The data processing device according to claim 1, wherein: The frame data generating unit further generates third frame data based on event data occurring within a third accumulation time from a third frame generation start time to a third frame generation end time, The frame period setting unit sets the first frame period and the second frame period to be different from each other, the second frame period being a period from the second frame generation start time to the third frame generation start time.

3. The data processing device according to claim 1, wherein: The accumulation time setting unit sets the first accumulation time and the second accumulation time to be different from each other.

4. A data processing device, comprising: a frame data generating unit configured to generate frame data based on event data indicating a change in an electrical signal of a pixel generating the electrical signal by performing photoelectric conversion, the event data occurring within a cumulative time from a frame generation start timing to a frame generation end timing; a frame period setting unit configured to set a frame period as an interval between a frame data generation start time of the first frame data and a frame data generation start time of the second frame data, the second frame data being generated continuously with the first frame data, and to provide the frame period to the frame data generating unit; and an accumulation time setting unit configured to set the accumulation time and provide the accumulation time to the frame data generating unit, wherein the accumulation time setting unit is further configured to switch between setting the accumulation time based on time and setting the accumulation time according to the number of event data in response to a user operation, The accumulation time setting unit sets the accumulation time in units of divided regions, wherein the divided regions are obtained by dividing the frame into a plurality of regions based on a plurality of objects corresponding to the frame of each of the first frame data and the second frame data.

5. The data processing device according to claim 4, wherein: The frame period setting unit sets the frame period based on time.

6. The data processing device according to claim 4, wherein: The frame period setting unit sets the frame period according to the amount of the event data.

7. The data processing device according to claim 4, wherein: The accumulation time setting unit sets the accumulation time according to an evaluation value indicating appropriateness of the accumulation time.

8. The data processing device according to claim 7, wherein: The accumulation time setting unit sets the accumulation time according to the evaluation value obtained using the frame data generated from the event data within the accumulation time.

9. The data processing device according to claim 4, wherein: The frame cycle setting unit switches between setting the frame cycle based on time and setting the frame cycle according to the number of event data.

10. A data processing method, comprising: a frame data generating unit generating frame data based on event data indicating a change in an electrical signal of a pixel generating the electrical signal by performing photoelectric conversion, the event data occurring within a cumulative time from a frame generation start timing to a frame generation end timing; Setting a frame period as an interval between a frame data generation start time of the first frame data and a frame data generation start time of the second frame data, and providing the frame period to the frame data generation unit, the second frame data being generated continuously with the first frame data; and an accumulation time setting unit to set the accumulation time and provide the accumulation time to the frame data generating unit, wherein the accumulation time setting unit is further configured to switch between setting the accumulation time based on time and setting the accumulation time according to the number of event data in response to a user operation, The accumulation time setting unit sets the accumulation time in units of divided regions, wherein the divided regions are obtained by dividing the frame into a plurality of regions based on a plurality of objects corresponding to the frame of each of the first frame data and the second frame data.

11. A program product for causing a computer to function as: a frame data generating unit configured to generate frame data based on event data indicating a change in an electrical signal of a pixel generating the electrical signal by performing photoelectric conversion, the event data occurring within a cumulative time from a frame generation start timing to a frame generation end timing; a frame period setting unit configured to set a frame period as an interval between a frame data generation start time of the first frame data and a frame data generation start time of the second frame data, the second frame data being generated continuously with the first frame data, and to provide the frame period to the frame data generating unit; and an accumulation time setting unit configured to set the accumulation time and provide the accumulation time to the frame data generating unit, wherein the accumulation time setting unit is further configured to switch between setting the accumulation time based on time and setting the accumulation time according to the number of event data in response to a user operation, The accumulation time setting unit sets the accumulation time in units of divided regions, wherein the divided regions are obtained by dividing the frame into a plurality of regions based on a plurality of objects corresponding to the frame of each of the first frame data and the second frame data.

12. A data processing device, comprising: a frame data generating unit configured to generate first frame data and second frame data in such a manner that the first accumulation time and the second accumulation time overlap, the first frame data is based on event data indicating a change in an electrical signal of a pixel that generates an electrical signal by performing photoelectric conversion and occurring within the first accumulation time from a first frame generation start time to a first frame generation end time, and the second frame data is based on the event data occurring within the second accumulation time from a second frame generation start time to a second frame generation end time; and an accumulation time setting unit configured to set the first accumulation time and the second accumulation time and provide the first accumulation time and the second accumulation time to the frame data generating unit, wherein the accumulation time setting unit is further configured to switch between setting the first accumulation time and the second accumulation time based on time and setting the first accumulation time and the second accumulation time depending on the number of event data in response to a user operation, The accumulation time setting unit sets the first accumulation time and the second accumulation time in units of divided regions, wherein the divided regions are obtained by dividing the frame into a plurality of regions based on a plurality of objects corresponding to the frame of each of the first frame data and the second frame data.

13. A data processing method, comprising: generating first frame data and second frame data in such a manner that the first accumulation time and the second accumulation time overlap, the first frame data is based on event data indicating a change in an electrical signal of a pixel that generates an electrical signal by performing photoelectric conversion and that occurs within the first accumulation time from a first frame generation start time to a first frame generation end time, and the second frame data is based on the event data that occurs within the second accumulation time from a second frame generation start time to a second frame generation end time, In response to a user operation, switching is performed between setting the first accumulation time and the second accumulation time based on time and setting the first accumulation time and the second accumulation time depending on the amount of the event data, dividing each of the first frame data and the second frame data into segmented areas, and The first accumulation time and the second accumulation time are set in units of the divided regions based on a plurality of objects corresponding to the frame.

14. A program product for causing a computer to function as: a frame data generating unit configured to generate first frame data and second frame data in such a manner that the first accumulation time and the second accumulation time overlap, The first frame data is based on event data indicating a change in an electrical signal of a pixel that generates an electrical signal by performing photoelectric conversion and occurring within the first accumulation time from a first frame generation start time to a first frame generation end time, and the second frame data is based on the event data occurring within the second accumulation time from a second frame generation start time to a second frame generation end time; and an accumulation time setting unit configured to set the first accumulation time and the second accumulation time and provide the first accumulation time and the second accumulation time to the frame data generating unit, in, The accumulation time setting unit is further configured to switch between setting the first accumulation time and the second accumulation time based on time and setting the first accumulation time and the second accumulation time depending on the number of event data in response to a user operation, The accumulation time setting unit sets the first accumulation time and the second accumulation time in units of divided regions, wherein the divided regions are obtained by dividing the frame into a plurality of regions based on a plurality of objects corresponding to the frame of each of the first frame data and the second frame data.

15. A data processing method, the method causing a data processing device to generate frame data based on event data indicating a change in an electrical signal of a pixel that generates an electrical signal by performing photoelectric conversion, The event data occurs within the cumulative time from the start time of frame generation to the end time of frame generation, in, The number of frame data generated per unit time varies according to the number of event data per unit time, In which, in response to user operation, the setting of the accumulation time is switched between setting the accumulation time based on time and setting the accumulation time according to the number of event data, wherein the accumulation time is set in units of segmented areas, and the segmented areas are obtained by dividing the frame into multiple areas based on multiple objects corresponding to the frame of the frame data.

16. The data processing method according to claim 15, wherein: The larger the number of the event data per unit time, the larger the number of frame data generated per unit time.

17. A data processing device, comprising: A data processing unit, the data processing unit comprising: an event generating unit having a pixel generating an electrical signal by performing photoelectric conversion and configured to generate event data indicating occurrence of an event, the event being a change in the electrical signal of the pixel; a frame data generating unit configured to generate frame data based on the event data occurring within a cumulative time from a frame generation start time to a frame generation end time, a frame cycle setting unit configured to set a frame cycle as an interval between a frame data generation start time of first frame data and a frame data generation start time of second frame data, the second frame data being generated continuously with the first frame data, and to provide the frame cycle to the frame data generating unit, and an accumulation time setting unit configured to set the accumulation time and provide the accumulation time to the frame data generating unit, wherein the accumulation time setting unit is further configured to switch between setting the accumulation time based on time and setting the accumulation time according to the number of event data in response to a user operation, The accumulation time setting unit sets the accumulation time in units of divided regions, wherein the divided regions are obtained by dividing the frame into a plurality of regions based on a plurality of objects corresponding to the frame of each of the first frame data and the second frame data.

18. The data processing apparatus according to claim 17, further comprising: An image processing unit is configured to perform image processing using the frame data.

19. The data processing apparatus according to claim 18, wherein: The image processing unit performs recognition processing on the frame data.

20. The data processing device according to any one of claims 17 to 19, wherein: The data processing device includes a semiconductor chip having a plurality of dies stacked on each other.

21. A data processing method, comprising: generating event data indicating occurrence of an event, the event being a change in an electrical signal of a pixel that generates an electrical signal by performing photoelectric conversion; The frame data generating unit generates frame data based on the event data occurring within a cumulative time from a frame generation start time to a frame generation end time; Setting a frame period as an interval between a frame data generation start time of the first frame data and a frame data generation start time of the second frame data, and providing the frame period to the frame data generation unit, the second frame data being generated continuously with the first frame data; an accumulation time setting unit to set the accumulation time and provide the accumulation time to the frame data generating unit, wherein the accumulation time setting unit is further configured to switch between setting the accumulation time based on time and setting the accumulation time according to the number of event data in response to a user operation, The accumulation time setting unit sets the accumulation time in units of divided regions, wherein the divided regions are obtained by dividing the frame into a plurality of regions based on a plurality of objects corresponding to the frame of each of the first frame data and the second frame data.

Images