Solid-state imaging devices and electronic equipment

Abstract

The present invention suppresses any reduction in the efficiency with which events are detected. A solid-state imaging device according to one embodiment of the present invention comprises: a pixel array unit (101) in which a plurality of pixels that each detect changes in the brightness of incident light are arrayed in a two-dimensional lattice shape; a signal processing unit (103) that generates first event data including position information for one or more pixels that detected a change in brightness; an adjustment unit (320) that adjusts the data amount of the first event data and generates second event data; and an output unit (109) that outputs the second event data to the outside, the adjustment unit being provided with a plurality of paths for adjusting the data amount of the first event data by mutually different methods and generating the second event data, and the output unit outputting, to the outside, at least one of the second event data outputted from each of the plurality of paths.

Description

[Technical Field]

[0001] This disclosure relates to solid-state imaging devices and electronic equipment. [Background technology]

[0002] In solid-state imaging devices using CMOS (Complementary Metal Oxide Semiconductor) and the like, an asynchronous solid-state image sensor has been proposed that detects brightness changes for each pixel as events in real time (for example, Patent Document 1). Such solid-state image sensors that detect events for each pixel are also called EVS (Event-based Vision Sensor) or DVS (Dynamic Vision Sensor). [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Special Publication No. 2017-535999 [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] EVS outputs event detection data for each pixel as event data in a predetermined format. However, when many events are detected at once, the amount of event data may exceed the transmission bandwidth. When this happens, the event data that cannot be transmitted must be discarded, which reduces the efficiency of event detection.

[0005] Therefore, this disclosure proposes a solid-state imaging device and electronic equipment capable of suppressing a decrease in event detection efficiency. [Means for solving the problem]

[0006] To solve the above problems, one embodiment of a solid-state imaging device according to the present disclosure comprises: a pixel array unit in which a plurality of pixels that detect changes in the brightness of incident light are arranged in a two-dimensional grid; a signal processing unit that generates first event data including position information of one or more pixels that have detected changes in brightness; an adjustment unit that generates second event data by adjusting the amount of data of the first event data; and an output unit that outputs the second event data to the outside. The adjustment unit comprises a plurality of paths that generate second event data by adjusting the amount of data of the first event data in different ways, and the output unit outputs at least one of the second event data output from each of the plurality of paths to the outside. [Brief explanation of the drawing]

[0007] [Figure 1] This is a schematic diagram showing a general configuration example of an electronic device equipped with a solid-state imaging device according to the first embodiment. [Figure 2] This is a block diagram showing an example system configuration of an electronic device according to the first embodiment. [Figure 3] This is a block diagram showing a schematic configuration example of an EVS device according to the first embodiment. [Figure 4] This is a circuit diagram showing a schematic configuration example of an event pixel according to the first embodiment. [Figure 5] This diagram illustrates cases where encoding increases the amount of data. [Figure 6] This is a block diagram showing a schematic configuration example of an event signal processing circuit according to the first embodiment. [Figure 7] This is a block diagram showing an example of a data volume adjustment unit according to the first embodiment. [Figure 8] This figure shows event data relating to the first example of the first embodiment. [Figure 9] This figure shows event data relating to a second example of the first embodiment. [Figure 10] This figure shows event data relating to a third example of the first embodiment. [Figure 11]This figure shows event data relating to the fourth example of the first embodiment. [Figure 12] This is a diagram (part 1) illustrating the coding of one line of event data using the first coding scheme (Hcomp coding) according to the first embodiment. [Figure 13] This is a diagram (part 2) illustrating the encoding of one line of event data using the first coding scheme (Hcomp coding) according to the first embodiment. [Figure 14] This is a diagram (part 3) illustrating the coding of one line of event data using the first coding scheme (Hcomp coding) according to the first embodiment. [Figure 15] This is a diagram (part 1) illustrating the coding of one line of event data using the second coding scheme (Run Length coding) according to the first embodiment. [Figure 16] This is a diagram (part 2) illustrating the coding of one line of event data using the second coding scheme (Run Length coding) according to the first embodiment. [Figure 17] This is a diagram (part 3) illustrating the coding of one line of event data using the second coding scheme (Run Length coding) according to the first embodiment. [Figure 18] This is a diagram (part 1) illustrating the encoding of one line of event data using the third coding scheme (Huffman coding) according to the first embodiment. [Figure 19] This is a diagram (part 2) illustrating the encoding of one line of event data using the third coding scheme (Huffman coding) according to the first embodiment. [Figure 20] This is a diagram (part 3) illustrating the encoding of one line of event data using the third coding scheme (Huffman coding) according to the first embodiment. [Figure 21]This is a diagram (part 1) illustrating the encoding of one line of event data using the fourth encoding scheme (Light Huffman coding) according to the first embodiment. [Figure 22] This is a diagram (part 2) illustrating the encoding of one line of event data using the fourth encoding scheme (Light Huffman coding) according to the first embodiment. [Figure 23] This is a diagram (part 3) illustrating the encoding of one line of event data using the fourth encoding scheme (Light Huffman coding) according to the first embodiment. [Figure 24] This is a diagram (part 1) illustrating the encoding of one line of event data using the fifth encoding scheme (Event Distance coding) according to the first embodiment. [Figure 25] This is a diagram (part 2) illustrating the encoding of one line of event data using the fifth encoding scheme (Event Distance coding) according to the first embodiment. [Figure 26] This is a diagram (part 3) illustrating the encoding of one line of event data using the fifth encoding scheme (Event Distance coding) according to the first embodiment. [Figure 27] This is a diagram (part 1) illustrating the encoding of one line of event data using the sixth encoding scheme (Entropy coding) according to the first embodiment. [Figure 28] This is a diagram (part 2) illustrating the encoding of one line of event data using the sixth encoding scheme (Entropy coding) according to the first embodiment. [Figure 29] This is a diagram (part 3) illustrating the encoding of one line of event data using the sixth encoding scheme (Entropy coding) according to the first embodiment. [Figure 30] This is a diagram illustrating the first signal processing (Optical Flow calculation) according to the first embodiment. [Figure 31]This is a diagram illustrating the second signal processing (feature point extraction) according to the first embodiment. [Figure 32] This is a diagram illustrating the third signal processing (ROI extraction) according to the first embodiment. [Figure 33] This figure shows an example of image data extracted by the third signal processing (ROI extraction) according to the first embodiment. [Figure 34] This is a schematic diagram showing the output format of frame data relating to the first output example of the first embodiment. [Figure 35] This is a schematic diagram showing the output format of frame data relating to the second output example of the first embodiment. [Figure 36] This is a schematic diagram showing the output format of frame data related to the third output example of the first embodiment. [Figure 37] This is a schematic diagram showing the output format of frame data relating to the fourth output example of the first embodiment. [Figure 38] This is a schematic diagram showing the output format of frame data related to the fifth output example of the first embodiment. [Figure 39] This is a block diagram showing a schematic configuration example of an event signal processing circuit according to the second embodiment. [Figure 40] This is a block diagram showing a schematic configuration example of an event signal processing circuit according to a modified example of the second embodiment. [Figure 41] This is a hardware configuration diagram showing an example of a computer that implements the functions of the information processing device related to this disclosure. [Modes for carrying out the invention]

[0008] Embodiments of the present disclosure will be described in detail below with reference to the drawings. In the following embodiments, the same parts will be denoted by the same reference numerals to avoid redundant descriptions.

[0009] Furthermore, this disclosure will be explained in the order of the items shown below. 1. First Embodiment 1.1 Example System Configuration 1.2 Example of an EVS device configuration 1.3 Example of circuit configuration for a unit pixel 1.4 Data Transmission 1.5 Example of a schematic configuration of an event signal processing circuit 1.6 Example of event data 1.7 Examples of encoding schemes 1.7.1 First encoding scheme 1.7.2 Second encoding scheme 1.7.3 Third Encoding Scheme 1.7.4 The fourth encoding scheme 1.7.5 The fifth encoding scheme 1.8 Examples of signal processing 1.8.1 First signal processing 1.8.2 Second signal processing 1.8.3 Third signal processing 1.9 Example of Frame Data Output 1.9.1 Example of First Output 1.9.2 Example of Second Output 1.9.3 Example of the third output 1.9.4 Example of the fourth output 1.9.5 Example of the 5th Output 1.10 Summary 2. Second Embodiment 2.1 Example of a schematic configuration of an event signal processing circuit 2.2 Summary 2.3 Variations 3. Hardware Configuration

[0010] 1. First Embodiment First, the first embodiment will be described in detail with reference to the drawings.

[0011] 1.1 Example System Configuration Figure 1 is a schematic diagram showing a general configuration example of an electronic device equipped with a solid-state imaging device according to the first embodiment, and Figure 2 is a block diagram showing a system configuration example of the electronic device according to the first embodiment.

[0012] As shown in Figure 1, the electronic device 1 according to this embodiment includes a laser light source 1010, an illumination lens 1030, an imaging lens 1040, an EVS device 100 as a solid-state imaging device, and a system control unit 1060.

[0013] As shown in Figure 2, the laser light source 1010 is composed of, for example, a vertical-cavity surface-emitting laser (VCSEL) 1012 and a light source drive unit 1011 that drives the VCSEL 1012. However, it is not limited to the VCSEL 1012, and various light sources such as LEDs (Light Emitting Diodes) may be used. Furthermore, the laser light source 1010 may be a point light source, a surface light source, or a line light source. In the case of a surface light source or a line light source, the laser light source 1010 may have a configuration in which, for example, a plurality of point light sources (e.g., VCSELs) are arranged in one or two dimensions. In this embodiment, the laser light source 1010 may emit light in a wavelength band different from the visible light wavelength band, such as infrared (IR) light.

[0014] The irradiation lens 1030 is positioned on the emission side of the laser light source 1010 and converts the light emitted from the laser light source 1010 into irradiation light with a predetermined divergence angle.

[0015] The imaging lens 1040 is positioned on the light-receiving surface side of the EVS device 100 and forms an image of the incident light onto the light-receiving surface of the EVS device 100. The incident light may also include reflected light emitted from the laser light source 1010 and reflected by the subject 901.

[0016] The EVS device 100, as detailed below, consists of, for example, a light-receiving unit 1022 in which event-detecting pixels (hereinafter referred to as event pixels) are arranged in a two-dimensional grid, and a sensor control unit 1021 that drives the light-receiving unit 1022 to generate frame data based on event data detected by the event pixels.

[0017] The system control unit 1060 is composed of, for example, a processor (CPU) and drives the VCSEL 1012 via the light source drive unit 1011. The system control unit 1060 also controls the EVS device 100 in synchronization with the control of the laser light source 1010, thereby acquiring event data detected in response to the emission / extinction of the laser light source 1010.

[0018] For example, the illumination light emitted from the laser light source 1010 is projected onto the subject (also called the object to be measured or an object) 901 through the illumination lens 1030. This projected light is reflected by the subject 901. The light reflected from the subject 901 then passes through the imaging lens 1040 and enters the EVS device 100. The EVS device 100 receives the reflected light from the subject 901 and generates event data, and based on the generated event data, generates frame data, which is a single image. The frame data generated by the EVS device 100 is supplied to the application processor 1070 of the electronic device 1. The application processor 1070, as an information processing device, performs predetermined processing such as image processing and recognition processing on the frame data input from the EVS device 100.

[0019] 1.2 Example of an EVS device configuration Next, a schematic example of the EVS device 100 will be described. Figure 3 is a block diagram showing a schematic example of the EVS device according to this embodiment. As shown in Figure 3, the EVS device 100 includes a pixel array section 101, X arbiters 104A and Y arbiters 104B, an event signal processing circuit 103, a system control circuit 105, and an output interface (I / F) 109.

[0020] The pixel array section 101 has a configuration in which multiple event pixels 20, each detecting an event based on a change in the brightness of incident light, are arranged in a two-dimensional grid. In the following description, the row direction (also called the row direction) refers to the arrangement direction of pixels in a pixel row (horizontal direction in the drawing), and the column direction (also called the column direction) refers to the arrangement direction of pixels in a pixel column (vertical direction in the drawing).

[0021] Each event pixel 20 is equipped with a photoelectric conversion element that generates an electric charge corresponding to the brightness of the incident light. When it detects a change in the brightness of the incident light based on the photocurrent flowing out from the photoelectric conversion element, it outputs a request to the X arbiter 104A and Y arbiter 104B to read the event signal, and outputs an event signal indicating that an event has been detected, in accordance with the arbitration by the X arbiter 104A and Y arbiter 104B.

[0022] Each event pixel 20 detects the presence or absence of an event based on whether or not a change occurs in the photocurrent corresponding to the brightness of the incident light that exceeds a predetermined threshold. For example, each event pixel 20 detects an event when the brightness change exceeds a predetermined threshold (positive event) or falls below a predetermined threshold (negative event).

[0023] When the event pixel 20 detects an event, it outputs a request to the X arbiter 104A and the Y arbiter 104B, respectively, requesting permission to output an event signal indicating the occurrence of the event. Then, when the event pixel 20 receives a response from the X arbiter 104A and the Y arbiter 104B indicating permission to output the event signal, it outputs an event signal to the event signal processing circuit 103.

[0024] The X arbiter 104A and the Y arbiter 104B mediate requests for the output of event signals supplied from each of the multiple event pixels 20, and send a response based on the mediation result (permission / denial of event signal output), as well as a reset signal to reset event detection, to the event pixel 20 that output the request.

[0025] The event signal processing circuit 103 generates and outputs event data by performing predetermined signal processing on the event signal input from the event pixel 20.

[0026] As described above, the change in photocurrent generated at the event pixel 20 can also be understood as a change in the amount of light (brightness change) incident on the photoelectric conversion part of the event pixel 20. Therefore, an event can also be said to be a change in the amount of light (brightness change) of the event pixel 20 that exceeds a predetermined threshold. The event data representing the occurrence of an event includes at least positional information such as coordinates representing the position of the event pixel 20 where the change in amount of light as an event occurred. In addition to positional information, the event data can also include the polarity of the change in amount of light.

[0027] Regarding the sequence of event data output from event pixel 20 at the time an event occurs, as long as the intervals between event data are maintained at the time the event occurred, it can be said that the event data implicitly contains time information representing the relative time when the event occurred.

[0028] However, if the interval between event data is no longer maintained as it was at the time the event occurred, such as when the event data is stored in memory, the time information implicitly included in the event data will be lost. For this reason, the event signal processing circuit 103 may include time information representing the relative time the event occurred, such as a timestamp, in the event data before the interval between event data is no longer maintained as it was at the time the event occurred.

[0029] (Other components) The system control circuit 105 is composed of a timing generator that generates various timing signals, and based on the various timings generated by the timing generator, it controls the driving of the X arbiter 104A, the Y arbiter 104B, and the event signal processing circuit 103.

[0030] The output interface 109 sequentially outputs the event data, which is output line by line from the event signal processing circuit 103, to the processing unit 200.

[0031] 1.3 Example of circuit configuration for a unit pixel Next, an example of the circuit configuration of the event pixel 20 will be described. Figure 4 is a circuit diagram showing a schematic example of the event pixel configuration according to the first embodiment. Figure 4 shows an example configuration in which a single comparator performs time-division multiplexing for the detection of positive and negative events.

[0032] Here, events may include, for example, positive events indicating that the change in photocurrent has exceeded an upper threshold, and negative events indicating that the change has fallen below a lower threshold. In this case, the event data representing the occurrence of an event may include, for example, one bit indicating the occurrence of the event and one bit indicating the polarity of the event that occurred. The event pixel 20 can be configured to have a function to detect only positive events, or it can be configured to have a function to detect only negative events.

[0033] As shown in Figure 4, the event pixel 20 includes, for example, a photoelectric conversion unit PD and an address event detection circuit 210. The photoelectric conversion unit PD is composed of, for example, a photodiode, and converts the charge generated by photoelectric conversion of incident light into a photocurrent I photo It is released as a photocurrent I. photo This then flows into the address event detection circuit 210.

[0034] The address event detection circuit 210 includes a light receiving circuit 212, a memory capacity 213, a comparator 214, a reset circuit 215, an inverter 216, and an output circuit 217.

[0035] The light receiving circuit 212 is composed of, for example, a current-voltage conversion circuit, and the photocurrent I that flows out from the photoelectric conversion unit PD is received. photo Voltage V pr Convert to this. Here, the voltage V is the same as the light intensity (luminance). pr The relationship is usually logarithmic. That is, the photoreceiving circuit 212 receives the photocurrent I corresponding to the intensity of the light irradiated onto the photoreceiving surface of the photoelectric conversion unit PD. photo The voltage V is a logarithmic function. pr Convert to [this]. However, the photocurrent I photo and voltage Vpr The relationship with [it] is not limited to a logarithmic relationship.

[0036] The photocurrent I output from the light-receiving circuit 212 photo corresponding voltage V pr After passing through the memory capacity 213, becomes the inverted (-) input which is the first input of the comparator 214 as the voltage V diff The comparator 214 is usually composed of differential pair transistors. The comparator 214 uses the threshold voltage V b given from the system control circuit 105 as the non-inverted (+) input which is the second input, and performs detection of a positive event and detection of a negative event in a time-sharing manner. Also, after detection of a positive event / negative event, the event pixel 20 is reset by the reset circuit 215.

[0037] The system control circuit 105 outputs the voltage V b as the voltage V in the stage of detecting a positive event in a time-sharing manner, outputs the voltage V on in the stage of detecting a negative event, and outputs the voltage V off in the stage of performing reset. The voltage V reset is output. The voltage V reset is a value between the voltage V on and the voltage V off and is preferably set to a value intermediate between the voltage V on and the voltage V off Here, the "intermediate value" includes not only the case of being strictly the intermediate value but also the case of being substantially the intermediate value, and various variations occurring in design or manufacture are tolerated.

[0038] Also, the system control circuit 105 outputs an ON selection signal to the event pixel 20 in the stage of detecting a positive event, outputs an OFF selection signal in the stage of detecting a negative event, and outputs a global reset signal (Global Reset) in the stage of performing reset. The ON selection signal is a selection switch SW onThe OFF selection signal is provided as a control signal to the selector switch SW provided between comparator 214 and output circuit 217. off It is given as a control signal to that.

[0039] The comparator 214, in the stage of detecting a positive event, uses the voltage V on and voltage V diff Compare this with the voltage V diff Voltage V on When it exceeds the limit, the photocurrent I photo The positive event information "On" is output as the comparison result, indicating that the amount of change has exceeded the upper threshold. The positive event information "On" is inverted by the inverter 216, and then selected by the select switch SW on It is supplied to the output circuit 217 through this.

[0040] The comparator 214, in the stage of detecting a negative event, uses the voltage V off and voltage V diff Compare this with the voltage V diff Voltage V off When it falls below this value, the photocurrent I photo The negative event information Off, which indicates that the amount of change has fallen below the lower threshold, is output as the comparison result. The negative event information Off is selected by the select switch SW off It is supplied to the output circuit 217 through this.

[0041] The reset circuit 215 is a reset switch SW RS The configuration includes a 2-input OR circuit 2151 and a 2-input AND circuit 2152. Reset switch SW RS This is connected between the inverting (-) input terminal and the output terminal of comparator 214, and when it is turned ON (closed), it selectively short-circuits the inverting input terminal and the output terminal.

[0042] OR circuit 2151 is a selection switch SW on After the positive event information is turned On, and the selection switch SW offThe negative event information Off, obtained through the process, is used as the second input. The AND circuit 2152 takes the output signal of the OR circuit 2151 as one input and the global reset signal provided by the system control circuit 105 as the other input, and when either positive event information On or negative event information Off is detected and the global reset signal is active, the reset switch SW RS Set it to the ON (closed) state.

[0043] In this way, the output signal of the AND circuit 2152 becomes active, and the reset switch SW RS This short-circuits the inverting input terminal and output terminal of the comparator 214, performing a global reset on the event pixel 20. As a result, the reset operation is performed only on the event pixel 20 where an event was detected.

[0044] The output circuit 217 has a configuration comprising a negative event output transistor NM1, a positive event output transistor NM2, and a current source transistor NM3. The negative event output transistor NM1 has a memory (not shown) at its gate for holding negative event information Off. This memory consists of the gate parasitic capacitance of the negative event output transistor NM1.

[0045] Similar to the negative event output transistor NM1, the positive event output transistor NM2 has a memory (not shown) at its gate for holding positive event information "On". This memory consists of the gate parasitic capacitance of the positive event output transistor NM2.

[0046] During the readout phase, the negative event information (Off) held in the memory of the negative event output transistor NM1 and the positive event information (On) held in the memory of the positive event output transistor NM2 are transferred to the readout circuit 130 via output lines nRxOff and nRxOn for each pixel row of the pixel array section 101 by a row selection signal being applied from the system control circuit 105 to the gate electrode of the current source transistor NM3. The readout circuit 130 is, for example, a circuit provided within the event signal processing circuit 103 (see Figure 3).

[0047] As described above, the event pixel 20 has an event detection function that uses one comparator 214 to perform time-division detection of positive events and negative events under the control of the system control circuit 105.

[0048] 1.4 Data Transmission From the EVS device 100, which has the above configuration, event data generated at each pixel is output to the outside via the output I / F 109. However, if an event data volume exceeding the transmission bandwidth is generated, the event data that cannot be transmitted must be discarded, which presents a problem as it reduces the efficiency of event detection.

[0049] Another method for efficiently transmitting event data without exceeding the transmission bandwidth is to compress the event data. Various encoding schemes can be used to compress event data, but each encoding scheme has its own limitations. Therefore, if event data generated in a scenario where the encoding scheme is inadequate is encoded, the amount of data may increase compared to the original data.

[0050] Figure 5 illustrates the case where encoding increases the amount of data. In Figure 5, (a) and (b) show the amount of data when one line of event data output from the EVS device is not encoded, and (c) and (d) show the amount of data when one line of event data is encoded. Furthermore, (a) and (c) show the case where the number of events detected in one line is small (1 pixel), and (b) and (d) show the case where the number of events detected in one line is large (all pixels).

[0051] As shown in Figures 5(a) and (b), without encoding, the amount of event data for one line is the same whether the number of events detected on a line is small (a) or large (b). On the other hand, as shown in (c) and (d), with encoding, when the number of events detected on a line is small (c), the amount of data can be reduced compared to the case without encoding (a), but when the number of events is large (d), the amount of data increases compared to the case without encoding (b). Thus, in scenarios where the adopted encoding method is not well-suited, encoding the event data can increase the amount of data.

[0052] In such cases, to avoid sensor malfunctions caused by the inability to transmit the encoded and compressed data, event data that could not be transmitted had to be discarded, which presented a challenge in that it reduced the efficiency of event detection.

[0053] Therefore, in this embodiment, the amount of event data is reduced using multiple methods, and the event data that has been reduced to an amount that does not exceed the transmission bandwidth is transmitted via the output I / F 109. This makes it possible to suppress the occurrence of event data that cannot be transmitted, and thus suppress the decrease in event detection efficiency.

[0054] 1.5 Example of a schematic configuration of an event signal processing circuit The reduction of the amount of event data is performed, for example, in the event signal processing circuit 103 of the EVS device 100. Figure 6 is a block diagram showing a schematic configuration example of the event signal processing circuit according to this embodiment. As shown in Figure 6, the event signal processing circuit 103 includes a signal processing unit 310, a data amount adjustment unit 320, and a selection unit 330.

[0055] (Signal processing unit 310) The signal processing unit 310 generates event data (x, y, p, t) that includes the location information (x address, y address) of the pixel where the event was detected, the polarity (p) of the detected event, and a timestamp (t) indicating the time the event was detected, based on the event signals output from the pixel array unit 101, for example, on a row-by-row basis. The X arbiter 104A in the EVS device 100 is configured to permit the output of event signals on a pixel row-by-row basis. In this case, event signals are output to the event signal processing circuit 103 from event pixels in one or more pixel sequences to which the permitted output pixel row belongs, and which are permitted to output by the Y arbiter 104B.

[0056] (Data volume adjustment unit 320) The data volume adjustment unit 320 reduces the amount of data output from the signal processing unit 310, for example, in line units, using one of several methods. Figure 7 shows an example of the data volume adjustment unit 320 according to this embodiment.

[0057] As shown in Figure 7, the data volume adjustment unit 320 includes, for example, an uncompressed unit 321, a first compression unit 322, a second compression unit 323, a first signal processing unit 324, and a second signal processing unit 325. In other words, the data volume adjustment unit 320 in this example is configured to output a total of five types of event data: the original event data (hereinafter also referred to as uncompressed data) and four types of event data whose data volume has been adjusted by compression and / or signal processing. Note that the types of event data output by the data volume adjustment unit 320 are not limited to five types, but may be two or more types.

[0058] The uncompressed section 321 is composed of, for example, a buffer, and outputs the input event data (hereinafter also simply referred to as input data) as is.

[0059] The first compression unit 322 is a compression unit that compresses event data using a first encoding scheme. The second compression unit 323 is a compression unit that compresses event data using a second encoding scheme different from the first encoding scheme. The second encoding scheme may include the same encoding scheme as the first encoding scheme, but with different parameters. Furthermore, the encoding schemes that the first and second compression units 322 and 323 may employ may be reversible encoding schemes that allow the original event data to be restored, or irreversible encoding schemes that do not allow the original event data to be restored. Examples of encoding schemes that the first and second compression units 322 and 323 may employ will be described later.

[0060] The first signal processing unit 324 is a signal processing unit that adjusts the amount of event data by a first signal processing. The second signal processing unit 325 is a signal processing unit that adjusts the amount of event data by a second signal processing that is different from the first signal processing. The second signal processing may include the same signal processing as the first signal processing, but with different parameters. Furthermore, the signal processing that the first signal processing unit 324 and the second signal processing unit 325 may employ may be reversible signal processing that allows the original event data to be restored, or irreversible signal processing that does not allow the original event data to be restored. Examples of signal processing that the first signal processing unit 324 and the second signal processing unit 325 may employ will be described later.

[0061] (Selection section 330) The selection unit 330 selects event data from among two or more types of event data output from the data volume adjustment unit 320 that does not exceed the transmission bandwidth of the output I / F 109, and outputs the selected event data to the output I / F 109. For example, the selection unit 330 may output the event data with the smallest data volume from among two or more types of event data output from the data volume adjustment unit 320 to the output I / F 109.

[0062] Furthermore, for example, if the amount of uncompressed event data exceeds the transmission bandwidth of the output I / F 109, the selection unit 330 may output the event data with the smaller amount of data from the first compression unit 322 and the second compression unit 323 to the output I / F 109. In addition, if both the event data output from the first compression unit 322 and the second compression unit 323 exceed the transmission bandwidth of the output I / F 109, the selection unit 330 may output the event data with the smaller amount of data from the first signal processing unit 324 and the second signal processing unit 325 to the output I / F 109.

[0063] Furthermore, the selection unit 330 may output event data via a path specified externally. For example, if uncompressed data (or uncompressed unit 321) is selected externally, the selection unit 330 may output event data via the uncompressed unit 321; if compressed data (either the first compression unit 322 or the second compression unit 323) is selected externally, the selection unit 330 may output compressed event data via the compression unit of the selected encoding scheme; or if signal processing (either the first signal processing unit 324 or the second signal processing unit 325) is selected externally, the selection unit 330 may output data with reduced data volume due to the selected signal processing (for the sake of explanation, this signal processing result will also be referred to as event data). Note that external specification may be made, for example, via the system control circuit 105 (see Figure 3), or directly by input.

[0064] Furthermore, the selection unit 330 may output two or more types of event data from among the multiple types of event data that have been input to the output interface. For example, the selection unit 330 may output uncompressed event data and compressed or signal-processed event data to the output interface. Also, the output of two or more types of event data may be specified externally.

[0065] 1.6 Example of event data Next, we will explain some examples of event data that is input from the signal processing unit 310 to the data volume adjustment unit 320.

[0066] (Example 1) Figure 8 shows event data according to a first example of this embodiment. As shown in Figure 8, the event data generated by the signal processing unit 310 may be pixel-specific event data (x,y,p,t) including x and y addresses (x,y) indicating the position of the event pixel 20 on the pixel array unit 101 where the event was detected, polarity information (p) indicating the polarity of the detected event (positive event or negative event), and a timestamp (t) indicating the time when the event was detected. Alternatively, the signal processing unit 310 may input pixel-specific event data to the data amount adjustment unit 320 in order according to the time series of event occurrences.

[0067] (Example 2) Figure 9 shows event data according to a second example of this embodiment. As shown in Figure 9, the signal processing unit 310 may generate a count map M1 that maps the number of events detected at each pixel within one frame period according to the arrangement of event pixels 20. In generating the count map M1, positive events and negative events may be counted separately, or positive events and negative events may be counted without distinction.

[0068] (Example 3) Figure 10 shows event data according to a third example of this embodiment. As shown in Figure 10, the signal processing unit 310 may generate image data (hereinafter also referred to as frame data) M2 that indicates in 2 bits whether a positive event or a negative event was detected at each pixel, based on the event signals output from each pixel within one frame period. In this case, for pixels where both a positive event and a negative event are detected within the same frame period, the pixel value of the corresponding pixel may be determined based on the polarity of the last detected event.

[0069] (Example 4) Figure 11 shows event data according to a fourth example of this embodiment. As shown in Figure 11, the signal processing unit 310 may generate frame data M3 consisting of the address (x,y) of the event pixel 20 that detected an event within one frame period and the polarity (p) of the detected event. In this case, because of the limitation of being within one frame period, the pixel value of each pixel does not need to include a timestamp (t).

[0070] 1.7 Examples of encoding schemes Next, we will explain some examples of encoding schemes that can be applied to the first and second compression units 322 and 323 described above. In the following, we will illustrate Hcomp coding, Run Length coding, Huffman coding, Light Huffman coding, Event Distance coding, and Entropy coding, but we are not limited to these, and various encoding schemes may be adopted, such as the encoding scheme related to ISSCC (IEEE International Solid-State Circuits Conference) 2017 4.1 and the encoding scheme related to ISSCC 2020 5.10. Furthermore, in the following explanation, we will illustrate a case where one line consists of 16 pixels, a positive event is detected at the first pixel (leftmost pixel) of a line, negative events are detected at the 12th to 14th pixels from the beginning, and no events are detected at the other pixels. However, the input data to be encoded (compressed) is not limited to event data for one line, but may be transformed in various ways, such as frame data.

[0071] 1.7.1 First encoding scheme As a first coding scheme, we illustrate Hcomp coding, which converts event data into a data sequence represented by the address of the pixel where the event occurred and the polarity of the event. Figures 12 to 14 are diagrams illustrating the coding of one line of event data using the first coding scheme (Hcomp coding).

[0072] As shown in Figure 12, when one line of input data is input to the compression unit 3201 employing Hcomp coding, the compression unit 3201 stores the y address 'y' indicating the corresponding line at the beginning, followed by the x address 'x' of the pixel where the event was detected. i '(i is a non-negative integer) and the polarity of the detected event' i 'Pair with (x i ,p i Compressed data containing ) (hereinafter referred to as output data) is output.

[0073] To illustrate this with a concrete example, the values ​​following the initial y-address 'y' are, as shown in Figure 13, the value of the first pixel where a positive event was detected (x0,p0)=(0,0), the value of the 12th pixel where a negative event was detected (x1,p1)=(11,1), the value of the 13th pixel where a negative event was also detected (x2,p2)=(12,1), and the value of the 14th pixel where a negative event was also detected (x3,p3)=(13,1). Note that polarity p=0 indicates a positive event, and p=1 indicates a negative event.

[0074] Therefore, as shown in Figure 14, the output data ((y),(x0,p0),(x1,p1),(x2,p2),(x3,p3)) output from the compression unit 3201 is a data sequence of ((y),(0,0),(11,1),(12,1),(13,1)).

[0075] 1.7.2 Second encoding scheme As a second coding method, we illustrate Run Length coding, which converts event data into a data sequence represented by the data type and its number of consecutive occurrences. Figures 15 to 17 illustrate the coding of one line of event data using the second coding method (Run Length coding).

[0076] As shown in Figure 15, when one line of input data is input to the compression unit 3202 which employs Run Length coding, the compression unit 3202 stores the y address 'y' indicating the corresponding line at the beginning, followed by a value 'd' indicating the type of data (presence or absence of an event and its polarity). i 'and the number of consecutive values ​​(consecutive numbers)' c i ' and pair (d i ,c i Compressed data (output data) containing ) is output.

[0077] To illustrate this with a concrete example, the values ​​following the initial y address 'y' are as follows, as shown in Figure 16: (d0,c0)=(1,1) indicating that a positive event was detected at the first pixel; (d1,c1)=(0,10) indicating that no events were detected at the 10 consecutive pixels from the 2nd to the 11th; (d2,c2)=(2,3) indicating that a negative event was detected at the 3 consecutive pixels from the 12th to the 14th; and (d3,c3)=(0,2) indicating that no events were detected at the 2 consecutive pixels from the 15th to the 16th. Note that data type d=0 indicates that no event was detected, d=1 indicates a positive event, and d=2 indicates a negative event.

[0078] Therefore, as shown in Figure 17, the output data ((y),(d0,c0),(d1,c1),(d2,c2),(d3,c3)) output from the compression unit 3202 is a data sequence of ((y),(1,1),(0,11),(2,3),(0,2)).

[0079] 1.7.3 Third Encoding Scheme As a third coding scheme, we will illustrate Huffman coding, which converts event data into a data sequence that represents events according to their frequency of occurrence. Figures 18 to 20 illustrate the coding of one line of event data using the third coding scheme (Huffman coding).

[0080] As shown in Figure 18, when one line of input data is input to the compression unit 3203 which employs Huffman coding, the compression unit 3203 stores the y address 'y' indicating the corresponding line at the beginning, followed by the Huffman code 'h' associated with the bit sequence. i The output is compressed data (output data) that stores the '' and has a frequency table at the end that maps the bit sequence to the Huffman code.

[0081] To illustrate this with a concrete example, the input data (1,0,0,0,0,0,0,0,0,0,0,0,2,2,2,0,0) can be divided into bit sequences consisting of '2' which appears 3 times, '0000000000' which appears once, '00' which also appears once, and '1' which also appears once. Note that '0' indicates no event, '1' indicates a positive event, and '2' indicates a negative event. Therefore, as illustrated in Figure 19, if we follow the rule of assigning small Huffman codes to bit sequences that appear frequently, for example, by assigning the Huffman code '0' to bit sequence '2', the Huffman code '10 (=2)' to bit sequence '00', the Huffman code '110 (=6)' to bit sequence '00', and the Huffman code '111 (=7)' to bit sequence '1', the input data can be represented as (7,2,0,0,0,6).

[0082] Therefore, as shown in Figure 20, the output data ((y), h0, h1, h2, h3, h4) output from the compression unit 3203 will be a data sequence of ((y), 7, 2, 0, 0, 0, 6, (frequency table)).

[0083] 1.7.4 The fourth encoding scheme As a fourth coding scheme, we will illustrate Light Huffman coding, which converts event data into a data sequence where no event is represented by 1 bit and an event is represented by 2 bits. Figures 21 to 23 are diagrams illustrating the coding of one line of event data using the fourth coding scheme (Light Huffman coding).

[0084] As shown in Figure 21, when one line of input data is input to the compression unit 3204, which employs Light Huffman coding, the compression unit 3204 outputs output data in which no event is represented by 1 bit and an event is represented by 2 bits.

[0085] Specifically, as illustrated in Figure 22, if the input data is represented by 2 bits each for positive events ('01'), negative events ('10'), and no events ('00'), then, as illustrated in Figure 23, the output data is converted into a bit sequence where positive events are represented by 2 bits ('11'), negative events by 2 bits ('10'), and no events by 1 bit ('0'). However, the pixels converted to 1 bit are not limited to pixels with no events; for example, they may be the type of event that appears most frequently within a line (positive event, negative event, or no event).

[0086] 1.7.5 The fifth encoding scheme As a fifth coding method, we will illustrate Event Distance coding, which converts event data into a data sequence represented by the distance between events. Figures 24 to 26 are diagrams illustrating the coding of one line of event data using the fifth coding method (Event Distance coding).

[0087] As shown in Figure 24, when one line of input data is input to the compression unit 3205, which employs Event Distance coding, the compression unit 3205 stores the address 'y' indicating the line at the beginning, followed by the distance (in bits) from the previous event, 'l'. i 'and, the polarity of the event' i 'and the number of consecutive events of the same polarity' c i ' and pair (l i ,p i ,c i Compressed data (output data) containing ) is output.

[0088] To illustrate this with a concrete example, the values ​​following the initial y-address 'y' are, as shown in Figure 25, (l0,p0,c0)=(0,0,1), indicating that a positive event was detected in the first pixel, and (l1,p1,c1)=(10,1,3), indicating that a negative event was detected in three consecutive pixels, from the 12th to the 14th pixel, which are 10 pixels away from the initial positive event pixel. Note that polarity p=0 indicates a positive event, and p=1 indicates a negative event.

[0089] Therefore, as shown in Figure 26, the output data ((y),(l0,p0,c0),(l1,p1,c1)) output from the compression unit 3205 is a data sequence of ((y),(0,0,1),(10,1,3)).

[0090] 1.7.6 The sixth encoding scheme As a sixth coding method, we will illustrate entropy coding, which converts event data into a data sequence represented by the distance between events and the data pattern. Figures 27 to 29 are diagrams illustrating the coding of one line of event data using the sixth coding method (entropy coding). For the sake of understanding, this explanation will illustrate a case where one line consists of 16 pixels, positive events are detected at the 1st, 4th, and 6th pixels from the beginning of the line, negative events are detected at the 5th, 7th, and 12th to 14th pixels from the beginning, and no events are detected at the remaining pixels.

[0091] As shown in Figure 27, when one line of input data is input to the compression unit 3206 employing entropy coding, the compression unit 3206 groups the bit sequence of the input data according to a predetermined rule. Based on this group, the address 'y' indicating the corresponding line is stored at the beginning, followed by a value 'l' indicating the distance from the previous event of the same polarity. i 'and, the polarity of the event' i 'and values ​​indicating the presence or absence of same-polarity events within the group's i 'pt' is a value that indicates the sequence pattern of same-polarity events within the group. i ' and pair (li ,p i ,s i ,pt i It outputs compressed data (output data) containing the data.

[0092] To illustrate this with a concrete example, as shown in Figure 28, the compression unit 3206 separates the input data shown in (a) into a sequence of positive events (P) and a sequence of negative events (N), as shown in (b). In the sequence of positive events (P) and the sequence of negative events (N) shown in (b), pixels containing events are indicated by '1'.

[0093] Next, the compression unit 3206 defines a group for each event sequence. In defining a group, the pixel where the first event is detected is used as the starting point, and a predetermined number of pixels from that pixel onward are set as one group. Therefore, for the event sequence of positive events (P) shown in Figure 28(b), from left to right, a total of four pixels (i.e., pixels 1 to 4) starting from the first pixel where the first positive event appears, which is a predetermined number of pixels (3 in this example), are defined as group Gp0. Following group Gp0, a total of four pixels (i.e., pixels 6 to 9) starting from the sixth pixel where the first positive event appears, which is a predetermined number of pixels (=3), are defined as group Gp1. Similarly, for a sequence of negative events (N), starting from the 5th pixel where the first negative event appears, a total of four pixels (i.e., pixels 5 through 8) are defined as group Gn0, and following group Gn0, a total of four pixels (i.e., pixels 12 through 15) starting from the 12th pixel where the first negative event appears are defined as group Gn1, a total of three pixels (i.e., pixels 12 through 15).

[0094] In this way, when the event sequences of positive events (P) and negative events (N) are grouped separately, the compression unit 3206 generates event data (l0,p0,s0,pt0)=(0,0,0,1001) from group Gp0 in the event sequence of positive events (P) with l0='0', p0='0', s0='0', and pt0='1001'. The compression unit 3206 also generates event data (l1,p1,s1)=(2,0,1) from group Gp1 with l1='2', p1='0', and s1='1'. Similarly, the compression unit 3206 generates event data (l2,p2,s2,pt2)=(5,1,0,1010) from group Gn0 in the event sequence of negative events (N), where l2='5', p2='1', s2='0', and pt2='1010'. The compression unit 3206 also generates from group Gn1, l i ='4', p i ='1', s i ='0', pt i Generate event data (l3,p3,s3,pt3)=(4,1,0,1110) for ='1110'.

[0095] Note, p i ='0' indicates a positive event, p i ='1' indicates a negative event. Also, s i ='0' indicates that events of the same polarity exist within the same group, s i ='1' indicates that there are no events of the same polarity within the same group. If there are no events of the same polarity within the same group, the value 'pt' indicates the array pattern of events of the same polarity within the group, as in the event data for group Gp1. i The ' can be omitted.

[0096] Therefore, as shown in Figure 29, the output data ((y),(l0,p0,s0,pt0),(l1,p1,s1),(l2,p2,s2,pt2),(l3,p3,s3,pt3)) output from the compression unit 3206 is a data sequence of ((y),(0,0,0,1001),(2,0,1),(5,1,0,1010),(4,1,0,1110)).

[0097] 1.8 Examples of signal processing Next, we will explain some examples of signal processing that can be applied to the first signal processing unit 324 and the second signal processing unit 325 described above. The signal processing performed by the first signal processing unit 324 and the second signal processing unit 325 are examples of methods for adjusting the amount of event data. In addition, we will illustrate Optical Flow calculation, feature point extraction, and ROI (Region of Interest) extraction below, but we are not limited to these and various signal processing such as object recognition and noise reduction may be applied, and signal processing using machine learning such as DNN (Deep Neural Network), CNN (Convolutional Neural Network), and RNN (Recurrent Neural Network) may also be applied.

[0098] 1.8.1 First signal processing As an example of the first signal processing method, we will illustrate the case where Optical Flow calculation is performed on frame data reconstructed from input event data. Figure 30 is a diagram illustrating the first signal processing method (Optical Flow calculation).

[0099] The input data to the signal processing unit 3301, which employs Optical Flow calculation, may be either the event data exemplified using Figures 8 to 11 above, or other event data such as frame data. In this explanation, we will illustrate the case where the input data is the event data related to the first example illustrated using Figure 8.

[0100] As shown in Figure 30, the signal processing unit 3301 reconstructs frame data from event data input during one frame period, and calculates, for example, block-level optical flow from the frame data of one or more previous frames that were previously reconstructed and the frame data of the currently reconstructed frame. The calculated optical flow may include, for example, the amount of movement and movement speed of each block. Note that a block may be a single pixel or an area of ​​M × N pixels (where M and N are integers of 1 or more).

[0101] In this way, when the optical flow is calculated on a block-by-block basis, the signal processing unit 3301 calculates the address (x) indicating the position of each block that makes up the frame data. i ,y i ) and dis_x which shows the optical flow in the x direction of the block. i And, dis_y, which shows the optical flow in the y direction of the block. i Output data including (x i ,y i ,dis_x i ,dis_y i ) generates. Note that the output data includes a timestamp 't' as time information indicating the time of the current frame. j '(j is an integer greater than or equal to 1) may be included.

[0102] 1.8.2 Second signal processing As a second signal processing method, we will illustrate the case where feature point extraction is performed on frame data reconstructed from input event data. Figure 31 is a diagram illustrating the second signal processing method (feature point extraction).

[0103] The input data to the signal processing unit 3302, which employs feature point extraction, may be either event data or other event data such as frame data, as illustrated using Figures 8 to 11 above, similar to the first signal processing which employs Optical Flow calculation. In this explanation, we will illustrate the case where the input data is the event data related to the first example illustrated using Figure 8.

[0104] As shown in FIG. 31, the signal processing unit 3302 executes a process of reconstructing frame data from the event data input during one frame period and extracting feature points from the reconstructed frame data. Then, the signal processing unit 3302 outputs data (x i , y i ) including the address (x i , y i ) indicating the position of the extracted feature points. Note that the output data may include a time stamp 't j HID=11]]' as time information indicating the time of the current frame.

[0105] 1.8.3 Third Signal Processing As an example of the third signal processing, a case where ROI extraction is performed on the frame data reconstructed from the input event data is illustrated. FIG. 32 is a diagram for explaining the third signal processing (ROI extraction).

[0106] The input data to the signal processing unit 3303 for which ROI extraction is adopted may be any of the other event data such as the event data or frame data exemplified above using FIGS. 8 to 11 in the same manner as the first and second signal processings described above. In this description, a case where the event data according to the first example exemplified using FIG. 8 is the input data is illustrated.

[0107] As shown in FIG. 32, the signal processing unit 3303 executes a process of reconstructing frame data from the event data input during one frame period and extracting an ROI from the reconstructed frame data. Then, the signal processing unit 3302 outputs data (x_sta i , y_sta i ) and an address (x_end i , y_end i ) indicating the end point position, and output data (x_sta i , x_end i , y_sta i , y_end iGenerate j The output data may include a time stamp 't' as time information indicating the time of the current frame.

[0108] In addition to or instead of the output data indicating the position of the ROI calculated as described above, the signal processing unit 3303 may output the image data G31 cut out as the ROI from the frame data as shown in FIG. 33.

[0109] 1.9 Output Example of Frame Data Next, several examples will be given and explained for the output form of the frame data by the output I / F 109.

[0110] The output I / F 109 can adopt interface standards such as MIPI (Mobile Industry Processor Interface) CSI (Camera Serial Interface)-2 (C-PHY, D-PHY, etc.), I3C (Improved Inter Integrated Circuits), and ALPDP (Acquisition Law Professional Development Program). When these interface standards are adopted, the output I / F 109 can output the event data after data volume adjustment to the outside in units of frames. Therefore, below, several examples will be given and explained for the output of the frame data when MIPI CSI-2 is adopted.

[0111] 1.9.1 First Output Example Figure 34 is a schematic diagram showing the output format of frame data related to the first output example. As shown in Figure 34, in the first output example, a frame header FS indicating the beginning of the frame data, a line header PH indicating the beginning of each line data, a line footer PF indicating the end of each line data, line data Event sandwiched between the line header PH and the line footer PF, and a frame footer FE indicating the end of the frame data are output. Between the frame header FS and the frame footer FE, the line data Event of all lines constituting the frame data is included.

[0112] Furthermore, each line data Event may include event data for all pixels constituting each line (e.g., positive event, negative event, or no event), as well as a y-address indicating the position of the line, and flags indicating whether the line data is uncompressed, compressed using a specific encoding scheme, or the result of a particular signal processing method.

[0113] However, as mentioned above, in this embodiment, the line data Event is encoded compressed data. Therefore, it is also possible to use the output format exemplified below.

[0114] 1.9.2 Example of Second Output Figure 35 is a schematic diagram showing the output format of frame data related to the second output example. As shown in Figure 35, in the second output example, in the same format as the output format related to the first output example, the amount of line data Event sandwiched between the line header PH and the line footer PF is reduced by processing by the data volume adjustment unit 320. As a result, the amount of data in one frame output from the output I / F 109 is reduced.

[0115] 1.9.3 Example of the third output Figure 36 is a schematic diagram showing the output format of frame data for the third output example. As shown in Figure 36, in the third output example, in the same format as the output format for the first output example, the output of line data Event is omitted for lines where no events have been detected. As a result, the amount of data per frame output from output I / F109 is reduced.

[0116] 1.9.4 Example of the fourth output Figure 37 is a schematic diagram showing the output format of frame data related to the fourth output example. As shown in Figure 37, in the fourth output example, by combining the second and third output examples described above, the amount of line data Events sandwiched between the line header PH and the line footer PF is reduced, and the output of line data Events is omitted for lines where no events have been detected. As a result, the amount of data in one frame output from the output I / F 109 is further reduced.

[0117] 1.9.5 Example of the 5th Output Figure 38 is a schematic diagram showing the output format of frame data related to the fifth output example. In the second and fourth output examples described above, the amount of line data Event sandwiched between the line header PH and the line footer PF is reduced, thus shortening the data transfer time for one line. Therefore, in the fifth output example, the output interval of the horizontal synchronization signal is made variable according to the amount of data transferred for each line. For example, for each line, the horizontal synchronization signal is output when the output of the line footer PF is completed. This makes it possible to shorten the transfer time for each line, and thus shorten the transfer time for one frame of data. Needless to say, if data transfer for lines where no events have been detected is omitted, the transfer time will be shortened even further.

[0118] 1.10 Summary As described above, this embodiment includes a configuration that adjusts the amount of event data to be output using multiple methods (including uncompressed data), and selectively outputs compressed data (including uncompressed data) whose data volume does not exceed the transmission bandwidth. This reduces the possibility of event data that cannot be transmitted, and thus suppresses the decrease in event detection efficiency due to the discarding of event data.

[0119] 2. Second Embodiment Next, a second embodiment will be described in detail with reference to the drawings. In the following description, redundant explanations will be omitted by referencing the embodiments described above for configurations similar to those described above.

[0120] The electronic equipment and solid-state imaging device according to this embodiment may be the same as the electronic equipment 1 and solid-state imaging device (EVS device 100) described with reference to Figures 1 to 4 in the first embodiment. However, in this embodiment, the event signal processing circuit 103 in the EVS device 100 is replaced with the event signal processing circuit 203 shown in Figure 39.

[0121] 2.1 Example of a schematic configuration of an event signal processing circuit Figure 39 is a block diagram showing a schematic configuration example of an event signal processing circuit according to this embodiment. As shown in Figure 39, the event signal processing circuit 203 includes a signal processing unit 310, a scene determination unit 340, and a data volume adjustment unit 320. The signal processing unit 310 and the data volume adjustment unit 320 may be the same as those in the first embodiment, for example.

[0122] (Scene determination unit 340) The scene determination unit 340 determines the current scene based on event data input from the signal processing unit 310, and based on the determined scene, selects an encoding method (including uncompressed data) and / or signal processing method (hereinafter also referred to as a path) suitable for the current scene from the data volume adjustment unit 320. The scene determination unit 340 then inputs the event data input from the signal processing unit 310 to one or more selected paths (one or more of the uncompressed unit 321, first compression unit 322, second compression unit 323, first signal processing unit 324, and second signal processing unit 325). As a result, compressed data (including uncompressed data) generated by encoding or signal processing is output to the output I / F 109 from the path to which the event data was input.

[0123] The scene determination unit 340 may employ various methods for scene determination, such as an event count analysis algorithm, an event firing rate analysis algorithm, a cluster analysis algorithm, or a machine learning-based scene determination. For example, the event count analysis algorithm determines the current scene based on the number of event data points detected within a predetermined period (e.g., one frame period). The event firing rate analysis algorithm determines the current scene by analyzing the event firing rate using a probabilistic model. The cluster analysis algorithm determines the current scene by classifying event data into clusters based on factors such as the pixel address where the event was detected or the polarity of the event, and then analyzing the results. Machine learning-based scene determination determines the current scene by inputting the received event data into a trained model and evaluating the results. However, the method is not limited to these, and various other scene determination methods may be used.

[0124] 2.2 Summary As described above, in this embodiment, before encoding / signal processing the event data, the current scene is determined, and the event data is input to a suitable encoding method (including uncompressed) and / or signal processing method estimated from the determined scene. This makes it possible to stop the operation of unused paths (one or more of the uncompressed unit 321, first compression unit 322, second compression unit 323, first signal processing unit 324, and second signal processing unit 325), thereby suppressing the increase in power consumption of the EVS device 100.

[0125] Furthermore, since the other configurations, operations, and effects may be the same as those of the embodiments described above, a detailed explanation is omitted here.

[0126] 2.3 Variations Furthermore, as shown in Figure 40, it is also possible to combine the configuration according to the first embodiment and the configuration according to the second embodiment described above. In that case, the selection unit 330 may output to the output I / F 109 one or more compressed data (including uncompressed data) with a data volume that does not exceed the transmission bandwidth from two or more compressed data (including uncompressed data) input via each of the two or more paths selected by the scene determination unit 340.

[0127] 3. Hardware Configuration The processing units 200 / 600 according to the embodiments, modifications thereof, and application examples described above can be realized by a computer 1000 having a configuration such as that shown in Figure 41. Figure 41 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the processing units 200 / 600. The computer 1000 has a CPU 1100, RAM 1200, ROM (Read Only Memory) 1300, HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The various parts of the computer 1000 are connected by a bus 1050.

[0128] The CPU 1100 operates based on programs stored in the ROM 1300 or HDD 1400, and controls various parts. For example, the CPU 1100 loads the programs stored in the ROM 1300 or HDD 1400 into the RAM 1200 and executes processing corresponding to various programs.

[0129] ROM1300 stores boot programs such as the BIOS (Basic Input Output System) executed by CPU1100 when computer 1000 starts up, as well as programs that depend on the computer 1000's hardware.

[0130] HDD1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU1100 and data used by such programs. Specifically, HDD1400 is a recording medium that records a projection control program according to this disclosure, which is an example of program data 1450.

[0131] The communication interface 1500 is an interface for the computer 1000 to connect to an external network 1550 (e.g., the Internet). For example, the CPU 1100 can receive data from other devices or transmit data it has generated to other devices via the communication interface 1500.

[0132] The input / output interface 1600 includes the I / F unit 18 described above and is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from input devices such as keyboards and mice via the input / output interface 1600. The CPU 1100 also transmits data to output devices such as displays, speakers, and printers via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs recorded on a predetermined recording medium (media). Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase Change Rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, or semiconductor memory.

[0133] For example, when computer 1000 functions as a processing unit 200 / 600 according to the above embodiment, the CPU 1100 of computer 1000 implements at least one of the following functions by executing a program loaded onto RAM 1200: event data processing unit 202, RGB data processing unit 402, ToF data processing unit 502, object recognition processing unit 204, and display information generation unit 205. The HDD 1400 stores the program and the like according to this disclosure. The CPU 1100 reads and executes the program data 1450 from HDD 1400, but as another example, these programs may be obtained from other devices via an external network 1550.

[0134] While embodiments of this disclosure have been described above, the technical scope of this disclosure is not limited to the embodiments described above, and various modifications are possible without departing from the gist of this disclosure. Furthermore, components from different embodiments and modifications may be combined as appropriate.

[0135] Furthermore, the effects described in each embodiment of this specification are merely illustrative and not limiting, and other effects may also occur.

[0136] Furthermore, this technology can also be configured as follows. (1) A pixel array section in which multiple pixels that detect changes in the brightness of incident light are arranged in a two-dimensional grid, A signal processing unit that generates first event data including position information of one or more pixels that have detected a change in brightness, An adjustment unit that adjusts the amount of data in the first event data to generate second event data, An output unit that outputs the aforementioned second event data to the outside, Equipped with, The adjustment unit comprises a plurality of paths that generate second event data by adjusting the amount of data of the first event data in different ways, The output unit outputs at least one of the second event data output from each of the multiple paths to the outside. Solid-state imaging device. (2) Each of the aforementioned multiple paths includes one or more first paths that generate the second event data by encoding the first event data. The solid-state imaging device described in (1) above. (3) The one or more first passes include a pass that encodes the first event data using at least one of Hcomp coding, Run Length coding, Huffman coding, Light Huffman coding, Event Distance coding, and Entropy coding. The solid-state imaging device described in (2) above. (4) Each of the aforementioned multiple paths includes one or more second paths that generate the second event data by signal processing the first event data. A solid-state imaging device as described in any one of (1) to (3) above. (5) The one or more second passes include a pass that performs signal processing on the first event data using at least one of the following: Optical Flow calculation, feature point extraction, and ROI (Region of Interest) extraction. The solid-state imaging device described in (4) above. (6) The aforementioned multiple paths include a third path that outputs the first event data as is. A solid-state imaging device as described in any one of (1) to (5) above. (7) The adjustment unit inputs the first event data received from the signal processing unit to each of the plurality of paths, and inputs the second event data output from each of the plurality of paths to the output unit. The output unit outputs one or more of the second event data output from each of the multiple paths to the outside. A solid-state imaging device as described in any one of (1) to (6) above. (8) The output unit outputs to the outside a portion of the second event data, from among the second event data output from each of the multiple paths, such that the amount of data does not exceed the transmission bandwidth. The solid-state imaging device described in (7) above. (9) The output unit outputs the second event data with the smallest data volume from among the second event data output from each of the multiple paths to the outside. The solid-state imaging device described in (7) above. (10) The system further includes a determination unit that determines the scene based on the first event data, Based on the scene determined based on the first event data, the determination unit inputs the first event data into at least one of the multiple paths. A solid-state imaging device as described in any one of (1) to (6) above. (11) The output unit outputs the second event data to the outside on a frame-by-frame basis. A solid-state imaging device as described in any one of (1) to (10) above. (12) The output unit outputs frame data to the outside, in which the data length of each line has been adjusted by the adjustment unit. The solid-state imaging device described in (11) above. (13) The output unit outputs frame data to the outside that does not include line data for lines where no pixels with detected brightness changes in incident light exist. The solid-state imaging device described in (11) or (12) above. (14) The output unit outputs a horizontal synchronization signal as soon as the output of line data for each line is complete. A solid-state imaging device as described in any one of (11) to (13) above. (15) A solid-state imaging device as described in any one of (1) to (14) above, An information processing device that processes the second event data output from the solid-state imaging device, Electronic devices equipped with these features. [Explanation of Symbols]

[0137] 1 Electronic equipment 20 event pixels 100 EVS devices (solid-state imaging devices) 101 Pixel Array Section 103, 203 Event signal processing circuit 104A X Arbiter 104B Y Arbiter 105 System Control Circuit 109 Output I / F 310 Signal Processing Unit 320 Data Volume Adjustment Unit 321 Uncompressed section 322 First Compression Section 323 Second Compression Section 324 First Signal Processing Unit 325 Second Signal Processing Unit 330 Selection Section 340 Scene determination unit 3201~3205 Compression section 3301~3303 Signal Processing Unit 1010 Laser light source 1011 Light source drive unit 1012 VCSEL 1021 Sensor Control Unit 1022 Light receiving section 1030 Irradiation Lens 1040 imaging lens 1060 System Control Unit 1070 Application Processor

Claims

1. A pixel array section in which multiple pixels that detect changes in the brightness of incident light are arranged in a two-dimensional grid, A signal processing unit that generates first event data including position information of one or more pixels that have detected a change in brightness, An adjustment unit that adjusts the amount of data in the first event data to generate second event data, An output unit that outputs the aforementioned second event data to the outside, A determination unit that determines the scene based on the first event data, Equipped with, The adjustment unit comprises a plurality of paths that generate second event data by adjusting the amount of data of the first event data in different ways, The output unit outputs at least one of the second event data output from each of the plurality of paths to the outside. The determination unit inputs the first event data into one of the plurality of paths, which adjusts the amount of data of the first event data in a manner suitable for the scene determined based on the first event data, based on the scene determined based on the first event data. Solid-state imaging device.

2. Each of the aforementioned multiple paths includes one or more first paths that generate the second event data by encoding the first event data. The solid-state imaging apparatus according to claim 1.

3. The one or more first passes include a pass that encodes the first event data using at least one of Hcomp coding, Run Length coding, Huffman coding, Light Huffman coding, Event Distance coding, and Entropy coding. The solid-state imaging apparatus according to claim 2.

4. Each of the aforementioned plurality of paths includes one or more second paths that generate the second event data by signal processing the first event data. The solid-state imaging apparatus according to claim 1.

5. The one or more second passes include a pass that performs signal processing on the first event data using at least one of Optical Flow calculation, feature point extraction, and ROI (Region of Interest) extraction. The solid-state imaging apparatus according to claim 4.

6. The aforementioned multiple paths include a third path that outputs the first event data as is. The solid-state imaging apparatus according to claim 1.

7. The output unit outputs the second event data to the outside on a frame-by-frame basis. The solid-state imaging apparatus according to claim 1.

8. The output unit outputs frame data to the outside, in which the data length of each line has been adjusted by the adjustment unit. The solid-state imaging apparatus according to claim 7.

9. The output unit outputs frame data to the outside that does not include line data for lines where no pixels with detected brightness changes in incident light exist. The solid-state imaging apparatus according to claim 7.

10. The output unit outputs a horizontal synchronization signal as soon as the output of line data for each line is complete. The solid-state imaging apparatus according to claim 7.

11. A solid-state imaging device according to claim 1, An information processing device that processes the second event data output from the solid-state imaging device, Electronic devices equipped with these features.

Images