Imaging device, data processing method, information processing device, information processing method, and recording medium

Abstract

A sensor unit having: an event sensor portion in which event detection pixels for detecting a change in a light reception amount as an event are two-dimensionally arrayed and which generates an event signal indicating a detection result of the event; an imaging sensor portion which has a photoelectric conversion element and in which imaging pixels for generating a light reception signal indicating the light reception amount are two-dimensionally arrayed so as to obtain an imaged image; an event-influence presence / absence data generation portion which generates event-influence presence / absence data indicating presence / absence of an influence of a noise with respect to the event signal on the basis of a result of determination on whether or not a read operation of the imaging sensor portion at detection of an event by the event sensor portion; an imaging-influence presence / absence data generation portion which generates imaging-influence presence / absence data indicating presence / absence of an influence of a noise with respect to the light reception signal on the basis of a result of determination on whether or not the read operation of the event signal was performed at the reading-out of the light reception signal of the imaging sensor portion; and a transmission portion for transmitting the event-influence presence / absence data and the imaging-influence presence / absence data is provided.

Description

Imaging device, data processing method, information processing device, information processing method, and recording medium

[0001] The present technology relates to an imaging device, a data processing method, an information processing device, an information processing method, and a recording medium, and in particular to a technology for dealing with a sensor unit as a mixed sensor in which an event sensor unit and an imaging sensor unit are implemented, when noise components are included in the event data obtained by the event sensor unit and the captured image data obtained by the imaging sensor unit.

[0002] For example, event sensors that detect changes in the amount of received light as events, such as event-based vision sensors (EVS) and dynamic vision sensors (DVS), are known. An event sensor has a two-dimensional array of event detection pixels that detect changes in the amount of received light as events, and generates an event signal that indicates the result of the event detection.

[0003] In addition, some event sensors take the form of so-called mixed sensors, in which an imaging sensor that captures images such as RGB (R: red, G: green, B: blue) images is mounted within the same sensor unit (sensor device) (see, for example, Patent Documents 1 and 2 listed below).

[0004] International Publication No. WO 2020 / 158583 International Publication No. WO 2023 / 106232

[0005] In the above-described hybrid sensor, the event sensor and the image sensor are arranged close to each other, and therefore the signal readout operation of one sensor may act as a noise source for the other sensor. Specifically, the readout operation of the light-receiving signal in the image sensor may act as a noise source for the event sensor, inducing erroneous event detection, and the readout operation of the event signal in the event sensor may act as a noise source for the image sensor, inducing noise superposition in the captured image.

[0006] This technology has been developed in consideration of the above circumstances, and aims to enable a downstream processing unit to perform appropriate noise countermeasure processing when the event data and captured image data output by a sensor unit equipped with an event sensor and an image sensor contain noise components.

[0007] An imaging device according to the present technology includes a sensor unit having an event sensor section in which event detection pixels are two-dimensionally arranged, each detecting a change in the amount of received light as an event, and generating an event signal indicating a detection result of the event; an imaging sensor section in which imaging pixels, each having a photoelectric conversion element and generating a light reception signal indicating the amount of received light, are two-dimensionally arranged, and an image is obtained; an event influence presence / absence data generation section that generates event influence presence / absence data indicating whether or not the event signal is affected by noise, based on a result of determining whether or not a readout operation of the imaging sensor section was being executed when the event was detected by the event sensor section; an imaging influence presence / absence data generation section that generates imaging influence presence / absence data indicating whether or not the light reception signal is affected by noise, based on a result of determining whether or not a readout operation of the event signal was being executed when the light reception signal was being read out by the imaging sensor section; and a transmission section that transmits the event influence presence / absence data and the imaging influence presence / absence data. According to the above configuration, for a sensor unit as a mixed sensor in which an event sensor unit and an imaging sensor unit are implemented, in cases where noise components are contained in the event data obtained by the event sensor unit and the captured image data obtained by the imaging sensor unit, it is possible to output influence presence / absence data indicating the presence or absence of influence of a noise source to a downstream processing unit.

[0008] The information processing device according to the present technology includes an event sensor unit in which event detection pixels that detect a change in the amount of received light as an event are two-dimensionally arranged and generate an event signal that indicates a detection result of the event; an imaging sensor unit in which imaging pixels that have photoelectric conversion elements and generate a light reception signal that indicates the amount of received light are two-dimensionally arranged to obtain an image; an event influence presence / absence data generation unit that generates event influence presence / absence data that indicates whether the event signal is influenced by noise based on a result of determining whether a readout operation of the imaging sensor unit has been executed when the event sensor unit detected an event; and a detection unit that generates event influence presence / absence data that indicates whether a readout operation of the event signal has been executed when the light reception signal of the imaging sensor unit has been read out. and a data processing unit that receives as input at least one of event data obtained based on the event signal and captured image data obtained based on the light receiving signal, the event data being transmitted by a sensor unit having an imaging influence presence / absence data generating unit that generates imaging influence presence / absence data indicating whether or not noise has had an effect on the light receiving signal based on a result of determining whether or not noise has been present in the light receiving signal, and a transmitting unit that transmits the event influence presence / absence data and the imaging influence presence / absence data, and performs processing based on the influence presence / absence data corresponding to the input data, out of the event influence presence / absence data and the imaging influence presence / absence data, as processing for the input data as input data.As a result, at least one of the event data and the captured image data obtained by the sensor unit as a mixed sensor is processed based on the corresponding influence presence / absence data at a stage subsequent to the sensor unit.

[0009] 1 is a block diagram showing an example of the overall configuration of an imaging device as an embodiment. FIG. 1 is a diagram showing an example of an implementation form of an event sensor unit and an imaging sensor unit in an embodiment. FIG. 2 is an explanatory diagram of noise being superimposed on an image captured by the imaging sensor unit in response to a readout operation by the event sensor unit. FIG. 2 is a block diagram for explaining an example of the configuration of a sensor unit as a first embodiment. FIG. 3 is a diagram showing an example of region division of a sensing surface. FIG. 4 is a diagram showing an example of event-side statistical information. FIG. 5 is a diagram showing an example of imaging-side statistical information. FIG. 6 is a diagram showing an example of the structure of transmission data of captured image data. FIG. 7 is a diagram showing an example of the structure of transmission data of event data. FIG. 8 is a diagram showing an example of storage of event-side statistical information in transmission data of event data. FIG. 9 is a diagram showing an example of storage of imaging-side statistical information in transmission data of captured image data. FIG. 10 is an explanatory diagram of another example of region division. FIG. 11 is a flowchart showing the flow of processing executed by a sensor unit to transmit event data and event-side statistical information to a subsequent stage. FIG. 12 is a diagram showing an example of the data structure of imaging-side statistical information indicating whether or not each captured pixel is affected by noise. FIG. 13 is a diagram for explaining an example of the data structure of imaging-side statistical information in the case where statistical information relating to whether or not each pixel group of one horizontal line is affected by noise is shown. FIG. 14 is a diagram showing an example of the data structure of statistical information in the case where area definition information is included. FIG. 10 is a diagram for explaining an example data structure of event-side statistical information when showing temporal statistics. FIG. 11 is an explanatory diagram of an example data structure of transmission data when statistical information is stored for each minimum unit data. FIG. 12 is a diagram showing an example data structure of statistical information when statistical information is stored for each minimum unit data. FIG. 13 is a block diagram for explaining an example configuration of an imaging device as a second embodiment. FIG. 14 is an explanatory diagram of an implementation example of an electronic circuit unit in the second embodiment. FIG. 15 is an explanatory diagram of an example where an event sensor unit and an imaging sensor unit are formed in the same semiconductor layer. FIG. 16 is an explanatory diagram of an implementation form in which imaging pixels and event detection pixels are mixed in the same pixel array unit. FIG. 17 is a block diagram for explaining an example configuration of an imaging device as a modified example. FIG. 18 is a diagram showing an example data structure of a transmission frame in a modified example. FIG. 19 is a block diagram for explaining an example configuration for countermeasures taking into account saturation of event detection in the event sensor unit.

[0010] Hereinafter, with reference to the accompanying drawings, embodiments according to the present technology will be described in the following order: <1. First embodiment> [1-1. Example of overall configuration of imaging device] [1-2. Example of internal configuration of sensor unit] [1-3. Processing procedure] [1-4. Regarding subsequent countermeasure processing] [1-5. Another example of transmitted data] <2. Second embodiment> <3. Modified example> <4. Summary of embodiments> <5. Present technology>

[0011] 1 is a block diagram showing an example of the overall configuration of an imaging device 1 that is an embodiment of an imaging device according to the present technology. The imaging device 1 includes a sensor device in the form of a so-called hybrid sensor that includes, as a sensor unit 3 for sensing a subject, an event sensor unit 6 for detecting an event that is a change in the amount of received light that is equal to or greater than a predetermined amount, and an imaging sensor unit 7 for obtaining a captured image.

[0012] The event sensor unit 6 is a sensor unit that has a two-dimensional array of event detection pixels (event detection pixels Pe, which will be described later) that detect changes in the amount of received light as events, such as those known as an EVS (Event-based Vision Sensor) or a DVS (Dynamic Vision Sensor), and generates an event signal that indicates the result of the event detection.

[0013] Furthermore, with regard to the imaging sensor unit 7, the term "imaging" as used herein broadly refers to obtaining image data by capturing a subject using a light-receiving element. The light-receiving element here is not limited to one that receives visible light, but may also include one that receives invisible light. Furthermore, the term "image data" as used herein collectively refers to data consisting of multiple pixel data. The pixel data may include not only data indicating the amount of light received from the subject using a predetermined number of gradation levels, but also various other data related to the subject, such as data indicating the distance to the subject, data indicating polarization information, and data indicating temperature. In other words, the "image data" obtained by "imaging" may include data as a gradation image indicating the gradation value of the amount of light received for each pixel, data as a distance image indicating information on the distance to the subject for each pixel, data as a polarization image indicating polarization information for each pixel, and data as a thermal image indicating information on temperature for each pixel. Hereinafter, as an example, the imaging sensor unit 7 is assumed to be configured as a gradation sensor that obtains a gradation image, similar to a general image sensor. Examples of the gradation sensor include a CCD (Charge Coupled Device) image sensor and a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

[0014] To accommodate the sensor unit 3 as a mixed sensor, the imaging device 1 of this example is provided with an event detection optical system 2e and an imaging optical system 2i as optical systems for guiding light from the subject to the sensor. The event detection optical system 2e collects light from the subject and guides it to the sensing surface (light-receiving surface) of the event sensor unit 6. The imaging optical system 2i collects light from the subject and guides it to the sensing surface (light-receiving surface) of the imaging sensor unit 7.

[0015] Although the example shown here shows a configuration in which light from the subject is guided to each sensor unit via separate optical systems, it is also possible to adopt a configuration in which light from the subject is guided to each sensor unit via a common optical system.

[0016] The sensor unit 3 is provided with a transmitter 8 for transmitting the event data obtained by the event sensor 6 and the captured image data obtained by the imaging sensor 7 to the outside of the sensor unit 3. Although details will be described later, the event data is data indicating the detection result of an event, and means data indicating at least the detection time (occurrence time) of the event and the position where the event occurred on the sensing surface.

[0017] The sensor unit 3 in this example is configured to transmit event data and captured image data via separate transmission systems (transmission channels), and the transmission unit 8 has an event data transmission unit 8e for transmitting event data and an image data transmission unit 8i for transmitting captured image data.

[0018] The event data and captured image data transmitted from the sensor unit 3 are received by the receiving unit 4. In response to the fact that the event data and the captured image data are transmitted via separate systems as described above, the receiving unit 4 has an event data receiving unit 4e for receiving event data and an image data receiving unit 4i for receiving captured image data.

[0019] The event data and captured image data received by the receiving unit 4 are input to the data processing unit 5, where predetermined processing is performed. A specific example of the processing by the data processing unit 5 will be described later, but the data processing unit 5 in this example has an event data processing unit 5e that processes the event data input via the receiving unit 4, and an captured data processing unit 5i that processes the captured image data input via the receiving unit 4.

[0020] Here, in the imaging device 1 of this example, the sensor unit 3 has a plurality of semiconductor layers Ls, and the event sensor unit 6 and the image sensor unit 7 are formed on different semiconductor layers Ls. Specifically, as illustrated in the schematic diagram of Fig. 2, the sensor unit 3 of this example has at least two semiconductor layers Ls, that is, a first semiconductor layer Ls1 and a second semiconductor layer Ls2, and the event sensor unit 6 is formed on the first semiconductor layer Ls1 and the image sensor unit 7 is formed on the second semiconductor layer Ls2, so that the event sensor unit 6 and the image sensor unit 7 are formed on adjacent semiconductor layers Ls.

[0021] In the sensor unit 3 serving as a mixed sensor, the event sensor unit 6 and the image capture sensor unit 7 tend to be disposed close to each other, for example, formed on adjacent semiconductor layers Ls as shown in Fig. 2. For this reason, the signal readout operation of one sensor unit may act as a noise source for the other sensor unit. Specifically, the readout operation of the light-receiving signal in the image capture sensor unit 7 may act as a noise source for the event sensor unit 6, inducing erroneous event detection, and the readout operation of the event signal in the event sensor unit 6 may act as a noise source for the image capture sensor unit 7, inducing noise superimposition on the captured image.

[0022] 3A and 3B are diagrams illustrating how noise is superimposed on an image captured by the image sensor unit 7 in response to a readout operation by the event sensor unit 6. Fig. 3A schematically illustrates one frame of captured image data obtained by the image sensor unit 7, as well as horizontal lines of the captured image data. For illustrative purposes, the captured image data includes four horizontal lines, L1 to L4.

[0023] 3B is a schematic diagram illustrating the readout operation of the light reception signals for each horizontal line performed by the image sensor unit 7. In the hybrid sensor, while the light reception signals are being readout for each horizontal line, an event signal associated with an event detection by the event sensor unit 6 may be readout. During the period in which the event signal is being readout, noise resulting from the readout of the event signal may be superimposed on the pixels from which the light reception signals were readout during that period. For example, if an event signal is readout as shown in the illustrated example during the readout of one frame of light reception signals by the image sensor unit 7, noise may be superimposed on the captured image data in the shaded area in FIG. 3A.

[0024] On the other hand, when the image sensor unit 7 reads out the light reception signal, the operation of reading out the light reception signal may become a noise source for the event sensor unit 6, potentially inducing erroneous event detection. In other words, it can be estimated that an event detected during the period when the image sensor unit 7 reads out the light reception signal is likely to have been erroneously detected due to noise.

[0025] Therefore, in this embodiment, a method is adopted in which influence presence / absence data indicating whether or not the event data and the captured image data are affected by noise is generated and output so that appropriate countermeasure processing can be applied at a later stage of the sensor unit 3 to the event data and the captured image data that have become noise components due to the readout operations of each other as described above.

[0026] 4 is a block diagram illustrating an example of the configuration of the sensor unit 3 as an embodiment that generates and outputs the influence presence / absence data as described above. Note that FIG. 4 also shows the receiving unit 4 together with the example of the configuration of the sensor unit 3.

[0027] As shown in the figure, the sensor unit 3 includes, in addition to the event sensor unit 6 and the imaging sensor unit 7 shown in FIG. 1, an event-side statistical information generator 16e, an imaging-side statistical information generator 16i, an event-side readout / non-readout signal generator 17e, and an imaging-side readout / non-readout signal generator 17i.

[0028] The event sensor unit 6 includes a pixel array unit 10e in which event detection pixels Pe that detect changes in the amount of received light as events are arranged two-dimensionally, a Y arbiter 11, an X arbiter 12, and an event-side readout unit 15e.

[0029] In the pixel array unit 10e, each event detection pixel Pe includes a photoelectric conversion element, such as a photodiode, that performs photoelectric conversion on incident light to generate an electric charge, and an event detection circuit that detects an event based on the difference between the level of the charge signal obtained by the photoelectric conversion element and a reference level. In this example, the event detection circuit is capable of distinguishing between a positive event, in which the amount of received light increases by a certain amount or more, and a negative event, in which the amount of received light decreases by a certain amount or more.

[0030] When each event detection pixel Pe detects an event, regardless of whether it is a positive event or a negative event, it outputs a request signal to the Y arbiter 11 and the X arbiter 12 to request that an event signal be read (output) from itself. Then, each event detection pixel Pe outputs the event signal in accordance with arbitration by the Y arbiter 11 and the X arbiter 12 in response to the request signal. Specifically, in response to receiving a response signal output by the Y arbiter 11 in response to the request signal and a response signal output by the X arbiter 12 in response to the request signal, each event detection pixel Pe outputs a positive event signal indicating the detection result of a positive event and a negative event signal indicating the detection result of a negative event to the event-side readout unit 15e. This corresponds to the event signal readout operation.

[0031] It is not essential to provide both the Y arbiter 11 and the X arbiter 12, and it is also possible to provide only one of them.

[0032] The event-side readout unit 15e generates event data for each event detection pixel Pe based on the event signals (in this example, positive event signals and negative event signals) input (read out) from the event detection pixels Pe. The generated event data includes at least position information of the event detection pixel Pe that detected the event (address information in a two-dimensional coordinate system serving as the sensing surface: hereinafter referred to as "position information of the event occurrence pixel") and "detection time information" indicating the time the event was detected. In this example, both positive and negative events can be detected as events, and to accommodate this, the generated event data includes the position information and detection time information of the event occurrence pixel, as well as "event type information" indicating the type of the detected event (positive event or negative event).

[0033] For convenience of illustration, the figure shows only the signal line from the event detection pixel Pe where an event is detected as the signal line for reading out an event signal from the pixel array unit 10e to the event-side readout unit 15e, but the signal line is actually wired to the event-side readout unit 15e for each event detection pixel Pe.

[0034] The imaging sensor unit 7 includes a pixel array unit 10i in which imaging pixels Pi, each having a photoelectric conversion element such as a photodiode and generating a light reception signal indicating the amount of received light, are arranged two-dimensionally, a horizontal control circuit 13, and a vertical control circuit 14. The horizontal control circuit 13 and the vertical control circuit 14 are control circuits that perform control to realize exposure and readout for each horizontal line (each pixel row) of the pixel array unit 10i. Specifically, the horizontal control circuit 13 performs control to select a pixel row to be exposed or readout, and the vertical control circuit 14 performs control to select a pixel column to be exposed or readout.

[0035] The image pickup side readout unit 15i receives a light reception signal from each image pickup pixel Pi in a selected pixel row as a readout signal, and performs at least A / D (Analog to Digital) conversion on the readout signal to obtain a digital image signal, i.e., image data, for each pixel row.

[0036] For convenience of illustration, only a portion of the signal lines for outputting readout signals from the pixel array unit 10i to the imaging side readout unit 15i on the imaging sensor unit 7 side are shown, but these signal lines are wired for each pixel column of the pixel array unit 10i.

[0037] The event-side read / unread signal generator 17e generates an event-side read / unread signal that indicates whether the event signal is being read by the event-side reader 15e. Specifically, the event-side read / unread signal generator 17e generates a signal that indicates whether the event signal is being read by the event-side reader 15e, such as a signal that is H level when the event signal is being read by the event-side reader 15e and L level when the event signal is not being read.

[0038] The imaging-side readout / non-readout signal generator 17i generates an imaging-side readout / non-readout signal that indicates whether the imaging-side readout unit 15i is reading out the light-receiving signal. Specifically, the imaging-side readout / non-readout signal generator 17i generates a signal that indicates whether the light-receiving signal is being read out, such as a signal that is H level when the imaging-side readout unit 15i is reading out the light-receiving signal and L level when the light-receiving signal is not being read out.

[0039] The event-side statistical information generator 16e generates event-side statistical information, which is information indicating statistics relating to the presence or absence of noise influence, for the event data obtained by the event-side reader 15e. This event-side statistical information is an example of event-related influence presence or absence data indicating the presence or absence of noise influence on the event signal.

[0040] Here, the event-related influence data can be generated as data indicating whether or not noise is affecting each detected event, but in this example, information indicating the degree of noise influence is generated for each area Ar formed by dividing the sensing surface of the event sensor unit 6 as event-related influence data, as event-side statistical information.

[0041] 5 shows an example of dividing the region Ar. For example, as shown in the figure, the sensing surface of the event sensor unit 6 may be divided into three regions in the horizontal direction (horizontal line direction: row direction) to form three regions Ar: a first region Ar1, a second region Ar2, and a third region Ar3. In the example shown, the first region Ar1 and the third region Ar3, which are located at both ends in the horizontal direction, have the same width, and the width of the second region Ar2 is wider than the widths of the first region Ar1 and the third region Ar3.

[0042] 4, the event-side statistical information generator 16e generates, as event-side statistical information, information indicating the number of event-occurrence pixels estimated to be affected by noise for each of the first area Ar1, the second area Ar2, and the third area Ar3. Here, in this example, the event-occurrence pixels estimated to be affected by noise are identified as event detection pixels Pe from which light reception signals were being read out by the image sensor unit 7 at the time the event was detected.

[0043] In order to generate the event-side statistical information as described above, the event-side statistical information generator 16e determines, for each event-occurrence pixel, whether or not the light-receiving signal was being read out from the imaging sensor unit 7 at the time the event signal was read out. This determination is made using the imaging-side readout presence / absence signal generated by the imaging-side readout presence / absence signal generator 17i described above.

[0044] As shown in the figure, the event-side statistical information generator 16e in this example has branched signal lines for reading out event signals from each event detection pixel Pe in the pixel array unit 10e. The event-side statistical information generator 16e determines whether the imaging-side readout signal for an event detection pixel Pe from which an event signal was read was at an H level when the event signal was read out, thereby determining whether the event detection pixel Pe is an event detection pixel Pe that is estimated to be affected by noise. Furthermore, the event-side statistical information generator 16e identifies address information (coordinate information) for an event detection pixel Pe that is determined (estimated) to be affected by noise based on this determination. In this case, the address information can be identified based on which signal line the readout signal is on.

[0045] Here, the event detection pixel Pe estimated to be affected by noise as described above can be said to be a pixel that may have erroneously detected an event due to the influence of noise. In this sense, hereinafter, an event detected by the event detection pixel Pe will be referred to as a "noise event."

[0046] The event-side statistical information generation unit 16e generates information indicating the number of noise events for each of the first area Ar1, the second area Ar2, and the third area Ar3 as event-side statistical information based on the results of detecting noise events and identifying the address information of the event detection pixel Pe in which the noise event occurred based on the imaging-side readout presence / absence signal as described above.

[0047] 6 is a diagram showing an example of event-side statistical information. As shown in the figure, the event-side statistical information in this example can be generated as information in which, for example, the number of noise events that occurred in a first area Ar1 is stored in a data area with a data address of "1," the number of noise events that occurred in a second area Ar2 is stored in a data area with a data address of "2," and the number of noise events that occurred in a third area Ar3 is stored in a data area with a data address of "3."

[0048] Next, the imaging-side statistical information generator 16i will be described. The imaging-side statistical information generator 16i generates imaging-side statistical information, which is information indicating statistics relating to the presence or absence of noise influence, for the captured image data obtained by the imaging-side readout unit 15i. The imaging-side statistical information is an example of imaging influence presence / absence data indicating the presence or absence of noise influence on the light reception signal of the imaging sensor unit 7.

[0049] The imaging-side statistical information generator 16i in this example generates, as imaging-side statistical information, information indicating the degree of noise influence for each imaging pixel Pi. However, in this example, the imaging-side statistical information used as imaging-side statistical information is information indicating the degree of noise influence for each region Ar, similar to the event-side statistical information described above. Although not shown, the regions Ar in this example are formed by dividing the sensing surface of the imaging sensor unit 7. For example, in this example, the sensing surface of the imaging sensor unit 7 is divided into three regions, a first region Ar1, a second region Ar2, and a third region Ar3, in the row direction, as in the example of FIG. 5 . The imaging-side statistical information generator 16i in this example generates, as imaging-side statistical information, information indicating the number of imaging pixels Pi estimated to be affected by noise for each of the first region Ar1, second region Ar2, and third region Ar3.

[0050] In generating such imaging-side statistical information, the imaging-side statistical information generator 16i determines, for each imaging pixel Pi, whether or not an event signal has been read out from the event sensor unit 6 while the light-receiving signal is being read out from the pixel array unit 10i. This determination is made using the event-side readout presence / absence signal generated by the event-side readout presence / absence signal generator 17e.

[0051] As shown in the figure, the imaging-side statistical information generator 16i in this example has branched signal lines for reading out light-receiving signals from each pixel column in the pixel array unit 10i. When reading out light-receiving signals for each horizontal line, the imaging-side statistical information generator 16i determines whether the event-side readout presence / absence signal was at H level when the light-receiving signal was read out for each imaging pixel Pi. This makes it possible to determine (estimate) the presence or absence of noise influence due to the readout of the event signal for each imaging pixel Pi.

[0052] Hereinafter, the image pixel Pi that is estimated to be affected by noise as described above will be referred to as a "noise pixel."

[0053] The imaging-side statistical information generation unit 16i generates information indicating the number of noise pixels for each of the first area Ar1, the second area Ar2, and the third area Ar3 as event-side statistical information based on the results of detecting noise pixels based on the imaging-side readout presence / absence signal as described above.

[0054] 7 is a diagram showing an example of imaging-side statistical information. As shown in the figure, the imaging-side statistical information in this example can be generated as information in which, for example, the number of noise pixels generated in a first area Ar1 is stored in a data area with a data address of "1," the number of noise pixels generated in a second area Ar2 is stored in a data area with a data address of "2," and the number of noise pixels generated in a third area Ar3 is stored in a data area with a data address of "3."

[0055] In Figure 4, the transmitting unit 8 transmits to the receiving unit 4 the event data obtained by the event-side reading unit 15e and the captured image data obtained by the imaging-side reading unit 15i, as well as the event-side statistical information generated by the event-side statistical information generating unit 16e and the capturing-side statistical information generated by the imaging-side statistical information generating unit 16i.

[0056] As explained above in Figure 1, in this example, it is assumed that the event data and the captured image data are transmitted through separate systems, so the transmitting unit 8 has an event data transmitting unit 8e and an image data transmitting unit 8i.

[0057] Here, examples of the data structures of the transmission data of captured image data and event data will be described with reference to Figures 8 and 9. In this example, it is assumed that the MIPI (Mobile Industry Processor Interface) format is adopted as the transmission data format of the captured image data and event data in the transmission unit 8.

[0058] FIG. 8 shows an example of the structure of transmission data for captured image data. In this example, the transmission data format defines a transmission frame consisting of start data Dst, a plurality of minimum unit data Da, and end data Den, as shown in the figure. The minimum unit data here refers to the smallest unit of data that can be transmitted. The minimum unit data includes a header area for storing header information, a payload area following the header area as an area for storing actual data, and a footer area following the payload area for storing footer information. For example, one transmission frame transmits one frame's worth of image data for captured image data. In this case, each minimum unit data Da corresponds to one horizontal line in the captured image, and the payload area of ​​each minimum unit data Da stores image data (line data) for the corresponding horizontal line in the captured image data.

[0059] 9 shows an example of the structure of transmission data for event data. Specifically, it shows an example of the structure of transmission data in units of transmission frames described above. In the event sensor unit 6, events occur in response to the movement of a subject, and the timing and location of their occurrence are undefined. That is, unlike captured image data in which data for all pixels is output for each frame period, event data is not of the nature of being able to obtain data for all pixels within a certain unit time. For this reason, a method of transmitting event data is employed in which a certain unit period is defined as one transmission frame period, and the event data obtained within that period is sequentially transmitted in units of minimum unit data Da for each transmission frame period.

[0060] Specifically, time t1 in the diagram indicates the start timing of one transmission frame period. The event data transmitter 8e outputs start data Dst at time t1. Thereafter, each time the number of event data that can be stored in the minimum unit data Da is obtained, the minimum unit data Da is generated by storing the event data in the payload area and output to the receiver 4. Time t2 in the diagram indicates the end timing of one transmission frame period. The event data transmitter 8e outputs end data Den at the end timing of one transmission frame period. This completes the output of one transmission frame's worth of data.

[0061] It should be noted that one transmission frame period may be defined as, for example, a period of several msec or several tens of msec.

[0062] In this example, the event data transmitter 8e and the imaging data transmitter 8i store the event-side statistical information and the imaging-side statistical information in the transmission frame and transmit them, assuming that data transmission is performed in transmission frame units as described above. Specifically, as shown in FIG. 10, the event data transmitter 8e stores minimum unit data Db, in which the event-side statistical information is stored in the payload area, as the last minimum unit data of the transmission frame. In other words, the minimum unit data Db is output immediately before the end data Den. Furthermore, as shown in FIG. 11, the imaging data transmitter 8i stores the imaging-side statistical information as the last minimum unit data of the transmission frame and transmits it.

[0063] 4, the event data receiving unit 4e in the receiving unit 4 receives the event data and event-side statistical information transmitted by the event data transmitting unit 8e as described above, and outputs them to the event data processing unit 5e in the data processing unit 5 shown in Fig. 1. Also, the imaging data receiving unit 4i in the receiving unit 4 receives the captured image data and imaging-side statistical information transmitted by the imaging data transmitting unit 8i as described above, and outputs them to the imaging data processing unit 5i in the data processing unit 5 shown in Fig. 1.

[0064] The process performed by the event data processing unit 5e on the event data based on the event-side statistical information and the process performed by the imaging data processing unit 5i on the captured image data based on the imaging-side statistical information will be described in detail later.

[0065] Although the above example illustrates an example in which the area Ar is divided into three regions on both the event side and the imaging side, the number of regions Ar is not limited to this. For example, it is also possible to divide the area Ar into nine regions, as illustrated in Fig. 12. Fig. 12 illustrates an example in which, for the first area Ar1, second area Ar2, and third area Ar3 illustrated in Fig. 5, the first area Ar1 is vertically divided into three regions to create a new first area Ar1, fourth area Ar4, and seventh area Ar7; the second area Ar2 is vertically divided into three regions to create a new second area Ar2, fifth area Ar5, and eighth area Ar8; and the third area Ar3 is vertically divided into three regions to create a new third area Ar3, sixth area Ar6, and ninth area Ar9.

[0066] Regarding the division of the areas Ar, how to divide the target sensing surface is arbitrary, and can be determined appropriately depending on, for example, the processing mode of the downstream data processing unit 5. Also, it can be configured so that the user can select how to divide the target sensing surface, or can be configured so that the user can select whether to divide the sensing surface into multiple areas Ar in the first place.

[0067] [1-3. Processing Procedure] For confirmation, the flow of processing executed by the sensor unit 3 to transmit event data and event-side statistical information to a downstream side will be described with reference to the flowchart in Fig. 13. First, in step S101, the event-side statistical information generator 16e stores area division setting information in response to the sensing surface (pixel array unit 10e) of the event sensor unit 6 being divided into a predetermined number of areas Ar.

[0068] Next, in step S102, the event sensor unit 6 starts detecting an event. For example, the event sensor unit 6 starts detecting an event based on a user operation or the like.

[0069] In step S103 following step S102, the event-side statistical information generating unit 16e clears a counter for counting the number of noise events.

[0070] In step S104 following step S103, the event-side statistical information generation unit 16e waits for an event to be read out, that is, until a readout signal for an event signal is obtained on one of the signal lines wired from the pixel array unit 10e to each event detection pixel Pe.

[0071] If an event is read in step S104, the event side readout unit 15e performs an event data generation process in step S105, that is, a process of generating event data based on the read event signal (in this example, data including the position information of the event occurrence pixel, the detection time information, and the event type information).

[0072] Next, in step S106, the event-side statistical information generating unit 16e determines whether the imaging-side readout presence / absence signal is at H level. This corresponds to determining whether the event whose readout was confirmed in the previous step S104 is a noise event.

[0073] If it is determined in step S106 that the imaging-side readout presence / absence signal is at H level (i.e., the event that has occurred is a noise event), the event-side statistical information generator 16e proceeds to step S107 to perform an event-occurrence region identification process, i.e., a process of identifying the region Ar in which the event has occurred among the first region Ar1, the second region Ar2, and the third region Ar3.

[0074] In step S108 following step S107, the event-side statistical information generating unit 16e performs a process of adding 1 to the counter for the identified area Ar.

[0075] If the event that has occurred is a noise event, the process proceeds to step S108. If the event that has occurred is not a noise event (No in step S105), the process proceeds to step S105. In step S109, the event data transmitter 8e determines whether a predetermined number of pieces of event data have been accumulated. The predetermined number here is the upper limit number of pieces of event data that can be stored in the minimum unit data Da.

[0076] If the predetermined number of event data have not been accumulated, the process returns to the previous step S104, in which case the event-side statistical information generating unit 16e waits until the next event is read.

[0077] If it is determined that a predetermined number of event data have been accumulated, the event data transmitter 8e outputs minimum unit data Da in step S110. That is, the minimum unit data Da, in which a predetermined number of event data have been stored in the payload area, is output to the receiver 4 (event data receiver 4e).

[0078] Next, in step S111, the event data transmitter 8e determines whether a predetermined time has elapsed, i.e., whether one transmission frame period has ended. If one transmission frame period has not ended, the process returns to step S104. That is, the event-side statistical information generator 16e waits until the next event is read.

[0079] On the other hand, if it is determined that one transmission frame period has ended, the event data transmitter 8e performs processing to output statistical data in step S112. That is, the event data transmitter 8e outputs minimum unit data Db, in which the event-side statistical information generated based on the counting processing in step S108 is stored in the payload area, to the receiver 4 (event data receiver 4e).

[0080] When the process of step S112 is executed, the process returns to step S103, whereby the generation and transmission of event data and event-side statistical information are repeated for each predetermined frame.

[0081] 11, if one transmission frame period ends with the number of accumulated event data items less than a predetermined number, minimum unit data Da may be output in which only event data items less than the predetermined number are stored in the payload area. In this case, the remaining area of ​​the payload area may be padded with 0s or 1s to reshape the data to the same size as the other minimum unit data items before sending, or minimum unit data Da with a small amount of data in the payload area may be sent without padding.

[0082] [1-4. Countermeasure Processing in Later Stages] The countermeasure processing of the data processing unit 5 will be described. The data processing unit 5 may be configured as a processing unit that performs inference processing (hereinafter referred to as "AI processing") on input data using an AI (Artificial Intelligence) model. Specifically, the event data processing unit 5e may be configured to perform inference processing, such as object detection processing, using an AI model, with event data from the event data receiving unit 4e as input data. Furthermore, the imaging data processing unit 5i may be configured to perform inference processing, such as object recognition processing, using an AI model, with captured image data from the imaging data receiving unit 4i as input data. In this case, the imaging data processing unit 5i may use information about the object detection area obtained as a result of the inference processing of the event data processing unit 5e to perform object recognition processing using only the image of the object detection area.

[0083] In AI processing, it is common to calculate likelihood information indicating the reliability of an inference result as information on the inference result. For example, in object recognition processing using an AI model, the label of a subject and its likelihood are calculated in the inference processing. The label here refers to information indicating the classification of the subject according to a predetermined classification mode. For example, when subjects are classified into categories such as "people," "cars," "bicycles," "motorcycles," "traffic lights," "dogs," and "cats," the label is information indicating the classification of the subject.

[0084] When the data processing unit 5 performs AI processing on input data in this manner, a possible noise countermeasure process is to adjust the likelihood calculated as the inference result of the AI ​​processing based on the event-side statistical information and the influence data as the imaging-side statistical information. Specifically, for example, in an AI processing performed using captured image data output from the imaging data receiving unit 4i as input data, the imaging data processing unit 5i adjusts the likelihood calculated as the inference result based on the imaging-side statistical information output from the imaging data receiving unit 4i. In this case, the likelihood adjustment is performed for each region Ar of the captured image data, since the imaging-side statistical information indicates the number of noise pixels for each region Ar. Specifically, for example, a process is performed to reduce by a predetermined value the likelihood calculated for an object recognized in a region Ar where the number of noise pixels is equal to or greater than a predetermined number. This reduces the likelihood for an object recognized in a region Ar estimated to have a high degree of noise superposition, making it less likely that a recognition result will be output. In other words, the accuracy of the inference processing can be improved by preventing erroneous inference results from being output.

[0085] Although the above example shows a process of reducing the likelihood calculated for an object recognized in an area Ar where the number of noise pixels is equal to or greater than a predetermined number by a predetermined value, it is also possible to reduce the likelihood value of an object recognized in each area Ar by a subtraction value corresponding to the number of noise pixels for that area Ar. There are various possible methods for adjusting the likelihood based on statistical information indicating the degree of noise superposition for each area Ar, and the present invention is not limited to a specific method.

[0086] Although the above example shows that AI processing for event data and AI processing for captured image data are performed separately, it is also possible to configure a single AI processing to be performed using event data and captured image data as input data. In this case, it is possible to adjust the likelihood calculated as a result of inference processing using the event data and captured image data as input data, based on event-side statistical information and capture-side statistical information.

[0087] Alternatively, the AI ​​processing may be performed using fusion data of event data and captured image data as input data. When the fusion data is used as input data, the noise reduction processing and frame thinning processing described below may be performed on the fusion data.

[0088] The data processing unit 5 may be configured as a processing unit that performs noise reduction processing on input data. In this case, noise countermeasure processing may involve adjusting the strength of the noise reduction processing performed on the input data based on influence data as event-side statistical information or imaging-side statistical information.

[0089] Specifically, for example, in this case, the imaging data processing unit 5i adjusts the noise reduction effect for each region Ar based on the imaging-side statistical information output from the imaging data receiving unit 4i when performing noise reduction processing using the captured image data output from the imaging data receiving unit 4i as input data. In this case, the adjustment of the noise reduction effect for each region Ar can be performed so as to enhance the noise reduction effect for regions Ar with a high degree of noise superposition. For example, a noise reduction effect level can be defined for each region Ar according to the number of noise pixels, and noise reduction processing can be performed for each region Ar at an effect level corresponding to the number of noise pixels in that region Ar. Note that the number of stages to which the noise reduction effect level is adjusted according to the number of noise pixels can be determined arbitrarily. Furthermore, the adjustment of the noise reduction effect can be achieved, for example, by changing the coefficients of the filter used for noise reduction.

[0090] By adjusting the noise reduction effect based on the influence data as described above, it is possible to realize appropriate noise reduction processing according to the degree of noise superposition in the input data. In particular, by adjusting the noise reduction effect for each region Ar based on statistical information representing the degree of noise superposition for each region Ar, it is possible to avoid unnecessarily increasing the noise reduction effect for regions Ar with a small degree of noise superposition, thereby suppressing deterioration in image quality.

[0091] Another possible countermeasure against noise is to perform frame thinning processing on input data. In this case, the data processing unit 5 performs frame thinning processing on input data obtained in predetermined frame units as event data and captured image data, based on event-side statistical information and influence data as imaging-side statistical information.

[0092] Specifically, for example, in this case, the imaging data processing unit 5i calculates the total number of noise pixels in the entire frame for the imaging image data output from the imaging data receiving unit 4i based on the imaging side statistical information output from the imaging data receiving unit 4i, and if the calculated total number is greater than or equal to a predetermined threshold, it does not input the imaging image data, and if the calculated total number is less than the predetermined threshold, it inputs the imaging image data.

[0093] As a result, when predetermined processing such as the AI ​​processing described above is performed later on the captured image data output from the sensor unit 3, frame images with a high degree of noise superposition can be prevented from being input as images to be processed in the predetermined processing, thereby preventing a decrease in the processing accuracy of the predetermined processing and achieving appropriate noise countermeasure processing.

[0094] Here, it is possible to configure the system to perform both the frame thinning process described above and the noise reduction effect adjustment process described above.It is also possible to configure the system to perform these frame thinning process, noise reduction effect adjustment process, and likelihood adjustment together.

[0095] Although the above description only exemplifies the noise countermeasure processing in the data processing unit 5 by the imaging data processing unit 5i, the event data processing unit 5e can also perform countermeasure processing similar to the countermeasure processing of the imaging data processing unit 5i exemplified above. It is not essential that the noise countermeasure processing in the data processing unit 5 be performed on both the captured image data side and the event data side; it is sufficient to perform at least one of them. Needless to say, the countermeasure processing on the captured image data side is performed based on the imaging-side statistical information, and the countermeasure processing on the event data side is performed based on the event-side statistical information.

[0096] [1-5. Other Examples of Transmission Data] Here, in the above, an example was given in which the event-side statistical information and the imaging-side statistical information are generated as information indicating the number of noise events and the number of noise pixels for each region Ar, respectively. However, the event-side statistical information and the imaging-side statistical information can also be generated as information indicating the presence or absence of an influence for each event (for each event detection pixel Pe) and for each imaging pixel Pi, rather than for each region Ar.

[0097] FIG. 14 shows an example of the imaging-side statistical information in this case. The imaging-side statistical information in this case basically only needs to be generated as information capable of identifying noise pixels. In the illustrated example, as an example of imaging-side statistical information capable of identifying noise pixels, information is generated that includes information indicating the position of each noise pixel as well as information indicating the total number of noise pixels (the total number of noise pixels in one frame). Specifically, the imaging-side statistical information in this case is generated as information in which the total number of noise pixels is stored in a data area with data address "1," and information indicating the position of one noise pixel per address is stored in data areas with data addresses "2" and onward. In this case, the more noise pixels are detected, the greater the number of data areas with data address "2" and onward.

[0098] Incidentally, when the event-side statistical information is information indicating whether or not each event is affected, a data structure similar to that shown in FIG. 14 can be adopted.

[0099] Furthermore, the imaging-side statistical information may be generated not by region Ar but by horizontal line. In this case, the imaging-side statistical information generator 16i generates statistical information indicating the presence or absence of noise influence for each pixel group of one horizontal line. A specific example of the imaging-side statistical information is generated as information indicating lines containing noise pixels, as shown in FIGS. 15A and 15B . In the example of FIG. 15A , information indicating the position of only horizontal lines containing noise pixels is stored. In the example of FIG. 15B , information indicating the presence or absence of noise pixels (flag information) is stored for each horizontal line in the captured image data. Note that in both the examples of FIGS. 15A and 15B , the imaging-side statistical information may also include information indicating the total number of horizontal lines containing noise pixels, as shown in the figure.

[0100] Furthermore, although the above description assumes that the downstream processing unit (data processing unit 5) is aware of the division pattern of the region Ar, it is also possible that the downstream processing unit does not understand the division pattern of the region Ar. In this case, the statistical information may be generated as information including definition information for each region Ar, as shown in FIG. 16 . Specifically, FIG. 16 illustrates an example data structure of event-side statistical information corresponding to the division into three regions, the first region Ar1, the second region Ar2, and the third region Ar3, as shown in FIG. 5 . In this case, the event-side statistical information stores "3," the number of divisions of the region Ar, in the data region with data address "1." Furthermore, the data regions with data addresses "2" through "6" store information for identifying the position and size of the first region Ar1 and the number of noise events that occurred in the first region Ar1. Specifically, the data area with data address "2" stores the x-coordinate of the reference position of the first region Ar1 (e.g., the pixel position in the upper left corner of the first region Ar1), and the data area with data address "3" stores the y-coordinate. The data area with data address "4" stores information on the width (number of pixels) of the first region Ar1, and the data area with data address "5" stores information on the height (number of pixels) of the first region Ar1. Furthermore, the data area with data address "6" stores the number of noise events that occurred in the first region Ar1. Similarly, the data areas with data addresses "7" through "11" store similar information about the second region Ar2, and the data areas with data addresses "12" through "16" store similar information about the third region Ar3. Note that if similar statistical information is used for the imaging-side statistical information, information indicating the number of noise pixels for each region Ar can be stored instead of information indicating the number of noise events for each region Ar.

[0101] Furthermore, the statistical information regarding the presence or absence of noise influence is not limited to information indicating spatial statistics for each region Ar, as exemplified above, but may also be information indicating temporal statistics, for example, between previous and next frames. FIG. 17 shows an example data structure in which the statistical information includes a difference from the previous frame and an integrated value from the previous frame. Specifically, FIG. 17 shows an example in which, corresponding to a case where the image is divided into three regions, a first region Ar1 to a third region Ar3, information indicating the difference from the previous frame for the number of noise events (or the number of noise pixels) for each region Ar, and information indicating the integration value for the past two frames (the current frame and the immediately preceding frame) are stored. The figure shows an example of statistical information for two consecutive frames (frame #1, frame #2).

[0102] Although the above example illustrates storing statistical information in the last minimum unit data of a transmission frame, the manner in which statistical information is stored for a transmission frame is not limited to this. For example, as illustrated in FIG. 18, statistical information may be stored within each minimum unit data. FIG. 18 illustrates an example transmission frame for event data. In this case, each minimum unit data within the transmission frame is transmitted as minimum unit data Dc, which stores a predetermined number of event data and statistical information related to the predetermined number of event data in its payload area. In this case, the statistical information may be, for example, information storing the number of noise events for each area Ar within a period during which a predetermined number of event data were obtained, as illustrated in FIG. 19.

[0103] The statistical information stored in the minimum unit data Dc is not limited to statistical information within the period in which a predetermined number of event data are obtained as described above, but may also be statistical information relating to the number of noise events that occurred over a longer period, such as one transmission frame period.

[0104] Regarding the imaging-side statistical information, it is conceivable to generate information for each minimum unit data and store it for each minimum unit data as in the example of Fig. 18. For example, it is conceivable to generate information indicating the total number of noise pixels for each horizontal line as imaging-side statistical information and store the imaging-side statistical information for each minimum unit data.

[0105] 2. Second Embodiment Fig. 20 is a block diagram for explaining an example configuration of an imaging device 1A as a second embodiment. The imaging device 1A of the second embodiment has the same configuration as the imaging device 1 of the first embodiment, except that a sensor unit 3A is provided instead of the sensor unit 3. For this reason, Fig. 20 mainly shows an example configuration of the sensor unit 3A, and does not illustrate the event detection optical system 2e, the imaging optical system 2i, and the data processing unit 5. Note that in the following explanation, parts that are similar to parts that have already been explained will be assigned the same reference numerals and explanations thereof will be omitted.

[0106] The second embodiment relates to a measure for the case where an electronic circuit section 20 that is a third noise source other than the event sensor section 6 and the image sensor section 7 is present.

[0107] The sensor unit 3A differs from the sensor unit 3 in the first embodiment in that it is provided with an electronic circuit section 20 and a circuit operation presence / absence signal generation section 21, and in that the event-side statistical information generation section 16e and the imaging-side statistical information generation section 16i are replaced with an event-side statistical information generation section 16e and an imaging-side statistical information generation section 16i, respectively.

[0108] The electronic circuit section 20 is a circuit section mounted on the sensor unit 3A and serves as a third noise source other than the event sensor section 6 and the imaging sensor section 7. A specific example of the electronic circuit section 20 is a DSP (Digital Signal Processor) that performs digital image processing such as predetermined filtering on captured image data obtained by the imaging sensor section 7. Another example of the electronic circuit section 20 is a semiconductor memory used to temporarily store captured image data and event data, specifically a DRAM (Dynamic Random Access Memory) circuit section (hereinafter referred to as the "memory circuit section").

[0109] 21 is an explanatory diagram of an example of implementation of the electronic circuit section 20 in the sensor unit 3A. As shown in the figure, the electronic circuit section 20 may be formed on a semiconductor layer Ls that is separate from the semiconductor layer Ls (referred to as a first semiconductor layer Ls1 in the figure) on which the event sensor section 6 is formed and the semiconductor layer Ls (referred to as a second semiconductor layer Ls2 in the figure) on which the image sensor section 7 is formed. The figure shows an example in which the electronic circuit section 20 is formed on a third semiconductor layer Ls3 below the second semiconductor layer Ls2.

[0110] If the electronic circuit unit 20 is a DSP, the operation of the DSP may act as a noise source for at least one of the event sensor unit 6 and the image sensor unit 7. If the electronic circuit unit 20 is the memory circuit unit described above, the read and write operations of the DRAM performed by the memory circuit unit may act as a noise source for at least one of the event sensor unit 6 and the image sensor unit 7. In addition, in the DRAM, the memory circuit unit performs a refresh process to retain stored information in the memory cells, and this refresh process may also act as a noise source for at least one of the event sensor unit 6 and the image sensor unit 7.

[0111] 20, the sensor unit 3A of this embodiment is provided with a circuit operation signal generation unit 21, which generates a circuit operation signal indicating whether the electronic circuit unit 20 is operating. Specifically, if the electronic circuit unit 20 is a DSP, the circuit operation signal generation unit 21 generates a signal indicating whether the DSP is operating, for example, a signal with an H level indicating that the DSP is operating and an L level indicating that the DSP is not operating, as the circuit operation signal. Alternatively, if the electronic circuit unit 20 is a memory circuit, the circuit operation signal generation unit 21 generates a signal indicating whether at least one of a DRAM read process, write process, and refresh process is being performed, for example, a signal with an H level indicating that one of the processes is being performed and an L level indicating that the process is not being performed, as the circuit operation signal.

[0112] The circuit operation presence / absence signal generated by the circuit operation presence / absence signal generator 21 is input to the event-side statistical information generator 16eA and the imaging-side statistical information generator 16iA. The event-side statistical information generator 16eA determines whether at least one of the readout operation of the light-receiving signal in the imaging sensor unit 7 and the processing of the electronic circuit unit 20 was being executed when the event signal was read out, based on the circuit operation presence / absence signal input from the circuit operation presence / absence signal generator 21 and the imaging-side readout presence / absence signal input from the imaging-side readout presence / absence signal generator 17i, and generates event-side statistical information based on the determination result. This makes it possible to appropriately generate influence presence / absence data indicating the presence or absence of noise influence in cases where both the readout operation of the light-receiving signal in the imaging sensor unit 7 and the processing of the electronic circuit unit 20 can act as noise sources for the event sensor unit 6.

[0113] The imaging-side statistical information generator 16iA determines whether or not at least one of the event signal readout operation in the event sensor unit 6 and the processing in the electronic circuit unit 20 was being executed when the light-receiving signal was read out, based on the circuit operation presence / absence signal input from the circuit operation presence / absence signal generator 21 and the event-side readout presence / absence signal input from the event-side readout presence / absence signal generator 17e, and generates imaging-side statistical information based on the determination result. This makes it possible to appropriately generate influence presence / absence data indicating the presence or absence of noise influence in cases where both the event signal readout operation in the event sensor unit 6 and the processing in the electronic circuit unit 20 can act as noise sources for the imaging sensor unit 7.

[0114] In the above, an example was given in which statistical information (data on the presence or absence of noise influence) reflecting whether or not the electronic circuit unit 20 is operating is generated for both the event side and the imaging side. However, if the operation of the electronic circuit unit 20 acts as a noise source only on the event sensor unit 6 side, statistical information reflecting whether or not the electronic circuit unit 20 is operating can be generated only on the event side, and if the operation of the electronic circuit unit 20 acts as a noise source only on the imaging sensor unit 7 side, statistical information reflecting whether or not the electronic circuit unit 20 is operating can be generated only on the imaging side.

[0115] In the above, a DSP and a memory circuit unit are given as examples of the electronic circuit unit 20, but the second embodiment can be suitably applied when an electronic circuit unit that may emit electromagnetic noise when processing at least one of captured image data and event data is implemented in the sensor unit 3A.

[0116] 3. Modifications The embodiment is not limited to the specific example described above, and various modification configurations are possible. For example, the above describes an example of a method for determining whether an event signal is being read out (whether or not there is an influence of noise) for each imaging pixel Pi, in which the imaging-side statistical information generation unit 16i monitors whether or not a light-receiving signal is being read out from a signal line for each pixel column of the pixel array unit 10i. However, the method for determining whether or not an event signal is being read out for each imaging pixel Pi is not limited to this. For example, another possible method is to use a clock signal used to control the readout of the light-receiving signal to recognize the readout timing for each imaging pixel Pi based on the elapsed time from the readout start timing, and determine whether or not an event signal is being read out based on the event-side readout presence / absence signal at the readout timing for each imaging pixel Pi.

[0117] Furthermore, although the above example shows the determination of the influence of noise on a pixel-by-pixel basis, it is also possible to determine the influence of noise on the light-receiving signal of the image sensor unit 7 for each of several pixels. For example, it is possible to determine the influence of noise for each of several image pixels Pi, which is less than the number of image pixels Pi for one horizontal line. By making the determination for each of several image pixels Pi, which is less than the number of image pixels Pi for one horizontal line, it is possible to appropriately obtain statistical information for each region Ar, for example, when the region Ar is divided horizontally as shown in FIG. 5 .

[0118] Furthermore, while the above example illustrates a case in which the event sensor unit 6 and the imaging sensor unit 7 are stacked, the event sensor unit 6 and the imaging sensor unit 7 may be formed on the same semiconductor layer Ls, as illustrated in Fig. 22. Fig. 22 illustrates an example in which the event sensor unit 6 and the imaging sensor unit 7 are arranged separately on the same semiconductor layer Ls, but as an example of a mounting form of the event sensor unit 6 and the imaging sensor unit 7 on the same semiconductor layer Ls, as illustrated in Fig. 23, for example, a mounting form in which the imaging pixels Pi and the event detection pixels Pe are mixed within the same pixel array unit is also considered.

[0119] In addition, in the second embodiment, an example was given in which the electronic circuit unit 20 was formed on a semiconductor layer Ls different from that of the event sensor unit 6 and the image sensor unit 7 (Figure 21), but it is also possible that the electronic circuit unit 20 was formed on the same semiconductor layer Ls as the event sensor unit 6, or the same semiconductor layer Ls as the image sensor unit 7, or the semiconductor layer Ls on which the event sensor unit 6 and the image sensor unit 7 were formed.

[0120] Furthermore, in the above example, the event data and the captured image data are output to a subsequent stage via separate transmission systems, but it is also possible to output the event data and the captured image data via a single transmission system.

[0121] 24 is a block diagram illustrating an example of the configuration of an imaging device 1B as a modified example corresponding to the case where event data and captured image data are output through a single transmission system. Although not shown, the imaging device 1B also includes an event detection optical system 2e, an imaging optical system 2i, and a data processing unit 5, similar to the imaging device 1.

[0122] As shown in the figure, imaging device 1B has a sensor unit 3B instead of sensor unit 3, and sensor unit 3B differs from sensor unit 3 in that it has a transmitter 8A instead of transmitter 8 and a receiver 4B instead of receiver 4. Transmitter 8A stores event data obtained by event-side reader 15e, captured image data obtained by imaging-side reader 15i, event-side statistical information generated by event-side statistical information generator 16e, and imaging-side statistical information generated by imaging-side statistical information generator 16i in the same transmission frame and transmits the stored data.

[0123] 25 shows an example of the data structure of the transmission frame in this case. As shown in the figure, the transmission frame in this case stores minimum unit data Da including line data of the captured image data and minimum unit data Da including a predetermined number of event data, followed by minimum unit data Dd including existing EBD (embedded) data, minimum unit data De including event-side statistical information, and minimum unit data Df including imaging-side statistical information at the end of the frame.

[0124] In this case, size information of the subsequent event-side statistical information and imaging-side statistical information is stored in a predetermined data address in the existing EBD data in the minimum unit data Dd, thereby enabling a processing unit at a later stage of the sensor unit 3B to separately acquire the event-side statistical information and the imaging-side statistical information.

[0125] 24, the receiving unit 4B receives the above-mentioned transmission frame data transmitted by the transmitting unit 8 via a single transmission system. The receiving unit 4B can separately acquire event data and captured image data based on, for example, header information in the minimum unit data Da. Furthermore, the receiving unit 4B can separately acquire event-side statistical information and imaging-side statistical information based on size information in the above-mentioned existing EBD data.

[0126] Note that the above example illustrates a method of storing size information for the event-side statistical information and the imaging-side statistical information within existing EBD data in order to enable separate acquisition of the event-side statistical information and the imaging-side statistical information. However, other methods may also be used to enable separate acquisition of the event-side statistical information and the imaging-side statistical information, such as transmitting the imaging-side statistical information and the event-side statistical information in different Data Types, or assigning different virtual CHs to the imaging-side statistical information and the event-side statistical information.

[0127] Furthermore, it is also conceivable that the influence data for the event data is generated not only as data indicating whether or not the light reception signal has been read out by the imaging sensor unit 7, but also as data that takes into consideration saturation of event detection by the event sensor unit 6. For example, as shown in Fig. 26, a saturation detection circuit 25 is provided that determines whether or not event detection by the event sensor unit 6 is saturated. The saturation detection circuit 25 determines whether or not the above-mentioned saturation state has occurred by, for example, determining whether or not the number of events detected per unit time is equal to or greater than a predetermined number, and outputs a saturation presence / absence signal indicating the determination result, for example, a signal that is H level when saturated and L level when non-saturated, to the event-side statistical information generator 16eC.

[0128] The event-side statistical information generator 16eC determines whether at least one of the following has occurred: a readout of the light-receiving signal from the imaging sensor unit 7 or saturation of the event sensor unit 6, based on the imaging-side readout presence / absence signal and the saturation presence / absence signal, each time an event signal is read out from the pixel array unit 10e, and generates event-side statistical information based on the determination result. Events detected when event detection is saturated in the event sensor unit 6 may be erroneously detected. Therefore, the event-side statistical information is generated so that such events are treated as noise events. This allows for appropriate processing to be performed as a later-stage countermeasure, taking into account the presence of erroneously detected events due to saturation of event detection.

[0129] Although MIPI has been cited above as an example of a transmission data format, application of the present technology is not limited to MIPI. Furthermore, when implementing the present technology using a specified transmission data format, statistical information (impact data) may be stored in a vendor extension area, a reserved area, or an unused area provided in the specified data format.

[0130] In addition, in the above example, a data processing unit that performs processing based on influence data such as event-side statistical information and imaging-side statistical information is provided in an imaging device equipped with a sensor unit, but it is also possible to consider a configuration in which the data processing unit is provided in an external device to the imaging device.

[0131] 4. Summary of the embodiment As described above, the imaging device (1, 1A, 1B) of the embodiment includes an event sensor unit (6) in which event detection pixels (Pe) that detect a change in the amount of received light as an event are two-dimensionally arranged and generate an event signal indicative of the event detection result, an imaging sensor unit (7) in which imaging pixels (Pi) that have photoelectric conversion elements and generate a light reception signal indicative of the amount of received light are two-dimensionally arranged to obtain an image, and an event-related signal processor (10) that determines whether a readout operation of the imaging sensor unit was being executed when the event was detected by the event sensor unit and indicates whether noise has affected the event signal. The sensor unit (3, 3A, or 3B) includes an event-related influence data generator (event-side statistical information generator 16e or 16eA) that generates event-related influence data, an imaging-related influence data generator (imaging-side statistical information generator 16i or 16iA) that generates imaging-related influence data indicating whether or not a noise source is affecting a light-receiving signal based on a result of determining whether an event signal read operation was being performed when the light-receiving signal of the imaging sensor was read, and a transmitter (8 or 8B) that transmits the event-related influence data and the imaging-related influence data. According to the above configuration, for a sensor unit as a hybrid sensor incorporating an event sensor unit and an imaging sensor unit, influence data indicating whether or not a noise source is affecting the event data obtained by the event sensor unit and the image data obtained by the imaging sensor unit can be output to a downstream processing unit. This allows the downstream processing unit to perform appropriate noise countermeasure processing.

[0132] Furthermore, in the imaging device according to the embodiment, the imaging influence data generation unit determines for each imaging pixel whether an event signal read operation was being executed when the light reception signal was read. This makes it possible to obtain information indicating the presence or absence of noise influence due to the readout of the event signal at the pixel-by-pixel granularity for the image data captured by the imaging sensor unit. This improves the accuracy of the imaging influence data, thereby improving the accuracy of the countermeasure processing performed by the processing unit at a subsequent stage.

[0133] Furthermore, in the imaging device according to the embodiment, the imaging influence data generation unit generates statistical information relating to the presence or absence of noise influence as imaging influence data. This makes it possible to output statistical information relating to the presence or absence of noise influence to a downstream processing unit. For example, it is possible to output information indicating statistics based on a certain space, such as an image plane, or information indicating statistics based on a certain period, such as a period of multiple consecutive frames, regarding the presence or absence of noise influence. Therefore, it is possible to output appropriate imaging influence data according to the type of countermeasure processing performed by the downstream processing unit.

[0134] Furthermore, in the imaging device according to the embodiment, the imaging influence data generation unit generates, as statistical information, information indicating the degree of noise influence for each region formed by dividing the sensing surface of the imaging sensor unit. This makes it possible to output imaging influence data indicating the degree of noise influence for each region in response to a case in which a subsequent processing unit processes each region of the captured image. Therefore, this is suitable for a case in which a subsequent processing unit processes each region.

[0135] Furthermore, in the imaging device according to the embodiment, the imaging influence data generation unit generates statistical information indicating the degree of noise influence for each horizontal line of the imaging sensor unit (see FIG. 15 ). This makes it possible to output imaging influence data indicating the degree of noise influence for each horizontal line in response to a case where a subsequent processing unit processes each horizontal line of the captured image. This is therefore suitable for a case where a subsequent processing unit processes each horizontal line.

[0136] Furthermore, in the imaging device (1B) according to the embodiment, the transmitting unit (8B) stores the captured image data obtained based on the light receiving signal, the event data obtained based on the event signal, the event influence presence / absence data, and the imaging influence presence / absence data in the same transmission frame and transmits them (see FIGS. 24 and 25). This makes it possible to transmit the captured image data and event data for each unit period and the influence presence / absence data therefor via a single transmission system, thereby reducing the number of communication cables required when data transmission to a downstream processing unit is performed via a wired connection.

[0137] Furthermore, in the imaging device (1A) according to an embodiment, the sensor unit (3A) has an electronic circuit section (20) that processes at least one of the captured image data obtained based on the light-receiving signal and the event data obtained based on the event signal, and the event-influence-presence data generator (event-side statistical information generator 16eA) generates event-influence-presence data based on the result of determining whether at least one of the readout operation of the imaging sensor section and the processing of the electronic circuit section was being executed when the event sensor section detected an event. This makes it possible to generate data that appropriately indicates the presence or absence of noise influence as event-influence-presence data in response to cases where noise components are superimposed on the event data due to the operation of the event sensor section and the electronic circuit section implemented in the sensor unit together with the imaging sensor section.

[0138] Furthermore, in the imaging device (1A) as an embodiment, the sensor unit has an electronic circuit section that performs processing on at least one of captured image data obtained based on a light receiving signal and event data obtained based on an event signal, and the imaging influence presence / absence data generation section (imaging-side statistical information generation section 16iA) generates imaging influence presence / absence data based on the result of determining whether at least one of the event signal read operation and the processing of the electronic circuit section was executed when the light receiving signal of the imaging sensor section was read. This makes it possible to generate data that appropriately indicates the presence or absence of noise influence as imaging influence presence / absence data in response to cases where noise components are superimposed on the captured image data due to the operation of the electronic circuit section implemented in the sensor unit together with the event sensor section and the imaging sensor section.

[0139] Furthermore, in the imaging device according to the embodiment, the electronic circuit unit is a circuit unit that performs image processing on captured image data obtained based on a light reception signal. If the electronic circuit unit mounted in the sensor unit together with the event sensor unit and the imaging sensor unit is an electronic circuit unit that performs image processing, there is a risk that noise components will be superimposed on the event data and captured image data due to the processing operation of the electronic circuit unit. Therefore, when generating the influence data, it is preferable to consider not only whether a readout operation of the other sensor unit has been performed, but also whether processing by the electronic circuit unit has been performed.

[0140] Furthermore, in the imaging device according to the embodiment, the electronic circuit is a semiconductor memory circuit. If the electronic circuit mounted in the sensor unit together with the event sensor and the imaging sensor is a semiconductor memory circuit, there is a risk that noise components may be superimposed on the event data and the captured image data due to operations such as reading and writing from the semiconductor memory by the circuit. Therefore, when generating the influence data, it is preferable to consider not only whether a read operation from the other sensor has been performed, but also whether processing by the electronic circuit has been executed.

[0141] In addition, in the imaging device according to the embodiment, the sensor unit has multiple semiconductor layers, and the event sensor unit and the image sensor unit are formed in different semiconductor layers. When the event sensor unit and the image sensor unit are stacked, their respective signal readout operations tend to be a source of noise for each other. Therefore, it is preferable to generate and transmit influence data for both the event data and the captured image data based on whether the other sensor readouts a signal.

[0142] Furthermore, the imaging device according to the embodiment includes a data processing unit (5) that receives, from the sensor unit, at least one of event data obtained based on the event signal and captured image data obtained based on the light-receiving signal, and processes the input data based on the influence data corresponding to the input data, out of the event-related influence data and the image-related influence data. This allows noise countermeasure processing to be performed downstream of the sensor unit on at least one of the event data and captured image data obtained by the sensor unit as a mixed sensor, using the corresponding influence data.

[0143] Furthermore, in the imaging device according to the embodiment, the data processing unit adjusts the likelihood calculated as an inference result in AI processing performed on input data based on the influence data. This makes it possible to adjust the likelihood of an inference process, such as image recognition processing performed on captured image data or event data, according to the degree of noise superposition in the input data, for example, by adjusting the likelihood to lower the likelihood of recognition processing performed on a noisy area. This makes it possible to obtain appropriate inference results that take into account the reliability of the input data, thereby improving the accuracy of the AI ​​processing.

[0144] Furthermore, in the imaging device according to the embodiment, the data processing unit adjusts the strength of the noise reduction process performed on the input data based on the influence data. This makes it possible to perform appropriate noise reduction processing according to the degree of noise superposition in the input data, such as increasing the noise reduction effect when the input data is noisy, when the noise reduction process is performed on captured image data or event data. Therefore, appropriate noise reduction processing can be achieved for the captured image data and event data obtained by the sensor unit.

[0145] Furthermore, in the imaging device according to the embodiment, the data processing unit performs frame thinning processing on input data obtained in units of a predetermined number of frames based on the influence data. This makes it possible to perform appropriate frame thinning processing on the captured image data and event data input by the sensor unit according to the degree of noise superposition in the input data, such as thinning frames so that data with a lot of noise is not used in later processing. Therefore, appropriate noise reduction processing can be achieved for the event data and captured image data obtained by the sensor unit.

[0146] A data processing method according to an embodiment is a data processing method for a sensor unit including an event sensor unit having a two-dimensional array of event detection pixels that detect changes in the amount of received light as an event and generate an event signal indicative of the event detection result, and an imaging sensor unit having a two-dimensional array of imaging pixels that have photoelectric conversion elements and generate light reception signals indicative of the amount of received light, to obtain an image, the data processing method comprising: generating event-related influence presence / absence data indicating whether or not the event signal is affected by noise based on a result of determining whether or not a readout operation of the imaging sensor unit was being performed when the event was detected by the event sensor unit; and generating imaging-related influence presence / absence data indicating whether or not the light reception signal is affected by noise based on a result of determining whether or not a readout operation of the event signal was being performed when the light reception signal was read out by the imaging sensor unit, and transmitting the generated event-related influence presence / absence data and imaging-related influence presence / absence data. This data processing method can also achieve the same functions and effects as the imaging device according to the above-described embodiment.

[0147] an image sensor unit having a two-dimensional array of image pickup pixels each having a photoelectric conversion element and each generating a light reception signal indicating the amount of received light; an image pickup unit having a two-dimensional array of image pickup pixels each having a photoelectric conversion element and each generating a light reception signal indicating the amount of received light to obtain an image pickup image; an image pickup unit having an image pickup effect presence / absence data generation unit which generates image pickup effect presence / absence data indicating whether or not noise is affecting the event signal based on a result of determining whether or not a read operation of the image pickup sensor unit was being performed when the event was detected by the event sensor unit; an image pickup effect presence / absence data generation unit which generates image pickup effect presence / absence data indicating whether or not noise is affecting the light reception signal based on a result of determining whether or not a read operation of the event signal was being performed when the light reception signal was read out by the image pickup sensor unit; and a data processing unit (same as above) which receives the image pickup effect presence / absence data and the image pickup effect presence / absence data and performs processing based on the effect presence / absence data corresponding to the input data as processing of the input data. As a result, at least one of the event data and captured image data obtained by the sensor unit as a mixed sensor is processed based on the corresponding influence data in a stage downstream of the sensor unit. Therefore, when the event data and captured image data output by the sensor unit equipped with the event sensor and the image sensor contain noise components, it is possible to enable the downstream processing unit to perform appropriate noise countermeasure processing.

[0148] an image sensor unit having a two-dimensional array of image pickup pixels, each having a photoelectric conversion element, for generating a light reception signal indicating the amount of received light; an event sensor unit having a two-dimensional array of image pickup pixels, each having a photoelectric conversion element, for generating a light reception signal indicating the amount of received light, to obtain an image pickup image; an event influence presence / absence data generation unit having an image pickup influence presence / absence data generation unit generating an image pickup influence presence / absence data generation unit generating an image pickup influence presence / absence data generation unit generating an image pickup influence presence / absence data generation unit generating an image pickup influence presence / absence data generation unit This information processing method can also provide the same functions and effects as the information processing device of the above embodiment.

[0149] Here, as an embodiment, it is possible to consider a program that causes, for example, a CPU, a DSP, or a device including these, to execute the processing of the above-described data processing unit 5, or a recording medium on which the program is recorded. That is, the recording medium of the embodiment is a recording medium on which a computer-readable program is recorded, and includes an event sensor unit in which event detection pixels that detect a change in the amount of received light as an event are two-dimensionally arranged and generate an event signal indicative of the event detection result, an imaging sensor unit in which imaging pixels that have photoelectric conversion elements and generate a light reception signal indicative of the amount of received light are two-dimensionally arranged to obtain an image, an event influence presence / absence data generation unit that generates event influence presence / absence data indicative of the presence or absence of noise on the event signal based on the result of determining whether or not a read operation of the imaging sensor unit was executed when the event was detected by the event sensor unit, and an event signal when the light reception signal of the imaging sensor unit is read out. and a transmitter that transmits the event influence presence / absence data and the imaging influence presence / absence data, and receives at least one of event data obtained based on an event signal and captured image data obtained based on the light receiving signal, the event data being transmitted by a sensor unit having: an imaging influence presence / absence data generation unit that generates imaging influence presence / absence data indicating the presence or absence of noise influence on the light receiving signal based on a result of determining whether a read operation of the event influence presence / absence data has been executed; and a transmitter that transmits the event influence presence / absence data and the imaging influence presence / absence data, and the recording medium having recorded thereon a program that causes a computer device to realize a function of performing processing based on the influence presence / absence data corresponding to the input data as processing of the input data as the input data. By using such a recording medium, the functions of the data processing unit as the above-described embodiment can be realized by software processing in an apparatus that serves as a computer device.

[0150] The above-mentioned recording medium can be realized as an HDD (Hard Disk Drive) built into a device such as a computer, or as a ROM in a microcomputer having a CPU. Alternatively, it can be realized as a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, or a memory card. Such removable recording medium can be provided as a so-called package software.

[0151] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0152] Furthermore, the above-described examples may be combined in any manner, and even when various combinations are used, the above-described various operational effects can be obtained.

[0153] 5. The Present Technology The present technology may also be configured as follows: (1) An imaging device including a sensor unit having: an event sensor unit in which event detection pixels that detect a change in the amount of received light as an event are two-dimensionally arranged and that generates an event signal indicative of the detection result of the event; an imaging sensor unit in which imaging pixels that have photoelectric conversion elements and generate light reception signals indicative of the amount of received light are two-dimensionally arranged to obtain an image; an event influence presence / absence data generation unit that generates event influence presence / absence data indicative of the presence or absence of noise on the event signal based on a result of determining whether a readout operation of the imaging sensor unit was being executed when the event was detected by the event sensor unit; an imaging influence presence / absence data generation unit that generates imaging influence presence / absence data indicative of the presence or absence of noise on the light reception signal based on a result of determining whether a readout operation of the event signal was being executed when the light reception signal was being read out by the imaging sensor unit; and a transmitter that transmits the event influence presence / absence data and the imaging influence presence / absence data. (2) The imaging device according to (1), wherein the imaging influence presence / absence data generation unit determines for each imaging pixel whether or not the readout operation of the event signal was being executed when the light reception signal was being readout. (3) The imaging device according to (1) or (2), wherein the imaging influence presence / absence data generation unit generates, as the imaging influence presence / absence data, statistical information relating to the presence or absence of the influence of the noise. (4) The imaging device according to (3), wherein the imaging influence presence / absence data generation unit generates, as the statistical information, information indicating the degree of influence of the noise for each region formed by dividing the sensing surface of the imaging sensor unit. (5) The imaging device according to (3), wherein the imaging influence presence / absence data generation unit generates, as the statistical information, information indicating the degree of influence of the noise for each horizontal line of the imaging sensor unit. (6) The imaging device according to any one of (1) to (5), wherein the transmitting unit stores the captured image data obtained based on the light receiving signal, the event data obtained based on the event signal, the event influence presence / absence data, and the imaging influence presence / absence data in the same transmission frame and transmits the same.(7) The imaging device according to any of (1) to (6), wherein the sensor unit has an electronic circuit section that performs processing on at least one of captured image data obtained based on the light reception signal and event data obtained based on the event signal, and the event influence presence / absence data generation section generates the event influence presence / absence data based on a result of determining whether at least one of a readout operation of the imaging sensor section and processing by the electronic circuit section was executed when the event sensor section detected an event. (8) The imaging device according to any of (1) to (7), wherein the sensor unit has an electronic circuit section that performs processing on at least one of captured image data obtained based on the light reception signal and event data obtained based on the event signal, and the imaging influence presence / absence data generation section generates the imaging influence presence / absence data based on a result of determining whether at least one of a readout operation of the event signal and processing by the electronic circuit section was executed when the light reception signal of the imaging sensor section was read. (9) The imaging device according to (7) or (8), wherein the electronic circuit section is a circuit section that performs image processing on captured image data obtained based on the light reception signal. (10) The imaging device according to (7) or (8), wherein the electronic circuit unit is a circuit unit of a semiconductor memory. (11) The imaging device according to any of (1) to (10), wherein the sensor unit has a plurality of semiconductor layers, and the event sensor unit and the imaging sensor unit are formed in different semiconductor layers. (12) The imaging device according to any of (1) to (11), wherein at least one of event data obtained based on the event signal and captured image data obtained based on the light receiving signal is input from the sensor unit, and the input data is processed based on the event influence presence / absence data and the influence presence / absence data corresponding to the input data, as processing for the input data. (13) The imaging device according to (12), wherein the data processing unit is configured to adjust a likelihood calculated as an inference result in AI processing performed on the input data, based on the influence presence / absence data.(14) The imaging device according to (12) or (13), wherein the data processing unit performs a process of adjusting a strength of a noise reduction process performed on the input data based on the influence data. (15) The imaging device according to any one of (12) to (14), wherein the data processing unit performs a process of thinning out frames of the input data obtained in predetermined frame units based on the influence data. (16) A data processing method for a sensor unit having an event sensor section in which event detection pixels that detect a change in the amount of received light as an event are arranged in a two-dimensional array and generate an event signal indicative of the detection result of the event, and an imaging sensor section in which imaging pixels that have photoelectric conversion elements and generate light reception signals indicative of the amount of received light are arranged in a two-dimensional array to obtain an image, the data processing method comprising: generating event-related influence presence / absence data indicative of the presence or absence of noise on the event signal based on a result of determining whether a readout operation of the imaging sensor section was being executed when the event was detected by the event sensor section; and generating imaging-related influence presence / absence data indicative of the presence or absence of noise on the light reception signal based on a result of determining whether a readout operation of the event signal was being executed when the light reception signal was being read out by the imaging sensor section; and transmitting the generated event-related influence presence / absence data and imaging-related influence presence / absence data.(17) An event sensor unit in which event detection pixels that detect a change in the amount of received light as an event are two-dimensionally arranged and generate an event signal indicative of a detection result of the event; an imaging sensor unit in which imaging pixels that have photoelectric conversion elements and generate a light reception signal indicative of the amount of received light are two-dimensionally arranged to obtain an image; an event influence presence / absence data generation unit that generates event influence presence / absence data indicative of the presence or absence of noise influence on the event signal based on a result of determining whether a readout operation of the imaging sensor unit was being executed when the event was detected by the event sensor unit; an imaging influence presence / absence data generating unit that generates imaging influence presence / absence data indicating whether or not noise has had an effect on the light reception signal based on a result of determining whether or not noise has had an effect on the light reception signal, and a transmitting unit that transmits the event influence presence / absence data and the imaging influence presence / absence data, and a data processing unit that inputs at least one of event data obtained based on the event signal and captured image data obtained based on the light reception signal, which are transmitted by a sensor unit having the imaging influence presence / absence data generating unit that generates imaging influence presence / absence data indicating whether or not noise has had an effect on the light reception signal based on a result of determining whether or not noise has had an effect on the light reception signal, and(18) An information processing device, an event sensor unit having a two-dimensional array of event detection pixels that detect a change in the amount of received light as an event and generate an event signal indicative of the detection result of the event; an imaging sensor unit having a two-dimensional array of imaging pixels that have photoelectric conversion elements and generate light reception signals indicative of the amount of received light to obtain an image; an event influence presence / absence data generation unit that generates event-related influence data indicative of the presence or absence of noise on the event signal based on a result of determining whether a readout operation of the imaging sensor unit was being executed when the event was detected by the event sensor unit; an imaging influence presence / absence data generation unit that generates imaging influence data indicative of the presence or absence of noise on the light reception signal based on a result of determining whether a readout operation of the event signal was being executed when the light reception signal was being read out by the imaging sensor unit; and a transmitter that transmits the event influence presence / absence data and the imaging influence presence / absence data, Information processing methods.(19) A recording medium having a computer-readable program recorded thereon, comprising: an event sensor unit in which event detection pixels that detect a change in the amount of received light as an event are two-dimensionally arranged and generate an event signal indicating a detection result of the event; an imaging sensor unit in which imaging pixels that have photoelectric conversion elements and generate a light reception signal indicating the amount of received light are two-dimensionally arranged to obtain an image; an event influence presence / absence data generation unit that generates event influence presence / absence data indicating the presence or absence of an influence of noise on the event signal based on a result of determining whether a read operation of the imaging sensor unit was being executed when the event was detected by the event sensor unit; a sensor unit having an imaging influence presence / absence data generation unit that generates imaging influence presence / absence data indicating whether or not noise is affecting the received light signal based on the result of noise detection, and a transmission unit that transmits the event influence presence / absence data and the imaging influence presence / absence data, and inputs at least one of event data obtained based on the event signal and captured image data obtained based on the received light signal, and performs processing based on the influence presence / absence data corresponding to the input data out of the event influence presence / absence data and the imaging influence presence / absence data as processing of the input data as the input data.

[0154] 1, 1A, 1B Imaging device 2e Event detection optical system 2i Imaging optical system 3, 3A, 3B Sensor unit 5 Data processing unit 5e Event data processing unit 5i Imaging data processing unit 6 Event sensor unit 7 Imaging sensor unit 8, 8B Transmission unit 8e Event data transmission unit 8i Imaging data transmission unit Ls Semiconductor layer Ls1 First semiconductor layer Ls2 Second semiconductor layer Pe Event detection pixel Pi Imaging pixel 10e, 10i Pixel array unit 15e Event side readout unit 15i Imaging side readout unit 16e, 16eA, 16eC Event side statistical information generation unit 16i, 16iA Imaging side statistical information generation unit 17e Event side readout presence / absence signal generation unit 17i Imaging side readout presence / absence signal generation unit Dst Start data Den End data Da, Db, Dc, Dd, De, Df Minimum unit data Ar Area Ar1 to Ar9 First area to ninth area 20 Electronic circuit section 21 Circuit operation presence / absence signal generation section

Claims

1. An imaging device equipped with a sensor unit having: an event sensor section in which event detection pixels that detect a change in the amount of received light as an event are arranged in a two-dimensional array, and which generates an event signal indicative of the detection result of the event; an imaging sensor section in which imaging pixels having photoelectric conversion elements that generate light receiving signals indicative of the amount of received light are arranged in a two-dimensional array to obtain an image; an event influence presence / absence data generation section that generates event influence presence / absence data indicative of the presence or absence of noise on the event signal, based on a result of a determination as to whether a readout operation of the imaging sensor section was being executed at the time of event detection by the event sensor section; an imaging influence presence / absence data generation section that generates imaging influence presence / absence data indicative of the presence or absence of noise on the light receiving signal, based on a result of a determination as to whether a readout operation of the event signal was being executed at the time of readout of the light receiving signal of the imaging sensor section; and a transmission section that transmits the event influence presence / absence data and the imaging influence presence / absence data.

2. The imaging device according to claim 1, wherein the imaging effect data generation unit determines for each imaging pixel whether or not the read operation of the event signal was being executed when the light reception signal was being read out.

3. The imaging device according to claim 1, wherein the imaging influence presence / absence data generating section generates statistical information relating to the presence or absence of the influence of the noise as the imaging influence presence / absence data.

4. The imaging device according to claim 3, wherein the imaging influence data generation unit generates, as the statistical information, information indicating the degree of influence of the noise for each area formed by dividing the sensing surface of the imaging sensor unit.

5. The imaging device according to claim 3, wherein the imaging influence data generating section generates, as the statistical information, information indicating the degree of influence of the noise for each horizontal line of the imaging sensor section.

6. The imaging device according to claim 1, wherein the transmission unit stores and transmits the captured image data obtained based on the light receiving signal, the event data obtained based on the event signal, the event effect presence / absence data, and the imaging effect presence / absence data in the same transmission frame.

7. The imaging device according to claim 1, wherein the sensor unit has an electronic circuit section which performs processing on at least one of the captured image data obtained based on the light receiving signal and the event data obtained based on the event signal, and the event influence presence / absence data generation section generates the event influence presence / absence data based on a result of determining whether or not at least one of the readout operation of the imaging sensor section and the processing of the electronic circuit section was being performed when the event sensor section detected an event.

8. The imaging device according to claim 1, wherein the sensor unit has an electronic circuit section which performs processing on at least one of the captured image data obtained based on the light receiving signal and the event data obtained based on the event signal, and the imaging effect presence / absence data generation section generates the imaging effect presence / absence data based on a result of determining whether or not at least one of the read operation of the event signal and the processing of the electronic circuit section was being executed when the light receiving signal of the imaging sensor section was being read out.

9. The imaging device according to claim 7 or 8, wherein the electronic circuit section is a circuit section for performing image processing on captured image data obtained based on the light receiving signal.

10. An imaging device according to claim 7 or 8, wherein the electronic circuit section is a semiconductor memory circuit section.

11. The imaging device according to claim 1, wherein the sensor unit has a plurality of semiconductor layers, and the event sensor section and the imaging sensor section are formed in different semiconductor layers.

12. The imaging device according to claim 1, further comprising a data processing unit which inputs at least one of event data obtained based on the event signal and captured image data obtained based on the light receiving signal from the sensor unit, and processes the input data as the input data based on the event effect presence / absence data and the imaging effect presence / absence data corresponding to the input data.

13. The imaging device according to claim 12, wherein the data processing unit performs a process of adjusting the likelihood calculated as an inference result in AI processing performed on the input data based on the influence data.

14. The imaging device according to claim 12, wherein the data processing unit performs a process of adjusting the strength of noise reduction processing performed on the input data based on the influence data.

15. The imaging device according to claim 12, wherein the data processing unit performs frame thinning processing on the input data obtained in predetermined frame units based on the influence data.

16. A data processing method for a sensor unit having an event sensor section in which event detection pixels that detect a change in the amount of received light as an event are arranged in a two-dimensional array and generate an event signal indicative of the detection result of the event, and an imaging sensor section in which imaging pixels having photoelectric conversion elements and generating light receiving signals indicative of the amount of received light are arranged in a two-dimensional array to obtain an image, the data processing method comprising the steps of: generating event influence presence / absence data indicating the presence or absence of noise influence on the event signal based on a result of determining whether or not a read operation of the imaging sensor section was being executed at the time of event detection by the event sensor section; generating imaging influence presence / absence data indicating the presence or absence of noise influence on the light receiving signal based on a result of determining whether or not a read operation of the event signal was being executed at the time of reading out the light receiving signal of the imaging sensor section; and transmitting the generated event influence presence / absence data and imaging influence presence / absence data.

17. An event sensor section in which event detection pixels for detecting a change in the amount of received light as an event are arranged two-dimensionally and which generates an event signal indicative of a detection result of the event; an image sensor section in which image pickup pixels having photoelectric conversion elements and generating a light reception signal indicative of an amount of received light are arranged two-dimensionally to obtain an image; an event influence presence / absence data generation section for generating event influence presence / absence data indicative of the presence or absence of an influence of noise on the event signal based on a result of judging whether a read operation of the image sensor section was being executed at the time of event detection by the event sensor section; an imaging effect presence / absence data generating unit that generates imaging effect presence / absence data indicating whether or not noise has been affected by the light reception signal based on a result of determining whether or not noise has been affected by the light reception signal, and a transmitting unit that transmits the event effect presence / absence data and the imaging effect presence / absence data, said information processing device comprising: a data processing unit that inputs at least one of event data obtained based on the event signal and captured image data obtained based on the light reception signal, transmitted by a sensor unit having the imaging effect presence / absence data generating unit that generates imaging effect presence / absence data indicating whether or not noise has been affected by the light reception signal based on a result of determining whether or not noise has been affected by the light reception signal, 18. An information processing device, an event sensor section in which event detection pixels that detect a change in the amount of received light as an event are two-dimensionally arranged and that generates an event signal indicative of a detection result of the event; an imaging sensor section in which imaging pixels having photoelectric conversion elements and that generate light reception signals indicative of the amount of received light are two-dimensionally arranged to obtain an image; an event influence presence / absence data generation section that generates event influence presence / absence data indicative of the presence or absence of influence of noise on the event signal based on a result of determining whether a readout operation of the imaging sensor section was being executed when the event was detected by the event sensor section; an imaging influence presence / absence data generation section that generates imaging influence presence / absence data indicative of the presence or absence of influence of noise on the light reception signal based on a result of determining whether a readout operation of the event signal was being executed when the light reception signal was read out by the imaging sensor section; and a transmission section that transmits the event influence presence / absence data and the imaging influence presence / absence data Information processing methods.

19. A recording medium having a computer readable program recorded thereon, comprising: an event sensor section in which event detection pixels for detecting a change in the amount of received light as an event are arranged two-dimensionally and which generates an event signal indicating the detection result of the event; an image sensor section in which image pickup pixels having photoelectric conversion elements and generating a light reception signal indicating the amount of received light are arranged two-dimensionally to obtain an image; an event influence presence / absence data generation section for generating event influence presence / absence data indicating the presence / absence of noise influence on the event signal based on a result of determining whether or not a read operation of the image sensor section was being executed when the event was detected by the event sensor section; a transmission unit that transmits the event influence presence / absence data and the imaging influence presence / absence data, and inputs at least one of event data obtained based on the event signal and captured image data obtained based on the light receiving signal, the event data being transmitted by a sensor unit having: an imaging influence presence / absence data generating unit that generates imaging influence presence / absence data indicating the presence or absence of an influence of noise on the light receiving signal based on a result of the input of the event influence presence / absence data and a transmission unit that transmits the event influence presence / absence data and the imaging influence presence / absence data, and the input data being processed based on the event influence presence / absence data and the imaging influence presence / absence data corresponding to the input data.

Images