Information processing apparatus and information processing method

Abstract

The information processing apparatus includes: a first receiving unit that receives first data generated by a first sensor in synchronization with a first reference signal and second data generated by a second sensor; and a transmission unit that transmits at least a portion of the first data and the second data as third data synchronized with the second reference signal.

Description

Technical Field

[0001] The present disclosure relates to an information processing device and an information processing method. Background Art

[0002] In solid-state imaging devices including complementary metal oxide semiconductors (CMOS) and the like, asynchronous or synchronous solid-state imaging elements that detect brightness changes of each pixel as events in real time have been proposed. As described above, solid-state imaging elements that detect events of each pixel are also called event-based visual sensors (EVS).

[0003] Reference List

[0004] Patent Literature

[0005] Patent Document 1: 2021-508975 Summary of the invention

[0006] Technical issues

[0007] However, in a hybrid sensor that combines multiple sensors such as an image sensor and EVS or a time-of-flight (ToF) sensor and EVS, all data acquired by each sensor is not necessarily required in the processing circuit in the subsequent stage, and there is a case where useless data transmission occurs and transmission efficiency is reduced.

[0008] Therefore, the present disclosure proposes an information processing device and an information processing method capable of suppressing a decrease in transmission efficiency.

[0009] Solution to the problem

[0010] In order to solve the above problems, an information processing device according to one embodiment of the present disclosure includes: a first receiving unit, receiving first data generated by a first sensor in synchronization with a first reference signal and second data generated by a second sensor; and a transmission unit, transmitting at least a part of the first data and the second data as third data synchronized with the second reference signal. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 : is a functional block diagram showing a schematic configuration example of an imaging device according to an embodiment of the present disclosure.

[0012] Figure 2 is a timing chart for describing a schematic operation example of the imaging device according to the first operation example of the embodiment of the present disclosure.

[0013] Figure 3 is a timing chart for describing a schematic operation example of the imaging device according to the second operation example of the embodiment of the present disclosure.

[0014] Figure 4is a diagram illustrating an example of a data structure of third output data according to an embodiment of the present disclosure.

[0015] Figure 5 : is a diagram showing a configuration example of a first signal processing device and a second signal processing device according to a first example of an embodiment of the present disclosure.

[0016] Figure 6 : is a diagram showing a configuration example of a first signal processing device and a second signal processing device according to a second example of an embodiment of the present disclosure.

[0017] Figure 7 1 is a functional block diagram showing a schematic configuration example of an imaging device according to a first example of an embodiment of the present disclosure.

[0018] Figure 8 1 is a timing chart showing a schematic operation example of the imaging device according to the first example of the embodiment of the present disclosure.

[0019] Fig. 9 : is a functional block diagram showing a schematic configuration example of an imaging device according to a second example of an embodiment of the present disclosure.

[0020] Fig.10 : is a timing chart showing a schematic operation example of the imaging device according to the second example of the embodiment of the present disclosure.

[0021] Fig.11 : is a functional block diagram showing a schematic configuration example of an imaging device according to a third example of an embodiment of the present disclosure.

[0022] Fig.12 : is a timing chart showing a schematic operation example of the imaging device according to the third example of the embodiment of the present disclosure.

[0023] Fig.13 : is a functional block diagram showing a schematic configuration example of an imaging device according to a fourth example of an embodiment of the present disclosure.

[0024] Fig.14 : is a timing chart showing a schematic operation example of the imaging device according to the fourth example of the embodiment of the present disclosure.

[0025] Fig.15 2 is a diagram for describing deblurring correction and frame interpolation according to an embodiment of the present disclosure.

[0026] Fig.16 is a diagram for describing frame interpolation according to an embodiment of the present disclosure.

[0027] Fig.17 is a diagram for describing frame interpolation of depth information according to an embodiment of the present disclosure.

[0028] Fig.18 is a diagram for describing an effect of obtaining through frame interpolation of depth information according to an embodiment of the present disclosure (Part 1).

[0029] Fig.19 is a diagram for describing an effect of obtaining through frame interpolation of depth information according to an embodiment of the present disclosure (Part 2).

[0030] Fig. 20 is a diagram (Part 3) for describing the effect of obtaining through frame interpolation of depth information according to an embodiment of the present disclosure.

[0031] Fig.21 is a block diagram showing a configuration example of an imaging device according to a first configuration example of an embodiment of the present disclosure.

[0032] Fig. 22 is a block diagram showing a configuration example of an imaging device according to a second configuration example of an embodiment of the present disclosure.

[0033] Fig.23 : is a block diagram showing a configuration example of an imaging device according to a third configuration example of an embodiment of the present disclosure.

[0034] Fig.24 : is a schematic diagram showing a schematic configuration example of an imaging device according to an embodiment of the present disclosure.

[0035] Fig.25 is a block diagram showing a system configuration example of an imaging device according to an embodiment of the present disclosure.

[0036] Fig.26 is a block diagram showing a schematic configuration example of a hybrid sensor according to an embodiment of the present disclosure.

[0037] Fig. 27 is a block diagram showing another schematic configuration example of a hybrid sensor according to an embodiment of the present disclosure.

[0038] Fig.28 is a schematic diagram showing an example of a pixel array of a hybrid sensor according to an embodiment of the present disclosure.

[0039] Fig.29 is a schematic diagram illustrating an example of another pixel array of a hybrid sensor according to an embodiment of the present disclosure.

[0040] Fig.30 : is a diagram showing an exemplary stacking structure of a solid-state imaging device according to an embodiment of the present disclosure.

[0041] Fig.31is a block diagram depicting an example of a schematic configuration of a vehicle control system.

[0042] Fig.32 1 is a diagram of assistance in explaining an example of the installation positions of the vehicle exterior information detection portion and the imaging unit. DETAILED DESCRIPTION

[0043] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In addition, in the following embodiments, the same reference numerals are given to the same parts, and repeated descriptions are omitted.

[0044] Furthermore, the present disclosure will be described in the order of the following items.

[0045] 1. Implementation Method

[0046] 1.1 Schematic Configuration Example

[0047] 1.2 Schematic operation examples

[0048] 1.2.1 First operation example (asynchronous type)

[0049] 1.2.2 Second Operation Example (Synchronous Type)

[0050] 1.3 Example of the structure of the third output data

[0051] 1.4 Configuration example of signal processing device

[0052] 1.4.1 First Example

[0053] 1.4.2 Second Example

[0054] 1.5 Specific Examples of Imaging Devices

[0055] 1.5.1 First Example

[0056] 1.5.2 Second Example

[0057] 1.5.3 Third Example

[0058] 1.5.4 Fourth Example

[0059] 1.6 Deblurring Correction and Frame Interpolation

[0060] 1.7 Frame Interpolation of Depth Information

[0061] 1.8 Configuration example of imaging device

[0062] 1.8.1 First Configuration Example

[0063] 1.8.2 Second Configuration Example

[0064] 1.8.3 Third Configuration Example

[0065] 1.9 Specific Configuration Examples

[0066] 1.10 Configuration Example of Hybrid Sensors

[0067] 1.10.1 Another configuration example of hybrid sensor

[0068] 1.10.2 Example of a hybrid sensor pixel array

[0069] 1.10.3 Another Pixel Array Example of a Hybrid Sensor

[0070] 1.11 Example of stacked structure of solid-state imaging device

[0071] 2. Application examples for mobile objects

[0072] 1. Implementation Method

[0073] First, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0074] 1.1 Schematic Configuration Example

[0075] Figure 1 1 is a functional block diagram showing a schematic configuration example of an imaging device according to the present embodiment. Figure 1 As shown, the imaging device 1 includes a plurality of sensors (including a first sensor 11 and a second sensor 12 ), a first signal processing device 13 and a second signal processing device 14 .

[0076] In the present embodiment, the first sensor 11 may be, for example, various sensors such as an image sensor that acquires a color image (including a multispectral image including wavelength components other than the wavelength components of the three primary colors of RGB) and / or a monochrome image (including a special light image such as infrared light), a distance measurement sensor that acquires the distance to a subject in a viewing angle (also referred to as depth), and an inertial measurement device that detects acceleration, angular acceleration, etc. Note that in the following description, a complementary metal oxide semiconductor (CMOS) image sensor (CIS) is exemplified as an image sensor, and a ToF sensor of a direct ToF method or an indirect ToF method is exemplified as a distance measurement sensor.

[0077] Furthermore, in the present embodiment, a case where the second sensor 12 is an EVS will be exemplified. The EVS may be a synchronous type in which two or more pixels synchronously detect an event, or may be an asynchronous type in which pixels independently detect an event.

[0078] The first signal processing device 13 includes, for example, a processing device such as a field programmable gate array (FPGA), a microcontroller unit (MCU) or a microprocessor unit (MPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC) or an application-specific standard product (ASSP), and selectively outputs data required for processing in the second signal processing device 14 in a subsequent stage from the first output data output from the first sensor 11 and the second output data output from the second sensor 12 as third output data.

[0079] The second signal processing device 14 includes, for example, an application processor, etc., and performs various types of processing (hereinafter, also referred to as predetermined processing), such as deblur correction, frame interpolation, recognition processing, and simultaneous localization and mapping (SLAM), based on the third output data input from the first signal processing device 13. The second signal processing device 14 can specify the data for making a request of the first signal processing device 13 by transmitting the request to the first signal processing device 13.

[0080] 1.2 Schematic operation examples

[0081] Next, a schematic operation of the imaging device according to the present embodiment will be described. Hereinafter, a case where the second sensor 12 is an asynchronous sensor (first operation example) and a case where the second sensor 12 is a synchronous sensor (second operation example) will be exemplified.

[0082] 1.2.1 First operation example (asynchronous type)

[0083] Figure 2 1 is a timing chart for describing a schematic operation example of the imaging device according to the first operation example of the present embodiment. It should be noted that Figure 2 A case is shown where the first sensor 11 is an image sensor and the second sensor 12 is an asynchronous EVS.

[0084] like Figure 2 As shown, the first sensor 11 performs exposure of pixels and reads pixel signals from the exposed pixels during a frame period defined by the first reference signal in synchronization with the rising (or falling) first reference signal in a predetermined first frame period, so as to output pixel signals of one frame read from all pixels as the first output data D1. In addition, the first sensor 11 generates an exposure timing signal indicating that the pixel is in the exposure period, and inputs the exposure timing signal to the second sensor 12.

[0085] On the other hand, at the timing of detecting an event in each pixel, the second sensor 12 generates event data including position information (address, etc.) of the pixel in which the event has been detected, the polarity of the detected event (positive event or negative event), information indicating the time when the event has been detected (timestamp, etc.), and information indicating whether the time when the event has been detected is during the exposure period of the first sensor 11 (hereinafter, referred to as exposure period mark), so as to output the generated event data as the second output data (event stream) D2 as needed. Note that the exposure period mark can be generated based on the exposure timing signal input from the first sensor 11. In addition, the second output data D2 can be output after being synchronized (framed) by a reference signal having a period and / or phase different from that of the first reference signal.

[0086] The first signal processing device 13 generates frame data using the first output data D1 input from the first sensor 11 and the second output data D2 input from the second sensor 12 in synchronization with the second reference signal that rises (or falls) during a specified second frame period, and outputs the generated frame data as third output data D3.

[0087] At this time, the first signal processing device 13 can determine whether the event indicated by each event data in the second output data D2 is an event detected during the exposure period based on the exposure period mark, and can separately process the event data of the event detected during the exposure period (also referred to as the second a output data D2a) and the event data of the event detected outside the exposure period (also referred to as the second b output data D2b).

[0088] Furthermore, in the case where any one of the second a-output data D2a and the second b-output data D2b is requested in the request, the first signal processing device 13 may generate the third output data D3 by including the requested data in the frame data.

[0089] However, in the case where both the second a output data D2a and the second b output data D2b are requested in the request, in the case where no data is specified in the request, etc., the first signal processing device 13 may generate frame data including both the second a output data D2a and the second b output data D2b.

[0090] Note that the first frame period and the second frame period may be the same period or different periods. In addition, when the first frame period and the second frame period are the same period, the first reference signal and the second reference signal may have the same phase (i.e., rise (or fall) at the same timing), or may have different phases (i.e., the phases are shifted).

[0091] 1.2.2 Second Operation Example (Synchronous Type)

[0092] Figure 3 is a timing chart for describing a schematic operation example of the imaging device according to the second operation example of the present embodiment. It should be noted that Figure 3 A case is shown where the first sensor 11 is an image sensor and the second sensor 12 is a synchronous EVS.

[0093] exist Figure 3 In the embodiment, the first sensor 11 outputs the first output data D1 for one frame in synchronization with the first reference signal, as shown in FIG. Figure 2 In the first operation example shown in FIG.

[0094] On the other hand, as in Figure 2 In the first operation example shown in FIG, the second sensor 12 generates event data at the timing of detecting an event in each pixel. However, in the second operation example, the event data of each frame is accumulated based on a third reference signal that rises (or falls) in a predetermined third frame period, and a set of accumulated event data is output as output data D2 of one frame.

[0095] Note that the first frame period and the third frame period may be the same period or different periods. In addition, when the first frame period and the third frame period are the same period, the first reference signal and the third reference signal may have the same phase (i.e., rise (or fall) at the same timing), or may have different phases (i.e., the phases are shifted).

[0096] As in Figure 2 In the first operation example shown in FIG. 1 , the first signal processing device 13 generates and outputs the third output data D3 by using the first output data D1 and the second output data D2 that can be input from the second sensor 12 in synchronization with the second reference signal. At this time, the second a output data D2a and the second b output data D2b can be processed separately based on the exposure period mark.

[0097] Other operations may be similar to those of the first operation example, so descriptions thereof will be omitted here.

[0098] 1.3 Example of the structure of the third output data

[0099] Next, an example of the data structure of the third output data D3 output from the first signal processing device 13 will be described. Figure 4 : is a diagram showing an example of the data structure of third output data according to the present embodiment.

[0100] like Figure 4As shown in, for example, the third output data D3 has a structure in which a packet P1 including the first output data D1 and a packet P2 including the second output data D2 are arranged in a payload portion between a header portion FS indicating a header of frame data and a trailer portion FE indicating the end of the frame data. When the second a output data D2a and the second b output data D2b are distinguished, the packet P2 may be divided into a packet P2a of the second a output data D2a and a packet P2b of the second b output data D2b. In addition, a data packet P4 including embedded data may be arranged between the header portion FS and the trailer portion FE.

[0101] Each of the packets P1, P2 (P2a, P2b), and P4 includes a packet header PH indicating a head of the packet and a packet footer PF indicating an end of the packet, and has a structure in which each piece of data is arranged between the packet header PH and the packet footer PF.

[0102] The data length of each of the packets P1, P2 (P2a, P2b) and P4 may be a fixed length or a variable length. In the case of a fixed length, if the data is not contained in one packet, the data may be divided into a plurality of packets. In addition, in the case where the data is shorter than the data length of one packet, the deficiency may be filled with a fixed value.

[0103] In addition, when the second a output data D2a and the second b output data D2b are distinguished from each other, information for identifying the storage location of each of the second a output data D2a and / or the second b output data D2b can be stored in any one of the packet header PH or the packet trailer PF of the packet P2a and / or P2b, the header part FS or the trailer FE of the frame data or the payload part, and the embedded data.

[0104] 1.4 Configuration example of signal processing device

[0105] Next, the configurations of the first signal processing device 13 and the second signal processing device 14 will be described through some examples.

[0106] 1.4.1 First Example

[0107] The first example shows a case where generation of the third output data is performed in the first signal processing unit 13 and predetermined processing is performed on the third output data by the second signal processing device 14 . Figure 5 : is a diagram showing a configuration example of a first signal processing device and a second signal processing device according to the first example.

[0108] like Figure 5As shown, in the first example, the first signal processing device 13 includes a receiving unit 401, a synchronization processing unit 402, a transmission unit 403, and a communication control unit 404. On the other hand, the second signal processing device 14 includes a receiving unit 501, an application programming interface (API) processing unit 502, an API calling unit 503, a signal processing unit 504, an API calling unit 505, an API processing unit 506, and a communication control unit 507.

[0109] In the first signal processing device 13 , the receiving unit 401 receives the first output data output from the first sensor 11 in synchronization with the first reference signal, and receives the second output data output from the second sensor 12 asynchronously or synchronously.

[0110] The synchronization processing unit 402 generates third output data using the first output data and the second output data received by the receiving unit 401. Then, the transmission unit 403 transmits the third output data generated by the synchronization processing unit 402 to the second signal processing device 14.

[0111] In the second signal processing device 14 , the receiving unit 501 receives the third output data transmitted from the first signal processing device 13 .

[0112] When the receiving unit 501 receives the third output data, the API calling unit 503 calls an API for acquiring the third output data. In response to the API call, the API processing unit 502 provides the third output data received by the receiving unit 501 to the API calling unit 503. On the other hand, the signal processing unit 504 performs predetermined processing using the third output data acquired by the API calling unit 503. Note that the API calling unit 503 may call an API for selectively acquiring only the second a output data D2a generated during the exposure period of the first sensor 11 in the third output data. In this case, in response to the API call, the API processing unit 502 may selectively provide the second a output data D2a in the third output data received by the receiving unit 501 to the API calling unit 503. Alternatively, the API calling unit 503 may call an API for selectively acquiring only the second b output data D2b generated outside the exposure period of the first sensor 11 in the third output data. In this case, in response to the API call, the API processing unit 502 may selectively provide the second b output data D2 b among the third output data received by the receiving unit 501 to the API calling unit 503 .

[0113] Furthermore, in the second signal processing device 14, the API that transmits the request to the first signal processing device 13 accesses the API processing unit 506 via the API calling unit 505. The accessed API processing unit 506 transmits the request determined by the API to the communication control unit 404 of the first signal processing device 13 via the communication control unit 507. At this time, the API calling unit 505 may call the API for transmitting the request for requesting the first signal processing device 13 to provide only the second a output data D2a generated in the exposure period of the first sensor 11 as the third output data. On the other hand, the API processing unit 506 may cause the communication control unit 507 to transmit the request for transmitting only the second a output data D2a as the third output data to the communication control unit 404 in response to the API call. In this case, the first signal processing device 13 may transmit the second a output data D2a generated during the exposure period of the first sensor 11 to the second signal processing device 14 as the third output data. Alternatively, the API calling unit 505 may call an API for transmitting a request for requesting the first signal processing device 13 to provide only the second b-output data D2b generated outside the exposure period of the first sensor 11 as the third output data. On the other hand, in response to the API call, the API processing unit 506 may cause the communication control section 507 to transmit a request for transmitting only the second b-output data D2b as the third output data to the communication control section 404. In this case, the first signal processing device 13 may transmit the second b-output data D2b generated outside the exposure period of the first sensor 11 as the third output data to the second signal processing device 14.

[0114] 1.4.2 Second Example

[0115] The second example shows a case where, in addition to generating the third output data in the first signal processing unit 13 , at least a part of the predetermined processing for the third output data is performed, and the remaining part of the predetermined processing for the third output data is performed by the second signal processing unit 14 . Figure 6 : is a diagram showing a configuration example of a first signal processing device and a second signal processing device according to a second example.

[0116] like Figure 6 As shown, in the second example, the first signal processing device 13 includes an API processing unit 411, an API calling unit 412, a signal processing unit 413, an API calling unit 414, and an API processing unit 415 in addition to the receiving unit 401, the synchronization processing unit 402, and the transmission unit 403. At the same time, the second signal processing device 14 includes a receiving unit 501, an API processing unit 502, and an API calling unit 503.

[0117] In the first signal processing device 13, the synchronization processing unit 402 generates the third output data according to the first output data and the second output data received by the receiving unit 401. When the synchronization processing unit 402 generates the third output data, the API calling unit 412 calls the API for obtaining the third output data to obtain the third output data via the API processing unit 411. On the other hand, the signal processing unit 413 uses the third output data obtained by the API calling unit 412 to perform at least a part of the predetermined processing. Note that the API calling unit 412 can call an API for selectively obtaining only the second a output data D2a or the second b output data D2b in the third output data. In this case, in response to the API call, the API processing unit 411 can selectively provide the second a output data D2a or the second b output data D2b in the third output data generated by the synchronization processing unit 402 to the API calling unit 412.

[0118] On the other hand, the API calling unit 414 calls an API for transmitting the result of the predetermined processing of the signal processing unit 413, so that the API processing unit 415 transmits the processing result to the second signal processing device 14 via the transmission unit 403. Note that the API calling unit 414 can call an API for transmitting the third output data together with the processing result.

[0119] The receiving unit 501 of the second signal processing device 14 receives the processing result transmitted from the transmission unit 403. When the receiving unit 501 receives the third output data, the API calling unit 503 calls the API for acquiring the third output data. In response to the API call, the API processing unit 502 provides the third output data received by the receiving unit 501 to the API calling unit 503. Then, the API calling unit 503 performs the remaining predetermined processing by calling and executing the API.

[0120] It should be noted that in the following description, for the sake of clarity, a case where the first signal processing device 13 and the second signal processing device 14 have the configuration according to the first example will be exemplified as a first example.

[0121] 1.5 Specific Examples of Imaging Devices

[0122] Next, a more specific configuration of the imaging device 1 according to the present embodiment will be described by way of some examples.

[0123] 1.5.1 First Example

[0124] In the first example, an imaging device 1A configured to perform so-called deblur correction will be described, in which an image sensor is used as a first sensor 11, an EVS is used as a second sensor 12, and blur occurring in image data (first output data) acquired by the image sensor 11 is corrected using event data (second output data) from the EVS 12. Note that, hereinafter, the EVS 12 will be exemplified as an asynchronous type (see Figure 2 ), but the present invention is not limited thereto, and may be a synchronous type (see Figure 3 ).

[0125] As one of the factors causing blurring of the image data acquired by the image sensor, it is conceivable that the subject within the viewing angle moves during the exposure period. This is particularly problematic in cases where the exposure period is extended, such as imaging in a dark place. On the other hand, typically, the basic frame rate of the EVS is much higher than the basic frame rate of the image sensor. For example, when the general frame rate of the image sensor is 60 to 120 frames per second (fps), the EVS can achieve a frame rate of 1000fps or more. Therefore, the image data reconstructed based on the event data acquired by the EVS is hardly affected by the movement of the subject. Therefore, in the first example, the blur (deblur correction) occurring in the image data acquired by the image sensor 11 is corrected based on the event data acquired by the EVS 12.

[0126] Figure 7 : is a functional block diagram showing a schematic configuration example of an imaging device according to the first example. Figure 8 : is a timing chart showing a schematic operation example of the imaging device according to the first example.

[0127] like Figure 7 and Figure 8 As shown, in the first example, the first signal processing device 13 discards the event data (second b output data D2b) acquired outside the exposure period in the event stream (second output data D2) input from the EVS 12, and uses the image data (first output data D1) input from the image sensor 11 and the event data (second a output data D2a) acquired by the EVS 12 within the exposure period to generate frame data (third output data D3) to be output to the second signal processing device 14.

[0128] As described above, the first signal processing device 13 discards event data (second b-output data D2b) that is not used for processing (deblurring correction) by the second signal processing device 14, which can reduce the amount of data transmitted from the first signal processing device 13 to the second signal processing device 14. As a result, the transmission band can be suppressed, and therefore, faster and more efficient data transmission can be achieved.

[0129] The second signal processing device 14 uses the event data (second a output data D2a) acquired during the exposure period (deblur correction) to correct the blur generated in the image data (first output data D1) of the frame data (third output data D3). As a result, the second signal processing device 14 can acquire high-quality image data with reduced blur.

[0130] In addition, in this example, the movement of the subject outside the exposure period does not blur the image data, so the event data (second b output data D2b) acquired outside the exposure time is discarded, but it is not limited to this. For example, the event data (second b output data D2b) acquired outside the exposure time may also be included in the frame data (third output data D3), and the second signal processing device 14 may be configured to use the event data (second b output data D2b) acquired outside the exposure time to perform frame interpolation between the image data (first output data D1) arranged along the time series.

[0131] In addition, in this example, the case where the first signal processing device 13 discards the event data (second b-output data D2b) that is not necessary for the deblurring correction is illustrated, but it is not limited to this, and event data generated outside the exposure time in the EVS 12 may also be discarded. Alternatively, the EVS 12 may be configured not to generate event data outside the exposure time.

[0132] 1.5.2 Second Example

[0133] In the second example, an imaging device 1B will be described, which is configured to acquire distance information (depth information) to a subject through the EVS 12 in addition to acquiring image data (first output data) through the image sensor 11 using an image sensor as the first sensor 11 and using the EVS as the second sensor 12. Note that, hereinafter, the EVS 12 will be exemplified as an asynchronous type (see Figure 2 ), but the present invention is not limited thereto, and may be a synchronous type (see Figure 3 ).

[0134] In this example, depth information is acquired using a so-called structured light method, in which a predetermined pattern of light is projected onto a subject, and distortion, size, etc. generated in the shape of the predetermined pattern of light projected onto the subject are recognized to measure the distance to the subject. Although an image sensor that detects the shape of light projected onto a subject may also be used, the distance may be measured at a higher speed using an EVS with a higher frame rate.

[0135] However, the light projected by the structured light method interferes with the image data acquired by the image sensor. Therefore, in this example, as will be described later, light is projected outside the exposure period of the image sensor, and the shape of the light projected onto the subject is detected by the EVS.

[0136] Fig. 9 is a functional block diagram showing a schematic configuration example of an imaging device according to the second example. Fig.10 2 is a timing chart showing a schematic operation example of the imaging device according to the second example.

[0137] like Fig. 9 As shown in , the imaging device 1B according to the second example includes a light projection unit 16 that projects a predetermined pattern of light toward a subject, in addition to the same configuration as that of the imaging device 1A according to the first example.

[0138] like Fig.10 As shown, the light projection period during which the light projection unit 16 projects light is set outside the exposure period of the image sensor 11. Therefore, the image data (first output data D1) acquired by the image sensor 11 can be prevented from being affected by the light transmitted from the light projection unit 16.

[0139] The first signal processing device 13 discards the event data (second a output data D2a) acquired during the exposure period in the event stream (second output data D2) input from EVS12, and uses the image data (first output data D1) input from the image sensor 11 and the event data (second b output data D2b) acquired by EVS12 outside the exposure period to generate frame data (third output data D3) to be output to the second signal processing device 14.

[0140] As described above, event data (second a output data D2a) that is not used for processing (distance measurement processing) in the second signal processing device 14 is discarded by the first signal processing device 13, so that the amount of data transmitted from the first signal processing device 13 to the second signal processing device 14 can be reduced as in the first example.

[0141] The second signal processing device 14 reconstructs the image data (first output data D1) using the event data (second b output data D2b) in the frame data (third output data D3), and recognizes the distortion, size, etc. of the shape of the projection light captured in the image data, thereby generating depth information representing the distance to the subject. Other configurations and operations may be similar to those of the first example.

[0142] In addition, in this example, the event data (second output data D2a) acquired during the exposure time in which the light irradiated to the subject is not detected is discarded, but the present invention is not limited thereto. For example, the event data (second a output data D2a) acquired during the exposure time may also be included in the frame data (third output data D3), and the second signal processing device 14 may be configured to perform deblurring correction on the image data (first output data) acquired by the image sensor 11 using the event data (second a output data D2a) acquired during the exposure time.

[0143] In addition, in this example, an example has been given in which event data (second a output data D2a) unnecessary for ranging processing is discarded by the first signal processing device 13, but the present invention is not limited thereto, and event data generated within the exposure time in the EVS 12 may be discarded. Alternatively, the EVS 12 may be configured not to generate event data within the exposure time.

[0144] 1.5.3 Third Example

[0145] In the third example, as in the first example, deblur correction of the image data (first output data) is performed using event data acquired within the exposure time (second a output data D2a), and as in the second example, a predetermined pattern of light is projected onto the subject outside the exposure time, and depth information is acquired using event data acquired at that moment (second b output data).

[0146] Fig.11 is a functional block diagram showing a schematic configuration example of an imaging device according to the third example. Fig.12 : is a timing chart showing a schematic operation example of the imaging device according to the third example.

[0147] like Fig.11 As shown, as in the imaging device 1B according to the second example, in addition to the configuration of the imaging device 1A according to the first example, the imaging device 1C according to the third example further includes a light projection unit 16. However, in the imaging device 1C according to the third example, the first signal processing device 13 generates the third output data D3 including the second a output data D2a in addition to the first output data D1 and the second b output data D2b.

[0148] The second signal processing device 14 corrects (deblurring correction) blur generated in the image data (first output data D1) of the frame data (third output data D3) using the event data (second a output data D2a) acquired during exposure, and reconstructs the image data (first output data D1) using the event data (second b output data D2b) of the frame data (third output data D3) to obtain depth information indicating the distance to the subject. Other configurations and operations may be similar to those of the first example or the second example.

[0149] 1.5.4 Fourth Example

[0150] In the fourth example, using a ToF sensor as the first sensor 11 and an EVS as the second sensor 12, an imaging device 1D configured to interpolate depth information obtained by a structured light method using the EVS 12 between frames of depth information acquired using the ToF sensor 11 will be described. Note that, hereinafter, the EVS 12 will be exemplified as an asynchronous type (see Figure 2 ), but the present invention is not limited thereto, and may be a synchronous type (see Figure 3 ).

[0151] Fig.13 is a functional block diagram showing a schematic configuration example of an imaging device according to a fourth example. Fig.14 : is a timing chart showing a schematic operation example of the imaging device according to the fourth example.

[0152] like Fig.13 As shown, the imaging device ID according to the fourth example also includes a light emitting unit 17, which replaces the image sensor 11 with a ToF sensor 11 and emits a pulsed laser toward the subject in the same configuration as the imaging device 1B according to the second example. The ToF sensor 11 may be a direct ToF method or an indirect ToF method. In addition, in the case of the direct ToF method, each pixel constituting the ToF sensor 11 may be, for example, a macro pixel including a group of multiple single-photon avalanche diode (SPAD) pixels. It should be noted that the light projection unit 16 and the light emitting unit 17 may be the same light source or may be separate light sources.

[0153] like Fig.14 As shown, the light emitting unit 17 emits pulsed laser light at the beginning of each frame period defined by the first reference signal. On the other hand, the ToF sensor 11 as the first sensor repeatedly exposes light in a predetermined sampling period in a predetermined sampling period after the light emitting unit 17 emits light, thereby creating a histogram of each macro pixel reflecting the distance to the subject. The generated histogram is output to the first signal processing device 13 as the first output data D1.

[0154] On the other hand, the light projection unit 16 projects a predetermined pattern of light toward the subject outside the sampling period instead of outside the exposure period. Other operations may be similar to those of the structured light method described in the second example or the third example.

[0155] The first signal processing device 13 discards the event data (second a output data D2a) acquired within the sampling period in the event stream (second output data D2) input from EVS12, and uses the histogram (first output data D1) input from the ToF sensor 11 and the event data (second b output data D2b) acquired through EVS12 outside the sampling period to generate frame data (third output data D3) to be output to the second signal processing device 14.

[0156] The second signal processing device 14 generates depth information indicating the distance to the subject using the histogram D1 in the frame data (third output data D3). In addition, the second signal processing device 14 reconstructs the image data (first output data D1) using the event data (second b output data D2b) and generates depth information indicating the distance to the subject, interpolating between frames using the depth information acquired by the ToF sensor 11.

[0157] As described above, by interpolating frames between frames of depth information obtained using the ToF sensor 11 with depth information obtained by a structured light method using the EVS 12, dense depth information in the time axis direction can be obtained, and therefore, a more detailed distance to the subject can be recognized. Other configurations and operations may be similar to those in any of the first to third examples.

[0158] 1.6 Deblurring Correction and Frame Interpolation

[0159] Here, the deblurring correction and frame interpolation of the image data exemplified above will be described. Fig.15 2 is a diagram for describing deblurring correction and frame interpolation according to the present embodiment.

[0160] like Fig.15 As shown in (A), the frame data (third output data D3) output from the first signal processing device 13 includes the image data (first output data D1) acquired by the image sensor 11 and the event data (second a output data D2a) acquired during the exposure time in the EVS 12. While one image data (first output data D1) is generated during one frame period defined by the first reference signal, a plurality of image data (hereinafter, also referred to as event images) can be reconstructed from the event data (second a output data D2a) acquired during one frame period.

[0161] Therefore, if Fig.15As shown in (B), the movement of the subject during the exposure period (movement direction, movement distance, etc.) can be identified based on multiple event images reconstructed from the event data (second a output data D2a), and the blur generated in the image data (first output data D1) can be corrected based on the identified movement of the subject.

[0162] In addition, if Fig.15 As shown in (B), the optical flow of each area in the image data (first output data D1) can also be obtained using multiple event images reconstructed from the event data (second a output data D2a), and the image data can be interpolated between frames based on the obtained optical flow.

[0163] like Fig.16 As shown, frame interpolation may also be performed on image data outside the exposure time based on event data (second b-output data D2 b ) acquired by the EVS 12 outside the exposure time.

[0164] In addition, the generation of image data to be interpolated is not limited to the above-mentioned method based on optical flow, and for example, various methods such as a method using machine learning can be used, wherein the machine learning uses a deep neural network (DNN) using image data (first output data D1) and event data (second output data D2) as input.

[0165] 1.7 Frame interpolation of depth information

[0166] Next, the frame interpolation of the depth information exemplified above will be described. Fig.17 : is a diagram for describing frame interpolation of depth information according to the present embodiment. Figures 18 to 20 is a diagram for describing the effect of acquiring by frame interpolation of depth information according to the present embodiment. In the following description, for the sake of understanding, the reference symbol of the depth information generated from the first output data D1 (histogram) is D1, and the reference symbol of the depth information acquired from the image data reconstructed from the event data outside the sampling period (the second b output data D2b) is D2b. In addition, the reference symbol of the depth information finally acquired by performing frame interpolation on the depth information D1 using the depth information D2b is set to D4.

[0167] like Fig.17 As shown, the frame rate of the depth information D2b obtained from the image data reconstructed from the event data (the second b-output data D2b) is higher than the frame rate of the depth information D1 generated from the first output data D1 (histogram). Therefore, by interpolating frames between frames of the depth information D1 generated from the first output data D1 (histogram) using the depth information D2b obtained from the image data reconstructed from the event data (the second b-output data D2b), the depth information can be acquired at a higher rate.

[0168] Specifically, for example, a difference ΔD2b (second b-output data D2b) between the previous frame and the next frame of the depth information D2b obtained from the image data reconstructed according to the event data is obtained, and the difference ΔD2b is added to the depth information D1 generated from the first output data D1 (histogram), thereby generating the depth information D2b of the next frame of the depth information D1. Thereafter, the depth information D2b interpolated to the depth information D1 of the next frame can be obtained by adding the difference ΔD2b between the corresponding plurality of pieces of depth information D2b to the depth information D2b generated as the previous frame.

[0169] By repeating the above operation, the depth information D4 can be finally generated at a rate higher than that of the depth information D1.

[0170] For example, Fig.18 As shown, the depth information D4 of 30 fps may be generated by interpolating the depth information D2 b of 5 frames between the frames of the depth information D1 of 180 fps.

[0171] In addition, for example, Fig.19 As shown, the total delay between the depth information D4 depends on the amount of delay when generating event data (second b output data D2b), but since the EVS 12 can generate event data (second b output data D2b) at a higher rate than the ToF sensor 11, the total delay between the depth information D4 can be minimized.

[0172] In addition, for example, Fig. 20 As shown, by inserting a frame between the frames of the depth information D1 having the depth information D2b, the depth information D4 having a sufficient frame rate can be acquired even when the frame rate of the depth information D1 is reduced. As a result, the frame rate of the histogram (first output data D1) output from the ToF sensor 11 can be reduced, so that power consumption can also be reduced.

[0173] 1.8 Configuration example of imaging device

[0174] Next, the above reference will be described Figures 7 to 14 The configurations of the imaging devices 1A to 1D are illustrated.

[0175] 1.8.1 First Configuration Example

[0176] First, the above reference Figure 7 and Figure 8 A configuration example of the imaging device 1A of the first example is described. Fig.21 1 is a block diagram showing a configuration example of an imaging device according to a first configuration example of the present embodiment. Fig.21As shown in , in the present configuration example, the imaging device 1 includes, for example, a solid-state imaging device 10 and an application processor 14 as a second signal processing device.

[0177] The solid-state imaging device 10 includes a control section 15 , a light projection unit 16 , and an external interface (I / F) 19 in addition to an image sensor 11 as a first sensor, an EVS 12 as a second sensor, and an FPGA 13 as a first signal processing device.

[0178] The control section 15 includes, for example, an information processing device such as a central processing unit (CPU), and controls each unit of the solid-state imaging device 10 .

[0179] For example, the external I / F 19 may be a communication adapter for establishing communication with the external application processor 14 via a communication network conforming to any standard, in addition to a wireless local area network (LAN) or a wired LAN, any standard such as a controller area network (CAN), a local interconnect network (LIN), FlexRay (registered trademark), a mobile industry processor interface (MIPI), or a low voltage differential signal (LVDS).

[0180] Here, for example, in the case where the solid-state imaging device 10 is mounted on a vehicle or the like, the application processor 14 may be an engine control unit (ECU) or the like mounted on the vehicle or the like. Furthermore, in the case where the solid-state imaging device 10 is mounted on an autonomous mobile robot (such as a household pet robot) or an autonomous mobile object (such as a robot vacuum cleaner, an unmanned aircraft, or a follow-up conveyor robot), the application processor 14 may be a control device that controls the autonomous mobile object or the like. Furthermore, in the case where the solid-state imaging device 10 is mounted on an electronic device such as a mobile phone, a smart phone, or a tablet terminal, the application processor 14 may be a CPU incorporated in the electronic device, a server (including a cloud server, etc.) connected to the electronic device via a network, or the like.

[0181] The FPGA 13 receives the first output data D1 (image data) output from the image sensor 11 and the second output data D2 (event stream) output asynchronously from the EVS 12 in synchronization with the first reference signal. Then, the FPGA 13 generates the third output data D3 (frame data) containing the designated data in synchronization with the second reference signal according to the request input from the application processor 14 via the external I / F 19. The generated third output data is output to the application processor 14 via the external I / F 19.

[0182] The application processor 14 performs predetermined processing such as deblurring processing, recognition processing, and SLAM on the third output data output from the solid-state imaging device 10 .

[0183] 1.8.2 Second Configuration Example

[0184] Fig. 22 1 is a block diagram showing a configuration example of an imaging device according to a second configuration example of the present embodiment. Fig. 22 As shown in FIG. 1 , the imaging device 2 according to the present configuration example has a configuration in which the image sensor 11 and the EVS 12 in the solid-state imaging device 10 are replaced with a hybrid sensor 11A, and the configuration of the hybrid sensor 11A is similar to that in the first configuration example. Fig.21 The configuration of the imaging device 1 is described.

[0185] Although the details will be described later, for example, the hybrid sensor 11A according to the second configuration example is a sensor in which an image sensor and an EVS are integrated into one chip and can output image data (corresponding to the first output data) and event data (corresponding to the second output data). However, the hybrid sensor 11A according to the present embodiment is not limited thereto, and for example, a combination of various sensors (such as a ToF sensor and an EVS or an image sensor and a ToF sensor) used as the first sensor 11 and the second sensor 12 may be integrated into one chip.

[0186] The other configuration may be similar to that of the imaging device 1 according to the first configuration example, and therefore a detailed description thereof will be omitted here.

[0187] 1.8.3 Third Configuration Example

[0188] Next, the above reference will be described. Fig. 9 , Fig.10 , Fig.11 , Fig.12 , Fig.13 and Fig.14 Configuration examples of imaging devices 1B to 1D of second to fourth examples described. Note that in this configuration example, a case where the light projection unit 16 and the light emitting unit 17 in the imaging device 1D according to the fourth example are configured by a common light source 18 is exemplified.

[0189] Fig.23 1 is a block diagram showing a configuration example of an imaging device according to a third configuration example of the present embodiment. Fig.23 As shown in FIG. 1 , the imaging device 3 according to the present configuration example has a configuration in which the light source 18 is arranged in a manner similar to that in the reference first configuration example. Fig.21 A configuration similar to the configuration of the described imaging device 1 is added.

[0190] In the above second and third examples, the light source 18 projects a predetermined pattern of light L3 onto the subject 90 outside the exposure period of the image sensor 11. On the other hand, the EVS 12 detects the shape of the light L4 projected on the subject 90 and generates event data (second b-output data D2b).

[0191] In the fourth example described above, the light source 18 emits the pulse laser L1 during the exposure period of the image sensor 11. On the other hand, the ToF sensor 11 detects the reflected light L2 reflected by the subject 90 at a predetermined sampling period, and generates a histogram (first output data D1) reflecting the distance to the subject 90. In addition, the light source 18 projects a predetermined pattern of light L3 to the subject 90 outside the exposure period of the image sensor 11. On the other hand, the EVS 12 detects the shape of the light L4 projected on the subject 90, and generates event data (second b-output data D2b).

[0192] The other configuration may be similar to that of the imaging device 1 according to the first configuration example, and therefore a detailed description thereof will be omitted here.

[0193] 1.9 Specific Configuration Example

[0194] Next, a specific configuration example of the imaging device 1 according to the present embodiment will be described. Fig.24 is a schematic diagram showing a schematic configuration example of an imaging device according to the present embodiment, and Fig.25 is a block diagram showing a system configuration example of the imaging device according to the present embodiment.

[0195] like Fig.24 As shown in FIG. 1 , the imaging device 1 according to the present embodiment includes a light source 1010, an irradiation lens 1030, an imaging lens 1040, a solid-state imaging device 100, and a first signal processing device 1050. The light source 1010 corresponds to, for example, Fig.23 The light source 18 in the solid-state imaging device 100 corresponds to Fig.21 or Fig. 22 That is, in this example, the light source 18 is provided outside the solid-state imaging device 10. Note that the irradiation lens 1030 is not an essential component and may be omitted.

[0196] like Fig.25As shown, the light source 1010 includes, for example, a vertical cavity surface emitting laser (VCSEL) 1012 and a light source driving unit 1011 that drives the VCSEL 1012. However, the present invention is not limited to the VCSEL 1012, and various light sources such as light emitting diodes (LEDs) may be used. In addition, the light source 1010 may be any one of a point light source, a surface light source, and a line light source. In the case of a surface light source or a line light source, for example, the light source 1010 may have a configuration in which a plurality of point light sources (e.g., VCSELs) are arranged one-dimensionally or two-dimensionally. In the present embodiment, the light source 1010 emits, for example, infrared (IR) light. However, the present invention is not limited thereto, and the light emitted from the light source 1010 may be light of a wavelength band different from the wavelength band of visible light.

[0197] The illumination lens 1030 is arranged close to the emission surface of the light source 1010 , and converts the light emitted from the light source 1010 into illumination light having a predetermined divergence angle.

[0198] The imaging lens 1040 is disposed close to the light receiving surface of the solid-state imaging device 100, and forms an image by incident light on the light receiving surface of the solid-state imaging device 100. The incident light may also include reflected light emitted from the light source 1010 and reflected by the subject 901. For example, in the case of eye tracking, the subject 901 may be around the user's eyes including the eyeball, and in the case of SLAM, the object may be an object existing around its own position. However, the object is not limited thereto, and various objects may be the subject 901 depending on the purpose of use of the imaging device 1.

[0199] Although the details will be described later, Fig.25 As shown, the solid-state imaging device 100 includes, for example, a hybrid sensor 1022 in which pixels for detecting brightness (hereinafter, referred to as brightness pixels) and pixels (hereinafter, a sensor control section 1021 for detecting events are arranged in a two-dimensional grid, and by driving the hybrid sensor 1022, the sensor control section 1021 generates image data (first output data D1) based on brightness signals detected by the brightness pixels and generates data (second output data D2) based on event data detected by the event pixels. Note that although the details will be described later, a configuration including brightness pixels (also referred to as first pixels) may constitute an image sensor (first sensor 11), and a configuration including event pixels (also referred to as second pixels) may constitute an EVS (second sensor 12).

[0200] As described above, the first signal processing device 1050 includes a processing device such as an FPGA, an MCU or an MPU, a DSP, an ASIC, or an application specific standard product (ASSP), and drives the VCSEL 1012 via the light source driving unit 1011. In addition, the first signal processing device 1050 acquires the first output data D1 and the second output data D2 by controlling the solid-state imaging device 100, and generates the third output data D3 from the acquired data.

[0201] For example, the illumination light emitted from the light source 1010 is projected onto the subject (also referred to as the measured object or object) 901 through the illumination lens 1030. The subject 901 reflects the projected light. Then, the light reflected by the subject 901 passes through the imaging lens 1040 and enters the solid-state imaging device 100. The EVS (corresponding to the second sensor 12) in the hybrid sensor 1022 of the solid-state imaging device 100 receives the reflected light reflected by the subject 901 and generates event data (second output data D2). On the other hand, the image sensor (corresponding to the first sensor 11) in the hybrid sensor 1022 receives, for example, visible light in the incident light and generates image data (first output data D1).

[0202] The third output data D3 generated by the first signal processing device 1050 is provided to the second signal processing device 1100. As described above, the second signal processing device 1100 includes, for example, an information processing device (such as an application processor), and performs predetermined processing (such as deblurring correction, frame interpolation, image processing, recognition processing, and SLAM) based on the image data and event data input from the first signal processing device 1050.

[0203] 1.10 Configuration Example of Hybrid Sensor

[0204] Fig.26 1 is a schematic configuration example of a hybrid sensor according to the present embodiment and is a block diagram showing an example of a pixel separation configuration. Fig.26 As shown in, for example, a hybrid sensor 11A according to the present embodiment includes a pixel array unit 101, a vertical drive circuit 102A, a horizontal drive circuit 102B, an X arbitrator 104A, a Y arbitrator 104B, a luminance signal processing circuit 103A, an event signal processing circuit 103B, a system control circuit 105, a luminance data processing unit 108A, and an event data processing unit 108B.

[0205] The pixel array unit 101 has a configuration in which unit pixels 110 are arranged in the row direction and the column direction (i.e., in a two-dimensional lattice (also referred to as a matrix pattern), and each of the unit pixels 110 is a repeating unit in the pixel layout. The row direction (also referred to as the row direction) refers to the array direction of pixels in a pixel row (the lateral direction in the figure), and the column direction (also referred to as the column direction) refers to the array direction of pixels in a pixel column (the longitudinal direction in the figure). Note that the repeating unit here is different from the repeating unit in a color filter array such as a Bayer array or a quad-Bayer array, and may be a constituent unit repeated in an element design.

[0206] Each unit pixel 110 includes one or more brightness pixels 310 and one or more event pixels 320. In the present specification, the brightness pixel 310 and the event pixel 320 may be referred to as pixels, respectively. Although the details of the specific circuit configuration and the pixel structure of the unit pixel 110 will be described later, in the pixel separation configuration, the brightness pixel 310 and the event pixel 320 each include a separate photoelectric conversion unit. Specifically, the brightness pixel 310 includes a photoelectric conversion unit PD1 that generates and accumulates charge according to the brightness of the incident light, and generates a brightness signal of a voltage according to the brightness. On the other hand, the event pixel 320 includes a photoelectric conversion unit PD2 that generates a charge according to the brightness of the incident light, and in the case where a change in the brightness of the incident light is detected based on the photocurrent flowing from the photoelectric conversion unit PD2, a request for requesting to read from the event pixel is output, and event data indicating that an event has been detected is output according to the arbitration output by the arbitrator that receives the request.

[0207] In the pixel array unit 101, the pixel drive lines LD1 and LD2 are wired along the row direction of each pixel row, and the vertical signal lines VSL1 and VSL2 are wired along the column direction of each pixel column relative to the matrix pixel array. For example, the pixel drive line LD1 is connected to the brightness pixel 310 of each row, and the pixel drive line LD2 is connected to the event pixel 320 of each row. On the other hand, for example, the vertical signal line VSL1 is connected to the brightness pixel 310 of each column, and the vertical signal line VSL2 is connected to the event pixel 320 of each column. However, the present invention is not limited to this, and the pixel drive lines LD1 and LD2 may be wired to be orthogonal to each other. Similarly, the vertical signal lines VSL1 and VSL2 may be wired to be orthogonal to each other. For example, the pixel drive line LD1 may be wired in the row direction, the pixel drive line LD2 may be wired in the column direction, the vertical signal line VSL1 may be wired in the column direction, and the vertical signal line VSL2 may be wired in the row direction.

[0208] The pixel drive line LD1 transmits a control signal for performing driving when reading a luminance signal from the luminance pixel 310. The pixel drive line LD2 transmits a control signal for causing the event pixel 320 to enter an active state in which an event can be detected. Fig.26 , each of the pixel drive lines LD1 and LD2 is shown as one wiring, but the number is not limited to one. One end of each of the pixel drive line LD1 and the pixel drive line L62 is connected to an output terminal corresponding to each row of the vertical drive circuit 102A.

[0209] (Drive configuration of brightness pixels)

[0210] Each brightness pixel 310 includes: a photoelectric conversion unit PD1 that photoelectrically converts incident light to generate electric charges; and a pixel circuit 111 that generates a brightness signal having a voltage value corresponding to the charge amount of the electric charges generated in the photoelectric conversion unit PD1, and causes the brightness signal to occur in the vertical signal line VSL1 under the control of the vertical drive circuit 102A.

[0211] The vertical driving circuit 102A includes a shift register, an address decoder, etc., and drives the luminance pixels 310 of the pixel array unit 101 for all pixels at the same time or in units of rows. That is, the vertical driving circuit 102A and the system control circuit 105 that controls the vertical driving circuit 102A constitute a driving unit that controls the operation of each luminance pixel 310 of the pixel array unit 101. Although the specific configuration of the vertical driving circuit 102A is not shown, the vertical driving circuit generally includes two scanning systems, a reading scanning system and a scanning scanning system.

[0212] The read scanning system sequentially and selectively scans each pixel of the pixel array unit 101 in row units to read a signal from each pixel. The brightness signal read from each pixel is an analog signal. The scanning scanning system performs a scanning scan on the read row before the read scanning exposure time, and the read scanning system performs a read scan on the read row.

[0213] By the scanning scanning of the scanning scanning system, unnecessary charges are cleared from the photoelectric conversion unit PD1 of each brightness pixel 310 in the read row, whereby the photoelectric conversion unit PD1 is reset. Then, the unnecessary charges are cleared (reset) by the clear scanning system, and the so-called electronic shutter operation is performed. Here, the electronic shutter operation refers to an operation of discarding the charges of the photoelectric conversion unit PD1 and newly starting exposure (starting accumulation of charges).

[0214] The signal read by the reading scanning system through the reading operation corresponds to the amount of light received after the immediately preceding reading operation or the electronic shutter operation. Then, the time period from the reading timing by the immediately preceding reading operation or the scanning time by the electronic shutter operation to the reading timing by the current reading operation is the charge accumulation time period (also called the exposure time period) of each pixel.

[0215] The luminance signal output from each luminance pixel 310 of the pixel row selectively scanned by the vertical drive circuit 102A is input to the luminance signal processing circuit 103A through each vertical signal line VSL1 of each pixel column. The luminance signal processing circuit 103A performs predetermined signal processing on the luminance signal output from each luminance pixel 310 of the selected row through the vertical signal line VSL1 for each pixel column of the pixel array unit 101, and temporarily holds the luminance signal after the signal processing.

[0216] Specifically, as signal processing, the luminance signal processing circuit 103A performs at least noise removal processing, such as correlated double sampling (CDS) processing or double data sampling (DDS) processing. For example, fixed pattern noise unique to a pixel, such as reset noise and threshold variation of an amplifier transistor in a pixel, is removed by CDS processing. The luminance signal processing circuit 103A also has, for example, an analog-to-digital (AD) conversion function, and converts an analog luminance signal read from the photoelectric conversion unit PD1 into a digital signal to output the digital signal.

[0217] The horizontal drive circuit 102B includes a shift register, an address decoder, etc., and sequentially selects the pixel circuit 111 corresponding to the pixel column of the luminance signal processing circuit 103A. Through the selective scanning of the horizontal drive circuit 102B, the luminance signal subjected to the signal processing of each pixel circuit 111 by the luminance signal processing circuit 103A is sequentially output.

[0218] (Driver configuration of event pixels)

[0219] Each event pixel 320 includes: a photoelectric conversion unit PD2 that generates charge according to the brightness of incident light; and an address event detection circuit 210 that detects the presence or absence of an event based on a change in the current value of the photocurrent flowing out of the photoelectric conversion unit PD2, and detects the presence or absence of an event based on whether a change exceeding a predetermined threshold value occurs in the photocurrent according to the brightness of the incident light. For example, each event pixel 320 detects a change in brightness exceeding or falling below a predetermined threshold value as an event.

[0220] When an event is detected, the event pixel 320 outputs a request to each of the X arbiter 104A and the Y arbiter 104B for allowing output of event data indicating the occurrence of the event. Then, when a response indicating that output of the event data is allowed is received from each of the X arbiter 104A and the Y arbiter 104B, the event pixel 320 outputs the event data to the event signal processing circuit 103B.

[0221] The event signal processing circuit 103B performs predetermined signal processing on the event data input from the event pixel 320 to output the event data after the signal processing.

[0222] As described above, the change in the photocurrent generated by the event pixel 320 can also be considered as a change in the amount of light (a change in brightness) incident on the photoelectric conversion unit PD2 of the event pixel 320. Therefore, it can also be said that an event is a change in the amount of light (a change in brightness) of the event pixel 320 that exceeds a predetermined threshold. The event data indicating the occurrence of an event includes at least position information, for example, coordinates indicating the position of the event pixel 320 where the change in the amount of light as an event occurs. In addition to the position information, the event data may also include the polarity of the change in the amount of light.

[0223] For a series of event data output from the event pixel 320 when an event occurs, as long as the intervals between multiple pieces of event data are maintained when the event occurs, it can be said that the event data implicitly includes time information indicating the relative time when the event occurs.

[0224] However, when the intervals between multiple pieces of event data are not maintained when an event occurs when the event data is stored in a memory, etc., the time information implicitly included in the event data is lost. Therefore, before the intervals between the event data are not maintained as when the event occurs, the event signal processing circuit 103B may include time information indicating the relative time when the event occurs in the event data, such as a timestamp, etc.

[0225] (Other configurations)

[0226] The system control circuit 105 includes a timing generator that generates various timing signals and the like, and drives and controls the vertical drive circuit 102A, the horizontal drive circuit 102B, the X arbitrator 104A, the Y arbitrator 104B, the luminance signal processing circuit 103A, the event signal processing circuit 103B, and the like based on the various timings generated by the timing generator.

[0227] Each of the brightness data processing unit 108A and the event data processing unit 108B has at least an arithmetic processing function, and performs various signal processing such as arithmetic processing on the image data (image frame data and event frame data) output from the brightness signal processing circuit 103A or the event signal processing circuit 103B.

[0228] The image data output from the brightness data processing unit 108A or the event data processing unit 108B is output to, for example, the FPGA 13 arranged on the same chip or an external chip.

[0229] It should be noted that the hybrid sensor 11A may include a storage unit for temporarily holding data required for signal processing by the brightness data processing unit 108A and the event data processing unit 108B, data processed by any one or more of the brightness signal processing circuit 103A, the event signal processing circuit 103B, the brightness data processing unit 108A and the event data processing unit 108B, and the like.

[0230] 1.10.1 Another configuration example of hybrid sensor

[0231] Fig. 27 is a block diagram showing another schematic configuration example of a hybrid sensor according to the present embodiment and an example of a pixel sharing configuration. Fig.26 and Fig. 27 It can be seen that in the pixel sharing configuration, one or more brightness pixels 310 and one or more event pixels 320 in the unit pixel 110 share one photoelectric conversion unit PD. In other words, the pixel circuit 111 and the brightness signal processing circuit 103A corresponding to the first signal processing unit for acquiring the brightness signal and the address event detection circuit 210 and the event signal processing circuit 103B corresponding to the second signal processing unit for acquiring the event signal are connected to the shared photoelectric conversion unit PD. Other configurations and operations may be similar to those according to reference Fig.26 The configuration and operation of the hybrid sensor 11A of the pixel separation configuration are described, and therefore a detailed description thereof is omitted here.

[0232] 1.10.2 Example of a hybrid sensor pixel array

[0233] Fig.28 is a schematic diagram showing an example of a pixel array of a hybrid sensor according to this embodiment. Fig.28 As shown in Fig.26 In the pixel separation configuration shown in , for example, one of four pixels constituting the array pattern 110A as a repetition unit of the Bayer array may be constituted by the event pixel 320, and the remaining three may be constituted by the luminance pixels 310r, 310b, and 310g.

[0234] 1.10.3 Another Pixel Array Example of a Hybrid Sensor

[0235] Fig.29 FIG. 2 is a schematic diagram showing another example of a pixel array of a hybrid sensor according to the present embodiment. Fig.29 As shown in Fig. 27In the pixel sharing configuration shown in , each of the four pixels making up the array pattern 110A (a repeating unit of the Bayer array) may include a shared pixel of a shared brightness pixel 310 r , 310 g , or 310 b and an event pixel 320 .

[0236] However, in the pixel sharing configuration, not all pixels are necessarily shared pixels of the luma pixels 310 and the event pixels 320 , and some (or more than two) of the plurality of pixels constituting the array pattern 110A may be shared pixels.

[0237] As described above, the brightness pixels 310 constituting the image sensor and the event pixels 320 constituting the EVS are uniformly mixed and arranged in the pixel array unit 101 so that the optical axes of the image sensor and the EVS can coincide with each other or substantially coincide with each other. Therefore, even in the case where a sensor for eye tracking, SLAM, etc., for example, is switched between the image sensor and the EVS, processing such as geometric transformation of the coordinate system for matching the image data acquired by each sensor can be omitted or simplified, and therefore, suppression of an increase in processing speed can be prevented.

[0238] In addition, by using a configuration in which brightness pixels 310 and event pixels 320 are mixed in the pixel array unit 101, as in the case of configuring an image sensor and an EVS using different chips, power waste caused by unused sensors continuously entering a conductive state can be suppressed, and thus, an increase in power consumption can be suppressed.

[0239] 1.11 Example of stacked structure of solid-state imaging device

[0240] Fig.30 : is a diagram showing an exemplary stacking structure of a solid-state imaging device according to the present embodiment. Fig.30 As shown, the solid-state imaging device 10 may have a structure in which a pixel chip 140 and a circuit chip 150 are vertically stacked. The pixel chip 140 is, for example, a semiconductor chip including a photoelectric conversion unit of each of the luminance pixel 310 and the event pixel 320 and a part of the circuit configuration in the luminance pixel 310 and the event pixel 320, and the circuit chip 150 is a semiconductor chip including a configuration other than the configuration arranged in the pixel chip 140.

[0241] For the bonding of the pixel chip 140 and the circuit chip 150, for example, so-called direct bonding can be used, in which the bonding surfaces are flat and bonded to each other by electronic force. However, the present invention is not limited thereto, and for example, so-called Cu-Cu bonding, bump bonding, etc., in which copper (Cu) electrode pads formed on the bonding surfaces are bonded to each other, can also be used.

[0242] In addition, for example, the pixel chip 140 and the circuit chip 150 are electrically connected via a connection portion such as a through contact via (TCV) that penetrates the semiconductor substrate. For connection using TCV, for example, a so-called double TCV method (in which two TCVs of a TCV provided in the pixel chip 140 and a TCV provided from the pixel chip 140 to the circuit chip 150 are connected on the outer surface of the chip), a so-called shared TCV method (in which both are connected via a TCV that penetrates from the pixel chip 140 to the circuit chip 150), etc. can be used.

[0243] However, when Cu-Cu bonding or bump bonding is used to bond the pixel chip 140 and the circuit chip 150 , both may be electrically connected via the Cu-Cu bonding portion or the bump bonding portion.

[0244] 2. Application examples for mobile objects

[0245] The technology according to the present disclosure can be applied to various products. For example, the technology according to the present disclosure can be further applied to a device installed on any one of various mobile bodies (such as, automobiles, electric vehicles, hybrid electric vehicles, motorcycles, bicycles, personal mobility, airplanes, drones, ships, and robots).

[0246] Fig.31 : is a block diagram showing an example of a schematic configuration of a vehicle control system as an example of a mobile object control system to which the technology according to the embodiment of the present disclosure can be applied.

[0247] The vehicle control system 12000 includes a plurality of electronic control units connected to each other via a communication network 12001. Fig.31 In the example shown, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an external information detection unit 12030, an internal information detection unit 12040, and an integrated control unit 12050. In addition, as a functional structure of the integrated control unit 12050, a microcomputer 12051, a sound and image output unit 12052, and an in-vehicle network interface (I / F) 12053 are shown.

[0248] The drive system control unit 12010 controls the operation of devices related to the drive system of the vehicle according to various programs. For example, the drive system control unit 12010 serves as a control device for a drive force generating device (such as an internal combustion engine, a drive motor, etc.) for generating a drive force of the vehicle, a drive force transmitting mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, a braking device for generating a braking force of the vehicle, and the like.

[0249] The body system control unit 12020 controls the operation of various devices provided on the vehicle body according to various programs. For example, the body system control unit 12020 is used as a control device for a keyless entry system, a smart key system, a power window device, or various lights such as a headlight, a backup light, a brake light, a turn signal, a fog light, etc. In this case, a radio wave transmitted from a mobile device as a substitute for a key or a signal of various switches may be input to the body system control unit 12020. The body system control unit 12020 receives these input radio waves or signals, and controls the door lock device, the power window device, the lights, etc. of the vehicle.

[0250] The vehicle exterior information detection unit 12030 detects information outside the vehicle including the vehicle control system 12000. For example, the vehicle exterior information detection unit 12030 is connected to the imaging unit 12031. The vehicle exterior information detection unit 12030 causes the imaging unit 12031 to capture an image outside the vehicle and receive the captured image. In addition, the vehicle exterior information detection unit 12030 may also detect objects such as people, vehicles, obstacles, signs, and text on the road surface, and detect their distances based on the received image.

[0251] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. The imaging unit 12031 can output the electrical signal as an image, or can output the electrical signal as information related to the measured distance. In addition, the light received by the imaging unit 12031 can be visible light or invisible light such as infrared light.

[0252] The in-vehicle information detection unit 12040 detects information related to the interior of the vehicle. The in-vehicle information detection unit 12040 is connected to, for example, a driver state detection unit 12041 that detects the state of the driver. The driver state detection unit 12041 has, for example, a camera that photographs the driver. Based on the detection information input from the driver state detection unit 12041, the in-vehicle information detection unit 12040 can calculate the driver's fatigue or the driver's concentration, or can determine whether the driver is dozing off.

[0253] The microcomputer 12051 can calculate the control target value of the driving force generation device, the steering mechanism, or the braking device based on the information about the inside or outside of the vehicle acquired by the vehicle exterior information detection unit 12030 or the vehicle interior information detection unit 12040, and output a control command to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an advanced driver assistance system (ADAS), including collision prevention or shock absorption of the vehicle after driving based on vehicle-to-vehicle distance, vehicle speed maintenance, vehicle collision warning, vehicle lane departure warning, etc.

[0254] In addition, the microcomputer 12051 can perform collaborative control for automatic driving by controlling the driving force generating device, steering mechanism, braking device, etc. based on the information about the outside or inside of the vehicle obtained by the outside information detection unit 12030 or the inside information detection unit 12040, thereby enabling the vehicle to travel automatically without relying on the driver's operation, etc.

[0255] In addition, the microcomputer 12051 can output a control instruction to the body system control unit 12020 based on the vehicle exterior information acquired by the vehicle exterior information detection unit 12030. For example, the microcomputer 12051 can perform cooperative control for preventing glare by controlling the headlights to change from high beam to low beam based on the position of the front vehicle or the oncoming vehicle detected by the vehicle exterior information detection unit 12030.

[0256] The sound and image output unit 12052 transmits an output signal of at least one of sound and image to an output device, which can visually or auditorily notify the occupants of the vehicle or the outside of the vehicle of information. Fig.31 In the example of FIG. 1 , an audio speaker 12061, a display unit 12062, and a dashboard 12063 are shown as output devices. The display unit 12062 may include, for example, at least one of an on-board display and a head-up display.

[0257] Fig.32 is a diagram showing an example of the setting position of the imaging unit 12031.

[0258] exist Fig.32 In the figure, imaging unit 12031 includes imaging units 12101, 12102, 12103, 12104 and 12105.

[0259] Imaging units 12101, 12102, 12103, 12104, and 12105 are arranged, for example, at positions on the front nose, side mirrors, rear bumper, and rear door of the vehicle 12100, and at positions on the upper portion of the windshield inside the vehicle. The imaging unit 12101 arranged at the front nose in the interior of the vehicle and the imaging unit 12105 arranged at the upper portion of the windshield mainly acquire images in front of the vehicle 12100. The imaging units 12102 and 12103 arranged at the side mirrors mainly acquire images on the sides of the vehicle 12100. The imaging unit 12104 arranged to the rear bumper or rear door mainly acquires images of the rear of the vehicle 12100. The imaging unit 12105 arranged at the upper portion of the windshield inside the vehicle is mainly used to detect vehicles ahead, pedestrians, obstacles, signals, traffic signs, lanes, etc.

[0260] By the way, Fig.32 An example of the imaging ranges of the imaging units 12101 to 12104 is described. The imaging range 12111 indicates the imaging range of the imaging unit 12101 set to the front nose. The imaging ranges 12112 and 12113 respectively indicate the imaging ranges of the imaging units 12102 and 12103 set to the side mirrors. The imaging range 12114 indicates the imaging range of the imaging unit 12104 set to the rear bumper or the rear door. For example, by overlapping the image data captured by the imaging units 12101 to 12104, an overhead image of the vehicle 12100 when viewed from above is obtained.

[0261] At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera composed of a plurality of imaging elements, or may be an imaging element having pixels for phase difference detection.

[0262] For example, the microcomputer 12051 can determine the distance to each three-dimensional object within the imaging range 12111 to 12114 and the time change of the distance (relative speed relative to the vehicle 12100) based on the distance information obtained from the imaging units 12101 to 12104, thereby extracting the objects existing on the driving path of the vehicle 12100 and traveling in the same direction as the vehicle 12100 at a prescribed speed (e.g., equal to or greater than 0 km / hour). In addition, the microcomputer 12051 can pre-set the inter-vehicle distance to maintain in front of the leading vehicle, and perform automatic braking control (including follow-up stop control), automatic acceleration control (including follow-up start control), etc. Thus, coordinated control for automatic driving that allows the vehicle to travel automatically without relying on the driver's operation, etc. can be performed.

[0263] For example, the microcomputer 12051 can classify the three-dimensional object data of the three-dimensional object into three-dimensional object data of three-dimensional objects such as two-wheeled vehicles, standard vehicles, large vehicles, pedestrians, and electric poles based on the distance information obtained from the imaging units 12101 to 12104, extract the classified three-dimensional object data, and use the extracted three-dimensional object data for automatic obstacle avoidance. For example, the microcomputer 12051 identifies obstacles around the vehicle 12100 as obstacles that can be visually recognized by the driver of the vehicle 12100 and obstacles that are difficult for the driver of the vehicle 12100 to visually recognize. Then, the microcomputer 12051 determines a collision risk indicating a risk level of collision with each obstacle. In the case where the collision risk is equal to or higher than the set value and therefore there is a possibility of collision, the microcomputer 12051 outputs a warning to the driver via the audio speaker 12061 or the display unit 12062, and performs forced deceleration or evasive steering via the drive system control unit 12010, thereby assisting driving for collision avoidance.

[0264] At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. The microcomputer 12051 may identify a pedestrian by, for example, determining whether a pedestrian exists in the captured images of the imaging units 12101 to 12104. Such recognition of pedestrians is performed, for example, by a process of extracting feature points in the captured images of the imaging units 12101 to 12104 as infrared cameras and a process of determining whether it is a pedestrian by performing pattern matching processing on a series of feature points representing the outline of the subject. When the microcomputer 12051 determines that a pedestrian exists in the captured images of the imaging units 12101 to 12104 and thus identifies the pedestrian, the sound image output unit 12052 controls the display unit 12062 so that a square outline for emphasis is superimposed on the identified pedestrian. The sound image output unit 12052 may also control the display unit 12062 so that an icon representing a pedestrian or the like is displayed at a desired position.

[0265] An example of a vehicle control system to which the technology according to the present disclosure can be applied is described above. For example, the technology according to the present disclosure can be applied to the driver state detection unit 12041 and the imaging unit 12031 in the above configuration. By applying the technology according to the present disclosure to the driver state detection unit 12041, eye tracking of the driver or a fellow passenger can be performed quickly and accurately. In addition, by applying the technology according to the present disclosure to the imaging unit 12031, the driving position and posture of the vehicle 12100 can be quickly and accurately acquired.

[0266] The embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above embodiments, and various changes can be made without departing from the gist of the present disclosure. In addition, components on different embodiments and modifications can be appropriately combined.

[0267] Furthermore, the effects in each embodiment described in this specification are merely examples and are not limited, and other effects may exist.

[0268] The present technology may also be configured as follows. (1)

[0270] An information processing device, comprising:

[0271] a first receiving unit that receives first data generated by the first sensor and second data generated by the second sensor in synchronization with the first reference signal; and

[0272] The transmission unit transmits at least a portion of the first data and the second data as third data synchronized with the second reference signal. (2)

[0274] The information processing device according to (1), wherein:

[0275] The second sensor is an event-based vision sensor (EVS) including one or more pixels that detect changes in brightness of incident light as events. (3)

[0277] The information processing device according to (2), wherein:

[0278] The EVS generates second data without being synchronized with the first reference signal. (4)

[0280] The information processing device according to (2) or (3), wherein:

[0281] The second data includes second information indicating whether the event is an event detected during an exposure period of the first sensor. (5)

[0283] The information processing device according to (4), wherein:

[0284] The transmission unit transmits fourth data selected from the second data based on the second information and the first data as third data. (6)

[0286] The information processing device according to (5), wherein:

[0287] The fourth data is data generated based on events detected during the exposure period. (7)

[0289] The information processing device according to (5), wherein:

[0290] The fourth data is data generated based on an event detected outside the exposure period. (8)

[0292] The information processing device according to any one of (4) to (7), further comprising:

[0293] A control section that receives a request for data generated based on an event detected during an exposure period or data generated based on an event detected outside of the exposure period, wherein

[0294] The transmission unit transmits third data including the data requested in the request in the second data. (9)

[0296] The information processing device according to (4), wherein:

[0297] The transmission unit distinguishes data generated based on an event detected during an exposure period and data generated based on an event detected outside the exposure period in the second data based on the second information, and transmits third data including the distinguished two pieces of data. (10)

[0299] The information processing device according to any one of (1) to (9), wherein:

[0300] The first sensor is an image sensor including one or more pixels generating a grayscale signal according to the light amount of incident light. (11)

[0302] The information processing device according to any one of (1) to (9), wherein:

[0303] The first sensor is a distance measuring sensor including one or more pixels that generate first information about a distance to the object based on a flight time of light reflected by the object. (12)

[0305] The information processing device according to any one of (1) to (11), further comprising:

[0306] The processing unit generates third data by framing at least a part of the first data and the second data. (13)

[0308] The information processing device according to any one of (1) to (12), comprising:

[0309] a second receiving unit, receiving third data transmitted from the transmitting unit; and

[0310] A signal processing unit performs predetermined processing on the third data received by the second receiving unit. (14)

[0312] The information processing device according to (13), wherein:

[0313] The first data is image data, and

[0314] The predetermined processing is deblurring correction for correcting blur occurring in the first data. (15)

[0316] The information processing device according to (13), wherein:

[0317] The first data is image data or depth information, and

[0318] The predetermined processing is frame interpolation for interpolating frames between first data generated in synchronization with the first reference signal. (16)

[0320] The information processing device according to any one of (1) to (15), further comprising:

[0321] At least one of the first sensor and the second sensor. (17)

[0323] An information processing method, comprising:

[0324] receiving first data generated by a first sensor synchronized with a first reference signal and second data generated by a second sensor; and

[0325] The step of transmitting at least a portion of the first data and the second data as third data synchronized with the second reference signal. (18)

[0327] An information processing device, comprising:

[0328] a processing unit that generates third data including first data generated by the first sensor in synchronization with the first reference signal and at least a portion of second data generated by the second sensor; and

[0329] A transmission unit transmits at least one of the third data or a processing result generated by performing at least a part of a predetermined process on the third data. (19)

[0331] An information processing device, comprising:

[0332] a calling unit, which acquires third data, where the third data includes at least a portion of first data generated by the first sensor in synchronization with the first reference signal and second data generated by the second sensor; and

[0333] The signal processing unit performs at least a part of predetermined processing on the third data acquired by the calling unit.

[0334] Reference Numbers List

[0335] 1. 1A to 1D, 2. 3 Imaging device

[0336] 10.100 Solid-state imaging device

[0337] 11. First sensor (image sensor, TOF sensor)

[0338] 11a, 1022 hybrid sensor

[0339] 12 Second sensor (EVS)

[0340] 13. 1050 first signal processing device (FPGA)

[0341] 14. 1100 second signal processing device (application processor)

[0342] 15 Control Unit

[0343] 16 Light projection unit

[0344] 17 Lighting Unit

[0345] 18. 1010 Light Source

[0346] 19External I / F

[0347] 140 Pixel Chip

[0348] 150 Circuit Chip

[0349] 401, 501 receiving unit

[0350] 402 Synchronous Processing Unit

[0351] 403 Transmission Unit

[0352] 404, 507 Communication Control Unit

[0353] 411, 415, 502, 506 API processing unit

[0354] 412, 414, 503, 505 API call unit

[0355] 413, 504 signal processing unit

[0356] 1011 Light source driver unit

[0357] 1012VCSEL

[0358] 1030 Illumination lens

[0359] 1040 Imaging Lens

Claims

1. An information processing device, comprising: A first receiving unit receives first data generated by the first sensor in synchronization with the first reference signal and second data generated by the second sensor; as well as A transmission unit transmits at least a portion of the first data and the second data as third data synchronized with a second reference signal.

2. The information processing device according to claim 1, wherein: The second sensor is an event-based vision sensor EVS including one or more pixels, which detects brightness changes of incident light as events.

3. The information processing device according to claim 2, wherein: The EVS generates the second data without synchronization with the first reference signal.

4. The information processing device according to claim 2, wherein: The second data includes second information indicating whether the event is an event detected during an exposure period of the first sensor.

5. The information processing device according to claim 4, wherein: The transmission unit transmits fourth data selected from the second data based on the second information and the first data as the third data.

6. The information processing device according to claim 5, wherein: The fourth data is data generated based on an event detected during the exposure period.

7. The information processing device according to claim 5, wherein: The fourth data is data generated based on an event detected outside the exposure period.

8. The information processing device according to claim 4, further comprising: a control unit that receives a request for data generated based on an event detected during the exposure period or data generated based on an event detected outside the exposure period, wherein: The transmission unit transmits the third data including the data requested in the request among the second data.

9. The information processing device according to claim 4, wherein: The transmission unit distinguishes data generated based on an event detected during the exposure period and data generated based on an event detected outside the exposure period in the second data based on the second information, and transmits the third data including the distinguished two pieces of data.

10. The information processing device according to claim 1, wherein: The first sensor is an image sensor including one or more pixels generating a grayscale signal according to a light amount of incident light.

11. The information processing device according to claim 1, wherein: The first sensor is a distance measuring sensor including one or more pixels, and the distance measuring sensor generates first information about a distance to a subject based on a flight time of light reflected by the subject.

12. The information processing device according to claim 1, further comprising: The processing unit generates third data by framing at least a part of the first data and the second data.

13. The information processing device according to claim 1, comprising: a second receiving unit, receiving the third data transmitted from the transmitting unit; as well as A signal processing unit performs predetermined processing on the third data received by the second receiving unit.

14. The information processing device according to claim 13, wherein: The first data is image data, and The predetermined processing is deblurring correction for correcting blur occurring in the first data.

15. The information processing device according to claim 13, wherein: The first data is image data or depth information, and The predetermined processing is frame interpolation for interpolating frames between the first data generated in synchronization with the first reference signal.

16. The information processing device according to claim 1, further comprising: At least one of the first sensor and the second sensor.

17. An information processing method, comprising: The step of receiving first data generated by a first sensor in synchronization with a first reference signal and second data generated by a second sensor; as well as The step of transmitting at least a portion of the first data and the second data as third data synchronized with a second reference signal.

18. An information processing device, comprising: a processing unit that generates third data including first data generated by the first sensor in synchronization with the first reference signal and at least a portion of second data generated by the second sensor; as well as A transmission unit transmits at least one of the third data or a processing result generated by performing at least a part of a predetermined process on the third data.

19. An information processing device, comprising: a calling unit, which acquires third data, wherein the third data includes at least a portion of first data generated by the first sensor in synchronization with the first reference signal and second data generated by the second sensor; and A signal processing unit performs at least a part of a predetermined process on the third data acquired by the calling unit.

Images