Light detection device, light detection method, and program

The photodetection device corrects latency in event signal detection sensors by calculating light reception and adjusting time information, addressing latency issues in event data accuracy.

JP2024163483AInactive Publication Date: 2024-11-22SONY SEMICON SOLUTIONS CORP
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023079115
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-05-12
Publication Date
2024-11-22
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Event signal detection sensors experience latency due to transfer rate constraints and circuit response delays, which cannot be effectively reduced by existing methods.

Method used

A photodetection device and method that corrects latency in event data by calculating the amount of light received by event detection pixels and adjusting time information based on this amount, using visible light sensors or CMOS image sensors to determine the appropriate latency correction for each pixel.

Benefits of technology

The method effectively reduces latency in event signal detection by correcting time information in event data, ensuring accurate timing of event detection regardless of light intensity variations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024163483000001_ABST
    Figure 2024163483000001_ABST
Patent Text Reader

Abstract

To make it possible to reduce latency caused by an event signal detection sensor.SOLUTION: An amount of light received by each of a plurality of event detection pixels, which are arranged in a two-dimensional plane and generate an electric signal by performing photoelectric conversion, is calculated on the basis of a gradation signal detected by a CMOS image sensor disposed in the vicinity of an event signal detection sensor which outputs event data indicating an occurrence of an event, i.e. a change in the electric signal, detected by the event detection pixel. Information related to event data latency is corrected on the basis of the amount of light received by the event detection pixel. The present invention can be applied to a Hybrid EVS sensor.SELECTED DRAWING: Figure 10
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to a light detection device, a light detection method, and a program, and in particular to a light detection device, a light detection method, and a program that enable reduction in latency caused by an event signal detection sensor. [Background technology]

[0002] An event signal detection sensor has been proposed that detects an event from the result of comparing a change in the amount of light received by a pixel per unit time with a predetermined threshold value, and outputs the event data (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-136958 A Summary of the Invention [Problem to be solved by the invention]

[0004] Incidentally, it is known that event signal detection sensors suffer from a latency problem, where the time of occurrence of an event recorded in the extracted event data is delayed from the time the actual event occurred, due to "transfer rate constraints" and "circuit response delays that depend on the amount of light received by a pixel per unit time."

[0005] Among the types of latency, the latency caused by restrictions on the transfer rate can be reduced by reducing the amount of data to be transferred, but the reality is that other types of latency cannot be reduced.

[0006] The present disclosure has been made in view of these circumstances, and in particular, aims to reduce latency in an event signal detection sensor. [Means for solving the problem]

[0007] A photodetection device and a program according to one aspect of the present disclosure include an event detection unit that outputs event data representing the occurrence of an event, the event data being a change in an electrical signal detected by event detection pixels that are arranged in a two-dimensional plane and perform photoelectric conversion to generate an electrical signal, a received light amount calculation unit that calculates the amount of light received by the event detection pixels, and a latency correction unit that corrects information relating to latency in the event data based on the amount of light received by the event detection pixels calculated by the received light amount calculation unit.

[0008] An optical detection method according to one aspect of the present disclosure is an optical detection method including the steps of: outputting event data representing the occurrence of an event, the event data being a change in an electrical signal detected by event detection pixels arranged in a two-dimensional plane and performing photoelectric conversion to generate an electrical signal; calculating an amount of light received by the event detection pixels; and correcting information relating to latency in the event data based on the calculated amount of light received by the event detection pixels.

[0009] In one aspect of the present disclosure, event detection pixels are arranged in a two-dimensional plane and perform photoelectric conversion to generate an electrical signal. Event data representing the occurrence of an event, which is a change in the electrical signal, is detected by the event detection pixels, the event detection pixels perform photoelectric conversion to generate an electrical signal, and the event data is output. The amount of light received by the event detection pixels is calculated, and information relating to latency in the event data is corrected based on the calculated amount of light received by the event detection pixels. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a signal processing device having an event signal detection sensor. [Diagram 2] 2 is a diagram illustrating an example of the configuration of a pixel circuit of the event signal detection sensor of FIG. 1. [Diagram 3] FIG. 13 is a diagram illustrating the detection of an event. [Figure 4] FIG. 1 is a diagram for explaining the principle of latency generation. [Diagram 5] FIG. 11 is a diagram illustrating latency that changes depending on the amount of received light. [Figure 6] 11A and 11B are diagrams illustrating a comparison between latency in an image with a large amount of received light and latency in an image with a small amount of received light. [Figure 7] FIG. 1 is a diagram illustrating an example of a configuration of a signal processing device according to a first embodiment of the present disclosure. [Figure 8] 10 is a diagram illustrating a positional relationship between a visible light sensor and the optical axis of a lens in front of the event signal detection sensor. FIG. [Figure 9] FIG. 13 is a diagram illustrating a table of received light amounts and corresponding latencies. [Figure 10] 8 is a flowchart illustrating signal processing by the signal processing device of FIG. 7. [Figure 11] FIG. 11 is a diagram illustrating a configuration example of a signal processing device according to a second embodiment of the present disclosure. [Figure 12] 1 is a diagram illustrating the positional relationship of the optical axes between a collecting lens in front of a CMOS image sensor and a collecting lens in front of an event signal detection sensor; [Figure 13] 10 is a diagram for explaining an example of applying the amount of light received by a CMOS image sensor to the amount of light received by a pixel on a corresponding event signal detection sensor; [Figure 14] 1A and 1B are diagrams illustrating an example of estimating a spectrum based on a ratio of the amount of received light of RGB pixels of a CMOS image sensor. [Figure 15] 1A and 1B are diagrams illustrating original event data and event data in which a time stamp, which is time information, has been corrected according to a latency correction amount. [Figure 16] 11A and 11B are diagrams illustrating original event data and event data to which time information corrected according to a latency correction amount has been added; [Figure 17] 12 is a flowchart illustrating signal processing by the signal processing device of FIG. [Figure 18] 10A and 10B are diagrams illustrating original event data and event data to which information on the amount of received light has been added; [Figure 19]FIG. 13 is a diagram illustrating a configuration example of a signal processing device according to a third embodiment of the present disclosure. [Figure 20] 20 is a flowchart illustrating signal processing by the signal processing device of FIG. 19. [Figure 21] FIG. 1 is a diagram illustrating a CMOS image sensor and a HybridEVS sensor. [Figure 22] 1 is a diagram illustrating the relationship between a HybridEVS sensor and the optical axis of a collecting lens. [Figure 23] 13 is a diagram illustrating a first configuration example of a signal processing device using a HybridEVS sensor according to a fourth embodiment of the present disclosure. FIG. [Figure 24] FIG. 13 is a diagram illustrating a second configuration example of a signal processing device using a HybridEVS sensor according to the fourth embodiment of the present disclosure. [Diagram 25] FIG. 13 is a diagram illustrating a third configuration example of a signal processing device using a HybridEVS sensor according to the fourth embodiment of the present disclosure. [Figure 26] FIG. 13 is a diagram illustrating a configuration example of a signal processing device according to a fifth embodiment of the present disclosure. [Figure 27] 11A and 11B are diagrams illustrating an example of shifting an event data group according to latency in an application example of the present disclosure. [Figure 28] 11 is a flowchart illustrating signal processing for implementing a modified example of the present disclosure. [Figure 29] FIG. 11 is a diagram illustrating a first modified example of an example of shifting an event data group according to latency. [Diagram 30] FIG. 11 is a diagram illustrating a second modified example of an example of shifting an event data group according to latency. [Diagram 31] 1 is a block diagram showing an example of a schematic configuration of a vehicle control system; [Diagram 32] 4 is an explanatory diagram showing an example of the installation positions of an outside-vehicle information detection unit and an imaging unit; FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.

[0012] Hereinafter, an embodiment of the present technology will be described in the following order.

[0013] 1. Overview of this Disclosure 2. First embodiment 3. Second embodiment 4. Third embodiment 5. Fourth embodiment 6. Fifth embodiment 7. Application Examples 8. Mobile Applications <<1. Overview of this Disclosure>> <Event signal detection sensor> The present disclosure, among other things, reduces latency in event signal detection sensors.

[0014] First, the event signal detection sensor will be described. Fig. 1 shows an example of the configuration of a signal processing device including an event signal detection sensor.

[0015] The signal processing device 1 in FIG. 1 includes an event signal detection sensor 11 and a subsequent application 12.

[0016] The event signal detection sensor 11 is configured to be a so-called pixel array in which a plurality of pixel circuits 21, including pixels 31 that perform photoelectric conversion of incident light to generate electric signals, are arranged in a lattice pattern on a two-dimensional plane. The event signal detection sensor 11 performs photoelectric conversion of incident light in the pixels 31 to generate electric signals, generates event data that indicates the occurrence of an event, which is a change in the electric signals, and outputs the event data to a downstream application 12.

[0017] The subsequent application 12 is an application program that executes various processes based on the event data supplied from the event signal detection sensor 11 .

[0018] Here, in the event signal detection sensor 11, the portion in which the plurality of pixel circuits 21 are arranged is a portion that receives incident light and performs photoelectric conversion as a whole, and is therefore also referred to as a light receiving portion.

[0019] <Pixel circuit configuration example> FIG. 2 is a block diagram showing an example of the configuration of the pixel circuit 21 in FIG.

[0020] The pixel circuit 21 includes a pixel 31 and an event detection unit 32 .

[0021] The pixel 31 includes a photodiode (PD) 51 as a photoelectric conversion element. In the pixel 31, the PD 51 receives light incident on the PD 51, performs photoelectric conversion, and generates and outputs a photocurrent (Iph) as an electric signal.

[0022] When a change exceeding a predetermined threshold (including a change equal to or greater than the threshold as necessary) occurs in the photocurrent generated by photoelectric conversion of the pixel 31, the event detection unit 32 detects the change in the photocurrent as an event. The event detection unit 32 outputs event data in response to (the detection of) an event.

[0023] Here, since the change in the photocurrent generated in pixel 31 can also be regarded as a change in the amount of light incident on pixel 31, the event can also be said to be a change in the amount of light in pixel 31 (a change in the amount of light exceeding a threshold).

[0024] The event data can at least specify position information (coordinates, etc.) that indicates the position of the pixel 31 (pixel circuit 21) where the light amount change occurred as an event. In addition, the event data includes information that specifies a positive event or a negative event according to the polarity (positive or negative) of the light amount change.

[0025] For a series of event data output by the event detection unit 32 at the timing of detecting an event, time information indicating the (relative) time when the event occurred can be identified as long as the interval between the event data is maintained as it was when the event occurred. However, when the interval between the event data is no longer maintained as it was when the event occurred due to the event data being stored in a memory or the like, the time information is lost. Therefore, time information indicating the (relative) time when the event occurred, such as a timestamp, is added to the event data before the interval between the event data is no longer maintained as it was when the event occurred. The process of adding the time information to the event data may be performed by the event detection unit 32 or may be performed outside the event detection unit 32, as long as it is performed before the interval between the event data is no longer maintained as it was when the event occurred.

[0026] The event detection unit 32 includes a current-voltage conversion unit 41 , a difference detection unit 42 , and a comparator 43 .

[0027] The current-voltage converter 41 converts the photocurrent from the pixel 31 into a voltage Vo (hereinafter also referred to as a photovoltage) corresponding to the logarithm of the photocurrent, and outputs the voltage Vo to the difference detector .

[0028] The difference detection unit 42 calculates the difference between the current optical voltage Vo from the current-voltage conversion unit 41 and an optical voltage at a timing that differs from the current by a small amount of time, and outputs the difference signal Vout to the comparator 43.

[0029] The comparator 43 compares the difference signal Vout output by the difference detection unit 42 with predetermined thresholds (voltages) +Vth and -Vth used for detecting an event. When the difference signal Vout is equal to or greater than the threshold +Vth or equal to or less than the threshold -Vth, the comparator 43 determines that a change in the amount of light as an event has been detected (occurred), and outputs event data.

[0030] For example, when the differential signal Vout is equal to or greater than the threshold +Vth, the comparator 43 determines that a positive event has been detected and outputs event data of a positive event. On the other hand, when the differential signal Vout is equal to or less than the threshold -Vth, the comparator 43 determines that a negative event has been detected and outputs event data of a negative event.

[0031] <Event signal detection sensor latency> As described above, the event signal detection sensor 11 detects a change in the amount of light received by the pixel 31 as an event, and outputs the pixel coordinates at that time, the time, and information indicating a positive event or a negative event as event data.

[0032] When the event signal detection sensor 11 outputs event data, if latency occurs with respect to the timing at which the event actually occurred, a delay occurs in the timing at which the event is detected, and the time information of the event data is a delayed time with respect to the time at which the event actually occurred.

[0033] For example, consider the case where, as shown in the left part of Figure 3, an area in which a white object exists against a gray background moves leftward from area WZ1 shown by the dotted line to area WZ2, as indicated by the arrow.

[0034] In this case, at the boundary BZ2 which is the left end of the white object region WZ2, as shown in the upper center of Figure 3, a positive event is detected as the amount of received light increases due to the change from gray to white.

[0035] On the other hand, at boundary BZ1, which is the right end of the white object region WZ2, a negative event is detected as the amount of received light decreases due to a change from white to gray, as shown in the upper right part of Figure 3.

[0036] A negative event detected at boundary BZ1 and a positive event detected at boundary BZ2 are detected at the same time, but due to delay (latency), information about a time delayed from the time the event actually occurred is recorded in the event data.

[0037] In other words, even if the positive event and negative event shown in the upper center and upper right corner of Figure 3 actually occurred at time t11, for example, time t12, which is delayed by a predetermined delay time latency TL1, is recorded as the time information of the event data.

[0038] As a result, if a white object moves from area WZ1, where a white object exists, as shown in the top row of Figure 4 at time T1, to area WZ2, as shown in the second row from the top of Figure 4 at time T2, a positive event will be detected in area Ze1, as shown in the third row from the top of Figure 4, when there is no latency and the time when the event actually occurs is appropriately reflected.

[0039] However, due to the effect of latency, time information that is later than the time when the actual event occurred is recorded, and the area in which the positive event is detected becomes a past area Ze2, which is shifted to the right in the figure by a distance Δd corresponding to the latency TL1 from the actual area Ze1, as shown in the bottom row of Figure 4.

[0040] For this reason, it is conceivable to correct the time information of the event data by subtracting the latency T1, which is a predetermined delay time.

[0041] However, it is known that the latency is shorter when the amount of received light is greater, and conversely, the latency is longer when the amount of received light is smaller. Therefore, subtracting the delay time due to latency from the time information as a specific delay time does not necessarily result in appropriate correction.

[0042] That is, for example, when a positive event occurs in a state where the amount of received light is greater than in the case of FIG. 3, as shown in the left part of FIG. 5, when the positive event actually occurs at time t31, the time information of the event data is set to time t32 which is delayed by latency TL11, which is shorter than latency TL1 relative to time t31.

[0043] In contrast, for example, when a positive event occurs in a state where the amount of received light is low compared to the case in Figure 3, as shown in the right part of Figure 5, when the positive event actually occurs at time t41, the time information of the event data is time t42, which is delayed from time t41 by a latency TL12 that is longer than the latency TL1.

[0044] Therefore, for example, as shown in the upper left part of Figure 6, the latency LL that occurs when a foreground region LG2 that is lighter gray than background region LG1, which has a relatively high amount of received light, moves over background region LG1 that is represented in light gray, is smaller than the latency DL that occurs when a foreground region DG2 that is lighter gray than background region DG1, which has a relatively low amount of received light, moves over background region DG1 that is represented in dark gray, as shown in the upper right part of Figure 6.

[0045] That is, since the length of latency changes depending on the amount of received light, it is not possible to appropriately reduce the effect of latency by subtracting a specific time from the time information.

[0046] Therefore, in the present disclosure, the amount of received light detected by the event signal detection sensor is detected, and the time information in the event data is corrected by the amount of latency corresponding to the detected amount of received light.

[0047] This makes it possible to correct the latency according to the amount of received light, and reduces the effect of the latency appropriately according to the amount of received light.

[0048] As a result, for example, it is possible to achieve a correction such that the area in which a positive event is detected in FIG. 4 approaches area Za1 from area Za2, and it is possible to appropriately reduce the effects of latency according to the amount of received light.

[0049] <<2. First embodiment>> Next, a configuration example of a signal processing device according to the present disclosure will be described with reference to Fig. 7. The signal processing device 101 in Fig. 7 includes an event signal detection sensor 111, a visible light sensor 112, a subsequent stage ISP (Image Sensor Processor) / AP (Application Processor) 113, and a subsequent stage application 114.

[0050] The event signal detection sensor 111 has the same configuration as the event signal detection sensor 11 in FIG. 1, detects a positive event or a negative event according to a change in the amount of received light, and outputs event data corresponding to the detection result to the downstream ISP / AP 113.

[0051] The visible light sensor 112 is a sensor that detects the amount of visible light received by the event signal detection sensor 111, and outputs a sensor output signal corresponding to the detected amount of received light to the subsequent ISP / AP 113.

[0052] More specifically, as shown in FIG. 8, the visible light sensor 112 is disposed in front of the event signal detection sensor 111 and is not on the optical axis AX1 of the lens 141 that focuses the light incident on the event signal detection sensor 111, but is disposed relatively close to the optical axis AX1.

[0053] With this arrangement, the visible light sensor 112 detects an amount of received light that is considered to be close to the amount of received light of the event signal detection sensor 111, and outputs it as a sensor output signal to the downstream ISP / AP 113. Since the visible light sensor 112 detects a sensor output signal indicating the amount of received light at one point in the vicinity of the optical axis AX1, the amount of received light calculated based on the sensor output signal is calculated assuming that all pixels constituting the event signal detection sensor 111 receive the same amount of light.

[0054] The downstream ISP / AP 113 is composed of a processor consisting of hardware, and calculates the amount of light received by the event signal detection sensor 111 based on the sensor output signal from the visible light sensor 112, and determines the amount of latency correction for the event data according to the calculated amount of light received.

[0055] Then, the subsequent stage ISP / AP 113 corrects the latency of the event data from the event signal detection sensor 111 using the identified latency correction amount, and outputs the corrected event data to the subsequent stage application 114 .

[0056] At this time, the subsequent stage ISP / AP 113 outputs a sensor output signal supplied from the visible light sensor 112 to the subsequent stage application 114 .

[0057] More specifically, the rear stage ISP / AP 113 includes an event detection pixel received light amount calculation unit 121 and a latency correction unit 122.

[0058] The event detection pixel received light amount calculation unit 121 calculates the amount of light received per unit time of the event detection pixels (corresponding to pixels 31 in FIG. 2) that constitute the event signal detection sensor 111 based on the sensor output signal, and outputs the amount of light received per unit time to the latency correction unit 122.

[0059] The event detection pixel received light amount calculation unit 121 calculates the amount of light received per unit time of the event detection pixels that constitute the event signal detection sensor 111, for example, the number of photons per time incident on one event detection pixel that constitutes the event signal detection sensor 111, irradiance [W / m2], or illuminance [Lux] based on the sensor output signal.

[0060] 7 and 8 show an example in which one visible light sensor 112 is provided for each event signal detection sensor, but a configuration in which multiple visible light sensors 112 are provided for each event signal detection sensor is also possible. In the case in which multiple visible light sensors 112 are provided for each event signal detection sensor, the amount of received light for each pixel may be calculated by using interpolation from the positional relationship between each of the multiple visible light sensors 112 and the event signal detection sensor 111.

[0061] The latency correction unit 122 identifies the latency based on the amount of light received per unit time of the pixels that constitute the event signal detection sensor 111 supplied from the event detection pixel light reception amount calculation unit 121, corrects the timestamp, which is the time information included in the event data, using the identified latency as a correction amount (hereinafter also referred to as the latency correction amount), and outputs the event data with the corrected timestamp as corrected event data.

[0062] In this embodiment, when the event signal detection sensor 111 generates event data, it outputs the event data with a time stamp, which is time information. However, the latency correction unit 122 may add a time stamp to the event data output by the event signal detection sensor 111 and further add a correction according to the latency.

[0063] More specifically, the latency correction unit 122 stores a table in which the latency for each event corresponding to the amount of received light is registered, as shown in FIG. 9, reads out the latency near the amount of received light supplied from the event detection pixel received light amount calculation unit 121, and calculates the corresponding latency by interpolation.

[0064] In the table of FIG. 9, the amount of received light, the latency in detecting a rising edge which is a positive event, and the latency in detecting a falling edge which is a negative event are registered in order from the left.

[0065] In FIG. 9, it is registered that when the amount of received light is 1000, the latency is A seconds when it is a positive event, and the latency is W seconds when it is a negative event.

[0066] Also, when the amount of received light is 100, it is registered that the latency is B seconds when it is a positive event, and that the latency is X seconds when it is a negative event.

[0067] Furthermore, it is registered that when the amount of received light is 10, the latency is C seconds when it is a positive event, and the latency is Y seconds when it is a negative event.

[0068] Also, when the amount of received light is 1, it is registered that the latency is D seconds when it is a positive event, and that the latency is Z seconds when it is a negative event.

[0069] In a table such as that shown in FIG. 9, latencies obtained in advance through experiments, simulations, etc., for positive events and negative events corresponding to the amount of received light are registered.

[0070] For example, when the amount of received light is 500 and it is a positive event, the latency correction unit 122 reads out from the table A seconds in a positive event when the amount of received light is 1000, which is close to 500, and B seconds in a positive event when the amount of received light is 100, and calculates the latency when the amount of received light is 500 by interpolation using, for example, the logarithmic average, and corrects the timestamp, which is the time information, by the latency correction amount.

[0071] The subsequent application 114 is made up of software such as an application program, and executes signal processing based on the event data and the sensor output signal after the latency has been corrected.

[0072] In the signal processing device 101 of FIG. 7, as described above, a sensor output signal corresponding to the amount of received light is output from the visible light sensor 112, the amount of received light of the event detection pixel is calculated based on the sensor output signal, and the timestamp is corrected based on the calculated amount of received light of the event detection pixel and on a latency correction amount corresponding to the amount of received light that has been measured in advance by an experiment or the like.

[0073] As a result, a latency correction amount corresponding to the amount of light received by the event signal detection sensor 111 is identified, and the time stamp, which is the time information, is corrected by the identified latency correction amount, thereby making it possible to appropriately reduce the effects of latency.

[0074] <Signal processing by the signal processing device in Fig. 7> Next, signal processing by the signal processing device 101 in FIG. 7 will be described with reference to the flowchart in FIG.

[0075] In step S31, the event signal detection sensor 111 determines whether or not an event has been detected. If it is determined in step S31 that an event has been detected, the process proceeds to step S32.

[0076] In step S32, the event signal detection sensor 111 outputs the event data to the latency correction unit 122 of the subsequent stage ISP / AP 113.

[0077] In step S33, the visible light sensor 112 outputs a sensor output signal corresponding to the amount of received visible light to the event detection pixel received light amount calculation unit 121 of the subsequent stage ISP / AP 113. At this time, the sensor output signal is also output to the subsequent stage application 114.

[0078] In step S34, the event detection pixel light receiving amount calculation unit 121 calculates the amount of light received per unit time of the event detection pixels that constitute the event signal detection sensor 111 based on the sensor output signal corresponding to the amount of received visible light, and outputs the calculated amount of light received to the latency correction unit 122.

[0079] In step S35, the latency correction unit 122 reads out the latency registered in association with the amount of received light from the stored table, and specifies the latency correction amount by performing interpolation as necessary.

[0080] In step S36, the latency correction unit 122 corrects the time stamp, which is the time information registered in the event data, by the latency correction amount and outputs the corrected time stamp to the subsequent application 114.

[0081] In step S37, the subsequent application 114 executes signal processing based on the corrected event data and the sensor output signal, and outputs the execution result.

[0082] Moreover, if no event is detected in step S31, the processes of steps S32 to S37 are skipped.

[0083] In step S38, it is determined whether or not an instruction to end the process has been given. If an instruction to end the process has not been given, the process returns to step S31, and the subsequent steps are repeated.

[0084] Then, in step S38, if an instruction to end the process is given, the process ends.

[0085] Through the above process, the latency correction amount is determined according to the amount of received light, and the time stamp, which is time information of the event data, is corrected by the determined latency correction amount.

[0086] As a result, an appropriate latency correction amount is set for the latency that changes according to the amount of received light, and the time information is corrected, so that the effect of the latency can be appropriately reduced.

[0087] <<3. Second embodiment>> In the above, an example has been described in which visible light sensor 112 is provided near event signal detection sensor 111, the amount of light received by event signal detection sensor 111 is calculated based on the sensor output signal of visible light sensor 112, a latency correction amount is obtained according to the calculated amount of received light, and a timestamp, which is time information, is corrected by the latency correction amount according to the amount of received light.

[0088] However, since the amount of received light differs for each pixel constituting the event signal detection sensor 111, the latency of the event data differs for each pixel constituting the event signal detection sensor 111.

[0089] Therefore, instead of the visible light sensor 112, a CMOS (Complementary Metal Oxide Semiconductor) image sensor may be provided, and the amount of light received by each pixel in the event signal detection sensor 111 may be determined from the gray scale signal of each pixel, and the latency may be corrected on a pixel-by-pixel basis.

[0090] FIG. 11 shows an example configuration of a signal processing device in which a CMOS image sensor is provided instead of the visible light sensor 112, the amount of light received by each pixel in the event signal detection sensor 111 is identified from the gradation signal of each pixel, and the latency is corrected on a pixel-by-pixel basis.

[0091] In the signal processing device 101A in Fig. 11, components having the same functions as those in the signal processing device 101 in Fig. 7 are given the same reference numerals, and the description thereof will be omitted as appropriate. Also, in the signal processing device 101A in Fig. 11, components having similar functions to those in the signal processing device 101 in Fig. 7 are given the same reference numerals with "A" added to the end.

[0092] Signal processing device 101A in FIG. 11 differs from signal processing device 101 in FIG. 7 in that a CMOS image sensor 151 is provided instead of visible light sensor 112, and a rear-stage ISP / AP 113A is provided instead of rear-stage ISP / AP 113.

[0093] The CMOS image sensor 151 is made up of a plurality of pixels arranged in an array in a direction perpendicular to the incident direction of incident light, and outputs a gray scale signal according to the amount of incident light to the downstream ISP / AP 113 in units of pixels.

[0094] 12, a lens 141A-2 that collects incident light is provided in front of the CMOS image sensor 151. In addition, a lens 141A-1 that collects incident light to the event signal detection sensor 111 is also provided in front of the event signal detection sensor 111.

[0095] Although the optical axes AX11 and AX12 of the lenses 141A-1 and 141A-2 are not the same, they are relatively close to each other and are arranged so as to be parallel to each other. Therefore, parallax occurs between each pixel of the CMOS image sensor 151 and each pixel of the event signal detection sensor 111 for objects in a relatively close range from both of them, but the gradation signal at each pixel of the CMOS image sensor 151 can be considered to roughly correspond to the amount of light received by each pixel of the event signal detection sensor 111.

[0096] Therefore, the CMOS image sensor 151 outputs a gradation signal in units of pixels to the subsequent stage ISP / AP 113A as information for detecting the amount of received light that is assumed to be approximate to the amount of received light of each pixel of the event signal detection sensor 111. In other words, the gradation signal indicating the amount of received light output from the CMOS image sensor 151 is output as information indicating the amount of received light of a pixel at a corresponding position configured in the event signal detection sensor 111.

[0097] The rear-stage ISP / AP 113A has the same basic functions as the rear-stage ISP / AP 113 in Figure 7, but calculates the amount of received light for each pixel in the event signal detection sensor 111 based on the event data from the event signal detection sensor 111 and the gradation signal from the CMOS image sensor 151, and determines a latency correction amount for each pixel according to the calculated amount of received light.

[0098] Then, the subsequent stage ISP / AP 113A corrects the latency of the event data with the latency correction amount specified for each pixel, and outputs the corrected event data in units of pixels to the subsequent stage application 114.

[0099] At this time, the subsequent stage ISP / AP 113 A outputs a sensor output signal supplied from the visible light sensor 112 to the subsequent stage application 114 .

[0100] More specifically, the rear stage ISP / AP 113A includes an event detection pixel received light amount calculation section 121A and a latency correction section 122A.

[0101] The event detection pixel received light amount calculation unit 121A has the same functions as the event detection pixel received light amount calculation unit 121, but calculates the amount of light received per unit time for each event detection pixel that constitutes the event signal detection sensor 111 based on a gradation signal in pixel units, and outputs the amount of light received per unit time to a latency correction unit 122A.

[0102] The event detection pixel received light amount calculation section 121A calculates the amount of received light per unit time for each event detection pixel constituting the event signal detection sensor 111 based on the gray scale signal of each pixel of the CMOS image sensor 151.

[0103] For example, the amount of light received per pixel per unit time is expressed as irradiance [W / m 2 ], the calculation is done using the following procedure.

[0104] First, the amount of charge per unit time for each pixel is calculated using the following formula (1).

[0105] Qu = (Q_after-gain / gain) / Te (1)

[0106] Here, Qu is the charge amount per unit time in units of pixels, Q_after-gain is the charge amount after the gain is multiplied in units of pixels of the CMOS image sensor 151, and is expressed by the following formula (2). Gain is the value of the multiplied gain, and is expressed by the following formula (3).

[0107] Q_after-gain=(Vs-OPB) / slope slope=((2 10 -1)-OPB) / Qs (2)

[0108] gain=10^(Vg[dB] / 20) Vg[dB]=20log(gain) (3)

[0109] Here, Vs is the value of the gray scale signal, OPB is the gray scale signal of a light-shielded pixel called optical black (usually 64), Qs is the saturated charge amount of the pixel, and Vg is the gain value [dB]. Also, "^" represents the power.

[0110] Then, when the spectrum of the incident light is known, the event detection pixel received light amount calculation unit 121A calculates the amount of received light per unit time for each pixel as irradiance [W / m by calculating the following formula (4) based on the calculated charge amount per unit time for each pixel. 2 ] is calculated as follows.

[0111] IRR=Qu1 / Qg (4)

[0112] Here, IRR is the amount of light received per pixel per unit time expressed as irradiance [W / m 2 ], Qu1 is the amount of charge generated in a pixel per second, and Qg is 1 [W / m 2 ] is the amount of charge that can be generated by photoelectric conversion in one second when light with an irradiance of

[0113] Furthermore, 1[W / m 2The amount of charge Qg that can be generated in one second by photoelectric conversion from light with an irradiance of [A] must be determined in advance by experiment or theoretical calculation according to the spectrum.

[0114] Then, the event detection pixel received light amount calculation unit 121A obtains the amount of received light per unit time for each pixel of the CMOS image sensor 151, and performs coordinate conversion for each pixel to the corresponding pixel position on the event signal detection sensor 111 to obtain the amount of received light of the corresponding pixel.

[0115] More specifically, as shown in FIG. 13, the event detection pixel light receiving amount calculation unit 121A calculates the amount of light received per unit time of pixel PC of the CMOS image sensor 151, performs coordinate conversion on the corresponding pixel PE on the event signal detection sensor 111, and applies the amount of light received by the coordinate conversion to the pixel PE.

[0116] In addition, in the above, it is assumed that the spectrum is known, but when the CMOS image sensor 151 is composed of RGB pixels, the event detection pixel received light amount calculation unit 121A may predict a spectrum for each pixel of the event signal detection sensor 111 from the ratio of the charge amounts of the RGB pixels, and calculate the amount of received light per unit time for each pixel by the calculation of the above-mentioned equations (1) to (4) based on the predicted spectrum for each pixel.

[0117] For example, if the amount of charge in the B pixel is greater than the amount of charge in the G pixel and R pixel, there is a high possibility that the spectrum will have a peak in the short wavelength region, and therefore a spectral distribution such as that shown by the solid line in the waveform shown in Figure 14 is predicted.

[0118] Furthermore, for example, if the amount of charge in the R pixel is greater than the amount of charge in the G pixel and the B pixel, there is a high possibility that the spectrum will have a peak in the long wavelength region, and therefore, for example, a spectral distribution such as that shown by the dotted line in the waveform shown in Figure 14 is predicted.

[0119] The event detection pixel received light amount calculation section 121A may estimate the spectrum from the ratio of the charge amounts of the R, G and B pixels by the method as described with reference to FIG.

[0120] In FIG. 14, the vertical axis indicates intensity and the horizontal axis indicates wavelength, and an example of the waveform of the spectral distribution is shown as a dotted line and a solid line.

[0121] The latency correction unit 122A has the same basic functions as the latency correction unit 122, but it determines the amount of latency correction for each pixel based on the amount of light received per unit time for each pixel constituting the event signal detection sensor 111 supplied from the event detection pixel received light amount calculation unit 121A, corrects the timestamp, which is the time information included in the event data, for each pixel according to the determined amount of latency correction for each pixel, and outputs the event data with the timestamp corrected for each pixel as corrected event data.

[0122] That is, while the latency correction unit 122 corrects the time information of the event data by a latency correction amount common to all pixels, the latency correction unit 122A corrects the time information of the event data for each pixel by a latency correction amount for each pixel.

[0123] For example, as shown in the left part of FIG. 15, in the event data indicating, from the left, a timestamp which is time information in units of pixels, coordinate information which represents the pixel position, and information on the event type indicating a positive event (rising edge) or a negative event (falling edge), for example, when the latency is 1 second, the latency correction unit 122A corrects the time information as shown in the right part of FIG. 15.

[0124] In the event data on the left side of FIG. 15, the top row indicates that a positive event, a rising edge, was detected in the pixel at coordinate position (x, y)=(100, 200) at time information 00:00:10. The second row from the top indicates that a positive event, a rising edge, was detected in the pixel at coordinate position (x, y)=(200, 200) at time information 00:00:10. The third row from the top indicates that a positive event, a rising edge, was detected in the pixel at coordinate position (x, y)=(100, 200) at time information 00:00:11. The fourth row from the top indicates that a positive event, a rising edge, was detected in the pixel at coordinate position (x, y)=(200, 200) at time information 00:00:11.

[0125] In addition, in the event data on the right side of FIG. 15, the top row indicates that a rising edge, which is a positive event, was detected in the pixel at coordinate position (x, y)=(100, 200) at time information 00:00:09. The second row from the top indicates that a rising edge, which is a positive event, was detected in the pixel at coordinate position (x, y)=(200, 200) at time information 00:00:09. The third row from the top indicates that a rising edge, which is a positive event, was detected in the pixel at coordinate position (x, y)=(100, 200) at time information 00:00:10. The fourth row from the top indicates that a rising edge, which is a positive event, was detected in the pixel at coordinate position (x, y)=(200, 200) at time information 00:00:10.

[0126] That is, the time information in the original event data shown on the left side of FIG. 15 is corrected in the right side of FIG. 15 by subtracting one second, which is the latency correction amount.

[0127] In the above, an example has been described in which the time information in the event data is corrected by the latency correction amount, but the corrected time information may be added.

[0128] For example, corrected time information may be added to the event data shown in the left part of FIG. 16, as shown in the right part of FIG.

[0129] The left part of Figure 16 is identical to the left part of Figure 15, and the right part of Figure 16 has a column for corrected time information added between the time information column and the coordinate information column in the right part of Figure 15.

[0130] Since the amount of latency correction varies on a pixel-by-pixel basis, the column for corrected time information on the right side of FIG. 16 can be replaced with a column for recording the delay time corresponding to the latency, i.e., the amount of latency correction, and the original time information can actually be corrected by the amount of latency correction before use.

[0131] <Signal processing by the signal processing device in Fig. 11> Next, signal processing by the signal processing device 101A of FIG. 11 will be described with reference to the flowchart of FIG.

[0132] In step S51, the event signal detection sensor 111 determines whether or not an event has been detected. If it is determined in step S51 that an event has been detected, the process proceeds to step S52.

[0133] In step S52, the event signal detection sensor 111 outputs the event data to the latency correction unit 122A of the subsequent stage ISP / AP 113A.

[0134] In step S53, the CMOS image sensor 151 outputs a gradation signal for each pixel to the event detection pixel received light amount calculation unit 121A of the subsequent stage ISP / AP 113A. At this time, the gradation signal is also output to the subsequent stage application 114.

[0135] In step S54, the event detection pixel light receiving amount calculation unit 121A calculates the amount of light received per unit time of the event detection pixels constituting the event signal detection sensor 111 based on the gradation signal in units of pixels, and outputs the calculated amount of light received for each pixel to the latency correction unit 122A.

[0136] In step S55, the latency correction unit 122A reads out the latency registered in association with the amount of received light for each pixel from the stored table, and specifies the latency correction amount for each pixel by performing interpolation as necessary.

[0137] In step S56, the latency correction unit 122A corrects the time stamp, which is the time information registered in the event data, by the latency correction amount for each pixel, and outputs the corrected time stamp to the subsequent application 114.

[0138] In step S57, the subsequent application 114 executes signal processing based on the corrected event data and the gradation signal, and outputs the execution result.

[0139] Moreover, if no event is detected in step S51, the processes of steps S52 to S57 are skipped.

[0140] In step S58, it is determined whether or not an instruction to end the process has been given. If an instruction to end the process has not been given, the process returns to step S51, and the subsequent steps are repeated.

[0141] Then, in step S58, if an instruction to end the process is given, the process ends.

[0142] Through the above processing, the amount of latency correction for each pixel is determined according to the amount of light received for each pixel, and the timestamp, which is the time information of the event data for each pixel, is corrected by the amount of latency correction for the determined pixel.

[0143] As a result, an appropriate latency correction amount is set for the latency that changes depending on the amount of light received on a pixel-by-pixel basis, and the time information is corrected on a pixel-by-pixel basis, making it possible to appropriately reduce the effects of latency.

[0144] <<4. Third embodiment>> In the above, an example has been described in which the time stamp, which is the time information in the event data, is corrected by the latency correction amount corresponding to the amount of received light. However, it is also possible to add only the information on the amount of received light to the event data, and have the downstream application 114 that acquires the event data correct the latency according to the amount of received light.

[0145] That is, for example, information on the amount of received light is added to original event data as shown in the left part of Fig. 18 as shown in the right part of Fig. 18, and the data is output to the subsequent application 114. The subsequent application 114 corrects the latency based on the event data with the amount of received light added, and then executes processing.

[0146] In addition, the left part of Figure 18 is original event data similar to the left parts of Figures 15 and 16, and the right part of Figure 18 has a column for the amount of light received at the corresponding pixel between a column for a time stamp, which is the time information in the left part of Figure 18, and a column for coordinate information indicating the pixel position.

[0147] In the column of the amount of received light on the right side of FIG. 18, 1111, 1111, 1112, and 1112 are registered as the amounts of received light in order from the top.

[0148] That is, the top row on the right side of FIG. 18 indicates that at time information 00:00:10, a rising edge as a positive event is detected in the pixel at coordinate position (x, y) = (100, 200), and the amount of received light is 1111. The second row from the top indicates that at time information 00:00:10, a rising edge as a positive event is detected in the pixel at coordinate position (x, y) = (200, 200), and the amount of received light is 1111. The third row from the top indicates that at time information 00:00:11, a rising edge as a positive event is detected in the pixel at coordinate position (x, y) = (100, 200), and the amount of received light is 1112. The fourth row from the top indicates that at time information 00:00:11, a rising edge as a positive event is detected in the pixel at coordinate position (x, y) = (200, 200), and the amount of received light is 1112.

[0149] The subsequent application 114 obtains a latency correction amount corresponding to the amount of received light based on the event data corrected by adding the information on the amount of received light, corrects the time stamp, which is time information, and executes the process.

[0150] <Example of the configuration of a signal processing device that registers the amount of received light in event data> FIG. 19 shows an example of the configuration of a signal processing device that registers the amount of received light in the event data.

[0151] In the signal processing device 101B in Fig. 19, components having the same functions as those in the signal processing device 101 in Fig. 7 and the signal processing device 101A in Fig. 11 are given the same reference numerals, and descriptions thereof will be omitted as appropriate. Also, in the signal processing device 101B in Fig. 19, components having similar functions to those in the signal processing device 101 in Fig. 7 and the signal processing device 101A in Fig. 11 are given the same reference numerals with "B" added to the end.

[0152] 19 differs from the signal processing device 101 in FIG. 7 and the signal processing device 101A in FIG. 11 in that instead of the subsequent stage ISP / APs 113 and 113A and the subsequent stage application 114, a subsequent stage ISP / AP 113B and a subsequent stage application 114B are provided.

[0153] More specifically, the rear stage ISP / AP 113B includes an event detection pixel received light amount calculation section 121B and a latency correction section 122B.

[0154] The event detection pixel received light amount calculation unit 121B has the same function as the event detection pixel received light amount calculation unit 121A, and calculates the amount of light received per unit time for each event detection pixel that constitutes the event signal detection sensor 111, and outputs it to the latency correction unit 122B.

[0155] The latency correction unit 122B adds the amount of light received for each event detection pixel to the event data supplied from the event signal detection sensor 111, as described with reference to FIG. 18, and outputs the corrected event data to the subsequent application 114B.

[0156] That is, unlike the latency correction sections 122 and 122A, the latency correction section 122B does not correct the time stamp, which is the time information included in the event data, and outputs the event data by simply adding information on the amount of received light.

[0157] It should be noted that the latency correction unit 122B does not correct the time stamp, which is time information, but it does provide information on the amount of received light that is necessary to correct the time stamp, which is time information of the event data, and therefore is considered to perform the function of correcting the event data.

[0158] The latter stage application 114B has the same basic functions as the latter stage application 114, but furthermore, based on the event data to which information on the amount of received light is added, corrects the time stamp, which is time information, by the latency correction amount and executes the processing.

[0159] More specifically, the latter-stage application 114B includes a latency correction unit 171. The latency correction unit 171 includes a table (similar to the table in FIG. 9) consisting of the amount of received light and the corresponding latency (latency correction amount), and reads out the corresponding latency correction amount from the table based on the information on the amount of received light included in the event data, and corrects the time stamp, which is time information.

[0160] <Signal processing by the signal processing device in Fig. 19> Next, signal processing by the signal processing device 101B in FIG. 19 will be described with reference to the flowchart in FIG.

[0161] In step S71, the event signal detection sensor 111 determines whether or not an event has been detected. If it is determined in step S71 that an event has been detected, the process proceeds to step S72.

[0162] In step S72, the event signal detection sensor 111 outputs the event data to the latency correction unit 122B of the subsequent stage ISP / AP 113B.

[0163] In step S73, the CMOS image sensor 151 outputs a gradation signal for each pixel to the event detection pixel received light amount calculation unit 121B of the subsequent stage ISP / AP 113B. At this time, the gradation signal is also output to the subsequent stage application 114B.

[0164] In step S74, the event detection pixel light receiving amount calculation unit 121B calculates the amount of light received per unit time of the event detection pixels constituting the event signal detection sensor 111 based on the gradation signal in units of pixels, and outputs the calculated amount of light received for each pixel to the latency correction unit 122B.

[0165] In step S75, the latency correction unit 122B corrects the event data by adding information on the amount of received light to the event data for each pixel.

[0166] In step S76, the latency correction unit 122B outputs the event data corrected by adding the information on the amount of received light for each pixel to the subsequent application 114B.

[0167] In step S77, the subsequent application 114B acquires the event data corrected by adding the information on the amount of received light for each pixel.

[0168] In step S78, the latency correction unit 171 of the subsequent application 114B reads information on the amount of received light for each pixel from the event data, and searches for and identifies the corresponding latency correction amount from a table in which latencies for the amount of received light are recorded.

[0169] In step S79, the latency correction unit 171 corrects the time stamp, which is time information, of the event data by the latency correction amount for each pixel.

[0170] In step S80, the subsequent application 114B executes signal processing based on the event data in which the time stamp, which is time information, has been corrected and the gradation signal, and outputs the execution result.

[0171] Moreover, if no event is detected in step S71, the processes of steps S72 to S80 are skipped.

[0172] In step S81, it is determined whether or not an instruction to end the process has been given. If an instruction to end the process has not been given, the process returns to step S71, and the subsequent steps are repeated.

[0173] Then, in step S81, if an instruction to end the process is given, the process ends.

[0174] In the above processing, the latency correction amount for each pixel is determined according to the amount of light received for each pixel, and the timestamp, which is the time information of the event data for each pixel, is corrected by the determined latency correction amount for each pixel.

[0175] As a result, an appropriate latency correction amount is set for the latency that changes depending on the amount of light received on a pixel-by-pixel basis, and the time information is corrected on a pixel-by-pixel basis, making it possible to appropriately reduce the effects of latency.

[0176] <<5. Fourth embodiment>> In the above, we have described example configurations of the signal processing devices 101, 101A, and 101B that use a CMOS image sensor 151 and an event signal detection sensor 111, but it is also possible to use a HybridEVS (EVS: Event-base Vision Sensor) sensor that has the functions of both the CMOS image sensor 151 and the event signal detection sensor 111.

[0177] The HybridEVS sensor has a pixel structure as shown in the right part of Fig. 21. The left part of Fig. 21 shows the pixel configuration of a CMOS image sensor 151, and shows the difference in configuration from the HybridEVS 201 in the right part of Fig. 21.

[0178] That is, for example, as shown in the left part of FIG. 21, the CMOS image sensor 151 has a pixel arrangement in which RGB pixel units are arranged in a Bayer array, with 2 pixels×2 pixels being one pixel unit.

[0179] In contrast, the HybridEVS201 is configured such that two of the four pixels constituting the R pixel unit and the B pixel unit in the CMOS image sensor 151 that are aligned vertically are arranged in place of the event detection pixel (corresponding to pixel 31 in Figure 2) EVS in the event signal detection sensor 111.

[0180] With such a configuration, in the HybridEVS sensor 201, a harmonic signal is acquired in the same manner as the configuration in the CMOS image sensor 151 from the RGB pixels, and event data is acquired from the event detection pixels EVS.

[0181] Also, in the HybridEVS sensor 201, as shown in FIG. 22, since the condenser lens 141B for light collection is provided at the front end, the harmonic signal and event data acquired in the HybridEVS 201 can be acquired while maintaining an appropriate positional relationship in which no parallax or the like in the arrangement occurs with respect to the axis AX21 in the lens 141B. Note that the pixel arrangement of the HybridEVS201 in the right part of FIG. 21 is an example, and a pixel arrangement other than the illustrated pixel arrangement may be used.

[0182] <First Configuration Example of Signal Processing Apparatus Using HybridEVS> FIG. 23 shows a first configuration example of a signal processing apparatus using a HybridEVS sensor. The signal processing apparatus 101C in FIG. 23 is composed of a HybridEVS sensor 201 and a subsequent-stage application 114.

[0183] The HybridEVS sensor 201 is composed of an event data acquisition unit 221, a harmonic signal acquisition unit 222, an event detection pixel light reception amount calculation unit 223, and a latency correction unit 224.

[0184] The event data acquisition unit 221 acquires the event data output from the event detection pixels EVS in FIG. 21 and outputs it to the latency correction unit 224.

[0185] The harmonic signal acquisition unit 222 acquires the harmonic signal output from the RGB pixels in FIG. 21 and outputs it to the event detection pixel light reception amount calculation unit 223.

[0186] The event data acquisition unit 221 functions in the same manner as the event signal detection sensor 111. Also, the harmonic signal acquisition unit 222 functions in the same manner as the CMOS image sensor 151.

[0187] Further, the event detection pixel light reception amount calculation unit 223 and the latency correction unit 224 function as the event detection pixel light reception amount calculation unit 121A or 121B and the latency correction unit 122A or 122B, respectively.

[0188] That is, the signal processing device 101C in FIG. 23 can substantially function as the signal processing device 101A in FIG. 11 or the signal processing device 101B in FIG. 19.

[0189] Note that since the respective signal processes are the same, the description thereof is omitted.

[0190] <Second Configuration Example of Signal Processing Device Using HybridEVS> In the above, a configuration example in which the HybridEVS sensor 201 includes the event data acquisition unit 221, the harmonic signal acquisition unit 222, the event detection pixel light reception amount calculation unit 223, and the latency correction unit 224 has been described. However, a configuration in which only the event data acquisition unit 221 and the harmonic signal acquisition unit 222 are provided may be used.

[0191] FIG. 24 shows a configuration example of a signal processing device when only the event data acquisition unit 221 and the harmonic signal acquisition unit 222 are provided in the HybridEVS sensor 201.

[0192] The signal processing device 101D in FIG. 24 includes a HybridEVS sensor 201D, a subsequent stage ISP / AP 241, and a subsequent stage application 114.

[0193] The HybridEVS sensor 201D has the same basic functions as the HybridEVS sensor 201, but only the event data acquisition unit 221D and the harmonic signal acquisition unit 222D having the same functions as the respective event data acquisition unit 221 and harmonic signal acquisition unit 222 are provided.

[0194] The subsequent ISP / AP241 is provided with an event detection pixel light reception amount calculation unit 223D and a latency correction unit 224D, each having the same functions as the event detection pixel light reception amount calculation unit 223 and the latency correction unit 224, respectively.

[0195] That is, in the signal processing device 101D of FIG. 24, the functions of the HybridEVS sensor 201 of FIG. 23 are realized by the HybridEVS sensor 201D and the subsequent ISP / AP241.

[0196] With such a configuration, the signal processing device 101D in FIG. 24 also substantially functions as the signal processing device 101A in FIG. 11 or the signal processing device 101B in FIG. 19.

[0197] <The third configuration example of the signal processing device using HybridEVS> In the above, an example in which the HybridEVS sensor 201D is composed only of the event data acquisition unit 221D and the harmonic signal acquisition unit 222D has been described. However, the function of the event detection pixel light reception amount calculation unit 223 may be further provided.

[0198] FIG. 25 shows a configuration example of a signal processing device when the HybridEVS sensor 201 is realized by an event data acquisition unit 221, a harmonic signal acquisition unit 222, and an event detection pixel light reception amount calculation unit 223.

[0199] The signal processing device 101E in FIG. 25 is composed of a HybridEVS sensor 201E, a subsequent ISP / AP241E, and a subsequent application 114.

[0200] The basic functions of the HybridEVS sensor 201E are the same as those of the HybridEVS sensor 201, but it is provided with an event data acquisition unit 221E, a harmonic signal acquisition unit 222E, and an event detection pixel light reception amount calculation unit 223E, each having the same functions as the event data acquisition unit 221, the harmonic signal acquisition unit 222, and the event detection pixel light reception amount calculation unit 223, respectively.

[0201] The subsequent stage ISP / AP 241E is provided with a latency correction unit 224E having the same function as the latency correction unit 224.

[0202] That is, in the signal processing device 101E in FIG. 25, the function of the HybridEVS sensor 201 in FIG. 23 is realized by the HybridEVS sensor 201E and the subsequent stage ISP / AP 241E.

[0203] With this configuration, signal processing device 101E in FIG. 25 also substantially realizes the functions of signal processing device 101A in FIG. 11 or signal processing device 101B in FIG.

[0204] <<6. Fifth embodiment>> Although an example of the configuration of a signal processing device using the HybridEVS sensor 201 has been described above, it is also possible to acquire pixel signals of a pixel array constituting the CMOS image sensor 151 by switching between gradation signals and event data.

[0205] FIG. 26 shows an example of the configuration of a signal processing device that acquires, by switching, a gray scale signal and event data from pixel signals of a pixel array that constitutes the CMOS image sensor 151.

[0206] The signal processing device 101F in FIG. 26 includes a switching sensor 261, a rear stage ISP / AP 113A, and a rear stage application 114.

[0207] The switching sensor 261 includes a pixel array 270 that constitutes the CMOS image sensor 151, an event data detection unit 271 that acquires event data based on a signal output from the pixel array 270, and a gradation signal detection unit 272 that detects a gradation signal.

[0208] The signal output from the pixel array 270 is switched to be output to either the event data detection section 271 or the gradation signal detection section 272 .

[0209] Therefore, the event data detection unit 271 detects event data at the timing when a signal is output from the pixel array 270 to itself, and outputs the event data to the latency correction unit 122A of the subsequent ISP / AP 113.

[0210] Furthermore, the gradation signal detection unit 272 detects a gradation signal at the timing when a signal is output from the pixel array 270 to itself, and outputs the gradation signal to the event detection pixel received light amount calculation unit 121A of the subsequent ISP / AP 113.

[0211] That is, the switching sensor 261 functions as a configuration having the functions of both the CMOS image sensor 151 and the event signal detection sensor 111 described above.

[0212] As a result, with such a configuration, signal processing device 101F in FIG. 26 also realizes substantially the same functions as signal processing device 101A in FIG. 11 or signal processing device 101B in FIG.

[0213] <<7. Application Examples>> In the above, an example has been described in which the latency correction amount is set according to the amount of received light, and the time stamp, which is the time information in the event data, is corrected by the latency correction amount.

[0214] However, when event data is transmitted to the subsequent application 114 in units of an event data group consisting of multiple event data, the event data group that is shifted by a latency correction amount specified according to the amount of received light may be output to the subsequent application 114.

[0215] More specifically, as shown in the upper part of Figure 27, each rectangular frame represents one piece of event data, which is generated in chronological order to the right in the figure, and the effect of latency occurs for a period of time equivalent to two pieces of event data.

[0216] Here, consider the case where, in the upper part of FIG. 27, a predetermined number of event data items generated in chronological order are sequentially buffered, and then, for example, an event data group W1 consisting of event data D1 to D14 is output to the subsequent application 114.

[0217] In this case, if the event data group W1 is sent as is to the subsequent application 114, the subsequent application 114 will process the event data D1 and D2, which are already past event data at the time the event data is acquired, as is, resulting in latency effects.

[0218] Therefore, when the latency correction unit 122 of the subsequent stage ISP / AP 113 transmits the event data group, it may shift the event data group by a latency correction amount SF21 so as to skip two event data D1 and D2 which are expected to be affected by latency, as shown in the lower part of Figure 27, and transmit the event data group W2 consisting of the subsequent stage event data D3 to D16 to the subsequent stage application 114.

[0219] This process shifts the event data group and sends it with past event data skipped by the amount of latency. As a result, when the subsequent application 114 acquires the event data group W2, it is acquired as the event data group W2 from which the effect of latency has been removed, so that processing in which the effect of latency has been corrected can be achieved simply by sequentially processing the event data constituting the acquired event data group W2 in chronological order.

[0220] <Signal processing in application examples> Next, signal processing in an application example will be described with reference to the flowchart in Fig. 28. Note that, here, the processing will be described when the signal processing with reference to Fig. 27 is performed by the signal processing device 101 in Fig. 7, and it is assumed that all pixels receive the same amount of light, and the event data of all pixels is corrected with a unified latency correction amount.

[0221] In step S101, the event signal detection sensor 111 determines whether or not an event has been detected. If it is determined in step S101 that an event has been detected, the process proceeds to step S102.

[0222] In step S102, the event signal detection sensor 111 outputs the event data to the latency correction unit 122 of the subsequent stage ISP / AP 113. At this time, the latency correction unit 122 buffers the event data.

[0223] In step S103, the event signal detection sensor 111 determines whether the number of buffered event data is equal to or greater than a predetermined threshold, and if the number is not equal to or greater than the predetermined threshold, the process returns to step S101.

[0224] The predetermined threshold value here is, for example, a value obtained by adding the number of event data items corresponding to the expected latency to the number of event data items to be transmitted as the event data group described with reference to FIG.

[0225] Therefore, in the example of FIG. 27, since the latency is equivalent to two pieces of event data, the predetermined threshold is 16 or more.

[0226] If it is determined in step S103 that the number of buffered event data items is equal to or greater than the predetermined threshold, the process proceeds to step S104.

[0227] In step S104, the visible light sensor 112 outputs a sensor output signal according to the amount of received visible light to the event detection pixel received light amount calculation unit 121 of the subsequent stage ISP / AP 113.

[0228] In step S105, the event detection pixel light receiving amount calculation unit 121 calculates the amount of light received per unit time of the event detection pixels that constitute the event signal detection sensor 111 based on the sensor output signal corresponding to the amount of received visible light, and outputs the calculated amount of light received to the latency correction unit 122.

[0229] In step S106, the latency correction unit 122 reads out the latency registered in association with the amount of received light from the stored table, and specifies the latency correction amount by performing interpolation as necessary.

[0230] In step S107, the latency correction unit 122 shifts the buffered event data by the latency correction amount, and then composes an event data group consisting of a predetermined number of event data, thereby correcting the latency and outputting it to the downstream application 114.

[0231] In step S108, the subsequent application 114 executes signal processing by sequentially processing the event data constituting the event data group based on the event data group whose latency has been corrected, and outputs the execution result.

[0232] Moreover, if no event is detected in step S101, the processes of steps S102 to S108 are skipped.

[0233] In step S109, it is determined whether or not an instruction to end the process has been given. If an instruction to end the process has not been given, the process returns to step S101, and the subsequent steps are repeated.

[0234] Then, in step S109, if an instruction to end the process is given, the process ends.

[0235] By the above processing, the latency correction amount is determined according to the amount of received light, and the event data group is shifted by the determined latency correction amount. Therefore, in the subsequent application 114, it is possible to realize processing in which the effects of latency are corrected simply by processing the event data that constitute the acquired event data group in chronological order.

[0236] As a result, it is possible to appropriately reduce the effect of latency that changes depending on the amount of received light.

[0237] In the above, we have described the processing performed by signal processing device 101 of Figure 7 when signal processing with reference to Figure 27 is performed. However, similar processing can be realized on a pixel-by-pixel basis in any of signal processing device 101A of Figure 11, signal processing device 101C of Figure 23, signal processing device 101D of Figure 24, signal processing device 101E of Figure 25, and signal processing device 101F of Figure 26.

[0238] <First modified example of event data shift method> In the above, an example has been described in which the event data group to be transmitted, among the buffered event data, is shifted by the latency correction amount and output to the subsequent application 114.

[0239] However, there is a possibility that the processing may be affected if the event data D1 and D2 that were prevented from being transmitted needs to be referenced, and the data is lost.

[0240] Therefore, the subsequent event data may be increased by the latency correction amount with respect to the initial event data group, and output to the subsequent application 114 .

[0241] That is, as shown in the upper part of Figure 29, event data D1 to D14 consisting of event data group W1 that is initially expected to be sent to subsequent application 114 is transmitted as event data group W11 consisting of event data D1 to D16 to which subsequent event data D15, D16 have been added by the effect of latency, as shown in the lower part of Figure 29.

[0242] Then, the subsequent application 114 may selectively use the event data D3 to D16 from the event data group consisting of the event data D1 to D16, excluding the past event data D1 and D2 by the latency time, to execute processing.

[0243] That is, as shown in the lower part of Figure 29, an event data group W11 may be transmitted in which the event data D15, S16 are increased by a shift amount SF21 that is affected by latency, in comparison with the event data group W1 consisting of the uncorrected event data D1 to D14.

[0244] In this case, the subsequent application 114 can recognize the event data D1 and D2 at the beginning of the event data group W11, which are affected by latency, as past event data based on the timestamp or the like. Therefore, by excluding the event data from the processing in advance and executing the processing, it is possible to realize processing that is not affected by latency.

[0245] As described above, when it is necessary to refer to past event data D1 and D2 depending on the processing content, the data is transmitted as a group of event data without any loss, so that it can be referred to in the subsequent application 114.

[0246] <Second modified example of event data shift method> The event data group may be constructed by deleting only past event data that is affected by latency.

[0247] That is, as shown in the lower part of Figure 30, an event data group W21 may be transmitted in which only the event data D1 and D2 corresponding to the shift amount SF21 affected by latency is deleted from the uncorrected event data group W1 shown in the upper part of Figure 30.

[0248] In this case, the latter-stage application 114 can realize processing that is not affected by latency by executing the processing sequentially from the first event data of the event data group W21. In this case, the event data following the original event data group W11 in the time series is lost, but the impact is small because only the event data for the latency is lost.

[0249] <<8. Mobile Application Examples>> The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of moving body such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, a personal mobility device, an airplane, a drone, a ship, a robot, etc.

[0250] FIG. 31 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a moving object control system to which the technology according to the present disclosure can be applied.

[0251] The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in Fig. 31, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside-vehicle information detection unit 12030, an inside-vehicle information detection unit 12040, and an integrated control unit 12050. Also, as functional configurations of the integrated control unit 12050, a microcomputer 12051, an audio / video output unit 12052, and an in-vehicle network I / F (Interface) 12053 are illustrated.

[0252] The drivetrain control unit 12010 controls the operation of devices related to the drivetrain of the vehicle in accordance with various programs. For example, the drivetrain control unit 12010 functions as a control device for a drive force generating device for generating a drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force of the vehicle.

[0253] The body system control unit 12020 controls the operation of various devices mounted on the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as head lamps, back lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves or signals of various switches transmitted from a portable device that replaces a key may be input to the body system control unit 12020. The body system control unit 12020 receives the input of these radio waves or signals and controls the door lock device, power window device, lamps, and the like of the vehicle.

[0254] The outside-vehicle information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the outside-vehicle information detection unit 12030. The outside-vehicle information detection unit 12030 causes the imaging unit 12031 to capture an image outside the vehicle and receives the captured image. The outside-vehicle information detection unit 12030 may perform object detection processing or distance detection processing for a person, a car, an obstacle, a sign, or characters on a road surface, based on the received image.

[0255] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal according to the amount of light received. The imaging unit 12031 can output the electrical signal as an image, or as distance measurement information. The light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

[0256] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041 that detects the state of the driver is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 may calculate the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing.

[0257] The microcomputer 12051 can calculate control target values ​​for the driving force generating device, the steering mechanism, or the braking device based on the information inside and outside the vehicle acquired by the outside-of-vehicle information detection unit 12030 or the inside-vehicle information detection unit 12040, and output a control command to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the inter-vehicle distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane departure warning, etc.

[0258] In addition, the microcomputer 12051 can perform cooperative control for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on the driver's operation, by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle acquired by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040.

[0259] Furthermore, the microcomputer 12051 can output a control command to the body system control unit 12030 based on the information outside the vehicle acquired by the outside-vehicle information detection unit 12030. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside-vehicle information detection unit 12030, and perform cooperative control for the purpose of preventing glare, such as switching from high beams to low beams.

[0260] The audio / video output unit 12052 transmits at least one output signal of audio and video to an output device capable of visually or audibly notifying information to passengers in the vehicle or the outside of the vehicle. In the example of Fig. 31, an audio speaker 12061, a display unit 12062, and an instrument panel 12063 are exemplified as the output device. The display unit 12062 may include at least one of an on-board display and a head-up display, for example.

[0261] FIG. 32 is a diagram showing an example of the installation position of the imaging unit 12031.

[0262] In FIG. 32, the imaging unit 12031 includes imaging units 12101, 12102, 12103, 12104, and 12105.

[0263] The imaging units 12101, 12102, 12103, 12104, and 12105 are provided at positions such as the front nose, side mirrors, rear bumper, back door, and the upper part of the windshield in the vehicle interior of the vehicle 12100. The imaging unit 12101 provided at the front nose and the imaging unit 12105 provided at the upper part of the windshield in the vehicle interior mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 provided at the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 provided at the rear bumper or back door mainly acquires images of the rear of the vehicle 12100. The imaging unit 12105 provided at the upper part of the windshield in the vehicle interior is mainly used to detect a preceding vehicle, a pedestrian, an obstacle, a traffic light, a traffic sign, a lane, or the like.

[0264] 32 shows an example of the imaging ranges of the imaging units 12101 to 12104. An imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and an imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or back door. For example, an overhead image of the vehicle 12100 viewed from above is obtained by superimposing the image data captured by the imaging units 12101 to 12104.

[0265] At least one of the imaging units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera made up of a plurality of imaging elements, or may be an imaging element having pixels for detecting a phase difference.

[0266] For example, the microcomputer 12051 can extract, as a preceding vehicle, a three-dimensional object that is the closest three-dimensional object on the travel path of the vehicle 12100 and travels at a predetermined speed (for example, 0 km / h or more) in approximately the same direction as the vehicle 12100, by calculating the distance to each three-dimensional object in the imaging ranges 12111 to 12114 and the change over time of this distance (relative speed with respect to the vehicle 12100) based on the distance information obtained from the imaging units 12101 to 12104. Furthermore, the microcomputer 12051 can set a vehicle distance to be secured in advance in front of the preceding vehicle, and perform automatic brake control (including follow-up stop control) and automatic acceleration control (including follow-up start control). In this way, cooperative control can be performed for the purpose of automatic driving that travels autonomously without relying on the driver's operation.

[0267] For example, the microcomputer 12051 classifies and extracts three-dimensional object data on three-dimensional objects into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on distance information obtained from the imaging units 12101 to 12104, and can use the data for automatic obstacle avoidance. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. Then, the microcomputer 12051 determines a collision risk indicating the risk of collision with each obstacle, and when the collision risk is equal to or greater than a set value and a collision is possible, the microcomputer 12051 can provide driving assistance for collision avoidance by outputting an alarm to the driver via the audio speaker 12061 or the display unit 12062, or by performing forced deceleration or avoidance steering via the drive system control unit 12010.

[0268] At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether or not a pedestrian is present in the captured images of the imaging units 12101 to 12104. The recognition of such a pedestrian is performed, for example, by a procedure of extracting feature points in the captured images of the imaging units 12101 to 12104 as infrared cameras, and a procedure of performing pattern matching processing on a series of feature points that indicate the contour of an object to determine whether or not the object is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the captured images of the imaging units 12101 to 12104 and recognizes the pedestrian, the audio / image output unit 12052 controls the display unit 12062 to superimpose a rectangular contour line for emphasis on the recognized pedestrian. The audio / image output unit 12052 may also control the display unit 12062 to display an icon or the like indicating a pedestrian at a desired position.

[0269] An example of a vehicle control system to which the technology according to the present disclosure can be applied has been described above. The technology according to the present disclosure can be applied to a configuration in which the imaging unit 12031 and the vehicle outside information detection unit 12030 are combined among the configurations described above. Specifically, the signal processing device 101 in FIG. 7, the signal processing device 101A in FIG. 11, the signal processing device 101B in FIG. 19, the signal processing device 101C in FIG. 23, the signal processing device 101D in FIG. 24, the signal processing device 101E in FIG. 25, and the signal processing device 101F in FIG. 26 can be applied to a configuration in which the imaging unit 12031 and the vehicle outside information detection unit 12030 are combined. By applying the technology according to the present disclosure to a configuration in which the imaging unit 12031 and the vehicle outside information detection unit 12030 are combined, it is possible to reduce the effect of latency that may occur in the object detection process or the distance detection process.

[0270] In this specification, a system means a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same case. Therefore, multiple devices housed in separate cases and connected via a network, and a single device in which multiple modules are housed in a single case, are both systems.

[0271] Furthermore, the embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present disclosure.

[0272] For example, the present disclosure can take the form of cloud computing in which a single function is shared and processed collaboratively by multiple devices via a network.

[0273] Furthermore, each step described in the above flow chart can be executed by one device, or can be shared and executed by a plurality of devices.

[0274] Furthermore, when a single step includes multiple processes, the multiple processes included in the single step can be executed by a single device, or can be shared and executed by multiple devices.

[0275] The present disclosure can also be configured as follows. <1> an event detection unit that outputs event data representing the occurrence of an event, the event data being a change in an electrical signal detected by a plurality of event detection pixels that are arranged on a two-dimensional plane and generate an electrical signal by performing photoelectric conversion; a light reception amount calculation unit that calculates an amount of light received by the event detection pixel; a latency correction unit that corrects information relating to latency in the event data based on the amount of light received by the event detection pixel calculated by the light-receiving-amount calculation unit; A light detection device comprising: <2> The latency correction unit has a table in which the relationship between the amount of received light and the latency is registered, reads the latency corresponding to the amount of received light calculated by the amount of received light calculation unit from the table, and corrects the event data by a latency correction amount consisting of the read latency. <1> The optical detection device according to claim 1 . <3> the event data includes time information at which the event was detected by the event detection unit, The latency correction unit corrects the time information of the event data by the latency correction amount corresponding to the amount of received light. <2> The optical detection device according to claim 1 . <4> the event data further includes coordinate information representing pixel positions of the event detection pixels of the event detection unit, The latency correction unit corrects the time information of the event data by the latency correction amount corresponding to the amount of received light in association with each of the coordinate information of the event detection pixels. <3> The optical detection device according to claim 1 . <5> the event data further includes coordinate information representing pixel positions of the event detection pixels of the event detection unit, The latency correction unit adds corrected time information obtained by correcting the time information by the latency correction amount corresponding to the amount of received light in association with each coordinate information of the event detection pixel, or corrects the event data by adding the latency correction amount. <3> The optical detection device according to claim 1 . <6> the event data further includes information on a type of the event, the information indicating whether the event is a positive event in which the electrical signal detected by the event detection unit changes in a positive direction, or a negative event in which the electrical signal changes in a negative direction; The table registers a relationship between the amount of received light and the latency for each type of information of the event, The latency correction unit adds corrected time information obtained by correcting the time information by the latency correction amount corresponding to the amount of received light in accordance with the type of the event, in association with each coordinate position of the event detection pixel, or corrects the event data by adding the latency correction amount. <3> The optical detection device according to claim 1 . <7> a visible light detection unit that detects the amount of received visible light in the vicinity of an imaging surface formed by arranging the event detection pixels in a two-dimensional plane, and outputs an output signal corresponding to the detected amount of received light, The received light amount calculation unit calculates the amount of received light of the event detection pixel based on the output signal. <1> The optical detection device according to claim 1 . <8> the visible light detection unit detects the amount of received visible light at one point in the vicinity of an imaging plane formed by arranging the event detection pixels in a two-dimensional plane, and outputs the amount of received visible light as the corresponding output signal; the received light amount calculation unit calculates, based on the output signal, the amounts of received light of all the event detection pixels as being the same, The latency correction unit corrects the event data of all the event detection pixels by a latency correction amount corresponding to the same amount of received light. <7> The optical detection device according to claim 1 . <9> a gradation signal detection unit configured of a plurality of imaging pixels arranged in a two-dimensional plane, which detects and outputs a gradation signal corresponding to an amount of incident light in the vicinity of an imaging plane configured by arranging the event detection pixels in a two-dimensional plane; The received light amount calculation unit calculates the received light amount for each of the event detection pixels based on the gradation signal for each of the imaging pixels. <1> The optical detection device according to claim 1 . <10> the received light amount calculation unit calculates the amount of received light for each of the imaging pixels, and applies the amount of received light to the event detection pixel at a position corresponding to the position of the imaging pixel; The latency correction unit corrects the event data of each of the event detection pixels by a latency correction amount corresponding to the amount of received light applied to each of the event detection pixels. <9> The optical detection device according to claim 1 . <11> The received light amount calculation unit calculates the amount of received light of the event detection pixel based on a predetermined spectrum and the gradation signal. <10> The optical detection device according to claim 1 . <12> The predetermined spectrum is a spectrum estimated from the tone signal. <11> The optical detection device according to claim 1 . <13> The plurality of imaging pixels constituting the gradation signal detection unit are provided with a plurality of filters in a predetermined arrangement in front of each of the imaging pixels, which filters the incident light passing through the imaging pixels into a plurality of wavelength bands, and the received light amount calculation unit estimates a spectrum from a ratio of the received light amounts of the plurality of filters based on the gradation signal, and calculates the received light amount of the event detection pixel based on the estimated spectrum and the gradation signal. <12> The optical detection device according to claim 1 . <14> the received light amount calculation unit calculates the amount of received light for each of the imaging pixels, and applies the amount of received light to the event detection pixel at a position corresponding to the position of the imaging pixel; The latency correction unit performs correction by adding the amount of received light applied to each of the event detection pixels to the event data of each of the event detection pixels. <9> The optical detection device according to claim 1 . <15> The event detection pixels and the imaging pixels are arranged on the same two-dimensional plane, and the event detection unit and the gradation signal detection unit are integrated into one structure. <9> The optical detection device according to claim 1 . <16> In addition to the event detection unit and the gradation signal detection unit, the light reception amount calculation unit is integrated into the event detection unit. <15> The optical detection device according to claim 1 . <17> In addition to the event detection unit, the gradation signal detection unit, and the light reception amount calculation unit, the latency correction unit is integrated into the configuration. <16> The optical detection device according to claim 1 . <18> The latency correction unit corrects information related to the latency of the event data by shifting an event data group consisting of a predetermined number of the event data in a time series to be output to a subsequent stage by a number of event data corresponding to the latency, based on the amount of light received by the event detection pixel calculated by the light reception amount calculation unit. <1> The optical detection device according to claim 1 . <19> outputting event data representing the occurrence of an event, the event data being a change in an electrical signal detected by event detection pixels arranged in a two-dimensional plane and performing photoelectric conversion to generate an electrical signal; Calculating the amount of light received by the event detection pixel; Correcting information related to latency in the event data based on the calculated amount of light received by the event detection pixel. A light detection method comprising the steps of: <20> an event detection unit that outputs event data representing the occurrence of an event, the event data being a change in an electrical signal detected by a plurality of event detection pixels that are arranged on a two-dimensional plane and generate an electrical signal by performing photoelectric conversion; a light reception amount calculation unit that calculates an amount of light received by the event detection pixel; a latency correction unit that corrects information relating to latency in the event data based on the amount of light received by the event detection pixel calculated by the light-receiving-amount calculation unit; programs that make a computer function. [Explanation of symbols]

[0276] 101, 101A to 101F signal processing device, 111 event signal detection sensor, 112 visible light sensor, 113, 113A, 113B rear stage ISP / AP, 114, 114B rear stage application, 121, 121A, 121B event detection pixel light receiving amount calculation unit, 122, 122A, 122B latency correction unit, 141, 141A-1, 141A-2, 141B lens, 151 CMOS image sensor, 171 latency correction unit, 201, 201D, 201E HybridEVS sensor, 221, 221D, 221E event data acquisition unit, 222, 222D, 222E gradation signal acquisition unit, 223, 223D, 223E event detection pixel light receiving amount calculation unit, 224, 224D, 224E latency compensation unit, 241, 241E rear stage ISP / AP, 261 switching sensor, 270 pixel array, 271 event data detection unit, 272 gradation signal acquisition unit

Claims

1. an event detection unit that outputs event data representing an occurrence of an event, the event data being a change in an electrical signal detected by a plurality of event detection pixels that are arranged on a two-dimensional plane and generate an electrical signal by performing photoelectric conversion; a light reception amount calculation unit that calculates an amount of light received by the event detection pixel; a latency correction unit that corrects information relating to latency in the event data based on the amount of light received by the event detection pixel calculated by the light-receiving-amount calculation unit; A light detection device comprising:

2. The latency correction unit has a table in which the relationship between the amount of received light and the latency is registered, reads the latency corresponding to the amount of received light calculated by the amount of received light calculation unit from the table, and corrects the event data by a latency correction amount consisting of the read latency.

2. The optical detection device according to claim 1.

3. the event data includes time information at which the event was detected by the event detection unit, The latency correction unit corrects the time information of the event data by the latency correction amount corresponding to the amount of received light.

3. The optical detection device according to claim 2.

4. the event data further includes coordinate information representing pixel positions of the event detection pixels of the event detection unit, The latency correction unit corrects the time information of the event data by the latency correction amount corresponding to the amount of received light in association with each of the coordinate information of the event detection pixels.

4. The optical detection device according to claim 3.

5. the event data further includes coordinate information representing pixel positions of the event detection pixels of the event detection unit, The latency correction unit adds corrected time information obtained by correcting the time information by the latency correction amount corresponding to the amount of received light in association with each coordinate information of the event detection pixel, or corrects the event data by adding the latency correction amount.

4. The optical detection device according to claim 3.

6. the event data further includes information on a type of the event, the information indicating whether the event is a positive event in which the electrical signal detected by the event detection unit changes in a positive direction, or a negative event in which the electrical signal changes in a negative direction; The table registers a relationship between the amount of received light and the latency for each type of information of the event, The latency correction unit adds corrected time information obtained by correcting the time information by the latency correction amount corresponding to the amount of received light in accordance with the type of the event, in association with each coordinate position of the event detection pixel, or corrects the event data by adding the latency correction amount.

4. The optical detection device according to claim 3.

7. a visible light detection unit that detects the amount of received visible light in the vicinity of an imaging plane formed by arranging the event detection pixels in a two-dimensional plane, and outputs an output signal corresponding to the detected amount of received light, The received light amount calculation unit calculates the amount of received light of the event detection pixel based on the output signal.

2. The optical detection device according to claim 1.

8. the visible light detection unit detects the amount of received visible light at one point near an imaging plane formed by arranging the event detection pixels on a two-dimensional plane, and outputs the amount of received visible light as the corresponding output signal; the received light amount calculation unit calculates, based on the output signal, the amounts of received light of all the event detection pixels as being the same, The latency correction unit corrects the event data of all the event detection pixels by a latency correction amount corresponding to the same amount of received light.

8. The optical detection device according to claim 7.

9. a gradation signal detection unit configured of a plurality of imaging pixels arranged in a two-dimensional plane, which detects and outputs a gradation signal corresponding to an amount of incident light in the vicinity of an imaging plane configured by arranging the event detection pixels in the two-dimensional plane; The received light amount calculation unit calculates the received light amount for each of the event detection pixels based on the gradation signal for each of the imaging pixels.

2. The optical detection device according to claim 1.

10. the received light amount calculation unit calculates the amount of received light for each of the imaging pixels, and applies the amount of received light to the event detection pixel at a position corresponding to the position of the imaging pixel; The latency correction unit corrects the event data of each of the event detection pixels by a latency correction amount corresponding to the amount of received light applied to each of the event detection pixels.

10. The optical detection device according to claim 9.

11. The received light amount calculation unit calculates the amount of received light of the event detection pixel based on a predetermined spectrum and the gradation signal. The optical detection device according to claim 10.

12. The predetermined spectrum is a spectrum estimated from the tone signal. The optical detection device according to claim 11.

13. The plurality of imaging pixels constituting the gradation signal detection unit are provided with a plurality of filters in a predetermined arrangement in front of each of the imaging pixels, which filters the incident light passing through the imaging pixels into a plurality of wavelength bands, and the received light amount calculation unit estimates a spectrum from a ratio of the received light amounts of the plurality of filters based on the gradation signal, and calculates the received light amount of the event detection pixel based on the estimated spectrum and the gradation signal.

13. The optical detection device of claim 12.

14. the received light amount calculation unit calculates the amount of received light for each of the imaging pixels, and applies the amount of received light to the event detection pixel at a position corresponding to the position of the imaging pixel; The latency correction unit performs correction by adding the amount of received light applied to each of the event detection pixels to the event data of each of the event detection pixels.

10. The optical detection device according to claim 9.

15. The event detection pixels and the imaging pixels are arranged on the same two-dimensional plane, and the event detection unit and the gradation signal detection unit are integrated into one structure.

10. The optical detection device according to claim 9.

16. In addition to the event detection unit and the gradation signal detection unit, the light reception amount calculation unit is integrated into the event detection unit.

16. The optical detection device of claim 15.

17. In addition to the event detection unit, the gradation signal detection unit, and the light reception amount calculation unit, the latency correction unit is integrated into the configuration.

17. The optical detection device of claim 16.

18. The latency correction unit corrects information related to the latency of the event data by shifting an event data group consisting of a predetermined number of the event data in a time series to be output to a subsequent stage by a number of event data corresponding to the latency, based on the amount of light received by the event detection pixel calculated by the light reception amount calculation unit.

2. The optical detection device according to claim 1.

19. outputting event data representing the occurrence of an event, the event data being a change in an electrical signal detected by a plurality of event detection pixels arranged on a two-dimensional plane and generating an electrical signal by performing photoelectric conversion; Calculating the amount of light received by the event detection pixel; Correcting information related to latency in the event data based on the calculated amount of light received by the event detection pixel. A light detection method comprising the steps of:

20. an event detection unit that outputs event data representing an occurrence of an event, the event data being a change in an electrical signal detected by a plurality of event detection pixels that are arranged on a two-dimensional plane and generate an electrical signal by performing photoelectric conversion; a light reception amount calculation unit that calculates an amount of light received by the event detection pixel; a latency correction unit that corrects information relating to latency in the event data based on the amount of light received by the event detection pixel calculated by the light-receiving-amount calculation unit; programs that make a computer function.

Citation Information

Patent Citations

  • Event signal detection sensor and control method

    JP2020136958A