Light detection device, light detection method, and program

The photodetection device addresses latency issues in event signal detection sensors by using event detection pixels and a latency correction unit to adjust timestamps based on light amounts, enhancing the accuracy of event timing detection.

WO2024237022A1PCT designated stage expired Publication Date: 2024-11-21SONY SEMICON SOLUTIONS CORP
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/015731
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-12
Filing Date
2024-04-22
Publication Date
2024-11-21

AI Technical Summary

Technical Problem

Event signal detection sensors experience latency due to transfer rate constraints and circuit response delays, which cannot be effectively reduced by minimizing data transfer, leading to inaccuracies in event occurrence timing.

Method used

A photodetection device and method that includes event detection pixels arranged on a two-dimensional plane for photoelectric conversion, a received light amount calculation section to determine the light received by each pixel, and a latency correction unit that adjusts event data timestamps based on the calculated light amount to correct latency.

Benefits of technology

The solution effectively reduces latency in event signal detection by accurately correcting timestamp information in event data, aligning it with the actual event occurrence time, thereby improving the precision of event detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024015731_21112024_PF_FP_ABST
    Figure JP2024015731_21112024_PF_FP_ABST
Patent Text Reader

Abstract

The present disclosure relates to a light detection device, a light detection method, and a program that make it possible to reduce latency caused by an event signal detection sensor. The amount of light received by each of a plurality of event detection pixels, which are arranged in a two-dimensional plane and generate an electric signal by performing photoelectric conversion, is calculated on the basis of a gradation signal detected by a CMOS image sensor disposed in the vicinity of an event signal detection sensor which outputs event data indicating the occurrence of an event, i.e. a change in the electric signal, detected by the event detection pixel. Information related to event data latency is then corrected on the basis of the amount of light received by the event detection pixel. The present invention can be applied to a Hybrid EVS sensor.
Need to check novelty before this filing date? Find Prior Art

Description

Light detection device, light detection method, and program

[0001] The present disclosure relates to a light detection device, a light detection method, and a program, and more particularly to a light detection device, a light detection method, and a program that are capable of reducing latency caused by an event signal detection sensor.

[0002] An event signal detection sensor has been proposed that detects an event from the result of comparing the change in the amount of light received by a pixel per unit time with a predetermined threshold value and outputs the result as event data (see Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2020-136958

[0004] Incidentally, event signal detection sensors are known to suffer from latency, where the time of occurrence of an event recorded in the extracted event data is delayed relative to the time the actual event occurred, due to "transfer rate constraints" and "circuit response delays that depend on the amount of light received per unit time by a pixel."

[0005] Among the types of latency, latency caused by limitations on transfer rate can be reduced by reducing the amount of data to be transferred, but the reality is that other types of latency cannot be reduced.

[0006] The present disclosure has been made in view of such circumstances, and in particular, aims to reduce latency in an event signal detection sensor.

[0007] A photodetection device and a program according to one aspect of the present disclosure include an event detection unit that outputs event data representing the occurrence of an event, the event data being a change in an electrical signal detected by event detection pixels that are arranged in a two-dimensional plane and perform photoelectric conversion to generate an electrical signal; a received light amount calculation unit that calculates the amount of light received by the event detection pixels; and a latency correction unit that corrects information relating to latency in the event data based on the amount of light received by the event detection pixels calculated by the received light amount calculation unit.

[0008] An optical detection method according to one aspect of the present disclosure is an optical detection method including the steps of: outputting event data representing the occurrence of an event, the event data being a change in an electrical signal detected by event detection pixels arranged in a two-dimensional plane and performing photoelectric conversion to generate an electrical signal; calculating the amount of light received by the event detection pixels; and correcting information relating to latency in the event data based on the calculated amount of light received by the event detection pixels.

[0009] In one aspect of the present disclosure, event detection pixels are arranged in a two-dimensional plane and perform photoelectric conversion to generate electrical signals. Event data representing the occurrence of an event, which is a change in the electrical signal, is detected by the event detection pixels, the amount of light received by the event detection pixels is calculated, and latency-related information in the event data is corrected based on the calculated amount of light received by the event detection pixels.

[0010] 7 is a diagram illustrating an example configuration of a signal processing device having an event signal detection sensor. FIG. 7 is a diagram illustrating an example configuration of a pixel circuit of the event signal detection sensor of FIG. 1. FIG. 7 is a diagram illustrating event detection. FIG. 7 is a diagram illustrating the principle of latency generation. FIG. 7 is a diagram illustrating latency that changes depending on the amount of received light. FIG. 7 is a diagram illustrating a comparison between latency in an image with a large amount of received light and latency in an image with a small amount of received light. FIG. 7 is a diagram illustrating an example configuration of a signal processing device according to a first embodiment of the present disclosure. FIG. 7 is a diagram illustrating the positional relationship between a visible light sensor and the optical axis of a lens in front of the event signal detection sensor. FIG. 7 is a diagram illustrating a table of received light amounts and corresponding latency. FIG. 7 is a flowchart illustrating signal processing by the signal processing device of FIG. 7. FIG. 7 is a diagram illustrating an example configuration of a signal processing device according to a second embodiment of the present disclosure. FIG. 7 is a diagram illustrating the positional relationship between the optical axes of a collecting lens in front of a CMOS image sensor and a collecting lens in front of the event signal detection sensor. FIG. 7 is a diagram illustrating an example of applying the amount of received light of a CMOS image sensor to the amount of received light of a pixel on a corresponding event signal detection sensor. FIG. 7 is a diagram illustrating an example of estimating a spectrum using the ratio of the amount of received light of RGB pixels of a CMOS image sensor. 19 is a diagram illustrating original event data and event data in which a timestamp, which is time information, has been corrected according to a latency correction amount. FIG. 19 is a diagram illustrating original event data and event data to which time information after correction according to a latency correction amount has been added. FIG. 19 is a flowchart illustrating signal processing by the signal processing device of FIG. 11. FIG. 19 is a diagram illustrating original event data and event data to which information on the amount of received light has been added. FIG. 19 is a flowchart illustrating signal processing by the signal processing device of FIG. 19. FIG. 19 is a diagram illustrating a CMOS image sensor and a HybridEVS sensor. FIG. 19 is a diagram illustrating the relationship between a HybridEVS sensor and the optical axis of a collecting lens. FIG. 19 is a diagram illustrating a first configuration example of a signal processing device using a HybridEVS sensor according to a fourth embodiment of the present disclosure. FIG. 19 is a diagram illustrating a second configuration example of a signal processing device using a HybridEVS sensor according to the fourth embodiment of the present disclosure.FIG. 10 is a diagram illustrating a third configuration example of a signal processing device using a HybridEVS sensor according to a fourth embodiment of the present disclosure. FIG. 11 is a diagram illustrating a configuration example of a signal processing device according to a fifth embodiment of the present disclosure. FIG. 12 is a diagram illustrating an example of shifting an event data group according to latency in an application example of the present disclosure. FIG. 13 is a flowchart illustrating signal processing for realizing a modified example of the present disclosure. FIG. 14 is a diagram illustrating a first modified example of an example of shifting an event data group according to latency. FIG. 15 is a diagram illustrating a second modified example of an example of shifting an event data group according to latency. FIG. 16 is a block diagram illustrating an example of a schematic configuration of a vehicle control system. FIG. 17 is an explanatory diagram illustrating an example of installation positions of an outside-vehicle information detection unit and an imaging unit.

[0011] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0012] Hereinafter, embodiments of the present technology will be described in the following order.

[0013] 1. Overview of the present disclosure 2. First embodiment 3. Second embodiment 4. Third embodiment 5. Fourth embodiment 6. Fifth embodiment 7. Application example 8. Application example to a moving body

[0014] <<1. Overview of the Present Disclosure>> <Event Signal Detection Sensor> The present disclosure particularly relates to reducing latency in an event signal detection sensor.

[0015] First, the event signal detection sensor will be described. Fig. 1 shows an example of the configuration of a signal processing device equipped with an event signal detection sensor.

[0016] The signal processing device 1 in FIG. 1 comprises an event signal detection sensor 11 and a subsequent application 12 .

[0017] The event signal detection sensor 11 is configured as a pixel array in which a plurality of pixel circuits 21, each including a pixel 31 that performs photoelectric conversion of incident light to generate an electrical signal, are arranged in a lattice pattern on a two-dimensional plane. The event signal detection sensor 11 performs photoelectric conversion of incident light in the pixels 31 to generate an electrical signal, generates event data that indicates the occurrence of an event, which is a change in the electrical signal, and outputs the event data to a downstream application 12.

[0018] The subsequent application 12 is an application program that executes various processes based on the event data supplied from the event signal detection sensor 11 .

[0019] Here, in the event signal detection sensor 11, the portion where the plurality of pixel circuits 21 are arranged is a portion that receives incident light and performs photoelectric conversion, and is therefore also referred to as a light receiving portion.

[0020] <Configuration Example of Pixel Circuit> FIG. 2 is a block diagram showing a configuration example of the pixel circuit 21 in FIG.

[0021] The pixel circuit 21 includes a pixel 31 and an event detection unit 32 .

[0022] The pixel 31 includes a photodiode (PD) 51 as a photoelectric conversion element. The pixel 31 receives light incident on the PD 51, performs photoelectric conversion, and generates and outputs a photocurrent (Iph) as an electric signal.

[0023] When a change exceeding a predetermined threshold (including a change equal to or greater than the threshold as necessary) occurs in the photocurrent generated by photoelectric conversion of the pixel 31, the event detection unit 32 detects the change in the photocurrent as an event. The event detection unit 32 outputs event data in response to (the detection of) an event.

[0024] Here, the change in the photocurrent generated in pixel 31 can also be considered as a change in the amount of light incident on pixel 31, so the event can also be said to be a change in the amount of light in pixel 31 (a change in the amount of light exceeding a threshold value).

[0025] The event data can at least identify position information (coordinates, etc.) that indicates the position of the pixel 31 (pixel circuit 21) where a change in light amount as an event has occurred. In addition, the event data includes information that identifies a positive event or a negative event depending on the polarity (positive or negative) of the change in light amount.

[0026] For a series of event data output by the event detection unit 32 when an event is detected, time information indicating the (relative) time at which the event occurred can be identified as long as the interval between the event data is maintained as it was when the event occurred. However, if the interval between the event data is no longer maintained as it was when the event occurred, for example, due to the event data being stored in memory, the time information is lost. Therefore, time information indicating the (relative) time at which the event occurred, such as a timestamp, is added to the event data before the interval between the event data is no longer maintained as it was when the event occurred. The process of adding time information to the event data may be performed by the event detection unit 32 or external to the event detection unit 32, as long as it is performed before the interval between the event data is no longer maintained as it was when the event occurred.

[0027] The event detection unit 32 includes a current-voltage conversion unit 41 , a difference detection unit 42 , and a comparator 43 .

[0028] The current-voltage converter 41 converts the photocurrent from the pixel 31 into a voltage Vo (hereinafter also referred to as photovoltage) corresponding to the logarithm of the photocurrent, and outputs it to the difference detector 42 .

[0029] The difference detection unit 42 calculates the difference between the current optical voltage Vo from the current-voltage conversion unit 41 and the optical voltage at a timing that differs from the current by a small amount of time, and outputs the difference signal Vout to the comparator 43.

[0030] The comparator 43 compares the difference signal Vout output by the difference detection unit 42 with predetermined thresholds (voltages) +Vth and −Vth used for detecting an event. If the difference signal Vout is equal to or greater than the threshold +Vth or equal to or less than the threshold −Vth, the comparator 43 determines that a change in light amount as an event has been detected (occurred), and outputs event data.

[0031] For example, when the differential signal Vout is equal to or greater than the threshold +Vth, the comparator 43 determines that a positive event has been detected and outputs event data of a positive event. On the other hand, when the differential signal Vout is equal to or less than the threshold −Vth, the comparator 43 determines that a negative event has been detected and outputs event data of a negative event.

[0032] <Regarding latency of the event signal detection sensor> As described above, the event signal detection sensor 11 detects a change in the amount of light received by the pixel 31 as an event, and outputs the coordinates of the pixel at that time, the time, and information indicating whether it is a positive event or a negative event as event data.

[0033] When the event signal detection sensor 11 outputs event data, if latency occurs relative to the timing at which the event actually occurred, a delay occurs in the timing at which the event is detected, and the time information of the event data is set to a time delayed relative to the time at which the event actually occurred.

[0034] For example, consider the case shown in the left part of Figure 3 where an area containing a white object against a gray background moves leftward from area WZ1 shown by the dotted line to area WZ2, as indicated by the arrow.

[0035] In this case, at the boundary BZ2, which is the left end of the white object area WZ2, a positive event is detected as the amount of received light increases due to a change from gray to white, as shown in the upper center of Figure 3.

[0036] On the other hand, at the boundary BZ1, which is the right end of the white object area WZ2, a negative event is detected as the amount of received light decreases due to a change from white to gray, as shown in the upper right part of Figure 3.

[0037] A negative event detected at boundary BZ1 and a positive event detected at boundary BZ2 are detected at the same time, but due to a delay (latency), information about a time that is delayed from the time the event actually occurred is recorded in the event data.

[0038] In other words, even if the positive event and negative event shown in the upper center and upper right corner of Figure 3 actually occurred at time t11, time t12, which is delayed by, for example, a predetermined delay time, latency TL1, is recorded as the time information of the event data.

[0039] As a result, if a white object moves from area WZ1, where a white object exists, as shown in the top row of Figure 4 at time T1, to area WZ2, as shown in the second row from the top of Figure 4 at time T2, a positive event will be detected in area Ze1, as shown in the third row from the top of Figure 4, if there is no latency and the time at which the event actually occurs is properly reflected.

[0040] However, due to the influence of latency, time information that is later than the time when the actual event occurred is recorded, and the area in which the positive event is detected becomes the past area Ze2, which is shifted to the right in the figure by a distance Δd corresponding to the latency TL1 from the actual area Ze1, as shown in the bottom row of Figure 4.

[0041] Therefore, it is conceivable to correct the time information of the event data by subtracting the latency T1, which is a predetermined delay time.

[0042] However, it is known that the latency is shorter when the amount of received light is greater, and conversely, the latency is longer when the amount of received light is less. Therefore, subtracting the delay time due to latency from the time information as a specific delay time does not necessarily result in an appropriate correction.

[0043] That is, for example, when a positive event occurs in a state where the amount of received light is greater than in the case of Figure 3, as shown in the left part of Figure 5, when the positive event actually occurs at time t31, the time information of the event data is set to time t32, which is delayed by latency TL11, which is shorter than latency TL1, relative to time t31.

[0044] In contrast to this, for example, if a positive event occurs in a state where the amount of received light is low compared to the case in Figure 3, when the positive event actually occurs at time t41, as shown in the right part of Figure 5, the time information of the event data will be time t42, which is delayed from time t41 by latency TL12, which is longer than latency TL1.

[0045] Therefore, for example, as shown in the upper left part of Figure 6, the latency LL that occurs when a foreground area LG2 that is lighter gray than the background area LG1 moves over a background area LG1 that is represented in light gray and has a relatively high amount of received light is smaller than the latency DL that occurs when a foreground area DG2 that is lighter gray than the background area DG1 moves over a background area DG1 that is represented in dark gray and has a relatively low amount of received light, as shown in the upper right part of Figure 6.

[0046] That is, since the length of latency changes depending on the amount of received light, it is not possible to appropriately reduce the effect of latency by subtracting a specific time from the time information.

[0047] Therefore, in the present disclosure, the amount of received light detected by the event signal detection sensor is detected, and the time information in the event data is corrected by the amount of latency corresponding to the detected amount of received light.

[0048] This makes it possible to correct the latency according to the amount of received light, and to reduce the effect of the latency appropriately according to the amount of received light.

[0049] As a result, for example, it is possible to achieve a correction such that the area in which a positive event is detected in Figure 4 approaches area Za1 from area Za2, and the effects of latency can be appropriately reduced according to the amount of received light.

[0050] <<2. First embodiment>> Next, a configuration example of a signal processing device according to the present disclosure will be described with reference to Fig. 7. The signal processing device 101 in Fig. 7 includes an event signal detection sensor 111, a visible light sensor 112, a subsequent-stage ISP (Image Sensor Processor) / AP (Application Processor) 113, and a subsequent-stage application 114.

[0051] The event signal detection sensor 111 has the same configuration as the event signal detection sensor 11 in Figure 1, and detects a positive event or a negative event according to changes in the amount of received light, and outputs event data corresponding to the detection result to the downstream ISP / AP 113.

[0052] The visible light sensor 112 is a sensor that detects the amount of visible light received by the event signal detection sensor 111 , and outputs a sensor output signal corresponding to the detected amount of received light to the subsequent stage ISP / AP 113 .

[0053] More specifically, as shown in FIG. 8, the visible light sensor 112 is disposed in front of the event signal detection sensor 111 and is not on the optical axis AX1 of the lens 141 that focuses the incident light onto the event signal detection sensor 111, but is disposed relatively close to the optical axis AX1.

[0054] With this arrangement, the visible light sensor 112 detects an amount of received light that is considered to be approximate to the amount of received light of the event signal detection sensor 111, and outputs the amount as a sensor output signal to the downstream ISP / AP 113. Since the visible light sensor 112 detects a sensor output signal that indicates the amount of received light at one point in the vicinity of the optical axis AX1, the amount of received light calculated based on the sensor output signal is calculated assuming that all pixels comprising the event signal detection sensor 111 receive the same amount of light.

[0055] The subsequent stage ISP / AP 113 is composed of a processor made up of hardware, and calculates the amount of light received by the event signal detection sensor 111 based on the sensor output signal from the visible light sensor 112, and determines the latency correction amount for the event data according to the calculated amount of light received.

[0056] Then, the subsequent stage ISP / AP 113 corrects the latency of the event data from the event signal detection sensor 111 using the identified latency correction amount, and outputs the corrected event data to the subsequent stage application 114 .

[0057] At this time, the subsequent stage ISP / AP 113 outputs the sensor output signal supplied from the visible light sensor 112 to the subsequent stage application 114 .

[0058] More specifically, the subsequent stage ISP / AP 113 includes an event detection pixel received light amount calculation unit 121 and a latency correction unit 122 .

[0059] The event detection pixel light reception amount calculation unit 121 calculates the amount of light received per unit time of the event detection pixels (corresponding to pixels 31 in Figure 2) that make up the event signal detection sensor 111 based on the sensor output signal, and outputs the amount of light received per unit time to the latency correction unit 122.

[0060] The event detection pixel received light amount calculation unit 121 calculates the amount of light received per unit time by the event detection pixels that make up the event signal detection sensor 111, for example, the number of photons per time incident on one event detection pixel that makes up the event signal detection sensor 111, irradiance [W / m2], or illuminance [Lux], based on the sensor output signal.

[0061] 7 and 8 show an example in which one visible light sensor 112 is provided for one event signal detection sensor, but a configuration in which multiple visible light sensors 112 are provided for one event signal detection sensor is also possible. In a case in which multiple visible light sensors 112 are provided for one event signal detection sensor, the amount of received light for each pixel may be calculated by interpolation from the positional relationship between each of the multiple visible light sensors 112 and the event signal detection sensor 111.

[0062] The latency correction unit 122 determines the latency based on the amount of light received per unit time of the pixels that make up the event signal detection sensor 111, which is supplied from the event detection pixel light reception amount calculation unit 121, corrects the timestamp, which is the time information included in the event data, using the determined latency as a correction amount (hereinafter also referred to as the latency correction amount), and outputs the event data with the corrected timestamp as corrected event data.

[0063] In this embodiment, when the event signal detection sensor 111 generates event data, it outputs the event data with a timestamp, which is time information. However, the latency correction unit 122 may also add a timestamp to the event data output by the event signal detection sensor 111 and further add a correction according to the latency.

[0064] More specifically, the latency correction unit 122 stores a table in which the latency for each event corresponding to the amount of received light is registered, as shown in FIG. 9, for example, and reads out the latency near the amount of received light supplied from the event detection pixel received light amount calculation unit 121 and calculates the corresponding latency by interpolation.

[0065] In the table of FIG. 9, from the left, the amount of received light, the latency in detecting a rising edge which is a positive event, and the latency in detecting a falling edge which is a negative event are registered.

[0066] In FIG. 9, it is registered that when the amount of received light is 1000, the latency is A seconds if it is a positive event, and W seconds if it is a negative event.

[0067] Also, when the amount of received light is 100, it is registered that the latency is B seconds when it is a positive event, and X seconds when it is a negative event.

[0068] Furthermore, it is registered that when the amount of received light is 10, the latency is C seconds if it is a positive event, and Y seconds if it is a negative event.

[0069] Furthermore, when the amount of received light is 1, it is registered that the latency is D seconds if it is a positive event, and Z seconds if it is a negative event.

[0070] In the table shown in FIG. 9, latencies obtained in advance through experiments, simulations, etc., for positive events and negative events corresponding to the amounts of received light are registered.

[0071] For example, if the amount of received light is 500 and it is a positive event, the latency correction unit 122 reads from the table A seconds in a positive event when the amount of received light is 1000, which is close to 500, and B seconds in a positive event when the amount of received light is 100, calculates the latency when the amount of received light is 500 by interpolation using, for example, the logarithmic mean, sets the calculation result as the latency correction amount, and corrects the timestamp, which is the time information, by the latency correction amount.

[0072] The subsequent application 114 is composed of software such as an application program, and executes signal processing based on the event data and sensor output signals after the latency has been corrected.

[0073] In the signal processing device 101 of FIG. 7 , as described above, the visible light sensor 112 outputs a sensor output signal corresponding to the amount of received light, the amount of received light at the event detection pixel is calculated based on the sensor output signal, and the timestamp is corrected based on the calculated amount of received light at the event detection pixel and on a latency correction amount corresponding to the amount of received light that has been measured in advance by an experiment or the like.

[0074] As a result, a latency correction amount corresponding to the amount of light received by the event signal detection sensor 111 is identified, and the timestamp, which is the time information, is corrected by the identified latency correction amount, thereby making it possible to appropriately reduce the effects of latency.

[0075] <Signal Processing by the Signal Processing Device of FIG. 7> Next, signal processing by the signal processing device 101 of FIG. 7 will be described with reference to the flowchart of FIG.

[0076] In step S31, the event signal detection sensor 111 determines whether or not an event has been detected. If it is determined in step S31 that an event has been detected, the process proceeds to step S32.

[0077] In step S32 , the event signal detection sensor 111 outputs the event data to the latency correction unit 122 of the subsequent stage ISP / AP 113 .

[0078] In step S33, the visible light sensor 112 outputs a sensor output signal corresponding to the amount of received visible light to the event detection pixel received light amount calculation unit 121 of the subsequent-stage ISP / AP 113. At this time, the sensor output signal is also output to the subsequent-stage application 114.

[0079] In step S34, the event detection pixel light reception amount calculation unit 121 calculates the amount of light received per unit time of the event detection pixels that make up the event signal detection sensor 111 based on the sensor output signal corresponding to the amount of received visible light, and outputs the calculated amount of received light to the latency correction unit 122.

[0080] In step S35, the latency correction unit 122 reads out the latency registered in association with the amount of received light from the stored table, and specifies the latency correction amount by interpolating as necessary.

[0081] In step S36 , the latency correction unit 122 corrects the timestamp, which is the time information registered in the event data, by the latency correction amount and outputs the corrected timestamp to the subsequent application 114 .

[0082] In step S37, the subsequent application 114 executes signal processing based on the corrected event data and the sensor output signal, and outputs the execution result.

[0083] If no event is detected in step S31, steps S32 to S37 are skipped.

[0084] In step S38, it is determined whether or not an instruction to end the process has been given. If an instruction to end the process has not been given, the process returns to step S31, and the subsequent steps are repeated.

[0085] Then, in step S38, if an instruction to end the process is given, the process ends.

[0086] Through the above process, the latency correction amount is determined according to the amount of received light, and the timestamp, which is the time information of the event data, is corrected by the determined latency correction amount.

[0087] As a result, an appropriate latency correction amount is set for the latency that changes depending on the amount of received light, and the time information is corrected, so that the effect of latency can be appropriately reduced.

[0088] <<3. Second embodiment>> In the above, an example has been described in which the visible light sensor 112 is provided in the vicinity of the event signal detection sensor 111, the amount of light received by the event signal detection sensor 111 is calculated based on the sensor output signal of the visible light sensor 112, a latency correction amount is obtained in accordance with the calculated amount of light received, and a timestamp, which is time information, is corrected by the latency correction amount in accordance with the amount of light received.

[0089] However, since the amount of received light differs for each pixel constituting the event signal detection sensor 111, the latency of the event data differs for each pixel constituting the event signal detection sensor 111.

[0090] Therefore, instead of the visible light sensor 112, a CMOS (Complementary Metal Oxide Semiconductor) image sensor may be provided, and the amount of light received by each pixel in the event signal detection sensor 111 may be determined from the gradation signal of each pixel, and the latency may be corrected on a pixel-by-pixel basis.

[0091] FIG. 11 shows an example of the configuration of a signal processing device in which a CMOS image sensor is provided instead of the visible light sensor 112, the amount of light received by each pixel in the event signal detection sensor 111 is identified from the gradation signal of each pixel, and latency is corrected on a pixel-by-pixel basis.

[0092] In the signal processing device 101A in Fig. 11, components having the same functions as those in the signal processing device 101 in Fig. 7 are given the same reference numerals, and descriptions thereof will be omitted as appropriate. Also, in the signal processing device 101A in Fig. 11, components having similar functions to those in the signal processing device 101 in Fig. 7 are given the same reference numerals with the letter "A" added to the end.

[0093] The signal processing device 101A in FIG. 11 differs from the signal processing device 101 in FIG. 7 in that a CMOS image sensor 151 is provided instead of the visible light sensor 112, and a subsequent ISP / AP 113A is provided instead of the subsequent ISP / AP 113.

[0094] The CMOS image sensor 151 is made up of a plurality of pixels arranged in an array perpendicular to the direction of incidence of incident light, and outputs a gradation signal corresponding to the amount of incident light to the downstream ISP / AP 113 in units of pixels.

[0095] 12, a lens 141A-2 that focuses incident light is provided in front of the CMOS image sensor 151. In addition, a lens 141A-1 that focuses incident light onto the event signal detection sensor 111 is also provided in front of the event signal detection sensor 111.

[0096] Although the optical axes AX11 and AX12 of the lenses 141A-1 and 141A-2 are not the same, they are relatively close to each other and are arranged so as to be parallel. Therefore, although parallax occurs between each pixel of the CMOS image sensor 151 and each pixel of the event signal detection sensor 111 for objects within a relatively close range from both, it can be considered that the gradation signal at each pixel of the CMOS image sensor 151 generally corresponds to the amount of light received by each pixel of the event signal detection sensor 111.

[0097] Therefore, the CMOS image sensor 151 outputs a gradation signal in units of pixels to the subsequent ISP / AP 113A as information for detecting the amount of received light that can be considered to be approximate to the amount of received light of each pixel of the event signal detection sensor 111. In other words, the gradation signal indicating the amount of received light output from the CMOS image sensor 151 is output as information indicating the amount of received light of a pixel at a corresponding position configured in the event signal detection sensor 111.

[0098] The subsequent stage ISP / AP 113A has the same basic functions as the subsequent stage ISP / AP 113 in Figure 7, but it calculates the amount of light received in units of each pixel in the event signal detection sensor 111 based on the event data from the event signal detection sensor 111 and the gradation signal from the CMOS image sensor 151, and specifies a latency correction amount in units of pixels corresponding to the calculated amount of light received.

[0099] Then, the subsequent stage ISP / AP 113A corrects the latency of the event data using the latency correction amount specified for each pixel, and outputs the corrected event data in units of pixels to the subsequent stage application 114.

[0100] At this time, the subsequent stage ISP / AP 113 A outputs the sensor output signal supplied from the visible light sensor 112 to the subsequent stage application 114 .

[0101] More specifically, the subsequent stage ISP / AP 113A includes an event detection pixel received light amount calculation unit 121A and a latency correction unit 122A.

[0102] The event detection pixel light reception amount calculation unit 121A has the same functions as the event detection pixel light reception amount calculation unit 121, but calculates the amount of light received per unit time for each event detection pixel that constitutes the event signal detection sensor 111 based on a gradation signal in pixel units, and outputs the amount of light received per unit time to the latency correction unit 122A.

[0103] The event detection pixel received light amount calculation unit 121A calculates the amount of received light per unit time for each of the event detection pixels that make up the event signal detection sensor 111 based on the gray scale signal of each pixel of the CMOS image sensor 151.

[0104] For example, the amount of light received per pixel per unit time is expressed as irradiance [W / m 2 ], it is calculated using the following procedure.

[0105] First, the amount of charge per unit time for each pixel is calculated using the following equation (1).

[0106] Qu=(Q_after-gain / gain) / Te...(1)

[0107] Here, Qu is the amount of charge per unit time in units of pixels, Q_after-gain is the amount of charge after gain is applied in units of pixels of the CMOS image sensor 151, and is expressed by the following formula (2): Gain is the value of the gain applied, and is expressed by the following formula (3):

[0108] Q_after-gain=(Vs-OPB) / slope slope=((2 10 -1) -OPB) / Qs...(2)

[0109] gain=10^(Vg[dB] / 20) Vg[dB]=20log(gain)...(3)

[0110] Here, Vs is the value of the gradation signal, OPB is the gradation signal of a light-shielded pixel called optical black (generally 64), Qs is the saturated charge amount of the pixel, and Vg is the gain value [dB]. Also, "^" represents exponentiation.

[0111] Then, when the spectrum of the incident light is known, the event detection pixel received light amount calculation unit 121A calculates the amount of received light per unit time for each pixel as irradiance [W / m] by calculating the following equation (4) based on the calculated charge amount per unit time for each pixel. 2 ] is calculated as follows.

[0112] IRR=Qu1 / Qg...(4)

[0113] Here, IRR is the amount of light received per pixel per unit time expressed as irradiance [W / m 2 ], Qu1 is the amount of charge generated in a pixel per second, and Qg is 1 [W / m 2 ] is the amount of charge that can be generated by photoelectric conversion in one second when light with an irradiance of

[0114] Furthermore, 1[W / m 2 The amount of charge Qg that can be generated in one second by photoelectric conversion from light with an irradiance of [ ] must be determined in advance by experiment or theoretical calculation depending on the spectrum.

[0115] Then, the event detection pixel light reception amount calculation unit 121A calculates the amount of light received per unit time for each pixel of the CMOS image sensor 151, and then performs coordinate conversion for each pixel to the pixel position on the event signal detection sensor 111 corresponding to that pixel, to determine the amount of light received by the corresponding pixel.

[0116] More specifically, as shown in FIG. 13, the event detection pixel light reception amount calculation unit 121A calculates the amount of light received per unit time of pixel PC of the CMOS image sensor 151, then performs coordinate conversion on the corresponding pixel PE on the event signal detection sensor 111, and applies the amount of light received by the coordinate conversion to the pixel PE.

[0117] Furthermore, in the above, it is assumed that the spectrum is known. However, if the CMOS image sensor 151 is composed of RGB pixels, the event detection pixel received light amount calculation unit 121A may predict a spectrum for each pixel of the event signal detection sensor 111 from the ratio between the amounts of charge of the RGB pixels, and calculate the amount of received light per unit time for each pixel by performing the calculations of the above-mentioned equations (1) to (4) based on the predicted spectrum for each pixel.

[0118] For example, if the amount of charge in the B pixel is greater than the amount of charge in the G pixel and the R pixel, it is highly likely that the spectrum will have a peak in the short wavelength region, and therefore, for example, a spectral distribution such as that shown by the solid line in the waveform shown in Figure 14 is predicted.

[0119] Furthermore, for example, if the amount of charge in the R pixel is greater than the amount of charge in the G pixel and the B pixel, it is highly likely that the spectrum will have a peak in the long wavelength region, and therefore, for example, a spectral distribution such as that shown by the dotted line in the waveform shown in Figure 14 is predicted.

[0120] The event detection pixel received light amount calculation unit 121A may estimate the spectrum from the ratio of the charge amounts of the R, G, and B pixels using the method described with reference to FIG.

[0121] In FIG. 14, the vertical axis represents intensity and the horizontal axis represents wavelength, and an example of the waveform of the spectral distribution is shown as a dotted line and a solid line.

[0122] The latency correction unit 122A has the same basic functions as the latency correction unit 122, but it determines the latency correction amount for each pixel based on the amount of light received per unit time for each pixel that constitutes the event signal detection sensor 111, which is supplied from the event detection pixel light reception amount calculation unit 121A, corrects the timestamp, which is the time information included in the event data, for each pixel according to the determined latency correction amount for each pixel, and outputs the event data with the timestamp corrected for each pixel as corrected event data.

[0123] That is, while the latency correction unit 122 corrects the time information of the event data using a latency correction amount common to all pixels, the latency correction unit 122A corrects the time information of the event data for each pixel using a latency correction amount for each pixel.

[0124] For example, as shown in the left part of Figure 15, in the event data that shows, from left to right, a timestamp which is time information in units of pixels, coordinate information which represents the pixel position, and information on the event type which indicates a positive event (rising edge) or a negative event (falling edge), for example, when the latency is 1 second, the latency correction unit 122A corrects the time information as shown in the right part of Figure 15.

[0125] In the event data on the left side of FIG. 15 , the top row indicates that a rising edge, which is a positive event, was detected at the pixel with coordinate position (x, y) = (100, 200) at time information 00:00:10. The second row indicates that a rising edge, which is a positive event, was detected at the pixel with coordinate position (x, y) = (200, 200) at time information 00:00:10. The third row indicates that a rising edge, which is a positive event, was detected at the pixel with coordinate position (x, y) = (100, 200) at time information 00:00:11. The fourth row indicates that a rising edge, which is a positive event, was detected at the pixel with coordinate position (x, y) = (200, 200) at time information 00:00:11.

[0126] In addition, in the event data on the right side of FIG. 15 , the top row indicates that a rising edge, which is a positive event, was detected at the pixel with coordinate position (x, y) = (100, 200) at time information 00:00:09. The second row indicates that a rising edge, which is a positive event, was detected at the pixel with coordinate position (x, y) = (200, 200) at time information 00:00:09. The third row indicates that a rising edge, which is a positive event, was detected at the pixel with coordinate position (x, y) = (100, 200) at time information 00:00:10. The fourth row indicates that a rising edge, which is a positive event, was detected at the pixel with coordinate position (x, y) = (200, 200) at time information 00:00:10.

[0127] That is, the time information in the original event data shown on the left side of FIG. 15 is corrected in the right side of FIG. 15 by subtracting one second, which is the latency correction amount.

[0128] Furthermore, in the above description, an example has been described in which the time information in the event data is corrected by the latency correction amount, but the corrected time information may also be added.

[0129] For example, corrected time information may be added to the event data shown in the left part of FIG. 16, as shown in the right part of FIG.

[0130] The left part of Figure 16 is the same as the left part of Figure 15, and the right part of Figure 16 has a column for corrected time information added between the time information column and the coordinate information column in the right part of Figure 15.

[0131] Since the amount of latency correction varies on a pixel-by-pixel basis, the column for corrected time information on the right side of Figure 16 can be replaced with a column for recording the delay time corresponding to the latency, i.e., the amount of latency correction, and the original time information can actually be corrected by the amount of latency correction before use.

[0132] <Signal Processing by the Signal Processing Device of FIG. 11> Next, signal processing by the signal processing device 101A of FIG. 11 will be described with reference to the flowchart of FIG.

[0133] In step S51, the event signal detection sensor 111 determines whether or not an event has been detected. If it is determined in step S51 that an event has been detected, the process proceeds to step S52.

[0134] In step S52, the event signal detection sensor 111 outputs the event data to the latency correction unit 122A of the subsequent stage ISP / AP 113A.

[0135] In step S53, the CMOS image sensor 151 outputs a gradation signal for each pixel to the event detection pixel received light amount calculation unit 121A of the subsequent stage ISP / AP 113A. At this time, the gradation signal is also output to the subsequent stage application 114.

[0136] In step S54, the event detection pixel light reception amount calculation unit 121A calculates the amount of light received per unit time, measured in pixels, of the event detection pixels that make up the event signal detection sensor 111 based on the gradation signal measured in pixels, and outputs the calculated amount of light received per pixel to the latency correction unit 122A.

[0137] In step S55, the latency correction unit 122A reads out the latency registered for each pixel in correspondence with the amount of received light from the stored table, and determines the latency correction amount for each pixel by interpolating as necessary.

[0138] In step S56, the latency correction unit 122A corrects the time stamp, which is the time information registered in the event data, by the latency correction amount for each pixel and outputs the corrected time stamp to the subsequent application 114.

[0139] In step S57, the subsequent application 114 executes signal processing based on the corrected event data and the gradation signal, and outputs the execution result.

[0140] If no event is detected in step S51, the processes of steps S52 to S57 are skipped.

[0141] In step S58, it is determined whether or not an instruction to end the process has been given. If an instruction to end the process has not been given, the process returns to step S51, and the subsequent steps are repeated.

[0142] Then, in step S58, if an instruction to end the process is given, the process ends.

[0143] Through the above processing, the latency correction amount for each pixel is determined according to the amount of light received for each pixel, and the timestamp, which is the time information of the event data for each pixel, is corrected by the determined latency correction amount for each pixel.

[0144] As a result, an appropriate latency correction amount is set for the latency, which varies depending on the amount of light received per pixel, and the time information is corrected per pixel, making it possible to appropriately reduce the effects of latency.

[0145] <<4. Third Embodiment>> In the above, an example has been described in which the timestamp, which is the time information in the event data, is corrected by the latency correction amount corresponding to the amount of received light. However, it is also possible to add only the information on the amount of received light to the event data, and have the downstream application 114 that acquires the event data correct the latency according to the amount of received light.

[0146] That is, for example, original event data such as that shown in the left part of Fig. 18 is attached with information on the amount of received light as shown in the right part of Fig. 18 and output to the subsequent application 114. The subsequent application 114 corrects latency based on the event data with the amount of received light attached, and then executes processing.

[0147] The left part of Figure 18 is original event data similar to the left parts of Figures 15 and 16, and the right part of Figure 18 has a column for the amount of light received at the corresponding pixel between a column for timestamp, which is the time information in the left part of Figure 18, and a column for coordinate information, which indicates the pixel position.

[0148] In the column of the amount of received light on the right side of FIG. 18, 1111, 1111, 1112, and 1112 are registered as the amounts of received light in order from the top.

[0149] That is, the top row on the right side of FIG. 18 shows that at time information 00:00:10, a rising edge, which is a positive event, was detected at the pixel at coordinate position (x, y) = (100, 200), and the amount of received light was 1111. The second row from the top shows that at time information 00:00:10, a rising edge, which is a positive event, was detected at the pixel at coordinate position (x, y) = (200, 200), and the amount of received light was 1111. The third row from the top shows that at time information 00:00:11, a rising edge, which is a positive event, was detected at the pixel at coordinate position (x, y) = (100, 200), and the amount of received light was 1112. The fourth row from the top shows that at time information 00:00:11, a rising edge, which is a positive event, was detected at the pixel at coordinate position (x, y) = (200, 200), and the amount of received light was 1112.

[0150] The subsequent application 114 calculates a latency correction amount corresponding to the amount of received light based on the event data corrected by adding information about the amount of received light, corrects the timestamp, which is time information, and executes processing.

[0151] <Configuration Example of Signal Processing Device in Which Received Light Amount is Registered in Event Data> FIG. 19 shows a configuration example of a signal processing device in which the received light amount is registered in event data.

[0152] In the signal processing device 101B of Fig. 19, components having the same functions as those of the signal processing device 101 of Fig. 7 and the signal processing device 101A of Fig. 11 are given the same reference numerals, and descriptions thereof will be omitted as appropriate. Also, in the signal processing device 101B of Fig. 19, components having similar functions to those of the signal processing device 101 of Fig. 7 and the signal processing device 101A of Fig. 11 are given the same reference numerals with the letter "B" added to the end.

[0153] The signal processing device 101B in FIG. 19 differs from the signal processing device 101 in FIG. 7 and the signal processing device 101A in FIG. 11 in that a subsequent stage ISP / AP 113B and a subsequent stage application 114B are provided instead of the subsequent stage ISP / APs 113 and 113A and the subsequent stage application 114.

[0154] More specifically, the subsequent stage ISP / AP 113B includes an event detection pixel received light amount calculation unit 121B and a latency correction unit 122B.

[0155] The event detection pixel light reception amount calculation unit 121B has the same functions as the event detection pixel light reception amount calculation unit 121A, and calculates the amount of light received per unit time for each event detection pixel that constitutes the event signal detection sensor 111 and outputs it to the latency correction unit 122B.

[0156] The latency correction unit 122B adds the amount of light received for each event detection pixel to the event data supplied from the event signal detection sensor 111, as described with reference to Figure 18, and outputs the corrected event data to the subsequent application 114B.

[0157] That is, unlike the latency correction units 122 and 122A, the latency correction unit 122B does not correct the time stamp, which is the time information included in the event data, but simply adds information on the amount of received light and outputs it.

[0158] It should be noted that the latency correction unit 122B does not correct the timestamp, which is time information, but it does provide information on the amount of received light that is necessary to correct the timestamp, which is time information for the event data, and therefore is considered to perform the function of correcting the event data.

[0159] The latter application 114B has the same basic functions as the latter application 114, but furthermore, based on the event data to which information on the amount of received light is attached, corrects the timestamp, which is the time information, by the latency correction amount and executes the processing.

[0160] More specifically, the subsequent-stage application 114B includes a latency correction unit 171. The latency correction unit 171 includes a table (similar to the table in FIG. 9 ) consisting of received light amounts and corresponding latencies (latency correction amounts), and reads out the corresponding latency correction amount from the table based on the received light amount information included in the event data, and corrects the timestamp, which is time information.

[0161] <Signal Processing by the Signal Processing Device of FIG. 19> Next, signal processing by the signal processing device 101B of FIG. 19 will be described with reference to the flowchart of FIG.

[0162] In step S71, the event signal detection sensor 111 determines whether or not an event has been detected. If it is determined in step S71 that an event has been detected, the process proceeds to step S72.

[0163] In step S72, the event signal detection sensor 111 outputs the event data to the latency correction unit 122B of the subsequent stage ISP / AP 113B.

[0164] In step S73, the CMOS image sensor 151 outputs a gradation signal for each pixel to the event detection pixel received light amount calculation unit 121B of the subsequent stage ISP / AP 113B. At this time, the gradation signal is also output to the subsequent stage application 114B.

[0165] In step S74, the event detection pixel light reception amount calculation unit 121B calculates the amount of light received per unit time, measured in pixels, of the event detection pixels that make up the event signal detection sensor 111 based on the gradation signal measured in pixels, and outputs the calculated amount of light received per pixel to the latency correction unit 122B.

[0166] In step S75, the latency correction unit 122B corrects the event data by adding information on the amount of received light to the event data for each pixel.

[0167] In step S76, the latency correction unit 122B outputs the event data corrected by adding the information on the amount of received light for each pixel to the subsequent application 114B.

[0168] In step S77, the subsequent application 114B acquires the event data corrected by adding the information on the amount of received light for each pixel.

[0169] In step S78, the latency correction unit 171 of the subsequent application 114B reads information on the amount of received light for each pixel from the event data, and searches for and identifies the corresponding latency correction amount from a table in which latencies for received light amounts are recorded.

[0170] In step S79, the latency correction unit 171 corrects the time stamp, which is time information, of the event data for each pixel by the latency correction amount.

[0171] In step S80, the subsequent application 114B executes signal processing based on the event data, the time stamp of which is time information, corrected, and the tone signal, and outputs the execution result.

[0172] If no event is detected in step S71, the processes of steps S72 to S80 are skipped.

[0173] In step S81, it is determined whether or not an instruction to end the process has been given. If an instruction to end the process has not been given, the process returns to step S71, and the subsequent steps are repeated.

[0174] Then, in step S81, if an instruction to end the process is given, the process ends.

[0175] In the above processing, the latency correction amount for each pixel is determined according to the amount of light received for each pixel, and the timestamp, which is the time information of the event data for each pixel, is corrected by the determined latency correction amount for each pixel.

[0176] As a result, an appropriate latency correction amount is set for the latency, which varies depending on the amount of light received per pixel, and the time information is corrected per pixel, making it possible to appropriately reduce the effects of latency.

[0177] <<5. Fourth Embodiment>> In the above, configuration examples of the signal processing devices 101, 101A, and 101B using the CMOS image sensor 151 and the event signal detection sensor 111 have been described, but a HybridEVS (EVS: Event-based Vision Sensor) sensor that has the functions of both the CMOS image sensor 151 and the event signal detection sensor 111 may also be used.

[0178] The HybridEVS sensor has, for example, a pixel structure as shown in the right part of Fig. 21. The left part of Fig. 21 shows the pixel configuration of a CMOS image sensor 151, and indicates the difference in configuration from the HybridEVS 201 in the right part of Fig. 21.

[0179] That is, the CMOS image sensor 151 has a pixel arrangement in which RGB pixel units are arranged in a Bayer array, with 2 pixels x 2 pixels as one pixel unit, as shown in the left part of FIG. 21, for example.

[0180] In contrast, the HybridEVS 201 is configured such that two of the four pixels that make up the R pixel unit and the B pixel unit in the CMOS image sensor 151 that are aligned vertically are arranged in place of the event detection pixel EVS (corresponding to pixel 31 in Figure 2) in the event signal detection sensor 111.

[0181] With this configuration, in the hybrid EVS sensor 201, grayscale signals are acquired from the RGB pixels in the same manner as in the configuration of the CMOS image sensor 151, and event data is acquired from the event detection pixels EVS.

[0182] 22, the HybridEVS sensor 201 is configured such that the condenser lens 141B is provided at the front end, so that the gradation signal and event data acquired by the HybridEVS 201 can be acquired while maintaining an appropriate positional relationship with respect to the axis AX21 of the lens 141B so that no parallax or the like occurs due to the arrangement. Note that the pixel arrangement of the HybridEVS 201 on the right side of FIG. 21 is an example, and a pixel arrangement other than the illustrated pixel arrangement may be used.

[0183] <First Configuration Example of Signal Processing Device Using HybridEVS> Fig. 23 shows a first configuration example of a signal processing device using a HybridEVS sensor. A signal processing device 101C in Fig. 23 is configured by a HybridEVS sensor 201 and a subsequent application 114.

[0184] The HybridEVS sensor 201 is composed of an event data acquisition unit 221 , a gradation signal acquisition unit 222 , an event detection pixel received light amount calculation unit 223 , and a latency correction unit 224 .

[0185] The event data acquisition unit 221 acquires the event data output from the event detection pixel EVS in FIG. 21 and outputs it to the latency correction unit 224 .

[0186] The gradation signal acquisition unit 222 acquires the gradation signals output from the RGB pixels in FIG. 21 and outputs them to the event detection pixel received light amount calculation unit 223 .

[0187] The event data acquisition unit 221 functions in the same manner as the event signal detection sensor 111. The gradation signal acquisition unit 222 functions in the same manner as the CMOS image sensor 151.

[0188] The event detection pixel received light amount calculation unit 223 and the latency correction unit 224 function as the event detection pixel received light amount calculation unit 121A or 121B and the latency correction unit 122A or 122B, respectively.

[0189] That is, the signal processing device 101C in FIG. 23 can function substantially as the signal processing device 101A in FIG. 11 or the signal processing device 101B in FIG.

[0190] The signal processing is the same for each, so the description thereof will be omitted.

[0191] <Second configuration example of a signal processing device using HybridEVS> The above has described a configuration example in which the HybridEVS sensor 201 is equipped with an event data acquisition unit 221, a gradation signal acquisition unit 222, an event detection pixel received light amount calculation unit 223, and a latency correction unit 224, but a configuration in which only the event data acquisition unit 221 and the gradation signal acquisition unit 222 are provided may also be used.

[0192] FIG. 24 shows an example of the configuration of a signal processing device in which only an event data acquisition unit 221 and a gradation signal acquisition unit 222 are provided in a HybridEVS sensor 201.

[0193] The signal processing device 101D in FIG. 24 is composed of a HybridEVS sensor 201D, a subsequent stage ISP / AP 241, and a subsequent stage application 114.

[0194] The HybridEVS sensor 201D has the same basic functions as the HybridEVS sensor 201, but is only equipped with an event data acquisition unit 221D and a gradation signal acquisition unit 222D that have the same functions as the event data acquisition unit 221 and the gradation signal acquisition unit 222, respectively.

[0195] The subsequent stage ISP / AP 241 is provided with an event detection pixel received light amount calculation unit 223D and a latency correction unit 224D, which have the same functions as the event detection pixel received light amount calculation unit 223 and the latency correction unit 224, respectively.

[0196] That is, in the signal processing device 101D of FIG. 24, the function of the HybridEVS sensor 201 of FIG. 23 is realized by the HybridEVS sensor 201D and the subsequent stage ISP / AP 241.

[0197] With this configuration, the signal processing device 101D in FIG. 24 also substantially achieves the same functions as the signal processing device 101A in FIG. 11 or the signal processing device 101B in FIG.

[0198] <Third configuration example of a signal processing device using HybridEVS> In the above, we have described an example in which the HybridEVS sensor 201D is composed only of an event data acquisition unit 221D and a gradation signal acquisition unit 222D, but it may also be configured to have the function of an event detection pixel received light amount calculation unit 223.

[0199] FIG. 25 shows an example of the configuration of a signal processing device in the case where the HybridEVS sensor 201 is realized by an event data acquisition unit 221, a gradation signal acquisition unit 222, and an event detection pixel received light amount calculation unit 223.

[0200] The signal processing device 101E in FIG. 25 is composed of a HybridEVS sensor 201E, a subsequent stage ISP / AP 241E, and a subsequent stage application 114.

[0201] The HybridEVS sensor 201E has the same basic functions as the HybridEVS sensor 201, but is provided with an event data acquisition unit 221E, a gradation signal acquisition unit 222E, and an event detection pixel received light amount calculation unit 223E that have the same functions as the event data acquisition unit 221, the gradation signal acquisition unit 222, and the event detection pixel received light amount calculation unit 223, respectively.

[0202] The subsequent stage ISP / AP 241E is provided with a latency correction unit 224E having the same function as the latency correction unit 224.

[0203] That is, in the signal processing device 101E in FIG. 25, the function of the HybridEVS sensor 201 in FIG. 23 is realized by the HybridEVS sensor 201E and the subsequent stage ISP / AP 241E.

[0204] With this configuration, the signal processing device 101E in FIG. 25 also substantially realizes the same functions as the signal processing device 101A in FIG. 11 or the signal processing device 101B in FIG.

[0205] <<6. Fifth embodiment>> In the above, an example configuration of a signal processing device using the HybridEVS sensor 201 has been described, but it is also possible to switch between acquiring gradation signals and event data from pixel signals of a pixel array that constitutes the CMOS image sensor 151.

[0206] FIG. 26 shows an example of the configuration of a signal processing device that acquires, by switching, grayscale signals and event data from pixel signals of a pixel array that constitutes a CMOS image sensor 151 .

[0207] The signal processing device 101F in FIG. 26 comprises a switching sensor 261, a subsequent stage ISP / AP 113A, and a subsequent stage application 114.

[0208] The switching sensor 261 includes a pixel array 270 that constitutes the CMOS image sensor 151, an event data detection unit 271 that acquires event data based on a signal output from the pixel array 270, and a gradation signal detection unit 272 that detects a gradation signal.

[0209] The signal output from the pixel array 270 is switched to be output to either the event data detector 271 or the gradation signal detector 272 .

[0210] Therefore, the event data detection unit 271 detects event data at the timing when a signal is output from the pixel array 270 to itself, and outputs the event data to the latency correction unit 122A of the subsequent stage ISP / AP 113.

[0211] Furthermore, the gradation signal detector 272 detects a gradation signal at the timing when a signal is output from the pixel array 270 to itself, and outputs the gradation signal to the event detection pixel received light amount calculator 121 A of the subsequent ISP / AP 113 .

[0212] That is, the switching sensor 261 functions as a configuration that combines the functions of the CMOS image sensor 151 and the event signal detection sensor 111 described above.

[0213] As a result, with this configuration, the signal processing device 101F in FIG. 26 also substantially achieves the same functions as the signal processing device 101A in FIG. 11 or the signal processing device 101B in FIG.

[0214] <<7. Application Example>> In the above, an example has been described in which the latency correction amount is set according to the amount of received light, and the timestamp, which is the time information in the event data, is corrected by the latency correction amount.

[0215] However, when event data is transmitted to the subsequent application 114 in units of an event data group consisting of multiple event data, the event data group may be configured to be shifted by a latency correction amount determined according to the amount of received light, and output to the subsequent application 114.

[0216] More specifically, as shown in the upper part of Figure 27, each rectangular frame represents one piece of event data, which is generated in chronological order to the right in the figure, and the impact of latency occurs for a period of time equivalent to two pieces of event data.

[0217] Here, consider the case in the upper part of Figure 27 where, after a predetermined number of event data generated in chronological order have been buffered, an event data group W1 consisting of, for example, event data D1 to D14 is output to the subsequent application 114.

[0218] In this case, if the event data group W1 is sent as is to the subsequent application 114, the subsequent application 114 will process the event data D1 and D2, which are already past event data at the time the event data is acquired, as is, resulting in latency effects.

[0219] Therefore, when the latency correction unit 122 of the subsequent-stage ISP / AP 113 transmits the event data group, it may shift the event data group by a latency correction amount SF21 so as to skip two event data D1 and D2 that are expected to be affected by latency, as shown in the lower part of Figure 27, and transmit an event data group W2 consisting of subsequent-stage event data D3 to D16 to the subsequent-stage application 114.

[0220] This process shifts the event data group W2 by the amount of latency, skipping past event data. As a result, when the subsequent application 114 acquires the event data group W2, the event data group W2 is acquired as an event data group W2 from which the effects of latency have been removed. Therefore, processing in which the effects of latency have been corrected can be achieved simply by sequentially processing the event data constituting the acquired event data group W2 in chronological order.

[0221] <Signal Processing in Application Example> Next, signal processing in an application example will be described with reference to the flowchart in Fig. 28. Note that here, the processing will be described when signal processing with reference to Fig. 27 is performed by the signal processing device 101 in Fig. 7, and it is assumed that all pixels receive the same amount of light and that the event data of all pixels is corrected with a unified latency correction amount.

[0222] In step S101, the event signal detection sensor 111 determines whether or not an event has been detected. If it is determined in step S101 that an event has been detected, the process proceeds to step S102.

[0223] In step S102, the event signal detection sensor 111 outputs the event data to the latency correction unit 122 of the subsequent stage ISP / AP 113. At this time, the latency correction unit 122 buffers the event data.

[0224] In step S103, the event signal detection sensor 111 determines whether the number of buffered event data is equal to or greater than a predetermined threshold, and if it is not equal to or greater than the predetermined threshold, the process returns to step S101.

[0225] The predetermined threshold value here is, for example, a value obtained by adding the number of event data items corresponding to the expected latency to the number of event data items to be transmitted as the event data group described with reference to FIG.

[0226] Therefore, in the example of FIG. 27, since the latency is equivalent to two pieces of event data, the predetermined threshold is 16 or more.

[0227] If it is determined in step S103 that the number of buffered event data is equal to or greater than the predetermined threshold, the process proceeds to step S104.

[0228] In step S104 , the visible light sensor 112 outputs a sensor output signal corresponding to the amount of received visible light to the event detection pixel received light amount calculation unit 121 in the subsequent stage ISP / AP 113 .

[0229] In step S105, the event detection pixel light reception amount calculation unit 121 calculates the amount of light received per unit time of the event detection pixels that make up the event signal detection sensor 111 based on the sensor output signal corresponding to the amount of received visible light, and outputs the calculated amount of received light to the latency correction unit 122.

[0230] In step S106, the latency correction unit 122 reads out the latency registered in association with the amount of received light from the stored table, and specifies the latency correction amount by interpolating as necessary.

[0231] In step S107, the latency correction unit 122 shifts the buffered event data by the latency correction amount, and then composes an event data group consisting of a predetermined number of event data, thereby correcting the latency and outputting it to the downstream application 114.

[0232] In step S108, the subsequent application 114 executes signal processing by sequentially processing the event data constituting the event data group based on the event data group whose latency has been corrected, and outputs the execution result.

[0233] If no event is detected in step S101, the processes of steps S102 to S108 are skipped.

[0234] In step S109, it is determined whether or not an instruction to end the process has been given. If an instruction to end the process has not been given, the process returns to step S101, and the subsequent steps are repeated.

[0235] Then, in step S109, if an instruction to end the process is given, the process ends.

[0236] Through the above processing, the latency correction amount is determined according to the amount of received light, and the event data group is shifted and configured by the determined latency correction amount. Therefore, in the subsequent application 114, it is possible to achieve processing in which the effects of latency are corrected simply by processing the event data that make up the acquired event data group in chronological order.

[0237] As a result, it is possible to appropriately reduce the effect of latency that varies depending on the amount of received light.

[0238] Note that, in the above, we have described the processing when signal processing with reference to Figure 27 is performed by the signal processing device 101 of Figure 7, but similar processing can be realized on a pixel-by-pixel basis in any of the signal processing device 101A of Figure 11, the signal processing device 101C of Figure 23, the signal processing device 101D of Figure 24, the signal processing device 101E of Figure 25, and the signal processing device 101F of Figure 26.

[0239] <First Variant of Event Data Shifting Method> In the above, an example has been described in which a group of buffered event data to be transmitted is shifted by the latency correction amount and output to the subsequent application 114.

[0240] However, there is a possibility that the processing may be affected if the event data D1 and D2 that were prevented from being transmitted needs to be referenced, and the loss of the data occurs.

[0241] Therefore, the subsequent event data may be increased by the latency correction amount with respect to the initial event data group, and output to the subsequent application 114 .

[0242] That is, as shown in the upper part of Figure 29, event data D1 to D14 consisting of event data group W1 is initially expected to be sent to the subsequent application 114, and as shown in the lower part of Figure 29, event data group W11 consisting of event data D1 to D16 to which subsequent event data D15 and D16 have been added by the amount of the effect of latency is sent.

[0243] Then, the subsequent application 114 may selectively use the event data D3 to D16 from the event data group consisting of the event data D1 to D16, excluding the past event data D1 and D2 by the latency time, to perform processing.

[0244] That is, as shown in the lower part of Figure 29, an event data group W11 may be transmitted in which the event data D15 and S16 are increased by a shift amount SF21 that is affected by latency, compared to the event data group W1 consisting of the event data D1 to D14 before correction.

[0245] In this case, the subsequent application 114 can recognize the event data D1 and D2 at the beginning of the event data group W11, which are affected by latency, as past event data based on timestamps, etc., and therefore can perform processing that is not affected by latency by excluding them from the processing in advance.

[0246] Furthermore, as described above, if there is a need to refer to past event data D1 and D2 depending on the processing content, the event data is transmitted without any loss as a group of event data, and therefore can also be referenced in the subsequent application 114.

[0247] <Second Modification of Event Data Shifting Method> The event data group may be configured by deleting only past event data that is affected by latency.

[0248] That is, as shown in the lower part of Figure 30, an event data group W21 may be transmitted in which only the event data D1 and D2 corresponding to the shift amount SF21 that is affected by latency is deleted from the uncorrected event data group W1 shown in the upper part of Figure 30.

[0249] In this case, the subsequent application 114 can realize processing that is not affected by latency by sequentially executing processing from the first event data in the event data group W21. In this case, although event data that follows the initial event data group W11 in chronological order is missing, the impact is small because only event data equivalent to the latency is missing.

[0250] <<8. Application Examples to Mobile Bodies>> The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of mobile body, such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, personal mobility, an airplane, a drone, a ship, or a robot.

[0251] FIG. 31 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile object control system to which the technology according to the present disclosure can be applied.

[0252] The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in Fig. 31 , the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside-vehicle information detection unit 12030, an inside-vehicle information detection unit 12040, and an integrated control unit 12050. Also shown as functional components of the integrated control unit 12050 are a microcomputer 12051, an audio / video output unit 12052, and an in-vehicle network I / F (Interface) 12053.

[0253] The drivetrain control unit 12010 controls the operation of devices related to the drivetrain of the vehicle in accordance with various programs. For example, the drivetrain control unit 12010 functions as a control device for a drive force generating device for generating a drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force of the vehicle.

[0254] The body system control unit 12020 controls the operation of various devices equipped in the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as headlamps, backup lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves transmitted from a portable device that serves as a key or signals from various switches can be input to the body system control unit 12020. The body system control unit 12020 receives these radio waves or signals and controls the vehicle's door lock device, power window device, lamps, etc.

[0255] The outside-vehicle information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the outside-vehicle information detection unit 12030. The outside-vehicle information detection unit 12030 causes the imaging unit 12031 to capture images outside the vehicle and receives the captured images. The outside-vehicle information detection unit 12030 may perform object detection processing or distance detection processing for people, cars, obstacles, signs, characters on the road surface, etc. based on the received images.

[0256] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. The imaging unit 12031 can output the electrical signal as an image or as distance measurement information. The light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

[0257] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041 that detects the state of the driver is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 may calculate the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing off.

[0258] The microcomputer 12051 can calculate control target values ​​for the driving force generating device, steering mechanism, or braking device based on the information inside and outside the vehicle acquired by the outside-vehicle information detection unit 12030 or the inside-vehicle information detection unit 12040, and output control commands to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the distance between vehicles, maintaining vehicle speed, vehicle collision warning, vehicle lane departure warning, etc.

[0259] In addition, the microcomputer 12051 can perform cooperative control for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on driver operation, by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle obtained by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040.

[0260] Furthermore, the microcomputer 12051 can output a control command to the body system control unit 12030 based on the information outside the vehicle acquired by the outside information detection unit 12030. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside information detection unit 12030, and perform cooperative control aimed at preventing glare, such as switching from high beams to low beams.

[0261] The audio / video output unit 12052 transmits at least one of audio and video output signals to an output device capable of visually or audibly notifying information to vehicle occupants or the outside of the vehicle. In the example of Fig. 31, the output devices are exemplified by an audio speaker 12061, a display unit 12062, and an instrument panel 12063. The display unit 12062 may include, for example, at least one of an on-board display and a head-up display.

[0262] FIG. 32 is a diagram showing an example of the installation position of the imaging unit 12031.

[0263] In FIG. 32, the imaging unit 12031 includes imaging units 12101, 12102, 12103, 12104, and 12105.

[0264] The imaging units 12101, 12102, 12103, 12104, and 12105 are provided, for example, at positions such as the front nose, side mirrors, rear bumper, back door, and the top of the windshield inside the vehicle cabin of the vehicle 12100. The imaging unit 12101 provided on the front nose and the imaging unit 12105 provided on the top of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 provided on the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 provided on the rear bumper or back door mainly acquires images of the rear of the vehicle 12100. The imaging unit 12105 provided on the top of the windshield inside the vehicle cabin is mainly used to detect preceding vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc.

[0265] 32 shows an example of the imaging ranges of the imaging units 12101 to 12104. Imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or back door. For example, by overlaying the image data captured by the imaging units 12101 to 12104, an overhead image of the vehicle 12100 viewed from above can be obtained.

[0266] At least one of the image capturing units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the image capturing units 12101 to 12104 may be a stereo camera made up of multiple image capturing elements, or may be an image capturing element having pixels for phase difference detection.

[0267] For example, based on the distance information obtained from the imaging units 12101 to 12104, the microcomputer 12051 can calculate the distance to each three-dimensional object within the imaging ranges 12111 to 12114 and the change in this distance over time (relative speed with respect to the vehicle 12100), thereby extracting as a preceding vehicle, in particular, the three-dimensional object that is the closest three-dimensional object on the path of the vehicle 12100 and traveling in approximately the same direction as the vehicle 12100 at a predetermined speed (e.g., 0 km / h or higher). Furthermore, the microcomputer 12051 can set a vehicle-to-vehicle distance to be maintained in advance in front of the preceding vehicle, and perform automatic braking control (including follow-up stop control), automatic acceleration control (including follow-up start control), etc. In this way, cooperative control can be performed for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on driver operation.

[0268] For example, the microcomputer 12051 classifies and extracts three-dimensional object data regarding three-dimensional objects into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on distance information obtained from the imaging units 12101 to 12104, and can use the data for automatic obstacle avoidance. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. The microcomputer 12051 then determines a collision risk that indicates the risk of collision with each obstacle, and when the collision risk is equal to or greater than a set value and a collision is possible, the microcomputer 12051 can provide driving assistance for collision avoidance by outputting an alarm to the driver via the audio speaker 12061 or the display unit 12062, or by performing forced deceleration or avoidance steering via the drive system control unit 12010.

[0269] At least one of the image capturing units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether a pedestrian is present in the images captured by the image capturing units 12101 to 12104. Such pedestrian recognition is performed, for example, by extracting feature points from the images captured by the image capturing units 12101 to 12104 as infrared cameras and performing pattern matching on a series of feature points that indicate the outline of an object to determine whether the object is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the images captured by the image capturing units 12101 to 12104 and recognizes the pedestrian, the audio / image output unit 12052 controls the display unit 12062 to superimpose a rectangular outline on the recognized pedestrian for emphasis. The audio / image output unit 12052 may also control the display unit 12062 to display an icon or the like indicating the pedestrian at a desired position.

[0270] The above describes an example of a vehicle control system to which the technology according to the present disclosure can be applied. The technology according to the present disclosure can be applied to a configuration that combines an image capture unit 12031 and an outside vehicle information detection unit 12030, among the configurations described above. Specifically, the signal processing device 101 of FIG. 7 , the signal processing device 101A of FIG. 11 , the signal processing device 101B of FIG. 19 , the signal processing device 101C of FIG. 23 , the signal processing device 101D of FIG. 24 , the signal processing device 101E of FIG. 25 , and the signal processing device 101F of FIG. 26 can be applied to a configuration that combines the image capture unit 12031 and the outside vehicle information detection unit 12030. By applying the technology according to the present disclosure to a configuration that combines the image capture unit 12031 and the outside vehicle information detection unit 12030, it is possible to reduce the effects of latency that may occur in object detection processing or distance detection processing.

[0271] In this specification, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are contained in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device with multiple modules housed in a single housing, are both systems.

[0272] Furthermore, the embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure.

[0273] For example, the present disclosure can be configured as a cloud computing system in which a single function is shared and processed collaboratively by multiple devices via a network.

[0274] Furthermore, each step described in the above flowchart can be executed by one device, or can be shared and executed by a plurality of devices.

[0275] Furthermore, when one step includes multiple processes, the multiple processes included in that one step can be executed by one device or can be shared and executed by multiple devices.

[0276] The present disclosure may also be configured as follows: <1> A photodetection device comprising: an event detection unit that outputs event data representing the occurrence of an event, the event detection data being a change in an electrical signal detected by a plurality of event detection pixels that are arranged on a two-dimensional plane and perform photoelectric conversion to generate an electrical signal; a received-light-amount calculation unit that calculates an amount of light received by the event detection pixels; and a latency correction unit that corrects information related to latency in the event data based on the amount of light received by the event detection pixels calculated by the received-light-amount calculation unit. <2> The photodetection device described in <1>, wherein the latency correction unit has a table in which a relationship between the amount of received light and the latency is registered, and the latency corresponding to the amount of received light calculated by the received-light-amount calculation unit is read from the table, and the latency correction unit corrects the event data by an amount of latency correction consisting of the read-out latency. <3> The photodetector according to <2>, wherein the event data includes time information at which the event was detected by the event detection unit, and the latency correction unit corrects the time information of the event data by the latency correction amount corresponding to the amount of received light. <4> The photodetector according to <3>, wherein the event data further includes coordinate information indicating each pixel position of the event detection pixels of the event detection unit, and the latency correction unit corrects the time information of the event data by the latency correction amount corresponding to the amount of received light, in association with the coordinate information of each of the event detection pixels. <5> The photodetector according to <3>, wherein the event data further includes coordinate information indicating each pixel position of the event detection pixels of the event detection unit, and the latency correction unit corrects the event data by adding corrected time information obtained by correcting the time information by the latency correction amount corresponding to the amount of received light, in association with the coordinate information of each of the event detection pixels, or by adding the latency correction amount.<6> The photodetector according to <3>, wherein the event data further includes information on a type of event indicating whether the event is a positive event, in which the electrical signal detected by the event detection unit changes in a positive direction, or a negative event, in which the electrical signal changes in a negative direction, the table registers a relationship between the amount of received light and the latency for each piece of information on the type of event, and the latency correction unit associates the event data with a coordinate position of each of the event detection pixels and adds corrected time information obtained by correcting the time information by the latency correction amount corresponding to the amount of received light, or corrects the event data by adding the latency correction amount. <7> The photodetector according to <1>, further including a visible light detection unit that detects the amount of received visible light near an imaging surface formed by arranging the event detection pixels on a two-dimensional plane, and outputs an output signal corresponding to the detected amount of received light, and the received light amount calculation unit calculates the amount of received light of the event detection pixels based on the output signal. <8> The photodetection device according to <7>, wherein the visible light detection unit detects the amount of received visible light at one point near an imaging surface formed by arranging the event detection pixels in a two-dimensional plane, and outputs the amount of received visible light as the corresponding output signal, the received light amount calculation unit calculates the amount of received light of all the event detection pixels based on the output signal, assuming that the amount of received light of all the event detection pixels is the same, and the latency correction unit corrects the event data of all the event detection pixels by a latency correction amount corresponding to the same amount of received light. <9> The photodetection device according to <1>, further including a gradation signal detection unit made up of a plurality of imaging pixels arranged in a two-dimensional plane, which detects and outputs a gradation signal according to the amount of incident light near an imaging surface formed by arranging the event detection pixels in a two-dimensional plane, and the received light amount calculation unit calculates the amount of received light of each of the event detection pixels based on the gradation signal of each of the imaging pixels.<10> The photodetector according to <9>, wherein the received light amount calculation unit calculates the amount of received light for each imaging pixel and applies it as the amount of received light of the event detection pixel at a position corresponding to the position of the imaging pixel, and the latency correction unit corrects the event data of each of the event detection pixels by a latency correction amount corresponding to the amount of received light applied for each of the event detection pixels. <11> The photodetector according to <10>, wherein the received light amount calculation unit calculates the amount of received light of the event detection pixel based on a predetermined spectrum and the gradation signal. <12> The photodetector according to <11>, wherein the predetermined spectrum is a spectrum estimated from the gradation signal. <13> The photodetector according to <12>, wherein the plurality of imaging pixels constituting the gradation signal detector have a plurality of filters in a predetermined arrangement in front of each of the imaging pixels, the filters filtering the incident light that passes through into a plurality of wavelength bands, the received light amount calculator estimating a spectrum from a ratio of the received light amounts for the plurality of filters based on the gradation signal, and calculating the received light amount of the event detection pixel based on the estimated spectrum and the gradation signal. <14> The photodetector according to <9>, wherein the received light amount calculator calculates the received light amount for each of the imaging pixels and applies it as the received light amount of the event detection pixel at a position corresponding to the position of the imaging pixel, and the latency corrector corrects by adding the received light amount applied for each of the event detection pixels to the event data of each of the event detection pixels. <15> The photodetector according to <9>, wherein the event detection pixel and the imaging pixel are arranged on the same two-dimensional plane, and the event detection unit and the gradation signal detection unit are integrated together. <16> The photodetector according to <15>, wherein the event detection unit, the gradation signal detection unit, and the received light amount calculation unit are integrated together. <17> The photodetector according to <16>, wherein the event detection unit, the gradation signal detection unit, and the received light amount calculation unit are integrated together.<18> The photodetection device according to <1>, wherein the latency correction unit corrects information related to the latency of the event data by shifting an event data group consisting of a predetermined number of the event data in time series to be output to a subsequent stage by a number of event data corresponding to the latency, based on the amount of light received by the event detection pixel calculated by the amount-of-light-received calculation unit. <19> A photodetection method comprising the steps of: outputting event data indicating an occurrence of an event, the event data being a change in the electrical signal, detected by a plurality of event detection pixels arranged on a two-dimensional plane and performing photoelectric conversion to generate an electrical signal; calculating the amount of light received by the event detection pixels; and correcting information related to latency in the event data based on the calculated amount of light received by the event detection pixels. <20> A program that causes a computer to function as an event detection unit that outputs event data representing the occurrence of an event, the event data being a change in an electrical signal detected by a plurality of event detection pixels that are arranged on a two-dimensional plane and generate electrical signals by performing photoelectric conversion; a received light amount calculation unit that calculates the amount of light received by the event detection pixels; and a latency correction unit that corrects information related to latency in the event data, based on the amount of light received by the event detection pixels calculated by the received light amount calculation unit.

[0277] 101, 101A to 101F Signal processing device, 111 Event signal detection sensor, 112 Visible light sensor, 113, 113A, 113B Subsequent stage ISP / AP, 114, 114B Subsequent stage application, 121, 121A, 121B Event detection pixel received light amount calculation unit, 122, 122A, 122B Latency correction unit, 141, 141A-1, 141A-2, 141B Lens, 151 CMOS image sensor, 171 Latency correction unit, 201, 201D, 201E HybridEVS sensor, 221, 221D, 221E Event data acquisition unit, 222, 222D, 222E Gradation signal acquisition unit, 223, 223D, 223E Event detection pixel received light amount calculation unit, 224, 224D, 224E Latency correction unit, 241, 241E Subsequent stage ISP / AP, 261 Switching sensor, 270 Pixel array, 271 Event data detection unit, 272 Gradation signal acquisition unit

Claims

1. A photodetection device comprising: an event detection unit that outputs event data representing the occurrence of an event, which is a change in an electrical signal detected by event detection pixels arranged in a two-dimensional plane and performing photoelectric conversion to generate an electrical signal; a light reception amount calculation unit that calculates the amount of light received by the event detection pixels; and a latency correction unit that corrects information relating to latency in the event data based on the amount of light received by the event detection pixels calculated by the light reception amount calculation unit.

2. The optical detection device of claim 1, wherein the latency correction unit has a table in which the relationship between the amount of received light and the latency is registered, and the latency correction unit reads out from the table the latency corresponding to the amount of received light calculated by the amount of received light calculation unit, and corrects the event data by an amount of latency correction consisting of the latency read out.

3. The optical detection device according to claim 2, wherein the event data includes time information at which the event was detected by the event detection unit, and the latency correction unit corrects the time information of the event data by the latency correction amount corresponding to the amount of received light.

4. The optical detection device of claim 3, wherein the event data further includes coordinate information indicating the pixel position of each of the event detection pixels of the event detection unit, and the latency correction unit corresponds to the coordinate information of each of the event detection pixels and corrects the time information of the event data by the latency correction amount corresponding to the amount of received light.

5. The light detection device of claim 3, wherein the event data further includes coordinate information indicating the pixel position of each of the event detection pixels of the event detection unit, and the latency correction unit corresponds to the coordinate information of each of the event detection pixels and adds corrected time information in which the time information is corrected by the latency correction amount corresponding to the amount of received light, or corrects the event data by adding the latency correction amount.

6. The photodetection device of claim 3, wherein the event data further includes information on the type of event indicating whether the event is a plus event in which the electrical signal detected by the event detection unit changes in a positive direction, or a minus event in which the electrical signal changes in a negative direction, and the table registers the relationship between the amount of received light and the latency for each piece of event type information, and the latency correction unit correlates the event data with the coordinate positions of each of the event detection pixels and adds corrected time information in which the time information is corrected by the latency correction amount corresponding to the amount of received light according to the type of event, or corrects the event data by adding the latency correction amount.

7. The light detection device according to claim 1, further comprising a visible light detection section that detects the amount of received visible light in the vicinity of an imaging surface constituted by the event detection pixels being arranged in a two-dimensional plane and outputs an output signal corresponding to the detected amount of received light, wherein the received light amount calculation section calculates the amount of received light of the event detection pixels based on the output signal.

8. The light detection device according to claim 7, wherein the visible light detection unit detects the amount of visible light received at one point near an imaging surface formed by arranging the event detection pixels on a two-dimensional plane and outputs the amount as the corresponding output signal, the received light amount calculation unit calculates the amount of light received by all of the event detection pixels based on the output signal as being the same amount of light, and the latency correction unit corrects the event data of all of the event detection pixels by a latency correction amount corresponding to the same amount of light received.

9. The photodetection device according to claim 1, further comprising a gradation signal detection unit consisting of a plurality of imaging pixels arranged in a two-dimensional plane, which detects and outputs a gradation signal corresponding to the amount of incident light in the vicinity of an imaging surface formed by arranging the event detection pixels in a two-dimensional plane, and wherein the received light amount calculation unit calculates the amount of received light for each of the event detection pixels based on the gradation signal for each of the imaging pixels.

10. The light detection device according to claim 9, wherein the light reception amount calculation unit calculates the amount of light reception for each of the imaging pixels and applies it as the amount of light reception for the event detection pixel at a position corresponding to the position of the imaging pixel, and the latency correction unit corrects the event data for each of the event detection pixels by an amount of latency correction corresponding to the amount of light reception applied to each of the event detection pixels.

11. The light detection device according to claim 10, wherein the received light amount calculation section calculates the amount of received light of the event detection pixel based on a predetermined spectrum and the gradation signal.

12. The optical detection device according to claim 11, wherein the predetermined spectrum is a spectrum estimated from the tone signal.

13. The photodetection device according to claim 12, wherein a plurality of filters for filtering the transmitted incident light into a plurality of wavelength bands are provided in front of each of the plurality of imaging pixels in a predetermined arrangement for each imaging pixel, and the received light amount calculation unit estimates a spectrum from a ratio of the received light amounts of the plurality of filters based on the gradation signal, and calculates the received light amount of the event detection pixel based on the estimated spectrum and the gradation signal.

14. The light detection device according to claim 9, wherein the light reception amount calculation unit calculates the amount of light reception for each of the imaging pixels and applies it as the amount of light reception for the event detection pixel at a position corresponding to the position of the imaging pixel, and the latency correction unit performs correction by adding the amount of light reception applied for each of the event detection pixels to the event data of each of the event detection pixels.

15. The photodetection device according to claim 9, wherein the event detection pixels and the imaging pixels are arranged on the same two-dimensional plane, and the event detection section and the gradation signal detection section are integrated into one structure.

16. The photodetection device according to claim 15, wherein said event detection section, said gradation signal detection section, and said received light amount calculation section are integrated into one device.

17. The photodetection device according to claim 16, wherein the latency correction section is integrated into the event detection section, the gradation signal detection section, and the light reception amount calculation section.

18. The photodetection device according to claim 1, wherein the latency correction unit corrects information relating to the latency of the event data by shifting an event data group consisting of a predetermined number of event data in a time series to be output to a subsequent stage by a number of event data corresponding to the latency based on the amount of light received by the event detection pixel calculated by the light reception amount calculation unit.

19. A light detection method comprising the steps of: outputting event data representing the occurrence of an event, the event data being a change in an electrical signal detected by event detection pixels arranged in a two-dimensional plane and performing photoelectric conversion to generate an electrical signal; calculating an amount of light received by the event detection pixels; and correcting information relating to latency in the event data based on the calculated amount of light received by the event detection pixels.

20. A program that causes a computer to function as an event detection unit that outputs event data representing the occurrence of an event, which is a change in an electrical signal detected by event detection pixels that are arranged in a two-dimensional plane and generate electrical signals by performing photoelectric conversion; a light reception amount calculation unit that calculates the amount of light received by the event detection pixels; and a latency correction unit that corrects information related to latency in the event data based on the amount of light received by the event detection pixels calculated by the light reception amount calculation unit.

Citation Information

Patent Citations

  • Sensor system, image processing device, image processing method, and program

    WO2021038749A1

  • Delay equalization in event-based vision sensors

    WO2021044299A1

  • Imaging device and imaging method

    WO2021205937A1

  • Dynamic vision sensor, method, and noise filtering circuit for a dynamic vision sensor

    WO2022152619A1

  • High accuracy representation of events generated by an event-based vision sensor

    WO2023100092A1