Light detection device, light detection method, and denoise program

Abstract

The present invention improves denoise accuracy when detecting an event.　This light detection device comprises: an event pixel used for detecting an event; and a denoise processing unit for performing denoise processing of the event on the basis of an influence value affecting noise detected by the event pixel. The denoise processing unit may comprise: a spatial filter for calculating an event density in a filter kernel; an event density table for registering the relationship between an influence value and a noise event density; a denoise threshold value setting unit for setting a denoise threshold value on the basis of the relationship between the influence value and the noise event density; and a threshold value / thinning processing unit for denoising the event on the basis of the event density and the denoise threshold value.

Description

Photodetection device, photodetection method, and denoising program

[0001] The present technology relates to a light detection device, a light detection method, and a denoising program. More specifically, the present technology relates to a light detection device, a light detection method, and a denoising program that are capable of denoising events.

[0002] In an event-based vision sensor (EVS), noise may be erroneously detected as an event. To address this issue, a sensing system has been proposed that detects a change in the amount of received light as an event and corrects the event signal based on a predetermined light emission pattern (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2022-121991

[0004] However, the noise detected by the EVS depends on the brightness and the sensor parameters, so the above-mentioned conventional technology may result in a decrease in denoising accuracy depending on the conditions under which the EVS is used.

[0005] This technology was developed in light of these circumstances and aims to improve the accuracy of denoising during event detection.

[0006] The present technology has been made to solve the above-mentioned problems, and a first aspect thereof is a photodetection device including an event pixel used for detecting an event, and a denoising processing unit that performs denoising of the event based on an influence value that affects noise detected at the event pixel, thereby achieving the effect of performing denoising of the event while responding to noise fluctuations according to the use conditions of the EVS.

[0007] In the first aspect, the influence value may include at least one of brightness, a sensor threshold, and a temperature at the time of the event detection, thereby providing an effect that denoising processing of the event is performed in accordance with noise fluctuations in brightness and sensor parameters.

[0008] In addition, in the first aspect, the image processing device may further include a luminance information acquisition unit that acquires luminance information at the time of event detection, thereby achieving an effect of performing event denoising processing while responding to noise fluctuations according to luminance.

[0009] In the first aspect, the image sensor may further include gradation pixels used for detecting gradations, and the luminance information acquisition unit may acquire the luminance information from the gradation pixels, thereby achieving the effect of acquiring the luminance information from gradation pixels provided separately from event pixels.

[0010] In the first aspect, the image processing device may further include a gradation image generating unit that generates a gradation image based on the event image, and the luminance information acquiring unit may acquire the luminance information from the gradation image, thereby achieving the effect of acquiring the luminance information without providing a separate luminance sensor.

[0011] In addition, in the first aspect, the image processing device may further include an edge noise removal unit that removes noise adjacent to an edge of the event image based on the luminance information, thereby improving accuracy in determining noise adjacent to an edge.

[0012] In addition, in the first aspect, the device may further include a brightness sensor that detects brightness, and an application processor that generates brightness information based on the brightness value output from the brightness sensor and passes the brightness information to the brightness information acquisition unit, thereby achieving the effect of externally acquiring brightness information when an event occurs.

[0013] In addition, in the first aspect, the denoising processor may include a spatial filter that calculates an event density within a filter kernel, an event density table that registers a relationship between the influence value and the event density of noise, a denoising threshold setting unit that sets a denoising threshold based on the relationship between the influence value and the event density of noise, and a threshold / thinning processor that denoises the events based on the event density and the denoising threshold. This brings about the effect that the denoising process of the events is performed based on the denoising threshold that was set based on the influence value.

[0014] A second aspect of the present invention is a light detection method for detecting an event based on received light, and for performing denoising of the event based on an influence value that affects noise that is erroneously detected as the event. This provides the effect of performing denoising of the event while responding to noise fluctuations according to the use conditions of the EVS.

[0015] A third aspect is a denoising program that causes a computer to execute a process of referring to an influence value that affects noise that is erroneously detected as an event, and a denoising process of the event based on the reference result of the influence value, thereby achieving the effect of performing the denoising process of the event while responding to noise fluctuations according to the use conditions of the EVS.

[0016] 1 is a block diagram showing an example of the configuration of a light detection system according to a first embodiment. FIG. 2 is a block diagram showing an example of the configuration of an imaging device according to a first embodiment. FIG. 3 is a block diagram showing an example of the configuration of a solid-state imaging device according to a first embodiment. FIG. 4 is a block diagram showing a flow of denoising processing according to the first embodiment. FIG. 5 is a block diagram showing an example of the circuit configuration of an event detection unit according to the first embodiment. FIG. 6 is a block diagram showing an example of the configuration of a denoising processing unit according to the first embodiment. FIG. 7 is a block diagram showing an example of spatial filtering processing according to the first embodiment. FIG. 8 is a block diagram showing a first example of an event density table according to the first embodiment. FIG. 9 is a block diagram showing a second example of an event density table according to the first embodiment. FIG. 10 is a block diagram showing an example of the hardware configuration of an imaging device according to the first embodiment. FIG. 11 is a block diagram showing an example of the configuration of a solid-state imaging device according to a second embodiment. FIG. 12 is a block diagram showing an example of the configuration of a solid-state imaging device according to a third embodiment. FIG. 13 is a block diagram showing an example of generation of a gradation image using an event image according to the third embodiment. FIG. 14 is a block diagram showing an example of the configuration of a solid-state imaging device according to a fourth embodiment. FIG. 15 is a block diagram showing an example of an event image and a gradation image according to the fourth embodiment. FIG. 16 is a diagram showing an example of an edge noise removal method according to the fourth embodiment. FIG. 17 is a perspective view showing an example of stacking of a pixel array unit according to a fifth embodiment. FIG. 18 is a block diagram showing a schematic example of the configuration of a vehicle control system. FIG. 19 is an explanatory diagram showing an example of the installation position of an imaging unit. FIG. 19 is a block diagram showing an example of the configuration of a vehicle control system. 21 is a diagram showing an example of a sensing area of ​​an external recognition sensor of the vehicle control system of FIG. 20.

[0017] Hereinafter, modes for carrying out the present technology (hereinafter referred to as embodiments) will be described. The description will be made in the following order: 1. First embodiment (an example in which event denoising is performed based on a denoising threshold set based on luminance and a sensor threshold) 2. Second embodiment (an example in which event pixels and gradation pixels are mixed in a pixel array section) 3. Third embodiment (an example in which a gradation image is generated based on an event image) 4. Fourth embodiment (an example in which noise adjacent to the edge of an event image is removed based on luminance information) 5. Fifth embodiment (an example in which semiconductor chips on which solid-state imaging devices are formed are stacked) 6. Example of application to a moving body 7. Example of the configuration of a vehicle control system

[0018] 1. First Embodiment Fig. 1 is a block diagram showing an example of the configuration of a light detection system according to a first embodiment. In the following description, an image capture device is used as an example of a light detection device, but the light detection device is not necessarily limited to an image capture device and may be any other light detection device. For example, the light detection device may be a distance measuring device or the like.

[0019] In the figure, the light detection system includes a luminance sensor 10, an imaging device 12, and an application processor 13. The luminance sensor 10 measures the luminance of incident light and outputs a luminance value KD to the application processor 13. The luminance sensor 10 may be a CMOS (Complementary Metal Oxide Semiconductor) image sensor in which grayscale pixels are arranged in a matrix in row and column directions. In this case, the luminance sensor 10 converts an optical image formed on the light-receiving surface into an electrical signal for each grayscale pixel, digitizes the electrical signal, and outputs it to the application processor 13.

[0020] The image capture device 12 detects an event based on incident light and outputs an event signal EV to the application processor 13. The event signal EV is a signal that indicates a change in the luminance of incident light in the same direction as an event. The image capture device 12 may be an EVS in which event pixels are arranged in a matrix in the row and column directions. In this case, the image capture device 12 can perform event denoising processing based on an influence value that affects the noise detected in the event pixel. The influence value may include at least one of the luminance, the sensor threshold, and the temperature at the time of event detection.

[0021] The imaging device 12 includes a luminance information acquisition unit 12A and a denoising processing unit 12B. The luminance information acquisition unit 12A acquires luminance information KJ from the application processor 13. The denoising processing unit 12B performs denoising processing on the event based on an influence value that affects noise detected in the event pixel. At this time, the denoising processing unit 12B may use the luminance information KJ acquired by the luminance information acquisition unit 12A to perform the denoising processing.

[0022] The application processor 13 can execute various processes based on the event signal EV output from the imaging device 12. This process may be, for example, a droplet monitoring process, a person tracking process, a 3D measurement process, a metal processing monitoring process, or a vibration monitoring process. The application processor 13 also calculates brightness information KJ around the imaging device 12 based on the brightness value KD input from the brightness sensor 10, and outputs the information to the imaging device 12. The application processor 13 may include a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit).

[0023] This optical detection system may be used as a standalone unit, or may be incorporated into a mobile device such as a smartphone, an authentication device, a monitoring device, or a vehicle or drone.

[0024] 1 shows an example in which the luminance information KJ is passed to the imaging device 12 via the application processor 13, but the luminance information KJ may be passed to the imaging device 12 without the application processor 13. In this case, the application processor 13 may not be required. In addition, to prevent the influence of a frame delay between the luminance sensor 10 and the imaging device 12, the luminance sensor 10 and the imaging device 12 may be operated in synchronization with each other.

[0025] FIG. 2 is a block diagram showing an example of the configuration of the imaging apparatus according to the first embodiment.

[0026] 1, the imaging device 12 includes an optical system 91, a solid-state imaging device 92, an imaging control unit 93, an image processing unit 94, a storage unit 95, a display unit 96, an operation unit 97, and a communication unit 98. The imaging control unit 93, the image processing unit 94, the storage unit 95, the display unit 96, the operation unit 97, and the communication unit 98 are connected to one another via a bus 99.

[0027] The optical system 91 allows light from a subject to be incident on the solid-state imaging device 92, and forms an optical image on the light-receiving surface of the solid-state imaging device 92. The optical system 91 may include, for example, a focus lens, a zoom lens, an aperture, etc. The optical system 91 may also include multiple lenses, such as a wide-angle lens, a standard lens, and a telephoto lens.

[0028] The solid-state imaging device 92 converts light from a subject into an electrical signal for each event pixel, digitizes the electrical signal, and outputs it. The solid-state imaging device 92 can output an event signal that detects a change in the luminance of incident light in the same direction as an event. The solid-state imaging device 92 can perform denoising on the event signal based on an influence value that affects the noise detected in the event pixel.

[0029] The imaging control unit 93 controls imaging by the solid-state imaging device 92 based on instructions from the operation unit 97. At this time, the imaging control unit 93 can control the exposure time, exposure amount, imaging timing, etc. of the solid-state imaging device 92.

[0030] The image processing unit 94 performs image processing based on the output from the solid-state imaging device 92. The image processing unit 94 may include a processor that executes processing based on software.

[0031] The storage unit 95 stores images captured by the solid-state imaging device 92 and stores sensor parameters of the solid-state imaging device 92. The sensor parameters may include a sensor threshold for detecting a change in the luminance of incident light in the same direction as an event. The storage unit 95 may also store a denoising threshold used for denoising the event. The storage unit 95 can also store a program for operating the imaging device 12 based on software. The storage unit 95 may include a read-only memory (ROM), a random access memory (RAM), and a memory card.

[0032] The display unit 96 displays captured images and events, and various information that supports the image capturing operation. The display unit 96 may be a liquid crystal display or an organic EL (Electro Luminescence) display.

[0033] The operation unit 97 provides a user interface for operating the imaging device 12. The operation unit 97 may include, for example, buttons, dials, and switches provided on the imaging device 12. The operation unit 97 may be configured as a touch panel together with the display unit 96.

[0034] The communication unit 98 communicates with an external device outside the imaging device 12. The communication partner of the imaging device 12 may be the brightness sensor 10 or the application processor 13. At this time, the imaging device 12 can acquire the brightness information KJ via the communication unit 98.

[0035] Depending on the usage of the imaging device 12, some of the above-described functions may not be present, or conversely, the imaging device 12 may further include functions that are not disclosed.

[0036] FIG. 3 is a block diagram showing an example of the configuration of the solid-state imaging device according to the first embodiment.

[0037] In the figure, the solid-state imaging device 92 includes a pixel array section 111 , an event signal output section 113 , an access control section 115 , a timing control section 116 , and an event signal processing section 117 .

[0038] The pixel array unit 111 has event pixels 121 arranged in an array. The event pixels 121 may be arranged in a matrix in the row and column directions. The event pixels 121 output an event signal when an event is detected. The photoelectric conversion unit of the event pixel 121 may be a photodiode or a single photon avalanche diode (SPAD). The event pixels 121 can be driven row by row via horizontal control lines 123. The event pixels 121 can output an event signal column by column via vertical event signal lines 122.

[0039] The event signal output unit 113 outputs, in a predetermined format, the event signal output from the event pixel 121. For example, the event signal output unit 113 may digitize the event signal output from the event pixel 121 and output the digitized event signal.

[0040] The access control unit 115 controls access to the event pixels 121 via the horizontal control lines 123. At this time, the access control unit 115 can drive the event pixels 121 row by row via the horizontal control lines 123.

[0041] The timing control unit 116 can control the timing of exposure, readout, selection, reset, etc. of the event pixel 121 .

[0042] The event signal processing unit 117 processes the event signal output from the event pixel 121. For example, the event signal processing unit 117 may perform contour enhancement processing or the like. The event signal processing unit 117 may also be provided with a denoising processing unit 12B. In this case, the denoising processing unit 12B may set a denoising threshold based on, for example, the luminance information KJ and a sensor threshold, and perform denoising processing on the event signal based on the denoising threshold.

[0043] FIG. 4 is a block diagram showing the flow of the denoising process according to the first embodiment.

[0044] In the figure, the event pixel 121 is provided with a light receiving unit 201 and a difference detection circuit 151. The event signal processing unit 117 is provided with a denoising processing unit 12B and a control register 152.

[0045] The light receiving unit 201 detects incident light, converts it into an electrical signal, and outputs it to the difference detection circuit 151. The difference detection circuit 151 detects the difference between electrical signals detected at different times by the same event pixel 121, and outputs an event signal when the difference is equal to or greater than the sensor threshold value TS. The control register 152 outputs the sensor threshold value TS to the difference detection circuit 151 and the denoising processor 12B, and also outputs luminance information KJ to the denoising processor 12B. The control register 152 holds the sensor threshold value TS and the luminance information KJ.

[0046] FIG. 5 is a diagram illustrating an example of a circuit configuration of the event detection unit according to the first embodiment.

[0047] 1, the event pixel 121 includes a light receiving unit 201, a logarithmic converter 211, a buffer 221, a differentiator 231, and a comparator 241. The logarithmic converter 211, the buffer 221, the differentiator 231, and the comparator 241 can be provided in a difference detection circuit 151.

[0048] The light receiving unit 201 converts incident light into an electrical signal and transfers it to the logarithmic conversion unit 211. The light receiving unit 201 includes a photodiode PD and a transfer transistor 210. The photodiode PD is connected to the logarithmic conversion unit 211 via the transfer transistor 210. A transfer signal TG is applied to the gate of the transfer transistor 210.

[0049] The logarithmic converter 211 logarithmically converts the output of the light receiving unit 201. By logarithmically converting the output of the light receiving unit 201, it is possible to convert the same rate of change in luminance into the same voltage change regardless of the illuminance. The logarithmic converter 211 is connected downstream of the light receiving unit 201. The logarithmic converter 211 includes N-channel field effect transistors 212 and 214 and a P-channel field effect transistor 213. The P-channel field effect transistor 213 and the N-channel field effect transistor 214 are connected in series. The junction of the P-channel field effect transistor 213 and the N-channel field effect transistor 214 is used as the output of the logarithmic converter 211. The junction of the P-channel field effect transistor 213 and the N-channel field effect transistor 214 is connected to the gate of the N-channel field effect transistor 212. A bias voltage Vb1 is applied to the gate of the P-channel field effect transistor 213. The gate of the N-channel field effect transistor 214 is connected to the cathode of the photodiode PD via the transfer transistor 210, and is also connected to the source of the N-channel field effect transistor 212. The drains of the P-channel field effect transistor 213 and the N-channel field effect transistor 212 are supplied with the power supply Vdd.

[0050] The buffer 221 passes the output of the logarithmic converter 211 to the differentiator 231. The buffer 221 is connected to the subsequent stage of the logarithmic converter 211. The buffer 221 includes N-channel field effect transistors 222 and 223. The N-channel field effect transistors 222 and 223 are connected in series with each other. The connection point of the N-channel field effect transistors 222 and 223 is used as the output of the buffer 221. The gate of the N-channel field effect transistor 222 is connected to the output of the logarithmic converter 211. A bias voltage Vb2 is applied to the gate of the N-channel field effect transistor 223. A power supply Vdd is supplied to the drain of the N-channel field effect transistor 222.

[0051] The differentiator 231 is connected after the buffer 221. The differentiator 231 differentiates the output of the logarithmic converter 211 passed via the buffer 221. The differentiator 231 can detect changes in the luminance of incident light, for example, by determining the difference between the past output level of the logarithmic converter 211 and the current output level of the logarithmic converter 211, using the past output level of the logarithmic converter 211 as a reference level. Here, the differentiator 231 can detect events in which the luminance of incident light increases (positive events) and events in which the luminance of incident light decreases (negative events) in a time-division manner.

[0052] The differentiator 231 includes a capacitor 232, N-channel field-effect transistors 233 and 235, a P-channel field-effect transistor 234, and a switch 236. The capacitor 232 is connected between the output of the buffer 221 and the input of the differentiator 231. The P-channel field-effect transistor 234 and the N-channel field-effect transistor 235 are connected in series with each other. The connection point of the P-channel field-effect transistor 234 and the N-channel field-effect transistor 235 is used as the output of the differentiator 231. The connection point of the P-channel field-effect transistor 234 and the N-channel field-effect transistor 235 is connected to the gate of the P-channel field-effect transistor 234 via the N-channel field-effect transistor 233. A bias voltage Vb3 is applied to the gate of the N-channel field-effect transistor 233. The switch 236 is connected to the gate of the N-channel field-effect transistor 235. The power supply Vdd is supplied to the drain of the P-channel field-effect transistor 234.

[0053] The switch 236 switches among the reset signal AZ, the positive bias signal POS, and the negative bias signal NEG and applies them to the gate of the N-channel field effect transistor 235. At this time, the switch 236 can switch among the reset signal AZ, the positive bias signal POS, and the negative bias signal NEG based on the switching signal BS. The reset signal AZ, the positive bias signal POS, and the negative bias signal NEG are input to the differentiator 231 via the horizontal control line 123.

[0054] At this time, the differentiator 231 resets the reference level to the current output level of the logarithmic converter 211 based on the reset signal AZ. The differentiator 231 can detect a new change in the luminance of the incident light based on changes in the output level of the logarithmic converter 211 from the time the reference level is reset. After the reference level is reset based on the reset signal AZ, the differentiator 231 applies the positive bias signal POS and the negative bias signal NEG to the gate of the N-channel field effect transistor 235 in a time-division manner. This allows the differentiator 231 to detect changes in the luminance of the incident light in the increasing direction and changes in the luminance of the incident light in the decreasing direction in a time-division manner.

[0055] The comparator 241 outputs the detection value of the differentiator 231 in a time-division manner as an increasing change in the luminance of the incident light and a decreasing change in the luminance of the incident light. The comparator 241 is connected to the subsequent stage of the differentiator 231. The comparator 241 includes a P-channel field-effect transistor 242 and an N-channel field-effect transistor 243. The P-channel field-effect transistor 242 and the N-channel field-effect transistor 243 are connected in series with each other. The connection point of the P-channel field-effect transistor 242 and the N-channel field-effect transistor 243 is used as the output of the comparator 241. The connection point of the P-channel field-effect transistor 242 and the N-channel field-effect transistor 243 is connected to the vertical event signal line 122. The gate of the P-channel field-effect transistor 242 is connected to the output of the differentiator 231. A bias voltage Vb4 is applied to the gate of the N-channel field-effect transistor 243.

[0056] Here, by setting the positive bias signal POS and the negative bias signal NEG in a time-division manner in the differentiator 231, the comparator 241 can be shared for both an increasing change in the luminance of the incident light and a decreasing change in the luminance of the incident light. This eliminates the need to provide separate comparators 241 for detecting an increasing change in the luminance of the incident light and a decreasing change in the luminance of the incident light, thereby reducing the circuit size of the comparator 241.

[0057] The output of the photodiode PD is input to a logarithmic converter 211. Based on the source follower operation of the logarithmic converter 211, the output of the photodiode PD is logarithmically converted and input to a differentiator 231 via a buffer 221. After a reset signal AZ is applied to the differentiator 231, a positive bias signal POS and a negative bias signal NEG are applied in a time-division manner, thereby detecting changes in the luminance of the incident light in the increasing direction and the decreasing direction in the luminance of the incident light in a time-division manner. Based on the output of the differentiator 231, a comparator 241 detects positive events and negative events in a time-division manner.

[0058] FIG. 6 is a block diagram illustrating an example of the configuration of the denoising processor according to the first embodiment.

[0059] In the figure, the denoising processing unit 12B includes an event density table 301, a denoising threshold setting unit 302, a spatial filter 303, and a threshold / thinning-out processing unit 304.

[0060] The event density table 301 registers the relationship between the sensor threshold value TS and the luminance information KJ and the event density. In order to represent the noise variation, the event density table 301 may also register the average and standard deviation of the noise variation for each combination of the sensor threshold value TS and the luminance information KJ.

[0061] The denoising threshold setting unit 302 sets the denoising threshold TZ based on the relationship between the sensor threshold T S and the luminance information K J and the event density. For example, the denoising threshold setting unit 302 may set the denoising threshold T Z based on the average and standard deviation of noise variations specified by the combination of the sensor threshold T S and the luminance information K J.

[0062] The spatial filter 303 calculates an event density ME within a filter kernel (also called a tap). At this time, the spatial filter 303 can set the filter kernel to the imaging region based on the kernel size. Note that a spatiotemporal filter may be used instead of the spatial filter 303.

[0063] The threshold / decimation processor 304 denoises events based on the event density ME and the denoising threshold TZ, and outputs the denoised event signal EV. At this time, if the event density ME is equal to or less than the denoising threshold TZ, the threshold / decimation processor 304 can denoise the event signal PEV that was erroneously detected before denoising, and output the denoised event signal EV.

[0064] When changing the denoising intensity according to the brightness or sensor parameters, the denoising intensity may be changed uniformly across the entire screen, the denoising intensity may be changed according to the image height, or the denoising intensity may be changed for each pixel.

[0065] FIG. 7 is a diagram illustrating an example of spatial filtering processing according to the first embodiment.

[0066] In the figure, a filter kernel 402 is set in an imaging area 401. The spatial filter 303 counts the number of events 404 in the filter kernel 402 and calculates the event density in the filter kernel 402. If the event density in the filter kernel 402 is equal to or less than a predetermined value, the spatial filter 303 regards the event of the pixel of interest 403 as noise and discards it. At this time, the spatial filter 303 performs a raster scan SCN of the filter kernel 402 and calculates the event density for all pixels of interest 403 in the imaging area 401. Here, because noise depends on the sensor threshold and luminance, the event density in the filter kernel 402 varies depending on the sensor threshold and luminance. For this reason, the denoising processing unit 12B can improve denoising accuracy by setting the denoising threshold TZ depending on the sensor threshold and luminance.

[0067] 8 is a diagram showing a first example of an event density table according to the first embodiment, which shows an event density table for positive events.

[0068] In FIG. 11A, the event density table TBP1 registers the average noise variations PA00, PA01, PA02, PA10, PA11, PA12, PA20, PA21, PA22, etc. for each combination of sensor thresholds TS0, TS1, TS2, etc. and brightness TN0, TN1, TN2, etc.

[0069] In FIG. 1B, the event density table TBP2 registers standard deviations PS00, PS01, PS02, PS10, PS11, PS12, PS20, PS21, PS22, etc. of noise variation for each combination of sensor thresholds TS0, TS1, TS2, etc. and brightness TN0, TN1, TN2, etc.

[0070] In this case, the denoising threshold setting unit 302 can set the denoising threshold TZ for positive events based on the reference results of the event density tables TBP1 and TBP2. For example, when the sensor threshold is TS1 and the luminance is TN2, the denoising threshold TZ can be given by PA21 + a·PS21, where a is a coefficient.

[0071] 9 is a diagram showing a second example of the event density table according to the first embodiment, which shows an event density table for negative events.

[0072] In FIG. 11A, the event density table TBN1 registers the average noise variations NA00, NA01, NA02, NA10, NA11, NA12, NA20, NA21, NA22, etc. for each combination of sensor thresholds TS0, TS1, TS2, etc. and brightness TN0, TN1, TN2, etc.

[0073] In FIG. 1B, the event density table TBN2 registers standard deviations NS00, NS01, NS02, NS10, NS11, NS12, NS20, NS21, NS22, etc. of noise variation for each combination of sensor thresholds TS0, TS1, TS2, etc. and brightness TN0, TN1, TN2, etc.

[0074] In this case, the denoising threshold setting unit 302 can set the denoising threshold TZ for negative events based on the reference results of the event density tables TBN1 and TBN2. For example, when the sensor threshold is TS1 and the luminance is TN2, the denoising threshold TZ can be given by NA21+b·PS21, where b is a coefficient.

[0075] FIG. 10 is a block diagram illustrating an example of the hardware configuration of the imaging apparatus according to the first embodiment.

[0076] In the figure, the imaging device 12 includes a processor 521, a communication control device 522, a communication interface 523, a main memory unit 525, an auxiliary memory unit 526, and an input / output interface 524. The processor 521, the communication control device 522, the communication interface 523, the main memory unit 525, the auxiliary memory unit 526, and the input / output interface 524 are interconnected via an internal bus 527. The main memory unit 525 and the auxiliary memory unit 526 are accessible from the processor 521.

[0077] The imaging device 12 is connected to an input device 16 and an output device 17. In this case, the input device 16 and the output device 17 can be connected to an internal bus 527 via an input / output interface 524. The input device 16 is, for example, a keyboard, a mouse, a touch panel, a card reader, or a microphone. The output device 17 is, for example, a screen display device (such as a liquid crystal monitor, an organic EL (Electro Luminescence) display, or a graphics card), a speaker, or a printer.

[0078] The processor 521 is hardware that performs overall control of the imaging device 12. In this case, the processor 521 can cause the communication control device 522, the communication interface 523, the main memory 525, the auxiliary memory 526, and the input / output interface 524 to operate in cooperation with each other. The processor 521 may be a central processing unit (CPU) or a graphics processing unit (GPU). The processor 521 may be a single-core processor or a multi-core processor. The processor 521 may include a hardware circuit (for example, a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC)) such as an accelerator that performs part of the processing.

[0079] The main memory unit 525 may include, for example, a semiconductor memory such as an SRAM (Static Random Access Memory) or a DRAM (Dynamic Random Access Memory). The main memory unit 525 may store programs being executed by the processor 521 and may provide a work area for the processor 521 to execute programs.

[0080] The auxiliary storage unit 526 is a storage device with a large storage capacity, such as a flash memory or an SD card. The auxiliary storage unit 526 can store executable files for various programs and data used to execute the programs. The auxiliary storage unit 526 can store an event density table 526A and a denoising program 526B. The denoising program 526B may be software that can be installed in the image capture device 12, or may be incorporated into the image capture device 12 as firmware.

[0081] The communication control device 522 is hardware having a function of controlling communication with the outside. The communication control device 522 is connected to the network 15 via a communication interface 523. The network 15 may be a wide area network (WAN) such as the Internet, a local area network (LAN) such as Wi-Fi or Ethernet (registered trademark), or a combination of a WAN and a LAN. The network may also be a cellular network.

[0082] The input / output interface 524 converts data input from the input device 16 into a data format that can be processed by the processor 521 , and converts data output from the processor 521 into a data format that can be processed by the output device 17 .

[0083] The processor 521 can perform event denoising processing by reading the denoising program 526B into the main storage unit 525 and executing the denoising program 526B. In this denoising processing, the processor 521 can refer to the event density table 526A.

[0084] The execution of the denoising program 526B may be shared among multiple processors or computers. Alternatively, the processor 521 may instruct a cloud computer or the like via the network 15 to execute all or part of the denoising program 526B and receive the execution results.

[0085] In this way, in the first embodiment described above, the image capture device 12 sets the denoising threshold based on the luminance and the sensor threshold, which allows the image capture device 12 to perform event denoising processing in accordance with noise fluctuations relative to the luminance and the sensor threshold, thereby improving the denoising accuracy according to the EVS usage conditions.

[0086] 2. Second Embodiment In the first embodiment described above, the brightness sensor 10 and the application processor 13 are provided to pass the brightness information KJ to the imaging device 12. In this second embodiment, event pixels and gradation pixels are mixed in the pixel array unit so that the imaging device can acquire the brightness information KJ.

[0087] FIG. 11 is a block diagram showing an example of the configuration of a solid-state imaging device according to the second embodiment.

[0088] In the figure, the solid-state imaging device 602 is provided with a pixel array unit 611, an access control unit 615, and a timing control unit 616 instead of the pixel array unit 111, the access control unit 115, and the timing control unit 116 of the above-described first embodiment. Also, the solid-state imaging device 602 is provided with a gradation signal output unit 114 and a gradation signal processing unit 118 in addition to the solid-state imaging device 92 of the above-described first embodiment. Other configurations of the solid-state imaging device 602 of the second embodiment are similar to the configuration of the solid-state imaging device 92 of the above-described first embodiment.

[0089] The pixel array unit 611 has event pixels 121 and gradation pixels 131 arranged in an array. The event pixels 121 and gradation pixels 131 may be arranged in a matrix in the row and column directions. For example, three event pixels 121 may be arranged in a 2×2 pixel region, and one gradation pixel 131 may be arranged in one of the 2×2 pixel regions. The gradation pixel 131 detects a gradation signal. The gradation signal indicates a level corresponding to the luminance of incident light. A photodiode may be provided as the photoelectric conversion unit of the gradation pixel 131. The event pixels 121 and gradation pixels 131 can be driven row by row via a horizontal control line 182. The gradation pixels 131 can output a gradation signal column by column via a vertical gradation signal line 132.

[0090] The grayscale signal output unit 114 outputs the grayscale signal output from the grayscale pixel 131 in a predetermined format. For example, the grayscale signal output unit 114 may digitize and output the grayscale signal output from the grayscale pixel 131. Furthermore, the grayscale signal output unit 114 may support constant current readout or capacitive load readout.

[0091] The access control unit 615 controls access to the event pixels 121 and the gradation pixels 131 via the horizontal control lines 182. At this time, the access control unit 615 can drive the event pixels 121 and the gradation pixels 131 row by row via the horizontal control lines 182.

[0092] The timing control section 616 can control the timing of exposure, readout, selection, reset, etc. for the event pixel 121 and the gradation pixel 131 .

[0093] The gradation signal processing unit 118 processes the gradation signal output from the gradation signal output unit 114. The gradation signal processing unit 118 may, for example, perform correlated double sampling (CDS) processing based on the gradation signal output from the gradation signal output unit 114. The gradation signal processing unit 118 also generates luminance information KJ based on the gradation signal output from the gradation signal output unit 114 and outputs the luminance information KJ to the event signal processing unit 117.

[0094] At this time, the denoising processing unit 12B can acquire the luminance information KJ from the gradation signal processing unit 118. Then, the denoising processing unit 12B can set a denoising threshold based on, for example, the luminance information KJ and the sensor threshold, and perform denoising on the event signal based on the denoising threshold.

[0095] As described above, in the second embodiment, the solid-state imaging device 602 generates luminance information KJ based on gradation signals output from the gradation pixels 131 provided in the pixel array unit 611 together with the event pixels 121, and outputs the luminance information KJ to the denoising processing unit 12B. This eliminates the need to provide a luminance sensor 10 separate from the solid-state imaging device 602 in order for the denoising processing unit 12B to acquire the luminance information KJ, thereby simplifying the configuration of the denoising system.

[0096] 3. Third Embodiment In the above-described first embodiment, the brightness sensor 10 and the application processor 13 are provided to transfer the brightness information KJ to the imaging device 12. In this third embodiment, the imaging device generates a gradation image based on an event image to acquire the brightness information KJ.

[0097] FIG. 12 is a block diagram showing an example of the configuration of a solid-state imaging device according to the third embodiment.

[0098] In the figure, the solid-state imaging device 702 includes an event signal processing unit 717 instead of the event signal processing unit 117 of the first embodiment described above. Other configurations of the solid-state imaging device 702 of the third embodiment are similar to the configuration of the solid-state imaging device 92 of the first embodiment described above.

[0099] The event signal processing unit 717 is the same as the event signal processing unit 117 of the first embodiment except that a gradation image generating unit 717A is added to the event signal processing unit 117 of the first embodiment. Other configurations of the event signal processing unit 717 of the third embodiment are the same as the configuration of the event signal processing unit 117 of the first embodiment.

[0100] The gradation image generating unit 717A generates a gradation image based on the event image generated by the event signal processing unit 717. For example, at each pixel position of the event image, the gradation image generating unit 717A can add 1 to the pixel value when a positive event occurs, and subtract 1 from the pixel value when a negative event occurs. In this way, the gradation image generating unit 717A can generate a pseudo-luminance image by accumulating events in the time direction.

[0101] At this time, the denoising processing unit 12B can acquire the luminance information KJ from the gradation image generating unit 717A. Then, the denoising processing unit 12B can set a denoising threshold based on, for example, the luminance information KJ and the sensor threshold, and perform denoising on the event signal based on the denoising threshold.

[0102] FIG. 13 is a diagram showing an example of generating a gradation image using an event image according to the third embodiment.

[0103] In the same figure, the gradation image generation unit 717A can, for example, accumulate one frame to generate a luminance image KG1, accumulate six frames to generate a luminance image KG2, accumulate nine frames to generate a luminance image KG3, and accumulate 16 frames to generate a luminance image KG4.

[0104] As described above, in the third embodiment, the solid-state imaging device 702 generates a gradation image based on the event image in order to obtain the luminance information KJ. This eliminates the need to provide a luminance sensor 10 separate from the solid-state imaging device 702 in order for the denoising processing unit 12B to obtain the luminance information KJ, thereby simplifying the configuration of the denoising system.

[0105] 4. Fourth Embodiment In the first embodiment described above, the event denoising process is performed based on a denoising threshold set based on the luminance and the sensor threshold. In this fourth embodiment, noise adjacent to the edge of the event image is removed based on luminance information.

[0106] FIG. 14 is a block diagram showing an example of the configuration of a solid-state imaging device according to the fourth embodiment.

[0107] In the figure, the solid-state imaging device 802 includes an event signal processing unit 817 instead of the event signal processing unit 117 of the first embodiment described above. Other configurations of the solid-state imaging device 802 of the fourth embodiment are similar to the configuration of the solid-state imaging device 92 of the first embodiment described above.

[0108] The event signal processing unit 817 is the same as the event signal processing unit 117 of the first embodiment described above, except that an edge noise removal unit 817A is added. Other configurations of the event signal processing unit 817 of the fourth embodiment are the same as the configuration of the event signal processing unit 117 of the first embodiment described above.

[0109] The edge noise removal unit 817A removes noise adjacent to the edges of the event image based on the luminance information KJ. At this time, the edge noise removal unit 817A can use a bilateral filter that mixes and smoothes only pixels with similar luminances near the pixel of interest. At this time, by using the luminance information KJ, it is possible to prevent pixels with similar luminances near the pixel of interest from being mixed in at positions across edges, and to remove noise near the edges of the event image.

[0110] FIG. 15 is a block diagram showing an example of an event image and a gradation image according to the fourth embodiment.

[0111] In the scene indicated by a in the figure, for example, a black and white step edge is moving horizontally. In this scene, the solid-state imaging device 802 acquires a gradation image 850. In this case, the solid-state imaging device 802 may acquire the gradation image 850 from the luminance sensor 10 of the first embodiment described above. Alternatively, the solid-state imaging device 802 may be provided with the gradation pixel 131 of the second embodiment described above, and generate the gradation image 850 based on a pixel signal output from the gradation pixel 131. Alternatively, the solid-state imaging device 802 may generate the gradation image based on an event image, as shown in the third embodiment described above. In this case, the gradation image 850 includes a white image 851 and a black image 852 separated by an edge 854. Then, a pixel 853 of the white image 851 near the edge 854 is considered.

[0112] In FIG. 1B, the solid-state imaging device 802 generates an event image 860 corresponding to the gradation image 850. In generating the event image 860, an event 864 is detected corresponding to an edge 854 of the gradation image 850. It is also assumed that noise is erroneously detected as an event 866 near the edge 854, and that noise is erroneously detected as an event 865 at a position away from the edge 854. Then, attention is focused on an event 863 that is erroneously detected corresponding to the position of pixel 853 in the gradation image 850. It is assumed that this event 863 is noise that has been erroneously detected as the event 866.

[0113] The spatial filter 303 sets a tap 862 in the event image 860 so as to include the event 863. At this time, since the event 863 is located near the edge 854, the tap 862 is set so as to straddle the edge 854. When the tap 862 is set so as to straddle the edge 854, many events 864 are included in the tap 862. For this reason, when the event density is calculated by counting all the events 863 and 864 in the tap 862, the event density in the tap 862 increases, and it becomes impossible to discard the event 863 that was erroneously detected as noise.

[0114] For this reason, the edge noise removal unit 817A counts only events whose brightness is close to that of the event 863 when the spatial filter 303 counts the event density in the tap 862. At this time, the event 863 in the tap 862 is located in the white image 851, and the event 864 in the tap 862 is located in the black image 852. For this reason, the brightness of the event 863 in the tap 862 is different from the brightness of the event 864, and the event 864 is excluded when the event density in the tap 862 is counted. As a result, only the event 863 is counted in the tap 862, the event density in the tap 862 decreases, and the event 863, which was erroneously detected as noise, can be discarded.

[0115] FIG. 16 is a diagram illustrating an example of an edge noise removal method according to the fourth embodiment.

[0116] In the figure, it is assumed that an event image 860 is generated corresponding to the gradation image 850 in Fig. 15. It is assumed that an event 866 is erroneously detected near an event 864 corresponding to the position of an edge 854, and an event 865 is erroneously detected at a position distant from the event 864.

[0117] At a position away from the event 864, the spatial filter 303 can set a tap so that the event 864 is not included. Therefore, as shown in b in the same figure, the denoising processing unit 12B can denoise noise 865 that is away from the event 864. At this time, the denoising processing unit 12B sets a denoising threshold according to the luminance of the white image 851 and the luminance of the black image 852 included in the gradation image 850, respectively, and can deal with the luminance dependency of noise that is erroneously detected as the event 865.

[0118] In the spatial filter 303, taps are set near the event 864 so that the event 864 is included. At this time, when the event density is counted by the spatial filter 303, the edge noise removal unit 817A counts only events that have a brightness similar to that of the event 866. In this way, when the event 866 is of interest, the edge noise removal unit 817A can prevent the event 864 near the event 866 from being counted, and can discard the event 866 that has been erroneously detected as noise, as shown at c in the figure.

[0119] In this way, in the fourth embodiment described above, the solid-state imaging device 802 determines whether an event adjacent to an edge of an event image is noise based on the luminance information KJ. This allows the solid-state imaging device 802 to improve the accuracy of determining noise adjacent to the edge of an event image, thereby improving the accuracy of event detection.

[0120] 5. Fifth Embodiment In the first embodiment described above, event denoising is performed based on a denoising threshold set based on luminance and a sensor threshold. In this fifth embodiment, pixel array units are stacked, with the light receiving unit of each pixel provided on the upper layer and the circuit unit provided on the lower layer.

[0121] FIG. 17 is a perspective view showing an example of the configuration of a pixel in a pixel array unit according to the fifth embodiment.

[0122] In the figure, the pixel array section includes a light-receiving array section 901 and a circuit array section 911. The light-receiving array section 901 can be stacked on the circuit array section 911. The light-receiving array section 901 includes light-receiving pixels 902. The light-receiving pixels 902 are arranged in a matrix pattern in the row and column directions. The light-receiving pixels 902 can be provided with the light-receiving section 201 and logarithmic conversion section 211 shown in FIG. 5.

[0123] The circuit array unit 911 includes circuit units 912. The circuit units 912 are arranged in a matrix shape in the row and column directions. The circuit units 912 are connected to the light-sensitive pixels 902. A circuit unit 912 can be provided for each light-sensitive pixel 902. In this case, the circuit unit 912 can be arranged directly below the light-sensitive pixel 902. The circuit unit 912 can be provided with the buffer 221, differentiator 231, and comparator 241 shown in FIG. 5 .

[0124] The light-receiving array unit 901 can be formed on the upper chip, and the circuit array unit 911 can be formed on the lower chip. In this case, the upper chip and the lower chip may be directly bonded. Hybrid bonding can be used for directly bonding the upper chip and the lower chip. In this case, the upper chip and the lower chip may be electrically connected based on Cu-Cu connection. The material of the semiconductor substrate used for the upper chip and the lower chip may be Si, InGaAs, or InP.

[0125] In this way, in the fifth embodiment described above, the light-receiving array section 901 and the circuit array section 911 are stacked, which makes it possible to increase the area of ​​the light-receiving pixels 902 while suppressing an increase in chip size, thereby enabling the solid-state imaging device to be miniaturized while improving sensitivity.

[0126] 6. Application Examples to Mobile Bodies The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of mobile body, such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, personal mobility, an airplane, a drone, a ship, or a robot.

[0127] FIG. 18 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile object control system to which the technology according to the present disclosure can be applied.

[0128] The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in Fig. 18, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside-vehicle information detection unit 12030, an inside-vehicle information detection unit 12040, and an integrated control unit 12050. Also shown as functional components of the integrated control unit 12050 are a microcomputer 12051, an audio / video output unit 12052, and an in-vehicle network I / F (interface) 12053.

[0129] The drivetrain control unit 12010 controls the operation of devices related to the drivetrain of the vehicle in accordance with various programs. For example, the drivetrain control unit 12010 functions as a control device for a drive force generating device for generating a drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force of the vehicle.

[0130] The body system control unit 12020 controls the operation of various devices equipped in the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as headlamps, backup lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves transmitted from a portable device that serves as a key or signals from various switches can be input to the body system control unit 12020. The body system control unit 12020 receives these radio waves or signals and controls the vehicle's door lock device, power window device, lamps, etc.

[0131] The outside-vehicle information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the outside-vehicle information detection unit 12030. The outside-vehicle information detection unit 12030 causes the imaging unit 12031 to capture images outside the vehicle and receives the captured images. The outside-vehicle information detection unit 12030 may perform object detection processing or distance detection processing for people, cars, obstacles, signs, characters on the road surface, etc. based on the received images.

[0132] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. The imaging unit 12031 can output the electrical signal as an image or as distance measurement information. The light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

[0133] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041 that detects the state of the driver is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 may calculate the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing off.

[0134] The microcomputer 12051 can calculate control target values ​​for the driving force generating device, steering mechanism, or braking device based on the information inside and outside the vehicle acquired by the outside-vehicle information detection unit 12030 or the inside-vehicle information detection unit 12040, and output control commands to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the distance between vehicles, maintaining vehicle speed, vehicle collision warning, vehicle lane departure warning, etc.

[0135] In addition, the microcomputer 12051 can perform cooperative control for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on driver operation, by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle obtained by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040.

[0136] Furthermore, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the outside information detection unit 12030. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside information detection unit 12030, and perform cooperative control aimed at preventing glare, such as switching from high beams to low beams.

[0137] The audio / video output unit 12052 transmits at least one of audio and video output signals to an output device capable of visually or audibly notifying the passengers of the vehicle or the outside of the vehicle of information. In the example of Fig. 18, the output devices are exemplified by an audio speaker 12061, a display unit 12062, and an instrument panel 12063. The display unit 12062 may include, for example, at least one of an on-board display and a head-up display.

[0138] FIG. 19 is a diagram showing an example of the installation position of the imaging unit 12031.

[0139] In FIG. 19, the imaging unit 12031 includes imaging units 12101, 12102, 12103, 12104, and 12105.

[0140] The imaging units 12101, 12102, 12103, 12104, and 12105 are provided, for example, at positions such as the front nose, side mirrors, rear bumper, back door, and the top of the windshield inside the vehicle cabin of the vehicle 12100. The imaging unit 12101 provided on the front nose and the imaging unit 12105 provided on the top of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 provided on the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 provided on the rear bumper or back door mainly acquires images of the rear of the vehicle 12100. The imaging unit 12105 provided on the top of the windshield inside the vehicle cabin is mainly used to detect preceding vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc.

[0141] 19 shows an example of the imaging ranges of the imaging units 12101 to 12104. Imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or back door. For example, by overlaying the image data captured by the imaging units 12101 to 12104, an overhead image of the vehicle 12100 viewed from above can be obtained.

[0142] At least one of the image capturing units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the image capturing units 12101 to 12104 may be a stereo camera made up of multiple image capturing elements, or may be an image capturing element having pixels for phase difference detection.

[0143] For example, based on the distance information obtained from the imaging units 12101 to 12104, the microcomputer 12051 can calculate the distance to each three-dimensional object within the imaging ranges 12111 to 12114 and the change in this distance over time (relative speed with respect to the vehicle 12100), thereby extracting as a preceding vehicle, in particular, the three-dimensional object that is the closest three-dimensional object on the path of the vehicle 12100 and traveling in approximately the same direction as the vehicle 12100 at a predetermined speed (e.g., 0 km / h or higher). Furthermore, the microcomputer 12051 can set a vehicle-to-vehicle distance to be maintained in advance in front of the preceding vehicle, and perform automatic braking control (including follow-up stop control), automatic acceleration control (including follow-up start control), etc. In this way, cooperative control can be performed for the purpose of autonomous driving, which runs autonomously without relying on driver operation.

[0144] For example, the microcomputer 12051 classifies and extracts three-dimensional object data regarding three-dimensional objects into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on distance information obtained from the imaging units 12101 to 12104, and can use the data for automatic obstacle avoidance. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. The microcomputer 12051 then determines a collision risk that indicates the risk of collision with each obstacle, and when the collision risk is equal to or greater than a set value and a collision is possible, the microcomputer 12051 can provide driving assistance for collision avoidance by outputting an alarm to the driver via the audio speaker 12061 or the display unit 12062, or by performing forced deceleration or avoidance steering via the drive system control unit 12010.

[0145] At least one of the image capturing units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether a pedestrian is present in the images captured by the image capturing units 12101 to 12104. Such pedestrian recognition is performed, for example, by extracting feature points from the images captured by the image capturing units 12101 to 12104 as infrared cameras and performing pattern matching on a series of feature points that indicate the outline of an object to determine whether the object is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the images captured by the image capturing units 12101 to 12104 and recognizes the pedestrian, the audio / image output unit 12052 controls the display unit 12062 to superimpose a rectangular outline on the recognized pedestrian for emphasis. The audio / image output unit 12052 may also control the display unit 12062 to display an icon or the like indicating the pedestrian at a desired position.

[0146] An example of a vehicle control system to which the technology according to the present disclosure can be applied has been described above. The technology according to the present disclosure can be applied to the imaging unit 12031 of the above-described configuration. Specifically, for example, the imaging devices according to the first to fourth embodiments described above can be applied to the imaging unit 12031. By applying the technology according to the present disclosure to the vehicle control system 12000, it is possible to improve the accuracy of event detection.

[0147] 7. Configuration Example of Vehicle Control System FIG. 20 is a block diagram showing a configuration example of a vehicle control system 11, which is a non-limiting example of a mobility device control system to which the present technology is applied.

[0148] The vehicle control system 11 is provided in the vehicle 1 and performs processing related to automated driving of the vehicle 1. This automated driving includes levels 1 to 5 of automated driving, as well as remote driving and / or remote assistance of the vehicle 1 by a remote driver. The level of automated driving may refer to the Society of Automotive Engineers (SAE) J3016™ APL2021 Levels of Driving Automation, where SAE Level 0 denotes the lowest level of automated driving and SAE Level 5 denotes the highest level of automated driving. For example, SAE Level 1 automated driving may be composed of driver assistance functions that provide steering or braking / acceleration support to the driver, and SAE Level 5 automated driving may be composed of automated driving functions that can drive the vehicle under all conditions.

[0149] The vehicle control system 11 includes a vehicle control ECU (Electronic Control Unit) 21, a communication unit 22, a map information storage unit 23, a location information acquisition unit 24, an external recognition sensor 25, an in-vehicle sensor 26, a vehicle sensor 27, a memory unit 28, a driving automation control unit 29, a DMS (Driver Monitoring System) 30, an HMI (Human Machine Interface) 31, and a vehicle control unit 32.

[0150] Two or more (or in some cases, all) of the vehicle control ECU 21, communication unit 22, map information storage unit 23, position information acquisition unit 24, external recognition sensor 25, in-vehicle sensor 26, vehicle sensor 27, memory unit 28, driving automation control unit 29, DMS 30, HMI 31, and vehicle control unit 32 are communicatively connected to each other via a communication network 41. The communication network 41 is configured, for example, by an in-vehicle communication network or bus conforming to a digital bidirectional communication standard such as a Controller Area Network (CAN), a Local Interconnect Network (LIN), a Local Area Network (LAN), FlexRay (registered trademark), or Ethernet (registered trademark). In some embodiments, the communication network 41 may include two or more types of communication networks, and different types of communication networks may be used depending on the type of data being transmitted. For example, a CAN may be used for data related to vehicle control, and an Ethernet may be used for large-volume data. In some embodiments, two or more (or in some cases, all) units of the vehicle control system 11 may be directly connected using wireless communication (e.g., communication at a relatively short distance) without using the communication network 41. In some embodiments, the wireless communication may use a short-range wireless communication technology. Non-limiting examples of short-range wireless communication technologies include near field communication (NFC) and Bluetooth (registered trademark). In some embodiments, two or more (or in some cases, all) units of the vehicle control system 11 may be connected using the communication network 41 and a wireless communication technology (e.g., a short-range wireless communication technology).

[0151] Hereinafter, in an embodiment in which two or more units of the vehicle control system 11 communicate with each other via the communication network 41, the description of the communication network 41 will be omitted. For example, in an embodiment in which the vehicle control ECU 21 and the communication unit 22 communicate with each other via the communication network 41, it will simply be described that the vehicle control ECU 21 and the communication unit 22 communicate with each other.

[0152] The vehicle control ECU 21 is configured with various processors such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), etc. The vehicle control ECU 21 controls the entire or part of the functions of the vehicle control system 11.

[0153] The communication unit 22 communicates with various devices inside the vehicle 1 (hereinafter referred to as in-vehicle devices), various devices outside the vehicle 1 (hereinafter referred to as out-vehicle devices), other vehicles, base stations, etc., and transmits and receives various types of data. In some embodiments, the communication unit 22 may communicate using multiple communication technologies.

[0154] A non-limiting example of communication between the communication unit 22 and an external device will now be briefly described. In some embodiments, the communication unit 22 may communicate with a server (hereinafter referred to as an external server) or the like on an external network via a base station or an access point using wireless communication technology. Non-limiting examples of wireless communication technology include 5G (5th Generation Mobile Communication System), LTE (Long Term Evolution), DSRC (Dedicated Short Range Communications), etc. The external network with which the communication unit 22 can communicate is, for example, the Internet, a cloud network, or a network specific to an operator. The communication technology used by the communication unit 22 to communicate with the external network is not particularly limited as long as it is a wireless communication technology that enables digital two-way communication at a communication speed equal to or higher than a predetermined distance.

[0155] In some embodiments, the communication unit 22 may use P2P (Peer to Peer) technology to communicate with a terminal located near the vehicle. The terminal located near the vehicle may be, for example, a terminal attached to a mobile object moving at a relatively slow speed, such as a pedestrian or a bicycle, a terminal installed at a fixed location in a store, and / or an MTC (Machine Type Communication) terminal. In some embodiments, the communication unit 22 may perform V2X (Vehicle to Everything) communication. V2X communication generally refers to communication between the vehicle and another entity. Non-limiting examples of V2X communication include vehicle-to-vehicle communication with another vehicle, vehicle-to-infrastructure communication with a roadside unit, vehicle-to-home communication, and vehicle-to-pedestrian communication with a terminal carried or worn by a pedestrian.

[0156] In some embodiments, the communication unit 22 may receive a program for updating software that controls the operation of the vehicle control system 11 from outside the vehicle 1 (e.g., over the air). In some embodiments, the communication unit 22 may receive map information, traffic information, information about the surroundings of the vehicle 1, etc. from outside the vehicle 1. In some embodiments, the communication unit 22 may transmit information about the vehicle 1 or information about the surroundings of the vehicle 1, etc. to an external device or an external network. Non-limiting examples of information about the vehicle 1 that the communication unit 22 transmits to an external device or an external network include data indicating the status of the vehicle 1, recognition results by the recognition unit 73, etc. In some embodiments, the communication unit 22 may communicate with a vehicle emergency notification system. Non-limiting examples of a vehicle emergency notification system include eCall, etc.

[0157] In some embodiments, the communication unit 22 may receive electromagnetic waves transmitted by a road traffic information communication system. In some embodiments, the electromagnetic waves may be transmitted using a radio beacon, an optical beacon, FM multiplex broadcasting, or the like.

[0158] Non-limiting examples of communication with in-vehicle devices that can be performed by the communication unit 22 will now be briefly described. In some embodiments, the communication unit 22 may communicate with the in-vehicle devices using wireless communication. For example, in some embodiments, the communication unit 22 may communicate with the in-vehicle devices using wireless communication technology that enables bidirectional digital communication at a predetermined communication speed or higher. Non-limiting examples of wireless communication technology include wireless LAN, Bluetooth, NFC, and WUSB (Wireless USB). Alternatively, the communication unit 22 may communicate with the in-vehicle devices using wired communication (in addition to or as an alternative to wireless communication). For example, in some embodiments, the communication unit 22 may communicate with the in-vehicle devices using wired communication via a cable connected to a connection terminal (not shown). In some embodiments, the communication unit 22 may communicate with the in-vehicle devices using wired communication technology that enables bidirectional digital communication at a predetermined communication speed or higher. Non-limiting examples of wired communication technologies include Universal Serial Bus (USB), High-Definition Multimedia Interface (HDMI)®, and Mobile High-definition Link (MHL).

[0159] Here, the in-vehicle devices refer to, for example, devices inside the vehicle 1 that are not connected to the communication network 41. The in-vehicle devices are divided into devices that constitute the vehicle control system 11 and devices that do not constitute the vehicle control system 11. Non-limiting examples of in-vehicle devices that do not constitute the vehicle control system 11 include mobile devices and wearable devices carried by users of the vehicle 1 (for example, the driver or passengers), and information devices that are temporarily installed inside the vehicle 1. These devices can, for example, be moved outside the vehicle 1 and become external devices.

[0160] The map information storage unit 23 stores maps acquired from an external device or an external network and / or maps created by the vehicle 1. For example, the map information storage unit 23 may store a three-dimensional high-precision map, a global map that has lower precision than a high-precision map and covers a wide area, or the like.

[0161] The high-precision map may be, for example, a dynamic map, a point cloud map, a vector map, etc. The dynamic map may be, for example, a map consisting of four layers of dynamic information, quasi-dynamic information, quasi-static information, and static information, and may be provided to the vehicle 1 from an external server or the like. The point cloud map may be a map composed of a point cloud (point cloud data). The vector map may be, for example, a map adapted for automated driving by associating traffic information such as the positions of lanes and traffic lights with the point cloud map.

[0162] The point cloud map and the vector map may be provided, for example, from an external server or the like, or may be created in the vehicle 1 based on sensing results from the camera 51, radar 52, LiDAR 53, etc. as a map for matching with a local map described later, and stored in the map information storage unit 23. Furthermore, when a high-precision map is provided from an external server or the like, map data of, for example, an area of ​​several hundred square meters regarding the planned route along which the vehicle 1 will travel may be acquired from the external server or the like in order to reduce communication capacity.

[0163] The position information acquisition unit 24 acquires position information of the vehicle 1. The acquired position information may be supplied to the driving automation control unit 29. In some embodiments, the position information acquisition unit 24 may receive GNSS (Global Navigation Satellite System) signals from GNSS satellites. In some embodiments, the position information acquisition unit 24 may receive signals from beacons or the like.

[0164] The external recognition sensor 25 includes various sensors used to recognize the situation outside the vehicle 1, and supplies sensor data from one or more (or in some cases, all) sensors to one or more (or in some cases, all) units of the vehicle control system 11. The type and number of sensors included in the external recognition sensor 25 are arbitrary.

[0165] In some embodiments, the external recognition sensor 25 may include a camera 51, a radar 52, a LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging) 53, and an ultrasonic sensor 54. Without being limited to this, the external recognition sensor 25 may be configured to include one or more types of sensors selected from the camera 51, the radar 52, the LiDAR 53, and the ultrasonic sensor 54. The number of cameras 51, radars 52, LiDARs 53, and ultrasonic sensors 54 is not particularly limited as long as they are numbers that can be realistically installed on the vehicle 1. Furthermore, the types of sensors included in the external recognition sensor 25 are not limited to this example, and the external recognition sensor 25 may include other types of sensors. Examples of sensing areas of the sensors included in the external recognition sensor 25 will be described later.

[0166] The camera 51 may use any suitable imaging method. In some embodiments, the camera 51 may use an imaging method capable of distance measurement. Non-limiting examples of cameras using imaging methods capable of distance measurement include a time-of-flight (ToF) camera, a stereo camera, a monocular camera, and an infrared camera. However, the camera 51 may simply acquire an image without distance measurement.

[0167] In some embodiments, the external recognition sensor 25 may include an environmental sensor for detecting characteristics of the environment around the vehicle 1. Non-limiting examples of environmental characteristics that may be detected include weather, climate, brightness, etc. In some embodiments, the environmental sensor may include various sensors such as a rain sensor, a fog sensor, a sunlight sensor, a snow sensor, and an illuminance sensor.

[0168] In some embodiments, the external recognition sensor 25 may include a microphone used to detect sounds around the vehicle 1 and the location of sound sources.

[0169] The interior sensor 26 includes various sensors for detecting information about the interior of the vehicle 1, and supplies sensor data from one or more (or in some cases, all) sensors to one or more (or in some cases, all) units of the vehicle control system 11. The types and number of the various sensors included in the interior sensor 26 are not particularly limited as long as they are of the types and number that can be realistically installed in the vehicle 1.

[0170] In some embodiments, the interior sensor 26 may include one or more sensors selected from the group consisting of a camera, radar, a seating sensor, a microphone, and a biometric sensor. In some embodiments, the camera included in the interior sensor 26 may use an imaging method capable of measuring distances. Non-limiting examples of cameras using imaging methods capable of measuring distances include a Time of Flight (ToF) camera, a stereo camera, a monocular camera, and an infrared camera. The camera included in the interior sensor 26 may also be a camera simply used to acquire captured images, regardless of distance measurement. The biometric sensor included in the interior sensor 26 may be provided, for example, on a seat or a steering wheel, and may detect various types of biometric information of the user.

[0171] The vehicle sensor 27 includes various sensors for detecting the state of the vehicle 1, and supplies sensor data from one or more (or in some cases, all) sensors to one or more (or in some cases, all) units of the vehicle control system 11. The types and number of the various sensors included in the vehicle sensor 27 are not particularly limited as long as they are of the types and number that can be realistically installed on the vehicle 1.

[0172] In some embodiments, the vehicle sensor 27 may include a speed sensor, an acceleration sensor, an angular velocity sensor (gyro sensor), and / or an inertial measurement unit (IMU) that integrates these. In some embodiments, the vehicle sensor 27 may include a steering angle sensor that detects the steering angle of the steering wheel, a yaw rate sensor, an accelerator sensor that detects the amount of accelerator pedal operation (e.g., pedal force, pedal stroke), and / or a brake sensor that detects the amount of brake pedal operation (e.g., pedal force, pedal stroke). In some embodiments, the vehicle sensor 27 may include a rotation sensor that detects the number of rotations of the engine or motor, an air pressure sensor that detects tire air pressure, a slip ratio sensor that detects tire slip ratio, and / or a wheel speed sensor that detects the rotation speed of the wheels. In some embodiments, the vehicle sensor 27 may include a battery sensor that detects the remaining battery level and temperature, and / or an impact sensor that can detect external impacts.

[0173] The storage unit 28 includes at least one of a non-volatile storage medium and a volatile storage medium, and stores data and programs. Non-limiting examples of storage media include magnetic storage devices such as electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), and / or hard disk drives (HDDs), semiconductor storage devices, optical storage devices, and magneto-optical storage devices. The storage unit 28 stores various programs and data used by one or more (or in some cases, all) units of the vehicle control system 11. In some embodiments, the storage unit 28 may include an event data recorder (EDR) or a data storage system for automated driving (DSSAD), and may store information about the vehicle 1 before and after an event such as an accident, as well as information acquired by the in-vehicle sensors 26.

[0174] The driving automation control unit 29 controls the driving automation function of the vehicle 1. In some embodiments, the driving automation control unit 29 may include an analysis unit 61, an action planning unit 62, and an operation control unit 63.

[0175] The analysis unit 61 performs an analysis process of the vehicle 1 and / or the surrounding situation. The analysis unit 61 includes a self-position estimation unit 71, a sensor fusion unit 72, and a recognition unit 73.

[0176] In some embodiments, the self-position estimation unit 71 may estimate the self-position of the vehicle 1 based on sensor data from the external recognition sensor 25 and a high-precision map stored in the map information storage unit 23. For example, the self-position estimation unit 71 may generate a local map based on the sensor data from the external recognition sensor 25 and estimate the self-position of the vehicle 1 by matching the local map with the high-precision map. The position of the vehicle 1 may be based on, for example, the center of the rear wheel pair axle.

[0177] In some embodiments, the local map may be a three-dimensional high-precision map, an occupancy grid map, or the like created using a technique such as SLAM (Simultaneous Localization and Mapping). The three-dimensional high-precision map may be, for example, the point cloud map described above. The occupancy grid map may be a map obtained by dividing a three-dimensional or two-dimensional space around the vehicle 1 into grids of a predetermined size and indicating the occupancy status of objects on a grid-by-grid basis. The occupancy status of an object may be indicated, for example, by the presence or absence of an object or a probability of its presence. In some embodiments, the local map may also be used, for example, in the detection process and / or recognition process of the situation outside the vehicle 1 by the recognition unit 73.

[0178] In some embodiments, the self-position estimation unit 71 may estimate the self-position of the vehicle 1 based on the position information acquired by the position information acquisition unit 24 and / or sensor data from the vehicle sensor 27 .

[0179] The sensor fusion unit 72 performs sensor fusion processing to obtain information by combining multiple different types of sensor data (for example, image data supplied from the camera 51 and sensor data supplied from the radar 52). Methods for combining different types of sensor data include, but are not limited to, compounding, integration, fusion, and association.

[0180] The recognition unit 73 executes a detection process for detecting the situation outside the vehicle 1 and / or a recognition process for recognizing the situation outside the vehicle 1 .

[0181] For example, the recognition unit 73 may perform detection processing and / or recognition processing of the situation outside the vehicle 1 based on information from the external recognition sensor 25, information from the self-position estimation unit 71, information from the sensor fusion unit 72, etc.

[0182] Specifically, for example, the recognition unit 73 may perform a detection process and / or a recognition process of objects around the vehicle 1. The object detection process may be, for example, a process of detecting the presence or absence, size, shape, position, movement, etc. of an object. The object recognition process may be, for example, a process of recognizing attributes such as the type of object, or a process of identifying a specific object. The detection process and the recognition process are not necessarily clearly separated, and there may be at least a partial overlap.

[0183] In some embodiments, the recognition unit 73 may detect objects around the vehicle 1 by performing clustering to classify a point cloud based on sensor data from the radar 52 and / or the LiDAR 53, etc. into clusters of points. This makes it possible to detect the presence, size, shape, and position of objects around the vehicle 1.

[0184] In some embodiments, the recognition unit 73 may detect the movement of objects around the vehicle 1 by tracking the movement of clusters of point clouds classified by clustering. This makes it possible to detect the speed and / or traveling direction (movement vector) of objects around the vehicle 1.

[0185] In some embodiments, the recognition unit 73 may detect and / or recognize vehicles (including bicycles), people, obstacles, structures, roads, traffic lights, traffic signs, road markings, etc. based on image data supplied from the camera 51. In some embodiments, the recognition unit 73 may recognize the type of object around the vehicle 1 by performing recognition processing such as semantic segmentation.

[0186] In some embodiments, the recognition unit 73 may perform a recognition process of traffic rules around the vehicle 1 based on the map stored in the map information storage unit 23, the result of estimation of the self-position by the self-position estimation unit 71, and / or the result of recognition of objects around the vehicle 1 by the recognition unit 73. Through this process, the recognition unit 73 may recognize the position and / or state of traffic lights, the contents of traffic signs and / or road markings, the contents of traffic regulations, and / or lanes that can be traveled, etc.

[0187] In some embodiments, the recognition unit 73 may perform recognition processing of the environment around the vehicle 1. In some embodiments, the recognition unit 73 may recognize weather characteristics (temperature, humidity, brightness) and / or road surface conditions, etc.

[0188] The behavior planning unit 62 creates a behavior plan for the vehicle 1. For example, the behavior planning unit 62 may create a behavior plan by performing route planning and route tracking.

[0189] In some embodiments, path planning may include global path planning and local path planning. Global path planning may include a process of planning a rough route from a start to a goal. Local path planning, also referred to as trajectory planning, may include generating a trajectory that allows the vehicle 1 to proceed safely and smoothly along a planned route in the vicinity of the vehicle 1, taking into account the motion characteristics of the vehicle 1, the presence of any obstacles, and the like.

[0190] In some embodiments, the path following may be a planning of an operation for safely and accurately traveling along a route planned by the route planner within a planned time. The behavior planning unit 62 may, for example, calculate a target speed and / or a target angular velocity of the vehicle 1 based on the result of the path following process.

[0191] The operation control unit 63 controls the operation of the vehicle 1 in order to realize the action plan created by the action planning unit 62 .

[0192] For example, in some embodiments, the operation control unit 63 may control the steering control unit 81, the brake control unit 82, and / or the drive control unit 83 included in the vehicle control unit 32 (described later) to perform lateral vehicle motion control and / or longitudinal vehicle motion control so that the vehicle 1 travels along the trajectory calculated by the trajectory plan. For example, the operation control unit 63 may perform control (e.g., lateral vehicle motion control, longitudinal vehicle motion control) for one or more driver assistance functions and / or driving automation. Non-limiting examples of driver assistance functions include collision avoidance or impact mitigation, following distance control (e.g., control to maintain a specific distance from a vehicle traveling in front of the vehicle 1), vehicle speed control (e.g., control to maintain a specific speed), vehicle collision warning, and lane departure warning. Non-limiting examples of driving automation include driving without operation by a driver or a remote driver.

[0193] In some embodiments, the DMS 30 may perform a driver authentication process and / or a driver state recognition process based on sensor data from the in-vehicle sensors 26 and / or input data input to the HMI 31 (described later), etc. Non-limiting examples of the driver state that may be recognized include physical condition, alertness, concentration, fatigue, gaze direction, level of intoxication, driving operation, posture, etc.

[0194] In some embodiments, the DMS 30 may perform authentication processing of a user other than the driver (e.g., a passenger) and / or recognition processing of the state of the user. In some embodiments, the DMS 30 may perform recognition processing of the interior situation of the vehicle 1 based on sensor data from the interior sensors 26. Non-limiting examples of characteristics of the interior situation of the vehicle 1 that can be recognized include temperature, humidity, brightness, odor, etc.

[0195] The HMI 31 receives various data, instructions, etc. as input, and presents the various data to the user.

[0196] The input of data to the HMI 31 will be briefly described. The HMI 31 includes an input device through which a person inputs data, instructions, etc. The HMI 31 generates an input signal based on the data, instructions, etc. input via the input device and supplies the signal to one or more (or in some cases, all) units of the vehicle control system 11. In some embodiments, the HMI 31 may include a touch panel, buttons, switches, and / or levers as input devices. Without being limited thereto, the HMI 31 may also include an input device that allows information to be input by a method other than manual operation, such as voice or gestures. In some embodiments, the HMI 31 may include an input device such as a remote control device using infrared and / or radio waves, or an externally connected device that can operate the vehicle control system 11. Non-limiting examples of externally connected devices include a mobile device (e.g., a smartphone) and a wearable device (e.g., a smart watch).

[0197] The presentation of data by the HMI 31 will be briefly described. The HMI 31 generates visual information, auditory information, and / or tactile information for the user and / or a person outside the vehicle 1. The HMI 31 may also perform output control, controlling the output, output content, output timing, and / or output method of each piece of generated information. Non-limiting examples of visual information that can be generated and output by the HMI 31 include information displayed by images or lights, such as an operation screen, a status display of the vehicle 1, a warning display, and a monitor image showing the situation around the vehicle 1. Non-limiting examples of auditory information that can be generated and output by the HMI 31 include voice guidance, warning sounds, warning messages, etc. Non-limiting examples of tactile information that can be generated and output by the HMI 31 include information imparted to the user's sense of touch by force, vibration, movement, etc.

[0198] In some embodiments, the HMI 31 may include, as an output device capable of outputting visual information, a display device that presents visual information by displaying an image itself or a projector device that presents visual information by projecting an image. In some embodiments, the display device may be, in addition to or instead of a typical display device, a device that displays visual information within the user's field of view, such as a head-up display, a see-through display, or a wearable device with an augmented reality (AR) function. In some embodiments, the HMI 31 may include, as an output device capable of outputting visual information, a display device included in a navigation device, an instrument panel, a camera monitoring system (CMS), an electronic mirror, a lamp, or the like provided in the vehicle 1.

[0199] In some embodiments, the HMI 31 may include an audio speaker, headphones, or earphones as output devices capable of outputting auditory information.

[0200] In some embodiments, the HMI 31 may include a haptic element using haptic technology as an output device capable of outputting tactile information. The haptic element may be provided on a part of the vehicle 1 that the user comes into contact with, such as the steering wheel or the seat.

[0201] The vehicle control unit 32 controls one or more (or in some cases, all) units of the vehicle 1. The vehicle control unit 32 includes a steering control unit 81, a brake control unit 82, a drive control unit 83, a body system control unit 84, a light control unit 85, and a horn control unit 86.

[0202] The steering control unit 81 detects and / or controls the state of the steering system of the vehicle 1. The steering system includes, for example, a steering mechanism including a steering wheel, an electric power steering, etc. The steering control unit 81 includes, for example, a steering ECU that controls the steering system, an actuator that drives the steering system, etc.

[0203] The brake control unit 82 detects and / or controls the state of the brake system of the vehicle 1. The brake system includes, for example, a brake mechanism including a brake pedal, an antilock brake system (ABS), a regenerative brake mechanism, etc. The brake control unit 82 includes, for example, a brake ECU that controls the brake system, an actuator that drives the brake system, etc.

[0204] The drive control unit 83 detects and / or controls the state of the drive system of the vehicle 1. The drive system includes, for example, an accelerator pedal, a drive force generating device for generating drive force such as an internal combustion engine or a drive motor, and a drive force transmission mechanism for transmitting the drive force to the wheels. The drive control unit 83 includes, for example, a drive ECU for controlling the drive system, and an actuator for driving the drive system.

[0205] The body system control unit 84 detects and / or controls the states of the body system systems of the vehicle 1. The body system systems include, for example, a keyless entry system, a smart key system, a power window device, a power seat, an air conditioning system, an airbag, a seat belt, a shift lever, etc. The body system control unit 84 includes, for example, a body system ECU that controls the body system systems, an actuator that drives the body system systems, etc.

[0206] The light control unit 85 detects and / or controls the states of various lights of the vehicle 1. Non-limiting examples of lights that can be controlled by the light control unit 85 include headlights, backlights, fog lights, turn signals, brake lights, projector lights, and bumper indicators. The light control unit 85 includes a light ECU that controls the lights, an actuator that drives the lights, and the like.

[0207] The horn control unit 86 detects and / or controls the state of the car horn of the vehicle 1. The horn control unit 86 includes, for example, a horn ECU that controls the car horn, an actuator that drives the car horn, and the like.

[0208] Fig. 21 is a diagram showing an example of a sensing area by the camera 51, the radar 52, the LiDAR 53, the ultrasonic sensor 54, etc. of the external recognition sensor 25 in Fig. 20. Fig. 21 schematically shows the vehicle 1 as viewed from above.

[0209] Sensing area 101F and sensing area 101B show examples of sensing areas of the ultrasonic sensors 54. Sensing area 101F (e.g., sensing area of ​​the multiple ultrasonic sensors 54) covers the periphery of the front end of the vehicle 1. Sensing area 101B (e.g., sensing area of ​​the multiple ultrasonic sensors 54) covers the periphery of the rear end of the vehicle 1.

[0210] The sensing results in sensing area 101F and / or sensing area 101B may be used, for example, for parking assistance for vehicle 1.

[0211] Sensing area 102F, sensing area 102B, sensing area 102L, and sensing area 102R show examples of sensing areas of a short-range or medium-range radar 52. Sensing area 102F covers a position farther in front of the vehicle 1 than sensing area 101F. Sensing area 102B covers a position farther behind the vehicle 1 than sensing area 101B. Sensing area 102L covers the surrounding area behind the left side of the vehicle 1. Sensing area 102R covers the surrounding area behind the right side of the vehicle 1.

[0212] The sensing results in the sensing area 102F may be used, for example, to detect vehicles, pedestrians, etc. present in front of the vehicle 1. The sensing results in the sensing area 102B may be used, for example, for a collision prevention function behind the vehicle 1. The sensing results in the sensing area 102L and / or the sensing area 102R may be used, for example, to detect one or more objects in blind spots on the left and / or right sides of the vehicle 1.

[0213] Sensing area 103F, sensing area 103B, sensing area 103L, and sensing area 103R show examples of sensing areas sensed by camera 51. Sensing area 103F covers a position farther in front of vehicle 1 than sensing area 102F. Sensing area 103B covers a position farther behind vehicle 1 than sensing area 102B. Sensing area 103L covers the periphery on the left side of vehicle 1. Sensing area 103R covers the periphery on the right side of vehicle 1.

[0214] The sensing results in sensing area 103F may be used, for example, for recognizing traffic lights and traffic signs, a lane departure prevention assistance system, or an automatic headlight control system. The sensing results in sensing area 103B may be used, for example, for parking assistance and / or a surround view system. The sensing results in sensing area 103L and / or sensing area 103R may be used, for example, for a surround view system.

[0215] Sensing area 104 shows an example of the sensing area of ​​LiDAR 53. Sensing area 104 covers a position farther ahead of vehicle 1 than sensing area 103F. On the other hand, sensing area 104 has a narrower range in the left-right direction of vehicle 1 than sensing area 103F.

[0216] The sensing results in the sensing area 104 may be used to detect objects such as surrounding vehicles, for example.

[0217] Sensing area 105 shows an example of the sensing area of ​​the long-range radar 52. Sensing area 105 covers a position further ahead of the vehicle 1 than sensing area 104. On the other hand, sensing area 105 has a narrower range in the left-right direction of the vehicle 1 than sensing area 104.

[0218] The sensing results in the sensing area 105 may be used for, for example, adaptive cruise control (ACC), emergency braking, collision avoidance, and the like.

[0219] In some embodiments, the sensing area of ​​each of the external recognition sensors 25 (e.g., the camera 51, the radar 52, the LiDAR 53, and the ultrasonic sensor 54) may have various configurations other than the configuration shown in Fig. 21. Specifically, in some embodiments, the ultrasonic sensor 54 may also sense the sides of the vehicle 1, and the LiDAR 53 may sense the rear of the vehicle 1. Furthermore, the installation position of each sensor is not limited to the above-mentioned examples. Furthermore, the number of each sensor may be one or more.

[0220] An example of a vehicle control system to which the autonomous driving technology according to the present disclosure can be applied has been described above. The technology according to the present disclosure can be applied to the external recognition sensor 25 of the above-described configuration. Specifically, for example, the imaging devices according to the first to fourth embodiments described above can be applied to the external recognition sensor 25. By applying the technology according to the present disclosure to the vehicle control system 11, it is possible to improve the accuracy of event detection.

[0221] Note that the above-described embodiment shows an example for realizing the present technology, and the matters in the embodiment and the matters specifying the invention in the claims correspond to each other. Similarly, the matters specifying the invention in the claims and the matters in the embodiment of the present technology with the same title correspond to each other. However, the present technology is not limited to the embodiment, and can be realized by applying various modifications to the embodiment within the scope of the gist. Furthermore, the effects described in this specification are merely examples and are not limited, and other effects may also be present.

[0222] The present technology may also be configured as follows: (1) A photodetector comprising: an event pixel used to detect an event; and a denoising processing unit that performs denoising on the event based on an influence value that affects noise detected in the event pixel. (2) The influence value includes at least one of luminance, a sensor threshold, and a temperature at the time of event detection. (3) The photodetector according to (1) or (2), further comprising a luminance information acquisition unit that acquires luminance information at the time of event detection. (4) The photodetector according to (3), further comprising grayscale pixels used to detect gradations, wherein the luminance information acquisition unit acquires the luminance information from the grayscale pixels. (5) The photodetector according to (3) or (4), further comprising a grayscale image generation unit that generates a grayscale image based on an event image, wherein the luminance information acquisition unit acquires the luminance information from the grayscale image. (6) The photodetector according to any of (3) to (5), further comprising an edge noise removal unit that removes noise adjacent to an edge of the event image based on the luminance information. (7) The photodetector according to any one of (3) to (6), further comprising: a luminance sensor that detects luminance; and an application processor that generates luminance information based on a luminance value output from the luminance sensor and passes the luminance information to the luminance information acquisition unit. (8) The photodetector according to claim 1, wherein the denoising processing unit comprises: a spatial filter that calculates an event density in a filter kernel; an event density table that registers a relationship between the influence value and the event density; a denoising threshold setting unit that sets a denoising threshold based on the relationship between the influence value and the event density; and a threshold / thinning-out processing unit that denoises the event based on the event density and the denoising threshold. (9) A photodetection method that detects an event based on received light and performs denoising processing on the event based on an influence value that affects noise that is erroneously detected as the event. (10) A denoising program that causes a computer to perform a process of referring to an influence value that affects noise that is erroneously detected as an event, and a process of denoising the event based on a result of referring to the influence value.

[0223] REFERENCE SIGNS LIST 10 Brightness sensor 12 Imaging device 12A Brightness information acquisition unit 12B Denoising processing unit 13 Application processor 91 Optical system 92 Solid-state imaging device 93 Imaging control unit 94 Image processing unit 95 Storage unit 96 Display unit 97 Operation unit 98 Communication unit 99 Bus 111 Pixel array unit 113 Event signal output unit 115 Access control unit 116 Timing control unit 117 Event signal processing unit 121 Event pixel 122 Vertical event signal line 123 Horizontal control line

Claims

1. A photodetection device comprising: an event pixel used to detect an event; and a denoising processing unit that performs denoising on the event based on an influence value that affects the noise detected in the event pixel.

2. The light detection device according to claim 1, wherein the influence value includes at least one of the luminance, the sensor threshold, and the temperature at the time when the event is detected.

3. The light detection device according to claim 1, further comprising a luminance information acquisition unit that acquires luminance information at the time of event detection.

4. The light detection device according to claim 3, further comprising gradation pixels used for detecting gradations, wherein the luminance information acquisition unit acquires the luminance information from the gradation pixels.

5. The light detection device according to claim 3, further comprising a gradation image generating unit that generates a gradation image based on an event image, wherein the luminance information acquiring unit acquires the luminance information from the gradation image.

6. The light detection device according to claim 3, further comprising an edge noise removal unit that removes noise adjacent to an edge of the event image based on the luminance information.

7. The light detection device according to claim 3, further comprising: a luminance sensor that detects luminance; and an application processor that generates luminance information based on the luminance value output from the luminance sensor and passes the luminance information to the luminance information acquisition unit.

8. The optical detection device of claim 1, wherein the denoising processing unit comprises: a spatial filter that calculates an event density within a filter kernel; an event density table that registers the relationship between the influence value and the event density; a denoising threshold setting unit that sets a denoising threshold based on the relationship between the influence value and the event density; and a threshold / thinning-out processing unit that densifies the events based on the event density and the denoising threshold.

9. A light detection method comprising: detecting an event based on received light; and performing denoising processing on the event based on an influence value that affects noise that is erroneously detected as the event.

10. A denoising program that causes a computer to execute a process of referring to an influence value that affects noise that is erroneously detected as an event, and a denoising process of the event based on the result of referring to the influence value.

Images