Noise event removal device, optical detection device, and noise event removal method

Abstract

[Problem] To provide a noise event removal device making it possible to maintain the detection accuracy of true events while also removing flicker noise events. [Solution] An optical detection device according to one embodiment of the present disclosure is provided with: a noise event mask unit for performing mask processing to remove illumination flicker noise from an event signal indicating a result of detecting, as an event, a change in the amount of light reception at pixels when a subject lit by the illumination was imaged in a predetermined frame; and an event restoration unit for performing event restoration processing to restore a true event that was lost due to the mask processing.

Description

Noise event removal device, photodetection device, and noise event removal method

[0001] The present disclosure relates to a noise event removal device, a light detection device, and a noise event removal method.

[0002] Photodetection devices are known that detect changes in the amount of light received by pixels as events in an imaging scene capturing an image of a subject. Such photodetection devices are sometimes called event-based vision sensors (EVSs). When such EVSs are used in lighting environments where flicker occurs, the EVSs may output changes in the illuminance of the lighting as events. In such cases, the output of the EVSs contains flicker noise events. Therefore, techniques for removing these flicker noise events have been proposed. For example, a technique is known that changes the event detection threshold so that the EVSs do not respond to lighting flicker.

[0003] JP 2023-93778 A

[0004] Simply changing the event detection threshold in response to flicker, as in the above technique, may make it difficult to detect true events that are not flicker noise events, which may result in a decrease in the accuracy of detecting true events.

[0005] The present disclosure provides a noise event removal device, a light detection device, and a noise event removal method that can remove flicker noise events while maintaining the accuracy of detecting true events.

[0006] A noise event removal device according to one embodiment of the present disclosure includes a noise event masking unit that performs masking processing to remove lighting flicker noise from an event signal that indicates the result of detecting, as an event, a change in the amount of light received by a pixel when an image of a subject illuminated by lighting is captured in a specified frame, and an event restoration unit that performs event restoration processing to restore a true event that has been lost by the masking processing.

[0007] In the masking process, the noise event masking unit may subtract an event signal of a correction target frame from which the flicker noise is to be removed by an event signal of a previous frame by one cycle of the flicker noise.

[0008] In the masking process, the noise event mask unit may group frames into groups of an arbitrary number, extract past frames that belong to a group immediately before the group to which the correction target frame, from which the flicker noise is to be removed, belongs, extract, from the past frames, the positions of the pixels at which an event with a large number of events has been detected out of two types of events determined according to changes in the amount of received light, generate a mask frame based on the positions of the pixels, and mask the correction target frame with the mask frame.

[0009] In the event restoration process, the event restoration unit may generate an extraction frame by extracting one of two types of events determined according to a change in the amount of received light from a correction target frame that is the target for removing the flicker noise, and a past extraction frame by extracting the one event from a past frame that immediately precedes the correction target frame, perform an integration process on the extraction frame and the past extraction frame, and perform an addition process on the masked frame and the integration process on the frame.

[0010] When the intensity of the flicker noise exceeds a preset reference value, the noise event masking unit may perform the masking process, and the event restoration unit may perform the event restoration process.

[0011] The noise event removal device may be provided in a photodetection device that includes the pixel.

[0012] The noise event removal device may be provided in an application processor arranged in a stage subsequent to the photodetection device including the pixels.

[0013] An optical detection device according to one embodiment of the present disclosure includes an event detection device that outputs an event signal indicating the result of detecting, as an event, a change in the amount of light received by a pixel when an image of an object illuminated by illumination is captured in a predetermined frame; a flicker noise intensity calculation device that calculates the intensity of flicker noise included in the event signal; and a noise event removal device that performs a masking process that removes the flicker noise of the illumination from the event signal according to the intensity of the flicker noise, and an event restoration process that restores a true event that has been lost by the masking process.

[0014] The flicker noise intensity calculation device may count a number of events corresponding to the number of pixels in which events are detected from the event signal, and calculate the intensity of the flicker noise based on a result of a Fourier transform of the number of events.

[0015] The flicker noise intensity calculation device may calculate the intensity of the flicker noise based on the difference between the maximum number of events and the minimum number of events at any given time for two types of events determined according to the change in the amount of received light.

[0016] The event detection device may further include an event signal detection circuit that generates the event signal, and one event signal detection circuit may be provided for one pixel.

[0017] The event detection device may further include an event signal detection circuit that generates the event signal, and one event signal detection circuit may be provided for a plurality of the pixels that are arranged close to one another.

[0018] The event detection device may include an event detection pixel that detects the event, and a color pixel that detects color light.

[0019] The photodetector may further include a control device that determines whether to cause the noise event elimination device to perform the masking process based on the intensity of the flicker noise.

[0020] A noise event removal method according to one embodiment of the present disclosure performs a masking process to remove flicker noise from an event signal indicating the result of detecting, as an event, a change in the amount of light received by a pixel when an image of a subject illuminated by a light source is captured in a predetermined frame, and performs an event restoration process to restore a true event that was lost by the masking process.

[0021] 12. FIG. 13 is a block diagram showing an example of the configuration of a photodetector according to the first embodiment. FIG. 14 is a diagram showing an example of the circuit configuration of a pixel. FIG. 15 is a block diagram showing an example of the configuration of an event signal detection circuit. FIG. 16 is a diagram showing an example of an event detection target. FIG. 17 is a flowchart showing the processing operations of the photodetector and control device according to the first embodiment. FIG. 18 is a diagram showing an example of the correspondence relationship between lighting flicker and the output of an event signal. FIG. 19 is a diagram for explaining masking processing. FIG. 20 is a diagram for explaining the disappearance phenomenon of a true event due to masking processing. FIG. 21 is a diagram for explaining an example of event restoration processing. FIG. 22 is a diagram for explaining another example of event restoration processing. FIG. 23 is a diagram showing an example of the relationship between the presence or absence of flicker and the output of an event signal. FIG. 24 is a diagram for explaining a method of calculating a flicker frequency using an autocorrelation coefficient. FIG. 25 is a flowchart of masking processing according to a third embodiment. FIG. 26 is a diagram for explaining each step of the flowchart shown in FIG. 12. FIG. 17 is a block diagram showing an example of the configuration of a photodetector according to Modification 1. FIG. 27 is a plan view showing a pixel configuration according to Modification 2. FIG. 28 is a block diagram showing an example of the configuration of a photodetector according to Modification 3. FIG. 29 is a block diagram showing an example of the configuration of a photodetector according to Modification 4. FIG. 29 is a block diagram showing an example of the configuration of a photodetector according to Modification 5. FIG. 29 is a block diagram showing an example of the schematic configuration of a vehicle control system. FIG. 29 is an explanatory diagram showing an example of the installation positions of an outside vehicle information detection unit and an imaging unit.

[0022] Hereinafter, embodiments of a noise event removal device, a photodetection device, and a noise event removal method according to the present disclosure will be described with reference to the drawings. The following description will focus on the main components of the noise event removal device, the photodetection device, and the noise event removal method. However, the noise event removal device, the photodetection device, and the noise event removal method described below may include components and functions that are not shown or described. The following description does not exclude components and functions that are not shown or described.

[0023] First Embodiment Fig. 1 is a block diagram showing an example of the configuration of a photodetector according to a first embodiment. The photodetector 10 shown in Fig. 1 detects, as an event, a change in the amount of light received in a scene in which an object is captured at a predetermined frame rate, based on the control of a control device 210. In this embodiment, the control device 210 is provided in an application processor 20 arranged downstream of the photodetector 10. The photodetector 10 also includes an event detector 11, a flicker noise intensity calculator 12, and a noise event remover 13.

[0024] The event detection device 11 is an asynchronous imaging device called an EVS, and includes pixels 111 and an event signal detection circuit 112. In this embodiment, the event detection device 11 includes a pixel array unit in which a plurality of pixels 111 are two-dimensionally arranged in a matrix (array). One event signal detection circuit 112 is provided for each of the plurality of pixels 111.

[0025] 2 is a diagram showing an example of the circuit configuration of a pixel 111. The pixel 111 has a light receiving unit 113 and a pixel signal generating unit 114. The light receiving unit 113 photoelectrically converts incident light to generate a photocurrent. The light receiving unit 113 also supplies the generated photocurrent to the pixel signal generating unit 114 and the event signal detection circuit 112. The pixel signal generating unit 114 generates and outputs a pixel signal SIG based on a voltage corresponding to the photocurrent supplied from the light receiving unit 113. This pixel signal SIG is converted into a digital signal. This digital signal is also subjected to predetermined signal processing such as CDS (Correlated Double Sampling) processing and image recognition processing.

[0026] The light receiving unit 113 includes a photoelectric conversion element PD, a transfer transistor Q11, and an OFG (Overflow Gate) transistor Q12. The photoelectric conversion element PD may be, for example, a photodiode. The transfer transistor Q11 and the OFG transistor Q12 may be, for example, an N-type MOS (Metal Oxide Semiconductor) transistor. The transfer transistor Q11 and the OFG transistor Q12 are connected in series.

[0027] The photoelectric conversion element PD is connected between the common connection node Nd of the transfer transistor Q11 and the OFG transistor Q12 and the ground, and performs photoelectric conversion on the incident light to generate an amount of charge corresponding to the amount of incident light.

[0028] A transfer signal TRG is supplied to the gate electrode of the transfer transistor Q11 from the drive unit 115. The transfer transistor Q11 supplies the charge photoelectrically converted by the photoelectric conversion element PD to the pixel signal generation unit 114 in response to the transfer signal TRG.

[0029] A control signal OFG is supplied to the gate electrode of the OFG transistor Q12 from the driver 115. In response to the control signal OFG, the OFG transistor Q12 supplies the electrical signal generated by the photoelectric conversion element PD to the event signal detection circuit 112. The electrical signal supplied to the event signal detection circuit 112 is a photocurrent made up of electric charges.

[0030] The pixel signal generation unit 114 includes a reset transistor Q21, an amplifier transistor Q22, a select transistor Q23, and a floating diffusion layer FD. The reset transistor Q21, the amplifier transistor Q22, and the select transistor Q23 are, for example, N-type MOS transistors.

[0031] The pixel signal generation unit 114 receives charges photoelectrically converted by the photoelectric conversion element PD from the light receiving unit 113 via the transfer transistor Q11. The charges supplied from the light receiving unit 113 are accumulated in the floating diffusion layer FD. The floating diffusion layer FD generates a voltage signal having a voltage value corresponding to the amount of accumulated charge. In other words, the floating diffusion layer FD converts the charges into a voltage.

[0032] The reset transistor Q21 is connected to the power supply voltage V ＤＤ The reset transistor Q21 is connected between the power supply line and the floating diffusion layer FD. A reset signal RST is supplied to the gate electrode of the reset transistor Q21 from the drive unit 115. In response to the reset signal RST, the reset transistor Q21 initializes (resets) the amount of charge in the floating diffusion layer FD.

[0033] The amplifier transistor Q22 is connected to the power supply voltage V ＤＤ The amplifier transistor Q22 is connected in series with the selection transistor Q23 between the power supply line and the vertical signal line VSL. The amplifier transistor Q22 amplifies the voltage signal converted from charge to voltage by the floating diffusion layer FD.

[0034] A selection signal SEL is supplied to the gate electrode of the selection transistor Q23 from the drive unit 115. In response to the selection signal SEL, the selection transistor Q23 outputs the voltage signal amplified by the amplification transistor Q22 as a pixel signal SIG to the vertical signal line VSL.

[0035] In the pixel 111 configured as described above, when the control device 210 shown in FIG. 1 instructs the drive unit 115 to start detecting an event, the drive unit 115 supplies a control signal OFG to the OFG transistor Q12 of the light receiving unit 113. This drives the OFG transistor Q12, and a photocurrent is supplied to the event signal detection circuit 112.

[0036] Subsequently, when any of the multiple pixels 111 detects an event, the OFG transistor Q12 of that pixel 111 is turned off under the control of the drive unit 115. This stops the supply of photocurrent to the event signal detection circuit 112. Subsequently, when the transfer transistor Q11 is driven based on the transfer signal TRG supplied from the drive unit 115, the charge photoelectrically converted by the photoelectric conversion element PD is transferred to the floating diffusion layer FD.

[0037] The above-described configuration of the pixel 111 is merely an example, and the present invention is not limited to this configuration example. For example, a pixel configuration not including the pixel signal generation unit 114 is also possible. In this pixel configuration, the OFG transistor Q12 is omitted from the light receiving unit 113, and the function of the OFG transistor Q12 is given to the transfer transistor Q11.

[0038] The event signal detection circuit 112 converts the photocurrent from the light receiving unit 113 into a voltage and detects whether an event has occurred by comparing the amount of change in this voltage over a predetermined time period with a predetermined threshold. The event signal detection circuit 112 distinguishes between a rising (positive) event, which indicates that the amount of change in voltage has exceeded a predetermined positive threshold, and a falling (negative) event, which indicates that the amount of change in voltage has fallen below a predetermined negative threshold. Information indicating whether a rising event or a falling event has been detected is represented in units of one bit of the event signal.

[0039] Fig. 3 is a block diagram showing an example of the configuration of the event signal detection circuit 112. The event signal detection circuit 112 shown in Fig. 3 includes a current-voltage conversion unit 1121, a buffer 1122, a subtractor 1123, a quantizer 1124, and a transfer unit 1125. However, the configuration of the event signal detection circuit 112 shown in Fig. 3 is just an example, and the present invention is not limited to this example configuration.

[0040] The current-voltage converter 1121 converts the photocurrent from the light receiving unit 113 of the pixel 111 into a logarithmic voltage signal. The current-voltage converter 1121 supplies the converted voltage signal to the buffer 1122. The buffer 1122 buffers the voltage signal supplied from the current-voltage converter 1121 and supplies it to the subtractor 1123.

[0041] The subtractor 1123 receives a row drive signal from the driver 115. In accordance with the row drive signal, the subtractor 1123 reduces the level of the voltage signal supplied from the buffer 1122. The subtractor 1123 then supplies the voltage signal after the level reduction to the quantizer 1124.

[0042] The quantizer 1124 quantizes the voltage signal supplied from the subtractor 1123 into a digital signal and outputs it to the transfer unit 1125 as an event signal. The quantizer 1124 has a configuration including a comparator. This comparator receives the voltage signal from the subtractor 1123 as a non-inverting (+) input and a predetermined threshold voltage Vth as an inverting (-) input. This comparator also compares the voltage signal from the subtractor 1123 with the predetermined threshold voltage Vth and outputs a signal indicating the comparison result to the transfer unit 1125 as an event signal.

[0043] The transfer unit 1125 transfers the event signal supplied from the quantizer 1124 to the flicker noise intensity calculation device 12 and the like.

[0044] The flicker noise intensity calculation device 12 receives an event signal output from the event signal detection circuit 112 provided in each of the arrayed pixels 111, counts the number of pixels that detect an event (number of events), and calculates the intensity of the flicker noise contained in the event signal by performing a Fourier transform on the change in the number of events over time. The flicker noise intensity calculation device 12 also outputs the calculated flicker noise intensity to the control device 210. The control device 210 determines whether or not to perform noise event removal processing in the noise event removal device 13 depending on the flicker noise intensity.

[0045] The noise event removal device 13 has a noise event masking unit 131 and an event restoration unit 132. The noise event masking unit 131 removes an event signal indicating flicker noise based on the flicker intensity calculated by the flicker noise intensity calculation device 12. At this time, for reasons described below, not only the flicker noise but also the true event may be removed. Therefore, the event restoration unit 132 restores the true event removed by the noise event masking unit 131. The event signal from which the flicker noise has been removed and the true event has been restored is transmitted to the application processor 20.

[0046] 1 , the application processor 20 includes a subsequent-stage application device 211 and a control device 210. In addition to determining whether or not the noise event removal process is required, the control device 210 also sets a threshold value for event detection in the event signal detection circuit 112. Specifically, the control device 210 sets a predetermined threshold voltage Vth for the quantizer 1124 in the event signal detection circuit 112.

[0047] The subsequent stage application device 211 performs various information processing on the pixel signals and the event signals to recognize a subject included in an image capture scene captured by the pixels 111. For example, the event signal can be used to grasp the contour shape of the subject, and the subsequent stage application device 211 can recognize the subject by performing a known pattern matching process or the like. When multiple subjects are present in an image capture scene, the subsequent stage application device 211 may recognize these multiple subjects.

[0048] Fig. 4 is a diagram showing an example of an event detection target. In Fig. 4, a subject 310 moves in one direction (to the right in Fig. 4) above a background 320 in an environment illuminated by lighting 300. The brightness of the subject 310 is higher than the brightness of the background 320. That is, in Fig. 4, the bright subject 310 moves above the dark background 320. Here, a description will be given of the processing operation performed by the light detection device 10 and the control device 210 to detect the movement of the subject 310 in the environment shown in Fig. 4 as an event.

[0049] 5 is a flowchart showing the processing operations of the photodetector 10 and the control device 210 according to the first embodiment. In this flowchart, the event detector 11 of the photodetector 10 first performs an event detection process (step S1). In step S1, the event signal detection circuit 112 of the event detector 11 outputs an event signal indicating the rising event and the falling event, respectively, based on a change in the amount of light received by the pixel 111.

[0050] The number of events, which corresponds to the number of pixels in which an event was detected at each time, is stored in the flicker noise intensity calculation device 12 (step S2). The flicker noise intensity calculation device 12 determines whether the number of stored event counts exceeds a predetermined number (step S3). If the number of stored events is equal to or less than the predetermined number (step S3: No), the operations of steps S1 and S2 described above are repeated. If the number of stored events exceeds the predetermined number (step S3: Yes), the flicker noise intensity calculation device 12 then performs a flicker noise intensity calculation process (step S4). In step S4, the flicker noise intensity calculation device 12 calculates the flicker noise intensity by Fourier transforming the array of the stored event counts. Here, the flicker noise intensity calculation process will be described with reference to FIG. 6.

[0051] Fig. 6 is a diagram showing an example of the correspondence relationship between the flicker of the lighting 300 and the output of the event signal. In the graph shown at the top of Fig. 6, the horizontal axis represents time and the vertical axis represents the illuminance of the lighting 300. As shown in this graph, when the lighting 300 is a flicker lighting, the illuminance changes with a period T.

[0052] 6, the horizontal axis indicates time, and the vertical axis indicates the number of events in one sampling. In this graph, the number of rising events P １ is shown by a solid line, and the number of falling events N １ is shown by the dotted line.

[0053] As shown in FIG. 6, the number of rising events P １When the illuminance of the lighting 300 increases, the number of events N increases as the rate of increase increases, and decreases as the rate of increase decreases. １ When the illuminance of the light 300 decreases, the number of events P increases as the rate of decrease increases, and decreases as the rate of decrease decreases. １ and the number of events N １ changes in response to the period T of the flicker of the lighting 300.

[0054] In this embodiment, the flicker noise intensity calculation device 12 performs frequency analysis on the event signals of rising and falling events using a Fourier transform. The flicker frequency is typically a positive multiple of the power supply frequency of the lighting device 300. Therefore, for example, if the power supply frequency of the lighting device 300 is 50 Hz and the flicker frequency is twice the power supply frequency, the flicker noise intensity calculation device 12 performs a Fourier transform at 100 Hz and outputs the result as the flicker noise intensity to the control device 210. Alternatively, if the power supply frequency of the lighting device 300 is 60 Hz and the flicker frequency is twice the power supply frequency, the flicker noise intensity calculation device 12 performs a Fourier transform at 120 Hz and outputs the result as the flicker noise intensity to the control device 210. This completes the flicker noise intensity calculation process. After the flicker noise intensity calculation process is completed, the flicker noise intensity calculation device 12 deletes the stored event count information (step S5).

[0055] Next, the control device 210 determines whether the flicker noise intensity calculated by the flicker noise intensity calculation process exceeds a preset reference value (step S6). If the flicker noise intensity is equal to or less than the reference value (step S6: NO), it is highly likely that the flicker noise event is not included in the event signal output from the event signal detection circuit 112. In this case, the masking process by the noise event mask unit 131 and the event restoration process by the event restoration unit 132 are invalid, and the process according to this flowchart ends.

[0056] If the flicker noise intensity exceeds a preset reference value (step S3: YES), the masking process by the noise event masking unit 131 and the event restoration process by the event restoration unit 132 are enabled (step S8). In this way, when the control device 210 determines whether or not to perform masking process by the noise event removal device 13 based on the flicker noise intensity, the masking process is not performed all the time, but is performed only when there is a high probability that the event detection device 11 has detected a flicker noise event. This makes it possible to reduce the processing load on the entire photodetection device 10.

[0057] The masking process will be described in detail below with reference to FIG.

[0058] 7 is a diagram illustrating the masking process. As shown in FIG. 7, in a certain frame F1, the imaging target in an imaging area 311 changes from a background portion 320 to a subject 310, so the amount of light received by the pixel 111 increases. As a result, the event detection device 11 detects a true rising event.

[0059] In addition, in the imaging area 312 in frame F1, the imaging target changes from the subject 310 to the background 320, so the amount of light received by the pixel 111 decreases. As a result, the event detection device 11 detects a true falling event.

[0060] Furthermore, in imaging area 313 between imaging area 311 and imaging area 312, all of the objects to be imaged are subject 310. Therefore, in response to a change in the illuminance of the flicker of lighting 300, event detection device 11 outputs, for example, event signal Es1 indicating a rising event of flicker noise in frames F1 and F7, and outputs event signal Es2 indicating a falling event of flicker noise in frames F4 and F10, which are different from frames F1 and F7.

[0061] The intensity of the flicker noise increases or decreases in units of n frames (n = 6 in FIG. 7 ), which corresponds to the flicker period T of the lighting 300 (see FIG. 6 ). Therefore, the noise event mask unit 131 subtracts the event signal of a correction target frame, from which flicker noise is to be removed, from the event signal of a frame one flicker noise period back, in other words, n frames back. In this embodiment, the number of frames n is set by the control device 210. The control device 210 sets the number of frames n corresponding to the flicker period T, for example, based on the equation n = f1 / f2. In this equation, f1 represents the sampling frequency of the event detection device 11. Furthermore, f2 represents the frequency at which the flicker intensity calculated in the flicker noise intensity calculation process is highest. The number of frames n is optimized by the control device 210 according to the frequency of the flicker intensity.

[0062] The above-described masking process removes the event signal that detected the flicker noise event of the lighting 300 from the detection results of the event detection device 11, leaving the true event signals detected in the imaging areas 311 and 312. However, with the above-described masking process, if the location where the true event occurred passes through a location where flicker noise previously occurred as the subject 310 moves, the true event may be mistakenly lost. Here, the phenomenon of the true event disappearing due to the masking process will be described with reference to FIG. 8 .

[0063] FIG. 8 is a diagram illustrating the disappearance of a true event due to masking. In FIG. 8, multiple subjects 310 arranged at intervals in the movement direction move simultaneously on a background 320. For example, suppose that in a certain frame, the event detection device 11 detects flicker noise in an imaging area 314 and then detects a true event in an imaging area 315 located behind the imaging area 314 in the movement direction. Then, suppose that the event detection device 11 detects a true event in the imaging area 314 in a frame following the period T of the flicker noise, in other words, frame number n. In this case, in the above-described masking process, this true event is considered a flicker noise event and is mistakenly lost due to the subtraction process of the event signal.

[0064] Therefore, in this embodiment, the event restoration section 132 of the noise event removal device 13 performs event restoration processing. Here, the event restoration processing will be described in detail with reference to Figs. 9A and 9B.

[0065] 9A is a diagram illustrating an example of the event restoration process, in which the true event erroneously lost in the masking process is a falling event (Neg event).

[0066] First, the event restoration unit 132 saves M (M=2 in FIG. 9A) previous frames F11 and F12 immediately before the correction target frame F0 on which the event restoration process is performed (step S801).

[0067] Next, the event restoration unit 132 performs a falling event extraction process (step S802). In step S802, the event restoration unit 132 first generates an extracted frame F20 by extracting a falling event (first event) from the correction target frame F0. The event restoration unit 132 also generates a previously extracted frame F21 by extracting a falling event from a previous frame F11 that is one frame before the correction target frame F0. The event restoration unit 132 also generates a previously extracted frame F22 by extracting a falling event from a previous frame F12 that is two frames before the correction target frame F0.

[0068] Next, the event reconstruction unit 132 performs an integration process on the extracted frame F20 and the previous extracted frame F21 to generate an extracted frame F31 in which falling events occurring in both frames are extracted. Similarly, the event reconstruction unit 132 performs an integration process on the extracted frame F20 and the previous extracted frame F22 to generate an extracted frame F32 in which falling events occurring in both frames are extracted. In this way, falling events occurring in both the current and previous frames are highly likely not flicker noise.

[0069] Next, the event restoration unit 132 performs an event addition process (step S803). In step S803, the event restoration unit 132 first generates an extracted frame F41 by extracting a rising event (Pos event) from frame F40, which is obtained by masking the correction target frame F0 (step S4). The event restoration unit 132 also extracts a falling event from frame F40 and adds the extracted frame 32 and extracted frame 33 generated in step S802 to the extracted frame to generate an added frame F42.

[0070] Finally, the event restoration unit 132 adds the extracted frame F41 and the added frame F42 to complete the restored frame F50. The event restoration unit 132 then outputs the restored frame F50 to the subsequent application device 211.

[0071] 9B is a diagram illustrating another example of event restoration processing. Here, the following mainly describes the differences from the event restoration processing shown in FIG. 9A. In step S801 of this event restoration processing, the processing results immediately before the correction target frame F0 are used as past frames F11 and F12. That is, the past frames F11 and F12 are created from frames obtained by performing the processing of step S83 on the frame immediately before the correction target frame F0.

[0072] In the photodetector 10 according to this embodiment, if the threshold value for detecting an event by the event signal detection circuit 112 is increased, the amount of noise caused by flicker decreases, but as a drawback, the amount of true events that occur decreases.

[0073] Furthermore, the higher the flicker frequency, the greater the amount of noise generated per frame.

[0074] On the other hand, if the sampling frequency of the photodetector 10, which is the interval at which events are detected, is increased, the amount of noise generated per frame due to flicker can be reduced.

[0075] Therefore, from the viewpoint of noise reduction, the following processing can be considered for the sampling frequency of the photodetector 10 and the threshold value for event detection.

[0076] When the intensity of the flicker noise is high and it is predicted that the processing by the noise event elimination device 13 will not be sufficient to reduce the flicker noise, the control device 210 increases the threshold value, thereby reducing the occurrence of noise events.

[0077] In addition, if the intensity of high-frequency flicker noise is high, the amount of noise generated is large at the current sampling frequency, and it is predicted that the processing by the noise event removal device 13 will not be sufficient to reduce noise, the control device 210 sets the sampling frequency of the event detection device 11 to a high value.

[0078] Furthermore, in this embodiment, the accuracy of the masking process and the index indicating the probability of a true event vary depending on the sampling frequency of the event detection device 11. Therefore, it is desirable for the control device 210 to control the noise event removal device 13 so that it is driven optimally. For example, the greater the number M of past frames used in the event restoration process, the easier it is to restore noise caused by flicker. Therefore, the control device 210 sets the number M of past frames to a smaller value as the flicker intensity calculated in the flicker noise intensity calculation process by the flicker noise intensity calculation device 12 increases.

[0079] According to the present embodiment described above, even if a true event is lost in the masking process for removing the flicker noise event, the lost true event can be restored by the event restoration process. Therefore, it is possible to maintain the detection accuracy of the true event while removing the flicker noise event.

[0080] Second Embodiment A second embodiment of the present disclosure will be described. Here, differences from the first embodiment will be mainly described. Furthermore, components similar to those in the first embodiment will be assigned the same reference numerals, and redundant description will be omitted.

[0081] In this embodiment, the content of the flicker noise intensity calculation process performed by the flicker noise intensity calculation device 12 is different from that in the first embodiment. The flicker noise intensity calculation process according to this embodiment will be described below.

[0082] 10 is a diagram showing an example of the relationship between the presence or absence of flicker and the number of events, which corresponds to the number of pixels where an event is detected. In the graph shown in FIG. 10, the horizontal axis represents time, and the vertical axis represents the number of events. In this graph, the number of events P １ indicates the number of pixels where a rising event is detected under a lighting environment with flicker. １ indicates the number of pixels where a falling event is detected under a lighting environment with flicker. ０ indicates the number of pixels where a rising event is detected under a flicker-free lighting environment. ０ indicates the number of pixels where a falling event is detected under a flicker-free lighting environment.

[0083] As shown in Fig. 10, the number of events in a flicker-free lighting environment is close to the number of events during a time when the illuminance change is zero in a flicker-present environment, i.e., the minimum number of events. Therefore, in the flicker noise intensity calculation process according to this embodiment, the flicker noise intensity calculation device 12 calculates the difference ΔPN between the maximum number of events and the minimum number of events at any given time as the flicker intensity. However, in this process, the flicker frequency is unknown. Therefore, in this embodiment, the flicker frequency is calculated by calculating the autocorrelation coefficient.

[0084] The method of calculating the flicker frequency using the autocorrelation coefficient utilizes the property that the value of the autocorrelation coefficient, which is the sum of the product of an original signal V(t) indicating a change in the number of events and a signal g(t) = v(t + Δt) shifted in time from the original signal V(t) by Δt, becomes large when Δt = 1 / f (the reciprocal of the frequency). Below, the method of calculating the flicker frequency using the autocorrelation coefficient will be described with reference to FIG. 11.

[0085] Fig. 11 is a diagram illustrating a method for calculating a flicker frequency using an autocorrelation coefficient. The graph at the top of Fig. 11 shows the waveforms of the original signal V(t) and the signal g(t) = v(t + Δt) when Δt is not the reciprocal of the frequency f. The timing of the peaks differs between these two signal waveforms. As a result, the autocorrelation coefficient becomes smaller.

[0086] On the other hand, the graph at the bottom of Figure 11 shows the waveforms of the original signal V(t) and the signal g(t) = v(t + Δt) when Δt is the reciprocal of the frequency f. The timing of the peaks between these two signal waveforms matches, so the autocorrelation coefficient is large. In this embodiment, the flicker noise intensity calculation device 12 finds Δt when the autocorrelation coefficient is highest, and calculates the reciprocal of the found Δt as the flicker frequency f.

[0087] According to the present embodiment described above, as in the first embodiment, even if a true event is lost in the masking process for removing the flicker noise event, the lost true event is restored by the event restoration process. Therefore, it is possible to maintain the detection accuracy of the true event while removing the flicker noise event.

[0088] Third Embodiment A third embodiment of the present disclosure will be described. Here, differences from the first embodiment will be mainly described. Furthermore, components similar to those in the first embodiment will be assigned the same reference numerals, and redundant description will be omitted.

[0089] In this embodiment, the content of the masking process by the noise event masking unit 131 is different from that in the first embodiment. Hereinafter, the masking process according to this embodiment will be described with reference to Figs.

[0090] FIG. 12 is a flowchart of the masking process according to the third embodiment. FIG. 13 is a diagram illustrating each step of the flowchart shown in FIG. 12 . In the masking process according to this embodiment, the noise event masking unit 131 first performs a grouping process in which frames are grouped according to an arbitrary number M (step S811). In this grouping process, the arbitrary number M is set by the control device 210. The control device 210 sets the arbitrary number M based on, for example, the formula M=f1 / f2 using the frequencies f1 and f2 described in the first embodiment. When M=6, as shown in FIG. 13 , frames are grouped into groups of six, such as group A, group B, and group C.

[0091] Next, the noise event masking unit 131 performs a past frame extraction process to extract six past frames Fa1 to Fa6 that belong to group A, which is one group before group B to which the correction target frame Fb4, which is the target of flicker noise removal, belongs (step S812).

[0092] Next, the noise event masking unit 131 identifies, for each of the past frames Fa1 to Fa6, either the rising event or the falling event, whichever has the greater number of events, and performs pixel position extraction processing to extract the position of the pixel 111 at which the identified event was detected (step S813). Because the pixels 111 are arranged in a two-dimensional matrix, the noise event masking unit 131 can extract the planar position of the pixel 111 based on the row position and column position. The event with the greater number of events identified in step S43 is highly likely to be flicker noise.

[0093] Next, the noise event mask unit 131 generates a mask frame Fm by performing an OR operation on the positions of the pixels 111 extracted in step S813 (step S814). In the mask frame Fm shown in Fig. 13, the positions of the pixels 111 extracted by the noise event mask unit 131 are set to "0", which corresponds to black.

[0094] Next, the noise event masking unit 131 performs a masking process to mask the correction target frame Fb4 using the mask frame Fm generated in step S814 (step S815). In this masking process, the noise event masking unit 131 first separates the correction target frame Fb4 into a first polarity frame Fb41 consisting of only rising events and a second polarity frame Fb42 consisting of only falling events.

[0095] Next, the noise event masking unit 131 masks the polarity frame with the larger number of events from the first polarity frame Fb41 and the second polarity frame Fb42 using the mask frame Fm. In FIG. 13, since the number of falling events is larger than the number of rising events, the second polarity frame Fb42 is masked with the mask frame Fm to generate the frame Fb43. Finally, the noise event masking unit 131 adds the first polarity frame Fb41, which has the smaller number of events, to the frame Fb43. This generates the corrected frame Fb44.

[0096] According to the present embodiment described above, as in the first embodiment, even if a true event is lost in the masking process for removing the flicker noise event, the lost true event is restored by the event restoration process. Therefore, it is possible to maintain the detection accuracy of the true event while removing the flicker noise event.

[0097] Furthermore, the masking process according to this embodiment can remove flicker noise events even if the position of the pixel where the flicker noise is detected moves significantly over time.Furthermore, it can remove flicker noise events even if the flicker noise does not occur periodically and stably.

[0098] In the present disclosure, the content of the masking process by the noise event masking unit 131 is not limited to the first and second embodiments described above. For example, the noise event masking unit 131 may define the pixel 111 in which both a rising event and a falling event are detected as a flicker occurrence area, and perform processing using a Gaussian filter to ensure robustness in the spatial direction to generate mask information indicating the location of the flicker noise occurrence. In this masking process, if the location of the flicker noise crosses the pixel where the true event occurred in a short time, the true event may be lost. However, the event reconstruction unit 132 can restore the lost true event by performing event reconstruction processing, thereby maintaining the detection accuracy of the true event while removing the flicker noise event.

[0099] Below, several modifications of the first to third embodiments will be described. In the description of each modification, the same components as those in the above-described embodiments will be assigned the same reference numerals, and duplicated descriptions will be omitted.

[0100] (Modification 1) Fig. 14 is a block diagram showing an example of the configuration of a photodetector according to Modification 1. In a photodetector 10a according to this modification, the driving mode of the event detection device 11 differs from that of the first to third embodiments. In each embodiment, the photodetector 10 detects events in full driving. In a photodetector 10 in full driving, one event signal detection circuit is provided for each pixel 111. Therefore, the event signal detection circuit 112 outputs the event detection result of each pixel 111 individually as an event signal.

[0101] On the other hand, the photodetector 10a according to this modification detects events using binning driving. In the binning-driven photodetector 10a, one event signal detection circuit 112 is provided for multiple pixels 111 that are arranged close to each other in the pixel array. Therefore, the event signal detection circuit 112 processes the event detection results of the multiple pixels 111 collectively.

[0102] As in each embodiment, the intensity of an event signal indicating a flicker noise event detected by binning driving is calculated by a flicker noise intensity calculation device 12. Furthermore, in accordance with the calculated intensity of the flicker noise event, a noise event removal device 13 performs the masking process and event restoration process described in each embodiment.

[0103] Therefore, in this modification as well, it is possible to maintain the accuracy of detecting true events while removing flicker noise events.

[0104] (Modification 2) Modification 2 of the present disclosure will be described below. In Modification 2, the pixel configuration of the event detection device 11 differs from that of the first to third embodiments.

[0105] FIG. 15 is a plan view illustrating a pixel configuration according to Modification 2. In this modification, the pixel array unit 110 includes not only an event detection pixel 111E that detects a change in the amount of received light as an event, but also color pixels 111R to 111B that output gradation signals as pixel signals. Each color pixel is provided with a color mask (not shown) that transmits the color light to be detected. The red pixel 111R outputs a gradation signal indicating that red light has been received. The green pixel 111G outputs a gradation signal indicating that green light has been received. The blue pixel 111B outputs a gradation signal indicating that blue light has been received. Each gradation signal is input to a downstream application device 211.

[0106] In this modification, the event signal detection circuit 112 outputs an event signal based on the amount of light received by the event detection pixel 111E. As in each embodiment, the intensity of the flicker noise event included in the event signal is calculated by the flicker noise intensity calculation device 12. Furthermore, in accordance with the calculated intensity of the flicker noise event, the noise event removal device 13 performs the masking process and event restoration process described in each embodiment.

[0107] Therefore, in this modification as well, it is possible to maintain the accuracy of detecting true events while removing flicker noise events.

[0108] 16 is a block diagram showing an example of the configuration of a photodetector according to Modification 3. A photodetector 10c according to this modification differs from the photodetector 10 according to each embodiment in that it includes a control device 210.

[0109] As in the respective embodiments, the control device 210 according to this modification also determines whether or not to perform masking processing in the noise event removal device 13 based on the intensity of the flicker noise calculated by the flicker noise intensity calculation device 12. When the control device 210 determines to perform masking processing, the noise event removal device 13 performs event restoration processing after the masking processing.

[0110] Therefore, in this modification as well, it is possible to maintain the accuracy of detecting true events while removing flicker noise events. Furthermore, in this modification, the control device 210 is provided in the photodetector 10c. Therefore, compared to the embodiments in which the control device 210 is provided in the application processor 20, the signal transmission paths between the control device 210 and each device in the photodetector 10c (the event detector 11, the flicker noise intensity calculator 12, and the noise event remover 13) can be made shorter.

[0111] 17 is a block diagram showing an example of the configuration of a photodetector according to Modification 4. This modification differs from the respective embodiments in that the photodetector 10d includes an event detector 11 and a flicker noise intensity calculator 12, and the application processor 20 includes a noise event remover 13, a controller 210, and a subsequent application device 211.

[0112] In this modification, when the control device 210 determines to perform masking processing based on the flicker noise intensity calculated by the flicker noise intensity calculation device 12, the event signal generated by the event detection device 11 is provided to the application processor 20. In the application processor 20, the masking processing and event restoration processing are performed by the noise event removal device 13, as in the above-described embodiments.

[0113] Therefore, in this modification, it is possible to maintain the accuracy of detecting true events while removing flicker noise events. Furthermore, in this modification, the noise event removal device 13 is implemented in the application processor 20, which makes it possible to simplify the configuration of the photodetector 10e.

[0114] 18 is a block diagram showing an example of the configuration of a photodetector according to Modification 5. This modification differs from the respective embodiments in that the photodetector 10e includes an event detector 11, and the application processor 20 includes a flicker noise intensity calculator 12, a noise event remover 13, a control device 210, and a subsequent application device 211.

[0115] In this modification, the event signal generated by the event detection device 11 is provided to the application processor 20. In the application processor 20, similarly to the above-described embodiments, the flicker noise intensity calculation device 12 performs a flicker noise intensity calculation process, the control device 210 performs a determination process, and the noise event removal device 13 performs a mask process and an event restoration process.

[0116] Therefore, in this modification, it is also possible to maintain the accuracy of detecting true events while removing flicker noise events. Furthermore, in this modification, the flicker noise intensity calculation device 12 and the noise event removal device 13 are implemented in the application processor 20. Therefore, it is possible to further simplify the configuration of the photodetector 10e compared to the above-described modification 4.

[0117] <Application to a Mobile Body> The technology according to the present disclosure (the present technology) can be applied to various products. For example, the technology according to the present disclosure may be realized as a device mounted on any type of mobile body, such as an automobile, an electric vehicle, a hybrid electric vehicle, a motorcycle, a bicycle, personal mobility, an airplane, a drone, a ship, or a robot.

[0118] FIG. 19 is a block diagram showing a schematic configuration example of a vehicle control system, which is an example of a mobile object control system to which the technology according to the present disclosure can be applied.

[0119] The vehicle control system 12000 includes a plurality of electronic control units connected via a communication network 12001. In the example shown in Fig. 19, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an outside-vehicle information detection unit 12030, an inside-vehicle information detection unit 12040, and an integrated control unit 12050. Also shown as functional components of the integrated control unit 12050 are a microcomputer 12051, an audio / video output unit 12052, and an in-vehicle network I / F (interface) 12053.

[0120] The drivetrain control unit 12010 controls the operation of devices related to the drivetrain of the vehicle in accordance with various programs. For example, the drivetrain control unit 12010 functions as a control device for a drive force generating device for generating a drive force of the vehicle, such as an internal combustion engine or a drive motor, a drive force transmission mechanism for transmitting the drive force to the wheels, a steering mechanism for adjusting the steering angle of the vehicle, and a braking device for generating a braking force of the vehicle.

[0121] The body system control unit 12020 controls the operation of various devices equipped in the vehicle body according to various programs. For example, the body system control unit 12020 functions as a control device for a keyless entry system, a smart key system, a power window device, or various lamps such as headlamps, backup lamps, brake lamps, turn signals, and fog lamps. In this case, radio waves transmitted from a portable device that serves as a key or signals from various switches can be input to the body system control unit 12020. The body system control unit 12020 receives these radio waves or signals and controls the vehicle's door lock device, power window device, lamps, etc.

[0122] The outside-vehicle information detection unit 12030 detects information outside the vehicle equipped with the vehicle control system 12000. For example, an imaging unit 12031 is connected to the outside-vehicle information detection unit 12030. The outside-vehicle information detection unit 12030 causes the imaging unit 12031 to capture images outside the vehicle and receives the captured images. The outside-vehicle information detection unit 12030 may perform object detection processing or distance detection processing for people, cars, obstacles, signs, characters on the road surface, etc. based on the received images.

[0123] The imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. The imaging unit 12031 can output the electrical signal as an image or as distance measurement information. The light received by the imaging unit 12031 may be visible light or invisible light such as infrared light.

[0124] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041 that detects the state of the driver is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that captures an image of the driver, and the in-vehicle information detection unit 12040 may calculate the degree of fatigue or concentration of the driver based on the detection information input from the driver state detection unit 12041, or may determine whether the driver is dozing off.

[0125] The microcomputer 12051 can calculate control target values ​​for the driving force generating device, steering mechanism, or braking device based on the information inside and outside the vehicle acquired by the outside-vehicle information detection unit 12030 or the inside-vehicle information detection unit 12040, and output control commands to the drive system control unit 12010. For example, the microcomputer 12051 can perform cooperative control aimed at realizing the functions of an ADAS (Advanced Driver Assistance System), including vehicle collision avoidance or impact mitigation, following driving based on the inter-vehicle distance, vehicle speed maintenance driving, vehicle collision warning, vehicle lane departure warning, etc.

[0126] In addition, the microcomputer 12051 can perform cooperative control for the purpose of autonomous driving, which allows the vehicle to travel autonomously without relying on driver operation, by controlling the driving force generating device, steering mechanism, braking device, etc. based on information about the surroundings of the vehicle obtained by the outside vehicle information detection unit 12030 or the inside vehicle information detection unit 12040.

[0127] Furthermore, the microcomputer 12051 can output a control command to the body system control unit 12020 based on the information outside the vehicle acquired by the outside information detection unit 12030. For example, the microcomputer 12051 can control the headlamps according to the position of a preceding vehicle or an oncoming vehicle detected by the outside information detection unit 12030, and perform cooperative control aimed at preventing glare, such as switching from high beams to low beams.

[0128] The audio / video output unit 12052 transmits at least one of audio and video output signals to an output device capable of visually or audibly notifying the passengers of the vehicle or the outside of the vehicle of information. In the example of Fig. 19, the output devices are exemplified by an audio speaker 12061, a display unit 12062, and an instrument panel 12063. The display unit 12062 may include, for example, at least one of an on-board display and a head-up display.

[0129] FIG. 20 is a diagram showing an example of the installation position of the imaging unit 12031.

[0130] In FIG. 20 , a vehicle 12100 has imaging units 12101, 12102, 12103, 12104, and 12105 as an imaging unit 12031.

[0131] The imaging units 12101, 12102, 12103, 12104, and 12105 are provided, for example, at positions such as the front nose, side mirrors, rear bumper, back door, and the top of the windshield inside the vehicle cabin of the vehicle 12100. The imaging unit 12101 provided on the front nose and the imaging unit 12105 provided on the top of the windshield inside the vehicle cabin mainly acquire images of the front of the vehicle 12100. The imaging units 12102 and 12103 provided on the side mirrors mainly acquire images of the sides of the vehicle 12100. The imaging unit 12104 provided on the rear bumper or back door mainly acquires images of the rear of the vehicle 12100. The forward images acquired by the imaging units 12101 and 12105 are mainly used to detect preceding vehicles, pedestrians, obstacles, traffic lights, traffic signs, lanes, etc.

[0132] 20 shows an example of the imaging ranges of the imaging units 12101 to 12104. Imaging range 12111 indicates the imaging range of the imaging unit 12101 provided on the front nose, imaging ranges 12112 and 12113 indicate the imaging ranges of the imaging units 12102 and 12103 provided on the side mirrors, respectively, and imaging range 12114 indicates the imaging range of the imaging unit 12104 provided on the rear bumper or back door. For example, by overlaying the image data captured by the imaging units 12101 to 12104, an overhead image of the vehicle 12100 viewed from above can be obtained.

[0133] At least one of the image capturing units 12101 to 12104 may have a function of acquiring distance information. For example, at least one of the image capturing units 12101 to 12104 may be a stereo camera made up of multiple image capturing elements, or may be an image capturing element having pixels for phase difference detection.

[0134] For example, based on the distance information obtained from the imaging units 12101 to 12104, the microcomputer 12051 can calculate the distance to each three-dimensional object within the imaging ranges 12111 to 12114 and the change in this distance over time (relative speed with respect to the vehicle 12100), thereby extracting as a preceding vehicle, in particular, the three-dimensional object that is the closest three-dimensional object on the path of the vehicle 12100 and traveling in approximately the same direction as the vehicle 12100 at a predetermined speed (e.g., 0 km / h or higher). Furthermore, the microcomputer 12051 can set a vehicle-to-vehicle distance to be maintained in advance in front of the preceding vehicle, and perform automatic braking control (including follow-up stop control), automatic acceleration control (including follow-up start control), etc. In this way, cooperative control can be performed for the purpose of autonomous driving, which runs autonomously without relying on driver operation.

[0135] For example, the microcomputer 12051 classifies and extracts three-dimensional object data regarding three-dimensional objects into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on distance information obtained from the imaging units 12101 to 12104, and can use the data for automatic obstacle avoidance. For example, the microcomputer 12051 distinguishes obstacles around the vehicle 12100 into obstacles that are visible to the driver of the vehicle 12100 and obstacles that are difficult to see. The microcomputer 12051 determines the collision risk, which indicates the risk of collision with each obstacle, and when the collision risk is equal to or greater than a set value and a collision is possible, the microcomputer 12051 can provide driving assistance for collision avoidance by outputting an alarm to the driver via the audio speaker 12061 or the display unit 12062, or by performing forced deceleration or avoidance steering via the drive system control unit 12010.

[0136] At least one of the image capturing units 12101 to 12104 may be an infrared camera that detects infrared rays. For example, the microcomputer 12051 can recognize a pedestrian by determining whether a pedestrian is present in the images captured by the image capturing units 12101 to 12104. Such pedestrian recognition is performed, for example, by extracting feature points from the images captured by the image capturing units 12101 to 12104 as infrared cameras and performing pattern matching on a series of feature points that indicate the outline of an object to determine whether the object is a pedestrian. When the microcomputer 12051 determines that a pedestrian is present in the images captured by the image capturing units 12101 to 12104 and recognizes the pedestrian, the audio / image output unit 12052 controls the display unit 12062 to superimpose a rectangular outline on the recognized pedestrian for emphasis. The audio / image output unit 12052 may also control the display unit 12062 to display an icon or the like indicating the pedestrian at a desired position.

[0137] The above describes an example of a vehicle control system to which the technology according to the present disclosure can be applied. The technology according to the present disclosure can be applied to, for example, the image capture unit 12031 of the above-described configuration. Specifically, the image capture unit 12031 can be equipped with any of the photodetection devices according to the above-described embodiments and modifications. By applying the technology according to the present disclosure to the image capture unit 12031, it becomes possible to remove flicker noise events while maintaining the accuracy of detecting true events. As a result, the performance of the vehicle 12100 can be improved.

[0138] Note that the above-described embodiments are examples for realizing the present technology, and the matters in the embodiments correspond to the matters specifying the invention in the claims. Similarly, the matters specifying the invention in the claims correspond to the matters in the embodiments of the present technology having the same names. However, the present technology is not limited to the embodiments, and can be realized by applying various modifications to the embodiments within the scope of the gist of the present technology.

[0139] The present technology can be configured as follows:

[0140] (1) A noise event removal device comprising: a noise event masking unit that performs masking to remove flicker noise from an event signal indicating the result of detecting, as an event, a change in the amount of light received by a pixel when an image of a subject illuminated by a light source is captured in a predetermined frame; and an event restoration unit that performs event restoration to restore a true event that has been lost by the masking.

[0141] (2) The noise event removal device according to (1), wherein in the masking process, the noise event masking unit subtracts an event signal of a correction target frame from which the flicker noise is to be removed by an event signal of a previous frame by one cycle of the flicker noise.

[0142] (3) The noise event removal device according to (1), wherein in the masking process, the noise event masking unit groups frames into groups of an arbitrary number, extracts past frames that belong to a group immediately before the group to which the correction target frame, from which the flicker noise is to be removed, belongs, extracts, for the past frames, the positions of the pixels at which an event with a large number of events has been detected out of two types of events determined according to changes in the amount of received light, generates a mask frame based on the positions of the pixels, and masks the correction target frame with the mask frame.

[0143] (4) The noise event removal device according to any one of (1) to (3), wherein in the event restoration process, the event restoration unit generates an extraction frame in which one of two types of events determined according to a change in the amount of received light is extracted from a correction target frame that is the target for removing the flicker noise, and a past extraction frame in which the one event is extracted from a past frame immediately preceding the correction target frame, performs an integration process on the extraction frame and the past extraction frame, and performs an addition process on the masked frame and the integration process on the frame.

[0144] (5) The noise event removal device according to any one of (1) to (4), wherein, when the intensity of the flicker noise exceeds a predetermined reference value, the noise event masking unit performs the masking process and the event restoration unit performs the event restoration process.

[0145] (6) The noise event removal device according to any one of (1) to (5), wherein the noise event removal device is provided in a photodetection device including the pixel.

[0146] (7) The noise event removal device according to any one of (1) to (5), wherein the noise event removal device is provided in an application processor arranged in a stage subsequent to a photodetection device including the pixel.

[0147] (8) A light detection device comprising: an event detection device that outputs an event signal indicating the result of detecting, as an event, a change in the amount of light received by a pixel when an image of a subject illuminated by lighting is captured in a predetermined frame; a flicker noise intensity calculation device that calculates the intensity of flicker noise included in the event signal; and a noise event removal device that performs a masking process to remove the flicker noise of the lighting from the event signal according to the intensity of the flicker noise, and an event restoration process to restore a true event that has been lost by the masking process.

[0148] (9) The photodetector according to (8), wherein the flicker noise intensity calculation device counts a number of events corresponding to a number of pixels in which events are detected from the event signal, and calculates the intensity of the flicker noise based on a result of a Fourier transform of the number of events.

[0149] (10) The light detection device according to (8), wherein the flicker noise intensity calculation device calculates the intensity of the flicker noise based on the difference between the maximum number of events and the minimum number of events at any time for two types of events determined according to changes in the amount of received light.

[0150] (11) The photodetector according to any one of (8) to (10), wherein the event detection device further includes an event signal detection circuit that generates the event signal, and one event signal detection circuit is provided for one pixel.

[0151] (12) The photodetector according to any one of (8) to (10), wherein the event detection device further includes an event signal detection circuit that generates the event signal, and one event signal detection circuit is provided for a plurality of the pixels that are arranged close to each other.

[0152] (13) The light detection device according to any one of (8) to (10), wherein the event detection device includes: an event detection pixel that detects the event; and a color pixel that detects color light.

[0153] (14) The photodetector according to any one of (8) to (13), further comprising a control device that determines whether or not to cause the noise event elimination device to perform the masking process based on the intensity of the flicker noise.

[0154] (15) A noise event removal method that performs a masking process to remove flicker noise from an event signal that indicates the result of detecting, as an event, a change in the amount of light received by a pixel when an object illuminated by the light is imaged in a predetermined frame, and performs an event restoration process to restore a true event that has been lost by the masking process.

[0155] 10, 10a, 10c, 10d, 10e: Light detection device 11: Event detection device 12: Flicker noise intensity calculation device 13: Noise event removal device 20: Application processor 111: Pixel 112: Event signal detection circuit 131: Noise event mask unit 132: Event restoration unit 210: Control device 300: Lighting 310: Subject

Claims

1. A noise event removal device comprising: a noise event mask unit that performs a masking process for removing flicker noise from an event signal indicating a result of detecting a change in the amount of light received by pixels obtained by imaging a subject illuminated by lighting within a predetermined frame; and an event restoration unit that performs an event restoration process for restoring a true event that has disappeared due to the masking process.

2. The noise event removal device according to claim 1, wherein in the masking process, the noise event mask unit subtracts an event signal of a correction target frame that is a removal target of the flicker noise from an event signal of a frame in the past by one period of the flicker noise.

3. The noise event removal device according to claim 1, wherein in the masking process, the noise event mask unit groups frames by an arbitrary number, extracts a past frame belonging to one group before the group to which the correction target frame that is a removal target of the flicker noise belongs, extracts positions of pixels for which an event with a larger number of events is detected among two types of events determined according to the change in the amount of light received for the past frame, generates a mask frame based on the positions of the pixels, and masks the correction target frame with the mask frame.

4. The noise event removal device according to claim 1, wherein in the event restoration process, the event restoration unit generates an extraction frame obtained by extracting one of two types of events determined according to the change in the amount of light received from the correction target frame that is a removal target of the flicker noise, and a past extraction frame obtained by extracting the one event from a past frame immediately before the correction target frame, performs an integration process on the extraction frame and the past extraction frame, and performs an addition process on the masked frame and the integrated frame.

5. The noise event removal device according to claim 1, wherein when the intensity of the flicker noise exceeds a preset reference value, the noise event mask unit performs the masking process and the event restoration unit performs the event restoration process.

6. The noise event removal device according to claim 1, wherein the noise event removal device is provided in an optical detection device including the pixels.

7. The noise event removal device according to claim 1, wherein the noise event removal device is provided in an application processor arranged at a subsequent stage of an optical detection device including the pixel.

8. An optical detection device comprising: an event detection device that outputs an event signal indicating a result of detecting, as an event, a change in a light reception amount of a pixel obtained by imaging a subject illuminated by illumination in a predetermined frame; a flicker noise intensity calculation device that calculates an intensity of flicker noise included in the event signal; a noise event removal device that performs a mask process for removing the flicker noise of the illumination from the event signal according to the intensity of the flicker noise, and an event restoration process for restoring a true event disappeared by the mask process.

9. The optical detection device according to claim 8, wherein the flicker noise intensity calculation device counts the number of events corresponding to the number of pixels in which an event is detected from the event signal, and calculates the intensity of the flicker noise based on a result of performing a Fourier transform on the number of events.

10. The optical detection device according to claim 8, wherein the flicker noise intensity calculation device calculates the intensity of the flicker noise based on a difference between a maximum number of events and a minimum number of events at an arbitrary time of two types of events determined according to the change in the light reception amount.

11. The optical detection device according to claim 8, wherein the event detection device further includes an event signal detection circuit that generates the event signal, and one event signal detection circuit is provided for one of the pixels.

12. The optical detection device according to claim 8, wherein the event detection device further includes an event signal detection circuit that generates the event signal, and one event signal detection circuit is provided for a plurality of the pixels arranged adjacent to each other.

13. The optical detection device according to claim 8, wherein the event detection device includes an event detection pixel that detects the event and a color pixel that detects color light.

14. The optical detection device according to claim 8, further comprising a control device that determines whether or not to cause the noise event removal device to perform the mask process based on the intensity of the flicker noise.

15. A noise event removal method that performs a mask process to remove flicker noise of the illumination from an event signal indicating a result of detecting, as an event, a change in the amount of light received by pixels obtained by imaging a subject illuminated by the illumination within a predetermined frame, and performs an event restoration process to restore true events that disappeared due to the mask process.

Images