Imaging device and distance measurement image reliability detection method

Abstract

A distance measurement sensor (10) irradiates a distance measurement target with light and captures reflected light from the distance measurement target to acquire a distance measurement image. An event-based vision sensor (20) images the distance measurement target and detects brightness changes for each pixel. A control unit (30) controls a reset timing and an output timing of the event-based vision sensor in synchronization with an exposure period of the distance measurement sensor. A detection unit (40) detects, as pixels having low reliability, pixels in the distance measurement image for which a brightness change equal to or greater than a predetermined threshold was detected by the event-based vision sensor.

Description

Imaging device and distance measurement image reliability detection method

[0001] The present invention relates to an imaging device and a distance measurement image reliability detection method.

[0002] There is a ToF (Time of Flight) camera that calculates the distance to an object by irradiating the object with infrared light and measuring the time it takes for the reflected light to be received, and is used for object recognition and three-dimensional measurement of objects.

[0003] Japanese Patent Laid-Open No. 2003-124999 discloses a distance measurement technique that corrects distance data indicating the distance to an object or the reliability of the distance data based on the detection result of the movement of the object.

[0004] Japanese Patent Application Laid-Open No. 2021-081290

[0005] ToF cameras calculate the distance to an object by irradiating it with infrared light for a short period of time multiple times per measurement. Therefore, if the object being measured moves during one measurement, it becomes difficult to obtain accurate distance measurement data. Since inaccurate distance data cannot be used as is, it is necessary to ensure the reliability of the distance measurement data by identifying the location of the inaccurate distance measurement data on the screen and performing processing such as correction, replacement, or discarding.

[0006] There are two methods for detecting the movement of an object: (1) using exposure data from image frames before and after the infrared image used for distance measurement by a ToF camera, and (2) using images from a visible light camera captured simultaneously with the ToF camera.

[0007] To detect object movement using methods (1) and (2), it is necessary to detect the difference in contrast between the frame that is synchronized with the distance measurement data and the immediately preceding frame. Since two frames of video data, image analysis, and correction value creation processing are required, the output time of the object movement detection is delayed relative to the output time of the distance data. Furthermore, if the difference in contrast between a moving distance measurement target and a stationary distance measurement target is small, it may not be possible to detect object movement.

[0008] Furthermore, in the method (2), the exposure time of the visible light camera is not synchronized with the exposure time of the ToF camera, resulting in a discrepancy in the amount of movement of the object.

[0009] Furthermore, when video is not used to detect the movement of an object, there is a method (3) of using distance measurement data from previous and subsequent frames.

[0010] Method (3) also requires two frames of data, just like methods (1) and (2). In addition, distance measurement data for the edge of an object tends to be inaccurate in principle due to reduced infrared light reflection. Distance measurement data for a subject at a long distance also tends to be inaccurate, and it is difficult to detect the movement of an object using only distance measurement data due to the inaccuracy of the data.

[0011] The present embodiment has been made in view of the above circumstances, and its purpose is to provide a technology for detecting the movement of an object with high accuracy and acquiring highly reliable distance measurement data.

[0012] In order to solve the above problem, an imaging device of one aspect of this embodiment includes a ranging sensor that irradiates a ranging object with light and captures the reflected light from the ranging object to obtain a ranging image, an event-based vision sensor that captures the ranging object and detects luminance changes for each pixel, a control unit that controls the reset timing and output timing of the event-based vision sensor in synchronization with the exposure period of the ranging sensor, and a detection unit that detects pixels of the ranging image corresponding to pixels in which a luminance change greater than a predetermined threshold has been detected by the event-based vision sensor as pixels of low reliability in the ranging image.

[0013] Another aspect of this embodiment is a distance measurement image reliability detection method, which includes the steps of: irradiating a distance measurement target with light using a distance measurement sensor and capturing reflected light from the target to acquire a distance measurement image; capturing the target with an event-based vision sensor to detect a luminance change for each pixel; controlling the reset timing and output timing of the event-based vision sensor in synchronization with an exposure period of the distance measurement sensor; and detecting, as low-reliability pixels of the distance measurement image, pixels in which a luminance change equal to or greater than a predetermined threshold is detected by the event-based vision sensor.

[0014] Any combination of the above components, and any transformation of the present invention into a method, device, system, recording medium, computer program, etc., are also valid aspects of the present invention.

[0015] According to this embodiment, it is possible to provide a technique for detecting the movement of an object with high accuracy and acquiring highly reliable distance measurement data.

[0016] It is a block diagram of an imaging device according to the present embodiment, and is a diagram illustrating synchronization of an exposure period of a distance measurement sensor and a detection period of an event-based vision sensor.

[0017] 1 is a configuration diagram of an image capturing device 100 according to the present embodiment. The image capturing device 100 includes a distance measuring sensor 10, an event-based vision sensor (EVS) 20, a control unit 30, a detection unit 40, and a distance calculation unit 50.

[0018] The distance measurement sensor 10 detects the distance to the target by irradiating the target with light and measuring the time it takes for the reflected light to return from the target. Here, as an example, a ToF sensor is used, which irradiates the target with infrared light and captures the reflected light from the target to obtain a distance measurement image.

[0019] The EVS 20 captures an image of a target object and detects a change in luminance for each pixel. The EVS 20 detects the change in luminance for each pixel in the captured image at high speed and with low latency, and outputs the coordinates of pixels that have a change in luminance equal to or greater than a predetermined threshold, along with time information.

[0020] It is desirable for the EVS 20 to detect brightness changes due to the same infrared light as the distance measurement sensor 10, but because the infrared light emitted by the distance measurement sensor 10 repeatedly flashes during exposure, the EVS 20 reacts to the flashing infrared light. Therefore, the EVS 20 uses the visible light range to capture an image of the target to be measured and detect brightness changes for each pixel. It is preferable to insert an optical filter that transmits visible light but not infrared light in front of the light receiving surface of the EVS 20 so that the EVS 20 does not react to the wavelength of the infrared light used for distance measurement.

[0021] The control unit 30 controls the reset timing and output timing of the EVS 20 in synchronization with the exposure time of the distance measurement sensor 10. Specifically, the control unit 30 resets the EVS 20 in accordance with the measurement start timing of the distance measurement sensor 10, and extracts the output of the EVS 20 in accordance with the measurement end timing of the distance measurement sensor 10. When the distance measurement sensor 10 performs distance measurement N times (N is a natural number of 2 or more) per frame and accumulates detection values, the control unit 30 resets the EVS 20 at the measurement start timing of the first distance measurement of the N distance measurements, and extracts the output of the EVS 20 at the measurement end timing of the last distance measurement of the N distance measurements.

[0022] The detection unit 40 detects pixels in the distance measurement image corresponding to pixels in which the EVS 20 has detected a luminance change equal to or greater than a predetermined threshold as pixels of low reliability in the distance measurement image.

[0023] It is preferable that the optical axes of the distance measurement sensor 10 and the EVS 20 are coaxial in order to identify pixels in the distance measurement image corresponding to pixels in which a change in brightness has been detected by the EVS 20. If the optical axes of the distance measurement sensor 10 and the EVS 20 are misaligned, image processing may be used to correct this so that the pixels of the subject in the image captured by the EVS 20 and the distance measurement image by the distance measurement sensor 10 match.

[0024] The distance calculation unit 50 receives the distance measurement image from the distance measurement sensor 10 and calculates the distance to the object to be measured based on the distance measurement image in which the pixels with low reliability detected by the detection unit 40 have been invalidated.

[0025] FIG. 2 is a diagram illustrating synchronization between the exposure period of the distance measurement sensor 10 and the detection period of the EVS 20. In FIG.

[0026] The EVS 20 outputs the pixel position where the light amount changes in a short period (for example, 1 μs) and the direction of the luminance change (from dark to bright or from bright to dark). The output sensitivity is adjustable, and the EVS 20 can detect the movement of the target by capturing a luminance change of each pixel that is equal to or greater than a predetermined threshold during the distance measurement period of the distance measurement sensor 10.

[0027] The EVS 20 has a pixel reset function and an external trigger function. The pixel reset function is a function that resets the output change of each pixel of the EVS 20. The external trigger function is a function that extracts brightness change data of each pixel of the EVS 20 in synchronization with an external signal. Using the pixel reset function and the external trigger function, the EVS 20 can extract pixels whose brightness has changed during the exposure period of the distance measurement sensor 10 and obtain data that indicates the moving area of ​​the target to be measured in synchronization with the distance measurement period of the distance measurement sensor 10.

[0028] Figure 2(a) shows the emission timing of the infrared laser of the distance measuring sensor 10, and Figure 2(b) shows the exposure timing of the distance measuring sensor 10. By performing distance measurement by the distance measuring sensor 10 N times (N is a natural number greater than or equal to 2) per frame and accumulating the detected values, measurement accuracy can be improved. Figure 2(c) shows the emission period of the infrared laser N times per frame, and Figure 2(d) shows the exposure period of the distance measuring sensor 10 N times per frame.

[0029] 2(e) shows the reset timing of the EVS 20 synchronized with the measurement start timing of the distance measurement sensor 10, and Fig. 2(f) shows the output timing of the EVS 20 synchronized with the measurement end timing of the distance measurement sensor 10. When distance measurement is performed N times per frame by the distance measurement sensor 10, the EVS 20 is reset at the measurement start timing of the first distance measurement of the N distance measurements by the distance measurement sensor 10, and the output of the EVS 20 is taken at the measurement end timing of the last distance measurement of the N distance measurements by the distance measurement sensor 10.

[0030] 2( e), the EVS 20 resets the output change of each pixel of the EVS 20 by inputting a reset signal using a pixel reset function at the exposure start timing of the distance measurement sensor 10. In terms of implementation, it is preferable to input the reset signal to the EVS 20 a predetermined short time before the exposure start timing of the distance measurement sensor 10.

[0031] As shown in Figure 2 (f), the EVS 20 extracts brightness change data for each pixel of the EVS 20 during the exposure period of the distance measurement sensor 10 by inputting an output instruction signal from the outside to the EVS using an external trigger function at the timing when the exposure of the distance measurement sensor 10 ends.

[0032] 2G shows the detection period of the EVS 20, that is, the period from the reset timing to the output timing of the EVS 20. The detection period of the EVS 20 is synchronized with the exposure period of the distance measurement sensor 10, and is the same period.

[0033] In this way, the EVS 20 resets the output change of each pixel at the exposure start timing of the distance measurement sensor 10 and extracts the luminance change data of each pixel at the exposure end timing of the distance measurement sensor 10, thereby making it possible to extract pixels whose luminance has changed during the exposure period of the distance measurement sensor 10. From the positions of pixels whose luminance has changed, it is possible to identify the moving area of ​​the object to be measured that is synchronized with the exposure period of the distance measurement sensor 10.

[0034] Compared to methods that perform motion detection based on the difference in contrast between image frames, the brightness change data obtained from EVS20 can be used to detect the amount of actual movement of the object being measured during the ranging period, resulting in higher accuracy in motion detection and reduced invalid data in the ranging image, thereby increasing reliability.

[0035] Furthermore, when motion detection is performed between image frames, the motion is detected within one frame period, resulting in significant blur. Furthermore, because motion outside the exposure period of the ranging sensor 10 is also detected, the valid area of ​​the ranging image is likely to be determined as invalid. In contrast, the EVS 20 detects motion during the same period as the exposure period of the ranging sensor 10, so it can accurately determine and invalidate pixels with movement within the ranging image.

[0036] Furthermore, by using the EVS 20, it is possible to detect changes in brightness with high sensitivity and detect motion even in dark areas where reflected light is difficult to obtain.

[0037] Furthermore, since the detection period of the EVS 20 and the distance measurement period of the distance measurement sensor 10 are the same, pixels to be invalidated can be determined at the same time as distance measurement data is acquired, thereby minimizing delays.

[0038] As described above, the imaging device 100 according to an embodiment of the present invention can improve the reliability of ranging data without delaying the output of ranging data by determining inaccurate pixels in a ranging image caused by, for example, the movement of a ranging target based on the detection results of the EVS 20 at the same time as the ranging data is acquired. This makes it possible to identify the location of inaccurate ranging data, which would be impossible using ranging data alone, and improves the reliability of data processing in correction processing and subsequent processing. Furthermore, since the sampling period of the EVS 20 can be set with a high degree of freedom, motion detection can be performed at the same timing as the ranging period of the ranging sensor 10, and the detection data of the EVS 20 can be acquired before or at the same time as the output of ranging data, thereby accelerating the start of ranging data processing.

[0039] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of the components and treatment processes, and that such modifications are also within the scope of the present invention.

[0040] The present invention can be used in imaging technology and distance measurement image reliability detection technology.

[0041] 10 Range-measuring sensor, 20 EVS, 30 Control unit, 40 Detection unit, 50 Distance calculation unit, 100 Imaging device

Claims

1. An imaging device comprising: a distance measurement sensor that irradiates a distance measurement target with light and captures the reflected light from the distance measurement target to obtain a distance measurement image; an event-based vision sensor that captures the distance measurement target and detects luminance changes for each pixel; a control unit that controls the reset timing and output timing of the event-based vision sensor in synchronization with the exposure period of the distance measurement sensor; and a detection unit that detects pixels of the distance measurement image corresponding to pixels in which a luminance change equal to or greater than a predetermined threshold has been detected by the event-based vision sensor as pixels of low reliability in the distance measurement image.

2. The imaging device of claim 1, characterized in that when the distance measurement sensor performs distance measurement N times (N is a natural number greater than or equal to 2) per frame and accumulates detection values, the control unit resets the event-based vision sensor at the start of the first distance measurement of the N distance measurements, and extracts the output of the event-based vision sensor at the end of the last distance measurement of the N distance measurements.

3. The imaging device described in claim 1 or 2, characterized in that the distance measurement sensor irradiates the distance measurement target with infrared light and captures the reflected light from the distance measurement target to obtain the distance measurement image, and the event-based vision sensor captures the distance measurement target irradiated with visible light and detects brightness changes for each pixel.

4. The imaging device according to claim 3, further comprising an optical filter that transmits visible light but blocks infrared light, inserted in front of the light receiving surface of the event-based vision sensor.

5. A method for detecting reliability of a ranging image, comprising the steps of: irradiating a ranging object with light using a ranging sensor and capturing the reflected light from the ranging object to obtain a ranging image; capturing the ranging object with an event-based vision sensor and detecting brightness changes for each pixel; controlling the reset timing and output timing of the event-based vision sensor in synchronization with the exposure period of the ranging sensor; and detecting pixels of the ranging image corresponding to pixels in which a brightness change equal to or greater than a predetermined threshold has been detected by the event-based vision sensor as pixels of low reliability in the ranging image.

Images