Imaging apparatus and detection method for range image reliability

Abstract

To provide a technology for detecting the movement of an object with high accuracy and acquiring highly reliable range data.SOLUTION: A range sensor 10 irradiates a range target with light and obtains a range image by photographing reflected light from the range target. An event-based vision sensor 20 photographs the range target and detects a change in brightness of each pixel. A control unit 30 controls the reset timing and output timing of the event-based vision sensor in synchronization with the exposure period of the range sensor. A detection unit 40 detects pixels of the range image corresponding to the pixels for which the event-based vision sensor has detected a brightness change of a predetermined threshold value or more as an unreliable pixel in the range images.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an imaging apparatus and a distance measurement image reliability detection method. [Background technology]

[0002] There is a ToF (Time of Flight) camera that calculates the distance to an object by shining infrared light on the object and measuring the time it takes for the reflected light to be received, and is used for object recognition and three-dimensional measurement of objects.

[0003] Patent Document 1 discloses a distance measurement technique that corrects distance data indicating the distance to an object or the reliability of the distance data based on the detection result of the movement of the object. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-081290 Summary of the Invention [Problem to be solved by the invention]

[0005] A ToF camera irradiates an object with infrared light for a short period of time multiple times per distance measurement, exposing it to the object and calculating the distance to the object. Therefore, if the object being measured moves within one distance measurement time, it becomes difficult to obtain accurate distance measurement data. Since inaccurate distance data cannot be used as is, it is necessary to ensure the reliability of the distance measurement data by identifying the position of the inaccurate distance measurement data on the screen and performing processing such as correction, replacement, or discarding.

[0006] To detect the movement of an object, (1) A method of using exposure data from image frames before and after the infrared image used for distance measurement with a ToF camera; and (2) Using images captured by a visible light camera simultaneously with a ToF camera There is.

[0007] In order to detect object movement using methods (1) and (2), it is necessary to detect the difference in contrast between the frame with the same timing as the distance measurement data and the frame immediately before. Since two frames of video data, image analysis, and correction value creation processing are required, the output time of the object movement detection is delayed compared to the output time of the distance data. Also, if the difference in contrast between a moving distance measurement target and a non-moving distance measurement target is small, it may not be possible to detect the object movement.

[0008] In addition, in method (2), the exposure time of the visible light camera is not synchronized with the exposure time of the ToF camera, resulting in a discrepancy in the amount of object movement.

[0009] Furthermore, if video is not used to detect object motion, (3) Using distance measurement data from previous and next frames There is.

[0010] Method (3) also requires two frames of data, just like methods (1) and (2). In addition, distance measurement data for the edge of an object tends to be inaccurate in principle because of reduced infrared light reflection. Distance measurement data for subjects at a distance also has a similar tendency, and it is difficult to detect the movement of an object using only distance measurement data due to the inaccuracy of the data.

[0011] The present invention has been made in view of the above circumstances, and has as its object to provide a technique for detecting the movement of an object with high accuracy and acquiring highly reliable distance measurement data. [Means for solving the problem]

[0012] In order to solve the above problems, an imaging device of one embodiment of the present invention includes a distance measurement sensor that irradiates a distance measurement target with light and captures the reflected light from the distance measurement target to obtain a distance measurement image, an event-based vision sensor that captures the distance measurement target and detects luminance changes for each pixel, a control unit that controls the reset timing and output timing of the event-based vision sensor in synchronization with the exposure period of the distance measurement sensor, and a detection unit that detects pixels of the distance measurement image corresponding to pixels in which a luminance change of more than a predetermined threshold has been detected by the event-based vision sensor as pixels of low reliability in the distance measurement image.

[0013] Another aspect of the present invention is a distance measurement image reliability detection method, which includes the steps of: irradiating a distance measurement target with light by a distance measurement sensor and capturing reflected light from the distance measurement target to obtain a distance measurement image, capturing the distance measurement target with an event-based vision sensor to detect a luminance change for each pixel, controlling reset timing and output timing of the event-based vision sensor in synchronization with an exposure period of the distance measurement sensor, and detecting, as a pixel of the distance measurement image having low reliability, a pixel of the distance measurement image that corresponds to a pixel in which a luminance change equal to or greater than a predetermined threshold is detected by the event-based vision sensor.

[0014] Any combination of the above components, and any transformation of the present invention into a method, device, system, recording medium, computer program, etc., are also effective as aspects of the present invention. Effect of the Invention

[0015] According to the present invention, it is possible to provide a technique for detecting the movement of an object with high accuracy and acquiring highly reliable distance measurement data. [Brief description of the drawings]

[0016] [Figure 1] 1 is a configuration diagram of an imaging device according to an embodiment of the present invention. [Diagram 2] 1 is a diagram illustrating synchronization between an exposure period of a distance measurement sensor and a detection period of an event-based vision sensor. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0017] 1 is a configuration diagram of an image capturing device 100 according to the present embodiment. The image capturing device 100 includes a distance measuring sensor 10, an event-based vision sensor (EVS) 20, a control unit 30, a detection unit 40, and a distance calculation unit 50.

[0018] The distance measurement sensor 10 detects the distance to the target by irradiating the target with light and measuring the time it takes for the reflected light to return from the target. Here, as an example, a ToF sensor is used that irradiates the target with infrared light and captures the reflected light from the target to obtain a distance measurement image.

[0019] The EVS20 captures an image of a target and detects changes in luminance for each pixel. The EVS20 detects changes in luminance for each pixel of the captured image at high speed and with low latency, and outputs the coordinates of pixels with luminance changes equal to or greater than a predetermined threshold together with time information.

[0020] It is desirable for the EVS20 to detect luminance changes caused by the same infrared light as the distance measurement sensor 10, but the infrared light emitted by the distance measurement sensor 10 flashes repeatedly during exposure, causing the EVS20 to react to the flashing infrared light. Therefore, the EVS20 uses the visible light region to capture the target and detect the luminance change for each pixel. It is preferable to insert an optical filter that transmits visible light but not infrared light in front of the light receiving surface of the EVS20 so that the EVS20 does not react to the wavelength of infrared light used for distance measurement.

[0021] The control unit 30 controls the reset timing and output timing of the EVS 20 in synchronization with the exposure time of the distance measurement sensor 10. Specifically, the EVS 20 is reset in accordance with the measurement start timing of the distance measurement sensor 10, and the output of the EVS 20 is taken out in accordance with the measurement end timing of the distance measurement sensor 10. When the distance measurement sensor 10 performs distance measurement N times (N is a natural number of 2 or more) per frame and accumulates the detection values, the control unit 30 resets the EVS 20 at the measurement start timing of the first distance measurement out of the N distance measurements, and takes out the output of the EVS 20 at the measurement end timing of the last distance measurement out of the N distance measurements.

[0022] The detection unit 40 detects pixels of the distance measurement image corresponding to pixels in which a luminance change equal to or greater than a predetermined threshold has been detected by the EVS 20 as pixels of low reliability in the distance measurement image.

[0023] It is preferable that the optical axes of the distance measurement sensor 10 and the EVS 20 are coaxial in identifying pixels of the distance measurement image corresponding to pixels in which a luminance change has been detected by the EVS 20. If the optical axes of the distance measurement sensor 10 and the EVS 20 are misaligned, image processing may be used to correct the misalignment so that the pixels of the subject in the image captured by the EVS 20 and the distance measurement image by the distance measurement sensor 10 match.

[0024] The distance calculation unit 50 receives the distance measurement image from the distance measurement sensor 10, and calculates the distance to the object based on the distance measurement image in which the pixels with low reliability detected by the detection unit 40 are invalidated.

[0025] FIG. 2 is a diagram for explaining synchronization between the exposure period of the distance measurement sensor 10 and the detection period of the EVS 20. In FIG.

[0026] The EVS 20 outputs the pixel position where the light amount changes in a short period (for example, 1 μs) and the direction of the luminance change (from dark to light or from light to dark). The output sensitivity is adjustable, and the EVS 20 can be used to detect the movement of the target object by capturing a luminance change of each pixel that is equal to or greater than a predetermined threshold during the distance measurement period of the distance measurement sensor 10.

[0027] The EVS20 has a pixel reset function and an external trigger function. The pixel reset function is a function that resets the output change of each pixel of the EVS20. The external trigger function is a function that extracts brightness change data of each pixel of the EVS20 in synchronization with an external signal. Using the pixel reset function and the external trigger function, the EVS20 can extract pixels whose brightness has changed during the exposure period of the distance measurement sensor 10, and obtain data that indicates the moving area of ​​the distance measurement target synchronized with the distance measurement period of the distance measurement sensor 10.

[0028] Fig. 2(a) shows the emission timing of the infrared laser of the distance measurement sensor 10, and Fig. 2(b) shows the exposure timing of the distance measurement sensor 10. By performing distance measurement by the distance measurement sensor 10 N times (N is a natural number of 2 or more) per frame and accumulating the detection values, the measurement accuracy can be improved. Fig. 2(c) shows the emission period of the infrared laser N times per frame, and Fig. 2(d) shows the exposure period of the distance measurement sensor 10 N times per frame.

[0029] 2(e) shows the reset timing of the EVS 20 synchronized with the measurement start timing of the distance measurement sensor 10, and Fig. 2(f) shows the output timing of the EVS 20 synchronized with the measurement end timing of the distance measurement sensor 10. When distance measurement is performed N times per frame by the distance measurement sensor 10, the EVS 20 is reset at the measurement start timing of the first distance measurement of the N distance measurements by the distance measurement sensor 10, and the output of the EVS 20 is taken at the measurement end timing of the last distance measurement of the N distance measurements by the distance measurement sensor 10.

[0030] 2(e), the EVS 20 resets the output change of each pixel of the EVS 20 by inputting a reset signal using a pixel reset function at the exposure start timing of the distance measurement sensor 10. Note that, in terms of implementation, it is preferable to input a reset signal to the EVS 20 a predetermined short time before the exposure start timing of the distance measurement sensor 10.

[0031] As shown in FIG. 2(f), the EVS 20 extracts brightness change data of each pixel of the EVS 20 during the exposure period of the distance measurement sensor 10 by inputting an output instruction signal to the EVS from the outside using an external trigger function at the timing when the exposure of the distance measurement sensor 10 ends.

[0032] 2G shows the detection period of the EVS 20, that is, the period from the reset timing to the output timing of the EVS 20. The detection period of the EVS 20 is synchronized with the exposure period of the distance measurement sensor 10, and is the same period.

[0033] In this way, the EVS 20 resets the output change of each pixel at the exposure start timing of the distance measurement sensor 10 and extracts the luminance change data of each pixel at the exposure end timing of the distance measurement sensor 10, thereby making it possible to extract pixels whose luminance has changed during the exposure period of the distance measurement sensor 10. From the positions of the pixels whose luminance has changed, it is possible to identify the moving area of ​​the distance measurement target synchronized with the exposure period of the distance measurement sensor 10.

[0034] Compared to methods that detect motion based on the difference in contrast between image frames, the brightness change data obtained from EVS20 can be used to detect the amount of actual movement of the target during the ranging period, resulting in higher motion detection accuracy and reduced invalid data in the ranging images, improving reliability.

[0035] Furthermore, when motion detection is performed between image frames, the motion is for the period of one frame, resulting in greater blurring.Movement outside the exposure period of the distance measurement sensor 10 is also picked up, increasing the possibility that the effective area of ​​the distance measurement image will be determined to be invalid.In contrast, the EVS 20 detects motion during the same period as the exposure period of the distance measurement sensor 10, so it can determine and invalidate pixels with movement in the distance measurement image with high accuracy.

[0036] Furthermore, by using the EVS20, it is possible to detect luminance changes with high sensitivity and detect motion, even in dark areas where reflected light is difficult to obtain.

[0037] In addition, since the detection period of the EVS 20 and the ranging period of the distance measurement sensor 10 are the same, pixels to be invalidated can be determined at the same time as the ranging data is acquired, and delays can be minimized.

[0038] As described above, according to the imaging device 100 of the embodiment of the present invention, inaccurate pixels in the distance measurement image caused by the movement of the distance measurement target, etc., are determined based on the detection result of the EVS 20 at the same time as the distance measurement data is acquired, thereby improving the reliability of the distance measurement data output without delaying the output of the distance measurement data. This makes it possible to identify the position of inaccurate distance measurement data that was impossible to identify using only the distance measurement data, improving the reliability of data processing in correction processing and subsequent processing. In addition, since the sampling period of the EVS 20 can be set with a high degree of freedom, motion detection can be performed at the same timing as the distance measurement period of the distance measurement sensor 10, and the detection data of the EVS 20 can be acquired before or at the same time as the output of the distance measurement data, thereby accelerating the start of processing of the distance measurement data.

[0039] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each treatment process, and that such modifications are also within the scope of the present invention. [Explanation of symbols]

[0040] 10 distance measuring sensor, 20 EVS, 30 control unit, 40 detection unit, 50 distance calculation unit, 100 imaging device.

Claims

1. a distance measurement sensor that irradiates a distance measurement target with light and captures a reflected light from the distance measurement target to obtain a distance measurement image; an event-based vision sensor that captures an image of the target and detects a luminance change for each pixel; a control unit that controls a reset timing and an output timing of the event-based vision sensor in synchronization with an exposure period of the distance measuring sensor; and a detection unit that detects pixels of the ranging image corresponding to pixels in which a luminance change equal to or greater than a predetermined threshold has been detected by the event-based vision sensor as pixels of low reliability in the ranging image.

2. The imaging device described in claim 1, characterized in that when distance measurement using the distance measurement sensor is performed N times (N is a natural number greater than or equal to 2) per frame and detection values ​​are accumulated, the control unit resets the event-based vision sensor at the start of the first distance measurement of the N distance measurements, and extracts the output of the event-based vision sensor at the end of the last distance measurement of the N distance measurements.

3. the distance measurement sensor irradiates an object to be measured with infrared light and captures reflected light from the object to obtain the distance measurement image; 3. The imaging device according to claim 1, wherein the event-based vision sensor captures an image of the target object irradiated with visible light and detects a change in luminance for each pixel.

4. 4. The imaging device according to claim 3, further comprising an optical filter that transmits visible light but blocks infrared light, inserted in front of the light receiving surface of the event-based vision sensor.

5. A step of irradiating a distance measurement target with light by a distance measurement sensor and capturing reflected light from the distance measurement target to obtain a distance measurement image; An event-based vision sensor captures an image of the target to be measured and detects a change in luminance for each pixel; controlling a reset timing and an output timing of the event-based vision sensor in synchronization with an exposure period of the distance measuring sensor; a step of detecting pixels of the ranging image corresponding to pixels in which a luminance change equal to or greater than a predetermined threshold is detected by the event-based vision sensor as pixels of low reliability in the ranging image.

Images