Imaging device and interference detection method

Abstract

In the present invention, an RGB image acquisition unit (30) acquires an RGB image of an object. A ranging image acquisition unit (32) acquires, on the same optical axis, a ranging image of the object. An RGB-image movement detection unit (40) detects, with respect to each pixel, movement in the RGB image by acquiring the difference between the current frame of the RGB image and the previous frame thereof. A ranging-image movement detection unit (42) detects, with respect to each pixel, movement in the ranging image by acquiring the difference between the current frame of the ranging image and the previous frame thereof. An interference determination unit (50) determines, per pixel, the presence or absence of infrared interference at the time the ranging image was acquired, on the basis of the results regarding the presence or absence of movement in the RGB image and the presence or absence of movement in the ranging image.

Description

Imaging device and interference detection method

[0001] The present invention relates to an imaging device and an interference detection method.

[0002] There is a ToF (Time of Flight) camera that calculates the distance to an object by irradiating the object with infrared light and measuring the time it takes for the reflected light to be received, and is used for object recognition and three-dimensional measurement of objects.

[0003] When multiple RGB cameras with ToF sensors are used to simultaneously capture images of an object, the infrared light from the multiple RGB cameras with ToF sensors interferes with each other, making it impossible to obtain accurate distance images. Therefore, when a 3D model of the object is generated from the RGB images and distance images, the 3D model of the object is significantly distorted due to the use of abnormal distance data caused by the interference.

[0004] Patent document 1 discloses a distance measuring device that can reduce the effects of interference by invalidating the distance values ​​of pixels included in a saturation determination light-receiving pixel area of ​​a light-receiving sensor when it is determined that a pixel with saturated light intensity has occurred in the saturation determination light-receiving pixel area.

[0005] Japanese Patent Application Laid-Open No. 2023-4120

[0006] When using multiple RGB cameras with ToF sensors to measure and photograph an object in real time, it is necessary to synchronize the operation of each camera to avoid interference from the infrared light from each camera. For this purpose, a synchronization signal line or a system to manage synchronization is required.

[0007] The present embodiment has been made in view of the above circumstances, and its purpose is to provide a technique that can reduce the influence of infrared light interference during distance measurement photography.

[0008] In order to solve the above problem, an imaging device of one aspect of this embodiment includes an RGB image acquisition unit that acquires an RGB image of an object, a distance measurement image acquisition unit that acquires a distance measurement image of the object on the same optical axis, an RGB image motion detection unit that detects movement in the RGB image for each pixel by taking the difference between the current frame and the previous frame of the RGB image, a distance measurement image motion detection unit that detects movement in the distance measurement image for each pixel by taking the difference between the current frame and the previous frame of the distance measurement image, and an interference determination unit that determines whether or not there is interference from infrared light when the distance measurement image is acquired based on the results of determining whether or not there is movement in the RGB image and the distance measurement image.

[0009] Another aspect of this embodiment is an interference detection method, which includes the steps of detecting motion in an RGB image of an object for each pixel by taking a difference between a current frame and a previous frame of the RGB image, detecting motion in the distance measurement image for each pixel by taking a difference between a current frame and a previous frame of a distance measurement image of the object acquired on the same optical axis, and determining whether or not there is interference of infrared light when the distance measurement image is acquired based on the results of determining whether or not there is motion in the RGB image and the distance measurement image.

[0010] Any combination of the above components, and conversion of the expression of this embodiment into a method, device, system, recording medium, computer program, etc. are also valid aspects of this embodiment.

[0011] According to this embodiment, it is possible to provide a technique that can reduce the influence of infrared light interference during distance measurement photography.

[0012] FIG. 1 is a diagram illustrating the configuration of an imaging device according to the present embodiment. FIG. 2 is a diagram illustrating criteria for determining whether or not there is interference of infrared light on a pixel-by-pixel basis by the interference determination unit of FIG. 1. FIG. 3(a) to FIG. 3(c) are diagrams illustrating how the interference avoidance processing unit of FIG. 1 shifts the drive timing of the drive control unit. FIG. 3 is a flowchart illustrating interference determination processing according to the present embodiment. FIG. 4 is a flowchart illustrating distance data correction processing according to the present embodiment. FIG. 4 is a flowchart illustrating interference avoidance processing according to the present embodiment.

[0013] 1 is a configuration diagram of an image capturing apparatus 100 according to this embodiment. The image capturing apparatus 100 includes a lens 10, a light source unit 12, an RGB sensor 20, a distance measurement sensor 22, an RGB image acquisition unit 30, a distance measurement image acquisition unit 32, an RGB image motion detection unit 40, a distance measurement image motion detection unit 42, an interference determination unit 50, a distance data correction unit 60, an interference avoidance processing unit 70, a drive control unit 80, and a point cloud / mesh generation unit 90.

[0014] In this embodiment, a plurality of imaging devices 100 are used to measure the distance to an object, and a three-dimensional model of the object is generated based on the distance to the object.

[0015] The lens 10 focuses the light reflected from the object illuminated with visible light onto the RGB sensor 20 .

[0016] The light source unit 12 irradiates an object with infrared light. The lens 10 focuses the reflected light from the object onto the distance measurement sensor 22.

[0017] The distance measurement sensor 22 detects the distance to an object by measuring the time it takes for light irradiated onto the object to be reflected off the object and return. Here, as an example, a ToF sensor is used, which irradiates the object with infrared light and captures the light reflected from the object to obtain a distance measurement image.

[0018] Here, the RGB sensor 20 and the distance measurement sensor 22 share a single lens 10 with the same optical axis, and a monocular configuration in which light is split into two using a beam splitter or the like is illustrated, but a compound-eye configuration in which the lens of the RGB sensor 20 and the lens of the distance measurement sensor 22 are arranged in parallel may also be used. In the case of a compound-eye configuration, the two lenses can be made to have substantially the same optical axis by shifting the pixels by the amount of the shift in the optical axes. The term "same optical axis" includes cases in which the same physical lens is shared and cases in which the two lenses are arranged in parallel and the pixels are shifted.

[0019] The RGB image acquisition unit 30 acquires R, G, and B data of an image from the RGB sensor 20 and stores the RGB image in a frame buffer.

[0020] The distance measurement image acquisition unit 32 acquires distance measurement data for each pixel from the distance measurement sensor 22 and stores the distance measurement image in a frame buffer. The pixel values ​​of the distance measurement image are distance data (depth values).

[0021] The RGB image motion detector 40 detects motion in the RGB image for each pixel by taking the difference between the current frame and the previous frame of the RGB image.

[0022] The ranging image motion detection unit 42 detects motion in the ranging image for each pixel by calculating the difference between the current frame and the previous frame of the ranging image. Motion is detected in the ranging image not only when the object moves. Infrared light from multiple image capture devices 100 may interfere with each other, causing large changes between frames of the ranging images of the image capture devices 100, which may be detected as motion.

[0023] The interference determination unit 50 determines, on a pixel-by-pixel basis, whether or not there is interference of infrared light when the distance measurement image is acquired, based on the results of the presence or absence of motion in the RGB image and the presence or absence of motion in the distance measurement image.

[0024] Here, with reference to FIG. 2, the criteria for determining whether or not there is interference of infrared light on a pixel-by-pixel basis by the interference determination unit 50 will be described.

[0025] If there is no movement in the RGB image but there is movement in the distance measurement image, the interference determination unit 50 determines that there is interference of infrared light. This is because if there is no movement in the RGB image but there is movement in the distance measurement image, it is considered that this is not due to movement of the object but rather due to interference of infrared light from the multiple image capture devices 100 with each other.

[0026] If there is movement in both the RGB image and the distance measurement image, the interference determination unit 50 determines that there is infrared light interference and that it is unclear whether there is infrared light interference. This is because, when there is movement in both the RGB image and the distance measurement image, it is not clear whether the object is moving and therefore motion is detected in the distance measurement image, or whether the object is moving and there is infrared light interference and therefore motion is detected in the distance measurement image.

[0027] The interference determination unit 50 determines that there is no interference of infrared light when there is no movement in the RGB image and no movement in the distance measurement image. If there is no movement in the RGB image, the object is not moving, and there is no movement in the distance measurement image, it is considered that there is no interference of infrared light.

[0028] The interference determination unit 50 determines that there is no interference of infrared light if there is movement in the RGB image but no movement in the ranging image. Range-finding images generally have a lower frame rate or lower resolution than RGB images, and so even if movement is detected in the RGB image, there are cases where no movement is detected in the ranging image. If there is no movement in the ranging image, it is considered that there is no interference of infrared light.

[0029] For pixels in the ranging image determined to have infrared light interference, the distance data correction unit 60 replaces the distance data of the pixel in the current frame with the distance data of the corresponding pixel in the previous frame. This corrects abnormal distance data in the current frame caused by infrared light interference when the ranging image was acquired with normal distance data from the previous frame. When detecting motion by sampling several frames, the distance data of the pixel in the current frame may be replaced with the average value of distance data of pixels in several past frames without interference, rather than being limited to the previous frame.

[0030] If infrared light interference occurs temporarily, the problem can be resolved by correction processing by the distance data correction unit 60, but if infrared light interference continues for a certain period of time or more (for example, interference detection continues for several tens of frames), the interference avoidance processing unit 70 executes interference avoidance processing.

[0031] If the period during which it is determined that infrared light interference exists in the ranging image continues for a certain period of time or longer, the interference avoidance processing unit 70 executes interference avoidance processing to shift the drive timing of the drive control unit 80. The interference avoidance processing unit 70 issues a command to the drive control unit 80 to shift the drive timing by a predetermined time. The predetermined time is a maximum frame period, and is automatically adjusted until infrared light interference disappears. For example, if the frame rate is 30 fps (frames per second), the drive timing is shifted by a maximum of 33 milliseconds.

[0032] The manner in which the interference avoidance processing unit 70 shifts the drive timing of the drive control unit 80 will be described with reference to FIGS. 3(a) to 3(c).

[0033] Fig. 3(a) shows the emission period of infrared light by the light source unit 12 and the exposure period of the distance measurement sensor 22 before the change. The interference avoidance processing unit 70 issues a command to the drive control unit 80 to shift the drive timing of the distance measurement sensor 22 and the light source unit 12, thereby shifting the emission period and exposure period as shown in Fig. 3(b). The amount of shift in the emission timing is up to a frame period, and Fig. 3(c) shows the case where the emission timing is shifted by only a frame period. The amount of shift in the emission timing is automatically adjusted until there is no interference of infrared light.

[0034] The drive control unit 80 controls the light source unit 12 to emit infrared light at the drive timing instructed by the interference avoidance processing unit 70, and controls the distance measurement sensor 22 to perform exposure after the infrared light is emitted.

[0035] As another method, the interference avoidance processor 70 changes the drive frequency of the distance measurement sensor 22 when a period during which it is determined that infrared light interference exists in the distance measurement image continues for a certain period of time or longer. Generally, the distance measurement sensor 22 has distance measurement modes such as Near (short distance), Mid (medium distance), and Far (long distance) depending on the depth, each of which has a different drive frequency. Therefore, the interference avoidance processor 70 can change the drive frequency by switching the distance measurement mode of the distance measurement sensor 22. Changing the drive frequency changes the infrared light emission time and the sensor exposure time, thereby shifting the infrared light emission timing and avoiding interference.

[0036] The point cloud / mesh generation unit 90 generates point cloud data of the object using the RGB image acquired by the RGB image acquisition unit 30, the distance measurement image corrected by the distance data correction unit 60, and internal parameters specific to the camera such as the focal length and the image center, and generates mesh (polygon) data of the object from the point cloud data. As a method for generating mesh data from point cloud data, the Open3D Ball-Pivoting algorithm can be used, for example, but the method is not limited to this.

[0037] FIG. 4 is a flowchart illustrating the interference detection process according to this embodiment.

[0038] The RGB image motion detection unit 40 detects motion in the RGB image for each pixel by calculating the difference between the current frame and the previous frame of the RGB image (S10).

[0039] The ranging image motion detection unit 42 detects motion in the ranging image for each pixel by calculating the difference between the current frame and the previous frame of the ranging image (S12).

[0040] If there is movement in the RGB image (Y in S14) and there is movement in the distance measurement image (Y in S16), the interference determination unit 50 determines that it is unclear whether there is interference of infrared light (S20).

[0041] If there is movement in the RGB image (Y in S14) and there is no movement in the distance measurement image (N in S16), the interference determination unit 50 determines that there is no interference of infrared light (S22).

[0042] If there is no movement in the RGB image (N in S14) and there is movement in the distance measurement image (Y in S18), the interference determination unit 50 determines that there is interference of infrared light (S24).

[0043] If there is no movement in the RGB image (N in S14) and no movement in the distance measurement image (N in S18), the interference determination unit 50 determines that there is no interference of infrared light (S26).

[0044] FIG. 5 is a flowchart illustrating the distance data correction process of this embodiment.

[0045] The result of collision determination by the collision determination unit 50 is checked for each pixel of the distance measurement image (S30).

[0046] If interference exists for the pixel in the distance measurement image (A of S30), the distance data corrector 60 replaces the distance data in the current frame of the distance measurement image with the corresponding distance data in the previous frame (S34).

[0047] If there is no interference for the pixel in the distance measurement image (B of S30), the distance data corrector 60 uses the distance data in the current frame of the distance measurement image as is (S36).

[0048] If it is unclear whether or not there is interference with the pixel in the distance measurement image (C in S30), the presence or absence of interference with the adjacent pixels of the pixel in question is checked (S32).

[0049] If there is interference with the adjacent pixel of the pixel in question (Y in S32), the distance data corrector 60 replaces the distance data in the current frame of the distance measurement image with the corresponding distance data in the previous frame (S38).

[0050] If there is no interference with the adjacent pixels of the pixel in question (N in S32), the distance data corrector 60 uses the distance data of the current frame of the distance measurement image as is (S40).

[0051] When examining the presence or absence of interference with neighboring pixels of the pixel in step S32, the presence or absence of interference with pixels adjacent to the pixel in question or with surrounding pixels within a range of up to two pixels is examined. Alternatively, the presence or absence of interference with surrounding pixels of the pixel in question may be examined by limiting the examination to the region of a human body part or object detected in the image, or the presence or absence of interference with surrounding pixels of the pixel in question may be examined by limiting the examination to the region of the same color as the pixel in the RGB image.

[0052] FIG. 6 is a flowchart illustrating the interference avoidance process of this embodiment.

[0053] It is checked whether the period during which the interference determination unit 50 determines that there is interference of infrared light continues for a predetermined time or longer (S50).

[0054] If the period during which it is determined that there is interference of infrared light continues for a predetermined time or longer (Y at S50), the interference avoidance processing unit 70 shifts the drive timing of the drive control unit 80 (S52).

[0055] Returning to step S50, if the interference determination unit 50 continues to detect interference as a result of the interference determination process again (Y in S50), the drive timing of the drive control unit 80 is further shifted (S52). This loop is repeated, and if the interference no longer continues (N in S50), the interference avoidance process is terminated.

[0056] In step S52, instead of shifting the drive timing of the drive control unit 80, the drive frequency of the distance measurement sensor 22 may be changed.

[0057] As described above, according to the imaging device 100 of the embodiment of the present invention, when distance measurement photography of an object is performed using multiple imaging devices 100, each imaging device 100 can detect and avoid infrared light interference on its own, eliminating the need to install synchronization signal cables between the multiple imaging devices 100 and reducing the processing load during distance measurement photography. Furthermore, because distance data of a pixel experiencing infrared light interference in the current frame of a distance measurement image is replaced with distance data of the corresponding pixel in the previous frame, abnormal distance data is corrected to normal distance data, eliminating distortion due to interference in 3D video data of the object and enabling the generation of high-quality 3D video data.

[0058] The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of the components and treatment processes, and that such modifications are also within the scope of the present invention.

[0059] The present invention can be used in imaging and interference detection techniques.

[0060] REFERENCE SIGNS LIST 10 Lens, 12 Light source unit, 20 RGB sensor, 22 Distance measurement sensor, 30 RGB image acquisition unit, 32 Distance measurement image acquisition unit, 40 RGB image motion detection unit, 42 Distance measurement image motion detection unit, 50 Interference determination unit, 60 Distance data correction unit, 70 Interference avoidance processing unit, 80 Drive control unit, 90 Point cloud / mesh generation unit, 100 Imaging device.

Claims

1. An imaging device comprising: an RGB image acquisition unit that acquires an RGB image of an object; a distance measurement image acquisition unit that acquires a distance measurement image of the object on the same optical axis; an RGB image motion detection unit that detects motion in the RGB image for each pixel by taking the difference between a current frame and a previous frame of the RGB image; a distance measurement image motion detection unit that detects motion in the distance measurement image for each pixel by taking the difference between the current frame and the previous frame of the distance measurement image; and an interference determination unit that determines the presence or absence of interference of infrared light when the distance measurement image is acquired based on the result of the presence or absence of motion in the RGB image and the presence or absence of motion in the distance measurement image.

2. The imaging device according to claim 1, characterized in that the interference determination unit determines that there is interference of infrared light when there is no movement in the RGB image but there is movement in the distance measurement image.

3. An imaging device as described in claim 1 or 2, further comprising a distance data correction unit which replaces the distance data of a pixel in the current frame with the distance data of the corresponding pixel in the previous frame for a pixel determined to have infrared light interference in the distance measurement image.

4. An imaging device as described in claim 1 or 2, further comprising an interference avoidance processing unit that shifts the timing of emitting infrared light when a period during which it is determined that there is infrared light interference in the distance measurement image continues for a predetermined time or longer.

5. An interference detection method comprising the steps of: detecting motion in an RGB image for each pixel by taking the difference between a current frame and a previous frame of an RGB image of an object; detecting motion in the distance measurement image for each pixel by taking the difference between a current frame and a previous frame of a distance measurement image of the object acquired on the same optical axis; and determining the presence or absence of interference of infrared light when the distance measurement image was acquired based on the result of the presence or absence of motion in the RGB image and the presence or absence of motion in the distance measurement image.

Images