Imaging device and interference detection method

Abstract

To provide technology that makes it possible to reduce the effect of interference of infrared light on the occasion of range-finding imaging.SOLUTION: An RGB image acquisition unit 30 acquires an RGB image of an object. A range-finding image acquisition unit 32 acquires a range-finding image of the object on the same optical axis. An RGB image movement detection unit 40 detects a movement in the RGB image for each pixel by taking the difference between the current frame and the previous frame of the RGB image. A range-finding image movement detection unit 42 detects a movement in the ranging-finding image for each pixel by taking the difference between the current frame and the previous frame of the range-finding image. An interference determination unit 50 determines, in pixel units, the presence of an interference of infrared light at the time the range-finding image was acquired, on the basis of the results of detection of the presence of movement in the RGB image and the presence of movement in the range-finding image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an imaging device and an interference detection method.

Background Art

[0002] There is a ToF (Time of Flight) camera that calculates the distance to an object by measuring the time from when the object is irradiated with infrared light until the reflected light is received, and it is used for object recognition and three-dimensional measurement of objects.

[0003] When an object is simultaneously photographed using multiple RGB cameras with ToF sensors, the infrared light from the multiple RGB cameras with ToF sensors interferes with each other, and an accurate ranging image cannot be obtained. Therefore, when generating a three-dimensional model of the target part from the RGB image and the ranging image, there is a problem that the three-dimensional model of the object is greatly disturbed by using abnormal distance data due to interference.

[0004] Patent Document 1 discloses a ranging device that can reduce the influence of interference by setting the distance values of the pixels included in the saturation determination light-receiving pixel region to invalid values when it is determined that pixels with light amount saturation have occurred in the saturation determination light-receiving pixel region of the light-receiving sensor.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] When performing real-time ranging photography of an object using multiple RGB cameras with ToF sensors, it is necessary to synchronize the driving of each camera in order to avoid interference of infrared light from each camera. For this purpose, a synchronization signal line needs to be provided, or a system for managing synchronization is required.

[0007] The present invention has been made in view of such circumstances, and an object thereof is to provide a technique capable of reducing the influence of infrared light interference during distance measurement imaging.

Means for Solving the Problems

[0008] In order to solve the above problems, an imaging device according to an aspect of the present invention includes an RGB image acquisition unit that acquires an RGB image of an object, a distance measurement image acquisition unit that acquires a distance measurement image of the object on the same optical axis, an RGB image motion detection unit that detects motion in the RGB image for each pixel by taking a difference between the current frame and the previous frame of the RGB image, a distance measurement image motion detection unit that detects motion in the distance measurement image for each pixel by taking a difference between the current frame and the previous frame of the distance measurement image, and an interference determination unit that determines the presence or absence of infrared light interference at the time of acquisition of the distance measurement image in pixel units based on the results of the presence or absence of motion in the RGB image and the presence or absence of motion in the distance measurement image.

[0009] Another aspect of the present invention is an interference detection method. This method includes a step of detecting motion in the RGB image for each pixel by taking a difference between the current frame and the previous frame of the RGB image of the object, a step of detecting motion in the distance measurement image for each pixel by taking a difference between the current frame and the previous frame of the distance measurement image of the object acquired on the same optical axis, and a step of determining the presence or absence of infrared light interference at the time of acquisition of the distance measurement image in pixel units based on the results of the presence or absence of motion in the RGB image and the presence or absence of motion in the distance measurement image.

[0010] In addition, any combination of the above components, and those obtained by converting the expression of the present invention between a method, an apparatus, a system, a recording medium, a computer program, etc. are also effective as aspects of the present invention.

Effects of the Invention

[0011] According to the present invention, it is possible to provide a technique capable of reducing the influence of infrared light interference during distance measurement imaging.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Embodiments for Carrying Out the Invention

[0013] FIG. 1 is a configuration diagram of an imaging device 100 according to the present embodiment. The imaging device 100 includes a lens 10, a light source unit 12, an RGB sensor 20, a distance measurement sensor 22, an RGB image acquisition unit 30, a distance measurement image acquisition unit 32, an RGB image motion detection unit 40, a distance measurement image motion detection unit 42, an interference determination unit 50, a distance data correction unit 60, an interference avoidance processing unit 70, a driving control unit 80, and a point cloud / mesh generation unit 90.

[0014] In the present embodiment, distance measurement imaging of an object is performed by a plurality of imaging devices 100, and a three-dimensional model of the object is generated based on the distance to the object.

[0015] The lens 10 condenses the reflected light from the object irradiated with visible light onto the RGB sensor 20.

[0016]

[0017] The distance measurement sensor 22 detects the distance to the object by measuring the time until the light irradiated on the object is reflected back from the object. Here, as an example, a ToF sensor that irradiates infrared light on the object and captures the reflected light from the object to obtain a distance measurement image is used.

[0018] Here, the RGB sensor 20 and the distance measurement sensor 22 share a single lens 10 with the same optical axis, and a monocular configuration that splits the light into two using a beam splitter or the like is illustrated, but a compound eye configuration in which the lens of the RGB sensor 20 and the lens of the distance measurement sensor 22 are arranged in parallel may also be used. In the case of the compound eye configuration, it is possible to substantially make the optical axes the same by shifting the pixels by the amount of deviation of the optical axes of the two lenses. The "same optical axis" means the meaning including the case of sharing physically the same lens and the case of arranging two lenses in parallel and shifting the pixels.

[0019] The RGB image acquisition unit 30 acquires the R, G, and B data of the image from the RGB sensor 20 and stores the RGB image in the frame buffer.

[0020] The distance measurement image acquisition unit 32 acquires the distance measurement data for each pixel from the distance measurement sensor 22 and stores the distance measurement image in the frame buffer. The pixel value of the distance measurement image is distance data (depth value).

[0021] The RGB image motion detection unit 40 detects the motion in the RGB image for each pixel by taking the difference between the current frame and the previous frame of the RGB image.

[0022] The distance measurement image motion detection unit 42 detects the motion in the distance measurement image for each pixel by taking the difference between the current frame and the previous frame of the distance measurement image. The motion is detected in the distance measurement image not only when the object moves. As a result of the infrared light from a plurality of imaging devices 100 interfering with each other, a large change may occur between the frames of the distance measurement image of the imaging device 100, and that may be detected as motion.

[0023] The interference determination unit 50 determines the presence or absence of infrared light interference at the time of acquisition of the distance measurement image in terms of pixels based on the results of the presence or absence of movement in the RGB image and the presence or absence of movement in the distance measurement image.

[0024] Here, with reference to FIG. 2, the determination criteria for the presence or absence of infrared light interference in terms of pixels by the interference determination unit 50 will be described.

[0025] When there is no movement in the RGB image and there is movement in the distance measurement image, the interference determination unit 50 determines that there is infrared light interference. This is because when there is no movement in the RGB image but there is movement in the distance measurement image, it is considered that the object is not moving, but rather the infrared light from the plurality of imaging devices 100 has interfered with each other.

[0026] When there is movement in the RGB image and there is movement in the distance measurement image, the interference determination unit 50 determines that it is unclear whether there is infrared light interference. This is because when there is movement in both the RGB image and the distance measurement image, it is not clear whether the movement detected in the distance measurement image is due to the object moving, or whether the object is moving and there is also infrared light interference, resulting in the movement detected in the distance measurement image.

[0027] When there is no movement in the RGB image and there is no movement in the distance measurement image, the interference determination unit 50 determines that there is no infrared light interference. If there is no movement in the RGB image and the object is not moving, and there is also no movement in the distance measurement image, it is considered that the infrared light is not interfering.

[0028] When there is movement in the RGB image and there is no movement in the distance measurement image, the interference determination unit 50 determines that there is no infrared light interference. Generally, the frame rate or resolution of the distance measurement image is lower than that of the RGB image. Even if movement is detected in the RGB image, movement may not be detected in the distance measurement image. If there is no movement in the distance measurement image, it is considered that the infrared light is not interfering.

[0029] When it is determined that there is infrared light interference in the distance measurement image, the distance data correction unit 60 replaces the distance data of the pixels in the current frame with the distance data of the corresponding pixels in the previous frame for the pixels so determined. As a result, abnormal distance data in the current frame due to infrared light interference at the time of acquisition of the distance measurement image is corrected with the normal distance data of the previous frame. When performing motion detection by sampling several frames, instead of limiting it to the previous frame, the distance data of the pixels in several past frames without interference may be used to replace the distance data of the pixels in the current frame by taking the average value.

[0030] When infrared light interference occurs temporarily, it is completed by dealing with it through the correction process by the distance data correction unit 60. However, when the infrared light interference continues for a certain period of time or more (for example, the interference determination continues for several tens of frames), the interference avoidance processing unit 70 executes interference avoidance processing.

[0031] When the period in which it is determined that there is infrared light interference in the distance measurement image continues for a certain period of time or more, the interference avoidance processing unit 70 executes interference avoidance processing to shift the driving timing of the drive control unit 80. The interference avoidance processing unit 70 gives an instruction to the drive control unit 80 to shift the driving timing by a predetermined time. The predetermined time is at most the frame period and is automatically adjusted until the infrared light interference disappears. For example, when the frame rate is 30 fps (frames per second), the driving timing is shifted by up to 33 milliseconds.

[0032] With reference to FIGS. 3(a) to 3(c), the state in which the interference avoidance processing unit 70 shifts the driving timing of the drive control unit 80 will be described.

[0033] FIG. 3(a) shows the infrared light emission period by the light source unit 12 before the change and the exposure period of the distance measurement sensor 22. By giving an instruction to the drive control unit 80 for the interference avoidance processing unit 70 to shift the driving timing of the distance measurement sensor 22 and the light source unit 12, as shown in FIG. 3(b), the emission period and the exposure period shift. The shift amount of the emission timing is at most the frame period, and FIG. 3(c) shows the case where the emission timing is shifted by one frame period. The shift amount of the emission timing is automatically adjusted until the infrared light interference disappears.

[0034] The drive control unit 80 controls the light source unit 12 to emit infrared light at the drive timing instructed by the interference avoidance processing unit 70, and controls the distance measurement sensor 22 to perform exposure after the emission of the infrared light.

[0035] As another method, when the period in which interference of infrared light is determined to be present in the distance measurement image continues for a certain period or longer, the interference avoidance processing unit 70 changes the drive frequency of the distance measurement sensor 22. Generally, the distance measurement sensor 22 is provided with modes such as Near (short distance), Mid (medium distance), and Far (long distance) as distance measurement modes, each having a different drive frequency. Therefore, the interference avoidance processing unit 70 can change the drive frequency by switching the distance measurement mode of the distance measurement sensor 22. When the drive frequency is changed, the emission time of the infrared light and the exposure time of the sensor change, so that the emission timing of the infrared light shifts and interference can be avoided.

[0036] The point cloud and mesh generation unit 90 generates point cloud data of the object using the RGB image acquired by the RGB image acquisition unit 30, the distance measurement image corrected by the distance data correction unit 60, and the internal parameters specific to the camera such as the focal length and the image center, and generates mesh (polygon) data of the object from the point cloud data. As an example, the method of generating mesh data from point cloud data can use the Ball-Pivoting algorithm of Open3D, but the method is not limited to this.

[0037] FIG. 4 is a flowchart for explaining the interference determination process of the present embodiment.

[0038] The RGB image motion detection unit 40 detects the motion in the RGB image for each pixel by taking the difference between the current frame and the previous frame of the RGB image (S10).

[0039] The distance measurement image motion detection unit 42 detects the motion in the distance measurement image for each pixel by taking the difference between the current frame and the previous frame of the distance measurement image (S12).

[0040] When there is movement in the RGB image (Y in S14) and there is movement in the distance measurement image (Y in S16), the interference determination unit 50 determines that it is unknown whether there is infrared light interference (S20).

[0041] When there is movement in the RGB image (Y in S14) and there is no movement in the distance measurement image (N in S16), the interference determination unit 50 determines that there is no infrared light interference (S22).

[0042] When there is no movement in the RGB image (N in S14) and there is movement in the distance measurement image (Y in S18), the interference determination unit 50 determines that there is infrared light interference (S24).

[0043] When there is no movement in the RGB image (N in S14) and there is no movement in the distance measurement image (N in S18), the interference determination unit 50 determines that there is no infrared light interference (S26).

[0044] FIG. 5 is a flowchart for explaining the distance data correction process of the present embodiment.

[0045] For each pixel of the distance measurement image, the result of the interference determination by the interference determination unit 50 is examined (S30).

[0046] When there is interference for the pixel of the distance measurement image (A in S30), the distance data correction unit 60 replaces the distance data of the current frame of the distance measurement image with the corresponding distance data of the previous frame (S34).

[0047] When there is no interference for the pixel of the distance measurement image (B in S30), the distance data correction unit 60 uses the distance data of the current frame of the distance measurement image as it is (S36).

[0048] When it is unknown whether there is interference for the pixel of the distance measurement image (C in S30), the presence or absence of interference of the neighboring pixels of the pixel is examined (S32).

[0049] When there is interference with adjacent pixels of the pixel (Y in S32), the distance data correction unit 60 replaces the distance data of the current frame of the distance measurement image with the corresponding distance data of the previous frame (S38).

[0050] When there is no interference with adjacent pixels of the pixel (N in S32), the distance data correction unit 60 uses the distance data of the current frame of the distance measurement image as it is (S40).

[0051] When examining the presence or absence of interference of adjacent pixels of the pixel in step S32, examine the presence or absence of interference of pixels adjacent to the pixel or peripheral pixels in the range up to two pixels. Also, as another method, it is possible to examine the presence or absence of interference of peripheral pixels of the pixel by limiting to the body parts of a person or the area of an object detected in the image, or by limiting to the area of the same color as the pixel in the RGB image and examining the presence or absence of interference of the peripheral pixels of the pixel.

[0052] FIG. 6 is a flowchart for explaining the interference avoidance process of the present embodiment.

[0053] Examine whether the period during which the interference determination unit 50 determines that there is infrared light interference continues for a predetermined time or more (S50).

[0054] When the period during which it is determined that there is infrared light interference continues for a predetermined time or more (Y in S50), the interference avoidance processing unit 70 shifts the driving timing of the driving control unit 80 (S52).

[0055] Return to step S50. As a result of the interference determination process by the interference determination unit 50 again, if interference is continuously detected (Y in S50), further shift the driving timing of the driving control unit 80 (S52). Repeat this loop, and when the interference no longer continues (N in S50), end the interference avoidance process.

[0056] In step S52, instead of shifting the driving timing of the driving control unit 80, the driving frequency of the distance measurement sensor 22 may be changed.

[0057] As described above, according to the imaging device 100 of the embodiment of the present invention, when performing distance measurement imaging of an object by a plurality of imaging devices 100, since the imaging device 100 can detect and avoid infrared light interference alone, it is not necessary to install a synchronization signal cable between the plurality of imaging devices 100, and the processing load during distance measurement imaging is reduced. Further, since the distance data of pixels with infrared light interference in the current frame of the distance measurement image is replaced with the distance data of the corresponding pixels in the previous frame, abnormal distance data is corrected to normal distance data, and the three-dimensional video data of the object is free from interference-induced distortion, and high-quality three-dimensional video data can be generated.

[0058] As described above, the present invention has been described based on the embodiments. It is understood by those skilled in the art that the embodiments are illustrative, and various modifications are possible for each combination of these components and each processing process, and such modifications are also within the scope of the present invention.

Explanation of Reference Numerals

[0059] 10 lens, 12 light source unit, 20 RGB sensor, 22 distance measurement sensor, 30 RGB image acquisition unit, 32 distance measurement image acquisition unit, 40 RGB image motion detection unit, 42 distance measurement image motion detection unit, 50 interference determination unit, 60 distance data correction unit, 70 interference avoidance processing unit, 80 drive control unit, 90 point cloud / mesh generation unit, 100 imaging device.

Claims

1. an RGB image acquisition unit that acquires an RGB image of an object; a distance measurement image acquisition unit that acquires a distance measurement image of the object on the same optical axis; an RGB image motion detection unit that detects the motion in the RGB image for each pixel by taking a difference between the current frame and the previous frame of the RGB image; a distance measurement image motion detection unit that detects the motion in the distance measurement image for each pixel by taking a difference between the current frame and the previous frame of the distance measurement image; an imaging device comprising: an interference determination unit that determines, for each pixel, the presence or absence of infrared light interference at the time of acquisition of the distance measurement image based on the results of the presence or absence of motion in the RGB image and the presence or absence of motion in the distance measurement image.

2. The imaging device according to claim 1, wherein the interference determination unit determines, for each pixel, that there is infrared light interference when there is no motion in the RGB image and there is motion in the distance measurement image.

3. The imaging device according to claim 1 or 2, further comprising a distance data correction unit that replaces the distance data of the pixel in the current frame with the distance data of the corresponding pixel in the previous frame for the pixel determined to have infrared light interference in the distance measurement image.

4. The imaging device according to claim 1 or 2, further comprising an interference avoidance processing unit that shifts the light emission timing of the infrared light when the period in which it is determined that there is infrared light interference in the distance measurement image continues for a predetermined time or longer.

5. a step of detecting the motion in the RGB image for each pixel by taking a difference between the current frame and the previous frame of the RGB image of the object; a step of detecting the motion in the distance measurement image for each pixel by taking a difference between the current frame and the previous frame of the distance measurement image of the object acquired on the same optical axis; an interference detection method comprising: a step of determining, for each pixel, the presence or absence of infrared light interference at the time of acquisition of the distance measurement image based on the results of the presence or absence of motion in the RGB image and the presence or absence of motion in the distance measurement image.

Images