3D imaging device

Abstract

To provide a three-dimensional video imaging device capable of properly measuring a wide range of distance.SOLUTION: A three-dimensional video imaging device comprises an infrared light source unit, a 3-CCD imaging element unit, and a processing unit. The 3-CCD imaging element unit comprises an imaging element which receives visible light, a short-range TOF sensor which receives a portion of infrared light, and a long-distance TOF sensor which receives the remaining infrared light. The short-range TOF sensor and the long-distance TOF sensor include a first part which accumulates charges at a first exposure timing and a second part which accumulates charges at a second exposure timing. The processing unit calculates first distance data based on the charges accumulated in the short-range TOF sensor, calculates second distance data based on the charges accumulated in the long-distance TOF sensor, determines whether a distance between the three-dimensional video imaging device and a subject is included in a short-distance range or a long-distance range, and switches an output of distance data based on the determined result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This invention relates to a three-dimensional image capturing device. [Background technology]

[0002] Patent Document 1 describes an electromagnetic wave detection device in which the optical axes of multiple sensors can be aligned. Patent Document 1 also describes that a single object can be detected by a visible light image sensor, an infrared light image sensor, etc., that an incident electromagnetic wave is separated into electromagnetic waves having different wavelengths and incident on multiple sensors, and that each sensor detects the image of an object viewed from the same direction as images of multiple electromagnetic waves having different wavelengths. The electromagnetic wave detection device described in Patent Document 1 comprises a first prism, a second prism, a third prism, a first layer and a second layer (at least one of a visible light reflective coating, a half mirror, a beam splitter, a dichroic mirror, a cold mirror, a hot mirror, a metasurface, a short-pass filter, a long-pass filter, a band-pass filter and a deflection element).

[0003] Unlike the electromagnetic wave detection device described in Patent Document 1, an imaging device is conceivable that includes lenses and sensors for RGB (Red-Green-Blue) and lenses and sensors for infrared images, thereby obtaining RGB image data and obtaining distance from the infrared image data. However, in such an imaging device, parallax occurs between the RGB lens and the infrared TOF (Time of Flight) lens, so when applied to a system that combines RGB image data and distance data, processing to correct for parallax becomes necessary. Another approach is to use a single TOF sensor and vary the shutter speed (exposure period) and exposure timing of the TOF sensor to perform near-range and far-range measurements for each frame. However, with this method, the distance obtained from the measurement results does not switch smoothly when the object (subject) is moving. Furthermore, with this method, the frame rate of the distance obtained by combining the frame data obtained from the near-range measurement and the frame data obtained from the far-range measurement becomes half the frame rate of the original TOF sensor, making it unable to handle smooth motion. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-027174 [Overview of the project] [Problems that the invention aims to solve]

[0005] In view of the problems described above, the present invention aims to provide a three-dimensional image capturing device that can appropriately measure distances over a wide range. [Means for solving the problem]

[0006] One aspect of the present invention is a three-dimensional image capturing apparatus comprising: a light source unit that irradiates infrared light toward a subject; a three-chip image sensor unit into which reflected infrared light irradiated by the light source unit and reflected by the subject and reflected visible light from the subject are incident; and a processing unit that controls the light source unit and the three-chip image sensor unit and processes signals output from the three-chip image sensor unit, wherein the three-chip image sensor unit comprises: a first prism into which infrared light and visible light from the subject are incident; a reflective dichroic film that reflects visible light from the infrared light and visible light incident on the first prism and transmits infrared light; and a unit that receives the visible light reflected by the reflective dichroic film. The system comprises a first image sensor, a second prism into which infrared light transmitted by the reflective dichroic film is incident, a half-mirror that reflects a portion of the infrared light incident on the second prism and transmits the remainder of the infrared light incident on the second prism, a third prism into which infrared light transmitted by the half-mirror is incident, one of the second and third image sensors that receives the infrared light transmitted by the half-mirror, and the other of the second and third image sensors that receives the infrared light reflected by the half-mirror, wherein the second image sensor is a short-range TOF (Time-of-Flight) sensor used for measuring the short-range distance, which is the distance range from the short-range minimum value to the short-range maximum value. The third image sensor is a TOF sensor for long-range use, used to measure a long-range distance range that is a distance range farther than the short-range range and is a distance range from the minimum long-range value to the maximum long-range value, the second image sensor comprises a first portion that accumulates charge at a first exposure timing of the second image sensor, which is an exposure timing of the second image sensor in which at least most of the emission period of the light source is included in the exposure period of the second image sensor, and a second portion that accumulates charge at a second exposure timing of the second image sensor, which is an exposure timing of the second image sensor in which the exposure period of the second image sensor begins after the end of the emission period of the light source, the third image sensor comprises a first portion that accumulates charge at a first exposure timing of the third image sensor, which is an exposure timing of the third image sensor in which at least a part of the emission period of the light source is included in the exposure period of the third image sensor,The processing unit comprises a second part which accumulates charge at a second exposure timing of the third image sensor, which is an exposure timing of the third image sensor that starts after the end of the light emission period of the light source, and the processing unit comprises a light emission control unit which controls the light emission of the light source, a second image sensor control unit which controls the exposure timing of the second image sensor, a third image sensor control unit which controls the exposure timing of the third image sensor, a first distance data calculation unit which calculates first distance data indicating the distance between the 3D image imaging device and the subject based on the charge accumulated in at least the first part of the second image sensor and the charge accumulated in the second part of the second image sensor, and a second distance data calculation unit which calculates second distance data indicating the distance between the 3D image imaging device and the subject based on the charge accumulated in at least the first part of the third image sensor and the charge accumulated in the second part of the third image sensor, and the second image sensor control unit which starts the exposure period corresponding to the first exposure timing of the second image sensor at a timing after the offset time based on the short distance minimum value has elapsed from the start timing of the light emission period of the light source, and the light emission period of the light source From the end timing of the second exposure, the third image sensor control unit starts an exposure period corresponding to the second exposure timing of the second image sensor when the offset time based on the minimum value for short distance has elapsed from the start timing of the light emission period of the light source unit, which is set within the same frame as the exposure period corresponding to the first exposure timing of the second image sensor, and from the end timing of the light emission period of the light source unit, which is set within the same frame as the exposure period corresponding to the second exposure timing of the second image sensor, which is set within the same frame as the exposure period corresponding to the second exposure timing of the second image sensor, and the processing unit starts an exposure period corresponding to the second exposure timing of the third image sensor when the offset time based on the minimum value for long distance has elapsed from the end timing of the light emission period of the light source unit, which is set within the same frame as the exposure period corresponding to the second exposure timing of the second image sensor, and based on a signal indicating the charge accumulated in at least the first part of the second image sensor, a signal indicating the charge accumulated in the second part of the second image sensor, a signal indicating the charge accumulated in the first part of the third image sensor, a signal indicating the charge accumulated in the second part of the third image sensor, and a preset discrimination rule.The 3D image capturing device comprises: a measurement range determination unit that determines whether the distance between the 3D image capturing device and the subject, as measured by the 3D image capturing device, falls within the short-range or long-range; and a distance data output switching unit that outputs the first distance data calculated by the first distance data calculation unit when the measurement range determination unit determines that the distance between the 3D image capturing device and the subject, as measured by the 3D image capturing device, falls within the short-range; and outputs the second distance data calculated by the second distance data calculation unit when the measurement range determination unit determines that the distance between the 3D image capturing device and the subject, as measured by the 3D image capturing device, falls within the long-range.

[0007] One aspect of the present invention is a three-dimensional image capturing apparatus comprising: a light source unit that irradiates infrared light toward a subject; a three-chip image sensor unit into which reflected infrared light irradiated by the light source unit and reflected by the subject and reflected visible light from the subject are incident; and a processing unit that controls the light source unit and the three-chip image sensor unit and processes signals output from the three-chip image sensor unit, wherein the three-chip image sensor unit comprises: a first prism into which infrared light and visible light from the subject are incident; a reflective dichroic film that reflects visible light from the infrared light and visible light incident on the first prism and transmits infrared light; and the visible light reflected by the reflective dichroic film The system comprises a first image sensor that receives light, a second prism into which infrared light transmitted by the reflective dichroic film is incident, a half-mirror that reflects a portion of the infrared light incident on the second prism and transmits the remainder of the infrared light incident on the second prism, a third prism into which infrared light transmitted by the half-mirror is incident, one of the second and third image sensors that receives the infrared light transmitted by the half-mirror, and the other of the second and third image sensors that receives the infrared light reflected by the half-mirror, wherein the second image sensor is a short-range TOF (Time-of-Flight) used for measuring distances from the short-range minimum to the short-range maximum. The third image sensor is a TOF sensor for long-range use, used to measure a distance range from the minimum long-range value to the maximum long-range value, where a part of the distance range from the minimum long-range value to the maximum long-range value and a part of the distance range from the minimum long-range value to the maximum long-range value overlap to form an overlap measurement range, the distance range from the minimum long-range value to the maximum long-range value minus the overlap measurement range is the short-range measurement range, and the distance range from the minimum long-range value to the maximum long-range value minus the overlap measurement range is the long-range measurement range, and the second image sensor has a first part that accumulates charge at the first exposure timing of the second image sensor, which is the exposure timing of the second image sensor, in which at least the majority of the light emission period of the light source is included in the exposure period of the second image sensor,The third image sensor comprises a second portion that accumulates charge at a second exposure timing of the second image sensor, where the exposure period of the second image sensor is an exposure timing of the second image sensor that starts after the end of the light emission period of the light source, and the third image sensor comprises a first portion that accumulates charge at a first exposure timing of the third image sensor, where at least a portion of the light emission period of the light source is included in the exposure period of the third image sensor, and a second portion of the third image sensor that accumulates charge at a second exposure timing of the third image sensor, where the exposure period of the third image sensor is an exposure timing of the third image sensor that starts after the end of the light emission period of the light source The processing unit comprises a second part that accumulates charge at two exposure timings, and the processing unit includes a light emission control unit that controls the light emission of the light source unit, a second image sensor control unit that controls the exposure timing of the second image sensor, a third image sensor control unit that controls the exposure timing of the third image sensor, a first distance data calculation unit that calculates first distance data indicating the distance between the 3D image imaging device and the subject based on the charge accumulated in at least the first part of the second image sensor and the charge accumulated in the second part of the second image sensor, and the charge accumulated in at least the first part of the third image sensor The 3D image capturing device includes a second distance data calculation unit that calculates second distance data indicating the distance between the 3D image capturing device and the subject based on the charge and the charge accumulated in the second part of the third image sensor, wherein the second image sensor control unit starts an exposure period corresponding to the first exposure timing of the second image sensor at a timing when an offset time based on the near-distance minimum value has elapsed from the start timing of the emission period of the light source unit, starts an exposure period corresponding to the second exposure timing of the second image sensor at a timing when an offset time based on the near-distance minimum value has elapsed from the end timing of the emission period of the light source unit, and the third image sensor control unit starts an exposure period corresponding to the first exposure timing of the third image sensor, which is set within the same frame as the exposure period corresponding to the first exposure timing of the second image sensor, at a timing when an offset time based on the far-distance minimum value has elapsed from the start timing of the emission period of the light source unit, and starts an exposure period corresponding to the second exposure timing of the third image sensor, which is set within the same frame as the exposure period corresponding to the first exposure timing of the second image sensor, at a timing when an offset time based on the far-distance minimum value has elapsed from the end timing of the emission period of the light source unit.The processing unit starts an exposure period set within the same frame as the exposure period corresponding to the second exposure timing of the second image sensor, and determines whether the distance between the 3D image imaging device and the subject, as measured by the 3D image imaging device, is included in the near-range measurement range, the overlap measurement range, or the far-range measurement range, based on a signal indicating the charge accumulated in at least the first part of the second image sensor, a signal indicating the charge accumulated in the second part of the second image sensor, a signal indicating the charge accumulated in the first part of the third image sensor, a signal indicating the charge accumulated in the second part of the third image sensor, and a preset discrimination rule, and determines whether the distance between the 3D image imaging device and the subject, as measured by the 3D image imaging device, is included in the near-range measurement range This is a 3D image imaging device comprising: a distance data output switching unit that outputs the first distance data calculated by the first distance data calculation unit when the measurement range determination unit determines that the distance between the 3D image imaging device and the subject, as measured by the 3D image imaging device, is included in the long-distance measurement range when the measurement range determination unit determines that the distance between the 3D image imaging device and the subject, as measured by the 3D image imaging device, is included in the overlap measurement range when the measurement range determination unit determines that the distance between the 3D image imaging device and the subject, as measured by the 3D image imaging device, is included in the overlap measurement range when the measurement range determination unit determines that the distance between the 3D image imaging device and the subject, is mixed and output by the distance data output switching unit.

[0008] One aspect of the present invention is a three-dimensional image capturing apparatus comprising: a light source unit that irradiates infrared light toward a subject; a three-chip image sensor unit into which reflected infrared light irradiated by the light source unit and reflected by the subject and reflected visible light from the subject are incident; and a processing unit that controls the light source unit and the three-chip image sensor unit and processes signals output from the three-chip image sensor unit, wherein the three-chip image sensor unit comprises: a first prism into which infrared light and visible light from the subject are incident; a reflective dichroic film that reflects visible light from the infrared light and visible light incident on the first prism and transmits infrared light; and a unit that receives the visible light reflected by the reflective dichroic film. The system comprises a first image sensor, a second prism into which infrared light transmitted by the reflective dichroic film is incident, a half-mirror that reflects a portion of the infrared light incident on the second prism and transmits the remainder of the infrared light incident on the second prism, a third prism into which infrared light transmitted by the half-mirror is incident, one of the second and third image sensors that receives the infrared light transmitted by the half-mirror, and the other of the second and third image sensors that receives the infrared light reflected by the half-mirror, wherein the second image sensor is a short-range TOF (Time-of-Flight) sensor used for measuring the short-range distance, which is the distance range from the short-range minimum value to the short-range maximum value. The third image sensor is a TOF sensor for long-range use, used to measure a long-range distance range that is a distance range further than the short-range range and is a distance range from the minimum long-range value to the maximum long-range value, the maximum short-range value is smaller than the minimum long-range value, the second image sensor has a first portion that accumulates charge at the first exposure timing of the second image sensor, which is the exposure timing of the second image sensor in which at least most of the emission period of the light source is included in the exposure period of the second image sensor, and a second portion that accumulates charge at the second exposure timing of the second image sensor, which is the exposure timing of the second image sensor in which the exposure period of the second image sensor begins after the end of the emission period of the light source, the third image sensor has a first portion that accumulates charge at the first exposure timing of the third image sensor, which is the exposure timing of the third image sensor in which at least a part of the emission period of the light source is included in the exposure period of the third image sensor,The processing unit comprises a second part which accumulates charge at a second exposure timing of the third image sensor, which is an exposure timing of the third image sensor that starts after the end of the light emission period of the light source, and the processing unit comprises a light emission control unit which controls the light emission of the light source, a second image sensor control unit which controls the exposure timing of the second image sensor, a third image sensor control unit which controls the exposure timing of the third image sensor, a first distance data calculation unit which calculates first distance data indicating the distance between the 3D image imaging device and the subject based on the charge accumulated in at least the first part of the second image sensor and the charge accumulated in the second part of the second image sensor, and a second distance data calculation unit which calculates second distance data indicating the distance between the 3D image imaging device and the subject based on the charge accumulated in at least the first part of the third image sensor and the charge accumulated in the second part of the third image sensor, and the second image sensor control unit which starts the exposure period corresponding to the first exposure timing of the second image sensor at a timing after the offset time based on the short distance minimum value has elapsed from the start timing of the light emission period of the light source, and the light emission period of the light source From the end timing of the second exposure, the third image sensor control unit starts an exposure period corresponding to the second exposure timing of the second image sensor when the offset time based on the minimum value for short distance has elapsed from the start timing of the light emission period of the light source unit, which is set to be within the same frame as the exposure period corresponding to the second exposure timing of the second image sensor, and from the end timing of the light emission period of the light source unit, which is set to be within the same frame as the exposure period corresponding to the second exposure timing of the second image sensor, and the processing unit starts an exposure period corresponding to the second exposure timing of the third image sensor when the offset time based on the minimum value for long distance has elapsed from the end timing of the light emission period of the light source unit, which is set to be within the same frame as the exposure period corresponding to the second exposure timing of the second image sensor, and based on a signal indicating the charge accumulated in at least the first part of the second image sensor, a signal indicating the charge accumulated in the second part of the second image sensor, a signal indicating the charge accumulated in the first part of the third image sensor, a signal indicating the charge accumulated in the second part of the third image sensor, and a preset discrimination rule.The 3D image capturing device comprises: a measurement range determination unit that determines whether the distance between the 3D image capturing device and the subject, as measured by the 3D image capturing device, falls within the short-distance range, the long-distance range, or a range greater than the maximum short-distance value and less than the minimum long-distance value; a distance data output switching unit that outputs the first distance data calculated by the first distance data calculation unit when the measurement range determination unit determines that the distance between the 3D image capturing device and the subject, as measured by the 3D image capturing device, falls within the short-distance range; outputs the second distance data calculated by the second distance data calculation unit when the measurement range determination unit determines that the distance between the 3D image capturing device and the subject, as measured by the 3D image capturing device, falls within the long-distance range; and outputs zero as distance data when the measurement range determination unit determines that the distance between the 3D image capturing device and the subject, as measured by the 3D image capturing device, falls within a range greater than the maximum short-distance value and less than the minimum long-distance value. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a three-dimensional image capturing device that can appropriately measure distances over a wide range. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example of a three-dimensional image capturing device 1 according to the first embodiment. [Figure 2] This figure illustrates an example of a general technology having a configuration similar to that of the three-chip image sensor unit 12 shown in Figure 1. [Figure 3] This figure shows a magnified view of a part of the three-chip image sensor unit 12 shown in Figure 1. [Figure 4] Figure 3 shows an example of the reflection spectral characteristics of the reflective dichroic film 12C. [Figure 5] Figure 3 shows an example of the characteristics of the infrared 850nm bandpass filters 12G1 and 12G2. [Figure 6] This is a diagram for explaining the charge accumulation and distance measurement mechanisms of three exposures of the second imaging element 12E2 (and the third imaging element 12E3). [Figure 7] This is a diagram conceptually showing the relationship between the light source unit 11, the subject S, the lens 12A, and the second imaging element 12E2 (the third imaging element 12E3). [Figure 8] This is a diagram comparing and showing an example of the first exposure timing, the second exposure timing, and the third exposure timing of the second imaging element 12E2 when the distance between the three-dimensional video imaging device 1 and the subject S is the near-distance minimum value Near_min, and the first exposure timing, the second exposure timing, and the third exposure timing of the second imaging element 12E2 when the distance between the three-dimensional video imaging device 1 and the subject S is the near-distance maximum value Near_max. [Figure 9] This is a diagram comparing and showing an example of the first exposure timing, the second exposure timing, and the third exposure timing of the third imaging element 12E3 when the distance between the three-dimensional video imaging device 1 and the subject S is the far-distance minimum value Far_min, and the first exposure timing, the second exposure timing, and the third exposure timing of the third imaging element 12E3 when the distance between the three-dimensional video imaging device 1 and the subject S is the far-distance maximum value Far_max. [Figure 10] This is a diagram showing an example of the light emission of the light source unit 11 and the exposure order of the second imaging element 12E2 and the third imaging element 12E3 when the pulse count number N is set to 500 times. [Figure 11A] This is a diagram for explaining the basis of the discrimination rule shown in Table 1. [Figure 11B] This is a diagram for explaining the basis of the discrimination rule shown in Table 1. [Figure 11C] This is a diagram for explaining the basis of the discrimination rule shown in Table 1. [Figure 12] This is a diagram showing an example of the switching order of signals by the output mixing unit 13N. [Figure 13] This is a diagram showing an enlarged view of a part of the three-plate imaging element unit 12 of the first example of the three-dimensional video imaging device 1 of the second embodiment. [Figure 14]It is a diagram for explaining the pulse count of the first example of the three-dimensional video imaging device 1 according to the third embodiment. [Figure 15] It is a diagram for explaining the pulse count of the second example of the three-dimensional video imaging device 1 according to the third embodiment. [Figure 16] It is a diagram showing an example of the three-dimensional video imaging device 1 according to the fourth embodiment. [Figure 17A] It is a diagram showing the timing in an example of the three-dimensional video imaging device 1 according to the fourth embodiment. [Figure 17B] It is a diagram showing the timing in an example of the three-dimensional video imaging device 1 according to the fourth embodiment. [Figure 17C] It is a diagram showing the timing in an example of the three-dimensional video imaging device 1 according to the fourth embodiment. [Figure 17D] It is a diagram showing the timing in an example of the three-dimensional video imaging device 1 according to the fourth embodiment. [Figure 18A] It is a diagram showing the timing in an example of the three-dimensional video imaging device 1 according to the fifth embodiment. [Figure 18B] It is a diagram showing the timing in an example of the three-dimensional video imaging device 1 according to the fifth embodiment. [Figure 18C] It is a diagram showing the timing in an example of the three-dimensional video imaging device 1 according to the fifth embodiment. [Figure 18D] It is a diagram showing the timing in an example of the three-dimensional video imaging device 1 according to the fifth embodiment.

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments of the three-dimensional video imaging device of the present invention will be described with reference to the accompanying drawings.

[0012] <First Embodiment> FIG. 1 is a diagram showing an example of the three-dimensional video imaging device 1 according to the first embodiment. In the example shown in FIG. 1, the three-dimensional video imaging device 1 includes a light source unit 11, a three-plate imaging element unit 12, and a processing unit 13. The light source unit 11 irradiates infrared light (specifically, distance measurement light) toward the subject S. In the example shown in Figure 1, the light source 11 is composed of an infrared laser diode array. In other examples, the light source 11 may be composed of something other than an infrared laser diode array, such as an infrared LED (Light Emitting Diode). In the example shown in Figure 1, a laser light-emitting element with a peak light output at a wavelength of 850 nm is used.

[0013] In the example shown in Figure 1, the subject S is not only illuminated with infrared light from the light source unit 11, but also with visible light from the sun, lighting equipment, etc. (not shown). Therefore, the three-chip image sensor unit 12 is incident on the infrared light that is illuminated by the light source unit 11 and reflected by the subject S, and the visible light reflected from the subject S (i.e., the visible light that is illuminated by the sun, lighting equipment, etc. and reflected by the subject S).

[0014] Figure 2 is a diagram illustrating an example of a general technology having a configuration similar to that of the three-chip image sensor unit 12 shown in Figure 1. More specifically, Figure 2 shows an example of a prism in a typical RGB three-chip camera. In the example shown in Figure 2, a portion of the visible light that passes through the infrared cut filter is reflected by the green reflective dichroic film and incident on the green monochrome image sensor (CMOS (Complementary Metal Oxide Semiconductor) sensor) via the green trimming filter. Of the visible light that passes through the green reflective dichroic film, a portion of the remaining visible light that passes through the blue reflective dichroic film is incident on the red monochrome image sensor (CMOS sensor) via the red trimming filter, and the remaining visible light that passes through the green reflective dichroic film is reflected by the blue reflective dichroic film and incident on the blue monochrome image sensor (CMOS sensor) via the blue trimming filter. In other words, the wavelength of visible light is divided into the red wavelength band region, the green wavelength band region, and the blue wavelength band region, and incident on the respective monochrome image sensors, resulting in the R signal, G signal, and B signal.

[0015] Figure 3 is a magnified view of a portion of the three-chip image sensor unit 12 shown in Figure 1. In the example shown in Figure 3, the three-chip image sensor unit 12 comprises a lens 12A (see Figure 1), a first prism 12B1, a second prism 12B2, a third prism 12B3, a reflective dichroic film 12C, a half mirror 12D, a first image sensor 12E1, a second image sensor 12E2, a third image sensor 12E3, an infrared cut filter 12F, an infrared 850nm bandpass filter 12G1, and an infrared 850nm bandpass filter 12G2. Infrared and visible light from the subject S (see Figure 1) enters the first prism 12B1 via lens 12A. The reflective dichroic film 12C reflects the visible light and transmits the infrared light from the infrared and visible light incident on the first prism 12B1. The first image sensor 12E1 receives the visible light that has been reflected by the reflective dichroic film 12C and transmitted through the infrared cut filter 12F. In the example shown in Figure 3, the first image sensor 12E1 is a color image sensor having a Bayer array. In other examples, the first image sensor 12E1 does not need to have a Bayer array.

[0016] Figure 4 shows an example of the reflection spectral characteristics of the reflective dichroic film 12C shown in Figure 3. In Figure 4, the horizontal axis represents wavelength, and the vertical axis represents reflectance. As shown in Figure 4, the reflective dichroic film 12C shown in Figure 3 reflects light in the visible light region and transmits infrared light (i.e., does not reflect infrared light).

[0017] In the example shown in Figure 3, a Bayer pattern color image sensor is used as the first image sensor 12E1 to obtain R, Gr, Gb, and B signals. In detail, in the example shown in Figure 3, a CMOS sensor with a global shutter (not shown) is used as the first image sensor 12E1. In other examples, an image sensor other than a CMOS sensor, such as a CCD (Charge Coupled Device) sensor with a global shutter, may be used as the first image sensor 12E1.

[0018] In the example shown in Figure 3, infrared light transmitted by the reflective dichroic film 12C is incident on the second prism 12B2. The half mirror 12D reflects a portion of the infrared light incident on the second prism 12B2 and transmits the remainder. The infrared light transmitted by the half mirror 12D is incident on the third prism 12B3. The second image sensor 12E2 receives infrared light that has been transmitted by the half mirror 12D and passed through the infrared 850nm bandpass filter 12G1. The third image sensor 12E3 receives infrared light that has been reflected by the half mirror 12D and transmitted through the infrared 850nm bandpass filter 12G2.

[0019] Figure 5 shows an example of the characteristics of the infrared 850nm bandpass filters 12G1 and 12G2 shown in Figure 3. In Figure 4, the horizontal axis represents wavelength and the vertical axis represents transmittance. In the example shown in Figure 5, the infrared 850nm bandpass filters 12G1 and 12G2 shown in Figure 3 have a center wavelength of 850nm and a full width at half maximum of approximately 80nm.

[0020] In the example shown in Figure 3, the infrared 850nm bandpass filter 12G2 is constructed, for example, by coating a dielectric film on the exit surface of the second prism 12B2, and the infrared 850nm bandpass filter 12G1 is constructed, for example, by coating a dielectric film on the exit surface of the third prism 12B3. In the example shown in Figure 3, the second image sensor 12E2 is a near-range TOF sensor used to measure the near-range range Near_min to Near_max, which is the distance range from the near-range minimum value Near_min to the near-range maximum value Near_max. The third image sensor 12E3 is a far-range TOF sensor used to measure the far-range range Far_min to Far_max, which is a distance range further than the near-range range Near_min to Near_max, and is the distance range from the far-range minimum value Far_min to the far-range maximum value Far_max. The second image sensor 12E2 and the third image sensor 12E3 have two-dimensional array light-receiving pixels and have good spectral sensitivity characteristics for near-infrared wavelengths. In the example shown in Figure 3, CMOS sensors with a global shutter (not shown) are used as the second image sensor 12E2 and the third image sensor 12E3. In other examples, image sensors other than CMOS sensors, such as a CCD sensor with a global shutter, may be used as the second image sensor 12E2 and the third image sensor 12E3. In another example, the second image sensor 12E2 may be a TOF sensor for long-range applications, and the third image sensor 12E3 may be a TOF sensor for short-range applications.

[0021] In the example shown in Figure 3, the infrared image received by the third image sensor 12E3 is inverted vertically in the half-mirror 12D. Therefore, the third image sensor 12E3 is positioned so that the right side of Figure 3 corresponds to the upper side of the sensor screen of the third image sensor 12E3. As a result, the image captured by the third image sensor 12E3 is inverted only horizontally (i.e., not vertically).

[0022] In the example shown in Figure 3, the three-chip image sensor unit 12 is constructed using a gapless prism, but in other examples, the three-chip image sensor unit 12 may be constructed using a Philips-type prism.

[0023] In the example shown in Figure 1, the processing unit 13 controls the light source unit 11 and the three-chip image sensor unit 12, and processes the signals output from the three-chip image sensor unit 12. The processing unit 13 includes, for example, a light emission control unit 13A, a first image sensor control unit 13B, a second image sensor control unit 13C, a third image sensor control unit 13D, an RGB camera process processing unit 13E, a first distance data calculation unit 13F, a second distance data calculation unit 13G, a control unit 13H, a measurement range determination unit 13I, a distance data output switching unit 13J, a storage unit 13K, an IR data output switching unit 13L, an IR data processing unit 13M, an output mixing unit 13N, and an output interface unit 13P. The light emission control unit 13A controls the light emission of the light source unit 11. In other words, the light emission control unit 13A controls the light emission period Tp (see Figure 6, etc.) and the light emission timing of the light source unit 11. The light emission control unit 13A includes a light source drive control unit 13A1 and a light source unit light emission control pulse setting unit 13A2. The light source drive control unit 13A1 controls the driving of, for example, the infrared laser diode array that constitutes the light source unit 11. The light source unit light emission control pulse setting unit 13A2 generates pulses to cause the light source unit 11 (infrared laser diode array) to emit light.

[0024] Figure 6 is a diagram illustrating the charge accumulation and distance measurement mechanism of the second image sensor 12E2 (and the third image sensor 12E3) during three exposures. Specifically, Figure 6(A) shows the first part 12E21, the second part 12E22, and the third part 12E23 of the second image sensor 12E2 (and the first part 12E31, the second part 12E32, and the third part 12E33 of the third image sensor 12E3). Figure 6(B) shows the first exposure timing of the second image sensor 12E2, Figure 6(C) shows the second exposure timing of the second image sensor 12E2, and Figure 6(D) shows the third exposure timing of the second image sensor 12E2.

[0025] As shown in Figure 6(A), the second image sensor 12E2 comprises a first portion 12E21, a second portion 12E22, and a third portion 12E23. The first part 12E21 is the part that accumulates charge at the first exposure timing (S0' exposure timing) of the second image sensor 12E2 shown in Figure 6(B). As shown in Figure 6(B), the first exposure timing (S0' exposure timing) of the second image sensor 12E2 is the exposure timing of the second image sensor 12E2 in which most of the light emission period Tp of the light source unit 11 is included in the exposure period of the second image sensor 12E2 (the period shown as "Global Shutter" in Figure 6(B)). The second part 12E22 is the part that accumulates charge at the second exposure timing (S1' exposure timing) of the second image sensor 12E2 shown in Figure 6(C). As shown in Figure 6(C), the second exposure timing (S1' exposure timing) of the second image sensor 12E2 is the exposure timing of the second image sensor 12E2 that starts after the end of the light emission period Tp of the light source unit 11, during which the exposure period of the second image sensor 12E2 (the period indicated as "Global Shutter" in Figure 6(C)) of the second image sensor 12E2 is. The third portion 12E23 is the portion that accumulates charge at the third exposure timing (BG exposure timing) of the second image sensor 12E2, as shown in Figure 6(D). As shown in Figure 6(D), the third exposure timing (BG exposure timing) of the second image sensor 12E2 is the exposure timing of the second image sensor 12E2 when it does not receive reflected infrared light that has been illuminated by the light source 11 and reflected by the subject S.

[0026] As shown in Figure 6(A), the third image sensor 12E3 is configured similarly to the second image sensor 12E2. The third image sensor 12E3 comprises a first portion 12E31, a second portion 12E32, and a third portion 12E33. The first portion 12E31 is the portion that accumulates charge during the first exposure (S0' exposure) timing of the third image sensor 12E3 (see Figures 9(A) and 9(D)). The first exposure (S0' exposure) timing of the third image sensor 12E3 is the exposure timing of the third image sensor 12E3 in which a portion of the light emission period Tp of the light source unit 11 is included in the exposure period of the third image sensor 12E3 (the period indicated as "GS2" in Figures 9(A) and 9(D)). The second part 12E32 is the part that accumulates charge during the second exposure (S1' exposure) timing of the third image sensor 12E3 (see Figures 9(B) and 9(E)). The second exposure (S1' exposure) timing of the third image sensor 12E3 is the exposure timing of the third image sensor 12E3 that starts after the end of the light emission period Tp of the light source unit 11, during which the exposure period of the third image sensor 12E3 (indicated as "GS2" in Figures 9(B) and 9(E)) of the third image sensor 12E3 is the exposure timing of the third image sensor 12E3. The third portion 12E33 is the portion that accumulates charge during the third exposure (BG exposure) timing of the third image sensor 12E3 (see Figures 9(C) and 9(F)). The third exposure (BG exposure) timing of the third image sensor 12E3 is the exposure timing of the third image sensor 12E3 when it does not receive reflected infrared light that has been illuminated by the light source 11 and reflected by the subject S.

[0027] In the example shown in Figure 6, techniques similar to those described in, for example, International Publication No. 2014 / 207788 and Japanese Patent Publication No. 2004-294420 are used to ensure that the charges accumulated during the S0' exposure period, the charges accumulated during the S1' exposure period, and the charges accumulated during the BG exposure period are stored independently without mixing. In other words, in the example shown in Figure 6, the second image sensor 12E2 (third image sensor 12E3) includes a first part 12E21 (first part 12E31) that stores the charge at the S0' exposure timing for the photoelectrically converted charge of each pixel's photodiode, a second part 12E22 (second part 12E32) that stores the charge at the S1' exposure timing, and a third part 12E23 (third part 12E33) that stores the charge at the BG exposure timing, and has the function of switching between each of these parts for output. In the example shown in Figure 6 (the first example of the 3D image capturing device 1 of the first embodiment), the second image sensor 12E2 includes a third portion 12E23, and the third image sensor 12E3 includes a third portion 12E33. However, in other examples (the second example of the 3D image capturing device 1 of the first embodiment), the second image sensor 12E2 does not have a third portion 12E23, and the third image sensor 12E3 does not have a third portion 12E33.

[0028] Figure 7 is a conceptual diagram showing the relationship between the light source unit 11, the subject S, the lens 12A, and the second image sensor 12E2 (third image sensor 12E3). As shown in Figure 7, in the 3D image capturing device 1 of the first embodiment, the light source unit 11 emits infrared light during the emission period Tp shown in Figure 6(B). During the exposure period of the second image sensor 12E2 (the period shown as "Global Shutter" in Figure 6(B)), the second image sensor 12E2 receives reflected infrared light from the subject S, and a charge (signal) is accumulated in the first part 12E21. Furthermore, in the 3D image capturing device 1 of the first embodiment, the light source unit 11 emits infrared light during the emission period Tp shown in Figure 6(C). The exposure period of the second image sensor 12E2 (the period indicated as "Global Shutter" in Figure 6(C)) starts after the end of the emission period Tp of the light source unit 11. During the exposure period of the second image sensor 12E2 (the period indicated as "Global Shutter" in Figure 6(C)), the second image sensor 12E2 receives reflected infrared light from the subject S, and a charge (signal) is accumulated in the second part 12E22. Furthermore, in the 3D image capturing device 1 of the first embodiment, the light source unit 11 does not emit infrared light, and during the exposure period of the second image sensor 12E2 (the period indicated as "Global Shutter" in Figure 6(D)), that is, during the period indicated as "BG" in Figure 6(D), the second image sensor 12E2 receives background infrared light, and a charge (signal) is accumulated in the third part 12E23.

[0029] Similarly, in the 3D image capturing device 1 of the first embodiment, during the exposure period of the second image sensor 12E2 (the period indicated as "Global Shutter" in Figure 6(B)), that is, during the period indicated as "BG" in Figure 6(B), the second image sensor 12E2 receives background infrared light, and an electric charge (signal) is accumulated in the first part 12E21. Furthermore, in the 3D image capturing device 1 of the first embodiment, during the exposure period of the second image sensor 12E2 (the period indicated as "Global Shutter" in Figure 6(C)), that is, during the period indicated as "BG" in Figure 6(C), the second image sensor 12E2 receives background infrared light, and an electric charge (signal) is accumulated in the second part 12E22.

[0030] In the example shown in Figure 6(B) (S0' exposure timing of the second image sensor 12E2), the level S0 (=S0'-BG) of infrared emission reflection at the S0' exposure timing of the second image sensor 12E2 is obtained by subtracting the background infrared light charge "BG" accumulated in the first part 12E21 during the exposure period of the second image sensor 12E2 from the total charge "S0'" accumulated in the first part 12E21 during the exposure period of the second image sensor 12E2 (the period shown as "Global Shutter" in Figure 6(B)). In the example shown in Figure 6(C) (S1' exposure timing of the second image sensor 12E2), the level S1 (=S1'-BG) of infrared emission reflection at the S1' exposure timing of the second image sensor 12E2 is obtained by subtracting the background infrared light charge "BG" accumulated in the second part 12E22 during the exposure period of the second image sensor 12E2 (the period shown as "Global Shutter" in Figure 6(C)) from the total charge "S1'" accumulated in the second part 12E22. For example, similar to the technique described in paragraph 0033 of International Publication No. 2014 / 207788, the distance Z between the subject S and the 3D image acquisition device 1 can be obtained by using the following equation 1.

[0031] Z=C×△t / 2=C·Tp / 2×(S1 / S0)...Equation 1

[0032] In Equation 1 △t:TOF (Time of flight) C: Speed ​​of light Tp: Luminous period

[0033] In the example shown in Figure 1, the first image sensor control unit 13B controls the first image sensor 12E1. The first image sensor control unit 13B includes a first image sensor drive control unit 13B1 and a first image sensor drive control pulse setting unit 13B2. The first image sensor drive control unit 13B1 controls the driving of the first image sensor 12E1. The first image sensor drive control pulse setting unit 13B2 generates pulses for driving the first image sensor 12E1. The second image sensor control unit 13C controls the exposure timing of the second image sensor 12E2 as shown in Figures 6(B), 6(C), and 6(D). The second image sensor control unit 13C includes a second image sensor drive control unit 13C1 and a second image sensor drive control pulse setting unit 13C2. The second image sensor drive control unit 13C1 controls the driving of the second image sensor 12E2. The second image sensor drive control pulse setting unit 13C2 generates pulses for driving the second image sensor 12E2. The third image sensor control unit 13D controls the exposure timing of the third image sensor 12E3. The third image sensor control unit 13D includes a third image sensor drive control unit 13D1 and a third image sensor drive control pulse setting unit 13D2. The third image sensor drive control unit 13D1 controls the driving of the third image sensor 12E3. The third image sensor drive control pulse setting unit 13D2 sets pulses for driving the third image sensor 12E3.

[0034] The RGB camera processing unit 13E processes the output signal of the first image sensor 12E1. Specifically, the output signal of the first image sensor 12E1 is adjusted to an appropriate image level within the first image sensor 12E1, then converted to a digital signal, further converted to a serial signal, and transmitted to the RGB camera processing unit 13E using subLVDS (Low Voltage Differential Signaling). The RGB camera processing unit 13E comprises an interface unit 13E1, a black balance unit 13E2, a debayer unit 13E3, a color matrix unit 13E4, a gamma unit 13E5, a YCbCr conversion unit 13E6, a low-pass filter 13E7, and a multiplex unit 13E8. The interface unit 13E1 processes the serial signal sent from the first image sensor 12E1 back into parallel signals (R signal, Gr signal, Gb signal, B signal). The black balance unit 13E2 performs black level correction. Specifically, it corrects the level of black data in RGB image data to a fixed level, such as zero.

[0035] The debayering unit 13E3 performs debayering (demosaicing), which is the process of converting a Bayer image (R signal, Gr signal, Gb signal, B signal) into a full-color image (R signal, G signal, B signal). The color matrix unit 13E4 performs color matrix correction on the RGB signals output by the color matrix unit 13E4.

[0036] The gamma unit 13E5 performs gamma correction on the RGB signals output by the color matrix unit 13E4.

[0037] The YCbCr conversion unit 13E6 converts the red, green, and blue values ​​of the RGB image into luminance (Y) and chrominance (Cb and Cr) values ​​of the YCbCr image.

[0038] The low-pass filter 13E7 performs chromasampling (a process that reduces color information in the signal to favor the luminance data) on the data, including the luminance data and dye data, after the conversion by the YCbCr conversion unit 13E6.

[0039] The multiplexing unit 13E8 performs multiplexing of data (YCbCr4:2:2 data) that includes luminance data and color difference data after chromasampling by the low-pass filter 13E7. In the example shown in Figure 1, the data processed by the RGB camera processing unit 13E (YUV4:2:2 data) is sequentially stored in the first buffer memory 13K1 of the storage unit 13K. In other examples, the RGB camera processing unit 13E may perform any known RGB camera processing different from that shown in Figure 1 (for example, any of the processes described on the website indicated by the URL above).

[0040] In the example shown in Figure 1, the RGB camera processing unit 13E performs the above-described processing at a frame rate of 30 Fps. In other examples, the RGB camera processing unit 13E may perform the above-described processing at a frame rate different from 30 Fps.

[0041] In the example shown in Figure 1, the first distance data calculation unit 13F calculates first distance data Z1, which indicates the distance between the 3D image capturing device 1 and the subject S, based on the charge accumulated in the first part 12E21 of the second image sensor 12E2 ("S0'" shown in Figure 6(A)), the charge accumulated in the second part 12E22 of the second image sensor 12E2 ("S1'" shown in Figure 6(A)), and the charge accumulated in the third part 12E23 of the second image sensor 12E2 ("BG" shown in Figure 6(A)). In detail, inside the second image sensor 12E2, the S0' signal indicating the charge accumulated in the first part 12E21, the S1' signal indicating the charge accumulated in the second part 12E22, and the BG signal indicating the charge accumulated in the third part 12E23 are converted to digital signals using A / D conversion, further converted to serial signals, and transmitted to the first distance data calculation unit 13F via subLVDS. The first distance data calculation unit 13F includes an interface unit 13F1, a delay circuit 13F2, a delay circuit 13F3, a delay circuit 13F4, a timing adjustment unit 13F5, a timing adjustment unit 13F6, a subtraction unit 13F7, a subtraction unit 13F8, a calculation unit 13F9, and an addition unit 13FX. The interface unit 13F1 processes the serial signal sent from the second image sensor 12E2 back into a 12-bit parallel signal (S0' signal, S1' signal, BG signal).

[0042] The delay circuit 13F2 delays the S0' signal, which indicates the charge accumulated in the first part 12E21 of the second image sensor 12E2. The reason for delaying the S0' signal will be explained later. The delay circuit 13F3 delays the S1' signal, which indicates the charge accumulated in the second part 12E22 of the second image sensor 12E2. The delay circuit 13F4 delays the BG signal, which indicates the charge accumulated in the third part 12E23 of the second image sensor 12E2. The timing adjustment unit 13F5 adjusts the timing of the S0' signal processing so that the timing of the S0' signal processing on a pixel-by-pixel basis is the same as the timing of the BG signal processing. The timing adjustment unit 13F6 adjusts the timing of the S1' signal processing so that the timing of the S1' signal processing on a pixel-by-pixel basis is the same as the timing of the BG signal processing.

[0043] The subtraction unit 13F7 generates the S0t1 signal by performing the calculation "S0'-BG" based on the S0' signal and the BG signal. In other words, the subtraction unit 13F7 performs the process of subtracting the background infrared light charge "BG" accumulated in the first part 12E21 during the exposure period of the second image sensor 12E2 (the "Global Shutter" period in Figure 6(B)) from the total charge "S0'" accumulated in the first part 12E21 during the exposure period of the second image sensor 12E2 as explained with reference to Figure 6(B). The subtraction unit 13F8 generates the S1t1 signal by performing the calculation "S1'-BG" based on the S1' signal and the BG signal. In other words, the subtraction unit 13F8 performs the process of subtracting the background infrared light charge "BG" accumulated in the second part 12E22 during the exposure period of the second image sensor 12E2 (the "Global Shutter" period in Figure 6(C)) from the total charge "S1'" accumulated in the second part 12E22 during the exposure period of the second image sensor 12E2 as explained with reference to Figure 6(C). The arithmetic unit 13F9 performs an operation (S1t1 / S0t1) in which it divides the S1t1 signal generated by the subtraction unit 13F8 by the S0t1 signal generated by the subtraction unit 13F7. The addition unit 13FX performs an operation to add the offset value Zoffset set by the control unit 13H to the result of the calculation (S1t1 / S0t1) obtained by the calculation unit 13F9, and calculates the first distance data Z1 (=S1t1 / S0t1+Zoffset) which indicates the distance between the 3D image capturing device 1 based on the second image sensor 12E2 and the subject S. The offset value Zoffset will be described later.

[0044] In the example shown in Figure 1, the first distance data calculation unit 13F performs the above-described process at a frame rate of 30 Fps, synchronized with the processing by the RGB camera processing unit 13E. In other examples (where the processing by the RGB camera processing unit 13E is at a frame rate different from 30 Fps), the first distance data calculation unit 13F may perform the above-described process at a frame rate different from 30 Fps, synchronized with the processing by the RGB camera processing unit 13E.

[0045] Figure 8 is a diagram comparing an example of the first, second, and third exposure timings of the second image sensor 12E2 when the distance between the 3D image capturing device 1 and the subject S is the minimum near distance value Near_min, and the first, second, and third exposure timings of the second image sensor 12E2 when the distance between the 3D image capturing device 1 and the subject S is the maximum near distance value Near_max. In detail, Figures 8(A) and 8(D) correspond to Figure 6(B), Figures 8(B) and 8(E) correspond to Figure 6(C), and Figures 8(C) and 8(F) correspond to Figure 6(D). In the example shown in Figure 8, as shown in Figures 8(A) and 8(D), the second image sensor control unit 13C starts the exposure period GS1 corresponding to the first exposure timing (S0' exposure timing) of the second image sensor 12E2 when an offset time Toffset (=2 × (Near_min - Near_min × 0.01) / C) based on the near-field minimum value Near_min has elapsed from the start timing of the emission period Tp of the light source unit 11. In the example shown in Figure 8, the offset time Toffset is set so that when the distance between the 3D image capturing device 1 and the subject S is at the near-minimum value Near_min, the exposure period GS1 corresponding to the first exposure timing (S0' exposure timing) of the second image sensor 12E2 starts 1% earlier than the timing at which the reflected infrared light irradiated by the light source 11 and reflected by the subject S reaches the 3D image capturing device 1.

[0046] Furthermore, as shown in Figures 8(B) and 8(E), the second image sensor control unit 13C starts the exposure period GS1 corresponding to the second exposure timing (S1' exposure timing) of the second image sensor 12E2 when an offset time Toffset (=2 × (Near_min - Near_min × 0.01) / C) based on the near-field minimum value Near_min has elapsed from the end timing of the light emission period Tp of the light source unit 11. Furthermore, as shown in Figures 8(C) and 8(F), the second image sensor control unit 13C starts the exposure period GS1 corresponding to the third exposure timing (BG exposure timing) of the second image sensor 12E2 at a timing when the second image sensor 12E2 does not receive reflected infrared light that has been irradiated by the light source unit 11 and reflected by the subject S.

[0047] In the example shown in Figure 8, the length of the exposure period GS1 corresponding to the first exposure timing (S0' exposure timing) of the second image sensor 12E2 shown in Figures 8(A) and 8(D), the length of the exposure period GS1 corresponding to the second exposure timing (S1' exposure timing) of the second image sensor 12E2 shown in Figures 8(B) and 8(E), and the length of the exposure period GS1 corresponding to the third exposure timing (BG exposure timing) of the second image sensor 12E2 shown in Figures 8(C) and 8(F) are all set to the same length. The exposure period GS1 for the first exposure timing (S0' exposure timing), second exposure timing (S1' exposure timing), and third exposure timing (BG exposure timing) of the second image sensor 12E2 can be obtained using the following equation 2.

[0048] GS1=Tp+(Near_max-Near_min+2×Near_min×0.01) / C...Equation 2

[0049] In Equation 2 Near_min: Minimum value at close range Near_max: Maximum value at close range C: Speed ​​of light Tp: Luminous period

[0050] The offset value Zoffset, which corresponds to half the distance (one way) that infrared light travels during the offset time Toffset described above, can be expressed using the following equation 2A.

[0051] Zoffset=Toffset / 2×C...Formula 2A

[0052] The distance ZL1 between the 3D image acquisition device 1 based on the second image sensor 12E2 and the subject S can be obtained by using the following equation 3.

[0053] ZL1 = C × (△t + Toffset) / 2 =C Tp / 2×(S1 / S0)+Zoffset...Equation 3

[0054] In Equation 3 △t:TOF (Time of flight) C: Speed ​​of light Tp: Luminous period S0: Reflection level of infrared emission at the S0' exposure timing of the second image sensor 12E2 S1: Reflection level of infrared emission at the S1' exposure timing of the second image sensor 12E2

[0055] In the example shown in Figure 1, the second distance data calculation unit 13G calculates second distance data Z2, which indicates the distance between the 3D image capturing device 1 and the subject S, based on the charge accumulated in the first part 12E31 of the third image sensor 12E3 ("S0'" shown in Figure 6(A)), the charge accumulated in the second part 12E32 of the third image sensor 12E3 ("S1'" shown in Figure 6(A)), and the charge accumulated in the third part 12E33 of the third image sensor 12E3 ("BG" shown in Figure 6(A)). In detail, inside the third image sensor 12E3, the S0' signal indicating the charge accumulated in the first part 12E31, the S1' signal indicating the charge accumulated in the second part 12E32, and the BG signal indicating the charge accumulated in the third part 12E33 are converted into digital signals using A / D conversion, further converted into serial signals, and transmitted to the second distance data calculation unit 13G via subLVDS. The second distance data calculation unit 13G includes an interface unit 13G1, a horizontal inversion unit 13G2, a horizontal inversion unit 13G3, a horizontal inversion unit 13G4, a timing adjustment unit 13G5, a timing adjustment unit 13G6, a subtraction unit 13G7, a subtraction unit 13G8, a calculation unit 13G9, and an addition unit 13GX. The interface unit 13G1 processes the serial signal sent from the third image sensor 12E3 back into a 12-bit parallel signal (S0' signal, S1' signal, BG signal).

[0056] As described above, the image captured by the third image sensor 12E3 is inverted only horizontally. Therefore, the horizontal inversion unit 13G2 performs a process to horizontally invert the S0' signal, which indicates the charge accumulated in the first part 12E31 of the third image sensor 12E3, by using line memory. The horizontal inversion unit 13G3 performs a process to horizontally invert the S1' signal, which represents the charge accumulated in the second part 12E32 of the third image sensor 12E3. The horizontal inversion unit 13G4 performs a process to horizontally invert the BG signal, which represents the charge accumulated in the third part 12E33 of the third image sensor 12E3. As described above, the delay circuits 13F2, 13F3, and 13F4 perform signal delay processing in accordance with the time required for horizontal inversion processing by the horizontal inversion units 13G2, 13G3, and 13G4. The timing adjustment unit 13G5 adjusts the processing timing of the S0' signal so that the processing timing of the S0' signal on a pixel-by-pixel basis is the same as the processing timing of the BG signal. The timing adjustment unit 13G6 adjusts the processing timing of the S1' signal so that the processing timing of the S1' signal on a pixel-by-pixel basis is the same as the processing timing of the BG signal.

[0057] The subtraction unit 13G7 generates the S0t2 signal by performing the calculation "S0'-BG" based on the S0' signal and the BG signal. In other words, the subtraction unit 13G7 performs the process of subtracting the background infrared light charge "BG" accumulated in the first part 12E31 during the exposure period of the third image sensor 12E3 from the total charge "S0'" accumulated in the first part 12E31 during the exposure period of the third image sensor 12E3 (the period indicated as "GS2" in Figures 9(A) and 9(D)). The subtraction unit 13G8 generates the S1t2 signal by performing the calculation "S1'-BG" based on the S1' signal and the BG signal. In other words, the subtraction unit 13G8 performs the process of subtracting the background infrared light charge "BG" accumulated in the second part 12E32 during the exposure period of the third image sensor 12E3 from the total charge "S1'" accumulated in the second part 12E32 during the exposure period of the third image sensor 12E3 (the period indicated as "GS2" in Figures 9(B) and 9(E)). The arithmetic unit 13G9 performs an operation (S1t2 / S0t2) in which it divides the S1t2 signal generated by the subtraction unit 13G8 by the S0t2 signal generated by the subtraction unit 13G7. The addition unit 13GX performs an operation to add the offset value Zoffset2 set by the control unit 13H to the result of the calculation (S1t2 / S0t2) obtained by the calculation unit 13G9, and calculates the second distance data Z2 (=S1t2 / S0t2+Zoffset2) which indicates the distance between the 3D image capturing device 1 based on the third image sensor 12E3 and the subject S. The offset value Zoffset2 will be described later.

[0058] In the example shown in Figure 1, the second distance data calculation unit 13G performs the above-described process at a frame rate of 30 Fps, synchronized with the processing by the RGB camera processing unit 13E and the first distance data calculation unit 13F. In other examples (where the processing by the RGB camera processing unit 13E and the first distance data calculation unit 13F is at a frame rate different from 30 Fps), the second distance data calculation unit 13G may perform the above-described process at a frame rate different from 30 Fps, synchronized with the processing by the RGB camera processing unit 13E and the first distance data calculation unit 13F.

[0059] Figure 9 is a diagram comparing an example of the first, second, and third exposure timings of the third image sensor 12E3 when the distance between the 3D image imaging device 1 and the subject S is the minimum long-distance value Far_min, and the first, second, and third exposure timings of the third image sensor 12E3 when the distance between the 3D image imaging device 1 and the subject S is the maximum long-distance value Far_max. In the example shown in Figure 9, as shown in Figures 9(A) and 9(D), the third image sensor control unit 13D starts the exposure period GS2 corresponding to the first exposure timing (S0' exposure timing) of the third image sensor 12E3, which is set to be within the same frame as the exposure period GS1 (see Figures 8(A) and 8(D)) corresponding to the first exposure timing (S0' exposure timing) of the second image sensor 12E2, at a timing when an offset time Toffset2 (=2 × (Far_min - Far_min × 0.01) / C) based on the far distance minimum value Far_min has elapsed from the start timing of the emission period Tp of the light source unit 11. In the example shown in Figure 9, the offset time Toffset2 is set so that, when the distance between the 3D image capture device 1 and the subject S is the long-distance minimum value Far_min, the exposure period GS2 corresponding to the first exposure timing (S0' exposure timing) of the third image sensor 12E3 starts 1% earlier than the timing at which the reflected infrared light irradiated by the light source 11 and reflected by the subject S reaches the 3D image capture device 1.

[0060] Furthermore, as shown in Figures 9(B) and 9(E), the third image sensor control unit 13D starts an exposure period GS2 corresponding to the second exposure timing (S1' exposure timing) of the third image sensor 12E3, which is set within the same frame as the exposure period GS1 corresponding to the second exposure timing (S1' exposure timing) of the second image sensor 12E2 (see Figures 8(B) and 8(E)), at the timing when an offset time Toffset2 (=2 × (Far_min - Far_min × 0.01) / C) based on the far distance minimum value Far_min has elapsed from the end timing of the emission period Tp of the light source unit 11. Furthermore, as shown in Figures 9(C) and 9(F), the third image sensor control unit 13D starts the exposure period GS2 corresponding to the third exposure timing (BG exposure timing) of the third image sensor 12E3 at a timing when the third image sensor 12E3 does not receive reflected infrared light that has been irradiated by the light source unit 11 and reflected by the subject S.

[0061] In the example shown in Figure 9, the length of the exposure period GS2 corresponding to the first exposure timing (S0' exposure timing) of the third image sensor 12E3 shown in Figures 9(A) and 9(D), the length of the exposure period GS2 corresponding to the second exposure timing (S1' exposure timing) of the third image sensor 12E3 shown in Figures 9(B) and 9(E), and the length of the exposure period GS2 corresponding to the third exposure timing (BG exposure timing) of the third image sensor 12E3 shown in Figures 9(C) and 9(F) are all set to the same length. The exposure period GS2 for the first exposure timing (S0' exposure timing), second exposure timing (S1' exposure timing), and third exposure timing (BG exposure timing) of the third image sensor 12E3 can be obtained using the following equation 4.

[0062] GS2=Tp+(Far_max-Far_min+2×Far_min×0.01) / C...Equation 4

[0063] In Equation 4 Far_min: Minimum distance Far_max: Maximum distance C: Speed ​​of light Tp: Luminous period

[0064] The offset value Zoffset2, which corresponds to half the distance (one way) that infrared light travels during the offset time Toffset2 described above, can be expressed using the following equation 4A.

[0065] Zoffset2=Toffset2 / 2×C...Formula 4A

[0066] The distance ZL2 between the 3D image acquisition device 1 based on the third image sensor 12E3 and the subject S can be obtained by using the following equation 5.

[0067] ZL2 = C × (△t + Toffset2) / 2 =C Tp / 2×(S1 / S0)+Zoffset2...Equation 5

[0068] In Equation 5 △t:TOF (Time of flight) C: Speed ​​of light Tp: Luminous period S0: Reflection level of infrared emission at the S0' exposure timing of the third image sensor 12E3. S1: Reflection level of infrared emission at the S1' exposure timing of the third image sensor 12E3.

[0069] In the example shown in Figure 1, the light emission period Tp [nsec] of the light source unit 11 is set using the light emission period equivalent length Tp_period [mm], which is the larger of the near-range range Near_min~Near_max (i.e., the near-range maximum value Near_max minus the near-range minimum value Near_min) and the far-range range Far_min~Far_max (i.e., the far-range maximum value Far_max minus the far-range minimum value Far_min). In other words, when the near-range (Near_min to Near_max) is greater than (longer than) the far-range (Far_min to Far_max) (longer than) the luminescence duration equivalent length Tp_period is equal to the near-range (Near_min to Near_max). On the other hand, if the far-range range Far_min to Far_max is greater (longer) than the near-range range Near_min to Near_max, the equivalent length of the emission period Tp_period becomes equal to the far-range range Far_min to Far_max. The light emission period Tp [nsec] of the light source unit 11 is set by using the following equation 6.

[0070] Tp=Tp_period / C+h...Formula 6

[0071] In Equation 6 C: Speed ​​of light (3×10 11 [mm / s]) h: Pulse gradient correction (here, it is set to 2 [nsec]). In reality, the pulses indicating the light emission period Tp of the light source unit 11 in Figure 9(A), etc., do not rise vertically, but rather rise at an angle with respect to the vertical axis in Figure 9(A), etc. Therefore, it is necessary to perform correction using a pulse tilt correction component h.

[0072] In the examples shown in Figures 8 and 9 above, the minimum near-range value Near_min is set to 300 [mm], the maximum near-range value Near_max is set to 1000 [mm], the near-range range Near_min to Near_max is set to 700 [mm] (= 1000 [mm] - 300 [mm]), the minimum far-range value Far_min is set to 1000 [mm], the maximum far-range value Far_max is set to 4000 [mm], and the far-range range Far_min to Far_max is set to 3000 [mm] (= 4000 [mm] - 1000 [mm]), therefore the light source unit 11 The light period Tp becomes 12 [nsec], the exposure period GS1 corresponding to the first exposure timing (S0' exposure timing), second exposure timing (S1' exposure timing), and third exposure timing (BG exposure timing) of the second image sensor 12E2 becomes 14.35 [nsec], and the exposure period GS2 corresponding to the first exposure timing (S0' exposure timing), second exposure timing (S1' exposure timing), and third exposure timing (BG exposure timing) of the third image sensor 12E3 becomes 22.07 [nsec].

[0073] Thus, the light emission period Tp [nsec] of the light source unit 11, the exposure period GS1 [nsec] of the second image sensor 12E2, and the exposure period GS2 [nsec] of the third image sensor 12E3 are all very short. Therefore, normally, the exposure of the second image sensor 12E2 at the first exposure timing (S0' exposure timing) is not performed only once. Instead, the light emission from the light source unit 11 and the exposure of the second image sensor 12E2 at the first exposure timing (S0' exposure timing) are repeated multiple times so that the relationship between the light emission period Tp of the light source unit 11 and the exposure period GS1 corresponding to the first exposure timing (S0' exposure timing) of the second image sensor 12E2 (see Figures 8(A) and 8(D)) remains the same each time, and charge is accumulated in the first part 12E21 of the second image sensor 12E2. Similarly, the relationship between the light emission period Tp of the light source unit 11 and the exposure period GS1 corresponding to the second exposure timing (S1' exposure timing) of the second image sensor 12E2 (see Figures 8(B) and 8(E)) is repeated multiple times so that the light emission of the light source unit 11 and the exposure at the second exposure timing (S1' exposure timing) of the second image sensor 12E2 are the same each time, and charge is accumulated in the second part 12E22 of the second image sensor 12E2. Furthermore, the exposure at the third exposure timing (BG exposure timing) of the second image sensor 12E2 is repeated multiple times so that the interval of the exposure period GS1 (see Figures 8(C) and 8(F)) corresponding to the third exposure timing (BG exposure timing) of the second image sensor 12E2 is the same each time, and charge is accumulated in the third part 12E23 of the second image sensor 12E2.

[0074] Furthermore, the exposure of the third image sensor 12E3 at the first exposure timing (S0' exposure timing) is repeated multiple times so that the relationship between the light emission period Tp of the light source unit 11 and the exposure period GS2 corresponding to the first exposure timing (S0' exposure timing) of the third image sensor 12E3 (see Figures 9(A) and 9(D)) is the same each time, and charge is accumulated in the first part 12E31 of the third image sensor 12E3. Furthermore, the exposure at the second exposure timing (S1' exposure timing) of the third image sensor 12E3 is repeated multiple times so that the relationship between the light emission period Tp of the light source unit 11 and the exposure period GS2 corresponding to the second exposure timing (S1' exposure timing) of the third image sensor 12E3 (see Figures 9(B) and 9(E)) is the same each time, and charge is accumulated in the second part 12E32 of the third image sensor 12E3. Furthermore, the exposure of the third image sensor 12E3 at the third exposure timing (BG exposure timing) is repeated multiple times so that the interval of the exposure period GS2 (see Figures 9(C) and 9(F)) corresponding to the third exposure timing (BG exposure timing) of the third image sensor 12E3 is the same each time, and charge is accumulated in the third part 12E33 of the third image sensor 12E3. The number of repetitions in this repeated exposure is called the pulse count, and it is repeated several hundred to tens of thousands of times depending on the sensitivity of the second image sensor 12E2 and the third image sensor 12E3. Here, the pulse count is represented by N.

[0075] In the example shown in Figure 1, first, the light emission from the light source 11 and the exposure of the second image sensor 12E2 and the third image sensor 12E3, corresponding to the first exposure timing (S0' exposure timing) of the second image sensor 12E2 and the first exposure timing (S0' exposure timing) of the third image sensor 12E3, are repeated for a pulse count of N, and charge is accumulated in the first part 12E21 of the second image sensor 12E2, and charge is also accumulated in the first part 12E31 of the third image sensor 12E3. Next, the light emission from the light source 11 and the exposure of the second image sensor 12E2 and the third image sensor 12E3, corresponding to the second exposure timing (S1' exposure timing) of the second image sensor 12E2 and the second exposure timing (S1' exposure timing) of the third image sensor 12E3, are repeated for a pulse count of N, and charge is accumulated in the second part 12E22 of the second image sensor 12E2, and charge is also accumulated in the second part 12E32 of the third image sensor 12E3. Next, exposure of the second image sensor 12E2 and the third image sensor 12E3 corresponding to the third exposure timing (BG exposure timing) of the second image sensor 12E2 and the third exposure timing (BG exposure timing) of the third image sensor 12E3 is repeated for a pulse count number N, and charge is accumulated in the third part 12E23 of the second image sensor 12E2, and charge is accumulated in the third part 12E33 of the third image sensor 12E3. In principle, the charge accumulated in the first part 12E21 of the second image sensor 12E2 and the first part 12E31 of the third image sensor 12E3 will reach saturation first. Therefore, the pulse count number N is set (adjusted) based on the number of repetitions of light emission from the light source 11 when the charge accumulated in the first part 12E21 of the second image sensor 12E2 and the first part 12E31 of the third image sensor 12E3 reaches saturation.

[0076] Figure 10 shows an example of the sequence of light emission from the light source unit 11 and exposure of the second image sensor 12E2 and the third image sensor 12E3 when the pulse count N is set to 500 times. In the example shown in Figure 10, first, the light source unit 11 emits light and the second image sensor 12E2 and the third image sensor 12E3 are exposed for a pulse count of N (500 times) corresponding to the first exposure timing (S0' exposure timing) of the second image sensor 12E2 and the first exposure timing (S0' exposure timing) of the third image sensor 12E3. Next, the light source unit 11 emits light and the second image sensor 12E2 and the third image sensor 12E3 are exposed for a pulse count of N (500 times) corresponding to the second exposure timing (S1' exposure timing) of the second image sensor 12E2 and the second exposure timing (S1' exposure timing) of the third image sensor 12E3. Next, exposure of the second image sensor 12E2 and the third image sensor 12E3, corresponding to the third exposure timing (BG exposure timing) of the second image sensor 12E2 and the third exposure timing (BG exposure timing) of the third image sensor 12E3, is repeated for a pulse count of N (500 times).

[0077] In the example shown in Figure 1, the light source unit emission control pulse setting unit 13A2 of the light emission control unit 13A of the processing unit 13 generates a pulse to cause the light source unit 11 (infrared laser diode array) to emit light. However, in other examples, the pulse to cause the light source unit 11 (infrared laser diode array) to emit light may be generated inside the light source unit 11. In the example shown in Figure 1, the first image sensor drive control pulse setting unit 13B2 of the first image sensor control unit 13B of the processing unit 13 generates pulses for driving the first image sensor 12E1. However, in other examples, the pulses for driving the first image sensor 12E1 may be generated inside the first image sensor 12E1. In the example shown in Figure 1, the second image sensor drive control pulse setting unit 13C2 of the second image sensor control unit 13C of the processing unit 13 generates pulses for driving the second image sensor 12E2. However, in other examples, the pulses for driving the second image sensor 12E2 may be generated inside the second image sensor 12E2.

[0078] In the example shown in Figure 1, the measurement range determination unit 13I receives a signal indicating the charge accumulated in the first part 12E21 of the second image sensor 12E2 (a signal corresponding to the S0' signal output from the interface unit 13F1), a signal indicating the charge accumulated in the second part 12E22 of the second image sensor 12E2 (a signal corresponding to the S1' signal output from the interface unit 13F1), a signal indicating the charge accumulated in the third part 12E23 of the second image sensor 12E2 (a signal corresponding to the BG signal output from the interface unit 13F1), a signal indicating the charge accumulated in the first part 12E31 of the third image sensor 12E3 (a signal corresponding to the S0' signal output from the interface unit 13G1), and the third image sensor Based on a signal indicating the charge accumulated in the second part 12E32 of 12E3 (a signal corresponding to the S1' signal output from the interface unit 13G1), a signal indicating the charge accumulated in the third part 12E33 of the third image sensor 12E3 (a signal corresponding to the BG signal output from the interface unit 13G1), and the pre-set discrimination rules shown in Table 1, the 3D image imaging device 1 determines whether the distance between the 3D image imaging device 1 and the subject S measured by the 3D image imaging device 1 falls within the short-range range Near_min to Near_max, and whether the distance between the 3D image imaging device 1 and the subject S measured by the 3D image imaging device 1 falls within the long-range range Far_min to Far_max.

[0079] The distance data output switching unit 13J switches the output of distance data according to the determination result of the measurement range determination unit 13I, for example, by outputting the first distance data Z1 (=S1t1 / S0t1+Zoffset) calculated by the first distance data calculation unit 13F, or by outputting the second distance data Z2 (=S1t2 / S0t2+Zoffset2) calculated by the second distance data calculation unit 13G.

[0080] Table 1 shows an example of a discrimination rule used by the measurement range discrimination unit 13I of the processing unit 13 of the 3D image capturing device 1 of the first embodiment.

[0081] [Table 1]

[0082] In the example shown in Table 1, when the measurement range determination unit 13I determines that the distance between the 3D image capturing device 1 and the subject S, as measured by the 3D image capturing device 1, falls within the near-range range Near_min to Near_max, the measurement range determination unit 13I outputs a 3-bit signal of AREA[2:0] (specifically, a signal where AREA[2:0]=1) as the determination result, and the distance data output switching unit 13J outputs the first distance data Z1 (=S1t1 / S0t1+Zoffset) (>0) calculated by the first distance data calculation unit 13F as distance data S1wide / S0wide. If the measurement range determination unit 13I determines that the distance between the 3D image capturing device 1 and the subject S, as measured by the 3D image capturing device 1, falls within the long-distance range Far_min to Far_max, the measurement range determination unit 13I outputs a signal AREA[2:0]=3 as the determination result, and the distance data output switching unit 13J outputs the second distance data Z2 (=S1t2 / S0t2+Zoffset2)(>Z1) calculated by the second distance data calculation unit 13G as distance data S1wide / S0wide. If the distance between the 3D image capturing device 1 and the subject S is less than the near distance minimum value Near_min, the measurement range determination unit 13I outputs a signal AREA[2:0]=0 as the determination result, indicating that measurement is not possible, and the distance data output switching unit 13J outputs zero (minimum value) which is smaller than the first distance data Z1 as distance data S1wide / S0wide. If the distance between the 3D image capturing device 1 and the subject S is greater than or equal to the long-distance maximum value Far_max, the measurement range determination unit 13I outputs a signal AREA[2:0]=4 as the determination result, indicating that measurement is impossible, and the distance data output switching unit 13J outputs a predetermined maximum value greater than the second distance data Z2 as distance data S1wide / S0wide. For example, in the case of a 16-bit unsigned integer, the distance data output switching unit 13J outputs 65535 as the predetermined maximum value greater than the second distance data Z2. In other words, in the example shown in Table 1, the distance data output switching unit 13J outputs either the first distance data Z1, the second distance data Z2, zero (minimum value), or the maximum value as distance data S1wide / S0wide.

[0083] Figures 11A to 11C are diagrams that explain the basis for the discrimination rules shown in Table 1. Although not shown in Figures 11A to 11C, when the distance between the 3D image imaging device 1 and the subject S is less than the near-min value, the charge S0' accumulated in the first part 12E21 of the second image sensor 12E2 at the first exposure timing (S0' exposure timing) of the second image sensor 12E2, and the charge accumulated in the third part 12E23 of the second image sensor 12E2 at the third exposure timing (BG exposure timing) of the second image sensor 12E2 The charge S0 (=S0'-BG), which represents the difference with the charge BG, is not zero, and the charge S1 (=S1'-BG), which represents the difference between the charge S1' accumulated in the second part 12E22 of the second image sensor 12E2 at the second exposure timing (S1' exposure timing) of the second image sensor 12E2 and the charge BG accumulated in the third part 12E23 of the second image sensor 12E2 at the third exposure timing (BG exposure timing) of the second image sensor 12E2, is zero. Therefore, in the example shown in Table 1, the measurement range determination unit 13I can determine whether the distance between the 3D image acquisition device 1 and the subject S is less than the near distance minimum value Near_min (whether AREA[2:0]=0) based on the charges S0 (=S0'-BG) and S1 (=S1'-BG) of the second image sensor 12E2. In the second example of the three-dimensional image capturing device 1 of the first embodiment described above (i.e., an example in which the second image sensor 12E2 does not have a third portion 12E23), the measurement range determination unit 13I determines whether the distance between the three-dimensional image capturing device 1 and the subject S is less than the near distance minimum value Near_min (whether AREA[2:0]=0) based on the charges S0' and S1' of the second image sensor 12E2.

[0084] As shown in Figure 11A, when the distance between the 3D image imaging device 1 and the subject S, as measured by the 3D image imaging device 1, falls within the short-range range Near_min to Near_max, the charge S0' accumulated in the first part 12E31 of the third image sensor 12E3 at the first exposure timing (S0' exposure timing) of the third image sensor 12E3, and the charge S0' accumulated in the third part 12E31 of the third image sensor 12E3 at the third exposure timing (BG exposure timing) of the third image sensor 12E3 The charge S0 (=S0'-BG), which represents the difference between the charge S1' accumulated in E33 and the charge BG accumulated in E33, is not zero, and the charge S1 (=S1'-BG), which represents the difference between the charge S1' accumulated in the second part 12E32 of the third image sensor 12E3 at the second exposure timing (S1' exposure timing) of the third image sensor 12E3 and the charge BG accumulated in the third part 12E33 of the third image sensor 12E3 at the third exposure timing (BG exposure timing) of the third image sensor 12E3, becomes zero. Furthermore, as shown in Figure 11A, when the distance between the 3D image imaging device 1 and the subject S, as measured by the 3D image imaging device 1, falls within the short-range range Near_min to Near_max, the charge S0' accumulated in the first part 12E21 of the second image sensor 12E2 at the first exposure timing (S0' exposure timing) of the second image sensor 12E2, and the charge S0' accumulated in the third part of the second image sensor 12E2 at the third exposure timing (BG exposure timing) of the second image sensor 12E2 The charge S0 (=S0'-BG), which represents the difference between the charge BG accumulated in minute 12E23 and the charge S0, is greater than or equal to the charge S1 (=S1'-BG), which represents the difference between the charge S1' accumulated in the second part 12E22 of the second image sensor 12E2 at the second exposure timing (S1' exposure timing) of the second image sensor 12E2 and the charge BG accumulated in the third part 12E23 of the second image sensor 12E2 at the third exposure timing (BG exposure timing) of the second image sensor 12E2 (S0≧S1). Furthermore, if the distance between the 3D image capturing device 1 and the subject S, as measured by the 3D image capturing device 1, falls within the short-range range of Near_min to Near_max, the distance Z between the subject S and the 3D image capturing device 1, obtained by applying the data (charges S0, S1) obtained by the second image sensor 12E2 to the above-mentioned equation 1, will be less than or equal to the long-range minimum value Far_min. By utilizing these properties, the measurement range determination unit 13I can determine whether the distance between the 3D image capturing device 1 and the subject S falls within the short-range range Near_min to Near_max (whether AREA[2:0]=1 or not). In the second example of the 3D image capturing device 1 of the first embodiment described above (i.e., an example in which the second image sensor 12E2 does not have a third part 12E23), the measurement range determination unit 13I determines whether the distance between the 3D image capturing device 1 and the subject S is included in the near-range (Near_min~Near_max) by applying the data (charges S0', S1') obtained by the second image sensor 12E2 to the above-described equation 1.

[0085] As shown in Figure 11B, when the distance between the 3D image imaging device 1 and the subject S, as measured by the 3D image imaging device 1, falls within the far-distance range Far_min to Far_max, the charge S0' accumulated in the first part 12E21 of the second image sensor 12E2 at the first exposure timing (S0' exposure timing) of the second image sensor 12E2, and the charge S0' accumulated in the third part 12E2 of the second image sensor 12E2 at the third exposure timing (BG exposure timing) of the second image sensor 12E2 The charge S0 (=S0'-BG), which represents the difference between the charge S1' accumulated in part BG at part 3, is not zero, and the charge S1 (=S1'-BG), which represents the difference between the charge S1' accumulated in part 12E22 of the second image sensor 12E2 at the second exposure timing (S1' exposure timing) of the second image sensor 12E2 and the charge BG accumulated in part 12E23 of the second image sensor 12E2 at the third exposure timing (BG exposure timing) of the second image sensor 12E2, is not zero. Furthermore, as shown in Figure 11B, when the distance between the 3D image imaging device 1 and the subject S, as measured by the 3D image imaging device 1, falls within the far-distance range Far_min~Far_max, the charge S0' accumulated in the first part 12E31 of the third image sensor 12E3 at the first exposure timing (S0' exposure timing) of the third image sensor 12E3, and the third part of the third image sensor 12E3 at the third exposure timing (BG exposure timing) of the third image sensor 12E3 The charge S0 (=S0'-BG), which represents the difference between the charge BG accumulated in 12E33 and the charge S1' accumulated in the second part 12E32 of the third image sensor 12E3 at the second exposure timing (S1' exposure timing) of the third image sensor 12E3, is greater than or equal to the charge S1 (=S1'-BG), which represents the difference between the charge S1' accumulated in the second part 12E32 of the third image sensor 12E3 and the charge BG accumulated in the third part 12E33 of the third image sensor 12E3 at the third exposure timing (BG exposure timing) of the third image sensor 12E3 (S0≧S1). Furthermore, if the distance between the 3D image capturing device 1 and the subject S, as measured by the 3D image capturing device 1, falls within the far-distance range Far_min to Far_max, the distance Z between the subject S and the 3D image capturing device 1, obtained by applying the data (charges S0, S1) obtained by the third image sensor 12E3 to the above-mentioned equation 5, becomes greater than the far-distance minimum value Far_min. By utilizing these properties, the measurement range determination unit 13I can determine whether the distance between the 3D image capturing device 1 and the subject S is included in the long-distance range Far_min to Far_max (whether AREA[2:0]=3). In the second example of the 3D image capturing device 1 of the first embodiment described above (i.e., an example in which the third image sensor 12E3 does not have a third part 12E33), the measurement range determination unit 13I determines whether the distance between the 3D image capturing device 1 and the subject S is included in the long-distance range Far_min~Far_max (whether AREA[2:0]=3) by applying the data obtained by the third image sensor 12E3 (charges S0', S1') to the above-described equation 5.

[0086] As shown in Figure 11C, when the distance between the 3D image imaging device 1 and the subject S, as measured by the 3D image imaging device 1, is greater than or equal to the maximum long-distance value Far_max, the charge S0' accumulated in the first part 12E31 of the third image sensor 12E3 at the first exposure timing (S0' exposure timing) of the third image sensor 12E3, and the charge S0' accumulated in the third part 12E33 of the third image sensor 12E3 at the third exposure timing (BG exposure timing) of the third image sensor 12E3 The charge S0 (=S0'-BG), which represents the difference between the accumulated charge BG and the charge S1' accumulated in the second part 12E32 of the third image sensor 12E3 at the second exposure timing (S1' exposure timing) of the third image sensor 12E3, becomes smaller than the charge S1 (=S1'-BG), which represents the difference between the charge S1' accumulated in the second part 12E32 of the third image sensor 12E3 and the charge BG accumulated in the third part 12E33 of the third image sensor 12E3 at the third exposure timing (BG exposure timing) of the third image sensor 12E3 (S1>S0). By using this property, the measurement range determination unit 13I can determine whether the distance between the 3D image capturing device 1 and the subject S is greater than or equal to the long-distance maximum value Far_max (whether or not AREA[2:0]=4). In the second example of the 3D image capturing device 1 of the first embodiment described above (i.e., an example in which the third image sensor 12E3 does not have a third part 12E33), the measurement range determination unit 13I determines whether the distance between the 3D image capturing device 1 and the subject S is greater than or equal to the maximum far distance Far_max (whether or not AREA[2:0]=4) by using the property that when the distance between the 3D image capturing device 1 and the subject S measured by the 3D image capturing device 1 is greater than or equal to the maximum far distance Far_max, the charge S1' > charge S0'.

[0087] In other words, in the example shown in Figure 1, the measurement range determination unit 13I outputs a signal AREA[2:0]=0 as a determination result when it determines that the distance between the 3D image capture device 1 and the subject S is less than the minimum near distance value Near_min, outputs a signal AREA[2:0]=1 as a determination result when it determines that the distance between the 3D image capture device 1 and the subject S is included in the near distance range Near_min to Near_max, outputs a signal AREA[2:0]=3 as a determination result when it determines that the distance between the 3D image capture device 1 and the subject S is included in the far distance range Far_min to Far_max, outputs a signal AREA[2:0]=4 as a determination result when it determines that the distance between the 3D image capture device 1 and the subject S is greater than or equal to the maximum far distance value Far_max, and outputs a signal AREA[2:0]=0 as a determination result when none of the above apply.

[0088] In some cases, the signals output from the second image sensor 12E2 and the third image sensor 12E3 may contain a large amount of noise. In view of this, in other examples of the 3D image capturing device 1 of the first embodiment, the measurement range determination unit 13I may perform the processing described later. In this example, an adjustable threshold Zth is set. Furthermore, the measurement range determination unit 13I outputs a determination result of AREA[2:0]=0 when the level S0 (=S0'-BG) of the infrared emission reflection at the S0' exposure timing of the second image sensor 12E2 is greater than the threshold Zth, and the level S1 (=S1'-BG) of the infrared emission reflection at the S1' exposure timing of the second image sensor 12E2 is less than or equal to the threshold Zth. Furthermore, the measurement range determination unit 13I outputs a determination result of AREA[2:0]=1 when the level S0 of the infrared emission reflection of the third image sensor 12E3 at the S0' exposure timing is greater than the threshold Zth, the level S1 of the infrared emission reflection of the third image sensor 12E3 at the S1' exposure timing is less than or equal to the threshold Zth, the level S0 of the infrared emission reflection of the second image sensor 12E2 at the S0' exposure timing is greater than or equal to the level S1 of the infrared emission reflection of the second image sensor 12E2 at the S1' exposure timing (S0≧S1), and the distance Z between the subject S and the 3D image capturing device 1, obtained by applying the levels S0 of the infrared emission reflection of the second image sensor 12E2 at the S0' exposure timing and the level S1 of the infrared emission reflection of the second image sensor 12E2 at the S1' exposure timing to the above-described equation 1, is less than or equal to the minimum long-distance value Far_min. The measurement range determination unit 13I outputs the determination result AREA[2:0]=3 when the level of infrared emission reflection S0 at the S0' exposure timing of the third image sensor 12E3 and the level of infrared emission reflection S1 at the S1' exposure timing are greater than the threshold Zth, the level of infrared emission reflection S0 at the S0' exposure timing of the third image sensor 12E3 is greater than or equal to the level of infrared emission reflection S1 at the S1' exposure timing of the third image sensor 12E3 (S0≧S1), and the distance Z between the subject S and the 3D image imaging device 1, obtained by applying the levels of infrared emission reflection S0 at the S0' exposure timing of the third image sensor 12E3 and the level of infrared emission reflection S1 at the S1' exposure timing of the third image sensor 12E3 to the above-described equation 5, is greater than the minimum long-distance value Far_min. The measurement range determination unit 13I outputs a determination result of AREA[2:0]=4 when the level S0 of the infrared emission reflection of the third image sensor 12E3 at the S0' exposure timing is smaller than the level S1 of the infrared emission reflection of the third image sensor 12E3 at the S1' exposure timing (S1>S0). The measurement range discrimination unit 13I outputs a discrimination result of AREA[2:0]=0 when none of the conditions apply.

[0089] In the example shown in Figure 1, the distance data S1wide / S0wide output by the distance data output switching unit 13J is temporarily stored in the second buffer memory 13K2 of the storage unit 13K. Furthermore, in the example shown in Figure 1, the IR data output switching unit 13L switches between outputting IR data indicating an IR image captured by the second image sensor 12E2 or outputting IR data indicating an IR image captured by the third image sensor 12E3, depending on the determination result of the measurement range determination unit 13I. The IR data processing unit 13M processes the IR data input from the IR data output switching unit 13L. The IR data processing unit 13M includes a black level unit 13M1 and a gamma unit 13M2. The black level unit 13M1 performs black level correction on the IR image indicated by the IR data input from the IR data output switching unit 13L. The gamma unit 13M2 performs gamma correction on the IR image indicated by the IR data input from the IR data output switching unit 13L. The IR data processed by the IR data processing unit 13M is stored in the third buffer memory 13K3 of the storage unit 13K. The output mixing unit 13N mixes the output of data (YUV4:2:2 data) from the first buffer memory 13K1 of the storage unit 13K, the output of distance data S1wide / S0wide from the second buffer memory 13K2 of the storage unit 13K, and the output of IR data from the third buffer memory 13K3 of the storage unit 13K. The output interface unit 13P can output the data mixed by the output mixing unit 13N to an external device (not shown) of the 3D image imaging device 1.

[0090] In other words, in the example shown in Figure 1, the IR data output switching unit 13L can select the S0t1 signal obtained from the second image sensor 12E2 and the S0t2 signal obtained from the third image sensor 12E3 according to the AREA[2:0] signal and the IRsel[1:0] signal output from the control unit 13H. Furthermore, the S0t1 signal is also the infrared light image signal from the second image sensor 12E2, and the S0t2 signal is also the infrared light image signal from the third image sensor 12E3. Therefore, when the user of the 3D image imaging device 1 acquires distance data indicating the distance between the 3D image imaging device 1 and the subject S (either the first distance data Z1, the second distance data Z2, zero (minimum value), or the maximum value mentioned above), they can view how the IR image, generated by the infrared light actually irradiated from the light source unit 11, is acquired by the second image sensor 12E2 and the third image sensor 12E3. The IRsel[1:0] signal can be set by the user of the 3D image acquisition device 1 to the control unit 13H, and the breakdown of the IRsel[1:0] signal is as follows, for example.

[0091] When IRsel[1:0]=0, the IR data output switching unit 13L outputs as follows in accordance with the AREA[2:0] signal. When AREA[2:0]=0, the S0t1 signal is output. When AREA[2:0]=1, the S0t1 signal is output. When AREA[2:0]=3, the S0t2 signal is output. When AREA[2:0]=4, the S0t2 signal is output.

[0092] When IRsel[1:0]=1 The IR data output switching unit 13L always outputs the S0t1 signal.

[0093] When IRsel[1:0]=2 The IR data output switching unit 13L always outputs the S0t2 signal.

[0094] In the example shown in Figure 1, the output signal from the IR data output switching unit 13L is treated as a monochrome IR video signal, the black level is adjusted by the black level unit 13M1 of the IR data processing unit 13M, video gamma is applied by the gamma unit 13M2 of the IR data processing unit 13M, and the signal is temporarily stored in the third buffer memory 13K3 of the storage unit 13K.

[0095] Figure 12 shows an example of the signal switching sequence by the output mixing unit 13N. Signals temporarily stored in the first buffer memory 13K1 (YUV4:2:2 buffer memory), the second buffer memory 13K2 (DepthDATA buffer memory), and the third buffer memory 13K3 (IR_DATA buffer memory) of the memory unit 13K are switched in the output mixing unit 13N (OUTPUTMIX block) in the order shown in Figure 12, for example, and output sequentially frame by frame via the subsequent output interface unit 13P (USB3.0 I / F). When data is output from the output interface unit 13P (USB3.0 I / F), the data can be read sequentially each frame via a personal computer, for example, to perform image display or point cloud processing, and may be applied to 3D display, etc.

[0096] In other words, in the example shown in Figure 1, the RGB camera processing unit 13E generates an RGB image from the signal output by the first image sensor 12E1, while the second image sensor 12E2 and the third image sensor 12E3 function as infrared image sensors. Furthermore, in the example shown in Figure 1, the IR data output switching unit 13L outputs a near-range infrared image captured by the second image sensor 12E2 when the measurement range determination unit 13I determines that the distance between the 3D image imaging device 1 and the subject S, as measured by the 3D image imaging device 1, falls within the near-range (Near_min~Near_max). It also outputs a far-range infrared image captured by the third image sensor 12E3 when the measurement range determination unit 13I determines that the distance between the 3D image imaging device 1 and the subject S, as measured by the 3D image imaging device 1, falls within the far-range (Far_min~Far_max). The output mixing unit 13N synchronously outputs at least one of the first distance data Z1 (=S1t1 / S0t1+Zoffset) and the second distance data Z2 (=S1t2 / S0t2+Zoffset2) output from the distance data output switching unit 13J, the RGB video generated by the RGB camera processing unit 13E, and the IR video generated from the infrared image for the near range or the infrared image for the far range output from the IR data output switching unit 13L.

[0097] As described above, in the 3D image capturing device 1 of the first embodiment, both RGB images on the same optical axis and distance data output from the distance data output switching unit 13J (hereinafter, the distance data output from the distance data output switching unit 13J will be simply referred to as "distance data") can be obtained with a single lens 12A. Since there is no need to correct for differences in the optical axis due to parallax, both data can be obtained without any misalignment between the RGB images and the distance data images. In the image correction process after imaging, there is no need for correction processing to align the positions of the two images, and the 3D image capturing device 1 of the first embodiment can be easily applied to systems that obtain 3D information in moving videos and live shooting. Furthermore, in the 3D image acquisition device 1 of the first embodiment, by using two TOF sensors (second image sensor 12E2 and third image sensor 12E3), distance data for both near and far distances can be obtained simultaneously. This eliminates frame rate drops and image blur due to motion, allowing for the acquisition of distance information over a wider range without any unnaturalness in the image.

[0098] <Second Embodiment> A second embodiment of the 3D image acquisition device of the present invention will be described below. The 3D image capturing device 1 of the second embodiment is configured in the same way as the 3D image capturing device 1 of the first embodiment described above, except for the points described later. Therefore, the 3D image capturing device 1 of the second embodiment can achieve the same effects as the 3D image capturing device 1 of the first embodiment described above, except for the points described later.

[0099] In the three-dimensional image capturing device 1 of the first embodiment described above, there are no restrictions on the transmittance and reflectance of the half mirror 12D. In other words, in the three-dimensional image capturing device 1 of the first embodiment, a half mirror having any transmittance and reflectance can be used as the half mirror 12D. On the other hand, in the 3D image capturing device 1 of the second embodiment, the transmittance and reflectance of the half mirror 12D are limited for reasons that will be described later.

[0100] As in the 3D image capturing device 1 of the first embodiment described above, when infrared light is emitted from the light source unit 11 (infrared laser diode array), the light emission from the light source unit 11 and the exposure of the second image sensor 12E2 and the third image sensor 12E3 are repeated for a number of pulse counts according to the sensitivity of the second image sensor 12E2 and the third image sensor 12E3, thereby accumulating charge in the second image sensor 12E2 and the third image sensor 12E3. Generally, by using the amount of accumulated charge up to the point just before saturation, the bit width of the A / D converter of the second image sensor 12E2 and the third image sensor 12E3 can be fully utilized, and an improvement in the signal-to-noise ratio can be expected. However, as in the example above, a single light source 11 (infrared laser diode array) illuminates both the near and far ranges with infrared light, and the second image sensor 12E2 and the third image sensor 12E3 capture the reflected infrared light. Therefore, depending on the exposure period setting, the signal obtained from the infrared light captured by the third image sensor 12E3 for the far range will be dimmer (weaker) than the signal captured by the second image sensor 12E2 for the short range. This is because, under normal circumstances, light attenuates inversely proportional to the square of the distance, so the farther the subject S, the dimmer it becomes.

[0101] For example, when the 3D image acquisition device 1 is applied to the near-range (Near_min to Near_max) and far-range (Far_min to Far_max) areas, the light emission period Tp of the light source unit 11 is set based on the wider of the two ranges: the near-range (Near_min to Near_max) and the far-range (Far_min to Far_max). In other words, if the difference between the near-range maximum value Near_max and the near-range minimum value Near_min (Near_max - Near_min) is greater than the difference between the far-range maximum value Far_max and the far-range minimum value Far_min (Far_max - Far_min), then the equivalent length of the luminescence period Tp_period (=Near_max - Near_min) is equal to the near-range range Near_min ~ Near_max. On the other hand, if the difference between the near-range maximum value Near_max and the near-range minimum value Near_min (Near_max-Near_min) is smaller than the difference between the far-range maximum value Far_max and the far-range minimum value Far_min (Far_max-Far_min), then the equivalent length of the luminescence period Tp_period (=Far_max-Far_min) is equal to the far-range range Far_min~Far_max.

[0102] The light emission period Tp [nsec] of the light source unit 11 is expressed by the above-mentioned equation 6 (Tp = Tp_period / C + h). The exposure period GS1 of the second image sensor 12E2 is expressed by the above-mentioned equation 2 (GS1 = Tp + (Near_max - Near_min + 2 × Near_min × 0.01) / C). The exposure period GS2 of the third image sensor 12E3 is expressed by the above-mentioned equation 4 (GS2 = Tp + (Far_max - Far_min + 2 × Far_min × 0.01) / C). As described above, when the minimum near distance value Near_min is 300[mm], the maximum near distance value Near_max is 1000[mm], the near distance range Near_min~Near_max is 700[mm] (=1000[mm]-300[mm]), the minimum far distance value Far_min is 1000[mm], the maximum far distance value Far_max is 4000[mm], and the far distance range Far_min~Far_max is 3000[mm] (=4000[mm]-1000[mm]), the emission period Tp of the light source unit 11 becomes 12[nsec], the exposure period GS1 of the second image sensor 12E2 becomes 14.35[nsec], and the exposure period GS2 of the third image sensor 12E3 becomes 22.07[nsec].

[0103] The condition under which the charge of the second image sensor 12E2 is most saturated is at the first exposure timing (S0' exposure timing) of the second image sensor 12E2 when the distance between the 3D image acquisition device 1 and the subject S coincides with the near-min value. We assume that the charge of the second image sensor 12E2 used will saturate when the pulse count at that time is N. The condition under which the charge of the third image sensor 12E3 accumulates the most in the same pulse count N is when the distance between the 3D image acquisition device 1 and the subject S coincides with the long-distance minimum value Far_min. Although the exposure period GS2 (22.07 [nsec]) of the third image sensor 12E3 is longer than the exposure period GS1 (14.35 [nsec]) of the second image sensor 12E2, the attenuation of the reflected infrared light with respect to distance is (Near_min × 2) 2 / (Far_min×2) 2 The ratio is larger for the third image sensor 12E3. When multiplied by the exposure period, the third image sensor 12E3 performs better than the second image sensor 12E2 in terms of GS2 / GS1 × (Near_min × 2). 2 / (Far_min×2) 2 The doubling attenuation is significant.

[0104] Specifically, the exposure period GS2 of the third image sensor 12E3 is expressed by the above-mentioned equation 4 (GS2 = Tp + (Far_max - Far_min + 2 × Far_min × 0.01) / C). As described above, when the minimum near-range value Near_min is 300[mm], the maximum near-range value Near_max is 1000[mm], the near-range range Near_min~Near_max is 700[mm] (=1000[mm]-300[mm]), the minimum far-range value Far_min is 1000[mm], the maximum far-range value Far_max is 4000[mm], the far-range range Far_min~Far_max is 3000[mm] (=4000[mm]-1000[mm]), and the pulse count N=500, the attenuation of the third image sensor 12E3 and the attenuation of the second image sensor 12E2 are GS2 / GS1×(300×2) 2 / (1000×2) 2The signal is attenuated by a factor of 0.14, and the ratio becomes approximately 7:1 = 0.875:0.125. In view of this, in the first example of the 3D image capturing device 1 of the second embodiment, the 3-chip image sensor unit 12 is configured as shown in Figure 13.

[0105] Figure 13 is an enlarged view showing a part of the three-chip image sensor section 12 of the first example of the three-dimensional image capturing device 1 of the second embodiment. In the example shown in Figure 13, similar to the example shown in Figure 3, the second image sensor 12E2 receives infrared light transmitted by the half mirror 12D. In the example shown in Figure 3, there are no limitations on the transmittance and reflectance of the half mirror 12D. That is, in the example shown in Figure 3, the transmittance of the half mirror 12D is, for example, 50%. In contrast, in the example shown in Figure 13 (the first example of the 3D image capturing device 1 of the second embodiment), the transmittance of the half mirror 12D is less than 50%, specifically 12.5%.

[0106] As described above, when the same amount of infrared light is incident on both the second image sensor 12E2 and the third image sensor 12E3, the third image sensor 12E3, which is used to measure the long-distance range, becomes darker and less likely to saturate. As a result, the signal-to-noise ratio of the third image sensor 12E3 generally deteriorates, and the accuracy of distance measurement also deteriorates. In this state, if you switch between near-range and far-range measurements within a single frame, such as in the example where the transmittance of the half-mirror 12D is 50%, and then, for example, create point cloud data from the output of the 3D image acquisition device 1 and combine it with an RGB image to display a 3D image, differences in signal-to-noise ratio and distance measurement accuracy will appear, resulting in an unnatural appearance. To solve the above problem, instead of spectrally separating the infrared light incident on the three-chip image sensor unit 12 at a 1:1 ratio by the half mirror 12D, such that the transmittance of the half mirror 12D is 50%, in the example shown in Figure 13, it is spectrally separated at a ratio of, for example, 7:1. In other words, infrared light transmitted with a transmittance of 12.5% ​​by the half mirror 12D is incident on the second image sensor 12E2 in the short-range measurement range, and infrared light reflected with a reflectance of 87.5% by the half mirror 12D is incident on the third image sensor 12E3. In this way, the optical system can pre-correct for the difference in exposure between the second image sensor 12E2 and the third image sensor 12E3 due to distance and exposure period. In the example shown in Figure 13, infrared light transmitted through the half-mirror 12D with a transmittance of 12.5% ​​is incident on the second image sensor 12E2. By setting the pulse count N to, for example, 4000 times (8 times = 100% / 12.5%), both the near-field second image sensor 12E2 and the far-field third image sensor 12E3 can be used effectively up to their saturation limits, thereby increasing the signal-to-noise ratio of both the second image sensor 12E2 and the third image sensor 12E3. Even when switching between displaying near-field and far-field images on a single screen, as in the example described above, the perceived incongruity caused by the difference in the signal-to-noise ratio of the second image sensor 12E2 and the third image sensor 12E3 can be suppressed.

[0107] In the second example of the 3D image capturing device 1 of the second embodiment, unlike the first example of the 3D image capturing device 1 of the second embodiment shown in Figure 13, the third image sensor 12E3 for the long-range receives infrared light transmitted by the half mirror 12D. Furthermore, the transmittance of the half mirror 12D is set to a value greater than 50% (for example, 87.5%).

[0108] <Third Embodiment> A third embodiment of the three-dimensional image capturing device of the present invention will be described below. The 3D image capturing apparatus 1 of the third embodiment is configured in the same way as the 3D image capturing apparatus 1 of the first embodiment described above, except for the points described later. Therefore, the 3D image capturing apparatus 1 of the third embodiment can achieve the same effects as the 3D image capturing apparatus 1 of the first embodiment described above, except for the points described later.

[0109] As described above, in the 3D image imaging device 1 of the first embodiment, a common pulse count N is used for the exposure period GS1 of the second image sensor 12E2 for short distances and the exposure period GS2 of the third image sensor 12E3 for long distances. On the other hand, in the 3D image imaging device 1 of the third embodiment, the same effect as the 3D image imaging device 1 of the second embodiment can be achieved by making the pulse count during the exposure period GS2 of the third image sensor 12E3 for long distances greater than the pulse count during the exposure period GS1 of the second image sensor 12E2 for short distances.

[0110] Figure 14 is a diagram illustrating the pulse count of a first example of the 3D image capturing device 1 according to the third embodiment. In the example shown in Figure 14, the ratio of the pulse count during exposure period GS1 of the second image sensor 12E2 for short distances (500 pulses) to the pulse count during exposure period GS2 of the third image sensor 12E3 for long distances (3500 pulses) is set to 1:7.

[0111] In detail, at the first exposure timing (S0' exposure timing) of the second image sensor 12E2 and the third image sensor 12E3, the light source unit 11 emits light 3500 times. Also, synchronized with the 3500 light emission by the light source unit 11, the third image sensor 12E3 undergoes 3500 exposures at its first exposure timing (S0' exposure timing). Furthermore, synchronized with the first 500 exposures of the third image sensor 12E3 at its first exposure timing (S0' exposure timing), the second image sensor 12E2 undergoes 500 exposures at its first exposure timing (S0' exposure timing). Furthermore, at the second exposure timing (S1' exposure timing) of the second image sensor 12E2 and the third image sensor 12E3, the light source unit 11 emits light 3500 times. Also, synchronized with the 3500 light emission by the light source unit 11, the third image sensor 12E3 undergoes 3500 exposures at its second exposure timing (S1' exposure timing). Moreover, synchronized with the first 500 exposures of the 3500 exposures at the second exposure timing (S1' exposure timing) of the third image sensor 12E3, the second image sensor 12E2 undergoes 500 exposures at its second exposure timing (S1' exposure timing). Furthermore, at the third exposure timing (BG exposure timing) of the second image sensor 12E2 and the third image sensor 12E3, 3500 exposures are performed for the third exposure timing (BG exposure timing) of the third image sensor 12E3. In addition, 500 exposures are performed for the third exposure timing (BG exposure timing) of the second image sensor 12E2 in synchronization with the first 500 exposures of the 3500 exposures for the third exposure timing (BG exposure timing) of the third image sensor 12E3.

[0112] In the example shown in Figure 14 (the first example of the 3D image capturing device 1 of the third embodiment), the pulse count for the 500th exposure of the second image sensor 12E2 finishes first, and while the 501st to 3500th exposures of the third image sensor 12E3 are being performed, the second image sensor 12E2 is not exposed at all. As a result, the image blur of the second image sensor 12E2 for close-range shots and the image blur of the third image sensor 12E3 for long-range shots will differ for moving subjects, which may cause a sense of incongruity at the transition between close-range and long-range shots, or in the overall 3D image. Therefore, in the second example of the 3D image capturing device 1 of the third embodiment, the countermeasures described later are implemented.

[0113] Figure 15 is a diagram illustrating the pulse count of a second example of the 3D image capturing device 1 according to the third embodiment. In the example shown in Figure 15, the pulse count (500 times) for the exposure period GS1 of the exposure period GS1 of the second image sensor 12E2 for short distances is set by roughly equalizing the pulse count (3500 times) of the exposure period GS2 of the third image sensor 12E3 for long distances.

[0114] In detail, at the first exposure timing (S0' exposure timing) of the second image sensor 12E2 and the third image sensor 12E3, the light source unit 11 emits light 3500 times. In addition, synchronized with the 3500 light emission times by the light source unit 11, the third image sensor 12E3 undergoes 3500 exposures at its first exposure timing (S0' exposure timing). Furthermore, the first exposure of the second image sensor 12E2 at its first exposure timing (S0' exposure timing) is performed in synchronization with the first exposure of the third image sensor 12E3 at its first exposure timing (S0' exposure timing), then the second exposure of the second image sensor 12E2 at its first exposure timing (S0' exposure timing) is performed in synchronization with the eighth exposure of the third image sensor 12E3 at its first exposure timing (S0' exposure timing), then the third exposure of the second image sensor 12E2 at its first exposure timing (S0' exposure timing) is performed in synchronization with the fifteenth exposure of the third image sensor 12E3 at its first exposure timing (S0' exposure timing), and similarly, the nth exposure of the second image sensor 12E2 at its first exposure timing (S0' exposure timing) is performed in synchronization with the (7 × (n-1) + 1)th exposure of the third image sensor 12E3 at its first exposure timing (S0' exposure timing) (n = 4 to 500).

[0115] Although not shown in Figure 15, the light source unit 11 emits light 3500 times at the second exposure timing (S1' exposure timing) of the second image sensor 12E2 and the third image sensor 12E3. In addition, synchronized with the 3500 light emission by the light source unit 11, the third image sensor 12E3 undergoes 3500 exposures at its second exposure timing (S1' exposure timing). Furthermore, the first exposure of the second exposure timing (S1' exposure timing) of the second image sensor 12E2 is performed in synchronization with the first exposure of the third image sensor 12E3 at the second exposure timing (S1' exposure timing), then the second exposure of the second image sensor 12E2 at the second exposure timing (S1' exposure timing) is performed in synchronization with the eighth exposure of the second exposure timing (S1' exposure timing) of the third image sensor 12E3, then the third exposure of the second exposure timing (S1' exposure timing) of the second image sensor 12E2 is performed in synchronization with the fifteenth exposure of the second exposure timing (S1' exposure timing) of the third image sensor 12E3, and similarly, the nth exposure of the second exposure timing (S1' exposure timing) of the second image sensor 12E2 is performed in synchronization with the (7 × (n-1) + 1) exposure of the second exposure timing (S1' exposure timing) of the third image sensor 12E3 (n = 4 to 500).

[0116] Furthermore, although not shown in Figure 15, at the third exposure timing (BG exposure timing) of the second image sensor 12E2 and the third image sensor 12E3, 3500 exposures are performed for the third exposure timing (BG exposure timing) of the third image sensor 12E3. Furthermore, the first exposure of the third exposure timing (BG exposure timing) of the third image sensor 12E3 is synchronized with the first exposure of the third exposure timing (BG exposure timing) of the third image sensor 12E3, then the second exposure of the third exposure timing (BG exposure timing) of the second image sensor 12E2 is synchronized with the eighth exposure of the third exposure timing (BG exposure timing) of the third image sensor 12E3, then the third exposure of the third exposure timing (BG exposure timing) of the second image sensor 12E2 is synchronized with the fifteenth exposure of the third exposure timing (BG exposure timing) of the third image sensor 12E3, and similarly, the nth exposure of the third exposure timing (BG exposure timing) of the second image sensor 12E2 is synchronized with the (7 × (n-1) + 1)th exposure of the third exposure timing (BG exposure timing) of the third image sensor 12E3 (n = 4 to 500).

[0117] In the example shown in Figure 15, by generating one pulse count for exposure of the second image sensor 12E2 for close range every seven pulse counts for exposure of the third image sensor 12E3 for long distances, there is no significant difference between the start and end of the pulse count. As a result, the blur of the image in the near and far ranges relative to a moving subject appears the same, and the sense of incongruity caused by differences in image blur is suppressed in the 3D image obtained by combining the distance data and RGB data obtained from the 3D image capturing device 1. In the examples shown in Figures 14 and 15, the transmittance and reflectance of the half mirror 12D are 50%. However, even in cases where the transmittance and reflectance of the half mirror 12D are different, by making the pulse count different for near-range and far-range measurements, there is the advantage that the CPU (control unit 13H) software can adjust the optimal saturation amount when changing the measurement range of the second image sensor 12E2 and the third image sensor 12E3 depending on the application. The processing unit 13 is equipped with hardware resources such as a CPU and memory, and operates according to the software stored in memory. If at least a portion of the components constituting the 3D image capturing device 1 are implemented by software, the components implemented by software may be implemented in a 3D image capturing device 1 with a general configuration by launching software or a program that defines the operation of those components. A 3D image capturing device 1 with a general configuration may include (i) a data processing device having a processor such as a CPU, ROM, RAM, and a communication interface, (ii) an input device such as a camera and various sensors, (iii) an output device, and (iv) a storage device such as memory, HDD, or SSD (including an external storage device).

[0118] <Fourth Embodiment> A fourth embodiment of the 3D image capturing device of the present invention will be described below. The 3D image capturing device 1 of the fourth embodiment is configured in the same way as the 3D image capturing device 1 of the first embodiment described above, except for the points described later. Therefore, the 3D image capturing device 1 of the fourth embodiment can achieve the same effects as the 3D image capturing device 1 of the first embodiment described above, except for the points described later.

[0119] Figure 16 shows an example of a three-dimensional image capturing device 1 according to the fourth embodiment. In the example shown in Figure 16, similar to the example shown in Figure 1, the 3D image capturing device 1 includes a light source unit 11 that irradiates infrared light toward the subject S, a three-chip image sensor unit 12 into which reflected infrared light irradiated by the light source unit 11 and reflected by the subject S, and reflected visible light from the subject S are incident, and a processing unit 13 that controls the light source unit 11 and the three-chip image sensor unit 12 and processes the signals output from the three-chip image sensor unit 12.

[0120] In the example shown in Figure 16, similar to the example shown in Figure 3, the three-chip image sensor unit 12 includes a first prism 12B1 into which infrared light and visible light from the subject S are incident, a reflective dichroic film 12C that reflects the visible light from the infrared and visible light incident on the first prism 12B1 and transmits the infrared light, a first image sensor 12E1 that receives the visible light reflected by the reflective dichroic film 12C, and an incident infrared light transmitted by the reflective dichroic film 12C. The system includes a second prism 12B2, a half-mirror 12D that reflects a portion of the infrared light incident on the second prism 12B2 and transmits the remainder of the infrared light incident on the second prism 12B2, a third prism 12B3 into which the infrared light transmitted by the half-mirror 12D is incident, a second image sensor 12E2 that receives the infrared light transmitted by the half-mirror 12D, and a third image sensor 12E3 that receives the infrared light reflected by the half-mirror 12D.

[0121] In the example shown in Figure 16, similar to the example shown in Figure 3, the second image sensor 12E2 is a short-range TOF sensor used to measure the distance range Near_min to Near_max, from the minimum value Near_min to the maximum value Near_max. The third image sensor 12E3 is a long-range TOF sensor used to measure the distance range Far_min to Far_max, which is further than the distance range Near_min to Near_max, from the minimum value Far_min to the maximum value Far_max.

[0122] In the example shown in Figure 16, unlike the example shown in Figure 1, a portion of the distance range Near_min to Near_max (from the short-range minimum value Near_min to the short-range maximum value Near_max) and a portion of the distance range Far_min to Far_max (from the far-range minimum value Far_min to the far-range maximum value Far_max) overlap to form an overlapping measurement range Far_min to Near_max. The near-range measurement range, Near_min to Far_min, is obtained by subtracting the overlap measurement range, Far_min to Near_max, from the distance range, Near_min to Near_max, which is the distance range from the nearest minimum value Near_min to the nearest maximum value Near_max. The long-distance measurement range Near_max to Far_max is obtained by subtracting the pre-overlap measurement range Far_min to Near_max from the distance range Far_min to Far_max, which is the distance range from the minimum long-distance value Far_min to the maximum long-distance value Far_max.

[0123] In the example shown in Figure 16, similar to the example shown in Figure 6, the second image sensor 12E2 includes a first portion 12E21 that accumulates charge at the first exposure timing (S0' exposure timing) of the second image sensor 12E2, which is the exposure timing of the second image sensor 12E2 in which at least most of the emission period Tp of the light source unit 11 is included in the exposure period of the second image sensor 12E2; a second portion 12E22 that accumulates charge at the second exposure timing (S1' exposure timing) of the second image sensor 12E2, which is the exposure timing of the second image sensor 12E2 in which the exposure period of the second image sensor 12E2 starts after the end of the emission period Tp of the light source unit 11; and a third portion 12E23 that accumulates charge at the third exposure timing (BG exposure timing) of the second image sensor 12E2, which is the exposure timing of the second image sensor 12E2 in which the second image sensor 12E2 does not receive reflected infrared light that has been irradiated by the light source unit 11 and reflected by the subject S.

[0124] In the example shown in Figure 16, similar to the example shown in Figure 6, the third image sensor 12E3 includes a first portion 12E31 that accumulates charge at the first exposure timing (S0' exposure timing) of the third image sensor 12E3, which is an exposure timing of the third image sensor 12E3 where at least a portion of the emission period Tp of the light source unit 11 is included in the exposure period of the third image sensor 12E3; a second portion 12E32 that accumulates charge at the second exposure timing (S1' exposure timing) of the third image sensor 12E3, which is an exposure timing of the third image sensor 12E3 where the exposure period of the third image sensor 12E3 starts after the end of the emission period Tp of the light source unit 11; and a third portion 12E33 that accumulates charge at the third exposure timing (BG exposure timing) of the third image sensor 12E3, which is an exposure timing of the third image sensor 12E3 where the third image sensor 12E3 does not receive reflected infrared light that has been irradiated by the light source unit 11 and reflected by the subject S. In the example shown in Figure 16 (the first example of the 3D image capturing device 1 of the fourth embodiment), the second image sensor 12E2 includes a third portion 12E23, and the third image sensor 12E3 includes a third portion 12E33. However, in other examples (the second example of the 3D image capturing device 1 of the fourth embodiment), the second image sensor 12E2 does not have a third portion 12E23, and the third image sensor 12E3 does not have a third portion 12E33.

[0125] In the example shown in Figure 16, similar to the example shown in Figure 1, the processing unit 13 includes a light emission control unit 13A that controls the light emission of the light source unit 11, a second image sensor control unit 13C that controls the exposure timing of the second image sensor 12E2 (S0' exposure timing, S1' exposure timing, BG exposure timing), a third image sensor control unit 13D that controls the exposure timing of the third image sensor 12E3 (S0' exposure timing, S1' exposure timing, BG exposure timing), and the charge accumulated in the first part 12E21 of the second image sensor 12E2, the charge accumulated in the second part 12E22 of the second image sensor 12E2, and the third part of the second image sensor 12E2 The system includes a first distance data calculation unit 13F that calculates first distance data Z1 (=S1t1 / S0t1+Zoffset) indicating the distance between the 3D image capturing device 1 and the subject S based on the charge accumulated in part 12E23, and a second distance data calculation unit 13G that calculates second distance data Z2 (=S1t2 / S0t2+Zoffset2) indicating the distance between the 3D image capturing device 1 and the subject S based on the charge accumulated in the first part 12E31 of the third image sensor 12E3, the charge accumulated in the second part 12E32 of the third image sensor 12E3, and the charge accumulated in the third part 12E33 of the third image sensor 12E3. In the second example of the 3D image capturing apparatus 1 of the fourth embodiment described above (i.e., an example in which the second image sensor 12E2 does not have a third portion 12E23), the second image sensor control unit 13C controls the exposure timing of the second image sensor 12E2 (S0' exposure timing, S1' exposure timing). Furthermore, in the second example of the 3D image capturing device 1 of the fourth embodiment described above (i.e., the example in which the third image sensor 12E3 does not have a third portion 12E33), the third image sensor control unit 13D controls the exposure timing of the third image sensor 12E3 (S0' exposure timing, S1' exposure timing).

[0126] In the example shown in Figure 16, similar to the example shown in Figure 1, the second image sensor control unit 13C starts the exposure period GS1 corresponding to the first exposure timing (S0' exposure timing) of the second image sensor 12E2 when an offset time Toffset (=2 × (Near_min - Near_min × 0.01) / C) based on the near-range minimum value Near_min has elapsed from the start timing of the emission period Tp of the light source unit 11, and starts the exposure period GS1 corresponding to the second exposure timing (S1' exposure timing) of the second image sensor 12E2 when an offset time Toffset (=2 × (Near_min - Near_min × 0.01) / C) based on the near-range minimum value Near_min has elapsed from the end timing of the emission period Tp of the light source unit 11.

[0127] In the example shown in Figure 16, similar to the example shown in Figure 1, the third image sensor control unit 13D, at the timing when an offset time Toffset2 (=2 × (Far_min - Far_min × 0.01) / C) based on the far-distance minimum value Far_min has elapsed from the start timing of the emission period Tp of the light source unit 11, sets the exposure period GS2 corresponding to the first exposure timing (S0' exposure timing) of the third image sensor 12E3, and within the same frame as the exposure period GS1 corresponding to the first exposure timing (S0' exposure timing) of the second image sensor 12E2. The set exposure period GS2 is started, and at the timing when the offset time Toffset2 (=2 × (Far_min - Far_min × 0.01) / C) based on the far distance minimum value Far_min has elapsed from the end timing of the emission period Tp of the light source unit 11, the exposure period GS2 corresponding to the second exposure timing (S1' exposure timing) of the third image sensor 12E3, which is set to be within the same frame as the exposure period GS1 corresponding to the second exposure timing (S1' exposure timing) of the second image sensor 12E2, is started.

[0128] In the example shown in Figure 16, unlike the example shown in Figure 1, the measurement range determination unit 13I determines the charge accumulated in the first part 12E21 of the second image sensor 12E2, the charge accumulated in the second part 12E22 of the second image sensor 12E2, the charge accumulated in the third part 12E23 of the second image sensor 12E2, the charge accumulated in the first part 12E31 of the third image sensor 12E3, and the charge accumulated in the second part 12E32 of the third image sensor 12E3. Based on a signal indicating the charge accumulated in the third part 12E33 of the third image sensor 12E3 and a preset discrimination rule shown in Table 2, the 3D image imaging device 1 determines whether the distance between the 3D image imaging device 1 and the subject S, as measured by the 3D image imaging device 1, falls within the short-range measurement range Near_min to Far_min, the overlap measurement range Far_min to Near_max, or the long-range measurement range Near_max to Far_max. In the second example of the 3D image capturing apparatus 1 of the fourth embodiment described above (that is, an example in which the second image sensor 12E2 does not have a third portion 12E23 and the third image sensor 12E3 does not have a third portion 12E33), the measurement range determination unit 13I determines the charge accumulated in the first portion 12E21 of the second image sensor 12E2, the charge accumulated in the second portion 12E22 of the second image sensor 12E2, the charge accumulated in the first portion 12E31 of the third image sensor 12E3, and the third Based on a signal indicating the charge accumulated in the second part 12E32 of the image element 12E3 and the preset discrimination rules shown in Table 2, the 3D image imaging device 1 determines whether the distance between the 3D image imaging device 1 and the subject S, as measured by the 3D image imaging device 1, falls within the near-range measurement range (Near_min to Far_min), the overlap measurement range (Far_min to Near_max), or the far-range measurement range (Near_max to Far_max).

[0129] [Table 2]

[0130] In the example shown in Figure 16, unlike the example shown in Figure 1, the distance data output switching unit 13J outputs the first distance data Z1 (=S1t1 / S0t1+Zoffset) calculated by the first distance data calculation unit 13F when the measurement range determination unit 13I determines that the distance between the 3D image imaging device 1 and the subject S, as measured by the 3D image imaging device 1, is included in the near-range measurement range Near_min~Far_min. When the measurement range determination unit 13I determines that the distance between the 3D image imaging device 1 and the subject S, as measured by the 3D image imaging device 1, is included in the far-range range Near_max~Far_max, the first distance data output switching unit 13J outputs the first distance data Z1 (=S1t1 / S0t1+Zoffset) calculated by the first distance data calculation unit 13F. The second distance data Z2 (=S1t2 / S0t2+Zoffset2) calculated by the distance data calculation unit 13G is output, and if the measurement range determination unit 13I determines that the distance between the 3D image capturing device 1 and the subject S, as measured by the 3D image capturing device 1, is included in the overlap measurement range Far_min~Near_max, then, as described later, the first distance data Z1 (=S1t1 / S0t1+Zoffset) calculated by the first distance data calculation unit 13F and the second distance data Z2 (=S1t2 / S0t2+Zoffset2) calculated by the second distance data calculation unit 13G are mixed and output.

[0131] In other words, the 3D image capturing device 1 of the fourth embodiment takes into consideration that the transition between the second image sensor 12E2 and the third image sensor 12E3 may not be smooth due to variations in their characteristics, noise, sensitivity, and shutter speed (exposure period). Therefore, in the example shown in Figure 16, the measurement range of the second image sensor 12E2 and the long-distance measurement range of the third image sensor 12E3 are slightly overlapped. In this overlapping portion, the mixing ratio of the outputs of the second image sensor 12E2 and the third image sensor 12E3 is changed according to the distance to gradually and smoothly transition between the distance result of the output of the second image sensor 12E2 in the close-distance range and the distance result of the output of the third image sensor 12E3 in the far-distance range. In the example shown in FIG. 16, the measurement range is set such that the near-distance maximum value Near_max > the far-distance minimum value Far_min, and for the distance range of the overlapping distance range, which is from the far-distance minimum value Far_min to the near-distance maximum value Near_max, distance data is acquired by both the second imaging device 12E2 and the third imaging device 12E3.

[0132] In the example shown in FIG. 17, which is an example of the three-dimensional video imaging device 1 of the fourth embodiment, the near-distance measurement range (near-distance minimum value Near_min to near-distance maximum value Near_max) is set to 300 mm to 1050 mm, the far-distance measurement range (far-distance minimum value Far_min to far-distance maximum value Far_max) is set to 950 mm to 4000 mm, the overlapping distance range (far-distance minimum value Far_min to near-distance maximum value Near_max) is set to 950 mm to 1050 mm, and for the overlapping distance range, distance data is acquired by both the second imaging device 12E2 and the third imaging device 12E3.

[0133] FIGS. 17A to 17D are diagrams showing the timing in an example of the three-dimensional video imaging device 1 of the fourth embodiment. In the example shown in FIGS. 17A to 17D, since Near_max - Near_min < Far_max - Far_min, based on Far_max - Far_min (= 3050 (mm)), the light emission period Tp of the light source unit 11 is obtained as follows. Tp = 3050 / C + h = 12.16 nsec C: speed of light (3 × 10 11 [mm / s]) h: pulse slope correction amount (here, it is set to 2 [nsec].)

[0134] The exposure period GS1 of the second imaging device 12E2 and the exposure period GS2 of the third imaging device 12E3 are obtained as follows. GS1 = Tp + (1050 mm - 300 mm + 2 × 300 mm × 0.01) / C = 14.69 nsec GS2 = Tp + (4000 mm - 950 mm + 2 × 950 mm × 0.01) / C = 22.4 nsec

[0135] In this way, the light emission period Tp of the light source unit 11, the exposure period GS1 of the second image sensor 12E2, and the exposure period GS2 of the third image sensor 12E3 are set. The second image sensor 12E2 is exposed at the timing of exposure period GS1 (shutter speed) to measure close distances, and the third image sensor 12E3 is exposed at the timing of exposure period GS2 (shutter speed) to measure long distances. The processing unit 13 shown in Figure 16 obtains the result of a calculation (S1t1 / S0t1) and a calculation (S1t2 / S0t2) that are proportional to the measured distance. The processing unit 13 adds the offset value Zoffset to the result of the calculation (S1t1 / S0t1) to obtain first distance data Z1 (=S1t1 / S0t1+Zoffset) that is proportional to the actual distance. Furthermore, the processing unit 13 obtains a second distance data Z2 (=S1t2 / S0t2+Zoffset2) that is proportional to the actual distance by adding the offset value Zoffset2 to the result of the calculation (S1t2 / S0t2). The control unit 13H sets a pre-calculated value (Toffset / 2×C) to the offset value Zoffset and inputs the set offset value Zoffset to the adder 13FX. Also, the control unit 13H sets a pre-calculated value (Toffset2 / 2×C) to the offset value Zoffset2 and inputs the set offset value Zoffset2 to the adder 13GX. In this way, the processing unit 13 in Figure 16 obtains a first distance data Z1 and a second distance data Z2 that are proportional to the measured distance. Next, the measurement range determination unit 13I determines whether the distance between the subject S corresponding to the pixel currently being measured and the 3D image capturing device 1 falls within the close distance measurement range, the far distance measurement range, or the overlap distance range. The distance data output switching unit 13J then switches whether to output the first distance data Z1, the second distance data Z2, or a mixture of the first distance data Z1 and the second distance data Z2, depending on the determination result of the measurement range determination unit 13I. As a result, distance data S1wide / S0wide, which is proportional to a wider range of distances than can be obtained with a single TOF sensor, is obtained. The measurement range determination unit 13I makes a determination based on the determination rules shown in Table 2 and outputs a 3-bit signal of AREA[2:0].

[0136] The measurement range determination unit 13I examines the following conditions in order to determine which of Table 2 the distance between the subject S being measured and the 3D image acquisition device 1 corresponds to. Figure 17A shows the light emission period Tp of the light source unit 11, the first exposure timing (S0' exposure timing), second exposure timing (S1' exposure timing), and third exposure timing (BG exposure timing) of the second image sensor 12E2, and the first exposure timing (S0' exposure timing), second exposure timing (S1' exposure timing), and third exposure timing (BG exposure timing) of the third image sensor 12E3 when AREA[2:0]=1. Figure 17B shows the light emission period Tp of the light source unit 11, the first exposure timing (S0' exposure timing), second exposure timing (S1' exposure timing), and third exposure timing (BG exposure timing) of the second image sensor 12E2, and the first exposure timing (S0' exposure timing), second exposure timing (S1' exposure timing), and third exposure timing (BG exposure timing) of the third image sensor 12E3 when AREA[2:0]=2. Figure 17C shows the light emission period Tp of the light source unit 11, the first exposure timing (S0' exposure timing), second exposure timing (S1' exposure timing), and third exposure timing (BG exposure timing) of the second image sensor 12E2, and the first exposure timing (S0' exposure timing), second exposure timing (S1' exposure timing), and third exposure timing (BG exposure timing) of the third image sensor 12E3 when AREA[2:0]=3. Figure 17D shows the light emission period Tp of the light source unit 11, the first exposure timing (S0' exposure timing), second exposure timing (S1' exposure timing), and third exposure timing (BG exposure timing) of the second image sensor 12E2, and the first exposure timing (S0' exposure timing), second exposure timing (S1' exposure timing), and third exposure timing (BG exposure timing) of the third image sensor 12E3 when AREA[2:0]=4. Refer to the timing diagrams shown in Figures 17A to 17D, and set the judgment conditions for AREA[2:0] shown in Table 2.

[0137] 1) When the level S0 of the infrared emission reflection of the second image sensor 12E2 at the S0' exposure timing is not zero, and the level S1 of the infrared emission reflection of the second image sensor 12E2 at the S1' exposure timing is zero. AREA[2:0]=0 2) When the level S0 of infrared emission reflection at the S0' exposure timing of the third image sensor 12E3 is not zero, the level S1 of infrared emission reflection at the S1' exposure timing of the third image sensor 12E3 is zero, and the relationship between the level S0 of infrared emission reflection at the S0' exposure timing of the second image sensor 12E2 and the level S1 of infrared emission reflection at the S1' exposure timing is S0≧S1, and the first distance data Z1 (=S1t1 / S0t1+Zoffset) calculated from the data obtained by the second image sensor 12E2 is less than or equal to Far_min(mm). AREA[2:0]=1 3) When the level S0 of infrared emission reflection of the third image sensor 12E3 at the S0' exposure timing is not zero, and the level S1 of infrared emission reflection of the third image sensor 12E3 at the S1' exposure timing is not zero, the relationship between the level S0 of infrared emission reflection of the third image sensor 12E3 at the S0' exposure timing and the level S1 of infrared emission reflection of the third image sensor 12E3 at the S1' exposure timing is S0 ≥ S1, and the level of infrared emission reflection of the second image sensor 12E2 at the S0' exposure timing is When the relationship between the radiation level S0 and the level of infrared emission reflection S1 at the S1' exposure timing is S0≧S1, the first distance data Z1 (=S1t1 / S0t1+Zoffset) calculated from the data obtained by the second image sensor 12E2 is greater than or equal to Far_min(mm), and the first distance data Z1 (=S1t1 / S0t1+Zoffset) calculated from the data obtained by the second image sensor 12E2 is less than or equal to Near_max(mm). AREA[2:0]=2 4) When the level S0 of infrared emission reflection at the S0' exposure timing of the third image sensor 12E3 is not zero, and the level S1 of infrared emission reflection at the S1' exposure timing of the third image sensor 12E3 is not zero, the relationship between the level S0 of infrared emission reflection at the S0' exposure timing of the third image sensor 12E3 and the level S1 of infrared emission reflection at the S1' exposure timing is S0≧S1, and the first distance data Z1 (=S1t1 / S0t1+Zoffset) calculated from the data obtained by the second image sensor 12E2 is greater than or equal to Near_max(mm) or S1>S0. AREA[2:0]=3 5) When the relationship between the level of infrared emission reflection S0 at the S0' exposure timing of the third image sensor 12E3 and the level of infrared emission reflection S1 at the S1' exposure timing is S1 > S0. AREA[2:0]=4 6) When the above conditions are not met AREA[2:0]=0

[0138] In some cases, the signals output from the second image sensor 12E2 and the third image sensor 12E3 may contain a large amount of noise. In view of this, in other examples of the 3D image capturing device 1 of the fourth embodiment, the measurement range determination unit 13I may perform the processing described later. In this example, an adjustable threshold Zth is set.

[0139] 1) When the level of infrared emission reflection S0 (=S0'-BG) at the S0' exposure timing of the second image sensor 12E2 is greater than the threshold Zth, and the level of infrared emission reflection S1 (=S1'-BG) at the S1' exposure timing of the second image sensor 12E2 is less than or equal to the threshold Zth. AREA[2:0]=0 2) When the level S0 of infrared emission reflection of the third image sensor 12E3 at the S0' exposure timing is greater than the threshold Zth, the level S1 of infrared emission reflection of the third image sensor 12E3 at the S1' exposure timing is less than or equal to the threshold Zth, and the level S0 of infrared emission reflection of the second image sensor 12E2 at the S0' exposure timing is greater than or equal to the level S1 of infrared emission reflection of the second image sensor 12E2 at the S1' exposure timing (S0≧S1), and the distance Z between the subject S and the 3D image capturing device 1 obtained by applying the levels S0 of infrared emission reflection of the second image sensor 12E2 at the S0' exposure timing and the level S1 of infrared emission reflection of the second image sensor 12E2 at the S1' exposure timing to the above-described equation 1 is less than or equal to the minimum long-distance value Far_min(mm). AREA[2:0]=1 3) The level of infrared emission reflection S0 at the S0' exposure timing of the third image sensor 12E3 and the level of infrared emission reflection S1 at the S1' exposure timing are greater than the threshold Zth, and the level of infrared emission reflection S0 at the S0' exposure timing of the third image sensor 12E3 is greater than or equal to the level of infrared emission reflection S1 at the S1' exposure timing of the third image sensor 12E3 (S0≧S1), and the levels of infrared emission reflection S0 at the S0' exposure timing of the second image sensor 12E2 and the S1' exposure timing are greater than or equal to the threshold Zth, and the level of infrared emission reflection S0 at the S0' exposure timing of the second image sensor 12E2 When the distance Z between the subject S and the 3D image capturing device 1, obtained by applying the level S1 of infrared emission reflection at the timing to the above-described equation 1, is greater than or equal to the far-distance minimum value Far_min, and the distance Z between the subject S and the 3D image capturing device 1, obtained by applying the level S0 of infrared emission reflection at the S0' exposure timing of the second image sensor 12E2 and the level S1 of infrared emission reflection at the S1' exposure timing to the above-described equation 1, is less than or equal to the near-distance maximum value Near_max. AREA[2:0]=2 4) The level S0 of the infrared emission reflection of the third image sensor 12E3 at the S0' exposure timing and the level S1 of the infrared emission reflection of the third image sensor 12E3 at the S1' exposure timing are greater than the threshold Zth, and the level S0 of the infrared emission reflection of the third image sensor 12E3 at the S0' exposure timing is greater than or equal to the level S1 of the infrared emission reflection of the third image sensor 12E3 at the S1' exposure timing (S0≧S1), and the level of the infrared emission reflection of the second image sensor 12E2 at the S0' exposure timing When a second distance data Z2 is calculated that satisfies the near-range maximum value Near_max or greater, obtained by applying the level of infrared emission reflection S1 at exposure timings S0 and S1' to the above-described equation 1, the distance Z between the subject S and the 3D image imaging device 1 is greater than or equal to Near_max, or when the level of infrared emission reflection S1 at exposure timing S1' of the second image sensor 12E2 is greater than the level of infrared emission reflection S0 at exposure timing S0' of the second image sensor 12E2 (S1>S0). AREA[2:0]=3 5) When the level S0 of the infrared emission reflection of the third image sensor 12E3 at the S0' exposure timing becomes smaller than the level S1 of the infrared emission reflection of the third image sensor 12E3 at the S1' exposure timing (S0 <S1)。 AREA[2:0]=4 6) When none of the above conditions are met. AREA[2:0]=0

[0140] Under these conditions, AREA[2:0] is set, and accordingly, the distance data output switching unit 13J (DepthMIX) sets or switches the distance data S1wide / S0wide and outputs the distance data S1wide / S0wide. This distance data S1wide / S0wide is once stored in the second buffer memory 13K2.

[0141] The distance data output switching unit 13J (DepthMIX) outputs the following according to the value of AREA[2:0].

[0142] When AREA[2:0]=0, the distance data output switching unit 13J determines that measurement is impossible and sets the distance data S1wide / S0wide to zero. When AREA[2:0]=1, the distance data output switching unit 13J outputs the first distance data Z1. When AREA[2:0]=2, the distance data output switching unit 13J changes the mixing ratio of the first distance data Z1 and the second distance data Z2 according to the distance and outputs them. When AREA[2:0]=3, the distance data output switching unit 13J outputs the second distance data Z2. When AREA[2:0]=4, the distance data output switching unit 13J determines that measurement is impossible and maximizes the distance data S1wide / S0wide. In other words, the distance data output switching unit 13J outputs the maximum possible value for the distance data S1wide / S0wide. For example, in the case of a 16-bit unsigned integer, the distance data output switching unit 13J outputs 65535.

[0143] When AREA[2:0]=2, the distance data output switching unit 13J gradually switches from the first distance data Z1 to the second distance data Z2 by mixing the data in a ratio proportional to the distance, as shown in Equation 7, in the portion where the measurement range of the second image sensor 12E2 and the measurement range of the third image sensor 12E3 overlap. Let Znear be the distance Z obtained from the first distance data Z1, as calculated by Equation 1. When AREA[2:0]=2, the distance data S1wide / S0wide is expressed by equation 7 below. S1wide / S0wide=(Znear-Far_min) / (Near_max-Far_min)×Z2+(Near_max-Znear) / (Near_max-Far_min)×Z1...Formula 7

[0144] To summarize the above, the distance data output switching unit 13J (DepthMIX) calculates the following formula and outputs distance data S1wide / S0wide. When AREA[2:0]=0 S1wide / S0wide=0 When AREA[2:0]=1 S1wide / S0wide=Z1 When AREA[2:0]=2 S1wide / S0wide=(Znear-Far_min) / (Near_max-Far_min)×Z2+(Near_max-Znear) / (Near_max-Far_min)×Z1 When AREA[2:0]=3 S1wide / S0wide=Z2 When AREA[2:0]=4 S1wide / S0wide=MAX For example, in the case of a 16-bit unsigned integer, the distance data output switching unit 13J outputs 65535 as the maximum value MAX.

[0145] In the example shown in Figure 16, similar to the example shown in Figure 1, the IR data output switching unit 13L can select the S0t1 signal obtained from the second image sensor 12E2 and the S0t2 signal obtained from the third image sensor 12E3 according to the AREA[2:0] signal and the IRsel[1:0] signal output from the control unit 13H. In addition, the S0t1 signal is also the infrared light image signal from the second image sensor 12E2, and the S0t2 signal is also the infrared light image signal from the third image sensor 12E3. Therefore, when users of the 3D image capturing device 1 acquire distance data indicating the distance between the 3D image capturing device 1 and the subject S (one of the following: zero when AREA[2:0]=0 as shown in Table 2, the first distance data Z1 when AREA[2:0]=1, the value shown in Equation 7 above when AREA[2:0]=2, the second distance data Z2 when AREA[2:0]=3, or the maximum value MAX when AREA[2:0]=4), they can view how the IR image produced by the infrared light irradiated from the light source 11 is actually acquired by the second image sensor 12E2 and the third image sensor 12E3.

[0146] The IR data output switching unit 13L (IRMIX) switches between the S0t1 signal and the S0t2 signal as shown below. The IRsel[1:0] signal can be set by the user of the 3D image acquisition device 1 to the control unit 13H, and the breakdown of the IRsel[1:0] signal is as follows, for example. When IRsel[1:0]=0, the IR data output switching unit 13L outputs as follows in accordance with the AREA[2:0] signal. When AREA[2:0]=0, the S0t1 signal is output. When AREA[2:0]=1, the S0t1 signal is output. When AREA[2:0]=2, the mixing ratio of the S0t1 signal from the second image sensor 12E2 and the S0t2 signal from the third image sensor 12E3 is changed according to the distance and output. When AREA[2:0]=3, the S0t2 signal is output. When AREA[2:0]=4, the S0t2 signal is output. When IRsel[1:0]=1, the S0t1 signal is output. When IRsel[1:0]=2, the S0t2 signal is output.

[0147] When IRsel[1:0]=0 and AREA[2:0]=2, the mixing ratio of the S0t1 signal from the second image sensor 12E2 and the S0t2 signal from the third image sensor 12E3 is changed according to the distance and output, as shown in equation 8 below. IRmix=(Znear-Far_min) / (Near_max-Far_min)×S0t2+(Near_max-Znear) / (Near_max-Far_min)×S0t1...Equation 8

[0148] In this way, the infrared images acquired by the second image sensor 12E2 and the third image sensor 12E3 are appropriately switched or mixed to output in a way that makes it impossible to tell where the images from the second image sensor 12E2 and the third image sensor 12E3 switch.

[0149] <Fifth Embodiment> A fifth embodiment of the three-dimensional image capturing device of the present invention will be described below. The 3D image capturing device 1 of the fifth embodiment is configured in the same way as the 3D image capturing device 1 of the first embodiment described above, except for the points described later. Therefore, the 3D image capturing device 1 of the fifth embodiment can achieve the same effects as the 3D image capturing device 1 of the first embodiment described above, except for the points described later.

[0150] As described above, in the 3D image capturing device 1 of the first embodiment, the near-range maximum value Near_max and the far-range minimum value Far_min are equal. On the other hand, in the 3D image capturing device 1 of the fifth embodiment, the near-range maximum value Near_max is smaller than the far-range minimum value Far_min.

[0151] In the 3D image capturing apparatus 1 of the fifth embodiment, the measurement range determination unit 13I determines the charge accumulated in the first portion 12E21 of the second image sensor 12E2, the charge accumulated in the second portion 12E22 of the second image sensor 12E2, the charge accumulated in the third portion 12E23 of the second image sensor 12E2, the charge accumulated in the first portion 12E31 of the third image sensor 12E3, the charge accumulated in the second portion 12E32 of the third image sensor 12E3, and the third portion of the third image sensor 12E3 Based on the signal indicating the charge accumulated in section 12E33 and the preset discrimination rules shown in Table 3, it is determined whether the distance between the 3D image capturing device 1 and the subject S, as measured by the 3D image capturing device 1, falls within the near-range (Near_min~Near_max), the far-range (Far_min~Far_max), or the range (Near_max~Far_min) which is greater than the maximum near-range value (Near_max) and less than the minimum far-range value (Far_min).

[0152] [Table 3]

[0153] In the three-dimensional video imaging device 1 of the fifth embodiment, when the distance data output switching unit 13J determines that the distance between the three-dimensional video imaging device 1 and the subject S measured by the three-dimensional video imaging device 1 is included in the short-distance range Near_min to Near_max as determined by the measurement range determination unit 13I, the first distance data Z1 (= S1t1 / S0t1 + Zoffset) calculated by the first distance data calculation unit 13F is output. When the distance between the three-dimensional video imaging device 1 and the subject S measured by the three-dimensional video imaging device 1 is included in the long-distance range Far_min to Far_max as determined by the measurement range determination unit 13I, the second distance data Z2 (= S1t2 / S0t2 + Zoffset2) calculated by the second distance data calculation unit 13G is output. When the distance between the three-dimensional video imaging device 1 and the subject S measured by the three-dimensional video imaging device 1 is included in the range Near_max to Far_min that is greater than the short-distance maximum value Near_max and less than the long-distance minimum value Far_min, zero is output as the distance data when determined by the measurement range determination unit 13I.

[0154] Figs. 18A to 18D are diagrams showing the timing in an example of the three-dimensional video imaging device 1 of the fifth embodiment. In the example shown in Figs. 18A to 18D, since Near_max - Near_min < Far_max - Far_min, based on Far_max - Far_min (= 3000 (mm)), the light emission period Tp of the light source unit 11 is obtained as follows. Tp = 3000 / C + h = 12.00 nsec C: speed of light (3 × 10 11 [mm / s]) h: pulse slope correction component (here, 2 [nsec]).

[0155] The offset time Toffset is 2 × (Near_min - Near_min × 0.01) / C, but the offset time Toffset2 is obtained as follows. If the difference between the near-range maximum value Near_max and the far-range minimum value Far_min is denoted as sd2 (=Far_min-Near_max), then the offset time Toffset2 can be calculated using the following formula. Toffset2=2×(Near_max+sd2-(Near_max+sd2)×0.01) / C Next, the measurement range determination unit 13I determines whether the distance between the subject S corresponding to the pixel currently being measured and the 3D image capturing device 1 falls within the close distance measurement range or the far distance measurement range. The distance data output switching unit 13J then switches between outputting the first distance data Z1 or the second distance data Z2 according to the determination result of the measurement range determination unit 13I. As a result, distance data S1wide / S0wide, which is proportional to the distances of two ranges that cannot be obtained with a single TOF sensor, is obtained. The measurement range determination unit 13I makes a determination based on the determination rules shown in Table 3 and outputs a 3-bit signal of AREA[2:0].

[0156] Figure 18A shows the light emission period Tp of the light source unit 11, the first exposure timing (S0' exposure timing), second exposure timing (S1' exposure timing), and third exposure timing (BG exposure timing) of the second image sensor 12E2, and the first exposure timing (S0' exposure timing), second exposure timing (S1' exposure timing), and third exposure timing (BG exposure timing) of the third image sensor 12E3 when AREA[2:0]=1. Figure 18B shows the light emission period Tp of the light source unit 11, the first exposure timing (S0' exposure timing), second exposure timing (S1' exposure timing), and third exposure timing (BG exposure timing) of the second image sensor 12E2, and the first exposure timing (S0' exposure timing), second exposure timing (S1' exposure timing), and third exposure timing (BG exposure timing) of the third image sensor 12E3 when AREA[2:0]=2. Figure 18C shows the light emission period Tp of the light source unit 11, the first exposure timing (S0' exposure timing), second exposure timing (S1' exposure timing), and third exposure timing (BG exposure timing) of the second image sensor 12E2, and the first exposure timing (S0' exposure timing), second exposure timing (S1' exposure timing), and third exposure timing (BG exposure timing) of the third image sensor 12E3 when AREA[2:0]=3. Figure 18D shows the light emission period Tp of the light source unit 11, the first exposure timing (S0' exposure timing), second exposure timing (S1' exposure timing), and third exposure timing (BG exposure timing) of the second image sensor 12E2, and the first exposure timing (S0' exposure timing), second exposure timing (S1' exposure timing), and third exposure timing (BG exposure timing) of the third image sensor 12E3 when AREA[2:0]=4. Refer to the timing diagrams shown in Figures 18A to 18D, and set the judgment conditions for AREA[2:0] shown in Table 3.

[0157] 1) When the level S0 of the infrared emission reflection of the second image sensor 12E2 at the S0' exposure timing is not zero, and the level S1 of the infrared emission reflection of the second image sensor 12E2 at the S1' exposure timing is zero. AREA[2:0]=0 2) When the level S1 of infrared emission reflection at the S1' exposure timing of the third image sensor 12E3 is zero, and the relationship between the level S0 of infrared emission reflection at the S0' exposure timing of the second image sensor 12E2 and the level S1 of infrared emission reflection at the S1' exposure timing is S0≧S1, and the first distance data Z1 (=S1t1 / S0t1+Zoffset) calculated from the data obtained by the second image sensor 12E2 is less than or equal to Near_max(mm). AREA[2:0]=1 3) When the level S0 of infrared emission reflection of the third image sensor 12E3 at exposure timing S0' is not zero, and the level S1 of infrared emission reflection of the third image sensor 12E3 at exposure timing S1' is not zero, the relationship between the level S0 of infrared emission reflection of the third image sensor 12E3 at exposure timing S0' and the level S1 of infrared emission reflection of the third image sensor 12E3 at exposure timing S1' is S0 ≥ S1, and the second distance data Z2 (= S1t2 / S0t2 + Zoffset2) calculated from the data obtained by the third image sensor 12E3 is greater than or equal to Far_min (mm). AREA[2:0]=3 4) When the level S0 of infrared emission reflection of the second image sensor 12E2 at exposure timing S0' is zero, the level S1 of infrared emission reflection of the second image sensor 12E2 at exposure timing S1' is zero, and the relationship between the level S0 of infrared emission reflection of the third image sensor 12E3 at exposure timing S0' and the level S1 of infrared emission reflection of the third image sensor 12E3 at exposure timing S1' is S1 > S0. AREA[2:0]=4 5) When the above conditions are not met AREA[2:0]=2

[0158] In some cases, the signals output from the second image sensor 12E2 and the third image sensor 12E3 may contain a large amount of noise. In view of this, in other examples of the 3D image capturing device 1 of the fifth embodiment, the measurement range determination unit 13I may perform the processing described later. In this example, an adjustable threshold Zth is set.

[0159] 1) When the level of infrared emission reflection S0 (=S0'-BG) at the S0' exposure timing of the second image sensor 12E2 is greater than the threshold Zth, and the level of infrared emission reflection S1 (=S1'-BG) at the S1' exposure timing of the second image sensor 12E2 is less than the threshold Zth. AREA[2:0]=0 2) When the level S1 of the infrared emission reflection of the third image sensor 12E3 at the S1' exposure timing is less than the threshold Zth, and the level S0 of the infrared emission reflection of the second image sensor 12E2 at the S0' exposure timing is greater than or equal to the level S1 of the infrared emission reflection of the second image sensor 12E2 at the S1' exposure timing (S0≧S1), and the distance Z between the subject S and the 3D image capturing device 1 obtained by applying the levels S0 of the infrared emission reflection of the second image sensor 12E2 at the S0' exposure timing and the level S1 of the infrared emission reflection at the S1' exposure timing to the above-described equation 1 is less than or equal to the near-range maximum value Near_max(mm). AREA[2:0]=1 3) When the level of infrared emission reflection S0 at the S0' exposure timing and the level of infrared emission reflection S1 at the S1' exposure timing of the third image sensor 12E3 are greater than the threshold Zth, and the level of infrared emission reflection S0 at the S0' exposure timing of the third image sensor 12E3 is greater than or equal to the level of infrared emission reflection S1 at the S1' exposure timing of the third image sensor 12E3 (S0≧S1), and the distance Z between the subject S and the 3D image capturing device 1 obtained by applying the levels of infrared emission reflection S0 at the S0' exposure timing and the level of infrared emission reflection S1 at the S1' exposure timing of the third image sensor 12E3 is greater than or equal to the minimum long-distance value Far_min. AREA[2:0]=3 4) When the level S0 of infrared emission reflection at the S0' exposure timing of the second image sensor 12E2 and the level S1 of infrared emission reflection at the S1' exposure timing are less than the threshold Zth, and the level S0 of infrared emission reflection at the S0' exposure timing of the third image sensor 12E3 is less than the level S1 of infrared emission reflection at the S1' exposure timing of the third image sensor 12E3 (S0 <S1)。 AREA[2:0]=4 5) When the above conditions are not met. AREA[2:0]=2

[0160] Under these conditions, AREA[2:0] is set, and accordingly, the distance data output switching unit 13J sets or switches the distance data S1wide / S0wide and outputs the distance data S1wide / S0wide. This distance data S1wide / S0wide is temporarily stored in the second buffer memory 13K2. In an example of the 3D image capturing device 1 of the fifth embodiment, the IR data output switching unit 13L can select the S0t1 signal obtained from the second image sensor 12E2 and the S0t2 signal obtained from the third image sensor 12E3 according to the AREA[2:0] signal and the IRsel[1:0] signal output from the control unit 13H. The S0t1 signal is also the infrared light image signal of the second image sensor 12E2, and the S0t2 signal is also the infrared light image signal of the third image sensor 12E3. Therefore, when users of the 3D image capturing device 1 acquire distance data indicating the distance between the 3D image capturing device 1 and the subject S (one of the following as shown in Table 3: zero when AREA[2:0]=0, first distance data Z1 when AREA[2:0]=1, zero when AREA[2:0]=2, second distance data Z2 when AREA[2:0]=3, or the maximum value MAX when AREA[2:0]=4), they can view how the IR image, generated by infrared light irradiated from the light source 11, is actually acquired by the second image sensor 12E2 and the third image sensor 12E3.

[0161] The IRsel[1:0] signal can be set by the user of the 3D image acquisition device 1 to the control unit 13H, and the breakdown of the IRsel[1:0] signal is as follows, for example. When IRsel[1:0]=0, the IR data output switching unit 13L outputs as follows in accordance with the AREA[2:0] signal.

[0162] When AREA[2:0]=0, the S0t1 signal is output. When AREA[2:0]=1, the S0t1 signal is output. When AREA[2:0]=2, output zero. When AREA[2:0]=3, the S0t2 signal is output. When AREA[2:0]=4, the S0t2 signal is output.

[0163] When IRsel[1:0]=1 The IR data output switching unit 13L always outputs the S0t1 signal. When IRsel[1:0]=2 The IR data output switching unit 13L always outputs the S0t2 signal.

[0164] In an example of the 3D video imaging device 1 of the fifth embodiment, the output signal from the IR data output switching unit 13L is treated as a monochrome IR video signal, the black level is adjusted by the black level unit 13M1 of the IR data processing unit 13M, video gamma is applied by the gamma unit 13M2 of the IR data processing unit 13M, and the signal is temporarily stored in the third buffer memory 13K3 of the storage unit 13K.

[0165] Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the invention. The configurations described in each of the embodiments and examples above may be combined as appropriate. [Explanation of symbols]

[0166] 1...3D image acquisition device, 11...Light source unit, 12...3-chip image sensor unit, 12A...Lens, 12B1...First prism, 12B2...Second prism, 12B3...Third prism, 12C...Reflective dichroic film, 12D...Half mirror, 12E1...First image sensor, 12E2...Second image sensor, 12E21...First part, 12E22...Second part, 12E23...Third part, 12E3...Third image sensor, 12E31...First part, 12E32...Second part, 12E33...Third part, 12F...Infrared cut filter, 12G1...Infrared 850nm bandpass filter, 12G2...Infrared 85 0nm bandpass filter, 13... Processing unit, 13A... Light emission control unit, 13A1... Light source drive control unit, 13A2... Light source light emission control pulse setting unit, 13B... First image sensor control unit, 13B1... First image sensor drive control unit, 13B2... First image sensor drive control pulse setting unit, 13C... Second image sensor control unit, 13C1... Second image sensor drive control unit, 13C2... Second image sensor drive control pulse setting unit, 13D... Third image sensor control unit, 13D1... Third image sensor drive control unit, 13D2... Third image sensor drive control pulse setting unit, 13E... RGB camera processing unit, 13E1... Inter Face section, 13E2...Black balance section, 13E3...Debayer section, 13E4...Color matrix section, 13E5...Gamma section, 13E6...YCbCr conversion section, 13E7...Low-pass filter, 13E8...Multiplex section, 13F...First distance data calculation section, 13F1...Interface section, 13F2...Delay circuit, 13F3...Delay circuit, 13F4...Delay circuit, 13F5...Timing adjustment section, 13F6...Timing adjustment section, 13F7...Subtraction section, 13F8...Subtraction section, 13F9...Calculation section, 13FX...Addition section, 13G...Second distance data calculation section, 13G 1…Interface unit, 13G2…Horizontal inversion unit, 13G3…Horizontal inversion unit, 13G4…Horizontal inversion unit, 13G5…Timing adjustment unit, 13G6…Timing adjustment unit, 13G7…Subtraction unit, 13G8…Subtraction unit, 13G9…Calculation unit, 13GX…Addition unit, 13H…Control unit, 13I…Measurement range determination unit, 13J…Distance data output switching unit, 13K…Storage unit, 13K1…First buffer memory, 13K2…Second buffer memory, 13K3…Third buffer memory, 13L…IR data output switching unit, 13M…IR data processing unit, 13M1…Black level unit, 13M2…Gamma unit,13N...Output mixing section, 13P...Output interface section, S...Subject,

Claims

1. A light source unit that illuminates the subject with infrared light, A three-chip image sensor unit into which reflected infrared light irradiated by the light source and reflected by the subject, and reflected visible light from the subject are incident. A three-dimensional image capturing apparatus comprising a processing unit that controls the light source unit and the three-chip image sensor unit and processes the signals output from the three-chip image sensor unit, The three-chip image sensor unit is A first prism into which infrared light and visible light from the subject are incident, A reflective dichroic film that reflects visible light and transmits infrared light from the infrared and visible light incident on the first prism, A first image sensor that receives visible light reflected by the aforementioned reflective dichroic film, A second prism into which infrared light transmitted by the aforementioned reflective dichroic film is incident, A half-mirror that reflects a portion of the infrared light incident on the second prism and transmits the remainder of the infrared light incident on the second prism, A third prism into which infrared light transmitted by the half-mirror enters, One of the second and third image sensors receives infrared light transmitted by the half-mirror, The system includes the other of the second and third image sensors, which receive infrared light reflected by the half-mirror. The second image sensor is a short-range TOF (Time of Flight) sensor used for measuring the short-range distance, which is the distance range from the minimum short-range value to the maximum short-range value. The third image sensor is a TOF sensor for long-distance range used to measure distances that are greater than the short-distance range, and are the distance range from the minimum long-distance value to the maximum long-distance value. The first image sensor is, A first portion that accumulates charge at the first exposure timing of the second image sensor, which is the exposure timing of the second image sensor, in which at least the majority of the light emission period of the light source is included in the exposure period of the second image sensor, The second image sensor comprises a second portion that accumulates charge at a second exposure timing of the second image sensor, which is an exposure timing of the second image sensor that starts after the end of the light emission period of the light source, The third image sensor is A first portion that accumulates charge at the first exposure timing of the third image sensor, which is the exposure timing of the third image sensor, in which at least a portion of the light emission period of the light source is included in the exposure period of the third image sensor, The third image sensor comprises a second portion which accumulates charge at a second exposure timing of the third image sensor, which is an exposure timing of the third image sensor that starts after the end of the light emission period of the light source, The aforementioned processing unit, A light emission control unit that controls the light emission of the light source unit, A second image sensor control unit that controls the exposure timing of the second image sensor, A third image sensor control unit that controls the exposure timing of the third image sensor, A first distance data calculation unit calculates first distance data indicating the distance between the three-dimensional image capturing device and the subject based on the charge accumulated in at least the first part of the second image sensor and the charge accumulated in the second part of the second image sensor, The system includes a second distance data calculation unit that calculates second distance data indicating the distance between the three-dimensional image capturing device and the subject based on the charge accumulated in at least the first portion of the third image sensor and the charge accumulated in the second portion of the third image sensor, The aforementioned processing unit, A measurement range determination unit that determines whether the distance between the three-dimensional image capturing device and the subject, as measured by the three-dimensional image capturing device, falls within the short-range or long-range, based on a signal indicating the charge accumulated in at least the first portion of the second image sensor, a signal indicating the charge accumulated in the second portion of the second image sensor, a signal indicating the charge accumulated in the first portion of the third image sensor, a signal indicating the charge accumulated in the second portion of the third image sensor, and a preset determination rule. The distance data output switching unit includes a measurement range determination unit that outputs the first distance data calculated by the first distance data calculation unit when the measurement range determination unit determines that the distance between the three-dimensional image capturing device and the subject, as measured by the three-dimensional image capturing device, falls within the short-distance range, and a distance data output switching unit that outputs the second distance data calculated by the second distance data calculation unit when the measurement range determination unit determines that the distance between the three-dimensional image capturing device and the subject, as measured by the three-dimensional image capturing device, falls within the long-distance range. A three-dimensional image acquisition device.

2. The first image sensor is a color image sensor having a Bayer array, The aforementioned processing unit, The system includes an RGB camera processing unit that generates RGB video from the signal output by the first image sensor, The second and third image sensors have functions as infrared image sensors. The aforementioned process is, An IR data output switching unit outputs an infrared image for the near-range captured by the second image sensor when the measurement range determination unit determines that the distance between the three-dimensional image imaging device and the subject, as measured by the three-dimensional image imaging device, falls within the near-range, and outputs an infrared image for the far-range captured by the third image sensor when the measurement range determination unit determines that the distance between the three-dimensional image imaging device and the subject, as measured by the three-dimensional image imaging device, falls within the far-range, The system includes an output mixing unit that synchronously outputs at least one of the first distance data and the second distance data output from the distance data output switching unit, the RGB video generated by the RGB camera processing unit, and an IR video generated from the near-range infrared image or the far-range infrared image output from the IR data output switching unit. The three-dimensional image capturing apparatus according to claim 1.

3. When the second image sensor receives infrared light transmitted by the half mirror, the transmittance of the half mirror is less than 50%. When the third image sensor receives infrared light transmitted by the half mirror, the transmittance of the half mirror is greater than 50%. The three-dimensional image capturing apparatus according to claim 1.

4. A light source unit that illuminates the subject with infrared light, A three-chip image sensor unit into which reflected infrared light irradiated by the light source and reflected by the subject, and reflected visible light from the subject are incident. A three-dimensional image capturing apparatus comprising a processing unit that controls the light source unit and the three-chip image sensor unit and processes the signals output from the three-chip image sensor unit, The three-chip image sensor unit is A first prism into which infrared light and visible light from the subject are incident, A reflective dichroic film that reflects visible light and transmits infrared light from the infrared and visible light incident on the first prism, A first image sensor that receives visible light reflected by the aforementioned reflective dichroic film, A second prism into which infrared light transmitted by the aforementioned reflective dichroic film is incident, A half-mirror that reflects a portion of the infrared light incident on the second prism and transmits the remainder of the infrared light incident on the second prism, A third prism into which infrared light transmitted by the half-mirror enters, One of the second and third image sensors receives infrared light transmitted by the half-mirror, The system includes the other of the second and third image sensors, which receive infrared light reflected by the half-mirror. The second image sensor is a short-range TOF (Time of Flight) sensor used to measure the distance range from the minimum short-range value to the maximum short-range value. The third image sensor is a long-range TOF sensor used to measure distances in a distance range greater than the distance range from the minimum near-range value to the maximum near-range value, and in the distance range from the minimum far-range value to the maximum far-range value. A portion of the distance range from the short-range minimum value to the short-range maximum value and a portion of the distance range from the long-range minimum value to the long-range maximum value overlap with each other, forming an overlapping measurement range. The short-range measurement range is the distance range from the short-range minimum value to the short-range maximum value, minus the overlap measurement range. The long-distance measurement range is the distance range from the minimum long-distance value to the maximum long-distance value, minus the overlap measurement range. The first image sensor is, A first portion that accumulates charge at the first exposure timing of the second image sensor, which is the exposure timing of the second image sensor, in which at least the majority of the light emission period of the light source is included in the exposure period of the second image sensor, The second image sensor comprises a second portion that accumulates charge at a second exposure timing of the second image sensor, where the exposure period of the second image sensor is an exposure timing of the second image sensor that starts after the end of the light emission period of the light source, The third image sensor is A first portion that accumulates charge at the first exposure timing of the third image sensor, which is the exposure timing of the third image sensor, in which at least a portion of the light emission period of the light source is included in the exposure period of the third image sensor, The third image sensor comprises a second portion which accumulates charge at a second exposure timing of the third image sensor, which is an exposure timing of the third image sensor that starts after the end of the light emission period of the light source, The aforementioned processing unit, A light emission control unit that controls the light emission of the light source unit, A second image sensor control unit that controls the exposure timing of the second image sensor, A third image sensor control unit that controls the exposure timing of the third image sensor, A first distance data calculation unit calculates first distance data indicating the distance between the three-dimensional image capturing device and the subject based on the charge accumulated in at least the first part of the second image sensor and the charge accumulated in the second part of the second image sensor, The system includes a second distance data calculation unit that calculates second distance data indicating the distance between the three-dimensional image capturing device and the subject based on the charge accumulated in at least the first portion of the third image sensor and the charge accumulated in the second portion of the third image sensor, The aforementioned processing unit, A measurement range determination unit that determines whether the distance between the three-dimensional image capturing device and the subject, as measured by the three-dimensional image capturing device, is included in the short-range measurement range, the overlap measurement range, or the long-range measurement range, based on a signal indicating the charge accumulated in at least the first portion of the second image sensor, a signal indicating the charge accumulated in the second portion of the second image sensor, a signal indicating the charge accumulated in the first portion of the third image sensor, a signal indicating the charge accumulated in the second portion of the third image sensor, and a preset determination rule. The distance data output switching unit outputs the first distance data calculated by the first distance data calculation unit when the measurement range determination unit determines that the distance between the three-dimensional image capturing device and the subject, as measured by the three-dimensional image capturing device, is included in the short-range measurement range; outputs the second distance data calculated by the second distance data calculation unit when the measurement range determination unit determines that the distance between the three-dimensional image capturing device and the subject, as measured by the three-dimensional image capturing device, is included in the long-range measurement range; and outputs a mixture of the first distance data calculated by the first distance data calculation unit and the second distance data calculated by the second distance data calculation unit when the measurement range determination unit determines that the distance between the three-dimensional image capturing device and the subject, as measured by the three-dimensional image capturing device, is included in the overlap measurement range. A three-dimensional image acquisition device.

5. A light source unit that illuminates the subject with infrared light, A three-chip image sensor unit into which reflected infrared light irradiated by the light source and reflected by the subject, and reflected visible light from the subject are incident. A three-dimensional image capturing apparatus comprising a processing unit that controls the light source unit and the three-chip image sensor unit and processes the signals output from the three-chip image sensor unit, The three-chip image sensor unit is A first prism into which infrared light and visible light from the subject are incident, A reflective dichroic film that reflects visible light and transmits infrared light from the infrared and visible light incident on the first prism, A first image sensor that receives visible light reflected by the aforementioned reflective dichroic film, A second prism into which infrared light transmitted by the aforementioned reflective dichroic film is incident, A half-mirror that reflects a portion of the infrared light incident on the second prism and transmits the remainder of the infrared light incident on the second prism, A third prism into which infrared light transmitted by the half-mirror enters, One of the second and third image sensors receives infrared light transmitted by the half-mirror, The system includes the other of the second and third image sensors, which receive infrared light reflected by the half-mirror. The second image sensor is a short-range TOF (Time of Flight) sensor used for measuring the short-range distance, which is the distance range from the minimum short-range value to the maximum short-range value. The third image sensor is a TOF sensor for long-distance range used to measure distances that are greater than the short-distance range, and are the distance range from the minimum long-distance value to the maximum long-distance value. The aforementioned short-range maximum value is smaller than the aforementioned long-range minimum value. The first image sensor is, A first portion that accumulates charge at the first exposure timing of the second image sensor, which is the exposure timing of the second image sensor, in which at least the majority of the light emission period of the light source is included in the exposure period of the second image sensor, The second image sensor comprises a second portion that accumulates charge at a second exposure timing of the second image sensor, where the exposure period of the second image sensor is an exposure timing of the second image sensor that starts after the end of the light emission period of the light source, The third image sensor is A first portion that accumulates charge at the first exposure timing of the third image sensor, which is the exposure timing of the third image sensor, in which at least a portion of the light emission period of the light source is included in the exposure period of the third image sensor, The third image sensor comprises a second portion which accumulates charge at a second exposure timing of the third image sensor, which is an exposure timing of the third image sensor that starts after the end of the light emission period of the light source, The aforementioned processing unit, A light emission control unit that controls the light emission of the light source unit, A second image sensor control unit that controls the exposure timing of the second image sensor, A third image sensor control unit that controls the exposure timing of the third image sensor, A first distance data calculation unit calculates first distance data indicating the distance between the three-dimensional image capturing device and the subject based on the charge accumulated in at least the first part of the second image sensor and the charge accumulated in the second part of the second image sensor, The system includes a second distance data calculation unit that calculates second distance data indicating the distance between the three-dimensional image capturing device and the subject based on the charge accumulated in at least the first portion of the third image sensor and the charge accumulated in the second portion of the third image sensor, The aforementioned processing unit, A measurement range determination unit that determines, at least, whether the distance between the three-dimensional image capturing device and the subject, as measured by the three-dimensional image capturing device, is included in the short-range range, included in the long-range range, or included in a range greater than the maximum short-range value and less than the minimum long-range value, based on a signal indicating the charge accumulated in at least the first portion of the second image sensor, a signal indicating the charge accumulated in the second portion of the second image sensor, a signal indicating the charge accumulated in the first portion of the third image sensor, a signal indicating the charge accumulated in the second portion of the third image sensor, and a preset determination rule. The distance data output switching unit includes: outputting the first distance data calculated by the first distance data calculation unit when the measurement range determination unit determines that the distance between the three-dimensional image capturing device and the subject, as measured by the three-dimensional image capturing device, falls within the short-distance range; outputting the second distance data calculated by the second distance data calculation unit when the measurement range determination unit determines that the distance between the three-dimensional image capturing device and the subject, as measured by the three-dimensional image capturing device, falls within the long-distance range; and outputting zero as distance data when the measurement range determination unit determines that the distance between the three-dimensional image capturing device and the subject, as measured by the three-dimensional image capturing device, falls within a range greater than the short-distance maximum value and less than the long-distance minimum value. A three-dimensional image acquisition device.

6. The second image sensor control unit is: At a time when the offset time based on the near-range minimum value has elapsed from the start timing of the light emission period of the light source, the exposure period corresponding to the first exposure timing of the second image sensor is started. At the time when the offset time based on the near-distance minimum value has elapsed from the end of the light emission period of the light source, the exposure period corresponding to the second exposure timing of the second image sensor is started. The third image sensor control unit is: At a timing after the offset time based on the minimum distance value has elapsed from the start timing of the light emission period of the light source, an exposure period corresponding to the first exposure timing of the third image sensor, which is set to be within the same frame as the exposure period corresponding to the first exposure timing of the second image sensor, is started. The three-dimensional image capturing apparatus according to claim 1, 4, or 5, wherein, at a timing after the offset time based on the minimum distance value has elapsed from the end timing of the light emission period of the light source, an exposure period corresponding to the second exposure timing of the third image sensor, which is set to be within the same frame as the exposure period corresponding to the second exposure timing of the second image sensor, is started.

Images