Solid-state imaging element, distance imaging device, and ranging method

Abstract

The present invention includes a pixel array (110) in which a plurality of pixels (120) including a plurality of first pixels and a plurality of second pixels are disposed in a two-dimensional manner, in which each of the plurality of first pixels is exposed to reflected light (L2) of irradiation light (L1) reflected by a target (2) on the basis of an exposure signal at a timing interlocking with a light-emission signal and generates a signal for distance detection, each of the plurality of second pixels generates a signal for phase-difference detection on the basis of light incident from a region in which the reflected light is pupil-split, the pixel array (110) includes one or more phase-difference detection regions (220) in which two or more second pixels of the plurality of second pixels are aligned along one direction, and at least some of the plurality of first pixels are, in the pixel array (110), disposed at positions different from the one or more phase-difference detection regions (220) and closer to a center of the pixel array (110) than at least one phase-difference detection region (220) of the one or more phase-difference detection regions (220).

Description

Solid-state imaging device, distance imaging device, and distance measuring method

[0001] The present disclosure relates to a solid-state imaging device, a distance imaging device, and a distance measuring method.

[0002] Japanese Patent Application Laid-Open No. 2003-124222 discloses a technique for correcting errors contained in a distance image obtained by imaging using a ToF (Time of Flight) camera.

[0003] Japanese Patent Application Laid-Open No. 2020-008531

[0004] For example, in the technology disclosed in Patent Document 1, the distance value to a reference object calculated based on a two-dimensional image of the reference object having multiple feature points with clear correlations in three-dimensional coordinates is compared with the distance value measured by a distance imaging device to calculate a correction amount and correct the distance value.

[0005] However, the technique disclosed in Patent Document 1 requires a reference object, and has the problem of being inconvenient when correction needs to be repeated, such as in situations where the surrounding environment is prone to change.

[0006] Therefore, the present disclosure provides a solid-state imaging device and the like that can be used in a distance measuring device that can easily correct distance measurement values.

[0007] One aspect of a solid-state imaging device according to the present disclosure includes a pixel array in which a plurality of pixels including a plurality of first pixels and a plurality of second pixels are arranged two-dimensionally, wherein each of the plurality of first pixels performs exposure of reflected light, which is illumination light reflected by an object, based on an exposure signal whose timing is linked to an emission signal, to generate a signal for distance detection, and each of the plurality of second pixels generates a signal for phase difference detection based on light incident on the reflected light from a pupil-divided region, the pixel array includes one or more phase difference detection regions in which two or more second pixels of the plurality of second pixels are arranged along one direction, and at least some of the first pixels of the plurality of first pixels are arranged at a position in the pixel array that is different from the one or more phase difference detection regions and that is closer to a center of the pixel array than at least one phase difference detection region of the one or more phase difference detection regions.

[0008] Furthermore, one aspect of a distance imaging device according to the present disclosure includes the above-described solid-state imaging element, a light source that irradiates an object with the irradiation light, a distance calculation unit that calculates a first distance to the object based on the distance detection signals generated by the plurality of first pixels, a drive control unit that outputs the light emission signal that instructs the light source to irradiate the irradiation light and outputs the exposure signal that instructs the solid-state imaging element to expose the plurality of pixels, and a correction processing unit that calculates a second distance to the object based on the phase difference detection signals generated by the plurality of second pixels and performs processing to correct the first distance based on the first distance and the second distance.

[0009] Another aspect of a distance imaging device according to the present disclosure includes a solid-state imaging element including a pixel array in which a plurality of pixels, including a plurality of first pixels and a plurality of second pixels, are arranged two-dimensionally; a light source that irradiates an object with illumination light; a distance calculation unit; a drive control unit; and a correction processing unit, wherein each of the plurality of first pixels performs exposure to light reflected from the object based on an exposure signal timed in conjunction with an emission signal to generate a distance detection signal; each of the plurality of second pixels generates a phase difference detection signal based on light incident on the reflected light from a pupil-divided region; the distance calculation unit calculates a first distance to the object based on the distance detection signals generated by the plurality of first pixels; the drive control unit outputs the emission signal to the light source and outputs the exposure signal to the solid-state imaging element; the correction processing unit calculates a second distance to the object based on the phase difference detection signals generated by the plurality of second pixels and performs processing to correct the first distance based on the first distance and the second distance.

[0010] Furthermore, one aspect of the distance measuring method according to the present disclosure is a distance measuring method using the above-mentioned solid-state imaging element, and includes a drive control step of outputting the light emission signal to a light source that irradiates the object with the irradiation light and outputting the exposure signal to the solid-state imaging element, a distance calculation step of acquiring the distance detection signal from the solid-state imaging element and calculating a first distance to the object based on the distance detection signal, and a correction processing step of calculating a second distance to the object based on the phase difference detection signals generated by the plurality of second pixels and performing processing to correct the first distance based on the first distance and the second distance.

[0011] Another aspect of the distance measuring method according to the present disclosure is a distance measuring method executed by a distance imaging device, the distance imaging device including: a light source that irradiates an object with illumination light; and a solid-state imaging element including a pixel array in which a plurality of pixels, including a plurality of first pixels and a plurality of second pixels, are arranged two-dimensionally, wherein each of the plurality of first pixels exposes reflected light of the illumination light reflected by the object based on an exposure signal timed in conjunction with an emission signal to generate a distance detection signal, and each of the plurality of second pixels generates a phase difference detection signal based on light incident on the reflected light from a pupil-divided region, the distance measuring method including: a drive control step of outputting the emission signal to the light source and the exposure signal to the solid-state imaging element; a distance calculation step of calculating a first distance to the object based on the distance detection signals generated by the plurality of first pixels; and a correction processing step of calculating a second distance to the object based on the phase difference detection signals generated by the plurality of second pixels, and performing processing to correct the first distance based on the first distance and the second distance.

[0012] The present disclosure can be realized not only as the distance measurement method, but also as a program for causing a computer to execute the distance measurement method, and further as a computer-readable recording medium storing the program.

[0013] The solid-state imaging device, distance imaging device, and distance measuring method according to the present disclosure can be used in a distance measuring device that can easily correct distance measurements.

[0014] FIG. 1 is a block diagram illustrating a functional configuration of a range imaging device according to an embodiment. FIG. 2 is a block diagram illustrating a functional configuration of a solid-state imaging element according to an embodiment. FIG. 3 is a circuit diagram illustrating an example of a circuit configuration of a pixel according to an embodiment. FIG. 4 is a diagram illustrating an example of a pixel arrangement in a pixel array according to an embodiment. FIG. 5 is a diagram illustrating a situation in which the range imaging device according to an embodiment is used as an endoscope system. FIG. 6 is a flowchart illustrating the operation of the range imaging device according to an embodiment. FIG. 7 is a diagram illustrating parallax detection using phase difference pixels. FIG. 8 is a timing chart illustrating an imaging operation before correction of the range imaging device according to an embodiment. FIG. 9 is a diagram illustrating the relationship between parallax and phase difference. FIG. 10 is a timing chart illustrating an imaging operation after correction of the range imaging device according to an embodiment. FIG. 11 is a diagram illustrating a first example of a pixel arrangement in a pixel array according to an embodiment. FIG. 12 is a diagram illustrating a second example of a pixel arrangement in a pixel array according to an embodiment. FIG. 13 is a diagram illustrating a third example of a pixel arrangement in a pixel array according to an embodiment. FIG. 14 is a diagram illustrating a fourth example of a pixel arrangement in a pixel array according to an embodiment. FIG. 15 is a diagram illustrating a fifth example of a pixel arrangement in a pixel array according to an embodiment. Fig. 16 is a diagram showing a sixth example of a pixel arrangement in a pixel array according to an embodiment, Fig. 17 is a diagram showing a seventh example of a pixel arrangement in a pixel array according to an embodiment, and Fig. 18 is a diagram showing an eighth example of a pixel arrangement in a pixel array according to an embodiment.

[0015] (Insights gained by the inventors) A TOF (Time of Flight) distance measuring sensor calculates the distance to a subject from the exposure amount of reflected light. In order to calculate the distance accurately, the phase of the irradiation pulse of the irradiation light that is irradiated onto the object and the exposure pulse that exposes the reflected light must be matched.

[0016] Here, when the control circuit and light source are positioned far from the TOF sensor that performs exposure, as in an endoscope device, a phase difference occurs due to a propagation delay between the illumination pulse and the exposure pulse. Therefore, it is necessary to perform a correction to align the phase of the illumination pulse with the phase of the exposure pulse, taking into account the phase difference due to the propagation delay.

[0017] However, since the phase difference due to propagation delay changes depending on the environment in which the range imaging device is used, such as temperature and the shape of the cable that transmits the pulse, adjustment at the time of shipment alone is insufficient; the phase difference must be detected and corrected during use.

[0018] Here, by providing an optical path such that the irradiation pulse is directly incident on a TOF sensor that performs exposure at the tip of the cable and performing distance measurement using the TOF sensor, it is possible to correct the distance measurement value to the target object using the distance measurement value obtained. In this case, the size of the tip of the cable becomes large. A large size of the tip of the cable is a disadvantage in an endoscope device.

[0019] Therefore, the inventors came up with the idea of ​​arranging phase difference pixels in the TOF sensor. The phase difference pixels can detect parallax and calculate the subject distance without being affected by propagation delays in cables, etc. Therefore, by correcting the distance measured by the TOF based on the distance measured by the phase difference pixels, it becomes possible to calculate an accurate distance using the TOF method.

[0020] (Summary of the Present Disclosure) A solid-state imaging element according to a first aspect of the present disclosure includes a pixel array in which a plurality of pixels including a plurality of first pixels and a plurality of second pixels are two-dimensionally arranged, and each of the plurality of first pixels performs exposure of reflected light, which is illumination light reflected by an object, based on an exposure signal having a timing linked to a light emission signal, to generate a signal for distance detection, and each of the plurality of second pixels generates a signal for phase difference detection based on light incident on the reflected light from a pupil-divided region, the pixel array includes one or more phase difference detection regions in which two or more second pixels of the plurality of second pixels are arranged along one direction, and at least some of the plurality of first pixels are arranged at a position in the pixel array different from the one or more phase difference detection regions and closer to a center of the pixel array than at least one phase difference detection region of the one or more phase difference detection regions.

[0021] Such a solid-state imaging device has a pixel array including TOF pixels that perform distance measurement by TOF and phase difference pixels that perform distance measurement by phase difference detection, and therefore can detect distance by two methods. Therefore, such a solid-state imaging device can be used in a distance measuring device that can easily correct distance measurements.

[0022] A solid-state imaging device according to a second aspect of the present disclosure is the solid-state imaging device according to the first aspect, wherein the number of the plurality of first pixels is greater than the number of the plurality of second pixels.

[0023] Such a solid-state image sensor can generate a TOF image with higher resolution.

[0024] A solid-state imaging device according to a third aspect of the present disclosure is the solid-state imaging device according to the first or second aspect, wherein the plurality of first pixels have a mode for generating signals for a visible light image.

[0025] Such a solid-state imaging device can change the distance measurement range when capturing a TOF image.

[0026] A solid-state imaging element according to a fourth aspect of the present disclosure is the solid-state imaging element according to any one of the first to third aspects, wherein the one or more phase difference detection regions include a phase difference detection region in which the second pixels are aligned along a first direction and a phase difference detection region in which the second pixels are aligned along a second direction perpendicular to the first direction, and the second pixels aligned in the first phase difference detection region receive light incident from a region formed by pupil division in the first direction, and the second pixels aligned in the second phase difference detection region receive light incident from a region formed by pupil division in the second direction.

[0027] Such a solid-state imaging device can perform phase difference detection in two orthogonal directions. Generally, phase difference detection may not be possible depending on the direction in which detection is performed, so using such a solid-state imaging device improves the efficiency of phase difference detection.

[0028] A solid-state imaging element according to a fifth aspect of the present disclosure is a solid-state imaging element according to any one of the first to fourth aspects, wherein the size of each of the plurality of second pixels is different from the size of each of the plurality of first pixels.

[0029] Such a solid-state imaging device can adjust the resolution of the TOF image and the phase difference detection pixels without changing the chip size.

[0030] A solid-state imaging element according to a sixth aspect of the present disclosure is a solid-state imaging element according to any one of the first to fifth aspects, wherein the pixel array includes an effective pixel area in which at least a portion of the plurality of first pixels are arranged two-dimensionally, and the one or more phase difference detection areas are arranged outside the effective pixel area.

[0031] Such a solid-state imaging element can prevent deterioration of the TOF image obtained in the effective pixel area. Furthermore, by arranging the phase difference detection area 220 outside the effective pixel area 210, it becomes easy to change and control the pixel size of the second pixel included in the phase difference detection area 220.

[0032] A solid-state imaging element according to a seventh aspect of the present disclosure is a solid-state imaging element according to any one of the first to fifth aspects, wherein the pixel array includes an effective pixel area in which at least a portion of the plurality of first pixels are arranged two-dimensionally, and the one or more phase difference detection areas are arranged within the effective pixel area.

[0033] Such a solid-state imaging device can reduce the chip size of the entire pixel array.

[0034] A solid-state imaging element according to an eighth aspect of the present disclosure is a solid-state imaging element according to any one of the first to fifth aspects, wherein the pixel array includes an effective pixel area in which at least a portion of the plurality of first pixels are arranged two-dimensionally, and the one or more phase difference detection areas include a phase difference detection area arranged outside the effective pixel area and a phase difference detection area arranged within the effective pixel area.

[0035] Such a solid-state imaging device can reduce the chip size of the entire pixel array while avoiding deterioration of the TOF image obtained in the effective pixel area.

[0036] A solid-state imaging element according to a ninth aspect of the present disclosure is a solid-state imaging element according to any one of the first to fifth aspects, wherein the pixel array includes an effective pixel area in which at least a portion of the plurality of first pixels are arranged two-dimensionally, and the one or more phase difference detection areas are arranged along sides of the effective pixel area.

[0037] In such a solid-state imaging device, there is no phase difference detection region in the center of the effective pixel region, so that deterioration of the TOF image obtained in the effective pixel region can be avoided.

[0038] A solid-state imaging element according to a tenth aspect of the present disclosure is a solid-state imaging element according to any one of the first to fifth aspects, wherein the one or more phase difference detection regions are a plurality of phase difference detection regions, and the plurality of phase difference detection regions are arranged along a predetermined direction.

[0039] In such a solid-state imaging device, the distance between the TOF pixel and the phase difference pixel can be reduced, and therefore the distance value obtained by the TOF pixel and the distance value obtained by the phase difference pixel can be reduced.

[0040] A solid-state imaging element according to an eleventh aspect of the present disclosure is a solid-state imaging element according to any one of the first to fifth aspects, wherein the one or more phase difference detection regions include a phase difference detection region in which the second pixels are aligned along a first direction, and a phase difference detection region in which the second pixels are aligned along a region in which the second pixels are aligned in a second direction perpendicular to the first direction.

[0041] Such a solid-state imaging device can perform phase difference detection in two orthogonal directions.

[0042] A distance imaging device according to a twelfth aspect of the present disclosure includes a solid-state imaging element according to any one of the first to eleventh aspects; a light source that irradiates an object with the irradiation light; a distance calculation unit that calculates a first distance to the object based on the distance detection signals generated by the plurality of first pixels; a drive control unit that outputs the light emission signal that instructs the light source to irradiate the irradiation light and outputs the exposure signal that instructs the solid-state imaging element to expose the plurality of pixels; and a correction processing unit that calculates a second distance to the object based on the phase difference detection signals generated by the plurality of second pixels and performs processing to correct the first distance based on the first distance and the second distance.

[0043] The distance image capturing device can correct the first distance calculated based on the distance detection signals generated by the TOF pixels, based on the second distance calculated based on the phase difference detection signals generated by the phase difference pixels. With this distance image capturing device, it is possible to easily correct the distance measurement value even when a time lag occurs between the irradiation timing and the exposure timing, such as when applied to an endoscope system.

[0044] A distance imaging device according to a thirteenth aspect of the present disclosure is a distance imaging device according to the twelfth aspect, wherein the correction processing unit adjusts the timing at which the drive control unit outputs the light emission signal and the exposure signal so as to correct the difference between the first distance and the second distance.

[0045] Such a distance imaging device can correct the distance measurement value by adjusting the timing at which the drive control unit issues a drive command, so that the correct distance image is output, which contributes to reducing the amount of processing that is performed after the distance image is generated.

[0046] A distance imaging device according to a fourteenth aspect of the present disclosure is the distance imaging device according to the twelfth aspect, wherein the correction processing unit corrects the value of the first distance based on the difference between the first distance and the second distance.

[0047] In such a distance imaging device, the correction processing unit calculates the difference between the first distance and the second distance and can correct the distance measurement value by subtracting this difference, so that correction can be performed by processing after the distance image is generated without changing the drive timing.

[0048] A distance imaging device according to a fifteenth aspect of the present disclosure is a distance imaging device according to any one of the twelfth to fourteenth aspects, wherein the correction processing unit performs processing to correct the first distance when there is a contrast difference in the reflected light incident on the one or more phase difference detection areas.

[0049] Such a range imaging device automatically calculates the second distance when it is possible to calculate the second distance, thereby making it possible to easily maintain the corrected state even in situations where the environment is changing rapidly.

[0050] A distance imaging device according to a sixteenth aspect of the present disclosure includes a solid-state imaging element having a pixel array in which a plurality of pixels, including a plurality of first pixels and a plurality of second pixels, are arranged two-dimensionally; a light source that irradiates an object with irradiation light; a distance calculation unit; a drive control unit; and a correction processing unit, wherein each of the plurality of first pixels exposes reflected light of the irradiation light reflected by the object based on an exposure signal timed in conjunction with an emission signal to generate a distance detection signal; each of the plurality of second pixels generates a phase difference detection signal based on light incident on the reflected light from a pupil-divided region; the distance calculation unit calculates a first distance to the object based on the distance detection signals generated by the plurality of first pixels; the drive control unit outputs the emission signal to the light source and outputs the exposure signal to the solid-state imaging element; the correction processing unit calculates a second distance to the object based on the phase difference detection signals generated by the plurality of second pixels and performs processing to correct the first distance based on the first distance and the second distance.

[0051] According to this range imaging device, the first distance calculated based on the distance detection signals generated by the TOF pixels can be corrected based on the second distance calculated based on the phase difference detection signals generated by the phase difference pixels. With this range imaging device, it is possible to easily correct the distance measurement value even when a time lag occurs between the irradiation timing and the exposure timing, such as when applied to an endoscope system.

[0052] A distance measuring method according to a seventeenth aspect of the present disclosure is a distance measuring method using a solid-state imaging element according to any one of the first to eleventh aspects, and includes a drive control step of outputting the light emission signal to a light source that irradiates the object with the irradiation light and outputting the exposure signal to the solid-state imaging element, a distance calculation step of acquiring the distance detection signal from the solid-state imaging element and calculating a first distance to the object based on the distance detection signal, and a correction processing step of calculating a second distance to the object based on the phase difference detection signals generated by the plurality of second pixels and performing processing to correct the first distance based on the first distance and the second distance.

[0053] According to this distance measurement method, the first distance calculated based on the distance detection signals generated by the TOF pixels can be corrected based on the second distance calculated based on the phase difference detection signals generated by the phase difference pixels. According to this distance imaging device, even when a time lag occurs between the irradiation timing and the exposure timing, such as when applied to an endoscope system, the distance measurement value can be easily corrected.

[0054] A distance measuring method according to an eighteenth aspect of the present disclosure is a distance measuring method performed by a distance imaging device, the distance imaging device including a light source that irradiates an object with irradiation light, and a solid-state imaging element including a pixel array in which a plurality of pixels, including a plurality of first pixels and a plurality of second pixels, are arranged two-dimensionally, each of the plurality of first pixels performing exposure to light reflected from the object by the irradiation light based on an exposure signal timed in conjunction with an emission signal to generate a distance detection signal, and each of the plurality of second pixels generating a phase difference detection signal based on light incident on the reflected light from a pupil-divided region, the distance measuring method including a drive control step of outputting the emission signal to the light source and the exposure signal to the solid-state imaging element, a distance calculation step of calculating a first distance to the object based on the distance detection signals generated by the plurality of first pixels, and a correction processing step of calculating a second distance to the object based on the phase difference detection signals generated by the plurality of second pixels, and performing processing to correct the first distance based on the first distance and the second distance.

[0055] According to this distance measurement method, the first distance calculated based on the distance detection signals generated by the TOF pixels can be corrected based on the second distance calculated based on the phase difference detection signals generated by the phase difference pixels. According to this distance imaging device, even when a time lag occurs between the irradiation timing and the exposure timing, such as when applied to an endoscope system, the distance measurement value can be easily corrected.

[0056] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0057] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, components, component placement and connection configurations, steps, and step sequences shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, among the components in the following embodiments, components not recited in independent claims are described as optional components. Each figure is a schematic diagram and is not necessarily an exact illustration. In each figure, substantially identical components are designated by the same reference numerals, and redundant descriptions may be omitted or simplified.

[0058] Furthermore, in this specification, terms indicating the relationship between elements, such as perpendicular, parallel, or coincident, terms indicating the shape of elements, such as circular or rectangular, and numerical ranges are not expressions that only express a strict meaning, but are expressions that also include a substantially equivalent range, for example, a difference of about a few percent.

[0059] Furthermore, in this specification, unless otherwise specified, ordinal numbers such as "first" and "second" do not refer to the number or order of components, etc., but are used for the purpose of avoiding confusion and distinguishing between components of the same type.

[0060] In addition, in this specification, "connection" between elements means electrical connection unless otherwise specified.

[0061] (Embodiment) [Configuration] First, the configuration of a range imaging device according to an embodiment will be described. The range imaging device according to the embodiment is applied to, for example, an endoscope system.

[0062] [Distance Imaging Apparatus] First, a distance imaging apparatus according to an embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the functional configuration of a distance imaging apparatus 1 according to this embodiment.

[0063] 1, the range imaging device 1 is an example of a range imaging device. As shown in FIG. 1, the range imaging device 1 can measure the distance to the object 2 by irradiating an object 2 with irradiation light L1 and receiving light L2 reflected by the object 2 from the irradiated irradiation light L1. The range imaging device 1 may also be capable of obtaining a visible light image of the object 2.

[0064] 1, the range imaging device 1 includes a main body 10 and an insertion section 20. The range imaging device 1 is used by inserting the insertion section 20 into the body of a human or animal, etc. In other words, the target object 2 is a part of the body.

[0065] The main body 10 is a part of the range imaging device 1 that is not inserted into the body. As shown in Fig. 1 , the main body 10 includes a light source 11, a light source drive circuit 12, an ISP (Image Signal Processor) 13, an output unit 14, a system control circuit 15, and a power supply IC (Integrated Circuit) 16.

[0066] The light source 11 emits light to illuminate the target object 2. Specifically, the light source 11 can emit flashing light of a predetermined wavelength for distance measurement. The predetermined wavelength is, for example, included in the visible light band. The visible light band is, for example, a wavelength band of 380 nm to 780 nm. The flashing light is light that periodically changes brightness. The flashing light has a period of, for example, 1 MHz to 200 MHz, for example, 50 MHz, but is not limited thereto. The light source 11 is realized by, for example, an LED (Light Emitting Diode), a semiconductor laser element, or an organic EL (Electroluminescence) element. For example, the light source 11 includes a blue LED or blue laser element that emits blue light and a yellow phosphor that is excited by the blue light to emit yellow light, and emits white light as a mixture of the blue light and the yellow light. The predetermined wavelength of the light emitted by the light source 11 may be included in the near-infrared band, and the light source 11 may be realized by, for example, a near-infrared laser element that emits near-infrared light as blinking light. Alternatively, the light source 11 may emit light that is not blinking light or does not blink.

[0067] The light source drive circuit 12 is a circuit that drives the light source 11, and specifically controls the timing of turning on and off the light source 11. The light source drive circuit 12 controls the timing of turning on and off the light source 11, for example, based on a light emission control pulse acquired from the ISP 13. The light source drive circuit 12 can control the timing of turning on and off the light source 11 by adjusting the timing of starting and stopping the supply of power to the light source 11.

[0068] The light source drive circuit 12 is configured by combining one or more of various electronic components such as ICs, resistors, transistors, diodes, capacitors, inductors, and transformers. The light source drive circuit 12 may be configured integrally with other components such as a power supply IC 16 or a system control circuit 15.

[0069] The ISP 13 is an example of an arithmetic circuit, and processes the signal output from the solid-state imaging device 100, i.e., the sensor output data. The ISP 13 calculates the distance to the object 2 using the sensor output data. The ISP 13 generates a distance image indicating the distance to the object 2 for each pixel. The distance image is a still image, but may also be a moving image (video). The ISP 13 may also generate a visible light image based on the sensor output data. The visible light image is, for example, an RGB image, which is an image including brightness values ​​of each of the RGB components. Specifically, the ISP 13 has a drive control unit 131, a distance calculation unit 132, and a correction processing unit 133.

[0070] The drive control unit 131 outputs a light emission control pulse to the light source drive circuit 12. The light emission control pulse is an example of a light emission signal. The light emission control pulse is a signal that instructs the light source 11 on the timing of light irradiation. The drive control unit 131 also outputs a sensor control pulse to the solid-state imaging element 100. The sensor control pulse is an example of an exposure signal. The sensor control pulse is a signal that instructs the solid-state imaging element 100 on the timing of exposure.

[0071] The distance calculation unit 132 calculates a distance value to the object 2 based on distance detection signals generated by a plurality of TOF pixels. The TOF pixels are an example of first pixels. The distance value to the object 2 calculated based on the distance detection signals is an example of a first distance.

[0072] The correction processing unit 133 calculates a distance value to the object 2 based on the phase difference detection signals generated by the multiple phase difference pixels. Furthermore, the correction processing unit 133 performs processing to correct the distance value to the object 2 calculated based on the distance detection signals, based on the distance value to the object 2 calculated based on the distance detection signals calculated by the distance calculation unit 132 and the distance value to the object 2 calculated based on the phase difference detection signals calculated by the correction processing unit 133. The phase difference pixels are an example of second pixels. The distance value to the object 2 calculated based on the phase difference detection signals is an example of a second distance.

[0073] The output unit 14 outputs the distance image generated by the ISP 13. For example, the output unit 14 is a communication IF for wired or wireless communication with an external device such as a display. For example, the output unit 14 is an output terminal to which a communication cable can be connected. Alternatively, the output unit 14 may include an antenna and a wireless processing circuit.

[0074] The system control circuit 15 performs overall control of the range imaging device 1. Specifically, the system control circuit 15 instructs the range imaging device 1 to perform distance measurement and correct the distance measurement value. The system control circuit 15 is realized, for example, by an integrated circuit, such as an LSI (Large Scale Integration). Note that the integrated circuit is not limited to an LSI and may be a dedicated circuit or a general-purpose processor. For example, the system control circuit 15 may be a microcontroller. The microcontroller includes, for example, a non-volatile memory storing a program, a volatile memory serving as a temporary storage area for executing the program, input / output ports, and a processor for executing the program. The system control circuit 15 may also be a programmable FPGA (Field Programmable Gate Array) or a reconfigurable processor in which the connections and settings of circuit cells within the LSI can be reconfigured. The functions performed by the system control circuit 15 may be realized by software or hardware.

[0075] The power supply IC 16 is a power supply circuit that processes power supplied from an external power source such as a commercial power source or a power storage device, etc. For example, the power supply IC 16 includes an AC / DC converter and / or a DC / DC converter.

[0076] 1, the range imaging device 1 may include an operation unit that accepts operation input from a user. The operation unit may be a physical operation button or operation switch, or may be a touch panel, etc. The operation unit may also be a communication IF that accepts operation input via a remote control terminal such as a remote controller.

[0077] The insertion section 20 is a flexible part, and when the distance imaging device 1 is used, at least the tip portion (the end opposite to the main body section 10) is inserted into the body. The length of the insertion section 20 is, for example, 1 m or more and 2 m or less when the object 2 is the upper and lower digestive tract. Note that the length of the insertion section 20 is not particularly limited. As shown in FIG. 1 , the insertion section 20 includes a light-guiding member 21, an objective lens 22, a condenser lens 23, and a solid-state image sensor (CIS (Contact Image Sensor)) 100.

[0078] The light guide member 21 guides the light emitted by the light source 11 to the tip portion, and emits the light as irradiation light L1 from the tip portion toward the object 2. The light guide member 21 is, for example, an optical fiber, but is not limited to this.

[0079] The objective lens 22 and the condenser lens 23 are an optical system for receiving reflected light L2 from the object 2 at the solid-state imaging element 100. The type and number of lenses are not particularly limited as long as the reflected light L2 can be received by the solid-state imaging element 100. Optical elements other than lenses may also be provided.

[0080] The solid-state imaging element 100 outputs an image signal obtained by photoelectrically converting the reflected light L2 from the object 2. Specifically, the solid-state imaging element 100 performs exposure at a timing based on a sensor control pulse obtained from the ISP 13, and generates an image signal. The solid-state imaging element 100 outputs the image signal to the ISP 13 as sensor output data.

[0081] The solid-state imaging device 100 is connected to the ISP 13 via one or more cables (not shown). The solid-state imaging device 100 operates based on a sensor control pulse transmitted from the ISP 13 via the cable. The sensor control pulse is, for example, a master clock MCLK, a vertical synchronization signal VD, or the like. In this embodiment, the solid-state imaging device 100 is a so-called CMOS (Complementary Metal-Oxide-Semiconductor) image sensor (CIS). The solid-state imaging device 100 is, for example, a back-illuminated CIS.

[0082] [Solid-State Image Pickup Device] Next, a specific configuration of the solid-state image pickup device 100 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the functional configuration of the solid-state image pickup device 100 according to this embodiment.

[0083] 2 , the solid-state imaging device 100 includes a pixel array 110, a drive circuit 130, and a signal output circuit 140. The solid-state imaging device 100 also includes a plurality of control lines 150, a plurality of power supply lines (not shown), and a plurality of vertical signal lines 160.

[0084] The pixel array 110 includes a plurality of pixels 120 arranged two-dimensionally in a matrix. The plurality of pixels 120 includes a plurality of first pixels and a plurality of second pixels. Each of the plurality of first pixels exposes reflected light L2, which is illumination light L1 reflected by an object 2, based on a sensor control pulse whose timing is linked to a light emission control pulse, to generate a signal for distance detection. Each of the plurality of second pixels generates a signal for phase difference detection based on light incident on the reflected light L2 from a pupil-divided region. The number of pixels 120 is, for example, several hundred or several thousand or more in each of the row and column directions, but is not limited thereto. The pixels 120 will be described in detail below. Each of the plurality of first pixels may also be capable of generating a signal for a visible light image.

[0085] 2, the area surrounded by a dashed line in the pixel array 110 is the effective pixel area. The effective pixel area is an area for generating a TOF image using first pixels. For example, dummy pixels are also called dummy pixels, and in FIG. 2, they are arranged around the effective pixel area. Note that dummy pixels do not necessarily have to be provided, and all of the pixels 120 included in the pixel array 110 may be first pixels.

[0086] The drive circuit 130 is a circuit that controls each of the multiple pixels 120. The drive circuit 130 and each pixel 120 are electrically connected by multiple control lines 150. The drive circuit 130 drives each control line 150. Specifically, the drive circuit 130 outputs a control signal to each control line 150 to control the exposure timing, signal output timing, etc. of each pixel included in each pixel 120.

[0087] The signal output circuit 140 is connected to each pixel 120 via a plurality of vertical signal lines 160. The signal output circuit 140 outputs each pixel 120 or an image signal read out from each pixel via the plurality of vertical signal lines 160 to the ISP 13 as sensor output data.

[0088] [Pixel] The configuration and arrangement of the pixel 120 will be described in detail. FIG. 3 is a circuit diagram showing an example of the circuit configuration of the pixel 120 according to the embodiment. The pixel 120 includes a photoelectric conversion unit 121, a transfer transistor 122, a charge accumulation unit 123, a charge discharge transistor 124, a source follower transistor 125, a selection transistor 126, and a reset transistor 127. The transfer transistor 122, the charge accumulation unit 123, the charge discharge transistor 124, the source follower transistor 125, the selection transistor 126, and the reset transistor 127 are, for example, N-channel MOSFETs (Metal Oxide Semiconductor FETs). The circuit configuration of the pixel 120 shown in FIG. 3 is applied to, for example, both a first pixel that generates a signal for distance detection and a second pixel that generates a signal for phase difference detection. Note that the circuit configuration of the pixel 120 shown in FIG. 3 is merely an example and can be modified as appropriate. Furthermore, the first pixel and the second pixel may have different circuit configurations.

[0089] The photoelectric conversion unit 121 generates electric charges by converting incident light incident on the pixel 120 into electric charges. The incident light incident on the pixel 120 includes, for example, pulsed light emitted from the light source 11 and reflected by the object 2. The photoelectric conversion unit 121 is composed of, for example, a photoelectric conversion element such as a photodiode. The photoelectric conversion unit 121 is connected to one of the source and drain of the transfer transistor 122 and one of the source and drain of the charge discharge transistor 124.

[0090] The transfer transistor 122 transfers the charges converted by the photoelectric conversion unit 121 from the photoelectric conversion unit 121 to the charge accumulation unit 123. The other of the source and drain of the transfer transistor 122 is connected to the charge accumulation unit 123. A transfer control signal TG that controls the on / off of the transfer transistor 122 is applied to the gate of the transfer transistor 122. The transfer control signal TG is a sensor exposure pulse generated inside the solid-state imaging device 100 based on a sensor control pulse obtained from the ISP 13.

[0091] The charge accumulation unit 123 accumulates the charges converted by the photoelectric conversion unit 121 and transferred by the transfer transistor 122. Specifically, the charge accumulation unit 123 accumulates the charges converted by the photoelectric conversion unit 121 while the pixel 120 is exposed to light. The charge accumulation unit 123 is, for example, an impurity region formed on a semiconductor substrate and capable of being in a floating state. The charge accumulation unit 123 may also include wiring capacitance.

[0092] The charge discharging transistor 124 discharges the charge converted by the photoelectric conversion unit 121 to the outside of the pixel 120. A power supply voltage VDD is applied to the other of the source and drain of the charge discharging transistor 124. A charge discharging control signal PRS that controls the on / off of the charge discharging transistor 124 is applied to the gate of the charge discharging transistor 124.

[0093] The source follower transistor 125 outputs a signal corresponding to the potential of the charge storage unit 123. That is, the source follower transistor 125 reads out the charge stored in the charge storage unit 123. The gate of the source follower transistor 125 is connected to the charge storage unit 123. A power supply voltage VDD is applied to one of the source and drain of the source follower transistor 125. The other of the source and drain of the source follower transistor 125 is connected to one of the source and drain of the selection transistor 126, and the source follower transistor 125 outputs a signal corresponding to the potential of the charge storage unit 123 to the bit line BL via the selection transistor 126. The source follower transistor 125 forms a source follower circuit together with a current source connected to the bit line BL.

[0094] The other of the source and drain of the selection transistor 126 is connected to a bit line BL, which is a signal line through which a signal from the pixel 120 is output. A selection control signal SEL that controls the on / off of the selection transistor 126 is applied to the gate of the selection transistor 126. When the selection transistor 126 is turned on, charge is read out by the source follower transistor 125. The signal from the pixel 120 output to the bit line BL is converted into a digital signal by, for example, an AD conversion circuit (not shown) and input to the distance calculation unit 132.

[0095] The reset transistor 127 resets the potential of the charge storage unit 123. One of the source and drain of the reset transistor 127 is connected to the charge storage unit 123. The power supply voltage VDD is applied to the other of the source and drain of the reset transistor 127. A reset control signal RS that controls the on / off of the reset transistor 127 is applied to the gate of the reset transistor 127. When the reset transistor 127 is turned on, the potential of the charge storage unit 123 is reset to the power supply voltage VDD.

[0096] The transfer control signal TG, the charge discharge control signal PRS, the selection control signal SEL, and the reset control signal RS are generated by the drive circuit 130 based on control signals from the drive control unit 131 (ISP 13), for example.

[0097] Next, a description will be given of the arrangement of the pixels 120. Fig. 4 is a diagram showing an example of the arrangement of the pixels 120 in the pixel array 110 according to the embodiment.

[0098] As shown in FIG. 4 , the pixel array 110 is provided with an effective pixel area 210 and a phase difference detection area 220 .

[0099] 4, the effective pixel area 210 is set in a rectangular shape in the center of the pixel array 110. The effective pixel area 210 is an area for generating a TOF image. First pixels that generate signals for distance detection are arranged in the effective pixel area 210.

[0100] 4 , the phase difference detection region 220 is arranged along the periphery of the rectangular effective pixel region 210 so as to surround the region. The phase difference detection region 220 is an area for detecting a phase difference. In the phase difference detection region 220, second pixels that generate signals for phase difference detection are arranged.

[0101] 4, the phase difference detection region 220 is realized by second pixels arranged in one row, but the second pixels may be arranged in multiple rows, which is disadvantageous in terms of chip size but makes it easier to detect distance values ​​using the second pixels.

[0102] 4, the phase difference detection region 220 is disposed along all four sides of the effective pixel region 210, but the phase difference detection region 220 does not have to be disposed on all four sides. This makes it difficult for the second pixel to detect the distance value, but is advantageous in terms of chip size.

[0103] Here, the light receiving regions of the second pixels are divided in the same direction as the arrangement of the phase difference detection regions 220. That is, the second pixels included in the phase difference detection regions 220 arranged in the left-right direction are divided into light receiving regions in the left-right direction, and the second pixels included in the phase difference detection regions 220 arranged in the up-down direction are divided into light receiving regions in the up-down direction. In the example shown in FIG. 4 , for each second pixel, the areas marked with R, L, B, and T are light receiving regions, i.e., openings, and the black regions are regions where light is not received. The regions where light is not received can be realized, for example, by providing a light blocking screen, but this is not limited thereto.

[0104] Note that pixels 120 other than the first pixels may be arranged in the effective pixel region 210. Second pixels, dummy pixels, etc. may be arranged in the effective pixel region 210. Furthermore, pixels 120 other than the second pixels may be arranged in the phase difference detection region 220. First pixels, dummy pixels, etc. may be arranged in the phase difference detection region 220.

[0105] Fig. 5 is a diagram illustrating a situation in which the distance imaging device 1 according to the embodiment is used as an endoscope system. Fig. 5(a) is a diagram illustrating the manner in which the distance imaging device 1 performs imaging. Fig. 5(b) is a diagram illustrating an example of a distance image generated by imaging in Fig. 5(a). In the distance image shown in Fig. 5(b), the darker the black, the greater the distance value.

[0106] Consider a situation in which the insertion section 20 of the distance imaging device 1 is inserted into a tubular object 2 to capture an image of the object 2 in the tubular direction, as shown in (a) of Figure 5. As shown in (b) of Figure 5, the distance image obtained is farther away the closer to the center, and closer to the periphery, as shown in (b) of Figure 5.

[0107] Here, it is generally known that a distance measurement characteristic of a phase difference pixel is that the closer the distance to the subject, the more accurately the phase difference pixel can measure the distance. Therefore, as shown in Fig. 4, by providing a phase difference detection region 220 on the periphery of the pixel array 110 and arranging second pixels (phase difference pixels), the distance measurement value by the second pixels can be calculated more accurately, which contributes to improving the accuracy of correction.

[0108] Furthermore, by placing the phase difference detection region 220 outside the effective pixel region 210, it becomes easier to change and control the pixel size of the second pixel included in the phase difference detection region 220, and deterioration of the TOF image obtained in the effective pixel region 210 can be avoided.

[0109] [Operation] Next, a description will be given of the operation of the distance imaging device 1 according to the embodiment. The distance imaging device 1 according to the embodiment can correct the phase difference between the irradiation pulse and the exposure pulse, which occurs when the light source 11 that emits light and the solid-state image sensor 100 that exposes the reflected light are disposed at separate positions.

[0110] The operation of the range imaging device 1 when used as an endoscope will now be described with reference to the flowchart of FIG.

[0111] First, 3D imaging (before correction) is started by the range imaging device 1 (S101). In other words, distance measurement is started by the range imaging device 1. That is, the drive control unit 131 outputs a light emission control pulse to the light source 11 and outputs a sensor control pulse to the solid-state imaging element 100.

[0112] Next, the endoscope is inserted into the body (S102). That is, the insertion section 20 of the distance imaging device 1 is inserted into the body. Since the light source 11 is provided in the main body 10 and the solid-state imaging element 100 is provided near the tip of the insertion section 20, the longer the insertion section 20, the greater the distance between the light source 11 and the solid-state imaging element 100, and the greater the phase difference due to propagation delay between the sensor exposure pulse and the irradiation light pulse at the tip of the cable. Furthermore, the phase difference changes depending on the bending of the insertion section 20, i.e., the shape inside the body. Furthermore, the phase difference also changes depending on the temperature inside the body, etc. Note that the order of the processes of steps S101 and S102 may be reversed, or they may be performed simultaneously.

[0113] Next, the distance calculation unit 132 calculates the TOF distance measurement value before correction (S103). Specifically, the distance calculation unit 132 calculates the first distance to the object 2 based on the distance detection signals generated by the plurality of first pixels. The procedure for calculating the first distance will be described later.

[0114] Here, it is determined whether or not there is an object with a contrast difference in the imaging range of the phase difference pixels (S104). FIG. 7 is a diagram showing how parallax is detected by the phase difference pixels. The solid line in the graph shown in FIG. 7 indicates the output by the right-opening pixel among the phase difference pixels, the dashed line in the graph indicates the output by the left-opening pixel among the phase difference pixels, and the double-headed arrow indicates the parallax detected by the phase difference pixels. As shown in FIG. 7, when the output by the right-opening pixel and the output by the left-opening pixel are equal, that is, when the focus position and the object distance are equal, no parallax occurs. When the object position is out of focus, the output by the right-opening pixel and the output by the left-opening pixel differ, and therefore parallax occurs. In other words, parallax can be detected only when there is a contrast difference in the object.

[0115] Here, the determination is made based on the actual imaging results obtained by the phase difference pixels, for example, or may be made by a user of the range imaging device 1, for example, a doctor.

[0116] If there is no subject with a contrast difference in the imaging range of the phase difference pixels (No in S104), the range imaging device 1 returns to the processing of step S103 and continues generating the TOF image.

[0117] If an object having a contrast difference is present in the imaging range of the phase difference pixels (Yes in S104), the correction processing unit 133 calculates a distance measurement value using the phase difference pixels (S105). Specifically, the correction processing unit 133 calculates a second distance to the object 2 based on the phase difference detection signals generated by the plurality of second pixels. The procedure for calculating the second distance will be described later.

[0118] Next, the correction processing unit 133 calculates a correction value (S106). Specifically, the correction processing unit 133 calculates the correction value for correcting the first distance based on the first distance calculated in step S103 and the second distance calculated in step S105.

[0119] Here, the correction processing unit 133 identifies, among the multiple second pixels, a second pixel that generated the phase difference detection signal used to calculate the second distance in step S105. The correction processing unit 133 identifies, among the multiple first pixels, a first pixel that is arranged near the identified second pixel. The correction processing unit 133 calculates a correction value based on the first distance calculated based on the distance detection signal generated by the identified first pixel and the second distance calculated based on the phase difference detection signal generated by the identified second pixel. This makes it possible to improve the accuracy of the correction.

[0120] Next, the correction processing unit 133 performs TOF correction (S107). The correction processing unit 133 performs TOF correction based on the calculated correction value. The TOF correction method will be described later.

[0121] Finally, the range imaging device 1 starts 3D imaging (after correction) (S108).

[0122] According to this operation, the phase difference between the timing of irradiation light and the timing of exposure, which is caused by the large distance between the light source 11 and the solid-state imaging element 100, can be corrected using the distance measurement value obtained by the phase difference pixels. In other words, the first distance calculated based on the distance detection signals generated by the TOF pixels can be corrected based on the second distance calculated based on the phase difference detection signals generated by the phase difference pixels. Therefore, according to this range imaging device 1, the distance measurement value can be easily corrected.

[0123] The correction of the first distance may be performed by, for example, pressing a button at a timing desired by the user. For example, the user may be able to instruct the correction to be performed via an operation unit provided in the range imaging device 1.

[0124] Furthermore, the correction of the first distance may be performed when there is an environmental change such as a change in temperature. For example, the insertion section 20 may be equipped with a temperature sensor, and the correction may be performed when there is a change in the temperature of the tip of the cable.

[0125] The correction of the first distance may be performed based on a correction value calculated in a previous correction. For example, the correction value may be stored in the system control circuit 15, and the correction may be performed based on the correction value.

[0126] Furthermore, the correction of the first distance may be performed while the insertion section 20 is set in a calibration case before being intubated into the body. The calibration case is, for example, a case that reproduces the environment, such as the shape and temperature, of the target object 2. In other words, the correction may be performed before the processing of step S102.

[0127] The correction of the first distance may be repeated at regular intervals. That is, the correction may be performed automatically at predetermined intervals so as to be able to respond to changes in the environment. The predetermined interval may be, for example, one minute or 30 seconds, but is not particularly limited here.

[0128] Furthermore, if there is no contrast difference in the imaging range of the phase difference pixel at the timing when the correction of the first distance is instructed, the user may be notified that the correction cannot be executed.

[0129] [Calculation of First Distance] The procedure for calculating the first distance will be described. That is, the procedure for the distance calculation unit 132 to calculate the first distance in step S103 will be described. Fig. 8 is a timing chart illustrating the imaging operation before correction of the range imaging device according to the embodiment.

[0130] 8, the drive control unit 131 of the ISP 13 outputs a sensor control pulse and an irradiation light pulse. The light source 11 repeatedly irradiates the target object 2 with pulsed light having a pulse width (period) Tp as irradiation light at predetermined time intervals.

[0131] The reflected light is delayed by a certain time ΔT from the pulsed light depending on the distance to the target object 2. In the pulse ToF method, light is received in at least two exposure intervals set at different times relative to the pulsed light. In the example shown in Fig. 8, a first exposure interval is set as an interval with a pulse width Tp that starts at the same timing as the pulsed light, and a second exposure interval is set as an interval with a pulse width Tp that starts simultaneously with the cessation of the pulsed light.

[0132] The first distance Z to the object 2 is expressed by the following equation (1) based on the delay time ΔT.

[0133]

[0134] A0 and A1 are the intensities of the reflected light received in the first exposure interval and the second exposure interval, respectively.

[0135] In the pulse ToF method, the accuracy of distance measurement can be improved by shortening the pulse width Tp. On the other hand, as the pulse width Tp becomes shorter, the measurable range (distance measurement range) becomes smaller. This is because the maximum measurable distance Z, Zmax, occurs when A0 = 0 in equation (1) and is expressed as (c × Tp) / 2. To address the reduction in the distance measurement range, the distance measurement range can be expanded by performing distance measurement with multiple exposures and combining the distance measurement results.

[0136] A third exposure interval may be set as an interval in which reflected light cannot be received. The third exposure interval is set to detect background light components. In this case, A0 is obtained by subtracting the signal intensity (background light component) obtained in the third exposure interval from the signal intensity obtained in the first exposure interval, and A1 is obtained by subtracting the signal intensity (background light component) obtained in the third exposure interval from the signal intensity obtained in the second exposure interval.

[0137] [Error Included in First Distance] FIG. 8 shows how the sensor control pulse generated by the drive control unit 131 of the ISP 13 and the irradiated light pulse irradiated by the light source 11 are delayed in propagation.

[0138] The drive control unit 131 generates a light emission control pulse and outputs it to the light source drive circuit 12. The light source drive circuit 12 drives the light source 11 in accordance with the light emission control pulse, and the light source 11 emits an irradiation light pulse. The irradiation light pulse emitted by the light source 11 is emitted from the distal end of the insertion section 20 via the light guiding member 21. As shown in Fig. 8 , the irradiation light pulse at the illumination lens at the distal end of the cable is delayed compared to the irradiation light pulse in the endoscope body by the time it takes for the pulse to pass through the light guiding member 21. In Fig. 8 , the magnitude of this delay is indicated by a diagonally shaded right arrow.

[0139] The drive control unit 131 generates a sensor control pulse and outputs it to the solid-state imaging device 100. The solid-state imaging device 100 generates a sensor exposure pulse based on the acquired sensor control pulse and performs exposure using reflected light. As shown in FIG. 8 , the sensor control pulse acquired by the solid-state imaging device 100 is delayed compared to the sensor control pulse output by the drive control unit 131 by the time it takes for the sensor control pulse to reach the tip of the cable. In FIG. 8 , the magnitude of this delay is indicated by a hollow right arrow. The sensor exposure pulse generated based on the sensor control pulse is delayed by the same amount as the delay of the sensor control pulse.

[0140] Here, as described above, in order to calculate the accurate value of the first distance Z based on equation (1), the first exposure section must start at the same timing as the pulsed light, and the second exposure section must start at the same time as the pulsed light stops.

[0141] However, as shown in Fig. 8, the magnitude of the delay in the illumination light pulse and the magnitude of the delay in the sensor exposure pulse differ, resulting in a phase difference between the illumination light pulse and the sensor exposure pulse. In Fig. 8, the magnitude of the phase difference between the illumination light pulse and the sensor exposure pulse is indicated by a hollow double-headed arrow. The first distance Z calculated in step S103 includes an error resulting from this phase difference.

[0142] [Calculation of Second Distance] The procedure for calculating the second distance will be described below, that is, the procedure for the correction processing unit 133 to calculate the second distance in step S105 of FIG.

[0143] 9A and 9B are diagrams illustrating the relationship between parallax and phase difference. Fig. 9A is a diagram illustrating the relationship between parallax and phase difference when the object 2 is located closer than the focus position. Fig. 9B is a diagram illustrating the relationship between parallax and phase difference when the object 2 is located farther than the focus position.

[0144] In FIG. 9, the distance from the lens to the object 2 is b'.

[0145] Here, if the focal length determined by the lens coefficient is f and the distance between the position where the real image of the object 2 is formed and the lens is a', then b' is given by the following equation (2).

[0146] b'=1 / (1 / f-1 / a') (2)

[0147] Here, if the focal position determined by the lens settings at the time of image capture is set to (a+b), the base line length of the lens determined by the lens coefficient is bl, and the parallax is D, then a' is expressed by the following equation (3).

[0148] a'=bl*a / (bl-D) (3)

[0149] Here, the parallax D is calculated from the output of the second pixel as shown in Fig. 7. Specifically, the parallax D is calculated from the distance between the second pixel that receives the incident light to the L aperture pixel and the second pixel that receives the incident light to the R aperture pixel.

[0150] Therefore, since a and b1 are constants, the value of b' can be obtained based on equations (2) and (3) by calculating the parallax D. In other words, the second distance (= a + b') can be obtained.

[0151] [Correction Method] Next, a TOF correction method will be described, that is, a correction method in the process of step S107 will be described.

[0152] Furthermore, for example, the correction processing unit 133 performs the correction by adjusting the timing at which the drive control unit 131 outputs the light emission control pulse and the sensor control pulse so as to correct the difference between the first distance and the second distance. Fig. 10 is a timing chart illustrating the imaging operation after correction of the distance imaging device according to the embodiment.

[0153] As shown in FIG. 10 , the correction processing unit 133 calculates the difference between the magnitude of the delay of the irradiated light pulse and the magnitude of the delay of the sensor control pulse shown in FIG. 8 , and performs correction by adjusting the timing at which the drive control unit 131 outputs the sensor control pulse to compensate for this difference. Specifically, in FIG. 10 , the drive control unit 131 advances the timing at which the sensor control pulse is output by the amount indicated by the hollow left arrow. In this way, the relationship between the timing of the irradiated light pulse and the timing of the sensor exposure pulse can be adjusted to be appropriate at the position of the solid-state imaging device 100. In other words, this correction method allows the correction of the distance measurement value by adjusting the timing at which the drive control unit 131 issues a drive instruction, thereby outputting a correct distance image through the correction. This contributes to reducing the amount of processing performed after generating the distance image.

[0154] For example, the correction processing unit 133 corrects the value of the first distance based on the difference between the first distance and the second distance. Specifically, the correction processing unit 133 calculates the difference between the first distance Z and the second distance (a+b') as the correction value, and performs the correction by subtracting the difference from the distance value of the entire TOF image.

[0155] According to this correction method, the correction processing unit 133 can correct the distance measurement value by calculating the difference between the first distance and the second distance and subtracting this difference, so that the correction can be performed by processing after the generation of the distance image without changing the drive timing.

[0156] [Modifications] Here, the arrangement of the pixels 120 in the pixel array 110 is not limited to the example described using Fig. 4. Below, examples of the arrangement of the pixels 120 in the pixel array 110, that is, the arrangement of the effective pixel area 210 and the phase difference detection area 220 in the pixel array 110, will be described using Figs.

[0157] A first example of the pixel 120 will be described below. Fig. 11 is a diagram showing a first example of the arrangement of the pixels 120 in the pixel array 110 according to the embodiment.

[0158] As shown in FIG. 11 , a rectangular effective pixel area 210 may be provided in the center of the pixel array 110 , and phase difference detection areas 220 may be provided outside the effective pixel area 210 along the four sides of the effective pixel area 210 .

[0159] A second example of the pixel 120 will be described below. Fig. 12 is a diagram showing a second example of the arrangement of the pixels 120 in the pixel array 110 according to the embodiment.

[0160] As shown in FIG. 12 , a rectangular effective pixel area 210 may be provided in the center of the pixel array 110 , and phase difference detection areas 220 may be provided inside the effective pixel area 210 along the four sides of the effective pixel area 210 .

[0161] A third example of the pixel 120 will be described below. Fig. 13 is a diagram showing a third example of the arrangement of the pixels 120 in the pixel array 110 according to the embodiment.

[0162] As shown in FIG. 13 , a rectangular effective pixel area 210 may be provided in the center of the pixel array 110, a phase difference detection area 220 may be provided outside the effective pixel area 210 along two of the four sides of the effective pixel area 210, and a phase difference detection area 220 may be provided inside the effective pixel area 210 along the other two of the four sides of the effective pixel area 210.

[0163] A fourth example of the pixel 120 will be described below. Fig. 14 is a diagram showing a fourth example of the arrangement of the pixels 120 in the pixel array 110 according to the embodiment.

[0164] As shown in FIG. 14 , a rectangular effective pixel area 210 may be provided in the center of the pixel array 110 , and phase difference detection areas 220 may be provided both outside and inside the effective pixel area 210 along the four sides thereof.

[0165] A fifth example of the pixel 120 will be described below. Fig. 15 is a diagram showing a fifth example of the arrangement of the pixels 120 in the pixel array 110 according to the embodiment.

[0166] As shown in FIG. 15 , a rectangular effective pixel area 210 may be provided in the center of the pixel array 110, and a phase difference detection area 220 may be provided outside the effective pixel area 210 along two of the four sides of the effective pixel area 210.

[0167] A sixth example of the pixel 120 will be described below. Fig. 16 is a diagram showing a sixth example of the arrangement of the pixels 120 in the pixel array 110 according to the embodiment.

[0168] As shown in FIG. 16 , a rectangular effective pixel area 210 may be provided in the center of the pixel array 110, and a phase difference detection area 220 may be provided outside the effective pixel area 210 along one of the four sides of the effective pixel area 210.

[0169] A seventh example of the pixel 120 will be described below. Fig. 17 is a diagram showing a seventh example of the arrangement of the pixels 120 in the pixel array 110 according to the embodiment.

[0170] As shown in FIG. 17 , a rectangular effective pixel area 210 may be provided in the center of the pixel array 110, and a plurality of phase difference detection areas 220 may be provided inside the effective pixel area 210, aligned in the left-right direction.

[0171] Although Figure 17 shows multiple phase difference detection regions 220 arranged side by side in the left-right direction within the effective pixel region 210, the multiple phase difference detection regions 220 may also be arranged side by side in the up-down direction within the effective pixel region 210.

[0172] An eighth example of the pixel 120 will be described below. Fig. 18 is a diagram showing an eighth example of the arrangement of the pixels 120 in the pixel array 110 according to the embodiment.

[0173] 18 , a rectangular effective pixel area 210 may be provided in the center of the pixel array 110, and a plurality of phase difference detection areas 220 arranged along an oblique direction may be provided inside the effective pixel area 210. Note that the oblique direction may be an oblique direction upward to the left or an oblique direction upward to the right.

[0174] The arrangement of the effective pixel area 210 and the phase difference detection area 220 in the pixel array 110 may be realized by any combination of the first to eighth examples. The effective pixel area 210 does not have to be rectangular.

[0175] The chip size can be reduced by providing the phase difference detection region 220 inside the effective pixel region 210. Furthermore, the distance between the first pixel and the second pixel used for correction can be reduced, allowing a more accurate correction value to be calculated.

[0176] Note that the arrangement of the effective pixel region 210 and the phase difference detection region 220 in the pixel array 110 is not limited to the above-described example. The arrangement of the pixels 120 is not particularly limited as long as a phase difference can be detected by the phase difference detection region 220. In other words, the arrangement of the pixels 120 is not particularly limited as long as at least one set of second pixels aligned along a certain direction is provided.

[0177] (Other Embodiments) Although the embodiments have been described above, the present disclosure is not limited to the above-described embodiments.

[0178] Although the control method for a range imaging device according to one or more aspects has been described above based on the embodiments, the present disclosure is not limited to these embodiments. As long as they do not deviate from the gist of the present disclosure, various modifications conceivable by those skilled in the art to the present embodiments and configurations constructed by combining components of different embodiments are also included within the scope of the present disclosure.

[0179] For example, in the above embodiment, the range imaging device calculates distance values ​​using the pulse TOF method, but it may also calculate distance values ​​using a CW (Continuous Wave) TOF method.

[0180] For example, in the above embodiment, an endoscope system is shown as an example of a distance imaging device, but the present invention is not limited to this. The distance imaging device may be, for example, a surveillance camera or an object detection device installed in a predetermined building or structure. Alternatively, the distance imaging device may be a sensor device mounted on a moving object such as an autonomous vehicle or a drone. In these cases, the target object 2 may be a stationary object or a moving object.

[0181] Furthermore, for example, the visible light image does not have to be an RGB image. The visible light image may be a monochromatic image such as a black and white image. Furthermore, for example, the first wavelength of the light used in the first operation mode and the second wavelength of the flashing light used in the second operation mode may be the same.

[0182] Furthermore, the communication method between the devices described in the above embodiment is not particularly limited. When wireless communication is performed between the devices, the wireless communication method (communication standard) is, for example, short-range wireless communication such as ZigBee (registered trademark), Bluetooth (registered trademark), or wireless LAN (Local Area Network). Alternatively, the wireless communication method (communication standard) may be communication via a wide-area communication network such as the Internet. Furthermore, wired communication may be performed between the devices instead of wireless communication. Specifically, wired communication is communication using power line communication (PLC) or a wired LAN.

[0183] Furthermore, in the above-described embodiments, the processing performed by a specific processing unit may be performed by another processing unit. Furthermore, the order of multiple processes may be changed, or multiple processes may be performed in parallel. Furthermore, the allocation of components included in a distance imaging device (ranging system) to multiple devices is an example. For example, components included in one device may be included in another device.

[0184] For example, the processing described in the above embodiments may be realized by centralized processing using a single device (system), or may be realized by distributed processing using multiple devices. Furthermore, the processor that executes the program may be a single processor or multiple processors. That is, centralized processing or distributed processing may be performed.

[0185] In the above-described embodiments, all or some of the components such as the control unit may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU (Central Processing Unit) or a processor reading and executing a software program recorded on a recording medium such as an HDD (Hard Disk Drive) or semiconductor memory.

[0186] Furthermore, the components such as the control unit may be configured with one or more electronic circuits, each of which may be a general-purpose circuit or a dedicated circuit.

[0187] The one or more electronic circuits may include, for example, a semiconductor device, an IC, or an LSI. The IC or LSI may be integrated on a single chip or on multiple chips. Although the IC or LSI is referred to here as an IC or LSI, the name may vary depending on the degree of integration, and may be called a system LSI, a VLSI (Very Large Scale Integration), or an ULSI (Ultra Large Scale Integration). An FPGA, which is programmed after the LSI is manufactured, can also be used for the same purpose.

[0188] Furthermore, the general or specific aspects of the present disclosure may be realized as a system, an apparatus, a method, an integrated circuit, or a computer program. Alternatively, the general or specific aspects may be realized as a computer-readable non-transitory recording medium such as an optical disk, a HDD, or a semiconductor memory on which the computer program is stored. Alternatively, the general or specific aspects of the present disclosure may be realized as any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

[0189] Furthermore, various modifications, substitutions, additions, omissions, etc. can be made to the above-described embodiments within the scope of the claims or their equivalents.

[0190] The present disclosure can be used as an imaging device that can be used for high-precision distance measurement, and can be used in, for example, an endoscope system, a surveillance camera, and the like.

[0191] REFERENCE SIGNS LIST 1 Range imaging device 2 Object 10 Main body 11 Light source 12 Light source driving circuit 13 ISP 131 Drive control unit 132 Distance calculation unit 133 Correction processing unit 14 Output unit 15 System control circuit 16 Power supply IC 20 Insertion unit 21 Light guide member 22 Objective lens 23 Condenser lens 100 Solid-state imaging element 110 Pixel array 120 Pixel 121 Photoelectric conversion unit 122 Transfer transistor 123 Charge storage unit 124 Charge discharge transistor 125 Source follower transistor 126 Selection transistor 127 Reset transistor 130 Drive circuit 140 Signal output circuit 150 Control line 160 Vertical signal line 210 Effective pixel area 220 Phase difference detection area L1 Irradiated light L2 Reflected light

Claims

1. A solid-state imaging device comprising a pixel array in which a plurality of pixels including a plurality of first pixels and a plurality of second pixels are arranged in a two-dimensional manner, wherein each of the plurality of first pixels performs exposure to light reflected by an object based on an exposure signal having a timing linked to an emission signal to generate a signal for distance detection, and each of the plurality of second pixels generates a signal for phase difference detection based on light incident on the reflected light from a pupil-divided region, wherein the pixel array includes one or more phase difference detection regions in which two or more second pixels of the plurality of second pixels are aligned in one direction, and at least a portion of the first pixels of the plurality of first pixels are arranged in a position different from the one or more phase difference detection regions in the pixel array and closer to a center of the pixel array than at least one phase difference detection region of the one or more phase difference detection regions.

2. The solid-state imaging device according to claim 1, wherein the number of the first pixels is greater than the number of the second pixels.

3. The solid-state imaging device according to claim 1 or 2, wherein the plurality of first pixels have a mode for generating signals for a visible light image.

4. The solid-state imaging element according to any one of claims 1 to 3, wherein the one or more phase difference detection regions include a first phase difference detection region in which the second pixels are aligned along a first direction, and a second phase difference detection region in which the second pixels are aligned along a second direction perpendicular to the first direction, and the second pixels aligned in the first phase difference detection region receive light incident from a region formed by pupil division in the first direction, and the second pixels aligned in the second phase difference detection region receive light incident from a region formed by pupil division in the second direction.

5. The solid-state imaging device according to any one of claims 1 to 4, wherein the size of each of the plurality of second pixels is different from the size of each of the plurality of first pixels.

6. A solid-state imaging element according to any one of claims 1 to 5, wherein the pixel array includes an effective pixel area in which at least a portion of the plurality of first pixels are arranged two-dimensionally, and the one or more phase difference detection areas are arranged outside the effective pixel area.

7. A solid-state imaging element according to any one of claims 1 to 5, wherein the pixel array includes an effective pixel area in which at least a portion of the plurality of first pixels are arranged two-dimensionally, and the one or more phase difference detection areas are arranged within the effective pixel area.

8. A solid-state imaging element as described in any one of claims 1 to 5, wherein the pixel array includes an effective pixel area in which at least a portion of the plurality of first pixels are arranged two-dimensionally, and the one or more phase difference detection areas include a phase difference detection area arranged outside the effective pixel area, and a phase difference detection area arranged within the effective pixel area.

9. A solid-state imaging element according to any one of claims 1 to 5, wherein the pixel array includes an effective pixel area in which at least a portion of the plurality of first pixels are arranged two-dimensionally, and the one or more phase difference detection areas are arranged along sides of the effective pixel area.

10. A solid-state imaging element according to any one of claims 1 to 5, wherein the one or more phase difference detection regions are a plurality of phase difference detection regions, and the plurality of phase difference detection regions are arranged in a predetermined direction.

11. A solid-state imaging element according to any one of claims 1 to 5, wherein the one or more phase difference detection regions include a phase difference detection region in which the second pixels are aligned along a first direction, and a phase difference detection region in which the second pixels are aligned along a region in which they are aligned in a second direction perpendicular to the first direction.

12. A distance imaging device comprising: a solid-state imaging element according to any one of claims 1 to 11; a light source which irradiates an object with the irradiation light; a distance calculation unit which calculates a first distance to the object based on the distance detection signals generated by the plurality of first pixels; a drive control unit which outputs the light emission signal which instructs the light source to irradiate the irradiation light, and outputs the exposure signal which instructs the solid-state imaging element to expose the plurality of pixels; and a correction processing unit which calculates a second distance to the object based on the phase difference detection signals generated by the plurality of second pixels, and performs processing to correct the first distance based on the first distance and the second distance.

13. The distance imaging device according to claim 12, wherein the correction processing section adjusts timing at which the drive control section outputs the light emission signal and the exposure signal so as to correct a difference between the first distance and the second distance.

14. The range imaging device according to claim 12, wherein the correction processing section corrects the value of the first distance based on a difference between the first distance and the second distance.

15. The distance imaging device according to any one of claims 12 to 14, wherein the correction processing unit performs processing to correct the first distance when there is a contrast difference in the reflected light incident on the one or more phase difference detection areas.

16. A distance imaging device comprising: a solid-state imaging element having a pixel array in which a plurality of pixels, including a plurality of first pixels and a plurality of second pixels, are arranged in a two-dimensional manner; a light source which irradiates an object with irradiation light; a distance calculation unit; a drive control unit; and a correction processing unit, wherein each of the plurality of first pixels performs exposure to light reflected by the object from the irradiation light based on an exposure signal having a timing linked to an emission signal to generate a signal for distance detection, each of the plurality of second pixels generates a signal for phase difference detection based on light incident on a pupil-divided region from the reflected light, the distance calculation unit calculates a first distance to the object based on the signal for distance detection generated by the plurality of first pixels, the drive control unit outputs the emission signal to the light source and outputs the exposure signal to the solid-state imaging element, and the correction processing unit calculates a second distance to the object based on the signal for phase difference detection generated by the plurality of second pixels, and performs processing to correct the first distance based on the first distance and the second distance.

17. A distance measuring method using a solid-state imaging element described in any one of claims 1 to 11, comprising: a drive control step of outputting the light emission signal to a light source that irradiates the object with the irradiation light, and outputting the exposure signal to the solid-state imaging element; a distance calculation step of acquiring the distance detection signal from the solid-state imaging element and calculating a first distance to the object based on the distance detection signal; and a correction processing step of calculating a second distance to the object based on the phase difference detection signals generated by the plurality of second pixels, and performing processing to correct the first distance based on the first distance and the second distance.

18. A distance measuring method performed by a distance imaging device, wherein the distance imaging device comprises: a light source which irradiates an object with irradiation light; and a solid-state imaging element having a pixel array in which a plurality of pixels, including a plurality of first pixels and a plurality of second pixels, are arranged two-dimensionally, wherein each of the plurality of first pixels exposes the reflected light of the irradiation light reflected by the object based on an exposure signal having a timing linked to an emission signal to generate a signal for distance detection, and each of the plurality of second pixels generates a signal for phase difference detection based on light incident on the reflected light from a pupil-divided region, and the distance measuring method comprises: a drive control step of outputting the emission signal to the light source and the exposure signal to the solid-state imaging element; a distance calculation step of calculating a first distance to the object based on the signals for distance detection generated by the plurality of first pixels; and a correction processing step of calculating a second distance to the object based on the signals for phase difference detection generated by the plurality of second pixels, and performing processing to correct the first distance based on the first distance and the second distance.

Images