Distance image capturing device and distance image capturing method

By setting multiple charge storage units in the distance image camera device and using waveform information to correct the charge quantity, the problem of reduced distance measurement accuracy caused by the distortion of rectangular signal waveforms is solved, and high-precision distance calculation is achieved.

CN116583712BActive Publication Date: 2026-11-10TOPPAN HOLDINGS INC
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202180081804.7
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2020-12-24
Filing Date
2021-12-23
Publication Date
2026-11-10
Estimated Expiration
2041-12-23

AI Technical Summary

Technical Problem

In distance image imaging devices, waveform distortion of rectangular signals leads to a decrease in distance measurement accuracy. The waveform distortion is particularly significant when the processing speed increases, making it difficult to correct distance deviations using polynomial approximation. Furthermore, the degree of deviation varies across multiple time windows, increasing processing complexity.

Method used

A distance image camera is used. Multiple charge accumulation units are set in pixels. The charge amount is corrected using waveform information to ensure that the charge accumulation units accumulate charge in a rectangular signal without distortion. The corrected charge amount is then used to calculate and measure the distance.

Benefits of technology

Even when the rectangular signal waveform is significantly distorted, it can still calculate and measure the distance with high accuracy, reducing the error caused by waveform distortion.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116583712B_ABST
    Figure CN116583712B_ABST
Patent Text Reader

Abstract

The present invention relates to a distance image imaging device (1), comprising: a light source unit (2); a light receiving unit (3) having a pixel (321) and a pixel driving circuit (323), the pixel (321) having a photoelectric conversion element (PD) and a plurality of charge storage units (CS); and a distance image processing unit (4), the distance image processing unit (4) acquiring waveform information (440) representing the degree of waveform distortion in a rectangular signal used in signal processing up to determining the distance, and based on the acquired waveform information, correcting the amount of charge (Q1, Q2) stored in the charge storage units (CS) to the amount of charge (Q1h, Q2h) stored when the waveform of the rectangular signal is undistorted, and using the corrected amount of charge (Q1h, Q2h) to determine the distance to be measured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to a distance image capturing apparatus and a distance image capturing method. This application is based on and claims priority to Japanese Patent Application No. 2020-215032, filed on December 24, 2020, the contents of which are incorporated herein by reference. Background Technology

[0002] For a long time, a technique for measuring the distance to an object has existed that measures the time of flight of a light pulse. This technique is called Time of Flight (TOF). In TOF, taking advantage of the known speed of light, a light pulse in the near-infrared region is irradiated onto an object. Then, the time difference between the moment the light pulse is irradiated and the moment the reflected light from the object is received is measured. The distance to the object is calculated based on this time difference. Ranging sensors that use photodiodes (photoelectric conversion elements) to detect the light used for distance measurement have been put into practical use.

[0003] Furthermore, in recent years, distance measuring sensors that can obtain not only the distance to an object but also the depth information of each pixel in a two-dimensional image containing the object—that is, three-dimensional information about the object—have been put into practical use. Such distance measuring sensors are also called distance image cameras. In a distance image camera, multiple pixels, including photodiodes, are arranged in a two-dimensional matrix on a silicon substrate, and each pixel receives light reflected from an object. In this distance image camera, by outputting an image quantity based on the photoelectric conversion signal of the amount of light (charge) received by each pixel, a two-dimensional image containing the object and the distance information of each pixel constituting the image can be obtained. For example, Patent Document 1 discloses a technique for calculating distance by setting three charge storage sections in one pixel and sequentially distributing the charge.

[0004] Existing technical documents Patent documents Patent Document 1: Japanese Patent No. 435729 Summary of the Invention

[0005] The problem that the invention aims to solve In this type of distance imaging device, there are short-pulse mode for intermittent illumination of light pulses and CW (Continuous Wave) mode for continuous illumination of light. In the short-pulse mode, it is important to accumulate charge while maintaining the waveform of the light pulse and reflected light as a rectangle for maintaining the accuracy of the measured distance. That is, the timing signal of the driver controlling the laser diode illuminating the light pulse and the gate signal for accumulating charge in the charge accumulation section need to be maintained as a rectangle. However, due to signal delays in the circuit and the charge transfer efficiency in the photoelectric conversion element, there is a delay in the rise or fall of the rectangle, resulting in waveform distortion. In particular, when the width of the rectangle (pulse width) is reduced for the purpose of improving processing speed, the waveform distortion becomes significant. When the waveform distortion becomes significant, there is a problem that the amount of charge accumulated changes significantly from the original amount, resulting in errors in the measured distance. As a countermeasure, it is possible to consider correcting the deviation (error) in the measured distance using polynomial approximation. However, there are problems such as generating processing load and deteriorating distance accuracy when noise in higher-order terms is amplified. Furthermore, when multiple time windows exist, the degree of deviation in distance measurements varies across each window. Therefore, when correcting distances using polynomial approximations across multiple time windows, the process becomes complex and computationally difficult.

[0006] The present invention is based on the above-mentioned problems, and its purpose is to provide a distance image capturing device and a distance image capturing method that can calculate the measured distance with high accuracy even when the waveform distortion of the rectangular signal used for distance measurement becomes significant.

[0007] Methods for solving problems The distance image capturing apparatus of the present invention includes: a light source unit that irradiates a light pulse into a measurement space in which a subject exists; a light receiving unit having a pixel and a pixel driving circuit, wherein the pixel has a photoelectric conversion element that generates a charge corresponding to the incident light and a plurality of charge storage units that store the charge, and the pixel driving circuit distributes the charge to the charge storage units in the pixel and stores it at a predetermined timing synchronized with the irradiation of the light pulse; and a distance image processing unit that uses the amount of charge stored in the charge storage units to determine the measurement distance to the subject. The distance image processing unit acquires waveform information representing the degree of waveform distortion in a rectangular signal used in signal processing up to the determination of the measurement distance, and based on the acquired waveform information, corrects the amount of charge stored in the charge storage units to the amount of charge stored when the waveform of the rectangular signal is undistorted, and uses the corrected amount of charge to determine the measurement distance.

[0008] In the distance image camera device of the present invention, the waveform information is a trailing ratio representing the ratio of the second area to the first area. The first area and the second area are derived by performing time integration on the rectangular signal. The first area is the integral value of the first signal (i.e., the rectangular signal without waveform distortion) from the start of rising to the start of falling. The second area is the integral value of the difference obtained by subtracting the second signal (i.e., the rectangular signal with waveform distortion) from the first signal from the start of rising to the end of rising, or the integral value of the second signal from the start of falling to the end of falling. The aforementioned distance image processing unit performs the following steps: for a first charge accumulation unit that first accumulates a charge corresponding to the reflected light from the light pulse reflected by the subject, and a second charge accumulation unit that then accumulates a charge in the first charge accumulation unit, the unit calculates a first distance calculation charge obtained by subtracting the charge corresponding to the external light component from the first charge accumulation unit, and a second distance calculation charge obtained by subtracting the charge corresponding to the external light component from the second charge accumulation unit. The unit then uses the waveform information to calculate a first correction charge in the second distance calculation charge that was not accumulated in the first charge accumulation unit but was accumulated in the second charge accumulation unit due to the distortion of the rectangular signal. Finally, the unit uses the first correction charge to correct the first distance calculation charge and the second distance calculation charge.

[0009] In the distance image imaging apparatus of the present invention, the aforementioned pixel includes three charge storage units. The distance image processing unit controls the timing of charge accumulation in the external light accumulation unit in such a way that no charge corresponding to the reflected light is accumulated in the external light accumulation unit, which is different from the first and second charge storage units, thereby making the amount of charge accumulated in the external light accumulation unit a charge amount corresponding to the external light component.

[0010] In the distance image imaging device of the present invention, the pixel has three charge storage units. The waveform information is a trailing ratio representing the ratio of the second area to the first area. The first area and the second area are derived by time integration of the rectangular signal. The first area is the integral value of the first signal (i.e., the rectangular signal without waveform distortion) from the start of rise to the start of fall. The second area is the integral value of the difference obtained by subtracting the second signal (i.e., the rectangular signal with waveform distortion) from the first signal from the start of rise to the end of rise, or the integral value of the second signal from the start of fall to the end of fall. The aforementioned distance image processing unit performs the following calculations for the three charge accumulation units: a first charge accumulation unit that first accumulates a charge corresponding to the reflected light from the light pulse reflected by the subject; a second charge accumulation unit that accumulates a charge corresponding to the reflected light from the subject; and a third charge accumulation unit that accumulates a charge corresponding to the reflected light from the subject. The calculations include: a first charge accumulation unit that first accumulates a charge corresponding to the reflected light from the subject; a second charge accumulation unit that first accumulates a charge corresponding to the subject; a third charge accumulation unit that first accumulates a charge corresponding to the reflected light from the subject; and a third charge accumulation unit that first accumulates a charge corresponding to the reflected light from the subject. The third distance calculation charge is obtained by considering the corresponding charge amount of the components. The waveform information is used to calculate the first correction charge amount in the second distance calculation charge amount that was not accumulated in the first charge accumulation portion but was accumulated in the second charge accumulation portion due to the distortion of the rectangular signal. The waveform information is used to calculate the second correction charge amount in the third distance calculation charge amount that was not accumulated in the second charge accumulation portion but was accumulated in the third charge accumulation portion due to the distortion of the rectangular signal. The first correction charge amount is used to correct the first distance calculation charge amount. The first correction charge amount and the second correction charge amount are used to correct the second distance calculation charge amount.

[0011] In the distance image imaging apparatus of the present invention, the pixel comprises four charge storage units. The distance image processing unit controls the timing of charge accumulation in the external light storage unit in such a way that no charge corresponding to the reflected light is accumulated in the external light storage unit, which is different from the first, second, and third charge storage units among the four charge storage units, and sets the amount of charge accumulated in the external light storage unit to be the amount of charge corresponding to the external light component.

[0012] The distance image capturing method of the present invention is a distance image capturing method performed by a distance image capturing device, which includes: a light source unit that irradiates a light pulse into a measurement space in which a subject exists; a light receiving unit having pixels and a pixel driving circuit, wherein the pixels have photoelectric conversion elements that generate charges corresponding to the incident light and a plurality of charge accumulation units that accumulate the charges, the pixel driving circuit distributing the charges to the charge accumulation units in the pixels and accumulating them at a predetermined timing synchronized with the irradiation of the light pulse; and a distance image processing unit that uses the amount of charge accumulated by the charge accumulation units to determine the measurement distance to the subject. The distance image processing unit acquires waveform information representing the degree of waveform distortion in a rectangular signal used in signal processing up to the determination of the measurement distance, and based on the acquired waveform information, corrects the amount of charge accumulated by the charge accumulation units to the amount of charge accumulated when the waveform of the rectangular signal is not distorted, and uses the corrected amount of charge to determine the measurement distance.

[0013] The effects of the invention According to the present invention, even when the waveform distortion of the rectangular signal used for distance measurement becomes significant, the measurement distance can be calculated with high accuracy. Attached Figure Description

[0014] Figure 1 This is a block diagram illustrating the general configuration of a distance image camera device according to an embodiment.

[0015] Figure 2 This is a block diagram illustrating the general configuration of a distance image sensor according to an embodiment.

[0016] Figure 3 This is a circuit diagram illustrating an example of the pixel configuration in an implementation method.

[0017] Figure 4 This is a timing diagram illustrating an example of timing the driving pixels in an implementation method.

[0018] Figure 5 This is a diagram illustrating waveform distortion in the implementation method.

[0019] Figure 6 This is a diagram illustrating waveform distortion in the implementation method.

[0020] Figure 7 This is a diagram illustrating waveform distortion in the implementation method.

[0021] Figure 8 This is a diagram illustrating an example of the waveform information configuration in an implementation method.

[0022] Figure 9This is a diagram illustrating the process of correcting charge quantity by the distance image processing unit in an embodiment.

[0023] Figure 10 This is a flowchart illustrating the processing flow performed by the distance image processing unit in the embodiment.

[0024] Figure 11 This is a timing diagram illustrating an example of timing the driving pixels in a modified embodiment.

[0025] Figure 12 This is a diagram illustrating the process of the distance image processing unit correcting the charge amount in a modified embodiment.

[0026] Figure 13 This is a diagram illustrating the effects of the implementation method.

[0027] Figure 14 This is a diagram illustrating two time windows of a variation of the implementation method. Detailed Implementation

[0028] Hereinafter, the distance image camera device according to the embodiment will be described with reference to the accompanying drawings.

[0029] <Implementation Method> First, the implementation method will be explained. Figure 1 This is a block diagram illustrating the schematic configuration of a distance image camera device according to the first embodiment of the present invention. Figure 1 The distance image imaging device 1 shown includes a light source unit 2, a light receiving unit 3, and a distance image processing unit 4. Figure 1 The text also indicates that there is an object, namely the subject OB, whose distance is measured in the distance imaging device 1.

[0030] The light source unit 2, under control from the distance image processing unit 4, irradiates light pulses PO into the space of the object being photographed, where the distance to the object being measured in the distance image imaging device 1 is located. The light source unit 2 is, for example, a surface-emitting semiconductor laser module such as a vertical-cavity surface-emitting laser (VCSEL). The light source unit 2 includes a light source device 21 and a diffuser plate 22.

[0031] The light source device 21 is a laser light source that emits a near-infrared wavelength (e.g., a wavelength range of 850 nm to 940 nm) that forms a light pulse PO that irradiates the subject OB. The light source device 21 is, for example, a semiconductor laser light-emitting element. The light source device 21 emits pulsed laser light according to the control from the timing control unit 41.

[0032] The diffuser plate 22 is an optical component that diffuses the near-infrared laser emitted by the light source device 21 into a surface area that irradiates the subject OB. The pulsed laser light diffused by the diffuser plate 22 is emitted as a light pulse PO and irradiates the subject OB.

[0033] The light-receiving unit 3 receives the reflected light RL from the light pulse PO reflected by the subject OB, whose distance is measured in the distance image capturing device 1, and outputs a pixel signal corresponding to the received reflected light RL. The light-receiving unit 3 includes a lens 31 and a distance image sensor 32.

[0034] Lens 31 is an optical lens that guides the incident reflected light RL to the distance image sensor 32. Lens 31 directs the incident reflected light RL toward the distance image sensor 32, so that the pixels in the light-receiving area of ​​the distance image sensor 32 are illuminated (incidentally).

[0035] The distance image sensor 32 is an imaging element used in the distance image imaging device 1. The distance image sensor 32 has multiple pixels in a two-dimensional light-receiving area. Each pixel of the distance image sensor 32 includes a photoelectric conversion element, multiple charge storage sections corresponding to this photoelectric conversion element, and elements for distributing charge to each charge storage section. In other words, a pixel is an imaging element configured to distribute and store charge to multiple charge storage sections.

[0036] The distance image sensor 32, under the control of the timing control unit 41, distributes the charge generated by the photoelectric conversion element to each charge storage unit. Furthermore, the distance image sensor 32 outputs a pixel signal corresponding to the amount of charge distributed to each charge storage unit. In the distance image sensor 32, multiple pixels are arranged in a two-dimensional matrix, and each pixel outputs a corresponding one-frame pixel signal.

[0037] The distance image processing unit 4 controls the distance image capturing device 1 and calculates the distance to the subject OB. The distance image processing unit 4 includes a timing control unit 41, a distance calculation unit 42, a measurement control unit 43, and a storage unit 44. In addition, a portion of the functional units of the distance image processing unit 4 (timing control unit 41, distance calculation unit 42, measurement control unit 43, and storage unit 44) can also be incorporated into the distance image sensor 32.

[0038] The timing control unit 41, under the control of the measurement control unit 43, controls the timing of various control signals required for output measurement. These control signals include, for example, signals controlling the irradiation of the light pulse PO, signals distributing and accumulating the reflected light RL to multiple charge accumulation sections, and signals controlling the number of distributions (accumulation counts) per frame. The number of distributions is directed towards the charge accumulation section CS (see reference). Figure 3The number of times the charge distribution process is repeated.

[0039] The distance calculation unit 42 uses the pixel signal output from the distance image sensor 32 and the waveform information 440 (described later) to calculate the distance to the subject OB, and outputs the calculated distance information. The distance calculation unit 42 uses the waveform information 440 to correct the amount of charge accumulated in the multiple charge storage units obtained from the pixel signal output from the distance image sensor 32, and calculates the distance to the subject OB based on the corrected charge amount. The waveform information 440 will be explained in detail later. Furthermore, the method by which the distance calculation unit 42 uses the waveform information 440 to determine the distance to the subject OB will be explained in detail later.

[0040] The measurement control unit 43 controls the timing control unit 41. For example, the measurement control unit 43 sets the number of times a frame is distributed and the accumulation time Ta, and controls the timing control unit 41 by recording images according to the set content.

[0041] The storage unit 44 is composed of a storage medium, such as an HDD (Hard Disk Drive), flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), RAM (Random Access Read / Write Memory), ROM (Read Only Memory), or any combination of these storage media. For example, the storage unit 44 stores waveform information 440.

[0042] According to this configuration, in the distance image camera device 1, the light receiving unit 3 receives the near-infrared light pulse PO irradiated by the light source unit 2 onto the subject OB by the reflected light RL reflected by the subject OB, and the distance image processing unit 4 outputs distance information that has been measured to the subject OB.

[0043] In addition, Figure 1 The diagram shows a distance image camera device 1 with a distance image processing unit 4 inside, but the distance image processing unit 4 may also be a component outside the distance image camera device 1.

[0044] Next, the configuration of the distance image sensor 32, which is used as an imaging element in the distance image imaging device 1, will be described. Figure 2 This is a block diagram showing the schematic configuration of the imaging element (distance image sensor 32) used in the distance image imaging device 1 in the embodiment.

[0045] like Figure 2As shown, the distance image sensor 32 includes, for example, a light-receiving area 320 with a plurality of pixels 321, a control circuit 322, a vertical scanning circuit 323 with a distribution operation, a horizontal scanning circuit 324, and a pixel signal processing circuit 325.

[0046] The light-receiving area 320 is an area configured with multiple pixels 321. Figure 2 The diagram shows an example of multiple pixels 321 arranged in an 8x8 two-dimensional matrix. Each pixel 321 accumulates a charge corresponding to the amount of light received. A control circuit 322 provides overall control of the distance image sensor 32. For example, the control circuit 322 controls the operation of the components of the distance image sensor 32 based on instructions from the timing control unit 41 of the distance image processing unit 4. Alternatively, the timing control unit 41 may directly control the components of the distance image sensor 32. In this case, the control circuit 322 can be omitted.

[0047] The vertical scanning circuit 323 controls the pixels 321 arranged in the light-receiving area 320 row by row according to the control from the control circuit 322. The vertical scanning circuit 323 outputs a voltage signal corresponding to the amount of charge stored in the charge storage section CS of each pixel 321 to the pixel signal processing circuit 325. In this case, the vertical scanning circuit 323 distributes the charge converted by the photoelectric conversion element to the charge storage section of each pixel 321. That is, the vertical scanning circuit 323 is an example of a "pixel driving circuit".

[0048] The pixel signal processing circuit 325 is a circuit that performs predetermined signal processing (e.g., noise suppression processing, A / D conversion processing, etc.) on the voltage signals output from the pixels 321 of each column to the corresponding vertical signal lines, according to the control from the control circuit 322.

[0049] The horizontal scanning circuit 324 is a circuit that, according to the control from the control circuit 322, sequentially outputs the signal from the pixel signal processing circuit 325 to the horizontal signal line. Thus, pixel signals equivalent to the charge accumulated for one frame are sequentially output to the distance image processing unit 4 via the horizontal signal line.

[0050] The following explanation assumes that the pixel signal processing circuit 325 performs A / D conversion processing and that the pixel signal is a digital signal.

[0051] Here, the configuration of the pixel 321 disposed within the light-receiving area 320 of the distance image sensor 32 will be described. Figure 3 This is a circuit diagram illustrating an example of the configuration of pixels 321 arranged within the light-receiving area 320 of the distance image sensor 32 in the embodiment. Figure 3This figure illustrates an example of the configuration of one pixel 321 among a plurality of pixels 321 disposed within the light-receiving area 320. In the example shown, pixel 321 is configured to have four pixel signal readout sections.

[0052] like Figure 3 As shown, pixel 321 includes one photoelectric conversion element PD, a drain-gate transistor GD, and four pixel signal readout units RU (pixel signal readout units RU1 to RU4). Each pixel signal readout unit RU outputs a voltage signal from the output terminal O.

[0053] In the following description, each pixel signal readout unit RU is distinguished by appending the numbers "1", "2", "3" or "4" after its symbol. Similarly, each component of the four pixel signal readout units RU is also distinguished by appending numbers after its symbol.

[0054] Each pixel signal readout unit RU includes a readout gate transistor G, a floating diffusion region FD, a charge storage capacitor C, a reset gate transistor RT, a source follower gate transistor SF, and a select gate transistor SL. In each pixel signal readout unit RU, the floating diffusion region FD and the charge storage capacitor C constitute a charge storage section CS. Specifically, pixel signal readout unit RU1 includes a readout gate transistor G1, a floating diffusion region FD1, a charge storage capacitor C1, a reset gate transistor RT1, a source follower gate transistor SF1, and a select gate transistor SL1. In pixel signal readout unit RU1, the floating diffusion region FD1 and the charge storage capacitor C1 constitute a charge storage section CS1. Pixel signal readout units RU2 to RU4 have the same configuration.

[0055] A photoelectric conversion element (PD) is an embedded photodiode that converts incident light into electrical charge and stores the resulting charge. The structure of a PD can be arbitrary. For example, a PD can be a PN photodiode with a P-type semiconductor and an N-type semiconductor bonded together, or a PIN photodiode with an I-type semiconductor sandwiched between the P-type and N-type semiconductors. Furthermore, a PD is not limited to a photodiode; it can also be a grating-type photoelectric conversion element.

[0056] In pixel 321, the charge generated by the photoelectric conversion element PD through photoelectric conversion of incident light is distributed to four charge storage units CS respectively, and each voltage signal corresponding to the amount of charge distributed is output to pixel signal processing circuit 325.

[0057] The composition of the pixels configured on the distance image sensor 32 is not limited to Figure 3 The configuration shown, which has four pixel signal readout units RU, can be any pixel having multiple pixel signal readout units RU. That is, the number of pixel signal readout units RU (charge storage units CS) provided by the pixel disposed at the distance image sensor 32 can be 2, 3, 5 or more.

[0058] In addition, Figure 3 In the pixel 321 shown, an example is illustrated where the charge storage section CS is composed of a floating diffusion region FD and a charge storage capacitor C. However, the charge storage section CS may be composed of at least a floating diffusion region FD, or the pixel 321 may not have a charge storage capacitor C.

[0059] In addition, Figure 3 The pixel 321 shown is an example of a configuration with a drain-gate transistor GD, but it is not limited to this. For example, if it is not necessary to discard the charge remaining in the photoelectric conversion element PD that has not been accumulated in the charge storage section CS, a configuration without a drain-gate transistor GD may be used.

[0060] Next, use Figure 4 The driving timing for pixel 321 is explained. Figure 4 This is a timing diagram showing the timing of the driving pixel 321 in the implementation method.

[0061] exist Figure 4 In this context, the time required for the charge accumulation unit CS to accumulate charge in a single allocation process is denoted as the "unit accumulation period". After the allocation process performed during the "unit accumulation period" is repeated an accumulation number of times equivalent to one frame, a process is performed to read out the amount of charge accumulated during that period. The time for performing this process of reading out the accumulated charge is denoted as the "readout period".

[0062] In addition, Figure 4 In this context, the timing of the illumination pulse PO is represented as "L", the timing of the reflected light RL being received is represented as "R", the timing of driving the readout gate transistor G1 is represented as "G1", the timing of driving the readout gate transistor G2 is represented as "G2", the timing of driving the readout gate transistor G3 is represented as "G3", the timing of driving the readout gate transistor G4 is represented as "G4", and the timing of the drive signal RSTD is represented as "GD".

[0063] The vertical scanning circuit 323, in a timing synchronized with the irradiation of the light pulse PO, causes the charge accumulation sections CS1 to CS4 to accumulate charge. Figure 4In the example, the charge accumulation section CS1 is charged at the same timing as the time when the light pulse PO is irradiated. After the charge accumulation section CS1 is charged, the charge accumulation sections CS2 to CS4 are charged in sequence.

[0064] exist Figure 4 The example shows a timing diagram of the case where the reflected light RL is received by the distance image sensor 32 after a delay time Td from the moment of illumination of the light pulse PO. According to the delay time Td, the charge corresponding to the reflected light RL is distributed and accumulated in charge accumulation units CS1 and CS2, or charge accumulation units CS2 and CS3. During the timing of charge accumulation in charge accumulation unit CS4, the reflected light RL is not received, and the charge corresponding to external light components such as background light is accumulated in charge accumulation unit CS4.

[0065] Specifically, first, the vertical scanning circuit 323 irradiates the device with a light pulse PO. At the same timing as the irradiation, the vertical scanning circuit 323 turns off the drain-gate transistor GD and turns on the readout gate transistor G1 for the accumulation time Ta. After turning on the readout gate transistor G1 for the accumulation time Ta, the vertical scanning circuit 323 turns it off. Thus, during the period when the readout gate transistor G1 is controlled to be on, the charge converted by the photoelectric conversion element PD is accumulated in the charge accumulation section CS1 via the readout gate transistor G1.

[0066] Next, when the vertical scanning circuit 323 turns off the readout gate transistor G1, it turns on the readout gate transistor G2 for the accumulation time Ta. After turning on the readout gate transistor G2 for the accumulation time Ta, the vertical scanning circuit 323 turns off the readout gate transistor G2. Thus, during the period when the readout gate transistor G2 is controlled to be on, the charge converted by the photoelectric conversion element PD is accumulated in the charge accumulation section CS2 via the readout gate transistor G2.

[0067] Next, when the vertical scanning circuit 323 turns the readout gate transistor G2 to the off state, it turns the readout gate transistor G3 to the on state for the accumulation time Ta. After turning the readout gate transistor G3 to the on state for the accumulation time Ta, the vertical scanning circuit 323 turns the readout gate transistor G3 to the off state. Thus, during the period when the readout gate transistor G3 is controlled to be on, the charge converted by the photoelectric conversion element PD is accumulated in the charge accumulation section CS3 via the readout gate transistor G3.

[0068] Next, when the accumulation of charge in the charge accumulation section CS3 ends, the vertical scanning circuit 323 turns on the readout gate transistor G4 for the accumulation time Ta. After turning on the readout gate transistor G4 for the accumulation time Ta, the vertical scanning circuit 323 turns off the readout gate transistor G4. When turning off the readout gate transistor G4, the vertical scanning circuit 323 turns on the drain gate transistor GD. Because the drain gate transistor GD is turned on, the charge converted by the photoelectric conversion element PD during this period is not accumulated in the charge accumulation section CS, but is discarded via the drain gate transistor GD.

[0069] The vertical scanning circuit 323 repeatedly performs the aforementioned driving a predetermined number of times per frame. Then, the vertical scanning circuit 323 outputs a voltage signal corresponding to the amount of charge stored in each charge storage unit CS. Specifically, the vertical scanning circuit 323 turns on the selection gate transistor SL1 for a predetermined time, thereby outputting a voltage signal corresponding to the amount of charge stored in the charge storage unit CS1 from the output terminal O1 via the pixel signal readout unit RU1. Similarly, the vertical scanning circuit 323 sequentially turns on the selection gate transistors SL2 to SL4, thereby outputting voltage signals corresponding to the amount of charge stored in the charge storage units CS2 to CS4 from the output terminals O2 to O4. Thus, an electrical signal equivalent to the amount of charge stored in each charge storage unit CS for one frame is output to the distance calculation unit 42.

[0070] Furthermore, the above description uses the case where the readout gate transistor G1 is turned on at the timing of the illumination pulse PO as an example. However, it is not limited to this. It is acceptable as long as the illumination pulse PO is applied at a timing at which the charge corresponding to the reflected light RL is distributed and accumulated to the charge accumulation sections CS1 and CS2, or CS2 and CS3.

[0071] exist Figure 4 In this embodiment, based on the relationship between the timing of the irradiated light pulse PO and the timing of charge accumulation in the charge accumulation units CS1 and CS2, the amount of charge corresponding to the reflected light RL and the external light components is allocated to and maintained. In this case, the charge accumulation unit CS1 is an example of a "first charge accumulation unit". The charge accumulation unit CS2 is an example of a "second charge accumulation unit". Furthermore, the charge accumulation unit CS4 maintains an amount of charge corresponding to the external light components such as background light. In this case, the charge accumulation unit CS4 is an example of a "charge accumulation unit for external light accumulation".

[0072] Furthermore, when reflected light RL is received from a subject OB located at a relatively far distance, the delay time Td increases. As a result, the charge accumulation sections CS2 and CS3 are allocated and maintained with an amount of charge corresponding to the reflected light RL and the external light component. In this case, charge accumulation section CS2 is an example of a "first charge accumulation section". Charge accumulation section CS3 is an example of a "second charge accumulation section".

[0073] The ratio of the amount of charge distributed to the charge storage units CS1 and CS2 is a ratio corresponding to the delay time Td from when the light pulse PO is reflected by the subject OB and incident on the image capturing device 1.

[0074] The distance calculation unit 42 uses this principle to calculate the delay time Td using the following equation (1). In equation (1), To is the time interval of the illumination light pulse PO, and R is the charge ratio representing the distribution ratio of the reflected light RL.

[0075] Td=To×R ……(1) in, R = Q2# / (Q1# + Q2#) Q1#=Q1-Qb Q2#=Q2-Qb Q4 = Qb To is the time interval of the illumination light pulse PO. R is the charge ratio calculated below. Q1# is the amount of charge stored in the charge storage section CS1 corresponding to the reflected light RL. Q2# is the amount of charge stored in the charge storage section CS2 corresponding to the reflected light RL. Qb is the amount of charge stored in the charge storage section CS, corresponding to the external light component. Q1 is the amount of charge stored in the charge storage section CS1. Q2 is the amount of charge stored in the charge storage section CS2. Q4 is the amount of charge stored in the charge storage section CS4. The distance calculation unit 42 multiplies the delay time Td obtained by equation (1) in the near-field light-receiving pixel with the speed of light (velocity), thereby calculating the round-trip distance to the subject OB. Then, the distance calculation unit 42 calculates the distance to the subject OB by halving the round-trip distance calculated above.

[0076] Here, we will explain the important factors that cause errors in the distance (measured distance) calculated based on the amount of accumulated charge.

[0077] One factor contributing to error is the distortion of waveforms in the rectangular signals used in various signal processing operations involved in distance measurement. In practical circuits, high-frequency characteristics during signal transmission deteriorate due to wiring resistance, parasitic capacitance, and other factors. Furthermore, the charge transfer efficiency of the photoelectric conversion element (PD) causes delays in charge transfer. If high-frequency characteristics deteriorate or charge transfer efficiency in the PD causes delays in charge transfer, this leads to delays during rapid changes in signal amplitude, such as rise and fall, resulting in waveform distortion. Consequently, the rectangle is deformed, transforming into a waveform with delayed rise and fall.

[0078] When the rectangular signal used in various distance measurement processes is distorted, it becomes a cause of error. For example, when the timing signal of the driver controlling the laser diode that illuminates the light pulse is distorted, the light intensity reaches the specified amount only after a slight delay after the light pulse is illuminated, and the light intensity of the light pulse PO becomes 0 (zero) only after a slight delay after the illumination stops. Such waveform distortion affects the reflected light RL, so the reflected light RL also produces a tailing waveform and is received by the pixel 321. Moreover, when the timing signal of the readout gate transistor G is distorted, the amount of charge that should be accumulated in the charge accumulation section CS1 is accumulated in the charge accumulation section CS2, or the amount of charge that should be accumulated in the charge accumulation section CS2 is accumulated in the charge accumulation section CS3. Using the amount of charge accumulated in the charge accumulation section CS that is different from the original charge accumulation section CS, the delay time Td calculated by equation (1) becomes a value different from the original delay time Td, which is a value containing error.

[0079] As a countermeasure, in this embodiment, waveform information 440 is used to correct the amount of charge corresponding to the reflected light RL stored in the charge storage section CS to the amount of charge stored when using a rectangular signal with no waveform distortion. Waveform information 440 is information indicating the degree of waveform distortion. Then, the delay time Td is calculated using the corrected amount of charge through equation (1). As a result, the error caused by the distortion of the rectangular signal can be reduced.

[0080] Here, use Figures 5-9 The method for correcting the charge quantity is explained in detail. Figures 5-7 This is a diagram illustrating waveform distortion in the implementation method. Figure 8 This is a diagram illustrating an example of the configuration of waveform information 440 in an implementation method. Figure 9 This diagram illustrates the process of correcting charge quantity by the distance image processing unit 4 in an embodiment.

[0081] Figure 5 The diagram schematically illustrates a rectangular signal H1 (hereinafter also referred to as "rectangular signal H1") with no waveform distortion. For example... Figure 5 As shown, at the beginning of the rise of signal H1 (Trs), the signal amplitude changes from Lo to Hi; at the beginning of the fall (Tds), the signal amplitude changes from Hi to Lo.

[0082] Figure 6 The diagram schematically illustrates a signal H2 with a distorted tail (hereinafter also referred to as "tailed signal H2"). For example... Figure 6 As shown, the signal H2 gradually changes from Lo to Hi in amplitude from the start of its rise (Trs) to the end of its rise (Tre). Similarly, the signal H2 gradually changes from Hi to Lo in amplitude from the start of its fall (Tds) to the end of its fall (Tde).

[0083] Figure 7 schematically showing that Figure 6 The waveform of the trailing signal H2 is divided into multiple regions. In this embodiment, as shown... Figure 7 As shown, the trailing signal H2 is divided into regions Sa to Sc according to its waveform, and the area of ​​each region is used to define the degree of distortion of the trailing signal H2.

[0084] Region Sa corresponds to the region resulting from the amplitude reduction caused by the rise delay of the trailing signal H2. The area of ​​region Sa is the integral value obtained by integrating the difference between the rectangular signal H1 and the trailing signal H2 from the start time Trs to the end time Tre.

[0085] Region Sb is equivalent to the area of ​​the rectangular signal H1 minus region Sa. The area of ​​region Sb is the integral value obtained by integrating the trailing signal H2 from the start of its rise Trs to the start of its fall Tds. Alternatively, it is the area obtained by subtracting the area of ​​region Sa from the integral value (rectangular area) obtained by integrating the rectangular signal H1 from the start of its rise Trs to the start of its fall Tds.

[0086] Region Sc corresponds to the tail region caused by the descent delay of the tail signal H2. The area of ​​region Sc is the integral value obtained by integrating the tail signal H2 from the start time Tds to the end time Tde of descent.

[0087] In this embodiment, the ratio of the trailing area (region Sc) to the rectangular area (region Sa + Sb) is set as the trailing ratio Re, and is used as an indicator of the "degree of distortion" in the waveform. In addition, the trailing ratio Re can be any value that at least represents the "degree of distortion" in the waveform. For example, the trailing ratio Re can also be the ratio of the rectangular area (region Sa + Sb) to the trailing area (region Sc).

[0088] Figure 8Here is an example of the structure of waveform information 440. Waveform information 440 includes items such as the tail ratio Re, a calculation formula, and notes. The tail ratio Re is an indicator of the "degree of distortion" in the waveform, and is the ratio of the tail area (region Sc) to the rectangular area (region Sa + Sb). The calculation formula is the formula for calculating the tail ratio Re.

[0089] In this embodiment, the distortions of various signal processing methods are concentrated on the reflected light RL, and the distorted reflected light RL is considered to have been received and corrected. That is, the charge corresponding to the distorted reflected light RL is accumulated in the charge accumulation section CS during the driving timing of the undistorted readout gate transistor G. Then, the amount of charge accumulated in the charge accumulation section CS is corrected based on the tailing ratio Re (distortion degree) of the reflected light RL. The corrected charge amount is the amount of charge accumulated in the charge accumulation section CS when the undistorted reflected light RL is driven by the undistorted readout gate transistor G.

[0090] Figure 9 This diagram illustrates the process of correcting charge quantity at the distance from the image processing unit 4. (For example...) Figure 9 As shown, in this embodiment, it is assumed that the distorted reflected light RL is accumulated in the charge accumulation section CS during the driving timing of the undistorted readout gate transistor G.

[0091] exist Figure 9 In, with Figure 4 Similarly, the reflected light RL arrives at the distance image sensor 32 with a delay time Td, and the charge corresponding to the reflected light RL is distributed and stored in the charge storage units CS1 and CS2. Additionally, in Figure 9 In this process, the waveform of the reflected light RL is distorted, and the charge corresponding to a portion of the reflected light RL generated due to the distortion is accumulated in the charge accumulation section CS3. In this case, the charge accumulation section CS1 is an example of a "first charge accumulation section". The charge accumulation section CS2 is an example of a "second charge accumulation section". The charge accumulation section CS3 is an example of a "third charge accumulation section".

[0092] Specifically, in the reflected light RL formed by the distorted trailing signal H2, the charge corresponding to a portion of region Sb (region Sb1) is accumulated in charge accumulation section CS1. Additionally, the charge corresponding to the remaining portion of region Sb (region Sb2) is accumulated in charge accumulation section CS2. Furthermore, the charge corresponding to a portion of region Sc (region Sc1) is accumulated in charge accumulation section CS2. Finally, the charge corresponding to the remaining portion of region Sc (region Sc2) is accumulated in charge accumulation section CS3.

[0093] In this embodiment, when correcting the charge amount, the irradiation time To of the irradiation pulse PO is assumed to be the same time interval as the charge accumulation time Ta in the charge accumulation section CS. Furthermore, in this embodiment, it is assumed that the area of ​​region Sa is the same as the area of ​​region Sc for calculation purposes.

[0094] In addition, Figure 9 The timing diagram for "G4" is omitted, but it is related to... Figure 4 The same applies to driving the readout gate transistor G4, and the amount of charge corresponding to the external light component is stored in the charge storage section CS4.

[0095] The distance calculation unit 42 acquires an electrical signal equivalent to the amount of charge stored in the charge storage unit CS for one frame. The distance calculation unit 42 calculates the total amount of charge QSUM corresponding to the reflected light RL using the following formula (2). Here, charge Q1# is the amount of charge stored in the charge storage unit CS1 corresponding to the reflected light RL, which is an example of "charge for first distance calculation". Charge Q2# is the amount of charge stored in the charge storage unit CS2 corresponding to the reflected light RL, which is an example of "charge for second distance calculation". Charge Q3# is the amount of charge stored in the charge storage unit CS3 corresponding to the reflected light RL, which is an example of "charge for third distance calculation".

[0096] QSUM=Q1#+Q2#+Q3#……(2) Q1#=Q1-Qb Q2#=Q2-Qb Q3#=Q3-Qb Q4 = Qb in, Q1# is the amount of charge stored in the charge storage section CS1 corresponding to the reflected light RL. Q2# is the amount of charge stored in the charge storage section CS2 corresponding to the reflected light RL. Q3# is the amount of charge stored in the charge storage section CS3 corresponding to the reflected light RL. Qb is the amount of charge stored in the charge storage section CS, corresponding to the external light component. Q1 is the amount of charge stored in the charge storage section CS1. Q2 is the amount of charge stored in the charge storage section CS2. Q3 is the amount of charge stored in the charge storage section CS3. Q4 is the amount of charge stored in the charge storage section CS4. The distance calculation unit 42 uses the total charge QSUM calculated by equation (2) and the tailing ratio Re stored in the waveform information 440 to calculate the charge Qs of the tailing portion by the following equation (3). Figure 7 The corresponding charge amount in region Sc.

[0097] Qs=QSUM×Re ……(3) in, QSUM = Q1# + Q2# + Q3# Q1# is the amount of charge stored in the charge storage section CS1 corresponding to the reflected light RL. Q2# is the amount of charge stored in the charge storage section CS2 corresponding to the reflected light RL. Q3# is the amount of charge stored in the charge storage section CS3 corresponding to the reflected light RL. Re is the trailing ratio. The distance calculation unit 42 separates the charge Qs of the trailing part calculated by equation (3) into the charge Q2s stored in the charge storage unit CS2 and the charge Q3# stored in the charge storage unit CS3 corresponding to the reflected light RL.

[0098] Qs = Q2s + Q3#……(4) in, Q2s is the amount of charge stored in the charge storage section CS2, corresponding to the trailing portion of the reflected light RL. Q3# is the amount of charge stored in the charge storage section CS3 corresponding to the reflected light RL. The distance calculation unit 42 solves equation (4) for the charge Q2s. The charge Qs of the trailing part and the charge Q3# stored in the charge storage unit CS3 corresponding to the reflected light RL are known, so the charge Q2s can be calculated by equation (5).

[0099] Q2s = Qs - Q3# ……(5) in, Q2s is the amount of charge stored in the charge storage section CS2, corresponding to the trailing portion of the reflected light RL. Q3# is the amount of charge stored in the charge storage section CS3 corresponding to the reflected light RL. Qs is the charge of the trailing portion. The distance calculation unit 42, assuming that the area of ​​region Sa is the same as the area of ​​region Sc, corrects the charge Q1# as in equation (6) and calculates the corrected charge Q1h. Additionally, the distance calculation unit 42 corrects the charge Q2# as in equation (7) and calculates the corrected charge Q2h.

[0100] Q1h=Q1#+Q2s……(6) Q2h=Q2#-Q2s+Q3#……(7) in, Q1h is the amount of charge stored in the corrected charge storage section CS1, corresponding to the reflected light RL. Q1# is the amount of charge stored in the charge storage section CS1 before correction, corresponding to the reflected light RL. Q2s is the amount of charge stored in the charge storage section CS2, corresponding to the trailing portion of the reflected light RL. Q2h is the amount of charge stored in the corrected charge storage section CS2, corresponding to the reflected light RL. Q2# is the amount of charge stored in the charge storage section CS2 before correction, corresponding to the reflected light RL. Q3# is the amount of charge stored in the charge storage section CS3 corresponding to the reflected light RL. Figure 10 This is a flowchart illustrating the processing flow performed by the distance image processing unit 4 in the embodiment. The distance calculation unit 42 obtains the charge amounts Q1 to Q4 accumulated by the charge accumulation units CS1 to CS4 respectively (step S10). The distance calculation unit 42 uses the obtained charge amount Q4 to calculate the charge amount Qb corresponding to the external light component (step S11).

[0101] The distance calculation unit 42 uses the charge amounts Q1 to Q3, the charge amount Qb, and the tail ratio Re to calculate the charge amount Qs corresponding to the tail component (step S12). The distance calculation unit 42 calculates the charge amount Qs by substituting the charge amounts Q1 to Q3, the charge amount Qb, and the tail ratio Re into equations (2) and (3).

[0102] The distance calculation unit 42 calculates the charge Q2s stored in the charge accumulation unit CS2 corresponding to the tail component (step S13). The distance calculation unit 42 calculates the charge Q2s using equations (4) and (5). The distance calculation unit 42 calculates the corrected charge Q1h, which is corrected for the charge Q1#, and calculates the corrected charge Q2h, which is corrected for the charge Q2# (step S14). The distance calculation unit 42 uses the corrected charge Q1h and Q2h to determine the measurement distance (step S15). The distance calculation unit 42 calculates the delay time Td by substituting the corrected charge Q1h into the charge Q1# in equation (1) and the corrected charge Q2h into the charge Q2# in equation (1). The distance calculation unit 42 calculates the measurement distance by multiplying the calculated delay time Td by the speed of light (velocity) to 1 / 2.

[0103] Furthermore, the above description uses the case where the pixel 321 of the distance image capturing device 1 has four charge storage sections CS1 to CS4 as an example. However, it is not limited to this. The pixel 321 of the distance image capturing device 1 may also be configured to have five or more (e.g., N, N≥5) charge storage sections CS.

[0104] When the pixel 321 of the distance image capturing device 1 has N (N≥5) charge storage units CS, in step S10, the distance calculation unit 42 obtains the charge amounts Q1 to QN stored in the charge storage units CS1 to CSN respectively. In step S11, the distance calculation unit 42 uses the obtained charge amounts Q1 to QN to calculate the charge amount Qb corresponding to the external light component. The method by which the distance calculation unit 42 calculates the charge amount Qb is the same as when the pixel 321 of the distance image capturing device 1 has four charge storage units CS1 to CS4, using the charge amounts stored in the charge storage units CS that have stored charge during a time when the reflected light RL is not received.

[0105] In step S12, the distance calculation unit 42 selects three charge storage units CS from charge storage units CS1 to CSN that have been allocated and have stored charge (including the trailing portion) corresponding to the reflected light RL. The method by which the distance calculation unit 42 selects the three charge storage units CS is, for example, to select the combination of the three charge storage units CS that continuously store charge, where the sum of the charge stored in each charge storage unit CS is the largest, as the three charge storage units CS that have been allocated and have stored charge corresponding to the reflected light RL. The distance calculation unit 42 uses the charge stored in each of the three charge storage units CS that have been allocated and have stored charge corresponding to the reflected light RL, and the charge Qb corresponding to the external light component, to calculate the charge Qs. The processing in steps S13 to S15 is the same as in the case where the pixel 321 of the distance image imaging device 1 has four charge storage units CS1 to CS4.

[0106] Alternatively, the pixel 321 of the distance image capturing device 1 may also be configured with three charge accumulation units CS1 to CS3. In this case, for each measurement, the distance image capturing device 1 performs two charge accumulation processes: a process that accumulates only the charge corresponding to the external light component (called the first process) and a process that accumulates the charge including the reflected light RL (called the second process). For example, the distance image capturing device 1 performs the first process in the first frame and the second process in the following frames. When performing the first process, the distance image capturing device 1 does not illuminate the light pulse PO and accumulates charge in the charge accumulation units CS1 to CS3 respectively. When performing the second process, the distance image capturing device 1 illuminates the light pulse PO and accumulates charge in the charge accumulation units CS1 to CS3 respectively.

[0107] In this case, in step S10, the distance calculation unit 42 obtains the charge amounts Q1ft to Q3ft accumulated by the charge accumulation units CS1 to CS3 respectively through the first processing. Additionally, the distance calculation unit 42 obtains the charge amounts Q1sd to Q3sd accumulated by the charge accumulation units CS1 to CS3 respectively through the second processing. In step S11, the distance calculation unit 42 uses any one or a combination of the obtained charge amounts Q1ft to Q3ft as the charge amount Qb corresponding to the external light component. In step S12, the distance calculation unit 42 uses the obtained charge amounts Q1sd to Q3sd and the charge amount Qb to calculate the charge amount Q2s. The method by which the distance calculation unit 42 calculates the charge amount Q2s is the same as when the pixel 321 of the distance image imaging device 1 has four charge accumulation units CS1 to CS4.

[0108] (Modifications of the implementation method) Here, a variation of the embodiment will be described. In this variation, as in the embodiment described above, the reflected light RL with waveform distortion is received. However, the difference from the embodiment described above is that the charge corresponding to the tail portion of the reflected light RL caused by distortion is accumulated in the charge accumulation section CS2.

[0109] Figure 11 This is a timing diagram illustrating an example of the timing of the driving pixel 321 in a modified embodiment. Figure 11 The "unit accumulation period" and "readout period" in the text are related to... Figure 4 They are the same, therefore their description is omitted. Additionally, Figure 11 The letters “L”, “R”, “G1” through “G3”, and “GD” are also related to… Figure 4 Since they are the same, their description is omitted.

[0110] exist Figure 11 In the example, with Figure 4 Similarly, a timing diagram is shown for the case where the reflected light RL is received by the distance image sensor 32 after a delay time Td from the moment of illumination of the light pulse PO. Figure 11 In the process, the charge corresponding to the reflected light RL is distributed and stored in charge storage sections CS1 and CS2. During the timing of charge storage section CS3 storing charge, the reflected light RL is not exposed to light, but the charge corresponding to external light components such as background light is stored in charge storage section CS3.

[0111] Figure 12 This diagram illustrates the process of the distance image processing unit 4 correcting the charge quantity in a modified example of the embodiment. Figure 12 In, with Figure 11Similarly, the reflected light RL arrives at the distance image sensor 32 with a delay time Td, and the charge corresponding to the reflected light RL is distributed and accumulated in charge accumulation units CS1 and CS2. In this case, charge accumulation unit CS1 is an example of a "first charge accumulation unit". Charge accumulation unit CS2 is an example of a "second charge accumulation unit".

[0112] Specifically, in the reflected light RL formed by the distorted trailing signal H20, the charge corresponding to a portion of region Sb (region Sb1) is accumulated in charge accumulation section CS1. Additionally, the charge corresponding to the remaining portion of region Sb (region Sb2) is accumulated in charge accumulation section CS2. Furthermore, the charge corresponding to region Sc is accumulated in charge accumulation section CS2.

[0113] When correcting the charge amount, in this modified example, as in the above embodiment, the irradiation time To of the irradiation pulse PO and the accumulation time Ta of the charge accumulating in the charge accumulation section CS are the same time interval. Furthermore, the area of ​​region Sa is the same as the area of ​​region Sc.

[0114] In addition, Figure 12 The timing diagram for "G3" is omitted, but it is consistent with... Figure 11 The same drive readout gate transistor G3 has a charge corresponding to the external light component stored in the charge storage section CS3.

[0115] The distance calculation unit 42 acquires an electrical signal equivalent to the amount of charge stored in the charge storage unit CS for one frame. The distance calculation unit 42 calculates the total amount of charge QSUM corresponding to the reflected light RL using the following equation (8).

[0116] QSUM=Q1#+Q2#……(8) Q1#=Q1-Qb Q2#=Q2-Qb Q3 = Qb in, Q1# is the amount of charge stored in the charge storage section CS1 corresponding to the reflected light RL. Q2# is the amount of charge stored in the charge storage section CS2 corresponding to the reflected light RL. Qb is the amount of charge stored in the charge storage section CS, corresponding to the external light component. Q1 is the amount of charge stored in the charge storage section CS1. Q2 is the amount of charge stored in the charge storage section CS2. Q3 is the amount of charge stored in the charge storage section CS3. The distance calculation unit 42 uses the total charge QSUM calculated by equation (8) and the tailing ratio Re stored in the waveform information 440 to calculate the charge Qs of the tailing portion by the following equation (9). Figure 12 The corresponding charge amount in region Sc.

[0117] Qs=QSUM×Re ……(9) in, QSUM = Q1# + Q2# Q1# is the amount of charge stored in the charge storage section CS1 corresponding to the reflected light RL. Q2# is the amount of charge stored in the charge storage section CS2 corresponding to the reflected light RL. Re is the trailing ratio. The distance calculation unit 42 sets the charge Qs of the trailing part calculated by equation (9) according to equation (10) as the charge Q2s stored by the charge storage unit CS2.

[0118] Qs = Q2s ……(10) in, Q2s is the amount of charge stored in the charge storage section CS2, corresponding to the trailing portion of the reflected light RL. The distance calculation unit 42 solves equation (9) for the charge Q2s. The charge Qs of the trailing part is known, so the charge Q2s can be calculated by equation (10).

[0119] Q2s = Qs ……(10) in, Q2s is the amount of charge stored in the charge storage section CS2, corresponding to the trailing portion of the reflected light RL. Qs is the charge of the trailing portion. The distance calculation unit 42 assumes that the area of ​​region Sa is the same as the area of ​​region Sc for calculation purposes, and performs the calculation by correcting the charge Q1# as in equation (11) and calculating the corrected charge Q1h. In addition, the distance calculation unit 42 corrects the charge Q2# as in equation (12) and calculates the corrected charge Q2h.

[0120] Q1h=Q1#+Q2s……(11) Q2h=Q2#-Q2s……(12) in, Q1h is the amount of charge stored in the corrected charge storage section CS1, corresponding to the reflected light RL. Q1# is the amount of charge stored in the charge storage section CS1 before correction, corresponding to the reflected light RL. Q2s is the amount of charge stored in the charge storage section CS2, corresponding to the trailing portion of the reflected light RL. Q2h is the amount of charge stored in the corrected charge storage section CS2, corresponding to the reflected light RL. Q2# is the amount of charge stored in the charge storage section CS2 before correction, corresponding to the reflected light RL. In the above description, the example given is that the pixel 321 of the distance image camera device 1 has three charge storage units CS1 to CS3, and one of the charge storage units CS3 stores only the charge corresponding to the external light component. However, it is not limited to this. The pixel 321 of the distance image camera device 1 may also be configured to have two charge storage units CS. In this case, the distance image camera device 1 performs two charge storage processes for each measurement: a process that stores only the charge corresponding to the external light component (called the first process) and a process that stores the charge including the reflected light RL (called the second process). For example, the distance image camera device 1 performs the first process in the first frame and the second process in the next frame. When performing the first process, the distance image camera device 1 does not irradiate the light pulse PO, and the charge storage units CS1 and CS2 store charge respectively. When performing the second process, the distance image camera device 1 irradiates the light pulse PO, and the charge storage units CS1 and CS2 store charge respectively.

[0121] In this case, in step S10, the distance calculation unit 42 obtains the charge amounts Q1ft and Q2ft accumulated by the charge accumulation units CS1 and CS2 respectively through the first processing. Additionally, the distance calculation unit 42 obtains the charge amounts Q1sd and Q2sd accumulated by the charge accumulation units CS1 and CS2 respectively through the second processing. In step S11, the distance calculation unit 42 uses either or both of the obtained charge amounts Q1ft and Q2ft as the charge amount Qb corresponding to the external light component. In step S12, the distance calculation unit 42 uses the obtained charge amounts Q1sd, Q2sd, and Qb to calculate the charge amount Q2s. In this modified example, the method by which the distance calculation unit 42 calculates the charge amount Q2s is the same as when the pixel 321 of the distance image imaging device 1 has three charge accumulation units CS1 to CS3.

[0122] As described above, the distance image imaging device 1 of the embodiment includes a light source unit 2, a light receiving unit 3, and a distance image processing unit 4. The light source unit 2 irradiates a light pulse PO into the measurement space where the subject OB exists. The light receiving unit 3 includes a pixel 321 and a vertical scanning circuit 323 (an example of a driving circuit). The pixel 321 includes a photoelectric conversion element PD and a plurality of charge storage units CS. The vertical scanning circuit 323 distributes charge to the charge storage units CS in the pixel 321 at a predetermined timing synchronized with the irradiation of the light pulse PO, and allows them to accumulate. The distance image processing unit 4 uses the amount of charge accumulated in each of the charge storage units CS to determine the measurement distance to the subject OB. The distance image processing unit 4 acquires waveform information 440. The waveform information 440 is information indicating the degree of waveform distortion in the rectangular signal used in signal processing up to the determination of the measurement distance. Based on the acquired waveform information 440, the distance image processing unit 4 corrects the amount of charge (charge Q1#, Q2#) stored in the charge storage unit CS to the amount of charge (charge Q1h, Q2h) stored in the rectangular signal without waveform distortion. The distance image processing unit 4 uses the corrected amount of charge to determine the measurement distance.

[0123] Therefore, the distance image camera 1 of the embodiment can correct the charge quantities (charge quantities Q1#, Q2#) that contain errors due to waveform distortion to charge quantities (charge quantities Q1h, Q2h) that reduce the errors. Thus, even when waveform distortion becomes significant in a rectangular signal, the measurement distance can be calculated with high accuracy.

[0124] Furthermore, in the distance image imaging device 1 of the embodiment, the waveform information 440 is a trailing ratio representing the proportion of the second area (the area of ​​region Sc) relative to the first area (the area of ​​region Sa + Sb) derived by time integration of signals H1 and H2 (an example of a rectangular signal). The first area is the integral value obtained by integrating the rectangular signal H1 (a rectangular signal with no waveform distortion, an example of the "first signal") from the start of rise Trs to the start of fall Tds. The second area is the integral value obtained by integrating the difference obtained by subtracting the trailing signal H2 (a rectangular signal with waveform distortion, an example of the "second signal") from the rectangular signal H1 from the start of rise Trs to the end of rise Tre. Alternatively, the second area is the integral value obtained by integrating the trailing signal H2 from the start of fall Tds to the end of fall Tde. The distance image processing unit 4 calculates the charge quantities Q1# and Q2#. The charge quantity Q1# is obtained by subtracting the charge quantity Qb corresponding to the external light component from the charge quantity Q1 (first charge quantity) stored in the charge storage unit CS1 (first charge storage unit). The charge quantity Q2# is obtained by subtracting the charge quantity Qb corresponding to the external light component from the charge quantity Q2 (second charge quantity) stored in the charge storage unit CS2 (second charge storage unit). The charge storage unit CS1 is the charge storage unit CS that first stores the charge quantity corresponding to the reflected light RL. The charge storage unit CS2 is the charge storage unit CS that then stores the charge quantity corresponding to the reflected light RL in the charge storage unit CS1. The distance image processing unit 4 uses waveform information 440 to calculate the charge quantity Q2s (first correction charge quantity). The charge quantity Q2s is the charge quantity Q2# stored in the charge storage unit CS2 corresponding to the reflected light RL that was not stored in the charge storage unit CS1 but was stored in the charge storage unit CS2 due to the distortion of the rectangular signal mentioned above. The distance image processing unit 4 uses the charge amount Q2s to correct the charge amount Q1#, for example, in equations (6) and (11), and calculates the charge amount Q1h. In addition, the distance image processing unit 4 uses the charge amount Q2s to correct the charge amount Q2#, for example, in equation (12), and calculates the charge amount Q2h.

[0125] Therefore, in the distance image camera device 1 of the embodiment, the amount of charge Q2s that was not accumulated in the charge accumulation section CS1 but was accumulated in the charge accumulation section CS2 due to waveform distortion can be calculated. As a result, the amount of charge that should have been accumulated in the charge accumulation section CS1 can be corrected back to its original state, and the measurement distance can be calculated with high accuracy.

[0126] Furthermore, in the distance image capturing device 1 of the embodiment, pixel 321 includes three charge storage units CS1 to CS3. The distance image processing unit 4 controls the timing of the charge storage, ensuring that the charge storage unit CS3 (an example of a "charge storage unit for external light storage"), which is different from charge storage units CS1 and CS2, does not store the charge corresponding to the reflected light RL. The distance image processing unit 4 sets the amount of charge stored in charge storage unit CS3 to a charge amount Qb corresponding to the external light component. Thus, in the distance image capturing device 1 of the embodiment, the amounts of charge Q1, Q2, and Qb stored for distributing the reflected light RL can be calculated in one frame. Therefore, compared to the case where the distance is determined by storing and reading the charge in two frames involving the first and second processing, the processing time can be shortened.

[0127] Furthermore, in the distance image imaging device 1 of the embodiment, pixel 321 has three charge storage units CS1 to CS3. Waveform information 440 is a trailing ratio representing the ratio of the second area (the area of ​​region Sc) to the first area (the area of ​​region Sa + Sb) derived by integrating the signals H1 and H2 (an example of a rectangular signal) over time. The first area is the integral value obtained by integrating the rectangular signal H1 (a waveform-undistorted rectangular signal, an example of the "first signal") from the rise start time Trs to the fall start time Tds. The second area is the integral value obtained by integrating the difference obtained by subtracting the trailing signal H2 (a waveform-distorted rectangular signal, an example of the "second signal") from the rectangular signal H1 from the rise start time Trs to the rise end time Tre. Alternatively, the second area is the integral value obtained by integrating the trailing signal H2 from the fall start time Tds to the fall end time Tde. The distance image processing unit 4 calculates the charge amounts Q1#, Q2#, and Q3#. The charge quantity Q1# is obtained by subtracting the charge quantity Qb corresponding to the external light component from the charge quantity Q1 (first charge quantity) stored in the charge storage unit CS1 (first charge storage unit). The charge quantity Q2# is obtained by subtracting the charge quantity Qb corresponding to the external light component from the charge quantity Q2 (second charge quantity) stored in the charge storage unit CS2 (second charge storage unit). The charge quantity Q3# is obtained by subtracting the charge quantity Qb corresponding to the external light component from the charge quantity Q3 (third charge storage unit) stored in the charge storage unit CS3 (third charge storage unit). The charge storage unit CS1 is the charge storage unit CS that first stores the charge quantity corresponding to the reflected light RL. The charge storage unit CS2 is the charge storage unit CS that subsequently stores the charge quantity corresponding to the reflected light RL from the charge storage unit CS1. The charge storage unit CS3 is the charge storage unit CS that subsequently stores the charge quantity corresponding to the reflected light RL from the charge storage unit CS2. The distance image processing unit 4 uses waveform information 440 to calculate the charge amount Q2s (first corrected charge amount). The distance image processing unit 4 uses waveform information 440 to calculate the charge amount Q3# (second corrected charge amount). The charge amount Q2s is the charge amount Q2# corresponding to the reflected light RL stored in the charge storage unit CS2 that was not stored in the charge storage unit CS1 but stored in the charge storage unit CS2 due to the distortion of the rectangular signal. The charge amount Q3# is the charge amount that was not stored in the charge storage unit CS2 but stored in the charge storage unit CS3 due to the distortion of the rectangular signal. The distance image processing unit 4 uses the charge amount Q2s to correct the charge amount Q1#, for example, according to equations (6) and (11), and calculates the charge amount Q1h. In addition, the distance image processing unit 4 uses the charge amounts Q2s and Q3# to correct the charge amount Q2#, for example, according to equation (7), and calculates the charge amount Q2h.

[0128] Therefore, the distance image camera device 1 of this embodiment can calculate the amount of charge Q3# accumulated in the charge accumulation section CS3 even if, assuming, due to waveform distortion, a charge corresponding to the reflected light RL is accumulated in the charge accumulation section CS3, which should not have accumulated therein. Thus, the amount of charge that should have been accumulated in the charge accumulation section CS2 can be corrected back to its original state, and the measurement distance can be calculated with high accuracy.

[0129] Furthermore, in the distance image capturing device 1 of the embodiment, pixel 321 has four charge storage units CS1 to CS4. The distance image processing unit 4 controls the timing of the charge storage, so that the charge storage unit CS (charge storage unit CS4, an example of an "external light storage charge storage unit") which is different from the charge storage units CS1 to CS3 among the four charge storage units CS1 to CS4 does not store the charge corresponding to the reflected light RL. The distance image processing unit 4 sets the amount of charge stored in the charge storage unit CS4 to the amount of charge Qb corresponding to the external light component. Therefore, in the distance image capturing device 1 of the embodiment, the amounts of charge Q1, Q2, Q3, and Qb allocated and stored by the reflected light RL can be calculated in one frame. Therefore, compared with the case where the distance is determined by accumulating and reading in two frames involving the first and second processing, the processing time can be shortened.

[0130] (Effects of the implementation method) Here, use Figure 13 The effects of the distance image camera device 1 according to the embodiment will be explained. Figure 13 This is a diagram illustrating the effects of the implementation method. Figure 13 This indicates the relationship between the actual distance (actual distance) and the measured distance. Figure 13 The horizontal axis represents the actual distance, and the vertical axis represents the measured distance. The distance here is the distance to the subject OB. Figure 13 In the figure, the uncorrected measurement distance, represented by the black circle, is calculated by substituting the charge amounts Q1 to Q3 into equation (1). The measured distance with correction, represented by the black triangle, is calculated using the corrected charge amounts Q1h and Q2h, calculated by correcting using waveform information 440, for example, equations (6) and (7). As shown in the figure, the measured distance with correction is consistent with the actual distance. On the other hand, the measured distance without correction is inconsistent with the actual distance, becoming a value containing error. That is, in the distance image camera device 1 of this embodiment, by using waveform information 440 to determine the measurement distance, a value closer to the actual distance can be calculated.

[0131] (Modifications of the implementation method) Figure 14 This is a diagram illustrating the characteristics of two time windows in a variation of the implementation method. The characteristics here represent the correspondence between the actual distance and the measured distance. Figure 14 The horizontal axis represents the actual distance, and the vertical axis represents the measured distance. Characteristic L0 represents the relationship between the ideal actual distance and the measured distance. Characteristic L1 represents the relationship between the actual distance and the measured distance in the first time window. Characteristic L2 represents the relationship between the actual distance and the measured distance in the second time window. As shown in the example in this figure, there are many cases where the correspondence between the actual distance and the measured distance differs between different time windows. Therefore, when using waveform information 440 that can be corrected with high accuracy in a certain time window to correct the distance in other time windows, high accuracy correction may not be possible.

[0132] As a countermeasure, in this modified example, waveform information 440 for each time window is pre-created in the storage unit 44 and stored therein. Therefore, even if the deviation (error) between the distance measured according to each time window and the actual distance differs, the distance can be easily and accurately calculated using the waveform information 440 corresponding to the time window.

[0133] All or part of the distance image capturing device 1 and the distance image processing unit 4 described in the above embodiments can also be implemented by a computer. In this case, the program for implementing this function can be recorded on a computer-readable recording medium, and the computer system can read and execute the program recorded on the recording medium. Furthermore, the "computer system" mentioned here includes hardware such as an operating system and peripheral devices. Additionally, the "computer-readable recording medium" refers to removable media such as floppy disks, optical disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into the computer system. Moreover, the "computer-readable recording medium" can include a recording medium that dynamically maintains the program for a short period of time, such as a communication line used to transmit the program via a network such as the Internet or a communication line such as a telephone line; or a recording medium that maintains the program for a certain period of time, such as volatile memory inside a computer system that serves as a server or client in this case. Furthermore, the program described above can be a program for implementing a part of the above function, a program that can implement the above function through a combination of programs already recorded in the computer system, or a program implemented using a programmable logic device such as an FPGA.

[0134] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the specific configuration is not limited to these embodiments, and also includes designs that do not depart from the spirit of the present invention.

[0135] Industrial availability According to the present invention, even when waveform distortion in the rectangular signal used for distance measurement becomes significant, the measurement distance can be calculated with high accuracy.

[0136] Symbol Explanation 1: Distance image camera device 2: Light source section 3: Light-receiving part 32: Distance Image Sensor 321: pixels 323: Vertical Scanning Circuit 4: Distance Image Processing Unit 41: Timing Control Department 42: Distance Calculation Unit 43: Measurement and Control Department 44: Storage Department 440: Waveform Information CS: Charge Accumulation Section PO: Light pulse

Claims

1. A distance image camera device, comprising: The light source irradiates a light pulse into the measurement space where the subject exists; The light-receiving section includes a pixel and a pixel driving circuit. The pixel comprises a photoelectric conversion element that generates a charge corresponding to the incident light and a plurality of charge storage sections for storing the charge. The pixel driving circuit distributes the charge to the charge storage sections in the pixel and stores it at a predetermined timing synchronized with the irradiation of the light pulse. The distance image processing unit uses the amount of charge stored in the charge storage section to determine the measurement distance to the subject. The aforementioned distance image processing unit is, Obtain waveform information representing the degree of waveform distortion in the rectangular signal used in the signal processing up to the point of determining the aforementioned distance. Based on the obtained waveform information, the amount of charge accumulated in the above-mentioned charge accumulation sections is corrected to the amount of charge accumulated when the waveform in the above-mentioned rectangular signal is undistorted. The measured distance is determined using the corrected charge amount described above. The waveform information described above represents the tailing ratio of the second area relative to the first area. The first area and the second area are derived by time integration of the rectangular signal. The aforementioned first area is the integral value of the rectangular signal (i.e., the first signal) with no waveform distortion, integrated from the start of its rise to the start of its fall. The aforementioned second area is the integral value obtained by integrating the difference between the first signal and the rectangular signal (i.e., the second signal) with waveform distortion, from the start of the rise to the end of the rise, or the integral value obtained by integrating the second signal from the start of the fall to the end of the fall. The aforementioned distance image processing unit is, The first charge accumulation section, which first accumulates a charge amount corresponding to the reflected light of the light pulse reflected by the subject, and the second charge accumulation section, which then accumulates charge in the first charge accumulation section, are among the plurality of charge accumulation sections described above. The first distance calculation charge is obtained by subtracting the charge corresponding to the external light component from the first charge accumulated in the first charge accumulation section, and the second distance calculation charge is obtained by subtracting the charge corresponding to the external light component from the second charge accumulated in the second charge accumulation section. The waveform information described above is used to calculate the first correction charge in the charge quantity used for the second distance calculation, which, due to the distortion of the rectangular signal, was not accumulated in the first charge accumulation section but was accumulated in the second charge accumulation section. The first correction charge is used to correct the first distance calculation charge and the second distance calculation charge.

2. The distance image camera device as described in claim 1, wherein, The aforementioned pixel has three of the aforementioned charge storage units. The aforementioned distance image processing unit is, The timing of charge accumulation in the external light accumulation section is controlled so that no charge corresponding to the reflected light is accumulated in the external light accumulation section, which is different from the first and second charge accumulation sections among the three charge accumulation sections. The amount of charge stored in the charge storage section for external light storage is made to correspond to the amount of charge of the external light component.

3. The distance image camera device as described in claim 1, wherein, The aforementioned pixel has three of the aforementioned charge storage units. The waveform information described above represents the tailing ratio of the second area relative to the first area. The first area and the second area are derived by time integration of the rectangular signal. The aforementioned first area is the integral value of the rectangular signal (i.e., the first signal) with no waveform distortion, integrated from the start of its rise to the start of its fall. The aforementioned second area is the integral value obtained by integrating the difference between the first signal and the rectangular signal (i.e., the second signal) with waveform distortion, from the start of the rise to the end of the rise, or the integral value obtained by integrating the second signal from the start of the fall to the end of the fall. The aforementioned distance image processing unit is, The three charge accumulation sections are: a first charge accumulation section that first accumulates a charge corresponding to the amount of reflected light from the light pulse reflected by the subject; a second charge accumulation section that accumulates charge in the first charge accumulation section; and a third charge accumulation section that accumulates charge in the second charge accumulation section. The calculations are as follows: a first distance calculation charge obtained by subtracting the charge corresponding to the external light component from the first charge accumulated in the first charge accumulation section; a second distance calculation charge obtained by subtracting the charge corresponding to the external light component from the second charge accumulated in the second charge accumulation section; and a third distance calculation charge obtained by subtracting the charge corresponding to the external light component from the third charge accumulated in the third charge accumulation section. The waveform information described above is used to calculate the first correction charge in the charge quantity used for the second distance calculation, which, due to the distortion of the rectangular signal, was not accumulated in the first charge accumulation section but was accumulated in the second charge accumulation section. The waveform information described above is used to calculate the second correction charge in the third distance calculation charge that, due to the distortion of the rectangular signal, was not accumulated in the second charge accumulation section but was accumulated in the third charge accumulation section. The first correction charge is used to correct the charge used for the first distance calculation. The charge used for the second distance calculation is corrected using the first and second correction charge quantities described above.

4. The distance image camera device as described in claim 3, wherein, The aforementioned pixel has four of the aforementioned charge storage units. The aforementioned distance image processing unit is, The timing of charge accumulation in the external light accumulation section is controlled so that no charge corresponding to the reflected light is accumulated in the external light accumulation section, which is different from the first, second, and third charge accumulation sections among the four aforementioned charge accumulation sections. The amount of charge stored in the charge storage section for external light storage is set as the amount of charge corresponding to the external light component.

5. A distance image capturing method, which is a distance image capturing method performed by a distance image capturing device, the distance image capturing device comprising: The light source irradiates a light pulse into the measurement space where the subject exists; The light-receiving section includes a pixel and a pixel driving circuit. The pixel comprises a photoelectric conversion element that generates a charge corresponding to the incident light and a plurality of charge storage sections for storing the charge. The pixel driving circuit distributes the charge to the charge storage sections in the pixel and stores it at a predetermined timing synchronized with the irradiation of the light pulse. The distance image processing unit uses the amount of charge stored in the charge storage section to determine the measurement distance to the subject. The aforementioned distance image processing unit is, Obtain waveform information representing the degree of waveform distortion in the rectangular signal used in the signal processing up to the point of determining the aforementioned distance. Based on the obtained waveform information, the amount of charge accumulated in the above-mentioned charge accumulation sections is corrected to the amount of charge accumulated when the waveform in the above-mentioned rectangular signal is undistorted. The measured distance is determined using the corrected charge amount described above. The waveform information described above represents the tailing ratio of the second area relative to the first area. The first area and the second area are derived by time integration of the rectangular signal. The aforementioned first area is the integral value of the rectangular signal (i.e., the first signal) with no waveform distortion, integrated from the start of its rise to the start of its fall. The aforementioned second area is the integral value obtained by integrating the difference between the first signal and the rectangular signal (i.e., the second signal) with waveform distortion, from the start of the rise to the end of the rise, or the integral value obtained by integrating the second signal from the start of the fall to the end of the fall. In the above-described distance image capturing method, the distance image processing unit, The first charge accumulation section, which first accumulates a charge amount corresponding to the reflected light of the light pulse reflected by the subject, and the second charge accumulation section, which then accumulates charge in the first charge accumulation section, are among the plurality of charge accumulation sections described above. The first distance calculation charge is obtained by subtracting the charge corresponding to the external light component from the first charge accumulated in the first charge accumulation section, and the second distance calculation charge is obtained by subtracting the charge corresponding to the external light component from the second charge accumulated in the second charge accumulation section. The waveform information described above is used to calculate the first correction charge in the charge quantity used for the second distance calculation, which, due to the distortion of the rectangular signal, was not accumulated in the first charge accumulation section but was accumulated in the second charge accumulation section. The first correction charge is used to correct the first distance calculation charge and the second distance calculation charge.

Citation Information

Patent Citations

  • Distance data generating method, distance image generating apparatus and photoelectronic sensor

    JP2007170856A