Distance measuring device, distance measuring method and storage medium

By determining the pattern through projection confidence and acquiring the pattern through distance, and combining the imaging device and data processing unit, the problem of inaccurate distance information in EVS during high-speed operation is solved, and more accurate and stable distance measurement is achieved.

CN123125461APending Publication Date: 2026-11-17SONY SEMICON SOLUTIONS CORP
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202580019157.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2024-03-11
Filing Date
2025-03-03
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing event-based vision sensors (EVS) struggle to consistently acquire accurate distance information during high-speed operation due to pixel response time characteristics and environmental factors, resulting in errors in the distance information.

Method used

A light source device projects a pattern to determine confidence and a distance to acquire the pattern. An imaging device captures the image. A data acquisition unit acquires the sensing information of each pixel. A confidence generation unit checks the pixel confidence and a processing unit selects the sensing information with the required confidence to determine the distance.

Benefits of technology

It improves the accuracy and stability of distance information, shortens distance calculation time, and reduces errors.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123125461A_ABST
    Figure CN123125461A_ABST
Patent Text Reader

Abstract

A ranging apparatus according to one embodiment of the present disclosure includes: a light source device that projects a confidence determination pattern and a distance acquisition pattern as illumination patterns onto a measurement object; a data acquisition unit that acquires sensing information of each pixel of an image obtained by imaging the measurement object onto which the confidence determination pattern or the distance acquisition pattern has been projected; a confidence generation unit that examines the sensing information of each pixel corresponding to the confidence determination pattern based on the confidence determination pattern, and generates a confidence level for each pixel based on the examination result; and a processing unit that selects sensing information of a pixel having a desired confidence level from multiple sensing information of the pixel corresponding to the distance acquisition pattern based on the confidence level of each pixel, and determines the distance to the measurement object using the selected sensing information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to ranging devices, ranging methods, and storage media. Background Technology

[0002] As a technique for determining the three-dimensional shape of an object, a technique known as spatial encoding method is known. In spatial encoding method, distance information (depth information) such as the three-dimensional shape is obtained by using multiple captured images, for example, by illuminating a stripe pattern with different periods. Since acquiring multiple captured images takes time, it needs to be accelerated. For example, a method for acquiring binary codes by using an event-based vision sensor (EVS) that senses changes in light at high speed has been disclosed (e.g., see Patent Document 1).

[0003] Reference List

[0004] Patent documents

[0005] PTL 1: JP 2021-124321 A Summary of the Invention

[0006] Technical issues

[0007] However, sensors such as EVS have pixel response time characteristics, and although they detect changes in light, some pixels react to the light with a delay. This occurs frequently when the sensor is operated at high speeds due to factors such as ambient illumination, distance to the subject, and the subject's reflectivity. Therefore, the sensor cannot properly capture reflected light from the object, potentially introducing errors in distance information. Because sensors such as EVS can only capture changes in brightness, it is difficult to obtain accurate distance information based on the response time characteristics described above. Therefore, it is difficult to consistently obtain accurate distance information.

[0008] Therefore, this disclosure provides a ranging device, a ranging method, and a storage medium capable of reliably obtaining accurate distance information.

[0009] Solution to the problem

[0010] A ranging apparatus according to one aspect of the present disclosure includes: a light source device configured to project a confidence determination pattern and a distance acquisition pattern as illumination patterns onto a target object to be measured; an imaging device configured to capture an image of the target object onto which the confidence determination pattern or the distance acquisition pattern is projected; a data acquisition unit configured to acquire sensing information of each pixel in the image acquired by the imaging device; a confidence generation unit configured to examine the sensing information of each pixel corresponding to the confidence determination pattern based on the confidence determination pattern, and generate a confidence level for each pixel based on the result of the examination; and a processing unit configured to select sensing information of pixels having a desired confidence level from the sensing information of each pixel corresponding to the distance acquisition pattern according to the confidence level of each pixel, and determine the distance to the target object to be measured by using the selected sensing information.

[0011] According to one aspect of the present disclosure, a ranging method executed by a computer includes: projecting a confidence determination pattern and a distance acquisition pattern as illumination patterns onto a target object to be measured; capturing an image of the target object onto which the confidence determination pattern or the distance acquisition pattern is projected; acquiring sensing information of each pixel in the image obtained by imaging; examining the sensing information of each pixel corresponding to the confidence determination pattern based on the confidence determination pattern, and generating a confidence level for each pixel based on the result of the examination; and selecting sensing information of pixels having a desired confidence level from the sensing information of each pixel corresponding to the distance acquisition pattern based on the confidence level of each pixel, and determining the distance to the target object by using the selected sensing information.

[0012] According to a computer-readable storage medium of one aspect of the present disclosure, a program is stored for causing a computer to perform the following operations: projecting a confidence determination pattern and a distance acquisition pattern as illumination patterns onto a measurement target object; capturing an image of the measurement target object onto which the confidence determination pattern or the distance acquisition pattern is projected; acquiring sensing information of each pixel in the image obtained by imaging; examining the sensing information of each pixel corresponding to the confidence determination pattern based on the confidence determination pattern, and generating a confidence level for each pixel based on the result of the examination; and selecting, based on the confidence level of each pixel, sensing information of pixels having a desired confidence level from the sensing information of each pixel corresponding to the distance acquisition pattern, and determining the distance to the measurement target object by using the selected sensing information. Attached Figure Description

[0013] [ Figure 1 [Illustration] is a diagram showing an example of the configuration of the ranging device according to the first embodiment.

[0014] [ Figure 2[Illustration] is a diagram showing an example of the configuration of the imaging apparatus according to the first embodiment.

[0015] [ Figure 3 [Illustration] is a diagram showing an example of the configuration of unit pixels according to the first embodiment.

[0016] [ Figure 4 [Illustration] is a diagram showing an example of the configuration of the detection unit according to the first embodiment.

[0017] [ Figure 5 [Illustration] is a diagram showing an example of the configuration of the current-voltage conversion unit according to the first embodiment.

[0018] [ Figure 6 [This is a diagram showing an example of the configuration of the subtractor and quantizer according to the first embodiment.]

[0019] [ Figure 7 [Illustration] is a diagram illustrating an operational example of the imaging apparatus according to the first embodiment.

[0020] [ Figure 8 [Illustration] is a diagram illustrating the operational details of the imaging apparatus according to the first embodiment.

[0021] [ Figure 9 [This is a diagram showing an example of the configuration of the overall control unit according to the first embodiment.]

[0022] [ Figure 10 [Illustration] is a diagram used to illustrate a timing control example according to the first embodiment.

[0023] [ Figure 11 [Illustration] is a diagram illustrating an example of an illumination pattern according to the first embodiment.

[0024] [ Figure 12 [Illustration] is a diagram illustrating a response example in the case of detecting white by performing pixel reset using a completely black image according to the first embodiment.

[0025] [ Figure 13 [Illustration] is a diagram illustrating a response example in the case of detecting black by performing pixel reset using a pure white image according to the first embodiment.

[0026] [ Figure 14 [Illustration] is a diagram illustrating a plurality of binary patterns according to the first embodiment.

[0027] [ Figure 15 [Illustration] is a diagram illustrating an example of encoding a plurality of binary patterns according to the first embodiment.

[0028] [ Figure 16 [Illustration] is a diagram used to illustrate the binary code according to the first embodiment.

[0029] [ Figure 17 [Illustration] is a diagram used to illustrate Gray code according to the first embodiment.

[0030] [ Figure 18 [Illustration] is a diagram showing an example of encoding a plurality of phase-shifting patterns (projection patterns) according to the first embodiment.

[0031] [ Figure 19 [Illustration] is a diagram showing an example of an ideal continuous pattern according to the first embodiment.

[0032] [ Figure 20 [Illustration] is a diagram showing an example of a discontinuous pattern according to the first embodiment.

[0033] [ Figure 21 [Illustration] is a diagram used to illustrate an example of depth calculation according to the first embodiment.

[0034] [ Figure 22 [This is a flowchart illustrating an operation example of the ranging device according to the first embodiment.]

[0035] [ Figure 23 [This is a diagram showing an example of the configuration of the overall control unit according to the second embodiment.]

[0036] [ Figure 24 [This is a diagram showing an example of the configuration of the overall control unit according to the third embodiment.]

[0037] [ Figure 25 [This is a diagram showing an example of the configuration of the overall control unit according to the fourth embodiment.]

[0038] [ Figure 26 [This is a diagram showing an example of the configuration of the overall control unit according to the fifth embodiment.]

[0039] [ Figure 27 [This is a diagram showing an example of the configuration of the overall control unit according to the sixth embodiment.]

[0040] [ Figure 28 [This is a diagram showing an example of the arrangement of unit pixels according to the sixth embodiment.]

[0041] [ Figure 29 [This is a diagram showing another arrangement example of a unit pixel according to the sixth embodiment.]

[0042] [ Figure 30 [Illustration] is a diagram showing an example of the imaging timing of the imaging apparatus according to the sixth embodiment.

[0043] [ Figure 31[This is a diagram showing another example of the imaging timing of the imaging apparatus according to the sixth embodiment.] Detailed Implementation

[0044] In the following, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Embodiments include examples, modifications, etc. It should be noted that the technology related to this disclosure is not limited to the embodiments. Furthermore, in the embodiments, repeated descriptions are omitted by labeling substantially the same parts with the same reference numerals.

[0045] One or more of the following embodiments can be implemented independently. Alternatively, at least a portion of each of the plurality of embodiments can be suitably combined with at least a portion of another example. These plurality of embodiments may include novel features that differ from each other. Therefore, each embodiment may serve a different purpose or solve a different problem and may produce different effects. It should be noted that the effects in the embodiments are merely illustrative and not limiting, and other effects may be provided.

[0046] This disclosure will be described in the following order.

[0047] 1. First Implementation Method

[0048] 1-1. Example of a distance measuring device configuration

[0049] 1-2. Imaging device

[0050] 1-2-1. Example of Imaging Device Configuration

[0051] 1-2-2. Example of unit pixel configuration

[0052] 1-2-3. Example of Detection Unit Configuration

[0053] 1-2-3-1. Configuration Example of Current-to-Voltage Conversion Unit

[0054] 1-2-3-2. Example of subtractor and quantizer configuration

[0055] 1-2-4. Operational Example of Imaging Device

[0056] 1-3. Overall control unit of the ranging device

[0057] 1-4. Timed Control Example

[0058] 1-5. Examples of Illuminated Patterns

[0059] 1-6. Examples of Depth Calculation

[0060] 1-7. Operation example of the distance measuring device

[0061] 2. Second Implementation Method

[0062] 2-1. Configuration Example of the Overall Control Unit

[0063] 3. Third Implementation Method

[0064] 3-1. Configuration Example of the Overall Control Unit

[0065] 4. Fourth Implementation Method

[0066] 4-1. Configuration Example of the Overall Control Unit

[0067] 5. Fifth Implementation Method

[0068] 5-1. Configuration Example of the Overall Control Unit

[0069] 6. Sixth Implementation Method

[0070] 6-1. Example of Imaging Device Configuration

[0071] 6-2. Example of unit pixel arrangement

[0072] 6-3. Operational Examples of Imaging Devices

[0073] 7. Functions and effects of each implementation method

[0074] 8. Other implementation methods

[0075] 9. Appendix

[0076] <1. First Implementation Method>

[0077] <1-1. Example of Distance Measuring Device Configuration>

[0078] Reference Figure 1 Here is an example of the configuration of the ranging device 1 according to this embodiment. Figure 1 This is a diagram showing an example of the configuration of the ranging device 1 according to this embodiment.

[0079] like Figure 1 As shown, the ranging device 1 according to this embodiment includes an imaging device 100, a projector 200, an overall control unit 300, and a storage unit 400. The ranging device 1 is, for example, a device that measures the three-dimensional shape of a target object A1 by measuring the distance from the ranging device 1 to the target object A1 using a technique called spatial encoding. It should be noted that the projector 200 is an example of a light source device, and the storage unit 400 is an example of a storage medium.

[0080] Projector 200 projects a predetermined image according to instructions from overall control unit 300. The predetermined image includes, for example, an illumination pattern. The illumination pattern includes, for example, a brightness and darkness pattern with different periods. Projector 200 emits illumination light towards the target object A1 based on the brightness and darkness pattern and projects the illumination pattern onto the target object A1. Projector 200 sequentially emits illumination light from each illumination pattern P0 to Pn as the predetermined projected image. Figure 1 In the example, four illumination patterns P0 to P3 are shown.

[0081] Illumination pattern P0 is an illumination pattern used to determine the threshold of imaging device 100. For example, illumination pattern P0 is an all-off pattern, that is, a black image where all areas are "dark" (completely black image). In other words, illumination pattern P0 is an illumination pattern that does not emit illumination light, and is also referred to as, for example, a background illumination pattern. Note that illumination pattern P0 can also be an all-on pattern, that is, a white image where all areas are "bright" (completely white image). Illumination patterns P1 to Pn are, for example, light and dark patterns in which the widths of the vertical stripes are different from each other.

[0082] The imaging device 100, according to instructions from the overall control unit 300, captures an image of the measurement target object A1 illuminated by the projector 200 with illumination light of each illumination pattern P0 to Pn. The imaging device 100 is, for example, an EVS and outputs a captured image for each pixel having a pixel value based on whether the amount of light received by the pixel exceeds a threshold.

[0083] exist Figure 1 In the example, the imaging device 100 sets (resets) a threshold according to the illumination timing of the illumination pattern P0 and obtains capture images S1 to Sn corresponding to each illumination pattern P1 to Pn. In other words, compared with the case of emitting the illumination pattern P0, the imaging device 100 outputs capture images S1 to Sn (hereinafter also referred to as differential images S1 to Sn) indicating whether the amount of light received by each pixel exceeds the threshold (whether the pixel becomes brighter).

[0084] In addition, Figure 1 In the example, for ease of understanding, the measured target object A1 is shown in the differential images S1 to Sn, but the actual differential images S1 to Sn indicate the case with the emitted illumination pattern P0 (see reference). Figure 1 The captured image S0) may be brightened or brightened compared to the image itself. Figure 1 The difference images S1 to Sn are different. This also applies to the following figures.

[0085] For example, when the imaging device 100 is a conventional image sensor that outputs brightness values ​​based on the amount of received light, it needs to output multiple captured images, each with multiple bits. Furthermore, when the imaging device 100 is a conventional image sensor, in subsequent signal processing via the overall control unit 300, it is necessary to calculate the difference in brightness values ​​for each pixel of the captured image and determine whether the pixel value of each pixel has become brighter. Therefore, calculating the distance to the target object A1 using a conventional image sensor via spatial encoding methods takes time.

[0086] However, the imaging apparatus 100 of this embodiment outputs differential images S1 to Sn, which indicate whether the amount of light received at each pixel during the illumination of each illumination pattern P1 to Pn has increased compared to the illumination of the illumination pattern P0. Therefore, the imaging apparatus 100 can output the differential images S1 to Sn to the overall control unit 300 at a higher speed than a conventional image sensor. Furthermore, in the subsequent signal processing performed by the overall control unit 300, the process of determining whether the pixel value of each pixel has brightened can be omitted. Thus, by using the ranging device 1 of this embodiment, the time required to calculate the distance to the target object A1 can be shortened.

[0087] The overall control unit 300 controls each component of the ranging device 1. The overall control unit 300 controls, for example, the projector 200 to emit illumination light with predetermined illumination patterns P0 to Pn. The overall control unit 300 also controls the imaging device 100 to reset its threshold while the projector 200 is emitting illumination light with illumination pattern P0. The overall control unit 300 controls the imaging device 100 to capture an image of the target object A1 while the projector 200 is emitting illumination light with predetermined illumination patterns P1 to Pn.

[0088] The overall control unit 300 acquires depth information (distance information to the target object A1) for each pixel of the multiple differential images S1 to Sn captured by the imaging device 100. Note that details of the method for acquiring (calculating) depth information by the overall control unit 300 will be described later.

[0089] Storage unit 400 stores various types of information. Storage unit 400 stores information required by the ranging device 1 for measuring the distance to the target object A1, such as illumination patterns P0 to Pn. Note that storage unit 400 can store, for example, programs that are read and executed by the overall control unit 300.

[0090] Note that here, the number of illumination patterns emitted by projector 200 is four, but not limited to this. The number of illumination patterns only needs to be multiple, and can be two or three, or even more than five. Furthermore, for example, the illumination pattern is described as a light and dark pattern of vertical stripes with different periods, but is not limited to this. For example, the illumination pattern can be a light and dark pattern of horizontal stripes. The illumination pattern can also be a light and dark pattern combining vertical and horizontal stripes. The illumination pattern can be any pattern, as long as the illumination pattern is binary encoded.

[0091] <1-2 Imaging Device>

[0092] <1-2-1. Example of Imaging Device Configuration>

[0093] The following will refer to Figure 2 Details of the imaging apparatus 100 according to this embodiment will be explained. Figure 2 This is a diagram showing an example of the configuration of the imaging apparatus 100 according to this embodiment.

[0094] like Figure 2 As shown, the imaging apparatus 100 according to this embodiment includes a pixel array unit 110, a control unit 111, and a signal processing unit 112.

[0095] In the pixel array unit 110, a plurality of unit pixels 1101 are arranged in a two-dimensional lattice. Each unit pixel 1101 includes, for example, a photoelectric conversion element (such as a photodiode) and a pixel circuit (in this embodiment, corresponding to the detection unit 1100 described later), which is configured to detect whether the current value of the photocurrent caused by the charge generated in the photoelectric conversion element (or the voltage value obtained by converting the current value) exceeds a predetermined threshold. Note that the pixel circuit can be shared by a predetermined number of photoelectric conversion elements. In this case, the unit pixel 1101 includes one photoelectric conversion element and a shared pixel circuit.

[0096] Multiple unit pixels 1101 can be divided into multiple pixel blocks, each pixel block including a predetermined number of unit pixels 1101. In the following text, in the horizontal direction ( Figure 2 A group of unit pixels 1101 or pixel blocks arranged in the left-right direction (in the image) is called a "row", and in the direction perpendicular to the row ( Figure 2 A group of unit pixels (1101) or pixel blocks arranged in the vertical direction (in the image) is called a "column".

[0097] A typical EVS is an asynchronous EVS, which is configured to read signals from the unit pixel 1101 corresponding to the detected pixel circuit when an event (address event) is detected in the pixel circuit. On the other hand, the imaging apparatus 100 of this embodiment is a synchronous EVS, which is configured to read signals from the unit pixel 1101 at predetermined timings according to control from the control unit 111.

[0098] It should be noted that the EVS detects an event when the brightness change of the light acquired by each unit pixel 1101 exceeds a predetermined threshold, and outputs the sensing information (e.g., coordinates, time, and polarity) of the unit pixel 1101 where the event occurred. Examples of events include bright response events (positive events) and dark response events (negative events). For example, the EVS can acquire an image (e.g., each difference image S1 to Sn) that includes the sensing information of the unit pixels 1101 where the event has occurred.

[0099] The control unit 111 controls the pixel array unit 110 and performs, for example, the reset of the threshold of each unit pixel 1101 and the reading of signals from the unit pixel 1101.

[0100] The signal processing unit 112 performs predetermined signal processing on the signal input from each unit pixel 1101, and provides the result of the signal processing as detection data (detection signal) to the overall control unit 300 through the signal line 112a.

[0101] <1-2-2. Example of unit pixel configuration>

[0102] The following will refer to Figure 3 This section describes a configuration example for each unit pixel 1101 according to this embodiment. Figure 3 This is a diagram illustrating an example of the configuration of each unit pixel 1101 according to this embodiment.

[0103] like Figure 3 As shown, each unit pixel 1101 according to this embodiment includes, for example, a light receiving unit 1103 and a detection unit 1100. Note that the control unit 111 may include, for example, logic circuitry.

[0104] For example, the light receiving unit 1103 includes a photoelectric conversion element 1104 such as a photodiode. The output of the light receiving unit 1103 is connected to the detection unit 1100.

[0105] The detection unit 1100 includes, for example, a current-to-voltage conversion unit 1110, a subtractor 1130, and a quantizer 1140. However, the detection unit 1100 also includes, for example, a buffer and a transmission unit. Note that details of the detection unit 1100 will be described later.

[0106] In this configuration, the photoelectric conversion element 1104 of the light receiving unit 1103 performs photoelectric conversion on the incident light to generate an electric charge. The charge generated by the photoelectric conversion element 1104 is input to the detection unit 1100 as a photocurrent having a current value according to the amount of charge.

[0107] <1-2-3. Configuration Example of Detection Unit>

[0108] The following will refer to Figure 4 Here is an example of the configuration of the detection unit 1100 according to this embodiment. Figure 4 This is a diagram showing an example of the configuration of the detection unit 1100 according to this embodiment.

[0109] like Figure 4 As shown, the detection unit 1100 includes the aforementioned current-to-voltage conversion unit 1110, subtractor 1130, and quantizer 1140 (see reference). Figure 3 In addition to the buffer 1120 and the transmission unit 1150, it also includes the buffer 1120 and the transmission unit 1150.

[0110] The current-to-voltage conversion unit 1110 converts the photocurrent (pixel signal) from the light receiving unit 1103 into a logarithmic voltage signal and outputs the generated voltage signal to the buffer 1120.

[0111] The buffer 1120 corrects the voltage signal from the current-to-voltage conversion unit 1110 and outputs the corrected voltage signal to the subtractor 1130.

[0112] Subtractor 1130 reduces the voltage level of the voltage signal from buffer 1120 according to the row drive signal from control unit 111, and outputs the reduced voltage signal to quantizer 1140.

[0113] The quantizer 1140 quantizes the voltage signal from the subtractor 1130 into a digital signal and outputs the generated digital signal as a detection signal to the transmission unit 1150.

[0114] The transmission unit 1150 transmits the detection signal from the quantizer 1140 to the control unit 111 and the signal processing unit 112.

[0115] <1-2-3-1. Configuration Example of Current-to-Voltage Conversion Unit>

[0116] The following will refer to Figure 5 Here is an example of the configuration of the current-voltage conversion unit 1110 according to this embodiment. Figure 5 This is a diagram showing an example of the configuration of the current-voltage conversion unit 1110 according to this embodiment.

[0117] like Figure 5As shown, the current-to-voltage conversion unit 1110 according to this embodiment includes an LG (LoG) transistor 1111, an amplifying transistor 1113, and a constant current circuit 1115. The current-to-voltage conversion unit 1110 can be a so-called source follower type current-to-voltage conversion unit.

[0118] For example, the source of LG transistor 1111 and the gate of amplifying transistor 1113 are connected to the light receiving unit 1103, in other words, the cathode of the photoelectric conversion element 1104 included in the light receiving unit 1103. The drain of LG transistor 1111 is connected to the power supply terminal VDD, for example. For example, the source of amplifying transistor 1113 is grounded, and the drain of amplifying transistor 1113 is connected to the power supply terminal VDD through a constant current circuit 1115.

[0119] Each of the LG transistor 1111 and the amplifying transistor 1113 may be constructed from, for example, an NMOS transistor. For example, the constant current circuit 1115 may be constructed from a loaded metal-oxide-semiconductor (MOS) transistor, such as a P-type MOS transistor.

[0120] A ring-shaped source follower circuit is used by, for example Figure 5 The connection relationships shown are configured accordingly. Therefore, the photocurrent from the light receiving unit 1103 is converted into a voltage signal with a logarithmic value according to its charge amount.

[0121] Note that the current-to-voltage conversion unit 1110 is not limited to a source follower type current-to-voltage conversion unit, but can also be, for example, a so-called gain-boost type current-to-voltage conversion unit. A gain-boost type current-to-voltage conversion unit includes, for example, two LG transistors, two amplifying transistors, and a constant current circuit. For example, a ring-shaped source follower circuit is also configured in a gain-boost type current-to-voltage conversion unit.

[0122] <1-2-3-2. Example of Subtractor and Quantizer Configuration>

[0123] The following will refer to Figure 6 This section describes a configuration example of the subtractor 1130 and the quantizer 1140 according to this embodiment. Figure 6 This is a diagram showing an example of the configuration of the subtractor 1130 and the quantizer 1140 according to this embodiment.

[0124] like Figure 6 As shown, the subtractor 1130 according to this embodiment includes a plurality of capacitors 1131 and 1133, an inverter 1132, and a switch 1134. The quantizer 1140 includes a comparator 1141.

[0125] One end of capacitor 1131 is connected to the output terminal of buffer 1120, and the other end of capacitor 1131 is connected to the input terminal of inverter 1132. Capacitor 1133 is connected in parallel with inverter 1132. Switch 1134 opens and closes the path connecting the two ends of capacitor 1133 according to the row drive signal. Inverter 1132 inverts the voltage signal input through capacitor 1131. Inverter 1132 outputs the inverted signal to the non-inverting input terminal (+) of comparator 1141.

[0126] When switch 1134 is turned on, the voltage signal Vinit is input to the buffer 1120 side of capacitor 1131. The opposite side is the virtual ground terminal. For convenience, it is assumed that the potential of the virtual ground terminal is zero. In this case, with the capacitance of capacitor 1131 being Cl, the potential Qinit stored in capacitor 1131 is represented by the following equation (1). Since the two ends of capacitor 1133 are short-circuited, the charge stored in it is zero.

[0127] Qinit = C1×Vinit...(1)

[0128] When switch 1134 is open and the voltage on the buffer 1120 side of capacitor 1131 becomes Vafter, the charge Qafter stored in capacitor 1131 is represented by the following equation (2).

[0129] Qafter = C1×Vafter...(2)

[0130] The charge Q02 stored in capacitor 1133 is represented by the following equation (3), where the output voltage is Vout.

[0131] Q02 = -C2×Vout...(3)

[0132] In this case, since the total charge of capacitors 1131 and 1133 does not change, the following equation (4) holds.

[0133] Qinit = Qafter + Q02...(4)

[0134] By substituting equations (1) to (3) into equation (4) and transforming them, we obtain equation (5).

[0135] Vout = - (C1 / C2) × (Vafter-Vinit)... (5)

[0136] Equation (5) represents the subtraction operation of the voltage signal, and the gain of the subtraction result is C1 / C2. Typically, maximizing the gain is desired, and therefore C1 is preferably designed to be large and C2 small. However, when C2 is too small, kTC noise increases and noise characteristics may deteriorate; therefore, the reduction in capacitance of C2 is limited to a tolerance range for noise. Furthermore, since each unit pixel 1101 is equipped with a detection unit 1100 including a subtractor 1130, each of capacitors C1 and C2 is area-constrained. The values ​​of capacitors C1 and C2 are determined taking these factors into account.

[0137] Comparator 1141 compares the voltage signal from subtractor 1130 with a predetermined threshold voltage Vth applied to the inverting input terminal (-). Comparator 1141 outputs a signal representing the comparison result as a detection signal to transmission unit 1150.

[0138] The overall gain A of the detection unit 1100 is expressed by the following formula (6), where the conversion gain of the current-voltage conversion unit 1110 is CG. log And the gain of buffer 1120 is "1".

[0139] [Mathematical Expression 1]

[0140] In equation (6), i photo _n is the photocurrent of the nth unit pixel 1101, and its unit is, for example, amperes (A). N is the number of unit pixels 1101 in the pixel block, and in this example, it is "1".

[0141] <1-2-4. Operational Example of Imaging Device>

[0142] The following will refer to Figure 7 An operational example of the imaging apparatus 100 according to this embodiment will be explained. Figure 7 This is a diagram illustrating an operation example (an operation example of reading detection signals (pixel values)) of the imaging apparatus 100 according to this embodiment. Figure 7 In the example, the horizontal axis represents time, and the vertical axis represents the vertical address (V address) of the unit pixel 1101 to be scanned.

[0143] like Figure 7 As shown, when light is emitted from the projector 200 into the illumination pattern P0 during time period T0, the imaging device 100 performs exposure synchronously with the emission of the projector 200 and generates charge Q0 according to the incident light.

[0144] Subsequently, during time period T1, the imaging device 100 sets (resets) the threshold Th of the detection unit 1100 based on the charge Q0 generated during time period T0. Specifically, the imaging device 100 is turned on. Figure 6 The switch 1134 shown in the figure inputs a voltage signal V0 generated by logarithmically transforming the amount of charge (photocurrent) generated during exposure using the irradiation pattern P0 to the buffer 1120 side of capacitor 1131. Therefore, the value subtracted from the voltage signal Vafter in the above equation (5) is set (voltage signal Vinit). Note that in Figure 7 The image shown is a captured image S0 with the charge Q0 generated in time period T1 as the pixel value.

[0145] When light is emitted from the projector 200 to illuminate the pattern P1 during time period T2, the imaging device 100 is exposed synchronously with the emission of the projector 200, and generates a charge Q1 according to the incident light.

[0146] During time period T3, the imaging device 100 subtracts the voltage signal V0 from the voltage signal V1 based on the charge Q1 generated in time period T2, and compares the subtraction result with a threshold voltage Vth to determine whether the charge Q1 generated in time period T2 is greater than the charge Q1 generated in time period T1. The imaging device 100 determines whether the pixel value of a unit pixel 1101 emitting light when illuminating pattern P1 is larger (brighter) than the pixel value of a unit pixel 1101 emitting light when illuminating pattern P0. In other words, by using the pixel value (voltage signal V0) emitting light in illuminating pattern P0 as the threshold Th, the imaging device 100 determines whether the pixel value (voltage signal V1) emitting light in illuminating pattern P1 is greater than the threshold Th.

[0147] During time period T4, the imaging device 100 acquires the differential image S1 by sequentially reading pixel values ​​indicating the comparison result (the result of the event) for each pixel. In this way, the imaging device 100 according to this embodiment is not an asynchronous operation that reads pixel values ​​when an event occurs, but a synchronous operation that reads pixel values ​​at predetermined intervals and performs a so-called global hold operation.

[0148] During time period T4, light is also emitted from the projector 200 to illuminate the pattern P2, and the imaging device 100 performs exposure synchronously with the illumination of the projector 200 and generates charge Q2 according to the incident light.

[0149] In a normal EVS, the threshold Th is reset when a comparison between the threshold Th and the voltage signal V1 is performed. When the EVS performs a global hold operation, the threshold Th is reset and set to the voltage signal V1 when a comparison between the threshold Th and the voltage signal V1 is performed during time period T3. Thus, when the threshold Th is reset, in the next threshold comparison, it is determined whether the pixel value has changed for each pixel value of the immediately preceding differential image S1.

[0150] Therefore, as Figure 7 As shown, in the imaging apparatus 100 according to this embodiment, the threshold Th is not reset after the threshold comparison in time period T3. Specifically, in time period T5, the imaging apparatus 100 performs a threshold comparison of the voltage signal V2 based on the charge Q2. Therefore, the imaging apparatus 100 determines whether the voltage signal V2 corresponding to the charge Q2 generated in time period T4 is greater than the threshold Th (voltage signal V0).

[0151] During time period T6, the imaging device 100 acquires a differential image S2 by sequentially reading pixel values ​​indicating the determination result for each pixel. This differential image S2 indicates the difference between the imaging result and the image obtained during the emission illumination pattern P0. Subsequently, the imaging device 100 similarly acquires differential images S3 to Sn.

[0152] <Details of the operation example of the imaging device>

[0153] The following will refer to Figure 8 The details of the operation of the imaging apparatus 100 according to this embodiment will be explained. Figure 8 This is a diagram illustrating the details of the operation of the imaging apparatus 100 according to this embodiment. Note that... Figure 8 The horizontal axis in the graph represents time.

[0154] Figure 8 (a) shows the illumination pattern of light emitted by projector 200. Figure 8 In example (a), the projector 200 emits light by using illumination pattern P0 as a reference illumination pattern, and then uses each illumination pattern P1 to Pn (in Figure 8 In the example, n=10) to emit light. Additionally, the projector 200, for example, during the calculation of the distance between the ranging device 1 and the target object A1, emits light... Figure 8 (a) shows the illumination patterns that emit light repeatedly. Note that the details of each illumination pattern P1 to Pn will be described later.

[0155] Figure 8 (b) shows the timing of the projector 200's illumination. Figure 8In (b), "High" indicates illumination (emission), and "Low" indicates no illumination (no emission). For example, the projector 200 repeats the high (illumination) and low (no emission) states at a predetermined cycle. Note that since the reference illumination pattern is an illumination pattern P0 without illumination, the projector 200 is in the "low" (no emission) state during the time period T00 corresponding to the reference illumination pattern.

[0156] Figure 8 (c) illustrates an example of the change in charge amount of the target unit pixel 1101 for a given unit pixel 1101. For example, when the projector 200 emits light in a reference illumination pattern, the charge amount of the target unit pixel 1101 is assumed to be "low". When the projector 200 emits light in a predetermined illumination pattern, the amount of light incident on the unit pixel 1101 that captures the image of the area illuminated by the illumination light is greater than the amount of incident light when the target unit pixel 1101 is illuminated with light from the reference illumination pattern. In this case, the amount of charge generated in the target unit pixel 1101 is greater than the amount of charge during illumination in the reference illumination pattern. Figure 8 In (c), the amount of charge added during the irradiation period relative to the reference irradiation pattern is denoted as "high".

[0157] Note that, according to each illumination pattern P1 to Pn, the target pixel 1101 can capture an image of the area not illuminated by the illumination light. In this case, the amount of light incident on the target pixel 1101 is the same as the amount of light incident on the target pixel 1101 during illumination in the reference illumination pattern, and the amount of charge generated in the target pixel 1101 is "low".

[0158] As described above, each illumination pattern P1 to Pn is a pattern including bright and dark vertical stripes. Therefore, even when the object unit pixel 1101 captures an image of an area not illuminated by the illumination light, other unit pixels 1101 can capture an image of an area illuminated by the illumination light. Figure 8 The dashed lines in (c) show the amount of charge generated in other unit pixels 1101 of the image captured in the area illuminated by the illumination light. In this way, the amount of charge generated by each unit pixel 1101 is different depending on the illumination pattern.

[0159] Figure 8 (d) shows the operating timing and threshold Th (reference voltage Vinit) of the imaging device 100. Figure 8In (d), the threshold Th is shown by a solid line. For example, when the projector 200 illuminates the pattern with light as a reference during time period T00 (during the non-illumination period), the imaging device 100 resets the threshold Th (global reference voltage reset). Subsequently, the imaging device 100 performs threshold determination (global detection) synchronously with the emission timing of the projector 200 during time periods T01 to T10, and outputs a detection signal for each unit pixel 1101.

[0160] For example, in object unit pixel 1101, such as Figure 8 As shown in (c), when the projector 200 illuminates the pattern P1 with emitted light, the charge increases. The imaging device 100 detects the increase in charge and outputs a detection signal indicating the increase. Figure 8 (The up arrow in (e)). For example, the imaging device 100 outputs "1" as a detection signal when the charge increases, and outputs "0" under other conditions.

[0161] It should be noted that, according to this embodiment, after the imaging device 100 resets the threshold Th in time period T00, it does not reset the threshold until time period T11. Therefore, the threshold Th of the imaging device 100 remains unchanged from time period T00 and is always kept "low" until time period T11.

[0162] As a comparative example, there is an asynchronous (arbitrator-type) imaging device. This imaging device is an asynchronous EVS and resets the threshold Th each time a threshold determination is performed. Specifically, the imaging device according to the comparative example outputs a detection signal when an event occurs, and resets the threshold Th each time a detection signal is output. It should be noted that even in synchronous EVSs, there are EVSs that reset the threshold Th each time a threshold determination is performed.

[0163] In a standard EVS, the projector 200 emits light repeatedly at predetermined intervals for distance calculation. Therefore, events occur simultaneously across a large number of unit pixels 1101, and these events are concentrated during the projector 200's emission timing. However, in an asynchronous EVS, the detection signal is output only when an event is initiated, resulting in a narrower output interface (I / F) bandwidth. Consequently, when events are concentrated at predetermined intervals, the detection signal cannot be output all at once, leading to misalignment between the EVS's detection of charge changes and the output of the detection signal, potentially causing the EVS to calculate an incorrect distance. In this situation, it's impossible to determine whether the distance calculation error is due to event concentration or movement of the target object A1. Furthermore, a standard EVS resets the threshold Th each time threshold determination is performed. Therefore, if threshold determination errors occur due to noise or other factors, the threshold Th is also incorrectly reset, potentially leading to subsequent threshold determination errors.

[0164] Therefore, in this embodiment, the readout timing of the imaging device 100 is synchronized at a predetermined timing, and a detection signal is output by a scanning method that simultaneously scans unit pixels 1101. Furthermore, in this embodiment, the threshold Th of the imaging device 100 is reset during illumination in the initial reference illumination pattern and is not reset thereafter. Therefore, the imaging device 100 according to this embodiment can align the timing of detecting changes in charge quantity (the timing of the projector 200 emitting light in the predetermined illumination pattern) with the timing of outputting the detection signal. Furthermore, the imaging device 100 according to this example can further reduce misdetermination in threshold determination.

[0165] <1-3. Overall Control Unit of the Distance Measuring Device>

[0166] The following will refer to Figure 9 Details of the overall control unit 300 of the ranging device 1 according to this example are explained. Figure 9 This is a diagram showing an example configuration of the overall control unit 300 according to this embodiment.

[0167] like Figure 9 As shown, the overall control unit 300 according to this embodiment includes a timing control unit 310, a projected image generation unit 320, a data acquisition unit 330, a code integration unit 340, a phase shift integration unit 350, a confidence generation unit 360, and a depth estimation unit 370. The overall control unit 300 performs the information processing functions and operations described below. Note that the depth estimation unit 370 is an example of a processing unit.

[0168] For example, the overall control unit 300 can be implemented by using random access memory (RAM) or the like as the working area, and executing the program stored in the ranging device 1 by a central processing unit (CPU), microprocessor unit (MPU), or the like. Alternatively, the overall control unit 300 can be implemented by an integrated circuit, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0169] Note that the internal configuration of the overall control unit 300 is not limited to... Figure 9 The configuration shown can be any other configuration that performs the information processing described later. For example, the overall control unit 300 can be connected to a predetermined network via wired or wireless means using a network interface card (NIC) or the like, and can receive various information from external servers or the like via the network.

[0170] The timing control unit 310 controls the imaging device 100 and the projector 200. The timing control unit 310 controls, for example, the illumination timing of the projector 200 and the imaging timing of the imaging device 100. The timing control unit 310 also performs, for example, control of switching the illumination pattern P0 to Pn of the light emitted by the projector 200.

[0171] The projection image generation unit 320, under the control of the timing control unit 310, transmits, for example, each of the illumination patterns P0 to Pn stored in the storage unit 400 to the projector 200. For example, the projection image generation unit 320 pre-generates various projection images, in other words, various illumination patterns P0 to Pn, and stores the data of the generated illumination patterns P0 to Pn in the storage unit 400.

[0172] For example, various illumination patterns P0 to Pn may include pixel reset patterns, binary patterns, confidence determination patterns, phase-shifting patterns, and moving object determination patterns. Light based on such patterns is emitted onto the measurement target object A1, in other words, projected onto the measurement target object A1, thereby obtaining various images (e.g., difference images S1 to Sn). In the difference images S1 to Sn, the difference image based on the illumination in the confidence determination pattern or the moving object determination pattern is used, for example, as a difference image for confidence generation. The difference image based on the illumination in the binary pattern or the phase-shifting pattern is used, for example, as a difference image for depth estimation. Note that the details of each pattern will be described later.

[0173] The data acquisition unit 330 acquires differential images S1 to Sn captured by the imaging device 100. The data acquisition unit 330 acquires, for example, the differential images S1 to Sn and timing information (frame ID) indicating the timing of capturing the differential images S1 to Sn, and outputs the acquired differential images S1 to Sn and the timing information. For example, when acquiring differential images S1 to Sn from the imaging device 100, the data acquisition unit 330 outputs multiple differential images from S1 to Sn used for confidence generation, along with the timing information of the differential images, to the confidence generation unit 360. The data acquisition unit 330 outputs multiple differential images from S1 to Sn used for depth estimation, along with the timing information of the differential images, to the code integration unit 340 and the phase shift integration unit 350. The data acquisition unit 330 may be, for example, a frame buffer storing the differential images S1 to Sn.

[0174] The code integration unit 340, for example, integrates the value of each pixel in multiple difference images (based on illumination difference images in a binary pattern) used for depth estimation from difference images S1 to Sn into a single code, thereby generating an integrated image. Each pixel in the difference image used for depth estimation indicates the difference with the background image (reference signal), with illuminated pixels represented by "1" and unilluminated pixels represented by "0". Therefore, for example, when the corresponding pixels in difference images S1 to Sn are all unilluminated pixels, the code integration unit 340 integrates the pixel values ​​as "00…0". For example, the code integration unit 340 obtains sensing information (e.g., coarse pixel position information) for each pixel from the integrated image based on the difference images used for depth estimation. The pixel position information is, for example, the coordinate information of the pixel where an event occurred.

[0175] Note that in a conventional rangefinder, the imaging device outputs a captured image with 10-bit pixel values ​​as a captured image corresponding to the illumination pattern. Therefore, the code integration unit included in a conventional rangefinder needs to determine whether each pixel of the captured image is an illuminated pixel or a non-illuminated pixel by using a threshold. However, the imaging device 100 according to this embodiment determines whether each pixel of the captured image is an illuminated pixel or a non-illuminated pixel by using a threshold (reference signal) and outputs the determination result as differential images S1 to Sn. Therefore, the code integration unit 340 according to this example only needs to integrate the code, and the thresholding processing of the differential images S1 to Sn can be omitted.

[0176] The phase-shift integration unit 350, for example, integrates the values ​​of corresponding pixels from multiple differential images (based on the illumination differential images in the phase-shift pattern) used for depth estimation from differential images S1 to Sn into a single code, thereby generating an integrated image. For example, the phase-shift integration unit 350 obtains sensing information (e.g., high-precision pixel position information) for each pixel from the integrated image based on the differential images used for depth estimation. The precision of the sensing information obtained by the phase-shift integration unit 350 is higher than the precision of the sensing information obtained by the code integration unit 340. For pixels that fail to properly acquire phase-shift pattern illumination, the phase-shift integration unit 350 sends abnormal information (e.g., a flag signal) indicating unreliable pixels to the confidence generation unit 360.

[0177] The confidence generation unit 360 calculates and generates the confidence score of each pixel in multiple differential images (differential images based on illumination in the confidence determination pattern and differential images based on illumination in the moving object determination pattern) among the differential images S1 to Sn for confidence score generation, based on the confidence determination pattern, the moving object determination pattern, and anomaly information, and outputs the generated confidence score of each pixel to the depth estimation unit 370. For example, the confidence generation unit 360 uses the confidence determination pattern, the moving object determination pattern, and the anomaly information obtained by the phase shift integration unit 350 to examine the sensing information of each pixel for each differential image, and generates the confidence score of each pixel based on the examination results.

[0178] It should be noted that, in addition to the multiple differential images acquired by the data acquisition unit 330 for confidence generation, the confidence generation unit 360 can also obtain the confidence of each pixel based on calibration information. The calibration information is information corresponding to, for example, the optical system and geometric position of the imaging device 100 and the projector 200, and is obtained through pre-calibration. The calibration information is pre-stored in, for example, the storage unit 400.

[0179] The depth estimation unit 370 estimates the distance (depth) to the target object A1 based on the integrated image generated by the code integration unit 340 and the integrated image generated by the phase-shift integration unit 350. For example, the illumination patterns P1 to Pn are binary encoded according to each illumination angle. Thus, the depth estimation unit 370 can associate each pixel with the illumination angle of the projector 200 by decoding the pixel values ​​of the pixels in the integrated image. The depth estimation unit 370 obtains information about the distance to the target object A1 by using the illumination angle of the projector 200 at each pixel.

[0180] For example, the depth estimation unit 370 estimates and obtains the distance to the target object A1 for pixels with the required confidence level generated by the confidence level generation unit 360. Pixels with the required confidence level are, for example, reliable pixels or pixels with a confidence level equal to or greater than a predetermined value. The depth estimation unit 370 obtains, for example, higher-precision sensing information by combining high-precision sensing information (e.g., pixel position information) obtained by the phase-shift integration unit 350 and coarse sensing information obtained by the code integration unit 340, and uses the higher-precision sensing information to estimate the distance to the target object A1.

[0181] For example, each of the phase-shift integration unit 350 and the code integration unit 340 acquires disparity information pixel by pixel. The phase-shift integration unit 350 acquires disparity information in 16 levels, such as -4.0, -3.5, -3.0, -2.5, -2.0, -1.5, -1.0, -0.5, 0.0, +0.5, +1.0, +1.5, +2.0, +2.5, +3.0, and +3.5, as high-precision sensing information. For example, the disparity information is repeated every 8 pixels, and therefore the same disparity information is output for the 0th, 8th, and 16th pixels. The code integration unit 340 acquires disparity information with an accuracy of approximately ±2 (pixels) (0, 4, 8, ...) as coarse sensing information (e.g., Gray code). This high-precision sensing information and coarse sensing information (low-precision sensing information) are integrated to obtain sensing information with higher precision. For example, when the output of the code integration unit 340 is 16 and the output of the phase shift integration unit 350 is +2.5, the integration result is 18.5.

[0182] This integration process will be described intuitively. For example, suppose the phase-shift integration unit 350 can measure distances from 0 cm to 99 cm in centimeters (cm), but cannot measure distances in meters (m). In this case, the phase-shift integration unit 350, for example, measures a distance of 50 cm and a distance of 1 m 50 cm as the same 50 cm. It is also assumed that the code integration unit 340 can measure distances only in meters (1 m, 2 m, 3 m, ...). When the coarse sensing information from the code integration unit 340 is integrated with the high-precision sensing information from the aforementioned phase-shift integration unit 350, for example, high-precision sensing information including both meter and centimeter units, such as 1 m 50 cm and 2 m 30 cm, can be obtained.

[0183] It should be noted that, in addition to the integrated image generated by the code integration unit 340 and the integrated image generated by the phase-shift integration unit 350, the depth estimation unit 370 can also calculate the distance to the target object A1 based on calibration information. The calibration information is the same as the calibration information described above. As mentioned above, for example, calibration information is obtained through pre-calibration and pre-stored in the storage unit 400.

[0184] <1-4. Example of timing control>

[0185] The following is for reference. Figure 10 This example illustrates the timing control method. Figure 10 This is a diagram illustrating a timing control example according to this embodiment.

[0186] like Figure 10As shown, between time t01 and time t12, the timing control unit 310 controls the imaging device 100 to perform imaging using the illumination pattern P0 and acquire the voltage signal Vinit (corresponding to...). Figure 10 (reference signal in the image). Because the illumination pattern P0 is the background illumination pattern in which the projector 200 does not emit light, the projection image generation unit 320 and the projector 200 do not operate. The timing control unit 310 controls the imaging device 100 to emit background light (illumination pattern P0) and acquires the voltage signal Vinit between time t01 and time t12 while the projector 200 is not emitting light.

[0187] The timing control unit 310 controls the projection image generation unit 320 to transmit the data of the illumination pattern P1 to the projector 200 between time t11 and time t12. The timing control unit 310 also controls the projector 200 to emit illumination light with the illumination pattern P1 between time t12 and time t13. The timing control unit 310 further controls the imaging device 100 to perform exposure between time t12 and time t13 and to detect the presence or absence of an event after time t13.

[0188] The timing control unit 310 controls the projection image generation unit 320 to transmit the data of the illumination pattern P2 to the projector 200 between time t21 and time t22. The timing control unit 310 also controls the projector 200 to emit illumination light with the illumination pattern P2 between time t22 and time t23. The timing control unit 310 further controls the imaging device 100 to perform exposure between time t22 and time t23 and to detect the presence or absence of an event after time t23.

[0189] It should be noted that, such as Figure 10 As shown, time t21 is the time between time t12 and time t13. Time t22 is the time after time t13. Therefore, since the timing control unit 310 starts illuminating the data transmission of pattern P2 before time t13, the light emission interval of the projector 200 and the exposure interval of the imaging device 100 can be shortened.

[0190] Similarly, the timing control unit 310 controls the projected image generation unit 320, the projector 200, and the imaging device 100 to perform data transmission of each illumination pattern P3 to Pn, light emission of the projector 200, and exposure of the imaging device 100 (detection of the presence or absence of an event).

[0191] <1-5. Examples of Illuminated Patterns>

[0192] The following will refer to Figures 11 to 20 An example illustration of the irradiation pattern according to this embodiment. Figure 11 This is a diagram illustrating an example of an illumination pattern according to this embodiment.

[0193] like Figure 11 As shown, the illumination pattern group P includes the pixel reset pattern group Pa, the confidence determination pattern group Pb, the binary pattern group Pc, the phase shift pattern group Pd, and the moving object determination pattern group Pe.

[0194] exist Figure 11 In the example, the pixel reset pattern group Pa includes two pixel reset patterns Pa1 and Pa2, the confidence determination pattern group Pb includes two confidence determination patterns Pb1 and Pb2, the binary pattern group Pc includes n binary patterns Pc1-Pcn, the phase shift pattern group Pd includes m phase shift patterns Pd1-Pdm, and the moving object determination pattern group Pe includes two moving object determination patterns Pe1 and Pe2. Each of these patterns serves as an example of an illumination pattern for a projected image.

[0195] The ranging device 1 resets all pixels of the imaging device 100 only once by continuously illuminating the pixel reset patterns Pa1 and Pa2. When illuminating other confidence determination patterns Pb1 and Pb2, binary patterns Pc1 to Pcn, phase shift patterns Pd1 to Pdm, and moving object determination patterns Pe1 and Pe2, only event detection is performed, and the above-mentioned reset of all pixels is not performed.

[0196] Note that in Figure 11 In the example, the pixel reset pattern group Pa, the confidence determination pattern group Pb, the binary pattern group Pc, the phase shift pattern group Pd, and the moving object determination pattern group Pe are used for illumination in the order described, but the order is not limited and can be changed appropriately. The number of patterns included in each pattern group is not limited. Depending on the use case, the phase shift pattern group Pd and the moving object determination pattern group Pe can be omitted.

[0197] (Pixel reset pattern)

[0198] Each pixel reset pattern Pa1, Pa2 is a pattern of light emitted from the projector 200 as a brightness reference for pixel reset. Figure 11 In the example, each pixel reset pattern Pa1, Pa2 is a completely white image (fully illuminated).

[0199] In addition, Figure 11In the example, two pixel reset patterns Pa1 and Pa2 are used, but only one pixel reset pattern Pa1 can be used. The number of patterns is not limited to this, and can be more than three. For example, in EVS, pixel response may be slowed down due to the influence of ambient light, subject reflectivity, etc., but pixel response can be stabilized by projecting the same pattern multiple times and performing pixel reset.

[0200] exist Figure 11 In the example, each pixel reset pattern Pa1, Pa2 is a completely white image, but it can also be a completely black image (completely non-illuminating). For example, in many ranging devices 1, the dark response is easier to handle than the bright response, so depending on the characteristics (EVS characteristics) of the ranging device 1, each pixel reset pattern Pa1, Pa2 can also be a completely black image. Each pixel reset pattern Pa1, Pa2 corresponds to, for example, the illumination pattern P0 described above (see reference). Figure 1 ).

[0201] Figure 12 This is a diagram illustrating an example of the response in the case of detecting white by performing pixel reset using a completely black image (completely non-emitting light) according to this embodiment. Figure 13 This is an illustration of an example of the response in the case of detecting black by performing pixel reset using a fully white image (fully illuminated) according to this embodiment.

[0202] like Figure 12 As shown, when white is detected by performing pixel reset using a completely black image—in other words, acquiring a bright response event—the ranging device 1 senses entire-surface emission with a very small delay; however, the event detection does not easily return to the initial state. Specifically, due to the pixel afterimage characteristics caused by acquiring strong light, the ranging device 1 may continuously output events for a long time.

[0203] On the other hand, such as Figure 13 As shown, when black is detected by performing pixel reset using a pure white image—in other words, when acquiring dark response events—a delay occurs in sensing of an entirely-surface non-emission environment (completely dark), but the recovery of event detection is rapid. Specifically, a delay may occur in sensing of an entirely non-emission environment, but event detection tends to recover to its original state faster than in a bright response.

[0204] (Confidence level determines the pattern)

[0205] return Figure 11Each confidence determination pattern Pb1 and Pb2 includes a completely black image and a completely white image used to determine the confidence of each pixel. The confidence of each pixel is, for example, information about each pixel indicating whether distance information (e.g., 3D information) can be accurately obtained by decoding the binary patterns Pc1 to Pcn and the phase-shifted patterns Pd1 to Pdm. For example, when each of the completely black and completely white images is projected only once, the confidence is considered as binary (1-bit) information indicating whether the pixel is reliable or unreliable.

[0206] The confidence generation unit 360 examines the sensing information (e.g., the occurrence of an event) for each pixel. The confidence generation unit 360 determines pixels that have experienced a dark response event (negative change) when the illumination pattern is a completely black image, and have not experienced an event (negative change or positive change) when the illumination pattern is a completely white image, as reliable pixels (pixels with high confidence). Specifically, the confidence generation unit 360 sets the confidence level of pixels that have experienced a dark response event when the illumination pattern is a completely black image, and have not experienced an event when the illumination pattern is a completely white image, to indicate that the pixel is reliable. Note that the confidence generation unit 360 may set the confidence level of pixels other than reliable pixels to indicate that the pixel is unreliable.

[0207] Due to response delays caused by factors such as light emitted from projector 200 not incident on the measurement target object A1, low reflectivity of the measurement target object A1, or a long distance between the object and projector 200, a dark response event may not occur when the illumination pattern is a completely black image, or a dark response event may occur when the illumination pattern is a completely white image. The confidence generation unit 360 can use these characteristics to generate a confidence level for each pixel. For example, the confidence generation unit 360 sets the confidence level of pixels that do not experience a dark response event when the illumination pattern is a completely black image, or the confidence level of pixels that experience a dark response event when the illumination pattern is a completely white image, as a confidence level indicating that the pixel is unreliable.

[0208] For example, even if the projector 200 emits light, the greater the distance from the projector 200 to the target object A1, the lower the brightness of the reflected light at the target object A1. Therefore, even at a location illuminated by light from the projector 200, the pixel signal may not exceed a threshold (reference signal), and the detection result may indicate the occurrence of a dark response event. Thus, when the illumination pattern is a completely white image, the confidence generation unit 360 sets the confidence of the pixels where a dark response event has occurred to the lowest value (e.g., zero).

[0209] Note that in Figure 11In the example, a set of images, including both a completely black image and a completely white image, is projected only once, but can be projected repeatedly (N times). In this case, the confidence level can, for example, take a value from 0 to N, indicating the number of times a normal response is obtained in N projections. Furthermore, a completely black image and a completely white image can be projected, but one or both of the completely black and completely white images can be projected multiple times. For example, a completely white image can be projected once, then a completely black image once, and finally a completely white image once. Alternatively, two completely white images can be projected consecutively, then a completely white image can be projected once, and finally a completely black image can be projected once.

[0210] (Binary pattern)

[0211] Figure 14 This is a diagram illustrating a plurality of binary patterns Pc1 to Pcn (n=5) according to this embodiment. Figure 15 This is a diagram illustrating an encoding example of multiple binary patterns Pc1 to Pcn (n=5) according to this embodiment. Each binary pattern Pc1 to Pcn is an example of a distance acquisition pattern.

[0212] like Figure 14 As shown, binary pattern Pc1 is a two-color shading pattern, where the left side of the pattern image is black (dark) and the right side is white (light). Binary pattern Pc2 is a shading pattern with two alternating black vertical stripes and two white vertical stripes. Binary pattern Pcn is one of the two... (n-1) One black vertical stripe and 2 (n-1) A pattern of alternating light and dark white vertical stripes. In this way, each binary pattern Pc1 to Pcn is a binary-coded pattern image.

[0213] like Figure 15 As shown, when black stripes are represented by "0" and white stripes by "1", each binary pattern Pc1 to Pcn follows the pattern image from the left ( Figure 15 The sequential codes (on the left side) are "00000", "00001", ... Note that in Figure 15 In the example, the binary patterns Pc1 to Pcn are arranged vertically. Figure 15 The downward arrow indicates time, and the horizontal arrow indicates the horizontal spatial distribution of black and white. Figure 15 In the diagram, the left side represents the least significant bit (LSB), and the right side represents the most significant bit (MSB).

[0214] The binary code is associated with each stripe in the horizontal direction of each binary pattern Pc1 to Pcn. In other words, each binary pattern Pc1 to Pcn is binary-coded for each illumination angle of light from the projector 200 in the horizontal direction.

[0215] Notice, Figure 14and Figure 15 The illumination pattern shown (e.g., binary patterns Pc1 to Pcn) is an example and is not limited thereto. The illumination pattern only needs to be a binary-coded pattern that allows identification of pixels corresponding to the captured image (e.g., differential images S1 to Sn), and can be, for example, a pattern of light and dark stripes rather than vertical stripes.

[0216] For example, when projector 200 illuminates any of the binary patterns Pc1 to Pcn, events generated by the periodic patterns may become thicker or thinner due to lens blurring of projector 200 or other reasons, and pixel position shifts may occur. The binary patterns Pc1 to Pcn can be converted into Gray-coded patterns to resolve pixel position shifts, thereby suppressing coding disorder.

[0217] However, even when converted to a Gray code pattern, the code can be shifted at the code boundary portions, and this problem is addressed by introducing phase shift patterns Pd1 to Pdm. Note that binary codes and Gray codes will be briefly described.

[0218] Figure 16 This is a diagram used to illustrate the binary code according to this embodiment. For example... Figure 16 As shown, binary code is the usual code (ideally) that is assigned in the order of binary numbers. In binary code, error codes may occur where multiple bits change between adjacent codes, which may lead to the calculation of depth with large errors (in reality: during strong reflections).

[0219] Figure 17 This is a diagram used to illustrate the Gray code according to this embodiment. For example... Figure 17 As shown, Gray code is a code where only one bit must change between adjacent codes (ideally). Therefore, even if the light spreads or narrows depending on the intensity of the reflected light, no false codes will be generated, thus minimizing depth errors (in reality: during periods of strong reflection).

[0220] (Phase-shifting pattern)

[0221] return Figure 11 Phase-shift patterns Pd1 to Pdm are used to illuminate the image while shifting the phase of a pattern with the same period, thereby improving the accuracy of the binary patterns Pcl to Pcn. In the case of a conventional image sensor, illumination is performed as a sine wave with grayscale; however, since the ranging device 1 (such as EVS) is a sensor that only captures changes that are present or absent, binary phase-shift patterns Pd1 to Pdm are used, each of which is binarized.

[0222] Figure 18 This is a diagram illustrating an example of the encoding of phase shift patterns Pdl to Pdm (projection patterns) according to this embodiment.

[0223] like Figure 18 As shown, the phase-shift integration unit 350 performs decoding by obtaining the centroids B1a of multiple patterns B1 in multiple images (images of corresponding projection patterns). For example, for each pattern B1 in each image acquired by illumination based on phase-shift patterns Pd1 to Pdm, the phase-shift integration unit 350 calculates the centroid B1a to generate high-precision pixel position information and outputs the generated high-precision pixel position information to the depth estimation unit 370.

[0224] As mentioned above, errors may occur in binary code patterns or Gray code patterns when the event becomes thicker or thinner. However, stable operation can be achieved in the phase-shift patterns Pd1 to Pdm used for centroid calculation even when the pattern thickness changes. In this way, each phase-shift pattern Pd1 to Pdm is a pattern used to calculate the centroid position from consecutive pixels (e.g., codes).

[0225] Figure 19 This is a diagram illustrating an example of an ideal continuous pattern B1 according to this embodiment. Figure 20 This is a diagram illustrating an example of a discontinuous pattern B1 according to this embodiment.

[0226] like Figure 19 As shown, an ideal continuous pattern B1 is typically obtained. In this case, centroid calculation is performed. On the other hand, as... Figure 20 As shown, for pixels that are not illuminated by light emitted from the projector 200 or have insufficient reflected light, an ideal continuous pattern B1 may not be obtained. In this case, centroid calculation is not performed, and a flag signal is output to the confidence generation unit 360 to indicate unreliable pixels. The flag signal is an example of anomaly information.

[0227] Note that, depending on the use case where the accuracy of 3D sensing may be low to a certain extent, the phase shift patterns Pd1 to Pdm can be simplified or omitted. As an example of simplifying the phase shift patterns Pd1 to Pdm, the number of patterns can be reduced.

[0228] (Move the object to determine the pattern)

[0229] return Figure 11 Each moving object defines patterns Pe1 and Pe2 as patterns used to check whether an event has occurred. Figure 11 In the example, each moving object determines patterns Pe1 and Pe2 as a completely white image (fully illuminated). Each moving object determines patterns Pe1 and Pe2 as, for example, the same patterns as the per-pixel reset patterns Pa1 and Pa2.

[0230] For example, when each moving object determines that patterns Pe1 and Pe2 are completely white images (fully illuminated), the confidence generation unit 360 considers pixels that have experienced dark response events as unreliable pixels and reduces the confidence of those pixels. Note that when each moving object determines that patterns Pe1 and Pe2 are completely black images (fully non-illuminated), pixels that have experienced bright response events are considered unreliable pixels and the confidence of those pixels is reduced.

[0231] Since the imaging device 100 can capture differential images S1 to Sn at high speed, the ranging device 1 can calculate the distance to the target object A1 even when the target object A1 moves to a certain extent. However, for example, if the target object A1 moves very fast, the distance calculation (detection) may fail. Therefore, the confidence generation unit 360 sets the confidence level of the position (pixel) where the target object A1 moves significantly to a lower confidence level.

[0232] Since the moving object determination patterns Pe1 and Pe2 are the same illumination patterns, when the target object A1 remains unchanged during illumination by both moving object determination patterns Pe1 and Pe2, the pixel value of each pixel in the difference image Sn indicates a dark response. When the target object A1 changes, such as when the target object A1 moves, the pixel values ​​at the changed locations in the difference image Sn become dark responses. Therefore, the confidence generation unit 360 sets the confidence of the pixels in the difference image Sn that have pixel values ​​representing dark responses to a low level, for example, to the lowest value.

[0233] Note that in Figure 11 The example uses two moving objects to determine patterns Pe1 and Pe2, but pattern Pe1 can be determined using only one moving object, and the number of patterns is unlimited, including more than three. For example, in EVS, pixel response may be slowed down due to ambient light, subject reflectivity, etc., but pixel response can be stabilized by projecting the same pattern multiple times and performing moving object determination.

[0234] exist Figure 11 In the example, each moving object determination pattern Pe1 and Pe2 is a completely white image (fully illuminated), but it can also be a completely black image (completely non-illuminated). For example, in many rangefinders 1, dark responses are easier to handle than bright responses, so depending on the characteristics of rangefinder 1 (EVS characteristics), each moving object determination pattern Pe1 and Pe2 can be a completely black image. Note that, depending on the condition of the subject being measured as the target object A1, the moving object determination patterns Pe1 and Pe2 can be omitted.

[0235] <1-6. Example of Depth Calculation>

[0236] The following will refer to Figure 21 This example demonstrates the depth calculation. Figure 21 This is a diagram used to illustrate an example of depth calculation according to this embodiment.

[0237] like Figure 21 As shown, the projector 200 projects the measurement target object A1 (corresponding to...) Figure 21 The angle of illumination when the "object" in the image emits light (corresponding to) Figure 21 The "light" in θ is derived from θ L This indicates that from the imaging device 100 (corresponding to Figure 21 The observation angle of the "camera" (in the text) when observing and measuring the target object A1 is θ c It is indicated that when the distance b between the projector 200 and the imaging device 100 is given as calibration information, the depth estimation unit 370 calculates the depth Z according to the following formula (7).

[0238] [Mathematical Expression 2]

[0239] Subsequently, the depth estimation unit 370 outputs the calculated depth Z as distance information. Note that the depth estimation unit 370 can output the confidence level associated with the calculated depth Z, set by the confidence generation unit 360, as the confidence level.

[0240] <1-7. Operational Examples of Distance Measuring Devices>

[0241] The following is for reference. Figure 22 This section describes an operational example of the ranging device 1 according to this example. Figure 22 This is a flowchart illustrating an operation example of the ranging device 1 according to this embodiment. Furthermore, the ranging device 1 repeatedly performs the following operations while measuring the distance to the target object A1: Figure 21 The operation shown.

[0242] like Figure 21 As shown, when the ranging device 1 illuminates the illumination pattern P0 (e.g., pixel reset patterns Pa1, Pa2) which serves as a background (non-luminous) pattern, it sets a reference signal (voltage signal Vinit) (step S101).

[0243] Subsequently, the ranging device 1 changes the illumination pattern to make the projector 200 emit light (step S102), and acquires a detection signal (step S103). The ranging device 1 determines whether all illumination patterns P1 to Pn (e.g., confidence determination patterns Pb1 and Pb2, binary patterns Pc1 to Pcn, phase shift patterns Pd1 to Pdm, and moving object determination patterns Pe1 and Pe2) have been illuminated (step S104).

[0244] When the projector 200 has not yet illuminated all the illumination patterns (No in step S104), the ranging device 1 returns to step S102. When the projector 200 has illuminated all the illumination patterns (Yes in step S104), the ranging device 1 integrates the acquired detection signals to generate an integrated image (step S105). Subsequently, the ranging device 1 sets the confidence level of each pixel based on the difference images S1 to Sn (step S106). The ranging device 1 estimates the depth of each pixel based on the integrated image and the confidence level (step S107).

[0245] As described above, the ranging device 1 according to this example resets the reference signal during the illumination of the illumination pattern P0, and acquires the differential images S1 to Sn without resetting the reference signal thereafter. Thus, the ranging device 1 is able to acquire the differential images S1 to Sn, representing the difference from the background image, at a higher speed.

[0246] Suppose that, for example, acquiring a 10-bit capture image requires the equivalent of 120 FPS (frames per second). Also assume that ranging device 1 acquires 10 capture images, excluding the background image, for distance calculation.

[0247] A typical rangefinder acquires a 10-bit capture image as a separate image from the background image. In this case, the typical rangefinder takes the equivalent of 11 FPS to acquire 11 capture images, including the background image. Therefore, between acquiring the initial background image and acquiring the final capture image, the target object A1 or the imaging device 100 may move, making it difficult to calculate the distance with high accuracy. Furthermore, because the typical rangefinder takes a long time to acquire the capture image, it is difficult to reduce the distance calculation time.

[0248] However, the imaging device 100 according to this embodiment is an EVS and therefore has a performance of 1000 FPS or higher. Therefore, the ranging device 1 can acquire differential images S1 to Sn at a higher speed. Thus, the possibility of the target object A1 or the imaging device moving between acquiring the initial differential image and acquiring the final differential image is low, enabling high-precision distance calculation. Furthermore, the distance calculation time of the ranging device 1 can be significantly reduced.

[0249] <2. Second Implementation Method>

[0250] <2-1. Configuration Example of the Overall Control Unit>

[0251] Reference Figure 23 An example of the configuration of the overall control unit 300 according to this embodiment is described. Figure 23This is a diagram illustrating an example configuration of the overall control unit 300 according to this embodiment. This embodiment is essentially the same as the first embodiment (see reference). Figure 9 However, the differences will be explained.

[0252] like Figure 23 As shown, the overall control unit 300 according to this embodiment includes, in addition to the components according to the first embodiment, a confidence summation unit 380 and a sensor control unit 390. By using the confidence summation unit 380 and the sensor control unit 390, the overall control unit 300 is configured to appropriately control the threshold for event detection of the imaging device 100 when the number of events for generating the confidence level (determination of confidence level) for each pixel is small.

[0253] The confidence summation unit 380 detects how many pixels have the required confidence level, for example, how many pixels have a high confidence level (the number of pixels with high confidence). For example, the confidence summation unit 380 calculates the number of pixels with a confidence level equal to or greater than a predetermined value as the number of pixels with high confidence. Furthermore, the predetermined value is, for example, preset, but can also be changed. The setting and changing of this predetermined value is also the same in other embodiments.

[0254] The sensor control unit 390 controls the imaging device 100 based on the number of pixels with high confidence obtained by the confidence summation unit 380. For example, when the number of pixels with high confidence (in other words, the number of pixels in the imaging device 100 that can detect events) is less than a predetermined value, the sensor control unit 390 sends a setting for lowering the event detection threshold of the imaging device 100 to the imaging device 100, thereby increasing the number of pixels in the imaging device 100 that can detect events.

[0255] <3. Third Implementation Method>

[0256] <3-1. Configuration Example of the Overall Control Unit>

[0257] Reference Figure 24 An example of the configuration of the overall control unit 300 according to this embodiment is described. Figure 24 This is a diagram illustrating an example configuration of the overall control unit 300 according to this embodiment. This embodiment is basically the same as the second embodiment (see reference). Figure 23 However, the differences will be explained.

[0258] like Figure 24As shown, the overall control unit 300 according to this embodiment, in addition to the components according to the second embodiment, also includes a light quantity control unit 395. The light quantity control unit 395 controls the light quantity of the projector 200 based on the number of pixels with high confidence obtained by the confidence summation unit 380. For example, when the number of pixels with high confidence (in other words, the number of pixels in the imaging device 100 capable of detecting events) is less than a predetermined value, the light quantity control unit 395 sends a setting to the projector 200 to increase the light quantity of the projector 200, thereby increasing the number of pixels in the imaging device 100 capable of detecting events.

[0259] <4. Fourth Implementation Method>

[0260] <4-1. Configuration Example of the Overall Control Unit>

[0261] Reference Figure 25 An example of the configuration of the overall control unit 300 according to this embodiment is described. Figure 25 This is a diagram illustrating an example configuration of the overall control unit 300 according to this embodiment. This embodiment and the third embodiment (see reference) Figure 24 They are basically the same, but the differences will be explained.

[0262] like Figure 25 As shown, the timing control unit 310 according to this embodiment controls the operation timing of each of the imaging device 100 and the projector 200 based on the number of pixels with high confidence obtained by the confidence summation unit 380. For example, when the number of normal pixels that have experienced a dark response event during the projection of a completely black image and have not experienced an event during the projection of a completely white image is less than a predetermined value, the timing control unit 310 delays the operation timing of each of the imaging device 100 and the projector 200. Therefore, even when there is a pixel response delay in the imaging device 100, the number of normal pixels performing normal event detection operations can be increased.

[0263] <5. Fifth Implementation Method>

[0264] <5-1. Configuration Example of the Overall Control Unit>

[0265] Reference Figure 26 An example of the configuration of the overall control unit 300 according to this embodiment is described. Figure 26 This is a diagram illustrating an example configuration of the overall control unit 300 according to this embodiment. This embodiment and the fourth embodiment (see reference) Figure 25 They are basically the same, but the differences will be explained.

[0266] like Figure 26As shown, the overall control unit 300 according to this embodiment includes, in addition to the components according to the fourth embodiment, an illuminance sensor 500. The illuminance sensor 500 acquires the ambient brightness. For example, the illuminance sensor 500 acquires the brightness (i.e., illuminance) around the target object A1. For example, a common image sensor or illuminance meter can be used as the illuminance sensor 500.

[0267] The timing control unit 310 controls the operation timing of the imaging device 100 and the projector 200 based on the illuminance obtained by the illuminance sensor 500. For example, the timing control unit 310 maintains the operation timing when the illuminance obtained by the illuminance sensor 500 is equal to or greater than a predetermined value, and delays the operation timing when the illuminance obtained by the illuminance sensor 500 is lower than the predetermined value.

[0268] Typically, in a ranging device 1 such as EVS, the delay characteristics deteriorate at low illumination. Therefore, when the illumination is low, the timing control unit 310 delays the operation timing, so that even when there is a pixel response delay in the imaging device 100, the number of pixels capable of detecting events can be increased.

[0269] <6. Sixth Implementation Method>

[0270] <6-1. Example of Imaging Device Configuration>

[0271] Reference Figure 27 Here is an example of the configuration of the imaging apparatus 100 according to this embodiment. Figure 27 This is a diagram illustrating an example configuration of the imaging apparatus 100 according to this embodiment. This embodiment is essentially the same as the first embodiment (see reference). Figure 2 (etc.), but the differences will be explained.

[0272] The difference is that, in addition to acquiring the distance to the target object A1, the ranging device 1 also acquires a color image (RGB image). The imaging device 100 is configured to output a detection signal, for example, when performing ranging, and not to output a detection signal when acquiring a color image.

[0273] like Figure 27 As shown, the imaging apparatus 100 according to this embodiment, in addition to the pixel array unit 110, the control unit 111, and the signal processing unit 112 (see reference 112), Figure 2 In addition to the above, it also includes a drive circuit 113, an output circuit 114, and a timing control unit 115 to obtain the brightness value (pixel signal) of the color image.

[0274] The driving circuit 113 includes a shift register and an address decoder, and drives each unit pixel 1101 of the pixel array unit 110. Electrical signals output from the unit pixels 1101 of the column selected and scanned by the driving circuit 113 are input to the output circuit 114 via corresponding output signal lines. The output circuit 114 includes column AD, and outputs the electrical signals input from the unit pixels 1101 as pixel signals to the external storage unit 400 or the overall control unit 300.

[0275] The timing control unit 115 includes a timing generator configured to generate various timing signals, and controls the drive circuit 113 and the output circuit 114 based on the various timing signals generated by the timing generator.

[0276] <6-2. Example of unit pixel arrangement>

[0277] Reference Figure 28 and Figure 29 An example of the arrangement of unit pixels 1101 according to this embodiment is described. Figure 28 This is a diagram showing an example of the configuration of unit pixel 1101 according to this embodiment. Figure 29 This is a diagram showing another configuration example of the unit pixel 1101 according to this embodiment.

[0278] like Figure 28 and Figure 29 As shown, the plurality of unit pixels 1101 according to this embodiment include a plurality of normal pixels R, G, and B that receive any one of red light, green light, and blue light, and a plurality of infrared light pixels IR that receive infrared light. In other words, the imaging apparatus 100 according to this embodiment includes normal pixels R, G, and B for capturing color images (RGB images) and infrared light pixels IR for capturing images for ranging.

[0279] Each of the normal pixels R, G, and B includes one of the R, G, and B color filters stacked on the light-receiving surface of the photoelectric conversion unit (not shown). The normal pixels R, G, and B constitute, for example, a Bayer array in pixel array unit 110.

[0280] Each infrared pixel IR includes a filter in which an infrared filter capable of receiving infrared light (in other words, light with wavelengths in the infrared region) is stacked on the light-receiving surface of a photoelectric conversion unit (not shown). The infrared pixels IR are arranged in a predetermined pixel row at predetermined intervals.

[0281] exist Figure 28 In the example, infrared pixels IR and normal pixels G are arranged alternately in a predetermined pixel row. Figure 29In the example, the infrared pixels IR are arranged sequentially at positions corresponding to the normal pixels G in the Bayer array of normal pixels R, G and B in the predetermined pixel row, and are adjacent to the normal pixels B in the same pixel row.

[0282] <6-3. Operational Examples of Imaging Devices>

[0283] The following will refer to Figure 30 and Figure 31 An operational example of the imaging apparatus 100 according to this embodiment will be explained. Figure 30 This is a diagram illustrating an example of the imaging timing of the imaging apparatus 100 according to this embodiment. Figure 31 This is a diagram illustrating another example of the imaging timing of the imaging apparatus 100 according to this embodiment.

[0284] like Figure 30 As shown, the imaging apparatus 100 according to this embodiment captures an RGB image as a color image, and then captures differential images S1 to Sn (in the image) for depth calculation. Figure 30 In the example, n=7). In other words, the imaging device 100 first captures an RGB image using normal pixels R, G, and B, and then captures differential images S1 to Sn for depth calculation using infrared pixels IR. Note that in Figure 30 A reference signal is set in S0 as shown.

[0285] More specifically, the imaging device 100 begins exposure of the normal pixels R, G, and B. Subsequently, the imaging device 100 sets a reference signal (S0) during the illumination of the illumination pattern P0. Then, the imaging device 100 acquires differential images S1 to S7 corresponding to the illumination patterns P1 to P7.

[0286] The imaging device 100 repeatedly acquires RGB images and differential images S1 to S7 for depth calculation, for example, in units of 1 / 30 to 1 / 120 of a second. Therefore, the imaging device 100 can output the differential images S1 to S7 for depth calculation at high speed. Thus, the RGB image and the differential images S1 to Sn for depth calculation can be captured by the same imaging device 100.

[0287] exist Figure 30 The example illustrates the case where the imaging device 100 sequentially captures an RGB image and differential images S1 to Sn for depth calculation, but this disclosure is not limited thereto. For example, as Figure 31 As shown, the imaging device 100 can capture RGB images and differential images S1 to Sn for depth calculation in parallel.

[0288] like Figure 31As shown, the imaging device 100 captures RGB images using normal pixels R, G, and B, and in parallel with the capture of the RGB images, captures differential images S1 to Sn (in the image) using infrared pixels IR for depth calculation. Figure 31 In the example, n=8). It should be noted that in image capture using infrared light pixels IR, the imaging device 100 first sets a reference signal (background) and then acquires differential images S1 to S8. The imaging device 100 repeatedly performs the capture of RGB images and differential images S1 to S8 for depth calculation, for example, in units of 1 / 30 second.

[0289] In this way, since the imaging device 100 simultaneously captures RGB images and differential images S1 to S8 for depth calculation, the RGB images and differential images S1 to Sn for depth calculation can be captured by the same imaging device 100.

[0290] <7. Functions and Effects of Each Implementation Method>

[0291] As described above, according to various embodiments, the ranging device 1 includes: a light source device (e.g., a projector 200) configured to project a confidence determination pattern (e.g., confidence determination patterns Pb1 and Pb2) and a distance acquisition pattern (e.g., binarized patterns Pc1 to Pcn) as illumination patterns onto the target object A1; an imaging device 100 configured to capture an image of the target object A1 onto which the confidence determination pattern or the distance acquisition pattern is projected; a data acquisition unit 330 configured to acquire sensing information of each pixel in the image acquired by the imaging device 100; a confidence generation unit 360 configured to examine the sensing information of each pixel corresponding to the confidence determination pattern based on the confidence determination pattern, and generate a confidence level for each pixel based on the examination result; and a processing unit (e.g., a depth estimation unit 370) configured to select sensing information of pixels with a desired confidence level from the sensing information of each pixel corresponding to the distance acquisition pattern according to the confidence level of each pixel, and determine the distance to the target object A1 by using the selected sensing information (see reference). Figure 9 (etc.). Therefore, for example, even when the ranging device 1 is operating at high speed, the distance to the target object A1 can be obtained using the sensing information of the pixels with the required confidence level, thereby enabling the stable acquisition of accurate distance information.

[0292] Confidence determination patterns (e.g., confidence determination patterns Pb1 and Pb2) can include fully dark patterns (fully black images) and fully bright patterns (fully white images) (see reference). Figure 11 Therefore, the confidence level of each pixel can be reliably obtained.

[0293] A light source device (e.g., projector 200) can project a completely dark pattern, and then can project a completely bright pattern (see reference). Figure 11 Therefore, the confidence level of each pixel can be reliably obtained.

[0294] The confidence generation unit 360 can set the confidence of a pixel that indicates a dark response event when the confidence determination pattern is a fully dark pattern and indicates no event when the confidence determination pattern is a fully bright pattern as the confidence of that pixel (see reference). Figure 9 and Figure 11 (etc.). Therefore, the confidence level of each pixel can be accurately obtained.

[0295] The confidence generation unit 360 can set the confidence level of a pixel that indicates no dark response event has occurred when the confidence level determines the pattern to be a completely dark pattern, or the confidence level of a pixel that indicates a dark response event has occurred when the confidence level determines the pattern to be a completely bright pattern, as the confidence level of that pixel as unreliable (see reference). Figure 9 and Figure 11 (etc.). Therefore, the confidence level of each pixel can be accurately obtained.

[0296] The processing unit (e.g., depth estimation unit 370) can set pixels with a confidence level equal to or greater than a predetermined value as pixels with the desired confidence level (see reference). Figure 9 and Figure 11 (etc.). Therefore, the confidence level of each pixel can be accurately obtained.

[0297] A light source device (e.g., projector 200) can project multiple phase-shift patterns Pd1 to Pdm as illumination patterns onto the measurement target object A1. The data acquisition unit 330 can acquire sensing information for each pixel in multiple images obtained by capturing images of the measurement target object A1 onto which multiple phase-shift patterns Pd1 to Pdm are sequentially projected. Furthermore, the confidence generation unit 360 can examine the sensing information of each pixel corresponding to the confidence determination pattern (e.g., confidence determination patterns Pb1 and Pb2) and the multiple phase-shift patterns Pd1 to Pdm based on the confidence determination pattern and the multiple phase-shift patterns Pd1 to Pdm, and can generate a confidence level for each pixel based on the result of this examination (see reference). Figure 9 and Figure 11 (etc.). Therefore, the confidence level of each pixel can be accurately obtained.

[0298] The ranging device 1 may further include: an acquisition unit (e.g., a phase-shift integration unit 350) configured to acquire abnormal information indicating that a pixel has not properly acquired sensing information based on the phase-shift pattern Pdm; and a confidence generation unit 360 may, based on a confidence determination pattern (e.g., confidence determination patterns Pb1 and Pb2) and the abnormal information, examine the sensing information of each pixel corresponding to the confidence determination pattern and the plurality of phase-shift patterns Pd1 to Pdm, and may generate a confidence level for each pixel based on the result of the examination (see reference). Figure 9 and Figure 11 (etc.). Therefore, the confidence level of each pixel can be accurately obtained.

[0299] The ranging device 1 may further include: a first integration unit (e.g., code integration unit 340) configured to acquire first sensing information (e.g., first pixel position information) for each pixel based on a first integrated image obtained by integrating multiple images corresponding to multiple distance acquisition patterns (e.g., binary patterns Pc1 to Pcn); and a second integration unit (e.g., phase-shift integration unit 350) configured to acquire second sensing information (e.g., second pixel position information) for each pixel, the second sensing information having higher accuracy than the first sensing information, based on a second integrated image obtained by integrating multiple images corresponding to multiple phase-shift patterns Pd1 to Pdm; and a processing unit (e.g., depth estimation unit 370) capable of selecting first sensing information and second sensing information of pixels with desired confidence levels from the first sensing information and second sensing information of each pixel based on the confidence level of each pixel, and capable of determining the distance to the measured target object A1 by using the selected first sensing information and the selected second sensing information (see reference). Figure 9 and 11 (etc.). Therefore, accurate distance information can be reliably obtained.

[0300] A light source device (e.g., projector 200) can project a moving object determination pattern (e.g., moving object determination patterns Pe1 and Pe2) as an illumination pattern onto the measurement target object A1. The data acquisition unit 330 can acquire sensing information for each pixel in an image obtained by capturing an image of the measurement target object A1 onto which the moving object determination pattern is projected. Furthermore, the confidence generation unit 360 can examine the sensing information for each pixel corresponding to the confidence determination pattern and the moving object determination pattern based on the confidence determination pattern (e.g., confidence determination patterns Pb1 and Pb2) and the moving object determination pattern, and generate a confidence level for each pixel based on the result of this examination (see reference). Figure 9 and Figure 11 (etc.). Therefore, the confidence level of each pixel can be accurately obtained.

[0301] The imaging device 100 can acquire an image as an image, including threshold-based sensing information for each pixel, and the ranging device 1 may further include: a confidence summing unit 380 configured to determine the number of pixels with a desired confidence level based on the confidence level of each pixel; and a control unit (e.g., a sensor control unit 390) configured to control the threshold based on the number determined by the confidence summing unit 380 (see reference). Figure 23 (etc.). Therefore, the number of pixels in the imaging device 100 that can detect events can be adjusted.

[0302] When the quantity determined by the confidence summation unit 380 is less than a predetermined value (see reference) Figure 23 (e.g., sensor control unit 390) can lower the threshold. Therefore, the number of pixels in the imaging device 100 capable of detecting events can be increased.

[0303] The ranging device 1 may further include: a confidence summing unit 380 configured to determine the number of pixels with a desired confidence level based on the confidence level of each pixel; and a light intensity control unit 395 configured to control the light intensity (reference) of a light source device (e.g., projector 200) based on the number determined by the confidence summing unit 380. Figure 24 (etc.). Therefore, the number of pixels in the imaging device 100 that can detect events can be adjusted.

[0304] When the quantity determined by the confidence summation unit 380 is less than a predetermined value, the light quantity control unit 395 may increase the light quantity (reference). Figure 24 (etc.). Therefore, the number of pixels in the imaging device 100 that can detect events can be increased.

[0305] The ranging device 1 may further include: a confidence summing unit 380 configured to determine the number of pixels with a desired confidence level based on the confidence level of each pixel; and a timing control unit 310 configured to base its determination on the confidence level of the pixels determined by the confidence summing unit 380 (see reference 310). Figure 25 The operating timing (e.g., imaging timing and illumination timing) of each of the imaging device 100 and the light source device (e.g., projector 200) is controlled by a determined number of parameters. Therefore, the operating timing of each of the imaging device 100 and the light source device can be adjusted.

[0306] When the quantity determined by the confidence summation unit 380 is less than a predetermined value (reference) Figure 25 When the timing control unit 310 delays the operation timing, it can increase the number of pixels capable of detecting events, even if there is a pixel response delay in the imaging device 100.

[0307] The ranging device 1 may further include: an illuminance sensor 500 configured to detect the illuminance for the target object A1; and a timing control unit 310 configured to, based on the illuminance detected by the illuminance sensor 500 (reference 310), determine the illuminance for the target object A1. Figure 26 The illuminance detected is used to control the operating timing (e.g., imaging timing and illumination timing) of each of the imaging device 100 and the light source device (e.g., projector 200). Therefore, the operating timing of each of the imaging device 100 and the light source device can be adjusted.

[0308] When the illuminance detected by the illuminance sensor 500 is lower than a predetermined value (reference) Figure 26 The timing control unit 310 can delay the operation timing. Therefore, even if there is a pixel response delay in the imaging device 100, the number of pixels capable of detecting events can be increased.

[0309] <8. Other Implementation Methods>

[0310] In addition to the embodiments described above, each configuration, process, etc., according to the above embodiments (including examples and variations) can be implemented in various forms. For example, the configuration and process are not limited to the examples described above, and various modes may also be adopted. Furthermore, for example, unless otherwise specified, the configurations, processes, specific names, or information containing various data and parameters described above and in the accompanying drawings may be modified as needed.

[0311] Furthermore, each configuration, process, etc., according to the above-described embodiments (including examples and variations) does not necessarily require the physical configuration shown. That is, the specific form of distribution or integration of each configuration, process, etc. is not limited to the examples shown, and all or part of them can be functionally or physically distributed or integrated in any unit according to various loads, usage conditions, etc.

[0312] Furthermore, each configuration, process, etc., according to the above embodiments (including examples and variations) can be appropriately combined. For example, at least a portion of one embodiment can be appropriately combined with at least a portion of another embodiment. Moreover, the effects described in the embodiments are merely illustrative and not limiting, and other effects may also be obtained.

[0313] For example, in each of the above embodiments, the imaging device 100 outputs a detection signal when the pixel signal is greater than a threshold, but this disclosure is not limited thereto. For example, the imaging device 100 may also output a detection signal when the pixel signal is less than a threshold. In this case, the imaging device 100 outputs a detection signal (e.g., "-1") indicating that the pixel signal has become less than the threshold (has become darker).

[0314] For example, when the illumination pattern P0 is a completely dark pattern (non-illumination pattern), during subsequent illumination periods of illumination patterns P1 to Pn, each pixel can output a detection signal indicating the same brightness as during the non-illumination period or a detection signal indicating that the pixel has become brighter. Therefore, as described above, when a detection signal indicating that the pixel has become darker is output, there is a possibility that the target object A1 or imaging device 100 has moved, or that the ambient light has changed, and the distance to the target object A1 cannot be accurately calculated. Therefore, the ranging device 1 can set the confidence level of pixels that output detection signals indicating that the pixel signal has become less than a threshold (darkened) to low. For example, the confidence level can be reduced based on the number of detection signals indicating "-1", and the confidence level can be set to a minimum value, for example, set to 0 when the number of detection signals indicating "-1" exceeds a predetermined threshold.

[0315] The technology disclosed herein can be applied to a variety of products. For example, the technology disclosed herein can be implemented as electronic devices installed on any type of mobile body, such as automobiles, electric vehicles, hybrid electric vehicles, motorcycles, bicycles, personal mobile devices, aircraft, drones, ships, robots, construction machines, or agricultural machines (tractors)). Furthermore, for example, the technology disclosed herein can be implemented as electronic devices installed on endoscopic surgical systems or microsurgical systems.

[0316] <9. Appendix>

[0317] It should be noted that this technology can be configured as follows.

[0318] (1) A ranging device, comprising: The light source device is configured to project a confidence determination pattern and a distance acquisition pattern as an illumination pattern onto the target object being measured. An imaging device is configured to capture an image of a measurement target object onto which a confidence determination pattern or a distance acquisition pattern is projected; The data acquisition unit is configured to acquire sensing information of each pixel in an image obtained by the imaging device; The confidence generation unit is configured to examine the sensing information of each pixel corresponding to the confidence determination pattern based on the confidence determination pattern, and generate a confidence level for each pixel based on the result of the examination; and The processing unit is configured to select, based on the confidence level of each pixel, the sensing information of the pixel with the required confidence level from the sensing information of each pixel corresponding to the distance acquisition pattern, and determine the distance to the target object to be measured by using the selected sensing information.

[0319] (2) According to the ranging device in (1), wherein, The confidence level determination pattern includes both fully dark and fully light patterns.

[0320] (3) According to the ranging device in (2), wherein, The light source device projects a completely dark pattern, and then projects a completely bright pattern.

[0321] (4) The ranging device according to (2) or (3), wherein, The confidence generation unit sets the confidence of pixels that indicate a dark response event has occurred when the confidence determination pattern is a fully dark pattern and that no event has occurred when the confidence determination pattern is a fully bright pattern as the confidence level of the pixel.

[0322] (5) A ranging device according to any one of (2) to (4), wherein, The confidence generation unit sets the confidence of pixels that do not have a dark response event when the confidence determination pattern is a fully dark pattern, or the confidence of pixels that have had a dark response event when the confidence determination pattern is a fully bright pattern, to the confidence level of pixels that are unreliable.

[0323] (6) A ranging device according to any one of (1) to (5), wherein, The processing unit sets pixels with a confidence level equal to or greater than a predetermined value as pixels with the required confidence level.

[0324] (7) A ranging device according to any one of (1) to (6), wherein, The light source device projects multiple phase-shift patterns as illumination patterns onto the target object being measured. The data acquisition unit acquires sensing information for each pixel in multiple images obtained by capturing images of a measurement target object on which multiple phase-shift patterns are sequentially projected. The confidence generation unit examines the sensing information of each pixel corresponding to the confidence determination pattern and multiple phase shift patterns based on the confidence determination pattern and multiple phase shift patterns, and generates the confidence of each pixel based on the result of the examination.

[0325] (8) The ranging device according to (7) further includes: an acquisition unit configured to acquire abnormal information indicating that a pixel has not properly acquired sensing information based on a phase-shift pattern, wherein, The confidence generation unit determines the pattern and anomaly information based on the confidence, examines the sensing information of each pixel corresponding to the confidence determination pattern and multiple phase shift patterns, and generates the confidence of each pixel based on the examination results.

[0326] (9) The ranging device according to (7) or (8) further includes: The first integration unit is configured to acquire first sensing information for each pixel based on a first integrated image obtained by integrating multiple images corresponding to multiple distance acquisition patterns; and The second integration unit is configured to acquire second sensing information for each pixel based on a second integrated image obtained by integrating multiple images corresponding to multiple phase-shift patterns, wherein the second sensing information has higher accuracy than the first sensing information. The processing unit selects the first and second sensing information of pixels with the required confidence level from the first and second sensing information of each pixel based on the confidence level of each pixel, and determines the distance to the target object by using the selected first and second sensing information.

[0327] (10) A ranging device according to any one of (1) to (9), wherein, The light source device projects the pattern defined by the moving object onto the target object as an illumination pattern. The data acquisition unit acquires sensing information for each pixel from an image obtained by capturing an image of a measurement target object onto which a moving object pattern is projected. The confidence generation unit examines the sensing information of each pixel corresponding to the confidence determination pattern and the moving object determination pattern based on the confidence determination pattern and the moving object determination pattern, and generates the confidence of each pixel based on the result of the examination.

[0328] (11) A ranging device according to any one of (1) to (10), wherein, The imaging device acquires an image as an image, including threshold-based sensing information for each pixel. The ranging device also includes: The confidence summation unit is configured to determine the number of pixels with the desired confidence level based on the confidence level of each pixel; and The control unit is configured to control the threshold based on the number determined by the confidence summation unit.

[0329] (12) According to the ranging device of (11), wherein, When the number determined by the confidence summation unit is less than a predetermined value, the control unit lowers the threshold.

[0330] (13) The ranging device according to any one of (1) to (12) further includes: The confidence summation unit is configured to determine the number of pixels with the desired confidence level based on the confidence level of each pixel; and The light quantity control unit is configured to control the light quantity of the light source device based on the quantity determined by the confidence summation unit.

[0331] (14) According to the ranging device of (13), wherein, The light quantity control unit increases the light quantity when the quantity determined by the confidence summation unit is less than a predetermined value.

[0332] (15) The ranging device according to any one of (1) to (14) further includes: The confidence summation unit is configured to determine the number of pixels with the desired confidence level based on the confidence level of each pixel; and The timing control unit is configured to control the operating timing of each of the imaging device and the light source device based on a number determined by the confidence summation unit.

[0333] (16) According to the ranging device of (15), wherein, The timing control unit delays the operation timing when the number determined by the confidence summation unit is less than a predetermined value.

[0334] (17) The ranging device according to any one of (1) to (16) further includes: An illuminance sensor is configured to detect the illuminance relative to a target object; and The timing control unit is configured to control the operating timing of each of the imaging device and the light source device based on the illuminance detected by the illuminance sensor.

[0335] (18) According to the ranging device of (17), wherein, The timing control unit delays operation timing when the illuminance detected by the illuminance sensor is lower than a predetermined value.

[0336] (19) A distance measurement method, executed by a computer, comprising: The confidence-determined pattern and the distance-acquired pattern are projected onto the target object as illumination patterns. Capture images of the target object to be measured, onto which confidence-determining patterns or distance-acquisition patterns have been projected; Acquire sensing information for each pixel in an image obtained through imaging; Based on the confidence level pattern, the sensing information of each pixel corresponding to the confidence level pattern is examined, and a confidence level for each pixel is generated based on the result of this examination; and Based on the confidence level of each pixel, the sensing information of pixels with the required confidence level is selected from the sensing information of each pixel corresponding to the distance acquisition pattern, and the distance to the target object is determined by using the selected sensing information.

[0337] (20) A computer-readable storage medium storing a program for causing a computer to perform the following operations: The confidence-determined pattern and the distance-acquired pattern are projected onto the target object as illumination patterns. Capture images of the target object to be measured, onto which confidence-determining patterns or distance-acquisition patterns have been projected; Acquire sensing information for each pixel in an image obtained through imaging; The confidence level determines the pattern by examining the sensing information of each pixel corresponding to the confidence level determination pattern, and generates the confidence level of each pixel based on the result of this examination; and Based on the confidence level of each pixel, the sensing information of the pixel with the required confidence level is selected from the sensing information of each pixel corresponding to the distance pattern, and the distance to the target object is determined by using the selected sensing information.

[0338] (21) A ranging method for using a computer as a ranging device according to any one of (1) to (18).

[0339] (22) A computer-readable storage medium storing a program for enabling a computer to implement a ranging device according to any one of (1) to (18).

[0340] Reference number list

[0341] 1. Distance measuring device

[0342] 100 Imaging Device

[0343] 110 pixel array unit

[0344] 111 Control Unit

[0345] 112 Signal Processing Unit

[0346] 113 Drive Circuit

[0347] 114 Output Circuit

[0348] 115 Timing Control Unit

[0349] 200 Projector

[0350] 300 Overall Control Unit

[0351] 310 Timing Control Unit

[0352] 320 Projection Image Generation Unit

[0353] 330 Data Acquisition Unit

[0354] 340 Code Integration Unit

[0355] 350 phase-shifting integration units

[0356] 360 Confidence Generation Unit

[0357] 370 depth estimation units

[0358] 380 confidence level summation unit

[0359] 390 Sensor Control Unit

[0360] 395 Light Quantity Control Unit

[0361] 400 storage units

[0362] 500 Illuminance Sensor

[0363] 1101 pixels

[0364] Pa1 Pixel Reset Pattern

[0365] Pa2 Pixel Reset Pattern

[0366] Pb1 confidence level determines the pattern

[0367] Pb2 confidence level determination pattern

[0368] Pc1 binary pattern

[0369] Pcn binary pattern

[0370] Pd1 phase shift pattern

[0371] Pdm phase shift pattern

[0372] Pe1 Moving Object Determines Pattern

[0373] Pe2 Move the object to define the pattern.

Claims

1. A ranging device, comprising: The light source device is configured to project a confidence determination pattern and a distance acquisition pattern as an illumination pattern onto the target object being measured. An imaging device is configured to capture an image of the measurement target object onto which the confidence determination pattern or the distance acquisition pattern is projected; The data acquisition unit is configured to acquire sensing information of each pixel in the image obtained by the imaging device; The confidence generation unit is configured to examine the sensing information of each pixel corresponding to the confidence determination pattern based on the confidence determination pattern, and generate a confidence level for each pixel based on the result of the examination. as well as The processing unit is configured to select, based on the confidence level of each pixel, the sensing information of a pixel having the required confidence level from the sensing information of each pixel corresponding to the distance acquisition pattern, and determine the distance to the measurement target object by using the selected sensing information.

2. The ranging device according to claim 1, wherein, The confidence level determination pattern includes both fully dark and fully light patterns.

3. The ranging device according to claim 2, wherein, The light source device projects the completely dark pattern, and then projects the completely bright pattern.

4. The ranging device according to claim 2, wherein, The confidence generation unit sets the confidence level of a pixel as reliable when the sensing information indicates a dark response event when the confidence determination pattern is the fully dark pattern and when the sensing information indicates no event when the confidence determination pattern is the fully bright pattern.

5. The ranging device according to claim 2, wherein, The confidence generation unit sets the confidence level of the pixel indicating that no dark response event has occurred when the confidence determination pattern is the all-dark pattern, or the confidence level of the pixel indicating that a dark response event has occurred when the confidence determination pattern is the all-bright pattern, as the confidence level of the pixel indicating that no dark response event has occurred.

6. The ranging device according to claim 1, wherein, The processing unit sets the pixel whose confidence level is equal to or greater than a predetermined value as the pixel with the required confidence level.

7. The ranging device according to claim 1, wherein, The light source device projects multiple phase-shift patterns as the illumination pattern onto the measurement target object. The data acquisition unit acquires, for each image, the sensing information of each pixel in a plurality of images obtained by capturing images of the measurement target object on which the plurality of phase-shift patterns are sequentially projected, and The confidence generation unit examines the sensing information of each pixel corresponding to the confidence determination pattern and the plurality of phase shift patterns based on the confidence determination pattern and the plurality of phase shift patterns, and generates the confidence of each pixel based on the examination results.

8. The ranging device according to claim 7, further comprising: The acquisition unit is configured to acquire abnormal information indicating that the sensing information based on the phase shift pattern has not been acquired normally, wherein, The confidence generation unit examines the sensing information of each pixel corresponding to the confidence determination pattern and the anomaly information based on the confidence determination pattern and the plurality of phase shift patterns, and generates a confidence level for each pixel based on the examination results.

9. The ranging device according to claim 7, further comprising: The first integration unit is configured to acquire first sensing information for each pixel based on a first integrated image obtained by integrating multiple images that correspond to multiple distance acquisition patterns respectively; as well as The second integration unit is configured to acquire second sensing information for each pixel based on a second integrated image obtained by integrating multiple images corresponding to the multiple phase-shift patterns, wherein the second sensing information has higher accuracy than the first sensing information. The processing unit selects the first and second sensing information of the pixel with the required confidence level from the first and second sensing information of each pixel based on the confidence level of each pixel, and determines the distance to the measurement target object by using the selected first and second sensing information.

10. The ranging device according to claim 1, wherein, The light source device projects the moving object determination pattern as the illumination pattern onto the measurement target object. The data acquisition unit acquires sensing information for each pixel of an image obtained by capturing an image of the measurement target object onto which the moving object determination pattern is projected, and The confidence generation unit examines the sensing information of each pixel corresponding to the confidence determination pattern and the moving object determination pattern based on the confidence determination pattern and the moving object determination pattern, and generates a confidence level for each pixel based on the examination results.

11. The ranging device according to claim 1, wherein, The imaging device acquires an image, including threshold-based sensing information for each pixel, as the image. The ranging device further includes: A confidence summation unit is configured to determine the number of pixels having the desired confidence level based on the confidence level of each pixel; and The control unit is configured to control the threshold based on the quantity determined by the confidence summation unit.

12. The ranging device according to claim 11, wherein, When the quantity determined by the confidence summation unit is less than a predetermined value, the control unit lowers the threshold.

13. The ranging device according to claim 1, further comprising: The confidence summation unit is configured to determine the number of pixels having the required confidence level based on the confidence level of each pixel; as well as The light quantity control unit is configured to control the light quantity of the light source device based on the quantity determined by the confidence summation unit.

14. The ranging device according to claim 13, wherein, The light quantity control unit increases the light quantity when the quantity determined by the confidence summation unit is less than a predetermined value.

15. The ranging device according to claim 1, further comprising: The confidence summation unit is configured to determine the number of pixels having the required confidence level based on the confidence level of each pixel; as well as The timing control unit is configured to control the operating timing of each of the imaging device and the light source device based on the quantity determined by the confidence summation unit.

16. The ranging device according to claim 15, wherein, The timing control unit delays the operation timing when the quantity determined by the confidence summation unit is less than a predetermined value.

17. The ranging device according to claim 1, further comprising: An illuminance sensor is configured to detect the illuminance for the target object being measured; as well as The timing control unit is configured to control the operating timing of each of the imaging device and the light source device based on the illuminance detected by the illuminance sensor.

18. The ranging device according to claim 17, wherein, The timing control unit delays the operation timing when the illuminance detected by the illuminance sensor is lower than a predetermined value.

19. A ranging method, executed by a computer, the ranging method comprising: The confidence-determined pattern and the distance-acquired pattern are projected onto the target object as illumination patterns. Capture an image of the measurement target object onto which the confidence determination pattern or the distance acquisition pattern is projected; Acquire sensing information for each pixel of the image obtained through imaging; Based on the confidence level, a pattern is determined, and the sensing information of each pixel corresponding to the confidence level is examined, and a confidence level for each pixel is generated based on the results of the examination. as well as Based on the confidence level of each pixel, the sensing information of pixels with the required confidence level is selected from the sensing information of each pixel corresponding to the distance acquisition pattern, and the distance to the measurement target object is determined by using the selected sensing information.

20. A computer-readable storage medium storing a program for causing a computer to perform the following operations: The confidence-determined pattern and the distance-acquired pattern are projected onto the target object as illumination patterns. Capture an image of the measurement target object onto which the confidence determination pattern or the distance acquisition pattern is projected; Acquire sensing information for each pixel of the image obtained through imaging; Based on the confidence level, a pattern is determined, and the sensing information of each pixel corresponding to the confidence level is examined, and a confidence level for each pixel is generated based on the results of the examination. as well as Based on the confidence level of each pixel, the sensing information of pixels with the required confidence level is selected from the sensing information of each pixel corresponding to the distance pattern, and the distance to the target object is determined by using the selected sensing information.

Citation Information

Patent Citations

  • Distance measuring device and distance measuring method

    JP2021124321A