Distance measuring device and its control method, and distance measuring system

Abstract

This invention relates to a distance measurement device, a method for controlling the same, and a distance measurement system that make it possible to arrange pixel array sample points such that more distance information can be obtained. This distance measurement device comprises a pixel array in which pixels for receiving reflected light resulting from the reflection of emitted light by an object are arranged in a matrix, a designation unit for designating some of the pixels of the pixel array as sample points for detecting distance information, and a storage unit for storing a sample point state table for storing sample point distance information and a sample point movement rule table for storing sample point movement rules. The designation unit updates sample point position information on the basis of the sample point state table and the sample point movement rule table. This invention can be applied to, for example, a distance measurement system for detecting depth-direction distances to a subject.

Description

[[Technical Field]]

[0001] The present technology relates to a distance measuring device, a control method therefor, and a distance measuring system, and particularly relates to a distance measuring device, a control method therefor, and a distance measuring system that enable arrangement of sample points in a pixel array so that more distance information can be obtained. [[Background Art]]

[0002] In recent years, distance measuring devices that perform distance measurement by the ToF (Time-of-Flight) method (hereinafter also referred to as depth cameras) have attracted attention. Among the ToF methods, there is a distance measuring device that adopts the direct ToF method, in which a light receiving element called SPAD (Single Photon Avalanche Diode) is arranged in each light receiving pixel, the flight time from the timing when irradiation light is emitted to the timing when reflected light is received is directly measured, and the distance to an object is calculated. In distance measurement by the direct ToF method, in order to suppress noise caused by ambient light, etc., the emission of irradiation light and the reception of its reflected light are repeated a predetermined number of times (for example, several to hundreds of times), a histogram of the flight time of the irradiation light is generated, and the distance to the object is calculated from the flight time corresponding to the peak of the histogram.

[0003] The circuit scale of a time measurement unit that measures flight time, a histogram generation unit that generates a histogram, and a peak detection unit that detects the peak of a histogram is relatively large, so it is generally difficult to provide these for all pixels. Therefore, the number of histograms that can be generated is smaller than the total number of pixels in the pixel array.

[0004] Therefore, it has been practiced to perform light receiving operation using only some pixels of the pixel array as sample points, or to regard a plurality of adjacent pixels as one large pixel (referred to as multi-pixel) and generate a histogram using the same as a sample point. In this case, the number of sample points for generating a histogram can be smaller than the total number of pixels in the pixel array.

[0005] When generating distance information using fewer sample points than the total number of pixels in a pixel array, how the sample points are arranged within the pixel array is crucial for obtaining more distance information.

[0006] For example, Patent Document 1 discloses a method in which the density of sample points increases as the distance to the object decreases, and decreases as the ambient light noise increases.

[0007] Furthermore, Patent Document 2 discloses a technique for irradiating a specific area of ​​the pixel array of a distance measuring device with illumination light. [Prior art documents] [Patent Documents]

[0008] [Patent Document 1] Japanese Patent Publication No. 2020-112443 [Patent Document 2] Japanese Patent Publication No. 2020-076619 [Overview of the project] [Problems that the invention aims to solve]

[0009] The technology disclosed in Patent Document 1 changes the sample points row by row of the pixel array, and since it selects from several pre-prepared sampling patterns, there are constraints on the arrangement of the sample points, and there is room for improvement.

[0010] This technology was developed in light of these circumstances, and aims to allow the arrangement of sample points in a pixel array to obtain more distance information. [Means for solving the problem]

[0011] The distance measuring device of the first aspect of this technology comprises a pixel array in which pixels that receive reflected light when an irradiated light is reflected by an object are arranged in a matrix, a determination unit that determines some of the pixels in the pixel array as sample points for detecting distance information, a sample point status table that stores the distance information of the sample points, and a storage unit that stores a sample point movement rule table that stores the movement rules of the sample points, wherein the determination unit updates the position information of the sample points based on the sample point status table and the sample point movement rule table.

[0012] A second aspect of this technology is a control method for a distance measuring device, in which a distance measuring device having a pixel array in which pixels that receive reflected light reflected by an object are arranged in a matrix determines some of the pixels in the pixel array as sample points for detecting distance information, stores the distance information of the sample points in a sample point state table, and updates the position information of the sample points based on the sample point state table and a sample point movement rule table in which the movement rules of the sample points are stored.

[0013] A third aspect of this technology is a distance measuring system comprising an illumination device that emits illumination light and a distance measuring device that receives reflected light when the illumination light is reflected by an object, wherein the distance measuring device comprises a pixel array in which pixels that receive the reflected light are arranged in a matrix, a determination unit that determines some of the pixels in the pixel array as sample points for detecting distance information, a sample point state table that stores the distance information of the sample points and a storage unit that stores the sample point movement rule table that stores the movement rules of the sample points, wherein the determination unit updates the position information of the sample points based on the sample point state table and the sample point movement rule table.

[0014] In the first to third aspects of this technology, some pixels in a pixel array, in which pixels that receive reflected light from an object are arranged in a matrix, are determined to be sample points for detecting distance information. The distance information of the sample points is stored in a sample point state table, and the position information of the sample points is updated based on the sample point state table and a sample point movement rule table in which the movement rules of the sample points are stored.

[0015] The distance measuring device and distance measuring system may be a standalone device or a module incorporated into another device. [Brief explanation of the drawing]

[0016] [Figure 1] This is a block diagram showing an example configuration of one embodiment of the distance measuring system disclosed herein. [Figure 2] This is a block diagram showing a detailed configuration example of a distance measuring system. [Figure 3] This is a diagram illustrating the sample point status table. [Figure 4] This is a diagram illustrating the sample point movement rule table. [Figure 5] This diagram illustrates the first example of the process for updating the location information of sample points. [Figure 6] This diagram illustrates the first example of the process for updating the location information of sample points. [Figure 7] This diagram illustrates the first example of the process for updating the location information of sample points. [Figure 8] This diagram illustrates a second example of the process for updating the location information of sample points. [Figure 9] This diagram illustrates a second example of the process for updating the location information of sample points. [Figure 10] This is a flowchart explaining the distance image generation process using a distance measuring system. [Figure 11] This diagram illustrates other examples of movement rule tables. [Figure 12] This diagram illustrates an example of calculating the confidence level of a distance. [Figure 13] This diagram illustrates an example of calculating the confidence level of a distance. [Figure 14] This block diagram shows an example configuration of another embodiment of the ranging system disclosed herein. [Figure 15] This is a block diagram showing the detailed configuration of the ranging device in brightness observation mode. [Modes for carrying out the invention]

[0017] Hereinafter, embodiments for carrying out this technology (hereinafter referred to as "embodiments") will be described with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration will be denoted by the same reference numerals to avoid redundant explanations. The explanation will proceed in the following order. 1. Example of a ranging system configuration 2. Detailed Configuration Example of a Rangefinder 3. First example of sample point update process 4. Second example of sample point update process 5. Flowchart for distance image generation process 6. Other examples of movement rules 7. Example of calculating the reliability of distance 8. Other configuration examples of the ranging system 9. Example of a configuration for brightness observation mode 10. Summary

[0018] <1. Example of a ranging system configuration> Figure 1 is a block diagram showing an example configuration of one embodiment of the distance measuring system of this disclosure.

[0019] The ranging system 1 shown in Figure 1 is a system that measures and outputs the distance to an object using, for example, the Time-of-Flight (ToF) method. Here, ranging system 1 performs ranging using the direct ToF method. The direct ToF method is a method that calculates the distance to an object by directly measuring the flight time from the time when the emitted light is released to the time when the reflected light is received.

[0020] This distance measuring system 1 can be used in conjunction with an RGB camera (not shown) that photographs subjects including objects 13, etc. When the distance measuring system 1 is used with an RGB camera as an external device, the distance measuring system 1 sets the distance measurement range to the same range as the imaging range of the RGB camera and generates a distance image as distance information for subjects captured by the RGB camera.

[0021] The distance measuring system 1 comprises an illumination device 11 and a distance measuring device 12, and measures the distance to a predetermined object 13 as the subject. More specifically, when the distance measuring system 1 receives a distance measuring instruction from a higher-level host device, it repeats the emission of illumination light and the reception of its reflected light a predetermined number of times (for example, several times to several hundred times). Based on the emission of illumination light and the reception of its reflected light, which are repeated a predetermined number of times, the distance measuring system 1 generates a histogram of the flight time of the illumination light, and calculates the distance to the object 13 from the flight time corresponding to the peak in the histogram.

[0022] The illumination device 11 irradiates a predetermined object 13 with illumination light based on the light emission control signal and light emission trigger supplied from the distance measuring device 12. For example, infrared light (IR light) with a wavelength in the range of approximately 850 nm to 940 nm is used as the illumination light. The illumination device 11 comprises at least a light emission unit 31 and a light emission drive unit 32. The illumination device 11 may also include a projection lens and a diffractive optical element (neither of which are shown).

[0023] The light-emitting unit 31 is composed of, for example, a VCSEL array in which multiple VCSELs (Vertical Cavity Surface Emitting Lasers) are arranged in a planar manner, and each VCSEL is turned on and off to emit light according to the control of the light-emitting drive unit 32. The size of the VCSEL light-emitting unit (light source) and the position of the VCSEL to emit light (light-emitting position) can be varied by the control of the light-emitting drive unit 32.

[0024] The light emission drive unit 32 includes, for example, a microprocessor, LSI, laser drive driver, etc., and controls the light emission unit (size of the light source) of the VCSEL and the position of the VCSEL to emit light (light emission position) based on the light emission control signal supplied from the control unit 51 of the distance measuring device 12. The light emission drive unit 32 also controls the light emission timing of the VCSEL to emit light according to the light emission trigger supplied from the control unit 51 of the distance measuring device 12. The light emission trigger is, for example, a pulse waveform consisting of two values, "High (1)" and "Low (0)", where "High" represents the timing at which the irradiation light is emitted.

[0025] When a distance measurement instruction is supplied, the distance measuring device 12 determines the light emission conditions, such as the size and position of the light source. Based on the determined light emission conditions, the distance measuring device 12 generates a light emission control signal and a light emission trigger and outputs them to the illumination device 11, causing it to emit light. The distance measuring device 12 also calculates the distance to the object 13 by receiving the reflected light from the object 13 and outputs the result as a distance image to the host device. The distance measuring device 12 comprises a control unit 51, a pixel driving unit 52, a light receiving unit 53, a signal processing unit 54, and an input / output unit 55.

[0026] The control unit 51 of the distance measuring device 12 is composed of, for example, an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), a microprocessor, etc. When the control unit 51 receives a distance measuring instruction from a higher-level host device via the input / output unit 55, it determines the light emission conditions and supplies a light emission control signal and a light emission trigger corresponding to the determined light emission conditions to the light emission drive unit 32 of the illumination device 11. The light emission trigger is also supplied to the signal processing unit 54 as a timing notification for the start of the flight time count.

[0027] Furthermore, the control unit 51 determines which pixels of the light-receiving unit 53 will be designated as active pixels in accordance with the determined light emission conditions, and supplies sample point control information to the pixel drive unit 52 to identify the active pixels. An active pixel is a pixel that detects the incidence of photons. Pixels that do not detect the incidence of photons are called inactive pixels.

[0028] The light-receiving unit 53 has a pixel array in which pixels are arranged in a matrix in a two-dimensional manner. Each pixel of the light-receiving unit 53 is equipped with a SPAD (Single Photon Avalanche Diode) as a photoelectric conversion element. The SPAD instantaneously detects a single photon by multiplying the carriers generated by photoelectric conversion in a high-electric-field PN junction region (multiplication region). When each active pixel of the light-receiving unit 53 detects the incidence of a photon, it outputs a detection signal to the signal processing unit 54 indicating that a photon has been detected.

[0029] The signal processing unit 54 generates a histogram of the time (count value) from when the illumination light is emitted until the reflected light is received, based on the emission of illumination light and the reception of the reflected light, which are repeated a predetermined number of times (for example, several times to several hundred times). The unit for generating the histogram (histogram generation unit) may be one pixel, or it may be a multi-pixel unit in which multiple adjacent pixels are considered as one large pixel (referred to as a multi-pixel). The signal processing unit 54 then detects the peak of the generated histogram to determine the time it takes for the light emitted from the illumination device 11 to be reflected back from the object 13, and calculates the distance to the object 13 based on the determined time and the speed of light, and generates a distance image. The generated distance image is output to a higher-level host device via the input / output unit 55. The signal processing unit 54 is composed of, for example, an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), and logic circuits.

[0030] The input / output unit 55 supplies distance measurement instructions from the higher-level host device to the control unit 51. The input / output unit 55 also outputs distance images supplied from the signal processing unit 54 to the higher-level host device. The input / output unit 55 can be configured, for example, with a communication interface compliant with MIPI (Mobile Industry Processor Interface).

[0031] The distance measuring device 12, configured as described above, has two operating modes: a distance measuring mode and a brightness observation mode. In the distance measuring mode, some of the pixels of the light-receiving unit 53 are set as active pixels and the remaining pixels are set as inactive pixels, and a distance image is generated and output based on the distance detected by the active pixels. On the other hand, in the brightness observation mode, all pixels of the light-receiving unit 53 are set as active pixels, and a brightness image is generated by counting the number of photons input during a certain period as a brightness value (pixel value).

[0032] In distance measurement mode, the distance measuring device 12 sets multiple sample points in the pixel array of the light receiving unit 53, generates a histogram for each sample point, and determines the distance to the object 13 by detecting the peaks in the histogram, thereby generating a distance image. A sample point may consist of a single pixel or multiple pixels. However, the number of sample points set in the pixel array is less than the total number of pixels in the pixel array. In this case, where only a portion of the pixels in the pixel array are used instead of all of them, how the sample points are arranged within the pixel array is important in order to obtain more distance information than that for the object 13. The distance measuring device 12 controls the arrangement of a number of sample points less than the total number of pixels in the pixel array to an optimal degree, and generates a distance image.

[0033] <2. Detailed Configuration Example of Rangefinder> Figure 2 is a block diagram of the distance measuring system 1, including a more detailed configuration example of the distance measuring device 12 when the operating mode is distance measuring mode.

[0034] The distance measuring device 12 comprises a control unit 51, a pixel driving unit 52, a light receiving unit 53, a signal processing unit 54, and an input / output unit 55. Note that in Figure 2, the control signal from the input / output unit 55 to the control unit 51 when a distance measuring instruction is input to the input / output unit 55 is omitted.

[0035] The control unit 51 includes a determination unit 61, a judgment unit 62, and a storage unit 63. The storage unit 63 stores a sample point status table 71 and a sample point movement rule table 72.

[0036] The signal processing unit 54 includes a multiplexer 80 and time measurement units 811 to 81 Q , histogram generation unit 821 to 82 Q , peak detection units 831 to 83 QThe signal processing unit 54 also has a distance calculation unit 84. Specifically, the signal processing unit 54 is provided with Q (Q>1) each of a time measurement unit 81, a histogram generation unit 82, and a peak detection unit 83, and is configured to generate Q histograms. The value of Q corresponds to the maximum number of sample points that can be set and is less than the total number of pixels in the pixel array of the light receiving unit 53. However, the value of Q may be the same as the total number of pixels in the light receiving unit 53, in which case, in order to reduce power consumption and speed up processing, the optimal placement control of sample points described below can be performed when the number of histograms to be generated is set to be less than the total number of pixels in the pixel array.

[0037] When the determination unit 61 receives a distance measurement instruction from a higher-level host device via the input / output unit 55, it determines the emission conditions for the light-emitting unit 31 of the illumination device 11. Specifically, the determination unit 61 determines the light-emitting unit (size of the light source) of the VCSEL and the position of the VCSEL to be emitted, and supplies a light-emitting control signal to the light-emitting drive unit 32 of the illumination device 11 indicating which VCSEL in the VCSEL array should be emitted. In this embodiment, for the sake of simplicity, all VCSELs in the VCSEL array are made to emit light with uniform brightness, but as will be described later, it is also possible to limit the emission position to a part of the VCSEL array depending on the arrangement of the sample positions, etc. After transmitting the light-emitting control signal, or by supplying a light-emitting trigger to the light-emitting drive unit 32 along with the light-emitting control signal, the determination unit 61 starts the emission of light by the light-emitting unit 31.

[0038] Furthermore, the determination unit 61 determines the initial position of the sample points for the pixel array of the light-receiving unit 53, generates a sample point status table 71 corresponding to the initial position, and stores it in the storage unit 63. Based on the sample point status table 71, the determination unit 61 supplies sample point control information that identifies the active pixels to the pixel driving unit 52 and the multiplexer 80. The sample point control information includes information indicating the active pixels of the pixel array of the light-receiving unit 53 and information indicating the constituent units of the multipixel.

[0039] When the initial distance image is generated using the sample points at their initial positions and the sample point status table 71 in the storage unit 63 is updated, the determination unit 62 notifies the decision unit 61 of the update to the sample point status table 71. Upon receiving the notification of the sample point status table 71 update from the determination unit 62, the decision unit 61 updates the sample point position information based on the sample point status table 71 and the sample point movement rule table 72. Specifically, the decision unit 61 updates the sample point position information based on the current distance information of each sample point recorded in the sample point status table 71 and the movement rules recorded in the sample point movement rule table 72. Along with the update of the sample point position information, the sample point status table 71 is also updated. Based on the updated sample point status table 71, the decision unit 61 generates sample point control information and supplies it to the pixel drive unit 52 and the multiplexer 80. By repeating the generation of the distance image and the updating of the sample point status table 71 based on the generated distance image a predetermined number of times, the sample points of the pixel array are updated to an optimal arrangement. Furthermore, in addition to the individual sample point movement rules for each sample point, Table 72 may also include overall rules that apply to all sample points.

[0040] The determination unit 62 determines whether the sample point status table 71 stored in the memory unit 63 has been updated. If the determination unit 62 determines that the sample point status table 71 has been updated, it notifies the decision unit 61 of the update of the sample point status table 71.

[0041] Referring to Figures 3 and 4, the sample point status table 71 and the sample point movement rule table 72 stored in the memory unit 63 will be described.

[0042] Figure 3 shows an example of the sample point status table 71.

[0043] The sample point status table 71 stores information about each of the multiple sample points set for the pixel array of the light-receiving unit 53. In the sample point status table 71 in Figure 3, n (n>0) sample points are set for the pixel array.

[0044] In the sample point status table 71, each of the n sample points set for the pixel array is assigned a sample point ID (sample point identification information) to identify the sample point. The sample point status table 71 also stores location information, distance information, confidence information, brightness information, and a rule ID for each sample point.

[0045] The position information indicates the location of the multi-pixels that make up the sample point. specific The position information includes the pixel position (X coordinate, Y coordinate) as the representative position of the multipixel, the multipixel width which is the number of pixels in the X direction of the multipixel, and the multipixel height which is the number of pixels in the Y direction of the multipixel. If the sample point consists of one pixel, the multipixel width and multipixel height will be "1".

[0046] Distance information refers to information about the distance calculated at the sample point in question. The distance information is configured to store distance information from at least the past two frames (two measurements) so that changes in distance can be detected. Specifically, the distance calculated at the most recent time t (t) and the distance calculated at the previous time (t-1) (t-1) can be recorded. It may also be possible to store distance information from three or more past frames.

[0047] Confidence information indicates the confidence level of the distance calculated at the given sample point. Like distance information, confidence information can be recorded for the same number of frames. Specific examples of confidence calculation will be discussed later, but for example, it can be calculated based on the difference between the count value (histogram height) of the bin where a peak was detected in the histogram and the count values ​​of the bins without peaks.

[0048] Brightness information is information indicating the brightness calculated at the sample point. Brightness information can also be recorded for the same number of frames as distance information. For example, the brightness information can be the count value (histogram height) of the bin in which a peak was detected in the histogram. Alternatively, brightness values ​​measured by changing the operating mode to brightness observation mode, or brightness values ​​from images captured by an external device such as an RGB camera may be used.

[0049] The Rule ID is rule identification information that indicates the rule applicable to the sample point in question. The specific rule corresponding to the Rule ID is described in Sample Point Movement Rule Table 72.

[0050] Figure 4 shows an example of the sample point movement rule table 72.

[0051] In Sample Point Movement Rule Table 72, it is stated that the following items can be defined for each rule ID: "Condition," "Action," and "Constraint."

[0052] "Condition" represents the conditions under which the sample point will perform the movement action defined in "Action". If the sample point does not meet the conditions, the movement action defined in the Action section will not be performed. "Unconditional" means that the movement action defined in "Action" will be performed without any conditions.

[0053] "Action" refers to the movement action performed on a sample point when the sample point meets the conditions described in "Conditions".

[0054] A "constraint" represents the conditions that must be met when performing the movement defined in "action." In other words, the action defined in "action" is performed while satisfying the conditions described in "constraint." Constraints can be omitted.

[0055] In the sample point movement rule table 72 in Figure 4, for example, rule ID=1 defines that the process of not moving the sample point is applied unconditionally. Rule ID=2 defines that the process of randomly moving the sample point is applied unconditionally.

[0056] Rule ID=3 defines that the process of moving the sample point in question to a position where it is closer to more of the eight surrounding pixels that have been detected as being closer than itself is unconditionally applied.

[0057] Rule ID=4 defines that, for the given sample point, provided that its own distance has not changed, the process of moving it to a position where it is more adjacent to any sample point where a distance change has been detected among the eight surrounding pixels is applied unconditionally.

[0058] Rule ID=5 defines that, for the given sample point, the distance has not changed over the past W frames (W>0), and the surrounding V pixels (V>0) are randomly moved.

[0059] In this way, by appropriately setting the desired movement rules in the sample point movement rule table 72, various sample point update algorithms can be implemented according to the purpose. The contents of the sample point movement rule table 72 can be changed, for example, from a higher-level host device.

[0060] Returning to the explanation of Figure 2, the determination unit 61 performs the action of the rule ID specified in the sample point status table 71 for each sample point, referring to the sample point movement rule table 72, and updates the position information of the sample points. Along with the update of the position information of the sample points, the sample point status table 71 is also updated.

[0061] The sample point movement rule table 72 shown in Figure 4 defines individual sample point movement rules for each sample point. However, separately from this, the decision unit 61 can define and execute an overall rule that applies to all sample points in common. When the decision unit 61 executes an overall rule, it first applies the individual rules for each sample point based on the sample point movement rule table 72. The overall rule may be pre-built as a common rule, or multiple overall rules may be stored in the storage unit 63, as in the sample point movement rule table 72, and the overall rule to be applied may be switched as appropriate.

[0062] For example, the following could be defined as an overall rule: (A) After applying the individual rules based on the sample point movement rule table 72, if there is a region where distance information has not been measured for a predetermined period (a predetermined number of frames), a sample point is placed speculatively in that region. (B) After applying the individual rules based on the sample point movement rule table 72, if there are any sample points whose distance information does not change for a predetermined period (a predetermined number of frames), those sample points are moved to their initial positions. In this case, it is necessary to store the initial position for each sample point.

[0063] The pixel drive unit 52 controls active and inactive pixels based on sample point control information supplied from the determination unit 61. In other words, the pixel drive unit 52 controls the on / off operation of each pixel in the light receiving unit 53. When an incident photon is detected in each pixel set as an active pixel in the light receiving unit 53, a detection signal indicating that a photon has been detected is output as a pixel signal to the multiplexer 80 of the signal processing unit 54.

[0064] The multiplexer 80 receives the pixel signals from the active pixels of the light-receiving unit 53 and, based on the sample point control information supplied from the determination unit 61, processes them using the time measurement units 811 to 811. Nallocates the signals to any of. More specifically, the multiplexer 80 appropriately selects pixel signals of one or more active pixels constituting the sample points of the light receiving unit 53, and outputs the selected pixel signals to the same time measurement unit 81 i performs control, for each sample point set in the light receiving unit 53, to supply the signal to (any one of i=1 to Q).

[0065] Although illustration is omitted in FIG. 2, the time measurement units 81 of the signal processing unit 54 i to 81 Q are also supplied with a light emission trigger output by the control unit 51 to the light emission driving unit 32 of the illumination device 11. The time measurement unit 81 i generates, based on the light emission timing indicated by the light emission trigger and the pixel signal supplied from each active pixel of the sample point, a count value corresponding to the time from when the light emitting unit 31 emits irradiation light to when the active pixel receives reflected light. The generated count value is supplied to the corresponding histogram generation unit 82 i The generated count value is supplied to the corresponding histogram generation unit 82. The time measurement unit 81 i is also called a TDC (Time to Digital Converter).

[0066] The histogram generation unit 82 i creates a histogram of count values based on the count values supplied from the time measurement unit 81 i Data of the generated histogram is supplied to the corresponding peak detection unit 83 i Data of the generated histogram is supplied to the corresponding peak detection unit 83.

[0067] The peak detection unit 83 i detects a peak of the histogram based on histogram data supplied from the histogram generation unit 82 i detects a peak of the histogram based on histogram data supplied from the histogram generation unit 82. The peak detection unit 83 i supplies the count value corresponding to the detected peak of the histogram to the distance calculation unit 84.

[0068] The distance calculation unit 84 obtains the count value from peak detection units 831 to 83 NFrom each of these, the flight time of the irradiated light is calculated based on the count value corresponding to the peak of the histogram, which is supplied on a sample point basis. Furthermore, the distance calculation unit 84 calculates the distance to the subject from the calculated flight time and generates a distance image in which the calculated distance is stored as a pixel value. The generated distance image is output to a higher-level host device via the input / output unit 55 and is also supplied to the control unit 51, where the distance information in the sample point status table 71 is updated.

[0069] When the operating mode is distance measurement mode, the distance measuring device 12 is configured as described above.

[0070] <3. First example of sample point update process> Next, we will explain the process by which the determination unit 61 updates the location information of the sample points based on the sample point status table 71 and the sample point movement rule table 72.

[0071] First, with reference to Figures 5 through 7, we will describe a first example of the sample point location information update process.

[0072] The first example shows how to update the position information of all sample points in a pixel array by applying rule ID=3 from the sample point movement rule table 72 shown in Figure 4.

[0073] Figure 5A shows the initial position of the sample point determined by the determination unit 61 for the pixel array of the light receiving unit 53.

[0074] In Figure 5A, for simplicity, assuming that each sample point is set to one pixel, white circles (○) represent pixels set as active pixels, i.e., sample points, and black circles (●) represent pixels set as inactive pixels. In the example in Figure 5A, the initial positions of the sample points are distributed evenly across the entire pixel array.

[0075] Figure 5B shows the movement rules applied to each sample point set in the pixel array.

[0076] The white circles in Figure 5B indicate that a movement rule, rule ID=3, is applied to the sample point, unconditionally moving it to a position where it touches more adjacent sample points detected as being closer than itself within the surrounding 8 pixels. The positions of the white circles in Figure 5B correspond to the sample points in Figure 5A.

[0077] The distance image DEP(t) generated at time t using the initial positions of the sample points shown in Figure 5A is shown on the left side of Figure 6.

[0078] In the distance image DEP(t) in Figure 6, the calculated distances to the sample points shown in Figure 5A are indicated by gray values.

[0079] Specifically, in the distance image DEP(t) in Figure 6, the calculated distances are classified into three categories: the closest (1st distance, hereinafter also referred to as short distance), the intermediate (2nd distance, hereinafter also referred to as medium distance), and the farthest (3rd distance, hereinafter also referred to as long distance). Each sample point is represented by a black, dot pattern, or white circle according to the calculated distance. Sample points represented in white are those where short distances were observed. Sample points represented by dot patterns are those where medium distances were observed. Sample points represented in black are those where long distances were observed. In relation to the object 13 as the subject, short distances are observed at sample points corresponding to the front of object 13 (the surface facing the distance measuring device 12), medium distances are observed at sample points corresponding to other surfaces of object 13, and long distances are observed at sample points corresponding to the background other than object 13. This distance image DEP(t) can be understood from the distance information stored in the sample point status table 71.

[0080] The determination unit 61 uses the distance information shown in the distance image DEP(t) to apply rule ID=3 to each sample point and update the position information of the sample points. After updating the position information, the sample points of the light receiving unit 53 will look like the right side of Figure 6.

[0081] In the sample points of the light-receiving unit 53 on the right side of Figure 6, the hatched circles represent sample points whose position has moved from the initial position, i.e., the position of the white circle (○) shown in A of Figure 5, due to the position information update process.

[0082] Looking at the position of the sample points after the update process, we see that the sample points have moved to the near-field portion of the captured scene, specifically to the region of object 13, allowing us to obtain a distance image with higher spatial resolution for near-field images.

[0083] An example of updating the positional information of 18 sample points included in region 101 from the numerous sample points of the distance image DEP(t) in Figure 6 by applying the movement rule with rule ID=3 will be explained with reference to Figure 7.

[0084] For convenience, the 18 sample points contained in region 101 are designated as a1 (the top leftmost sample point), a2 (the sample point to its right), and a2, a3, ... a18 in the order of the raster scan direction.

[0085] First, the decision unit 61 determines whether to move the sample position of sample point a1 based on the movement rule of rule ID=3. Specifically, the decision unit 61 focuses on the distance information of a 5x5 surrounding position centered on sample point a1 and determines whether moving to any of the 8 surrounding pixels would allow it to touch more nearby points than the current sample position. Among the 8 surrounding pixels, the position of the current sample point and areas outside the pixel array cannot be moved, so this determination can be omitted. For sample point a1, there is no position where it can touch more nearby points than the current sample position, so the decision unit 61 does not move the sample position.

[0086] Next, the decision unit 61 determines whether to move the sample position of sample point a2 based on the movement rule of rule ID=3. Specifically, the decision unit 61 focuses on the distance information of the surrounding 5x5 positions centered on sample point a2 and determines whether moving to any of the 8 pixels surrounding sample point a2 would allow it to touch more nearby points than the current sample position. For sample point a2, moving downwards from the current sample position would allow it to touch the medium-range sample points shown by the dot pattern, so the decision unit 61 moves the sample position to the downward position shown by the dashed line.

[0087] Next, the decision unit 61 determines whether to move the sample position of sample point a3 based on the movement rule of rule ID=3. Specifically, the decision unit 61 focuses on the distance information of the surrounding 5x5 positions centered on sample point a3 and determines whether moving to any of the 8 pixels surrounding sample point a3 would allow it to touch more nearby points than the current sample position. For sample point a3, moving downwards from the current sample position would allow it to touch the medium-range sample points shown by the dot pattern, so the decision unit 61 moves the sample position to the downward position shown by the dashed line.

[0088] Next, the decision unit 61 determines whether to move the sample position of sample point a4 based on the movement rule of rule ID=3. Specifically, the decision unit 61 focuses on the distance information of a 5x5 surrounding position centered on sample point a4 and determines whether moving to any of the 8 pixels surrounding sample point a4 would allow it to touch more nearby points than its current position. Since there is no position for sample point a4 that would allow it to touch more nearby points than its current position, the decision unit 61 does not move the sample position.

[0089] Similarly, for sample points a5 through a18, a decision is made to move their sample positions based on the movement rule of rule ID=3. If it is determined that the sample can touch more nearby points than its current position, the sample position is moved.

[0090] After the sample point update process is completed up to sample point a18, the region 101 in Figure 7 is the same as the region 101 of the light receiving unit 53 in Figure 6.

[0091] <4. Second example of sample point update process> Next, with reference to Figures 8 and 9, a second example of the sample point location information update process will be described.

[0092] In the first example described above, one rule, i.e., rule ID=3, was applied to all sample points in the pixel array. The second example illustrates how to apply multiple rules to the entire pixel array by dividing the entire pixel array into multiple regions and applying a different rule to each region.

[0093] More specifically, the determination unit 61 applies rule ID=1 to sample points in the outer region near the field of view of the pixel array, and applies rule ID=4 to sample points in the inner region further inside, thereby updating the position information of the sample points.

[0094] Figure 8A shows the initial position of the sample point determined by the determination unit 61 for the pixel array of the light receiving unit 53. This initial position is the same as in the first example described above, so no explanation is given. In the second example as well, for simplicity, the sample point is considered to be one pixel.

[0095] Figure 8B shows the movement rules applied to each sample point set in the pixel array.

[0096] The black circles in section B of Figure 8 represent sample points to which the movement rule with rule ID=1 is applied. The movement rule with rule ID=1 unconditionally executes a process that does not move the position of the sample points. In other words, the position of the sample points with black circles does not move regardless of the detected distance. This movement rule is set to prevent the failure to detect objects that appear outside the field of view of the distance measurement range during continuous distance measurement.

[0097] On the other hand, the white circles in Figure 8B represent sample points to which the movement rule of rule ID=4 is applied. The movement rule of rule ID=4 unconditionally moves the sample point to a position where it is in contact with more of the surrounding 8 pixels where a distance change has been detected.

[0098] According to the movement rule with rule ID=4, it is necessary to know the change in distance, so two distance images are required. Therefore, the determination unit 61 first generates two distance images, a distance image DEP(t-1) at time (t-1) and a distance image DEP(t) at the subsequent time (t), by repeating the irradiation and emission of light a predetermined number of times at the initial position sample point.

[0099] The distance images DEP(t-1) and DEP(t) in Figure 9 are images showing the calculated distance to the initial sample point as gray values. The meaning of the black, dot pattern, and white colors of the sample point shown in the two frames of distance images DEP(t-1) and DEP(t) is the same as in the first example. In the second example, as shown in the upper right of Figure 9, object 13 moves to the right, as indicated by the arrow, during the distance measurement over two frames. Therefore, the position of object 13 in the distance images DEP(t-1) and DEP(t) are different. These distance images DEP(t-1) and DEP(t) can be determined from the distance information stored in the sample point status table 71.

[0100] The determination unit 61 calculates the distance difference between corresponding sample points in the distance image DEP(t-1) and the distance image DEP(t), and generates a distance difference image DIF(t).

[0101] In the distance difference image DIF(t) shown in Figure 9, each sample point is represented in black, with a dot pattern, or in white. In this example, the calculated distance difference is classified into three categories: distance change from near to far (hereinafter also referred to as far-distance change), no change in distance, and distance change from far to near (hereinafter also referred to as short-distance change). Sample points represented in black are sample points where far-distance change was observed. Sample points represented with a dot pattern are sample points where no change in distance was observed. Sample points represented in white are sample points where short-distance change was observed.

[0102] Based on the distance difference image DIF(t), the determination unit 61 focuses on the distance difference of a 5x5 area surrounding the sample point to which rule ID=4 applies, similar to the process described in Figure 7, and determines whether moving to any of the 8 pixels surrounding the sample point would allow it to touch more distance change points than its current position. Once the position information update process is completed for all sample points to which rule ID=4 applies, the updated sample points of the light receiving unit 53 will look like the right side of Figure 9.

[0103] Looking at the position of the sample points after the update process, the sample points have moved around the object 13 that is moving in the shooting scene, making it possible to obtain a distance image with higher motion tracking capability.

[0104] In the first and second examples of the sample point update process described above, each sample point is composed of one pixel, and the determination unit 61 refers to the distance information of the 5x5 pixels surrounding the current position of the sample point and controls it to move to a pixel that matches the "action" in the sample point movement rule table 72 among the 3x3 surrounding 8 pixels. The 5x5 pixels, which are the reference range for the distance information, and the 3x3, which are the movable range, are merely examples and are not limited to them. Also, the reference range and movable range of the sample point's distance information may change depending on whether the sample point is composed of multiple adjacent pixels or a single pixel. The determination unit 61 applies the movement rule based on the distance information of the first surrounding area (reference range) centered on the sample point, determines whether or not to move the sample point to a predetermined position in a second surrounding area (movable range) that is smaller than the first surrounding area, and updates the position information of the sample point.

[0105] <5. Flowchart of distance image generation process> Next, the overall flow of the distance image generation process by the distance measuring system 1 will be explained with reference to the flowchart in Figure 10. This process is initiated, for example, when a distance measuring instruction is supplied from a higher-level host device.

[0106] First, in step S11, the determination unit 61 of the distance measuring device 12 determines the light emission conditions and outputs a light emission control signal to the light emission drive unit 32 of the illumination device 11 indicating which VCSELs of the VCSEL array should be illuminated based on the determined light emission conditions. For example, a light emission control signal that causes the entire VCSEL array to emit light with uniform brightness is output from the distance measuring device 12 to the illumination device 11.

[0107] In step S12, the determination unit 61 determines the initial position of the sample point for the pixel array of the light receiving unit 53, generates a sample point state table 71 corresponding to the determined initial position, and stores it in the storage unit 63.

[0108] In step S13, the determination unit 61 generates sample point control information based on the sample point status table 71 and supplies it to the pixel drive unit 52 and the multiplexer 80.

[0109] In step S14, the determination unit 61 generates a light emission trigger and outputs it to the light emission drive unit 32 of the illumination device 11, starting the emission of illumination light. Based on the light emission trigger, the light emission drive unit 32 turns on and off a predetermined VCSEL of the light emission unit 31. The light emission trigger is transmitted to the time measurement units 811 to 81 of the signal processing unit 54. Q It is also supplied to [another location].

[0110] In step S15, the distance measuring device 12 starts light reception and generates a distance image. More specifically, the pixel driving unit 52 drives a predetermined pixel as an active pixel based on sample point control information from the determination unit 61. When a photon is detected in an active pixel, a detection signal indicating this is output as a pixel signal to the signal processing unit 54 via the multiplexer 80. The multiplexer 80 of the signal processing unit 54 measures the pixel signals from each active pixel in a predetermined time measurement unit 81 in sample point units based on the sample point control information. i Controls the supply to be directed to the time measurement unit 81. i This generates a count value corresponding to the flight time of the irradiated light, and the corresponding histogram generation unit 82 i It supplies to the histogram generation unit 82. i The time measurement unit 81 i Based on the count values ​​supplied, a histogram of the count values ​​is created. Peak detection unit 83 i This is the histogram generation unit 82 i Based on the histogram data supplied from, the distance calculation unit 84 detects the peaks in the histogram. N From each of these, the flight time of the irradiated light is calculated based on the count value corresponding to the peak of the histogram, which is supplied on a sample point basis. The distance calculation unit 84 calculates the distance to the subject from the calculated flight time and generates a distance image in which the calculated distance is stored as a pixel value. The generated distance image is output to a higher-level host device via the input / output unit 55 and is also supplied to the control unit 51, and the distance information in the sample point status table 71 is updated. Output to a higher-level host device may be omitted.

[0111] In step S16, the determination unit 62 monitors the sample point status table 71 in the storage unit 63 and determines whether or not the sample point status table 71 has been updated. The determination unit 62 repeats the process in step S16 until it determines that the sample point status table 71 has been updated.

[0112] Then, if it is determined in step S16 that the sample point status table 71 has been updated, the process proceeds to step S17, and the determination unit 62 notifies the decision unit 61 of the update of the sample point status table 71.

[0113] In step S18, the determination unit 61 receives an update notification for the sample point status table 71 from the judgment unit 62 and updates the position information of each sample point based on the sample point status table 71 and the sample point movement rule table 72. More specifically, the position information of the sample points is updated based on the current distance information of each sample point recorded in the sample point status table 71 and the movement rules recorded in the sample point movement rule table 72. For example, when applying rule ID=3 to each sample point, the position information update process is performed as in the first example of the sample point update process described above. Along with the update of the position information of the sample points, the sample point status table 71 is also updated.

[0114] In step S19, the determination unit 61 applies the overall rule to all sample points and updates the position information of the sample points. For example, as part of the overall rule, if there is a region where distance information has not been sampled for a long period of time, the determination unit 61 performs a process to speculatively place a sample point in that region.

[0115] In step S20, the determination unit 61 of the control unit 51 determines whether or not to terminate the distance measurement. For example, the determination unit 61 determines to terminate the distance measurement when it has generated and output distance images a predetermined number of times. Alternatively, the determination unit 61 may determine to terminate the distance measurement when the number of sample points whose position information is updated (changed) falls below a predetermined value.

[0116] If it is determined in step S20 that the distance measurement is not yet complete, the process returns to step S13, and steps S13 through S20 described above are repeated. As a result, the distance measuring system 1 receives reflected light at the updated sample point and generates a distance image again.

[0117] On the other hand, if it is determined in step S20 that the distance measurement is complete, the process proceeds to step S21, and the control unit 51 or signal processing unit 54 outputs the last generated distance image to the higher-level host device as the final distance measurement result, and the distance image generation process ends. If the generated distance image is output to the higher-level host device each time in step S15 as described above, the process in step S20 can be omitted.

[0118] In the above explanation of the flowchart for the distance image generation process in Figure 10, an example was described in which the position information of the sample points is updated by applying rule ID=3 as the movement rule applied to the sample points.

[0119] In contrast, if, for example, rule ID=4 is applied as the movement rule applied to the sample point, two distance images are required, as explained in the second example of the sample point update process. In this case, the distance measuring device 12 generates two distance images in step S15 described above, and then performs the processing from step S16 onward.

[0120] According to the distance image generation process described above, the distance measuring device 12 can generate a distance image at a sample position suitable for the purpose of distance measurement by updating the position information of the sample points based on the sample point status table 71 and the sample point movement rule table 72. For example, the sample points can be moved to obtain a distance image with higher spatial resolution for short distances, or to obtain a distance image with higher motion tracking capability.

[0121] In the distance image generation process described above, the illumination device 11 emits light from the entire VCSEL array to emit illumination light. However, depending on the position of the sample point in the light receiving unit 53, the position of the VCSELs to emit light may be limited to a part of the VCSEL array. This can reduce the power consumption of the illumination device 11.

[0122] <6. Other examples of movement rules> Figure 11 shows other examples of “Conditions,” “Actions,” and “Constraints” in Sample Point Movement Rule Table 72.

[0123] For example, as a "condition," if there has been no change in distance during the past D1 frames (D1>0), one of the following "actions" can be applied. Additionally, one of the following "constraints" can be set.

[0124] In terms of "action", • Moves randomly within the range of surrounding R pixels (R>0). • Detects the brightness gradient within the distance measurement range and moves along the brightness gradient. • Obtain normal information for the distance measurement range and move toward the edge of the plane. • Move towards the direction with lower confidence in the measured distance or object recognition. Either of the following can be applied.

[0125] The luminance gradient of the distance measurement range may be obtained by acquiring images from an external RGB camera, or by obtaining luminance images obtained by operating the distance measuring device 12 in luminance observation mode. Alternatively, it may be obtained from luminance images based on the count values ​​of bins in which peaks in the histogram are detected. Normal information may be obtained from an external device that acquires normal information perpendicular to the surface (plane) of an object, or by a predetermined algorithm using luminance images and distance images. The confidence level of the distance can be determined using the confidence information from the sample point status table 71. The confidence level of object recognition can be obtained from the results of the recognition process using images from the RGB camera.

[0126] As for "constraints", • Make it difficult to approach sample points that have already been placed. • Make it easier to access areas that have not been sampled for a predetermined period. If the distance remains unchanged for D2 frames (D2 > 0), return to the initial sample position. • Do not move the sample points around the field of view, regardless of whether the distance changes or not. You can set one of the following:

[0127] The movement rules are not limited to the examples described above; other examples are also acceptable. Depending on the purpose of distance measurement, the characteristics of the object to be captured, the environmental conditions of the distance measurement range, etc., the sample point status table 71 and the sample point movement rule table 72 in the memory unit 63 can be modified as appropriate.

[0128] <7. Example of calculating the reliability of distance> Next, we will explain an example of how to calculate the confidence level of the distance, which is recorded in the sample point status table 71 in Figure 3.

[0129] The reliability of distance can be based on factors such as the error between the measured distance and the true distance, or the reliability of the signal-to-noise ratio (SNR) used to calculate the distance. Below, we will explain how to calculate distance reliability based on the SNR.

[0130] At a given sample point, the distance is calculated based on the count value of the bin that produces the highest histogram height due to reflected light. Since all bins contain noise components from ambient light, if the histogram height due to reflected light is insufficient, it becomes difficult to reliably select the bin to use for distance calculation. In other words, the higher the signal-to-noise ratio (SNR) when reflected light is treated as the signal and ambient light as the noise, the more reliable the distance measurement results are considered to be.

[0131] Therefore, for a given pixel, the confidence level is calculated by determining how far the count value (histogram height) of the bin that detected reflected light is from the count values ​​due to ambient light in all other bins.

[0132] Figure 12 shows a conceptual diagram of a histogram capturing ambient light and reflected light.

[0133] Assume that the count value λ of a given bin follows a Poisson distribution. When λ is greater than approximately 10, the Poisson distribution has a mean of λ and a variance of λ = σ. 2 It is known that this can be approximated by a normal distribution. Typically, in the distance measuring device 12, the count value λ is several thousand to tens of thousands, so the above approximation by the normal distribution can be used.

[0134] Specifically, when ambient light is constant, the average of the count values ​​of all bins that do not detect reflected light is λ. n Therefore, the count value of each bin will have a mean and variance of λ. n This results in a normal distribution. Furthermore, the mean λ of the count values ​​in the bin containing only reflected light is also determined. s Therefore, the count value of the bin that captures reflected light includes both reflected light and ambient light, so the mean and variance are (λ s +λ n This results in a normal distribution such that ).

[0135] The mean and variance of ambient light alone are λ n In a normal distribution, the standard deviation is σ n =√(λ n If we assume that the maximum count value is (λ n +σ n ), (λ n +2σ n ), (λ n +3σ n The probabilities of the results being less than or equal to ) are 68.27%, 95.45%, and 99.73%, respectively.

[0136] The mean and variance of the reflected light and ambient light are (λ s +λ n In a normal distribution, the standard deviation is σ s =√(λ s +λ n If we assume that the minimum count value is (λ s +λ n -σ s ), (λ s +λn -2σ s ), (λ s +λ n -3σ s The probabilities of these values ​​being greater than or equal to 68.27%, 95.45%, and 99.73%, respectively.

[0137] When the confidence interval is 1σ, the difference cntdiff between the maximum value of the ambient light count and the minimum value of the (reflected light + ambient light) count is given by the following equation (1). cntdiff=(λ s +λ n -σ s )-(λ n +σ n ) =λ s -σ s -σ n ··········(1)

[0138] The confidence level of distance cnf is calculated by multiplying the cntdiff in equation (1) above by a value corresponding to the confidence interval 1σ (σ). s +σ n It can be found using the following equation (2) obtained by dividing by ).

number

[0139] When the confidence level cnf in equation (2) is "0", there is a 68.27% probability that the count value in that bin is due to reflected light rather than ambient light noise. When the confidence level cnf is "1" or "2", it corresponds to a confidence interval of 2σ and 3σ, respectively, and the probability that the count value in that bin is due to reflected light rather than ambient light noise is 95.45% and 99.73%. Similarly, the larger the confidence level cnf, the greater the probability that the count value λ is due to reflected light. s This is the ambient light count value λ n Because it can be reliably separated, it becomes more reliable.

[0140] Furthermore, if the confidence level cnf in equation (2) is negative, the count value of the reflected light λ s or ambient light count value λ nDue to the variability, the ambient light count value λ is as shown in the histogram on the left of Figure 13. n The count value of the reflected light is λ s The probability of accidentally exceeding this threshold increases, leading to a judgment of low reliability. This occurs when the influence of ambient light is significant and the emission intensity of reflected light is weak, resulting in insufficient signal-to-noise ratio (SNR). In such cases, as shown in the histogram on the right of Figure 13, the SNR can be improved by increasing the number of times the irradiated light is emitted and the number of times the reflected light is counted. This is because the average count value of the bin containing reflected light is larger than that of the bin containing only ambient light, so increasing the number of counts relatively reduces the effect of variability. Thus, the distance measuring device 12 can also set the number of times the irradiated light is emitted (number of counts) to obtain the required SNR according to the emission intensity of the ambient light.

[0141] <8. Other configuration examples of distance measuring systems> Figure 14 is a block diagram showing an example configuration of another embodiment of the distance measuring system of this disclosure.

[0142] The distance measuring system 1 shown in Figure 14 has a configuration that further adds an external device 351 and a signal processing device 352 to the first embodiment shown in Figure 1, etc.

[0143] The external device 351 consists of, for example, an RGB camera that captures a subject including object 13. The RGB camera, as the external device 351, captures the subject and supplies the resulting image signal to the signal processing device 352.

[0144] The signal processing device 352 is composed of, for example, a general-purpose personal computer, FPGA, DSP, microprocessor, etc., and processes the image signal supplied from the external device 351. For example, the signal processing device 352 includes an image processing unit 361, which performs predetermined processing such as demosaicing, YUV conversion, normal detection, and object recognition of the input image signal. The color image generated by the signal processing device 352 is supplied to the control unit 51 via the input / output unit 55 of the distance measuring device 12.

[0145] The signal processing device 352 may also be incorporated as part of the external device 351.

[0146] If an RGB camera is provided as an external device 351, for example, the brightness information in the sample point status table 71 shown in Figure 3 can use the brightness values ​​of the color image supplied by the signal processing device 352. If information such as normal information or the reliability of object recognition is required as part of the "operation" of the movement rule, the image processing unit 361 may calculate this information and supply it to the distance measuring device 12. The determination unit 61 updates the position information of the sample points based on the sample point movement rule table 72, also using the data detected by the external device 351.

[0147] The external device 351 may be a device or sensor other than an RGB camera. Examples of devices other than RGB cameras include IR cameras that capture infrared (far-infrared, near-infrared) light, indirect ToF distance sensors (distance measuring devices), and EVS (event-based vision sensors). An indirect ToF distance sensor is a distance sensor that measures the distance to an object by detecting the time of flight from the moment the irradiated light is emitted to the moment the reflected light is received as a phase difference. An EVS is a sensor that has pixels that convert optical signals into photoelectric signals and output pixel signals, and outputs the temporal brightness change of the optical signal as an event signal (event data) based on the pixel signal. Unlike general image sensors that capture images in synchronization with a vertical synchronization signal and output frame data for one frame (screen) at the period of that vertical synchronization signal, an EVS is an asynchronous or address-controlled camera that outputs event data only when an event occurs.

[0148] For example, if a thermal camera that detects far-infrared rays is used as the external device 351, it is possible to detect temperatures within the same range as the distance measurement range, and an algorithm (movement rule) can be implemented that moves the sample point according to the temperature.

[0149] By providing the external device 351, the distance measuring device 12 can acquire information other than the position of the sample point from which it can acquire distance information, making it possible to interpolate the distance information using this information.

[0150] <9. Example Configuration for Brightness Observation Mode> As described above, the distance measuring device 12 has two operating modes: a distance measuring mode and a brightness observation mode. In brightness observation mode, a brightness image can be generated, so the brightness values ​​of the brightness image generated in brightness observation mode can be used in the brightness information of the sample point status table 71 shown in Figure 3.

[0151] Referring to Figure 15, the detailed configuration of the distance measuring device 12 when the operating mode is brightness observation mode will be described.

[0152] Figure 15 is a block diagram showing a detailed configuration example of the distance measuring device 12 when the operating mode is brightness observation mode.

[0153] In Figure 15, parts that are common to the configuration of the rangefinder 12 in the rangefinder mode shown in Figure 2 are denoted by the same reference numerals, and explanations of those parts are omitted as appropriate.

[0154] In the distance measuring device 12, which is in brightness observation mode, the signal processing unit 54 has photon counting units 3011 to 301 P , and a brightness image generation unit 302 is provided. Instead, time measurement units 811 to 81 Q , histogram generation unit 821 to 82 Q , peak detection units 831 to 83 Q The distance calculation unit 84 is omitted. The other configurations of the distance measuring device 12 are the same as in Figure 2.

[0155] When the operating mode is brightness observation mode, P (P>0) photon counting units 3011 to 301 are provided in the signal processing unit 54. PIn response, P pixels in the light-receiving unit 53 are set as active pixels. The multiplexer 80 connects the active pixels of the light-receiving unit 53 to the photon counting unit 301 on a one-to-one basis and supplies the pixel signal of each active pixel of the light-receiving unit 53 to the corresponding photon counting unit 301.

[0156] Photon counting unit 301 j (j=1 or P) counts the number of times the SPAD of the corresponding active pixel of the light-receiving unit 53 reacts within a predetermined period of time which is defined as one frame, i.e., the number of times a photon was incident. Then, the photon counting unit 301 j The counting results are supplied to the brightness image generation unit 302. Photon counting units 3011 to 301 P If the number of pixels P is equal to the total number of pixels in the light-receiving unit 53, one luminance image can be generated in one frame. However, if it is less than the total number of pixels in the light-receiving unit 53, one luminance image is generated over multiple frames by switching the active pixels. The luminance image generation unit 302 generates a luminance image using the photon count result measured at each pixel as the pixel value (luminance value) and supplies it to the control unit 51. As a result, the luminance information in the sample point status table 71 of the storage unit 63 is updated. The generated luminance image may also be output to a higher-level host device via the input / output unit 55.

[0157] Furthermore, the photon count results may be calculated on a multi-pixel (multiple pixel) basis, rather than on a single-pixel basis.

[0158] As described above, it is also possible to use the brightness information from the brightness image obtained by setting the operating mode to brightness observation mode. However, since it is necessary to switch between the operating mode and brightness observation mode, the frame rate for generating distance images will be reduced to less than half.

[0159] <10. Summary> The distance measuring device 12 can arrange the sample points of the pixel array to obtain more distance information by updating the position information of the sample points based on the sample point status table 71 and the sample point movement rule table 72.

[0160] More specifically, the determination unit 61 of the distance measuring device 12 updates the position information of the sample points based on the distance information of the sample points described in the sample point status table 71 and the sample point movement rules described in the sample point movement rule table 72.

[0161] For example, the system references the distance information of the 5x5 pixels surrounding the current sample point and moves to a pixel in the surrounding 3x3 pixels that matches the "action" in Sample Point Movement Rule Table 72. If the "action" is to move to a position that is closer to more sample points that have been detected as being closer than itself, a depth image with higher spatial resolution for close distances can be obtained. Also, if the "action" is to move to a position that is closer to more sample points that have been detected as having changed distances, a depth image with higher motion tracking capability can be obtained.

[0162] The sample point movement rule table 72 in the memory unit 63 can be rewritten according to the purpose of distance measurement. For example, the sample point movement rule table 72 is updated by transmission from an external device, such as a higher-level host device. This makes it possible to update the position information of the sample points using any algorithm. The determination unit 61 can store the information necessary for the movement rules described in the sample point movement rule table 72 in the sample point status table 71.

[0163] The embodiments of this technology are not limited to those described above, and various modifications are possible without departing from the spirit of this technology.

[0164] In this specification, a system means a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device containing multiple modules in one enclosure, are both considered systems.

[0165] Furthermore, the effects described herein are merely illustrative and not limiting, and other effects may also occur.

[0166] Furthermore, this technology can take the following configuration. (1) A pixel array in which pixels that receive reflected light from an object after the irradiated light has been reflected are arranged in a matrix, A determination unit that determines some of the pixels in the aforementioned pixel array as sample points for detecting distance information, A storage unit that stores a sample point status table that stores distance information of the sample points, and a sample point movement rule table that stores the movement rules of the sample points. Equipped with, The determination unit updates the position information of the sample points based on the sample point status table and the sample point movement rule table. Ranging device. (2) The sample point status table stores at least distance information for the sample point and rule identification information indicating the movement rule applied to the sample point. The aforementioned sample point movement rule table stores the movement rule corresponding to the rule identification information, The determination unit updates the position information of the sample point by executing the movement rule of the rule identification information described in the sample point movement rule table on the sample point. The distance measuring device described in (1) above. (3) The determination unit applies the movement rule based on the distance information of the first peripheral region centered on the sample point, determines whether or not to move the sample point to a predetermined position in a second peripheral region smaller than the first peripheral region, and updates the position information of the sample point. The distance measuring device described in (1) or (2) above. (4) The determination unit applies a single movement rule to the entire pixel array and updates the position information of the sample points. A distance measuring device according to any one of (1) to (3) above. (5) The determination unit divides the entire pixel array into multiple regions, applies different movement rules to each region, and updates the position information of the sample points. A distance measuring device according to any one of (1) to (4) above. (6) The aforementioned multiple regions are composed of an outer peripheral region around the field of view and an inner region inside that region. The distance measuring device described in (5) above. (7) The determination unit updates the position information of the sample points based on the sample point movement rule table, and then applies a general rule common to all the sample points to further update the position information of the sample points. A distance measuring device according to any one of (1) to (6) above. (8) The overall rule is to place a sample point at a location where the distance information has not been measured for a predetermined period of time. The distance measuring device described in (7) above. (9) The overall rule is a rule that updates sample points whose distance information has not changed for a predetermined period of time. The distance measuring device described in (7) above. (10) The aforementioned sample point movement rule table includes, as a movement rule, a rule that moves to a position in the surrounding area of ​​the sample point that is closer to more sample points detected as being closer than itself. A distance measuring device according to any one of (1) to (9) above. (11) The aforementioned sample point movement rule table includes, as a movement rule, a rule that moves to a position that is more in contact with the sample point where a short-range change has been detected in the area surrounding the sample point. A distance measuring device according to any one of (1) to (10) above. (12) The aforementioned sample point movement rule table includes, as movement rules, rules that move to a random position within the surrounding area of ​​the sample point. A distance measuring device according to any one of (1) to (11) above. (13) The aforementioned movement rule defines that a predetermined movement is performed when the sample point satisfies a predetermined condition. A distance measuring device according to any of (1) to (12) above. (14) The aforementioned movement rule includes an action that defines how to move the positional information of a sample point, and conditions for performing the said action. A distance measuring device according to any of (1) to (13) above. (15) The aforementioned movement rule includes an action that defines how to move the positional information of a sample point, conditions for performing the action, and constraints on the action. A distance measuring device according to any of (1) to (14) above. (16) The aforementioned sample point status table also stores the reliability of the distance information of the sample point, or the brightness information of the sample point. The determination unit updates the position information of the sample point using the reliability of the distance information of the sample point, or the brightness information of the sample point. A distance measuring device according to any one of (1) to (15) above. (17) The aforementioned determination unit also acquires data detected by an external device, The determination unit updates the position information of the sample point using the data detected by the external device. A distance measuring device according to any one of (1) to (16) above. (18) A distance measuring device comprising a pixel array in which pixels that receive reflected light reflected by an object are arranged in a matrix, A portion of the pixels in the aforementioned pixel array are designated as sample points for detecting distance information. The distance information of the aforementioned sample points is stored in the sample point status table. The position information of the sample points is updated based on the sample point status table and the sample point movement rule table, which stores the movement rules for the sample points. A control method for a distance measuring device. (19) A lighting device that emits light, A distance measuring device that receives reflected light from an object after the aforementioned irradiated light has been reflected by an object. Equipped with, The distance measuring device is A pixel array in which pixels that receive the reflected light are arranged in a matrix, A determination unit that determines some of the pixels in the aforementioned pixel array as sample points for detecting distance information, A storage unit that stores a sample point status table that stores distance information of the sample points, and a sample point movement rule table that stores the movement rules of the sample points. Equipped with, The determination unit updates the position information of the sample points based on the sample point status table and the sample point movement rule table. Distancing system. [Explanation of symbols]

[0167] 1: Distance measuring system, 11: Lighting device, 12: Distance measuring device, 13: Object, 31: Light-emitting unit, 32: Light-emitting drive unit, 51: Control unit, 52: Pixel drive unit, 53: Light-receiving unit, 54: Signal processing unit, 55: Input / output unit, 61: Determination unit, 62: Judgment unit, 63: Memory unit, 71: Sample point status table, 72: Sample point movement rule table, 80: Multiplexer, 81: Time measurement unit, 82: Histogram generation unit, 83: Peak detection unit, 84: Distance calculation unit, 301: Photon counting unit, 302: Brightness image generation unit, 351: External device, 352: Signal processing unit, 361: Image processing unit

Claims

1. A pixel array in which pixels that receive reflected light from an object after the irradiated light has been reflected are arranged in a matrix, A determination unit determines a plurality of sample points for the pixel array, where a portion of the pixels in the pixel array are used as one sample point for detecting distance information. A sample point status table stores the position information and distance information of the sample point and rule identification information indicating the movement rule applied to the sample point for each of the plurality of sample points, and a storage unit stores a sample point movement rule table that stores the movement rule corresponding to the rule identification information. Equipped with, The determination unit updates the position information of each sample point in the sample point status table by applying the movement rules of the rule identification information described in the sample point movement rule table to each of the plurality of sample points. Ranging device.

2. The determination unit applies the movement rule based on the distance information of the first peripheral region centered on the sample point, determines whether or not to move the sample point to a predetermined position in a second peripheral region smaller than the first peripheral region, and updates the position information of each sample point in the sample point status table. The distance measuring device according to claim 1.

3. The determination unit applies a single movement rule to the entire pixel array and updates the position information of each sample point in the sample point state table. The distance measuring device according to claim 1.

4. The determination unit divides the entire pixel array into multiple regions, applies different movement rules to each region, and updates the position information of each sample point in the sample point state table. The distance measuring device according to claim 1.

5. The aforementioned multiple regions are composed of an outer peripheral region around the field of view and an inner region inside that region. The distance measuring device according to claim 4.

6. The determination unit updates the position information of each sample point in the sample point status table based on the sample point movement rule table, and then applies a general rule common to all sample points to further update the position information of each sample point in the sample point status table. The distance measuring device according to claim 1.

7. The overall rule is to place a sample point at a location where the distance information has not been measured for a predetermined period of time. The distance measuring device according to claim 6.

8. The overall rule is a rule that updates sample points whose distance information has not changed for a predetermined period of time. The distance measuring device according to claim 6.

9. The aforementioned sample point movement rule table includes, as a movement rule, a rule that moves to a position in the surrounding area of ​​the sample point that is closer to more sample points detected as being closer than itself. The distance measuring device according to claim 1.

10. The aforementioned sample point movement rule table includes, as a movement rule, a rule that moves to a position that is more in contact with the sample point where a short-range change has been detected in the area surrounding the sample point. The distance measuring device according to claim 1.

11. The aforementioned sample point movement rule table includes, as movement rules, rules that move to a random position within the surrounding area of ​​the sample point. The distance measuring device according to claim 1.

12. The aforementioned movement rule defines that a predetermined movement is performed when the sample point satisfies a predetermined condition. The distance measuring device according to claim 1.

13. The aforementioned movement rule includes an action that defines how to move the positional information of a sample point, and conditions for performing the said action. The distance measuring device according to claim 1.

14. The aforementioned movement rule includes an action that defines how to move the positional information of a sample point, conditions for performing the action, and constraints on the action. The distance measuring device according to claim 1.

15. The aforementioned sample point status table also stores the reliability of the distance information of the sample point, or the brightness information of the sample point. The determination unit updates the position information of each sample point in the sample point status table, using the reliability of the distance information of the sample point, or the brightness information of the sample point. The distance measuring device according to claim 1.

16. The aforementioned determination unit also acquires data detected by an external device, The determination unit updates the position information of each sample point in the sample point status table, also using the data detected by the external device. The distance measuring device according to claim 1.

17. A distance measuring device comprising a pixel array in which pixels that receive reflected light reflected by an object are arranged in a matrix, A plurality of sample points are determined for the pixel array, with some of the pixels in the pixel array being used as one sample point for detecting distance information. For each of the multiple sample points, the location information and distance information of the sample point, and rule identification information indicating the movement rule applied to the sample point are stored in the sample point status table of the storage unit. Based on the sample point movement rule table in the storage unit, which stores the movement rule corresponding to the rule identification information, the position information of each sample point in the sample point status table is updated by applying the movement rule of the rule identification information to each of the multiple sample points. A control method for a distance measuring device.

18. A lighting device that emits light, A distance measuring device that receives reflected light from an object after the aforementioned irradiated light has been reflected by an object. Equipped with, The distance measuring device is A pixel array in which pixels that receive the reflected light are arranged in a matrix, A determination unit determines a plurality of sample points for the pixel array, where a portion of the pixels in the pixel array are used as one sample point for detecting distance information. A sample point status table stores the position information and distance information of the sample point and rule identification information indicating the movement rule applied to the sample point for each of the plurality of sample points, and a storage unit stores a sample point movement rule table that stores the movement rule corresponding to the rule identification information. Equipped with, The determination unit updates the position information of each sample point in the sample point status table by applying the movement rules of the rule identification information described in the sample point movement rule table to each of the plurality of sample points. Distancing system.

Images