Distance measurement device and offset noise removal method

Abstract

This distance measurement device (10) comprises: a light source (11); a light reception unit (12); a drive control unit (13); a distance calculation unit (14) that calculates a distance value for the distance to a subject (20); and an offset noise removal unit (15) or the like that performs filtering on the distance value calculated by the distance calculation unit (14), thereby generating a distance image removed of offset noise. The offset noise removal unit (15) or the like counts the number of invalid values among the distance values of pixels positioned around a relevant pixel, and when the number of obtained invalid values exceeds a number threshold value, outputs the distance value of the relevant pixel by changing same to an invalid value.

Description

Distance measuring device and offset noise removal method

[0001] The present disclosure relates to a distance measuring device and an offset noise removal method, and more particularly to a time-of-flight (TOF) distance measuring device and an offset noise removal method.

[0002] Conventionally, techniques for improving the reliability of TOF distance measuring devices have been proposed (see, for example, Patent Document 1).

[0003] Japanese Patent Application Laid-Open No. 2020-190418

[0004] However, in the technology of Patent Document 1, when an offset signal, which is a signal component excluding reflected light such as background light, dark current, and parasitic sensitivity, is generated, offset noise, which is a false distance value caused by shot noise due to the offset signal, may be generated, which may reduce the reliability of the obtained distance image as a whole. This may cause problems in the performance of detection and recognition processing in downstream applications (e.g., face recognition, driver monitoring, people flow detection, etc.) that use the distance image output from the distance measuring device.

[0005] Therefore, an object of the present disclosure is to provide a distance measuring device and an offset noise removal method that can suppress a decrease in the overall reliability of the obtained distance image even when offset noise occurs.

[0006] In order to achieve the above object, a distance measuring device according to one embodiment of the present disclosure is a distance measuring device for measuring a distance to a subject, comprising: a light source that emits illumination light toward the subject in accordance with an input illumination pulse; a light receiving unit having a plurality of pixels arranged two-dimensionally that receives light including reflected light from the subject during an exposure period determined by an input exposure pulse; a drive control unit that drives the light source by outputting the illumination pulse to the light source and drives the light receiving unit by outputting the exposure pulse to the light receiving unit; and a distance measuring unit that measures, for each of the plurality of pixels, a time period from when illumination light is emitted from the light source to when the reflected light is received by the light receiving unit based on the amount of reflected light received by the light receiving unit. The apparatus comprises a distance calculation unit that calculates a depth corresponding to the time difference as a distance value to the subject, and an offset noise removal unit that generates a distance image for each of the plurality of pixels by applying a filter process to the distance value calculated by the distance calculation unit using the distance values ​​of pixels located around the pixel in question, thereby removing offset noise caused by offset signals, which are signal components excluding reflected light.The offset noise removal unit counts the number of invalid values, which are distance values ​​of pixels located around the pixel in question that do not satisfy specified conditions regarding reliability, and if the number of invalid values ​​obtained exceeds a number threshold, changes the distance value of the pixel in question to an invalid value and outputs it.

[0007] In order to achieve the above object, an offset noise removal method according to one embodiment of the present disclosure is an offset noise removal method for removing offset noise caused by an offset signal, which is a signal component excluding reflected light, in a distance measuring device that measures a distance to a subject, the distance measuring device including a light source that emits irradiation light toward the subject in accordance with an input light emission pulse, a light receiving unit having a plurality of pixels arranged two-dimensionally that receives light including reflected light from the subject during an exposure period determined by an input exposure pulse, a drive control unit that drives the light source by outputting the light emission pulse to the light source and drives the light receiving unit by outputting the exposure pulse to the light receiving unit, and a light receiving unit that receives the reflected light received by the light receiving unit for each of the plurality of pixels. and a distance calculation unit that calculates, based on the amount of light emitted from the light source and the amount of reflected light received by the light receiving unit, as a distance value to the subject. The offset noise removal method includes a noise removal step that generates a distance image from which the offset noise has been removed by applying a filter process to the distance value calculated by the distance calculation unit for each of the plurality of pixels using distance values ​​of pixels located around the pixel in question, and in the noise removal step, the number of invalid values, which are distance values ​​that do not satisfy a predetermined condition regarding reliability, are counted among the distance values ​​of pixels located around the pixel in question, and if the number of invalid values ​​obtained exceeds a number threshold, the distance value of the pixel in question is changed to an invalid value and output.

[0008] The present disclosure provides a distance measuring device and an offset noise removal method that can suppress a decrease in the overall reliability of the obtained distance image even when offset noise occurs.

[0009] FIG. 1 is a block diagram showing the configuration of a distance measuring device according to an embodiment. FIG. 2 is a timing chart showing distance measurement operations by the distance measuring device according to an embodiment. FIG. 3 is a diagram explaining offset noise in the distance measuring device according to an embodiment. FIG. 4A is a block diagram showing only the offset noise removal unit extracted from the distance measuring device shown in FIG. 1 together with input and output information. FIG. 4B is a flowchart showing the operation of the offset noise removal unit included in the distance measuring device according to an embodiment. FIG. 4C is a diagram showing distance values ​​that are subject to spatial filtering processing, for explaining the operation of the offset noise removal unit included in the distance measuring device according to an embodiment. FIG. 5 is a diagram explaining characteristic components included in a distance measuring device according to a first modification of an embodiment. FIG. 6 is a diagram explaining characteristic components included in a distance measuring device according to a second modification of an embodiment. FIG. 7A is a diagram explaining characteristic components included in a distance measuring device according to a third modification of an embodiment. FIG. 7B is a diagram showing distance values ​​that are subject to spatial filtering, for explaining the operation of the offset noise removal unit and number threshold calculation unit included in the distance measuring device according to the third modification.

[0010] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that each of the embodiments described below represents a specific example of the present disclosure. Numerical values, circuit elements, components, component placement and connection configurations, steps, step order, etc. shown in the following embodiments are merely examples and are not intended to limit the present disclosure. Furthermore, each figure is not necessarily an exact illustration. In each figure, substantially identical configurations are assigned the same reference numerals, and redundant explanations are omitted or simplified. Furthermore, "connection" means electrical connection, and includes not only cases where two circuit elements are directly connected, but also cases where two circuit elements are indirectly connected with another circuit element inserted between them.

[0011] 1 is a block diagram showing the configuration of a distance measuring device 10 according to an embodiment. In addition to the distance measuring device 10, the diagram also shows an object 20, light 21 directed toward the object 20, and light 22 reflected from the object 20.

[0012] The distance measuring device 10 is a TOF distance measuring device having a function of removing offset noise, and includes a light source 11 , a light receiving unit 12 , a drive control unit 13 , a distance calculation unit 14 , and an offset noise removal unit 15 .

[0013] The light source 11 is a light source that emits irradiation light 21 such as infrared (IR) light toward the subject 20 in accordance with an input light emission pulse, and is, for example, an LED (light emitting diode) or an LD (laser diode).

[0014] The light receiving unit 12 is a circuit having a plurality of pixels arranged two-dimensionally that receive light including reflected light 22 from the subject 20 during an exposure period determined by the input exposure pulse, and outputs an electrical signal (RAW data) indicating the amount of light received by each pixel, and is, for example, a solid-state imaging element such as a CMOS sensor composed of pixels arranged two-dimensionally.

[0015] The drive control unit 13 is a circuit that drives the light source 11 by outputting an emission pulse to the light source 11 and drives the light receiving unit 12 by outputting an exposure pulse to the light receiving unit 12, and is composed of, for example, a timer circuit and a pulse generating circuit.

[0016] The distance calculation unit 14 is a circuit that calculates, for each of the multiple pixels in the light receiving unit 12, a depth corresponding to the time difference between when the illumination light 21 is emitted from the light source 11 and when the reflected light 22 is received by the light receiving unit 12, as a distance value to the subject 20, based on the amount of reflected light 22 received by the light receiving unit 12, and is configured, for example, with a frame memory and a processor that executes a built-in program. In addition to the distance value, the distance calculation unit 14 also outputs various information required by the offset noise removal unit 15, such as the amount of received light, to the offset noise removal unit 15.

[0017] The offset noise removal unit 15 is a circuit that performs spatial filtering using distance values ​​of pixels located around the pixel in question to reduce variations in the distance values ​​calculated by the distance calculation unit 14. This filter calculates a distance value (Depth) from which offset noise caused by offset signals, which are signal components excluding reflected light 22, is removed, and outputs the calculated distance value as a distance image. The circuit is, for example, composed of a product-sum operation circuit or a processor that executes a program stored in memory. Prior to spatial filtering, the offset noise removal unit 15 counts the number of invalid values, which are distance values ​​of pixels located around the pixel in question that do not satisfy a predetermined reliability condition. If the number of invalid values ​​exceeds a threshold value, the offset noise removal unit 15 changes the distance value of the pixel in question to an invalid value and outputs the invalid value. The predetermined reliability condition is, for example, that the net amount of received light excluding offset signal components (hereinafter also referred to as "IR level") exceeds a predetermined light reception threshold. In this embodiment, the threshold value is a fixed value.

[0018] FIG. 2 is a timing chart showing the distance measurement operation by the distance measuring device 10 according to the embodiment.

[0019] More specifically, FIG. 2A shows the timing of a vertical synchronization signal indicating a frame period.

[0020] 2(b) shows the timing of two types of operation periods ("light emission exposure period" and "readout period") in one frame. The light emission exposure period is a period during which signal charge is accumulated in each pixel of the light receiving unit 12 by repeatedly emitting irradiation light 21 from the light source 11 and receiving the light at the light receiving unit 12. The readout period is a period during which RAW data indicating the signal charge (amount of received light) accumulated in each pixel of the light receiving unit 12 during the light emission exposure period is read out from the light receiving unit 12 to the distance calculation unit 14.

[0021] Figure 2(c) shows the timing of six unit intervals β1 to β8 that make up the light emission exposure period in Figure 2(b). In each unit interval β1 to β8, the light emission and exposure shown in Figure 2(d) to 2(f) are repeated.

[0022] 2(d) shows the timing of two types of exposure periods ("A0 / A2 exposure" and "A1 / A3 exposure") in one unit section shown in FIG. 2(c). The A0 / A2 exposure is an exposure period accompanied by light emission for receiving reflected light 22, as shown in the left half of the timing in FIG. 2(e) and (f). On the other hand, the A1 / A3 exposure is an exposure period accompanied by no light emission for identifying an offset signal other than reflected light when calculating a distance value, during a time when there is no reflected light 22, as shown in the right half of the timing in FIG. 2(e) and (f).

[0023] 2(e) shows the timing at which light emission pulses are output from the drive control unit 13 to the light source 11 during the two types of exposure periods shown in FIG. 2(d). Here, light emission pulses are output α times only during the A0 / A2 exposure. The light emission pulse is a high active signal, and irradiation light 21 is emitted from the light source 11 during the section indicating high.

[0024] (f) of Fig. 2 shows the timing at which exposure pulses are output from the drive control unit 13 to the light receiving unit 12 during the two types of exposure periods shown in (d) of Fig. 2. Here, exposure pulses are output α × 2 times during each of the A0 / A2 exposure and A1 / A3 exposure. Note that the exposure pulse is a low active signal, and light is received by the light receiving unit 12 during the section in which it indicates low.

[0025] 2(g) shows a time-expanded timing diagram of a pair of a light emission pulse ("LD") and an exposure pulse ("A0" and "A2", "A1" and "A3") used to calculate a distance value of one of the light emission pulses shown in FIG. 2(e) and the exposure pulses shown in FIG. 2(f). As shown, in the A0 / A2 exposure in FIG. 2(d), two exposure pulses A0 and A2 having the same pulse width but different phases are output in synchronization with one light emission pulse ("LD") to receive reflected light 22. On the other hand, in the A1 / A3 exposure in FIG. 2(d), two exposure pulses A1 and A3 having the same pulse width but different phases are output without a light emission pulse ("LD") to identify an offset signal.

[0026] As described above, for each frame, the signal charge accumulated in each of the multiple pixels in the light receiving unit 12 by multiple light emission exposure sequences during the light emission exposure period of (b) in Figure 2 is read out from the light receiving unit 12 to the distance calculation unit 14 as RAW data indicating the accumulated signal charge (amount of received light) during the readout period of (b) in Figure 2, and the distance calculation unit 14 calculates the distance value by the following calculation.

[0027] That is, if the amounts of received light accumulated in the pixels by exposures A0, A1, A2, and A3 are respectively A0, A1, A2, and A3, the pulse width of the light emission pulse (i.e., the pulse width of each exposure pulse is also the same) is Tp, and the speed of light is c, then the distance value Depth is expressed by the following equation 1.

[0028] Depth=c×Tp / 2×(A2-A3) / {(A0-A1)+(A2-A3)}...(Formula 1)

[0029] This is because (A0-A1) in Equation 1 indicates the net amount of light received from one reflected light 22 excluding the offset signal component due to exposure in the same first half period Tp as the light emitting pulse LD, and (A2-A3) indicates the net amount of light received from one reflected light 22 excluding the offset signal component due to exposure in the second half period Tp following the period of the light emitting pulse LD, so {(A0-A1)+(A2-A3)} indicates the net amount of light received (i.e., "IR level") excluding the offset signal component for one reflected light 22. Therefore, (A2-A3) / {(A0-A1)+(A2-A3)} indicates the ratio of the net amount of received light in the latter half period Tp to the net amount of received light for one reflected light 22, and therefore is the time (TOF) from when the light emission pulse LD is output until the reflected light 22 is detected, that is, the time from when the illumination light 21 is emitted from the light source 11 until it is reflected by the subject 20 and reaches the light receiving unit 12 of the distance measuring device 10 (the time it takes for light to make a round trip). Therefore, the distance value Depth between the subject 20 and the distance measuring device 10 is expressed by the above formula 1.

[0030] 3 is a diagram illustrating offset noise in the distance measuring device 10 according to the embodiment. Here, examples of box plots are shown that indicate the range of variation in the amount of received light during imaging under various conditions. In this figure, the shaded graphs indicate the combined signal amount of the reflected light signal and the offset signal, and the white graphs indicate the signal amount of the offset signal. Each box plot also shows an example of the range of variation in the signal amount.

[0031] 3A shows an example of the amounts of light received A0 to A3 at exposures A0 to A3 when capturing an image of a subject. In this case, the variation in the reflected light 22 is generally greater than the variation in the offset signal.

[0032] 3B shows an example of the received light amounts A0 to A3 for exposures A0 to A3 when capturing an image at infinity and the offset signal is small. In this case, the variation in the offset signal is generally greater than the variation in the reflected light 22.

[0033] 3C shows an example of the received light amounts A0 to A3 for exposures A0 to A3 when capturing an image of infinity and the offset signal is large. In this case, the offset signal generally varies more than the reflected light 22.

[0034] FIG. 3D shows an example of the IR level in each of the cases shown in FIG. 3A to FIG. 3C. As shown in this figure, in the case of FIG. 3B, the IR level does not exceed the predetermined light receiving threshold. On the other hand, in the case of FIG. 3C, the IR level may exceed the light receiving threshold due to signal variations. When the IR level exceeds the threshold, the distance value calculated based on the IR level can be considered offset noise calculated due to the offset signal.

[0035] Therefore, in this embodiment, from the viewpoint of eliminating the offset signal, the offset noise elimination unit 15 sets a predetermined condition for the reliability of the pixel distance value, which is that the net amount of received light excluding the offset signal component (i.e., the IR level) must exceed a predetermined light-receiving threshold. In other words, the offset noise elimination unit 15 treats a distance value calculated using an IR level that exceeds the light-receiving threshold as a valid distance value, while treating a distance value calculated using an IR level that does not exceed the light-receiving threshold as an invalid value. Therefore, the larger the light-receiving threshold, the more effectively the influence of the offset signal can be eliminated, but the fewer cases in which the distance value (depth) of the subject can be calculated.

[0036] Next, the operation of the offset noise removal unit 15 included in the distance measuring device 10 according to the embodiment will be described.

[0037] Fig. 4A is a block diagram showing only the offset noise removal unit 15 extracted from the distance measuring device 10 shown in Fig. 1 together with input and output information. Fig. 4B is a flowchart showing the operation of the offset noise removal unit 15 provided in the distance measuring device 10 according to the embodiment (i.e., an offset noise removal method). Fig. 4C is a diagram showing distance values ​​(including invalid values) that are subject to spatial filtering processing, for explaining the operation of the offset noise removal unit 15 provided in the distance measuring device 10 according to the embodiment. Here, an example of distance values ​​of 5 x 5 pixels including a pixel with a distance value of interest is shown.

[0038] First, the offset noise removal unit 15 performs a process of replacing all distance values ​​calculated by the distance calculation unit 14 with invalid values ​​if the distance value does not satisfy a predetermined condition regarding reliability (here, the IR level is equal to or lower than the light receiving threshold value) (S10 in FIG. 4B). Note that this process may be performed by the distance calculation unit 14.

[0039] Next, the offset noise removal unit 15 repeats the following processing for each of all distance values ​​calculated by the distance calculation unit 14 (the distance value being focused on is referred to as the "target distance value") (S11 to S16 in Figure 4B).

[0040] First, the offset noise removal unit 15 determines whether the distance value of interest is an invalid value (that is, the distance value replaced with an invalid value in step S10 in FIG. 4B) (S12 in FIG. 4B).

[0041] As a result, if it is determined that the target distance value is an invalid value (Yes in S12 of Figure 4B), the offset noise removal unit 15 terminates the processing for this target distance value (one processing in the loop) (S16 of Figure 4B).On the other hand, if it is determined that the target distance value is not an invalid value (No in S12 of Figure 4B), the offset noise removal unit 15 counts the number of invalid values ​​(i.e., distance values ​​replaced with invalid values ​​in step S10 of Figure 4B) among the distance values ​​of pixels located around the pixel of the target distance value defined as a spatial filter (S13 of Figure 4B), and determines whether the number of invalid values ​​obtained by the count exceeds a predetermined number threshold (S14 of Figure 4B).

[0042] As a result, if it is determined that the number of invalid values ​​exceeds a predetermined threshold value (Yes in S14 of Figure 4B), the offset noise removal unit 15 replaces the target distance value with an invalid value (S15 of Figure 4B) and then terminates the processing for this target distance value (one processing in the loop) (S16 of Figure 4B).

[0043] On the other hand, if it is determined that the number of invalid values ​​does not exceed a predetermined threshold (No in S14 of Figure 4B), the offset noise removal unit 15 terminates the processing for this target distance value (one processing in the loop) (S16 of Figure 4B).

[0044] After the above loop processing (S11 to S16) is completed, the offset noise removal unit 15 performs a predetermined spatial filtering process on each of all distance values ​​(i.e., distance values ​​of interest) calculated by the distance calculation unit 14, and replaces the distance value of interest with the distance value obtained by the spatial filtering process (S17 in FIG. 4B). Note that the predetermined spatial filtering process is a product-sum operation (e.g., a Gaussian filter) using weighting coefficients (i.e., filter coefficients) associated with the positions of each pixel, on the distance value of interest and the distance values ​​of pixels (excluding invalid values) located around the pixel of the distance value of interest.

[0045] In the example shown in Fig. 4C , the distance value of interest has not been replaced with an invalid value (No in S12 in Fig. 4B ), so the number of invalid values ​​among the distance values ​​of the surrounding pixels is counted (in this example, 13 is counted) (S13 in Fig. 4B ), and the counted number of invalid values ​​(i.e., 13) is compared with a number threshold (e.g., 10) (S14 in Fig. 4B ). As a result, since the number of invalid values ​​(i.e., 13) exceeds the number threshold (i.e., 10) (Yes in S14 in Fig. 4B ), the distance value of interest is replaced with an invalid value (S15 in Fig. 4B ). Note that Fig. 4C shows an example in which a spatial filter consisting of 5 × 5 pixels is used, but the size of the spatial filter is not limited to this size.

[0046] In this way, according to the offset noise removal unit 15 provided in the distance measuring device 10 of the embodiment, the number of invalid values, which are distance values ​​that do not satisfy specified conditions regarding reliability, among the distance values ​​of pixels located around the pixel of the distance value of interest is counted, and if the number of invalid values ​​obtained exceeds a number threshold, the distance value of interest is replaced with an invalid value without performing spatial filtering processing.

[0047] As a result, in the spatial filtering process for removing offset noise, distance values ​​of interest that have many invalid values ​​in many surrounding pixels are replaced with invalid values ​​(i.e., excluded as distance values) and then spatial filtering process is performed to generate a distance image, so that, compared to conventional techniques that can remove noise due to the influence of false distance values, the overall reliability of the obtained distance image is suppressed from decreasing, and high accuracy is maintained for the distance measuring device 10. As a result, high performance in detection and recognition processing can be ensured in downstream applications (e.g., face recognition, driver monitoring, people flow detection, etc.) that use the distance image output from the distance measuring device 10.

[0048] Next, a distance measuring device according to a first modification of the embodiment will be described. The distance measuring device according to the first modification of the embodiment basically includes the components included in the distance measuring device 10 according to the embodiment shown in Fig. 1, but differs from the embodiment in that it includes an offset noise removal unit 15a instead of the offset noise removal unit 15, and further includes a number threshold calculation unit 16a (see Fig. 5). The following description will focus on the differences from the embodiment.

[0049] Fig. 5 is a diagram illustrating characteristic components of a distance measuring device according to a first modified example of the embodiment. More specifically, Fig. 5(a) is a block diagram illustrating an offset noise removal unit 15a and a number threshold calculation unit 16a included in the distance measuring device according to the first modified example, along with input and output information. Fig. 5(b) is a graph illustrating the relationship between the level indicated by the offset signal input to the number threshold calculation unit 16a (i.e., the offset signal level) and the number threshold output from the number threshold calculation unit 16a. Fig. 5(c) is a diagram illustrating distance values ​​(including invalid values) that are subject to spatial filtering, for explaining the operation of the offset noise removal unit 15a and the number threshold calculation unit 16a included in the distance measuring device according to the first modified example.

[0050] As shown in (a) of Figure 5, offset noise removal unit 15a basically has the same function as offset noise removal unit 15 according to the embodiment, but differs from offset noise removal unit 15 according to the embodiment in that the number threshold used to determine when replacing a distance value of interest with an invalid value is not a fixed value, but a number threshold calculated by number threshold calculation unit 16a.

[0051] The number threshold calculation unit 16a calculates a number threshold based on a signal dependent on offset noise and notifies the offset noise removal unit 15a of the calculated number threshold. Specifically, in this modification, the number threshold calculation unit 16a acquires an offset signal indicating the amount of offset noise in the amount of light received by the light receiving unit 12 as the offset noise-dependent signal, and calculates the number threshold based on the offset signal so that the larger the amount of offset noise indicated by the acquired offset signal, the smaller the number threshold. Here, the offset signal is, for example, the total amount of background light received in one frame for all pixels (i.e., the sum of the amount of light received in the A1 exposure and the amount of light received in the A3 exposure) or the average value per pixel of the total amount of background light received, and is calculated by the distance calculation unit 14 or based on the amount of light received acquired from the light receiving unit 12 by the number threshold calculation unit 16a.

[0052] More specifically, number threshold calculation unit 16a internally stores data indicating the relationship (i.e., negative correlation) between the magnitude of the offset signal (offset signal level) and the number threshold, as shown in Figure 5(b), and when the offset signal generated by distance calculation unit 14 is input, it refers to the relationship indicated by the internally stored data to determine a number threshold that becomes smaller as the level indicated by the input offset signal becomes larger, and notifies offset noise removal unit 15a of this number threshold. Then, offset noise removal unit 15a uses the number threshold notified by number threshold calculation unit 16a to determine whether to replace the distance value of interest with an invalid value (S14 in Figure 4B).

[0053] In the example shown in (c) of Figure 5, the distance value of interest is not replaced with an invalid value (No in S12 of Figure 4B), so the number of invalid values ​​among the distance values ​​of the surrounding pixels is counted (in this example, it is counted as 13) (S13 of Figure 4B), and the counted number of invalid values ​​(i.e., 13) is compared with the number threshold (e.g., 6) notified by the number threshold calculation unit 16a (S14 of Figure 4B).As a result, since the number of invalid values ​​(i.e., 13) exceeds the number threshold (i.e., 6) (Yes in S14 of Figure 4B), the distance value of interest is replaced with an invalid value (S15 of Figure 4B).

[0054] In this way, the offset noise removal unit 15a and number threshold calculation unit 16a provided in the ranging device of this modified example count the number of invalid values, which are distance values ​​that do not satisfy specified conditions regarding reliability, among the distance values ​​of pixels located around the pixel of the distance value of interest, and if the number of invalid values ​​obtained exceeds a number threshold determined depending on the offset signal (i.e., with negative correlation), the distance value of interest is replaced with an invalid value without performing spatial filtering processing.

[0055] As a result, in the spatial filtering process for removing offset noise, when the number of surrounding invalid values ​​is compared with the number threshold, the larger the offset signal, the smaller the number threshold (i.e., the higher the probability) is used to replace the target distance value with an invalid value (i.e., exclude it as a distance value), and then a distance image is generated. Therefore, compared to conventional techniques that can remove noise due to the influence of false distance values, the overall reliability of the obtained distance image is suppressed from decreasing, and high accuracy is maintained as a distance measuring device.

[0056] Next, a distance measuring device according to a second modification of the embodiment will be described. The distance measuring device according to the second modification of the embodiment basically includes the components included in the distance measuring device according to the first modification of the embodiment described using Fig. 5, but differs from the first modification of the embodiment in that it includes number threshold calculation unit 16b (see Fig. 6) instead of number threshold calculation unit 16a. The following description will focus on the differences from the first modification of the embodiment.

[0057] Fig. 6 is a diagram illustrating characteristic components of a distance measuring device according to a second modification of the embodiment. More specifically, Fig. 6(a) is a block diagram illustrating, together with input and output information, an offset noise removal unit 15a and a number threshold calculation unit 16b included in the distance measuring device according to the second modification. Fig. 6(b) is a graph illustrating the relationship between the temperature indicated by the temperature information input to the number threshold calculation unit 16b and the number threshold output from the number threshold calculation unit 16b. Fig. 6(c) is a diagram illustrating distance values ​​(including invalid values) that are subject to spatial filtering, for explaining the operation of the offset noise removal unit 15a and the number threshold calculation unit 16b included in the distance measuring device according to the second modification.

[0058] 6A, the number threshold calculation unit 16b receives temperature information indicating the temperature of the ranging device as a signal dependent on offset noise, calculates a number threshold based on the received temperature information, and notifies the calculated number threshold to the offset noise removal unit 15a. The temperature information is information indicating the temperature of the ranging device obtained from a temperature sensor (not shown), such as a thermistor, provided in the ranging device.

[0059] More specifically, number threshold calculation unit 16b internally stores data indicating the relationship (i.e., negative correlation) between the temperature indicated by the temperature information and the number threshold, as shown in (b) of Figure 6. When temperature information is input, number threshold calculation unit 16b refers to the relationship indicated by the internally stored data to determine a number threshold that decreases as the temperature indicated by the input temperature information increases, and notifies offset noise removal unit 15a of this number threshold. Then, offset noise removal unit 15a uses the number threshold notified by number threshold calculation unit 16b to determine whether to replace the target distance value with an invalid value (S14 of Figure 4B).

[0060] In the example shown in (c) of Figure 6, the distance value of interest is not replaced with an invalid value (No in S12 of Figure 4B), so the number of invalid values ​​among the distance values ​​of the surrounding pixels is counted (in this example, it is counted as 13) (S13 of Figure 4B), and the counted number of invalid values ​​(i.e., 13) is compared with the number threshold (e.g., 6) notified by the number threshold calculation unit 16b (S14 of Figure 4B).As a result, since the number of invalid values ​​(i.e., 13) exceeds the number threshold (i.e., 6) (Yes in S14 of Figure 4B), the distance value of interest is replaced with an invalid value (S15 of Figure 4B).

[0061] In this way, the offset noise removal unit 15a and number threshold calculation unit 16b provided in the distance measuring device of this modified example count the number of invalid values, which are distance values ​​that do not satisfy specified conditions regarding reliability, among the distance values ​​of pixels located around the pixel of the distance value of interest, and if the number of invalid values ​​obtained exceeds a number threshold determined depending on the temperature information (i.e., with a negative correlation), the distance value of interest is replaced with an invalid value without performing spatial filtering processing.

[0062] As a result, in the spatial filtering process for removing offset noise, when the number of surrounding invalid values ​​is compared with the number threshold, the higher the temperature of the ranging device, i.e., the greater the possibility of offset noise, the smaller the number threshold (i.e., the higher the probability) is for the target distance value to be replaced with an invalid value (i.e., excluded as a distance value) and the distance image is generated. Therefore, compared to conventional techniques that can remove noise due to the influence of false distance values, the overall reliability of the obtained distance image is suppressed from decreasing, and high accuracy is maintained as a ranging device.

[0063] Next, a distance measuring device according to a third modified embodiment will be described. The distance measuring device according to the third modified embodiment basically includes the components included in the distance measuring device according to the first modified embodiment described using Fig. 5, but differs from the first modified embodiment in that it includes an offset noise removal unit 15b instead of the offset noise removal unit 15a (see Fig. 7A). The following description will focus on the differences from the first modified embodiment.

[0064] 7A is a diagram illustrating characteristic components of a distance measuring device according to a third modification of the embodiment. More specifically, (a) of FIG. 7A is a block diagram illustrating, together with input and output information, an offset noise removal unit 15b and a number threshold calculation unit 16a included in the distance measuring device according to the third modification. (b) of FIG. 7A is a graph illustrating the relationship between the level of the offset signal input to the number threshold calculation unit 16a (i.e., the offset signal level) and the number threshold output from the number threshold calculation unit 16a.

[0065] 7A(a), offset noise removal unit 15b basically has the same function as offset noise removal unit 15a according to the first modified embodiment in that it determines when to replace a distance value of interest with an invalid value using the number threshold calculated by number threshold calculation unit 16a, but differs from offset noise removal unit 15a according to the first modified embodiment in that it determines when to replace a distance value of interest with an invalid value using not only the current frame but also past frames. More specifically, when the number of corresponding pixels that have invalid values ​​in the current frame and also in the past frames exceeds the number threshold given by number threshold calculation unit 16a, offset noise removal unit 15b changes the distance value of the pixel to an invalid value and outputs the invalid value.

[0066] 7B is a diagram showing distance values ​​(including invalid values) that are the subject of spatial filtering processing, for explaining the operation of the offset noise removal unit 15b and the number threshold calculation unit 16a provided in the distance measuring device according to the third modification. More specifically, (a) in FIG. 7B shows an example of distance values ​​in a past frame, (b) in FIG. 7B shows an example of corresponding distance values ​​in a current frame, and (c) in FIG. 7B shows pixels that are invalid in both the past frame and the current frame. The numerical value in (c) in FIG. 7B indicates the count (here, 1 to 12, a total of 12) of pixels that are invalid in both the past frame and the current frame.

[0067] In the example shown in FIG. 7B , the distance value of interest has not been replaced with an invalid value (No in S12 in FIG. 4B ), so the number of invalid values ​​among the distance values ​​of the surrounding pixels is counted (S13 in FIG. 4B ). At this time, the offset noise removal unit 15 b counts the number of corresponding pixels ((b) in FIG. 7B ) that were invalid in the past frame ((a) in FIG. 7B ) and that are invalid in the current frame ((c) in FIG. 7B ). In the example shown in (c) in FIG. 7B , a total of 12 pixels are counted. The offset noise removal unit 15 b then compares the counted number (12 in this example) with a number threshold (e.g., 10) provided by the number threshold calculation unit 16 a (S14 in FIG. 4B ). As a result, the number of invalid values ​​(i.e., 12) exceeds the number threshold (i.e., 10) (Yes in S14 in FIG. 4B ), so the distance value of interest is replaced with an invalid value (S15 in FIG. 4B ).

[0068] In this way, in this modification, when determining whether to replace a distance value of interest with an invalid value in spatial filtering processing for removing offset noise, not only the current frame but also past frames are used, and the number of surrounding invalid values ​​is compared with a number threshold. Therefore, the distance value of interest is replaced with an invalid value depending not on instantaneous offset noise but on the occurrence status of offset noise in consecutive frames, so that deterioration in reliability is more suppressed than in the embodiment, and higher accuracy is maintained as a distance measuring device.

[0069] In this modified example, the number threshold calculation unit 16a according to the first modified example, which calculates the number threshold based on an offset signal, is used as the number threshold calculation unit that determines the number threshold. However, instead of this, the number threshold calculation unit 16b according to the second modified example, which calculates the number threshold based on temperature information, may be used.

[0070] In this modification, the current frame and past frames are used in determining whether to replace the distance value of interest with an invalid value, but the number of past frames used is not limited to one, and for example, the current frame and the three most recent past frames may be used. The same applies to the time filtering process, and for example, the time filtering process may be performed on the current frame and the three most recent past frames.

[0071] As described above, the distance measuring device 10 according to the embodiment is a distance measuring device for measuring the distance to the subject 20, and includes a light source 11 that emits irradiation light 21 toward the subject 20 in accordance with an input light emission pulse, a light receiving unit 12 having a plurality of pixels arranged two-dimensionally that receives light including reflected light 22 from the subject 20 during an exposure period determined by the input exposure pulse, a drive control unit 13 that drives the light source 11 by outputting a light emission pulse to the light source 11 and drives the light receiving unit 12 by outputting an exposure pulse to the light receiving unit 12, and a drive control unit 13 that calculates the amount of reflected light 22 received by the light receiving unit 12 from the time the irradiation light 21 is emitted from the light source 11 until the time the reflected light 22 is received by the light receiving unit 12 for each of the plurality of pixels, based on the amount of reflected light 22 received by the light receiving unit 12. The apparatus includes a distance calculation unit 14 that calculates a depth corresponding to the time difference between the above two pixels as a distance value to the subject 20, and an offset noise removal unit 15 that generates a distance image from which offset noise caused by offset signals, which are signal components excluding reflected light 22, has been removed by applying a filter process to the distance value calculated by the distance calculation unit 14 for each of a plurality of pixels using the distance values ​​of pixels located around the pixel in question, and the offset noise removal unit 15 counts the number of invalid values, which are distance values ​​that do not satisfy predetermined conditions regarding reliability, among the distance values ​​of pixels located around the pixel in question, and if the number of invalid values ​​obtained exceeds a number threshold, changes the distance value of the pixel in question to an invalid value and outputs it.

[0072] As a result, in the spatial filtering process for removing offset noise, distance values ​​of interest that have many invalid values ​​in many surrounding pixels are replaced with invalid values ​​(i.e., excluded as distance values) and then spatial filtering process is performed to generate a distance image.Therefore, compared to conventional technologies that can remove noise due to the influence of false distance values, the overall reliability of the obtained distance image is suppressed from decreasing, and high accuracy is maintained for the distance measuring device 10.

[0073] Furthermore, in the third modification of the embodiment, the offset noise removal unit 15b performs filtering using distance values ​​of pixels located around a pixel in the current frame as well as distance values ​​of corresponding pixels in past frames. As a result, in the spatial filtering process for removing offset noise, when determining whether to replace a distance value of interest with an invalid value, not only the current frame but also past frames are used, and the number of surrounding invalid values ​​is compared with a number threshold. Therefore, the distance value of interest is replaced with an invalid value depending on the long-term occurrence status of offset noise in consecutive frames, rather than on instantaneous offset noise, so that a decrease in reliability is further suppressed, and higher accuracy is maintained as a distance measuring device.

[0074] More specifically, when the number of pixels that have invalid values ​​in the current frame and in the past frame exceeds a count threshold, the offset noise removal unit 15 b changes the distance value of the pixel to an invalid value and outputs the invalid value. This makes it possible to remove offset noise using the current frame and the past frame with a simple configuration.

[0075] In the embodiment, the predetermined condition for reliability is that the net amount of received light excluding the offset signal component exceeds the light-receiving threshold value, so that distance values ​​that do not meet the certain amount of received light are treated as invalid values.

[0076] In the embodiment, the number threshold is a fixed value, which allows a simple configuration to determine whether to replace a distance value of interest with an invalid value.

[0077] Furthermore, in the first to third modified examples of the embodiment, the distance measuring device further includes a number threshold calculation unit 16a or the like that calculates a number threshold based on a signal that depends on offset noise, and the offset noise removal unit 15a or the like changes the distance value of the pixel in question to an invalid value and outputs the invalid value when the number of invalid values ​​exceeds the number threshold calculated by the number threshold calculation unit 16a or the like. As a result, in the spatial filtering process for removing offset noise, when the number of surrounding invalid values ​​is compared with the number threshold, the distance value of interest is replaced with an invalid value (i.e., excluded as a distance value) using the number threshold that depends on offset noise. Therefore, because the distance value of interest is replaced with an invalid value depending on the magnitude of the offset noise, a decrease in the reliability of the obtained distance image as a whole can be more effectively suppressed compared to when a fixed number threshold is used.

[0078] Furthermore, in the first modified example of the embodiment, number threshold calculation unit 16a acquires an offset signal that indicates the amount of offset noise in the amount of light received by light receiving unit 12 as a signal that depends on offset noise, and calculates the number threshold depending on the offset signal so that the larger the amount of offset noise indicated by the acquired offset signal, the smaller the number threshold becomes. As a result, the larger the offset noise, the more likely it is that the distance value of interest will be replaced with an invalid value, thereby ensuring the reliability of the obtained distance image as a whole.

[0079] In the second modification of the embodiment, the number threshold calculation unit 16 b acquires temperature information indicating the temperature of the distance measuring device as a signal dependent on offset noise, and calculates the number threshold dependent on the temperature information so that the higher the temperature indicated by the acquired temperature information, the smaller the number threshold becomes. As a result, the higher the temperature of the distance measuring device, that is, the greater the offset noise in the environment, the greater the possibility that the distance value of interest will be replaced with an invalid value, thereby ensuring the reliability of the obtained distance image as a whole.

[0080] Furthermore, the offset noise removal method according to the embodiment is a method for removing offset noise caused by an offset signal, which is a signal component excluding reflected light 22, in a distance measuring device that measures the distance to an object 20, and the distance measuring device 10 includes a light source 11 that emits irradiation light 21 toward the object 20 in accordance with an input light emission pulse, a light receiving unit 12 having a plurality of pixels arranged two-dimensionally that receives light including reflected light 22 from the object 20 during an exposure period determined by the input exposure pulse, a drive control unit 13 that drives the light source 11 by outputting a light emission pulse to the light source 11 and drives the light receiving unit 12 by outputting an exposure pulse to the light receiving unit 12, and a light detection unit 13 that calculates an offset noise based on the amount of reflected light 22 received by the light receiving unit 12 for each of the plurality of pixels. and a distance calculation unit 14 that calculates, as a distance value to the subject 20, a depth corresponding to the time difference between when the irradiated light 21 is emitted from the light source 11 and when the reflected light 22 is received by the light receiving unit 12. The offset noise removal method includes a noise removal step that generates a distance image from which the offset noise has been removed by applying a filter process to the distance value calculated by the distance calculation unit 14 for each of a plurality of pixels using the distance values ​​of pixels located around the pixel in question. In the noise removal step, the number of invalid values, which are distance values ​​that do not satisfy predetermined conditions regarding reliability, among the distance values ​​of pixels located around the pixel in question are counted, and if the number of invalid values ​​obtained exceeds a number threshold, the distance value of the pixel in question is changed to an invalid value and output.

[0081] As a result, in the spatial filtering process for removing offset noise, distance values ​​of interest that have many invalid values ​​in many surrounding pixels are replaced with invalid values ​​(i.e., excluded as distance values) and then spatial filtering process is performed to generate a distance image.Therefore, compared to conventional technologies that can remove noise due to the influence of false distance values, the overall reliability of the obtained distance image is suppressed from decreasing, and high accuracy is maintained for the distance measuring device 10.

[0082] The distance measuring device and offset noise removal method according to the present disclosure have been described above based on the embodiments and modifications, but the present disclosure is not limited to these embodiment modifications. As long as they do not deviate from the gist of the present disclosure, various modifications that would occur to a person skilled in the art to the present embodiment and modifications, and other forms constructed by combining some of the components in the embodiments and modifications, are also included within the scope of the present disclosure.

[0083] For example, in the embodiment, when determining whether to replace a distance value of interest with an invalid value, if the number of invalid values ​​does not exceed a number threshold, spatial filtering is performed on the distance value of interest.However, in addition to such a filter, spatial filtering may also be performed by applying temporal filtering using past frames.

[0084] Furthermore, the offset noise removal method according to the present disclosure may be realized as a program or program product that causes a computer to execute the steps included in the offset noise removal method, as a non-transitory computer-readable recording medium on which the program is recorded, or as an offset noise removal device that includes a processor, memory, etc. that executes the steps included in the offset noise removal method.

[0085] The present disclosure can be used as a TOF-type distance measuring device having the function of removing offset noise, particularly as a distance measuring device that can suppress a decrease in the overall reliability of the obtained distance image even when offset noise occurs, for example, as a distance measuring device used in applications such as face recognition, driver monitoring, and people flow detection.

[0086] REFERENCE SIGNS LIST 10 Distance measuring device 11 Light source 12 Light receiving unit 13 Drive control unit 14 Distance calculation unit 15, 15a, 15b Offset noise removal unit 16a, 16b Number threshold calculation unit 20 Object 21 Irradiation light 22 Reflected light

Claims

1. A distance measuring device for measuring the distance to an object, comprising: a light source that emits illumination light toward the object in accordance with an input illumination pulse; a light receiving unit having a plurality of pixels arranged two-dimensionally that receive light including reflected light from the object during an exposure period determined by the input exposure pulse; a drive control unit that drives the light source by outputting the illumination pulse to the light source and drives the light receiving unit by outputting the exposure pulse to the light receiving unit; a distance calculation unit that calculates, for each of the plurality of pixels, a depth corresponding to the time difference between when illumination light is emitted from the light source and when the reflected light is received by the light receiving unit, as a distance value to the object, based on the amount of reflected light received by the light receiving unit; and an offset noise removal unit that generates a distance image from which offset noise caused by offset signals, which are signal components excluding reflected light, has been removed by filtering the distance value calculated by the distance calculation unit for each of the plurality of pixels using distance values ​​of pixels located around the pixel in question, The offset noise removal unit counts the number of invalid values, which are distance values ​​that do not satisfy predetermined conditions regarding reliability, among the distance values ​​of pixels located around the pixel in question, and if the number of the obtained invalid values ​​exceeds a number threshold, changes the distance value of the pixel in question to an invalid value and outputs it.

2. The distance measuring device according to claim 1, wherein the offset noise removal unit performs the filtering process using distance values ​​of pixels located around the pixel in question in the current frame as well as distance values ​​of corresponding pixels in previous frames.

3. The distance measuring device according to claim 2, wherein when the number of pixels that are invalid in the current frame and invalid in the past frame exceeds the number threshold, the offset noise removal unit changes the distance value of the pixel to an invalid value and outputs the invalid value.

4. The distance measuring device according to claim 1, wherein the predetermined condition regarding reliability is that the net amount of received light excluding the offset signal component exceeds a light receiving threshold value.

5. The distance measuring device according to any one of claims 1 to 4, wherein the number threshold is a fixed value.

6. A distance measuring device according to any one of claims 1 to 4, further comprising a number threshold calculation unit that calculates the number threshold based on a signal that depends on offset noise, and wherein when the number of invalid values ​​exceeds the number threshold calculated by the number threshold calculation unit, the offset noise removal unit changes the distance value of the pixel to an invalid value and outputs it.

7. A distance measuring device as described in claim 6, wherein the number threshold calculation unit acquires an offset signal indicating the amount of offset noise in the amount of light received by the light receiving unit as a signal dependent on the offset noise, and calculates the number threshold dependent on the offset signal so that the number threshold becomes smaller as the amount of offset noise indicated by the acquired offset signal increases.

8. The distance measuring device of claim 6, wherein the number threshold calculation unit acquires temperature information indicating the temperature of the distance measuring device as a signal dependent on the offset noise, and calculates the number threshold dependent on the temperature information so that the higher the temperature indicated by the acquired temperature information, the smaller the number threshold becomes.

9. An offset noise removal method for removing offset noise caused by an offset signal, which is a signal component excluding reflected light, in a distance measuring device that measures the distance to a subject, wherein the distance measuring device comprises: a light source that emits illumination light toward the subject in accordance with an input light emission pulse; a light receiving unit having a plurality of pixels arranged two-dimensionally that receives light including reflected light from the subject during an exposure period determined by an input exposure pulse; a drive control unit that drives the light source by outputting the light emission pulse to the light source and drives the light receiving unit by outputting the exposure pulse to the light receiving unit; and a distance calculation unit that calculates, for each of the plurality of pixels, a depth corresponding to the time difference between when illumination light is emitted from the light source and when the reflected light is received by the light receiving unit, as a distance value to the subject, based on the amount of reflected light received by the light receiving unit; and the offset noise removal method includes a noise removal step of generating a distance image from which the offset noise has been removed by filtering the distance value calculated by the distance calculation unit for each of the plurality of pixels using distance values ​​of pixels located around the pixel in question, In the noise removal step, the number of invalid values, which are distance values ​​that do not satisfy a predetermined condition regarding reliability, among the distance values ​​of pixels located around the pixel in question is counted, and if the number of the obtained invalid values ​​exceeds a number threshold, the distance value of the pixel in question is changed to an invalid value and output.

Images