Data processing apparatus, range imaging apparatus, and edge detection method

Abstract

To appropriately perform edge detection in a distance image.SOLUTION: The data processing device includes an acquisition unit that acquires a distance image based on a distance to an object present in a measurement target space, and an edge detection unit that performs edge detection on each pixel of the distance image based on a ratio of distance values of the pixel or a plurality of pixels around the pixel.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present invention relates to a data processing device, a distance image capturing device, and an edge detection method. [Background technology]

[0002] Taking advantage of the fact that the speed of light is known, a time-of-flight (hereinafter referred to as "TOF") type range imaging device has been realized that measures the distance between a measuring device and an object based on the time of flight of light in space (measurement space) (see, for example, Patent Document 1). In such a range imaging device, the delay time from the time a light pulse is emitted until the light pulse is reflected from the object and returns is found by accumulating charges generated by a photoelectric conversion element in multiple charge storage units, and the distance to the object is calculated using the delay time and the speed of light.

[0003] In range images obtained by such range imaging devices, if the reflected light from a front target and a rear target enters a single pixel at the edges of multiple targets (measurement objects) located at different distances, a flying pixel (FP) may occur. A common method is to detect edges by filtering based on the difference between adjacent pixels using a differential filter such as a Sobel filter, set a threshold for the edge size, and remove pixels with edges greater than the threshold as FPs. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 4235729 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in the edge detection based on the difference between adjacent pixels as described above, there is a problem that if the threshold is set small, FPs at small edges can be removed, but unnecessary invalid pixels other than FPs increase, and if the threshold is set large, unnecessary invalid pixels are reduced, but FPs at small edges remain. Therefore, a more appropriate method of edge detection is desired.

[0006] The present invention has been made in consideration of the above-mentioned problems, and one of its objects is to provide a data processing device, a distance image capturing device, and an edge detection method that can perform appropriate edge detection in a distance image. [Means for solving the problem]

[0007] A data processing device according to one aspect of the present invention includes an acquisition unit that acquires a distance image based on the distance to an object present in the space to be measured, and an edge detection unit that detects edges for each pixel of the distance image based on the ratio of the distance values ​​of the pixel or any multiple pixels surrounding the pixel.

[0008] In addition, a distance image capturing device according to one aspect of the present invention comprises a light receiving unit having a light source unit that irradiates a space to be measured with a light pulse, a pixel having a photoelectric conversion element that generates an electric charge in response to the incident light and a plurality of charge accumulation units that accumulate the electric charge, and a pixel driving circuit that allocates and accumulates the electric charge in each of the charge accumulation units in the pixel at a predetermined timing synchronized with the irradiation of the light pulse, a distance calculation unit that calculates the distance to an object in the space based on the amount of electric charge accumulated in each of the charge accumulation units to obtain a distance image, and an edge detection unit that detects edges for each pixel of the distance image based on the ratio of the distance values ​​of the pixel or any plurality of pixels surrounding the pixel.

[0009] Furthermore, a method for detecting edges of a distance image in a data processing device according to one aspect of the present invention includes a step in which an acquisition unit acquires a distance image based on the distance to an object present in the space to be measured, and a step in which an edge detection unit detects edges for each pixel of the distance image based on the ratio of the distance values ​​of the pixel or any two or more pixels surrounding the pixel. [Effects of the Invention]

[0010] According to the present invention, edge detection can be performed appropriately in a range image. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a block diagram showing a schematic configuration of a distance imaging device according to an embodiment. [Figure 2] 1 is a block diagram showing a schematic configuration of a range image sensor according to an embodiment. [Figure 3] FIG. 2 is a circuit diagram showing an example of the configuration of a pixel of the range image sensor according to the embodiment. [Figure 4] FIG. 1 is an explanatory diagram of the principle of occurrence of flying pixels (FPs). [Figure 5] FIG. 1 is a diagram illustrating an example of a flying pixel (FP). [Figure 6] FIG. 1 is an explanatory diagram of an edge detection method using a Sobel filter. [Figure 7] FIG. 10 is a diagram illustrating an example of removing flying pixels (FPs). [Figure 8] FIG. 1 is a diagram showing an example of the result of removing flying pixels (FPs) using a conventional edge detection method. [Figure 9] FIG. 1 is an explanatory diagram of a first example of a trade-off mechanism. [Figure 10] FIG. 10 is an explanatory diagram of a second example of a trade-off mechanism. [Figure 11] FIG. 1 is an explanatory diagram illustrating a basic concept of edge detection according to an embodiment. [Figure 12] 10 is a flowchart illustrating an example of edge detection processing according to the embodiment. [Figure 13] 10A to 10C are diagrams illustrating specific calculation examples of edge detection processing according to the embodiment. [Figure 14] FIG. 10 is a diagram showing an example of a result of removing flying pixels (FPs) by edge detection according to the embodiment. [Figure 15] 10A and 10B are diagrams illustrating another example of calculation of the edge detection processing according to the embodiment. [Figure 16] FIG. 2 is a diagram illustrating an example of a differential filter according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, the distance imaging device of this embodiment will be described with reference to the drawings.

[0013] Fig. 1 is a block diagram showing the schematic configuration of a distance imaging device according to this embodiment. The distance imaging device 1 is a distance imaging device that measures (range-finding) the distance to an object using the TOF method, and includes, for example, a light source unit 2, a light receiving unit 3, and a distance image processing unit 4. Fig. 1 also shows an object OB (subject), which is the object to which the distance is to be measured by the distance imaging device 1.

[0014] The light source unit 2 irradiates the space to be measured with a light pulse PO under the control of the distance image processor 4. The light source unit 2 is, for example, a surface-emitting semiconductor laser module such as a vertical cavity surface-emitting laser (VCSEL). The light source unit 2 includes a light source device 21 and a diffuser plate 22.

[0015] Light source device 21 is a light source that emits laser light in the near-infrared wavelength band (for example, a wavelength band of 850 nm to 940 nm) that becomes light pulses PO to be irradiated into the space to be measured. Light source device 21 is, for example, a semiconductor laser light-emitting element. Light source device 21 emits pulsed laser light in response to control from distance image processing unit 4.

[0016] The diffusion plate 22 is an optical component that diffuses the laser light in the near-infrared wavelength band emitted by the light source device 21 to the extent of the surface that is irradiated into the space to be measured. The pulsed laser light diffused by the diffusion plate 22 is emitted as a light pulse PO and is irradiated into the space to be measured.

[0017] When an object OB is present in a space to be measured for distance measurement in the distance image pickup device 1, the light receiving unit 3 receives reflected light RL of the light pulse PO that is reflected by the object OB after irradiating the light pulse PO from the light source unit 2, and outputs a pixel signal corresponding to the received reflected light RL. The light receiving unit 3 includes a lens 31 and a distance image sensor 32.

[0018] The lens 31 is an optical lens that guides the incident reflected light RL to the range image sensor 32. The lens 31 outputs the incident reflected light RL to the range image sensor 32 side, and causes the light to be received (incident) by pixels provided in the light receiving region of the range image sensor 32.

[0019] The range image sensor 32 is an imaging element used in the range image capturing device 1. The range image sensor 32 has a plurality of pixels in a two-dimensional light receiving area. Each pixel of the range image sensor 32 is provided with one photoelectric conversion element, a plurality of charge accumulation units corresponding to this one photoelectric conversion element, and components that distribute charge to each of the charge accumulation units. In other words, the pixel is an imaging element with a distribution configuration in which charge is distributed and stored in a plurality of charge accumulation units.

[0020] The range image sensor 32 distributes the charges generated by the photoelectric conversion elements to the respective charge accumulation sections under the control of the timing control section 41. The range image sensor 32 also outputs pixel signals according to the amount of charge distributed to the charge accumulation sections. The range image sensor 32 has multiple pixels arranged in a two-dimensional matrix, and outputs pixel signals for one frame corresponding to each pixel.

[0021] Here, the depth direction range (distance range) measurable in the space of the measurement target to measure the distance in the distance image capture device 1 is determined primarily by the light intensity of the light pulse PO emitted from the light source unit 2 and the light receiving sensitivity of the light receiving unit 3. The surface direction range measurable is determined by the irradiation angle (light spread) of the light pulse PO emitted from the light source unit 2 and the light receiving angle (angle at which light can be received) of the light receiving unit 3.

[0022] The distance image processing unit 4 functions as a data processing device that acquires a distance image based on the distance to an object present in the measurement space, performs edge detection on the acquired distance image, and removes FPs (flying pixels). For example, the distance image processing unit 4 controls the distance image capturing device 1 to calculate the distance to the object OB, acquires a distance image, performs edge detection on the acquired distance image, and removes FPs (flying pixels). For example, the distance image processing unit 4 includes a timing control unit 41, a distance calculation unit 42, a measurement control unit 43, an edge detection unit 44, and a noise reduction unit 45.

[0023] The timing control unit 41 controls the timing of outputting various control signals required for measurement in accordance with the control of the measurement control unit 43. The various control signals here include, for example, a signal that controls the irradiation of the light pulse PO, a signal that distributes and accumulates the reflected light RL in multiple charge accumulation units, and a signal that controls the number of accumulations per frame. The number of accumulations is the number of times that the process of distributing and accumulating electric charge in the charge accumulation units CS (see FIG. 3) is repeated. The accumulation time is the product of this number of accumulations and the time (accumulation time) for accumulating electric charge in each charge accumulation unit per process of distributing and accumulating electric charge.

[0024] The distance calculation unit 42 outputs distance information calculated based on the pixel signals output from the distance image sensor 32. The distance calculation unit 42 calculates the delay time from when the light pulse PO is emitted until when the reflected light RL is received based on the amount of charge accumulated in the multiple charge accumulation units. The distance calculation unit 42 calculates the distance to the object OB according to the calculated delay time and acquires a distance image.

[0025] The measurement control unit 43 controls the timing control unit 41. For example, the measurement control unit 43 sets the number of accumulations and accumulation time for one frame, and controls the timing control unit 41 so that imaging is performed according to the set contents.

[0026] With this configuration, in the distance image capturing device 1, the light source 2 irradiates an optical pulse PO in the near-infrared wavelength band onto the object OB, and the light receiving unit 3 receives the reflected light RL that is reflected by the object OB, and the distance image processing unit 4 outputs distance information that measures the distance to the object OB. For example, the distance image processing unit 4 outputs a distance image as distance information that measures the distance to the object OB.

[0027] The edge detection unit 44 detects edges based on the distance values ​​of multiple pixels in the distance image. For example, the edge detection unit 44 detects edges for each pixel in the distance image based on the ratio of the distance values ​​of the pixel or multiple pixels surrounding the pixel. Specifically, for example, the edge detection unit 44 detects edges using a threshold value for the calculated ratio value.

[0028] The noise reduction unit 45 removes flying pixels based on the edges detected by the edge detection unit.

[0029] Although Figure 1 shows a distance image capturing device 1 configured such that the distance image processing unit 4 is provided inside the distance image capturing device 1, the distance image processing unit 4 may also be a component provided outside the distance image capturing device 1.

[0030] Here, the configuration of the distance image sensor 32 used as an imaging element in the distance image pickup device 1 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing a schematic configuration of the imaging element (distance image sensor 32) used in the distance image pickup device 1 according to this embodiment.

[0031] As shown in FIG. 2, the distance image sensor 32 includes, for example, a light receiving area 320 in which a plurality of pixels 321 are arranged, a control circuit 322, a vertical scanning circuit 323 having a distribution operation, a horizontal scanning circuit 324, and a pixel signal processing circuit 325.

[0032] The light receiving area 320 is an area in which a plurality of pixels 321 are arranged, and FIG. 2 shows an example in which the pixels are arranged in a two-dimensional matrix of 8 rows and 8 columns. The pixels 321 accumulate electric charges corresponding to the amount of light they receive. The control circuit 322 comprehensively controls the range image sensor 32. The control circuit 322 controls the operation of the components of the range image sensor 32 in accordance with instructions from, for example, the timing control unit 41 of the range image processing unit 4. Note that the components of the range image sensor 32 may be directly controlled by the timing control unit 41, in which case the control circuit 322 may be omitted.

[0033] The vertical scanning circuit 323 is a circuit that controls the pixels 321 arranged in the light receiving region 320 for each row in accordance with control from the control circuit 322. The vertical scanning circuit 323 outputs a voltage signal corresponding to the amount of charge accumulated in each charge accumulation unit CS of the pixels 321 to the pixel signal processing circuit 325. In this case, the vertical scanning circuit 323 distributes and accumulates the charge converted by the photoelectric conversion element in each charge accumulation unit of the pixels 321. In other words, the vertical scanning circuit 323 is an example of a "pixel driving circuit."

[0034] The pixel signal processing circuit 325 is a circuit that performs predetermined signal processing (e.g., noise suppression processing, A / D conversion processing, etc.) on the voltage signals output from the pixels 321 in each column to the corresponding vertical signal lines in accordance with control from the control circuit 322.

[0035] Horizontal scanning circuit 324 is a circuit that sequentially outputs signals output from pixel signal processing circuit 325 to horizontal signal lines in accordance with control from control circuit 322. As a result, pixel signals corresponding to the amount of charge accumulated for one frame are sequentially output to distance image processing unit 4 via the horizontal signal lines.

[0036] In the following description, it is assumed that the pixel signal processing circuit 325 performs A / D conversion processing and the pixel signals are digital signals.

[0037] Here, the configuration of the pixel 321 arranged in the light receiving region 320 provided in the range image sensor 32 will be described with reference to Fig. 3. Fig. 3 is a circuit diagram showing an example of the configuration of the pixel 321 arranged in the light receiving region 320 of the range image sensor 32 according to this embodiment. Fig. 3 shows an example of the configuration of one pixel 321 out of the multiple pixels 321 arranged in the light receiving region 320. The pixel 321 is an example of a configuration including four pixel signal readout units.

[0038] The pixel 321 includes one photoelectric conversion element PD, a drain gate transistor GD, and four pixel signal readout units RU that output voltage signals from corresponding output terminals O. Each pixel signal readout unit RU includes a readout gate transistor G, a floating diffusion FD, a charge storage capacitance C, a reset gate transistor RT, a source follower gate transistor SF, and a select gate transistor SL. In each pixel signal readout unit RU, the floating diffusion FD and the charge storage capacitance C form a charge storage unit CS.

[0039] 3, the four pixel signal readout units RU are distinguished from one another by adding the numbers "1," "2," "3," or "4" after the symbol "RU" of each pixel signal readout unit RU. Similarly, the components of each of the four pixel signal readout units RU are distinguished from one another by adding the number representing each pixel signal readout unit RU after the symbol.

[0040] 3, the pixel signal readout unit RU1, which outputs a voltage signal from the output terminal O1, includes a readout gate transistor G1, a floating diffusion FD1, a charge storage capacitance C1, a reset gate transistor RT1, a source follower gate transistor SF1, and a select gate transistor SL1. In the pixel signal readout unit RU1, the floating diffusion FD1 and the charge storage capacitance C1 form a charge storage unit CS1. The pixel signal readout units RU2 to RU4 have a similar configuration.

[0041] The photoelectric conversion element PD is a buried photodiode that photoelectrically converts incident light to generate electric charges and accumulates the generated electric charges. The photoelectric conversion element PD may have any structure. For example, the photoelectric conversion element PD may be a PN photodiode having a structure in which a P-type semiconductor and an N-type semiconductor are joined together, or a PIN photodiode having a structure in which an I-type semiconductor is sandwiched between a P-type semiconductor and an N-type semiconductor. Furthermore, the photoelectric conversion element PD is not limited to a photodiode, and may be, for example, a photogate type photoelectric conversion element.

[0042] In pixel 321, the photoelectric conversion element PD photoelectrically converts incident light to generate electric charges, which are then distributed to each of the four charge accumulation units CS, and voltage signals corresponding to the amount of electric charge distributed are output to the pixel signal processing circuit 325.

[0043] The configuration of pixels arranged in the range image sensor 32 is not limited to the configuration including four pixel signal readout units RU as shown in Fig. 3, but may be any pixel configured to include multiple pixel signal readout units RU. In other words, the number of pixel signal readout units RU (charge accumulation units CS) provided in the pixels arranged in the range image sensor 32 may be two, three, five or more.

[0044] 3 shows an example in which the charge storage section CS is configured by a floating diffusion FD and a charge storage capacitance C. However, the charge storage section CS only needs to be configured by at least a floating diffusion FD, and the pixel 321 may not have a charge storage capacitance C.

[0045] Furthermore, in the pixel 321 having the configuration shown in FIG. 3, an example of a configuration including a drain gate transistor GD is shown, but if there is no need to discard the charge accumulated (remaining) in the photoelectric conversion element PD, the pixel 321 may have a configuration not including a drain gate transistor GD.

[0046] Next, the principle of occurrence of flying pixels (FP) present at the edge of an object OB in a range image will be described with reference to Fig. 4. Fig. 4 is an explanatory diagram of the principle of occurrence of flying pixels (FP). In the example shown in Fig. 4, a foreground object OB1 and a background object OB2 exist in the space to be measured by the range image pickup device 1. At this time, reflected light from the foreground object OB1 and reflected light from the background object OB2 enter one pixel of the range image sensor 32.

[0047] (a) shows the time-series timing of the incident light, reflected light, and gates. As shown in (a), reflected light from object OB1 in the foreground is incident first, with a delay from the timing of the incident light, followed by reflected light from object OB2 in the background. The two incident reflected lights are distributed to two gates, gate transistor G1 and gate transistor G2. As a result, as shown in (b), the signal amounts of the pixel signal from gate transistor G1 and the pixel signal from gate transistor G2 become closer and are averaged, generating distance information indicating the intermediate distance between objects OB1 and OB2 (distance information where no object actually exists), i.e., a flying pixel (FP).

[0048] An example of a flying pixel (FP) that occurs as distance information indicating the intermediate distance between object OB1 and object OB2 is shown in FIG. 5. FIG. 5 is a diagram showing an example of a flying pixel (FP). In FIG. 5, the vertical axis represents distance and the horizontal axis represents pixels, and the location where the flying pixel (FP) occurs in the distance image is shown. A flying pixel (FP) occurs at the edge between object OB1 in the foreground and object OB2 in the background.

[0049] Next, referring to Fig. 6 to Fig. 8, we will explain the problems with removing flying pixels (FPs) using conventional common edge detection methods. Fig. 6 is an explanatory diagram of an edge detection method using a Sobel filter. The Sobel filter is an example of a differential filter, and can detect edges by calculating the difference between adjacent pixels. Applying the Sobel filter to a range image enables edge detection and removal of flying pixels (FPs).

[0050] In Figure 6, kernel H is a Sobel filter in the horizontal direction, and kernel V is a Sobel filter in the vertical direction. By calculating gradient H, which is a product-sum operation of the coefficients of kernel H in the horizontal direction and the distance image, and gradient V, which is a product-sum operation of the coefficients of kernel V in the vertical direction and the distance image, respectively, Sobel filter processing in the vertical and horizontal directions is performed. Pixels with large calculated values ​​are edge parts, and pixels with small calculated values ​​are flat parts.

[0051] As shown in FIG. 7, flying pixels (FP) can be removed by determining the calculated value of the Sobel filter using a threshold and designating pixels above the threshold as invalid pixels. FIG. 7 is a diagram showing an example of removing flying pixels (FP). In FIG. 7, similar to FIG. 5, the vertical axis represents distance and the horizontal axis represents pixels, and (A) shows the calculated value of the Sobel filter for each pixel. By performing the Sobel filter processing shown in FIG. 6, the calculated value of the Sobel filter for pixels in edge areas (flying pixels (FP)) becomes high. For example, by invalidating pixels whose calculated value of the Sobel filter is above a threshold of 20, flying pixels (FP) can be removed as shown in (B).

[0052] However, in the removal of flying pixels (FP) using the conventional edge detection method described with reference to Figures 6 and 7, there is a problem in that, depending on the threshold setting, there is a trade-off between the removal of flying pixels (FP) and unnecessary invalid pixels other than flying pixels (FP) (loss of distance information of actually existing objects).

[0053] FIG. 8 shows an example of the results of removing flying pixels (FPs) using a conventional edge detection method. In the example shown in FIG. 8, a foreground object OB1 and a background object OB2 exist as two plates in the measurement target space, with the distance between them being approximately 20 cm. The example of a distance image shown in (A) is an example when the threshold for removing flying pixels (FPs) is small (e.g., 100). The example of a distance image shown in (B) is an example when the threshold for removing flying pixels (FPs) is large (e.g., 200). In both (A) and (B), the left side shows an example of a 2D display of the distance image, and the right side shows an example of a 3D display that makes the edges easier to see.

[0054] As shown in (A), when the threshold is set small, flying pixels (FP) can be removed from small edges, but unnecessary invalid pixels (black) other than the flying pixels (FP) increase in the background, etc. On the other hand, as shown in (B), when the threshold is set large, unnecessary invalid pixels (black) decrease, but flying pixels (FP) from small edges remain. In this way, depending on the threshold setting, a trade-off occurs between the removal of flying pixels (FP) and unnecessary invalid pixels other than the flying pixels (FP). The mechanism of this trade-off will be explained with reference to FIGS. 9 and 10.

[0055] Figure 9 is an explanatory diagram of a first example of the trade-off mechanism. The amount of reflected light varies depending on the distance of an object in the space being measured, so the signal amount of the pixel signal varies depending on the distance of the object. Objects at long distances have a small signal amount and therefore a large amount of noise, while objects at short distances have a large signal amount and therefore a small amount of noise. Because objects at long distances have a large amount of noise, if the threshold value of the Sobel filter is set small, they will be considered edges and determined to be invalid pixels. If the threshold value is increased in an attempt to make pixels of objects at long distances valid, small edge flying pixels (FP) will remain.

[0056] FIG. 10 is an explanatory diagram of a second example of the trade-off mechanism. As shown in the measurement example in (A), suppose that an object OB1 in the foreground (close distance) and an object OB2 in the background (far distance) exist with the same tilt. Here, since the far distance object occupies fewer pixels in the angle of view, the difference in distance between adjacent pixels is different, as shown in (B), even if the tilt is the same. Specifically, even if the tilt is the same, the difference in distance between adjacent pixels is larger for a far distance object compared to a near distance object (Δd far >Δd near Therefore, if a distant object has a tilt, it may be mistaken for an edge and removed as a flying pixel (FP) depending on the threshold setting.

[0057] Next, the edge detection process executed by the range image capturing device 1 according to this embodiment will be described. FIG. 11 is an explanatory diagram showing the basic concept of edge detection according to this embodiment. In FIG. 11, (A) shows conventional edge detection for comparison, and (B) shows edge detection according to this embodiment. For example, in the conventional edge detection of (A), in an example where three pixels D(-1,0), D(0,0), and D(1,0) are lined up, the difference ΔD between adjacent pixels is calculated by "ΔD=|D(1,0)-D(-1,0)|". If the threshold value is D th Then, ΔD is D thEdge determination is performed based on whether ΔD is greater than Dth or ΔD≦Dth. In the case of edge detection based on the difference between adjacent pixels, as described with reference to Figures 9 and 10, the edge determination is affected by noise, which can cause edge confusion depending on whether the distance is long or short, and the distance difference between adjacent pixels due to tilt, so edge detection may not be performed properly.

[0058] Therefore, in the edge detection according to this embodiment, the ratio of adjacent pixels is used instead of the difference between adjacent pixels, thereby making the judgment level uniform between long distances and short distances. For example, in the method of detecting edges based on the ratio of adjacent pixels (B), in an example where three pixels D(-1,0), D(0,0), and D(1,0) are lined up, the ratio ΔR of adjacent pixels is calculated by "ΔR=D(1,0) / D(-1,0)". If the threshold is R th Then, ΔR is R th Edge determination is performed based on whether ΔR is greater than Rth or ΔR≦Rth. By using the ratio of adjacent pixels, the distance difference between adjacent pixels due to noise and tilt, which can cause edge confusion depending on whether the distance is long or short, can be adjusted, thereby reducing erroneous edge determination.

[0059] Next, a specific example of edge detection processing according to this embodiment will be described with reference to Figs. 12 and 13. Fig. 12 is a flowchart showing an example of edge detection processing according to this embodiment. Fig. 13 is a diagram showing a specific calculation example of edge detection processing according to this embodiment.

[0060] (Step S101) Distance image processing unit 4 takes the logarithm of distance image D (see FIG. 13(1)) (see FIG. 13(2) logarithmic distance image log(D)), and then proceeds to step S103.

[0061] (Step S103) Distance image processing unit 4 performs filtering on the logarithmic distance image log(D) using a Sobel filter in the vertical and horizontal directions (see (3) in FIG. 13), and then proceeds to step S105.

[0062] (Step S105) The distance image processing unit 4 removes the logarithm of the vertical and horizontal Sobel filter calculation values ​​(difference values) with the larger absolute value (see (4) in FIG. 13) and sets the ratio value (see (5) in FIG. 13). The vertical and horizontal Sobel filter calculation values ​​(difference values) with the larger absolute value are set as the Sobel filter composite value, which is calculated using the following Equation 1:

[0063]

number

[0064] Furthermore, the ratio value R obtained by removing the logarithm of the composite value of the Sobel filter is calculated by the following formula 2.

[0065]

number

[0066] A ratio image R based on ratio values ​​calculated by applying a Sobel filter to the logarithmic distance image log(D) to remove the logarithm is shown in (6) of Fig. 13. Then, the process proceeds to step S107.

[0067] (Step S107) The distance image processing unit 4 calculates the ratio R by dividing the ratio R by the threshold R th For example, distance image processing unit 4 performs edge determination using the following equation 3.

[0068]

number

[0069] (Step S109) Based on the determination result of step S107, the distance image processing unit 4 determines whether the ratio value R is greater than or equal to the threshold value R th The pixel with the larger value is determined to be an edge pixel and set to "1" (see (7) in FIG. 13). Then, the distance image processing unit 4 executes invalidation processing to invalidate the pixel determined to be an edge pixel (the pixel set to "1") (see (8) in FIG. 13).

[0070] (Step S111) Based on the determination result of step S107, the distance image processing unit 4 determines whether the ratio value R is greater than or equal to the threshold value R th The following pixels are determined not to be edge pixels and are set to "0" (see (7) in FIG. 13). Then, distance image processing unit 4 executes a validation process to validate the pixels determined not to be edge pixels (pixels set to "0") (see (8) in FIG. 13).

[0071] 14 shows an example of the results of flying pixel (FP) removal using edge detection according to this embodiment. In FIG. 14, (A) shows the conventional result ((A) in FIG. 8) for comparison. The example of a distance image shown in (B) is an example of the results of edge detection based on the ratio of adjacent pixels according to this embodiment, in which flying pixels (FP) have been removed and there are few unnecessary invalid pixels (black). In other words, by using the edge detection according to this embodiment, it is possible to remove flying pixels (FP) and suppress unnecessary invalid pixels at the same time.

[0072] As described above, the distance image capture device 1 according to this embodiment includes a light source 2, a light receiving unit 3, and a distance image processor 4. The light source 2 irradiates a space to be measured with a light pulse PO. The light receiving unit 3 includes pixels 321 each having a photoelectric conversion element PD that generates an electric charge in response to the incident light and multiple charge storage units CS that store the electric charge. A vertical scanning circuit 323 (an example of a pixel drive circuit) distributes and stores the electric charge in each of the charge storage units CS of the pixels 321 at a predetermined timing synchronized with the irradiation of the light pulse PO. The distance image processor 4 includes a distance calculation unit 42 that calculates the distance to an object OB in space based on the amount of electric charge stored in each charge storage unit CS to acquire a distance image, and an edge detection unit 44 that performs edge detection for each pixel in the distance image based on the ratio of the distance values ​​of the pixel or any of multiple pixels surrounding the pixel.

[0073] As a result, the distance imaging device 1 performs edge detection using the ratio of adjacent pixels in the distance image, and therefore can suppress erroneous edge determination by aligning the distance difference between adjacent pixels due to noise and tilt, which can cause edge confusion depending on whether the distance is long or short, thereby enabling appropriate edge detection.In other words, the distance imaging device 1 can acquire a distance image with appropriate edge detection.

[0074] In other words, the distance image processor 4 according to this embodiment is an example of a data processing device that acquires a distance image based on the distance to an object present in the measurement space. For each pixel in the distance image, the distance image processor 4 detects edges based on the ratio of distance values ​​of the pixel or any two or more pixels surrounding the pixel. For example, the distance image processor 4 detects edges in the distance image based on the ratio of distance values ​​of any two or more pixels within an area to be filtered using a Sobel filter (an example of a differential filter).

[0075] As a result, the distance image processing unit 4 performs edge detection using the ratio of adjacent pixels in the distance image, and therefore can suppress erroneous edge determination by aligning the distance differences between adjacent pixels due to noise and tilt, which can cause edge confusion depending on whether the distance is long or short, thereby enabling appropriate edge detection.

[0076] For example, distance image processing unit 4 uses logarithms when calculating the ratio of the distance values ​​of each pixel in the distance image to the pixel itself or any of a plurality of pixels surrounding the pixel.

[0077] This allows distance image processing unit 4 to calculate the ratio of adjacent pixels in the distance image using logarithms and perform edge detection based on the calculated ratio, thereby enabling appropriate edge detection.

[0078] Specifically, for example, when calculating the ratio of the distance values ​​of each pixel in the distance image to the pixel itself or multiple pixels surrounding the pixel, the distance image processing unit 4 calculates the ratio of the distance values ​​of the multiple pixels by applying a Sobel filter (an example of a differential filter) to the logarithms of the distance values ​​of the multiple pixels and then removing the logarithms.

[0079] This allows the distance image processing unit 4 to calculate the ratio of adjacent pixels in the distance image using a logarithm and a Sobel filter (an example of a differential filter), and perform edge detection based on the calculated ratio, thereby enabling appropriate edge detection.

[0080] For example, distance image processing unit 4 detects edges using a threshold value for the calculated ratio value.

[0081] This allows the distance image processing unit 4 to perform appropriate edge detection by adjusting the distance differences between adjacent pixels due to noise and tilt, which can cause edge confusion depending on whether the distance is long or short, and then using a threshold value to determine the edge.

[0082] Furthermore, the distance image processing unit 4 removes flying pixels (FP) based on the detected edges.

[0083] This allows the distance image processing unit 4 to appropriately remove flying pixels (FP) while suppressing unnecessary invalid pixels.

[0084] Further, the distance image edge detection method in the distance image processing unit 4 (an example of a data processing device) according to this embodiment is as follows: an acquisition unit acquiring a distance image based on distances to objects present in a measurement target space; The method includes a step in which an edge detection unit detects edges for each pixel of the distance image based on a ratio of distance values ​​of the pixel or a plurality of pixels surrounding the pixel.

[0085] As a result, the distance image edge detection method in the distance image processing unit 4 performs edge detection using the ratio of adjacent pixels, so that the distance differences of adjacent pixels due to noise and tilt, which can cause edge confusion depending on whether the distance is long or short, can be aligned to suppress erroneous edge determination, and edge detection can be performed appropriately.

[0086] In the above embodiment, an example was shown in which logarithms (log) were used to calculate the ratio of adjacent pixels in a distance image (see FIG. 13), but logarithms do not have to be used. FIG. 15 shows another example of edge detection processing, illustrating an example of calculation for detecting edges by calculating the ratio of adjacent pixels without using logarithms. In this example, the ratio is calculated by dividing the difference value between adjacent pixels of a pixel of interest by the value of the pixel of interest.

[0087] In the above embodiment, an example has been shown in which a Sobel filter is used for edge detection, but the present invention is not limited to the Sobel filter, and other differential filters may be used. For example, a Prewitt filter or a Laplacian filter may be used instead of the Sobel filter. Fig. 16 is a diagram showing an example of a differential filter. The Sobel filter and the Prewitt filter are first-order differential filters, and the Laplacian filter is a second-order differential filter.

[0088] Furthermore, distance image processing unit 4 (an example of a data processing device) functions as an acquisition unit that acquires a distance image, but is not limited to a configuration in which it acquires a distance image by controlling light source unit 2 and light receiving unit 3. For example, distance image processing unit 4 (an example of a data processing device) may acquire a distance image from another device, and may similarly perform edge detection on the acquired distance image to remove flying pixels (FP). Furthermore, the distance image is not limited to a distance image measured (distance-measured) using the TOF method, but may be a distance image obtained by any method.

[0089] All or part of the depth image processing unit 4 in the above-described embodiment may be implemented by a computer. In this case, a program for implementing this function may be recorded on a computer-readable recording medium, and the program may be loaded into a computer system and executed. Note that the term "computer system" as used herein includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into a computer system. Furthermore, the term "computer-readable recording medium" may also include devices that dynamically store programs for a short period of time, such as communication lines used when transmitting programs via networks such as the Internet or telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within a computer system that serves as a server or client. The program may also be designed to implement part of the above-described functions, or may be capable of implementing the above-described functions in combination with a program already stored in the computer system, or may be implemented using a programmable logic device such as an FPGA.

[0090] Although an embodiment of the present invention has been described above in detail with reference to the drawings, the specific configuration is not limited to this embodiment, and includes designs within the scope of the gist of the present invention. [Explanation of symbols]

[0091] 1...Distance image capturing device 2...Light source section 21...Light source device 22...Diffuser 3...Light receiving section 31...Lens 32...Distance image sensor 320…Light receiving area 321...pixels 322...Control circuit 323...Vertical scanning circuit 324...Horizontal scanning circuit 325...Image signal processing circuit 4...Distance image processing section 41...Timing control section 42...Distance calculation section 43...Measurement control section 44...Edge detection section 45...Noise reduction section CS…Charge storage section PO...light pulse RL…Reflected light OB…object

Claims

1. an acquisition unit that acquires a distance image based on the distance to an object present in the measurement target space; an edge detection unit that detects an edge for each pixel of the distance image based on a ratio of distance values ​​of the pixel or any of a plurality of pixels surrounding the pixel; A data processing device comprising:

2. The edge detection unit a logarithm is used when calculating a ratio of a distance value of each pixel of the distance image to the distance value of the pixel or any of a plurality of pixels surrounding the pixel; 2. The data processing device according to claim 1.

3. The edge detection unit When calculating a ratio of distance values ​​of each pixel of the distance image to the pixel or a plurality of pixels surrounding the pixel, a differential filter is applied to the logarithms of the distance values ​​of the plurality of pixels, and then the logarithms are removed to calculate the ratio of the distance values ​​of the plurality of pixels.

3. The data processing device according to claim 2.

4. The edge detection unit Detecting edges using a threshold value for the calculated ratio value.

2. The data processing device according to claim 1.

5. a noise reduction unit that removes flying pixels based on the edges detected by the edge detection unit; The data processing apparatus of claim 1 , comprising:

6. a light source unit that irradiates a space to be measured with a light pulse; a light receiving unit including pixels each including a photoelectric conversion element that generates an electric charge according to incident light and a plurality of charge accumulation units that accumulate the electric charge, and a pixel drive circuit that distributes and accumulates the electric charge in each of the charge accumulation units in the pixel at a predetermined timing synchronized with the irradiation of the light pulse; a distance calculation unit that calculates a distance to an object present in the space based on the amount of charge accumulated in each of the charge accumulation units, and acquires a distance image; an edge detection unit that detects an edge for each pixel of the distance image based on a ratio of distance values ​​of the pixel or any of a plurality of pixels surrounding the pixel; A distance image capturing device comprising:

7. 1. A method for detecting edges of a distance image in a data processing device, comprising: an acquisition unit acquiring a distance image based on distances to objects present in a measurement target space; an edge detection unit detecting an edge for each pixel of the distance image based on a ratio of distance values ​​of the pixel or any of a plurality of pixels surrounding the pixel; An edge detection method including:

Images