Image processing device and image processing method

Abstract

To provide an image processing device and an image processing method capable of accurately detecting tip coordinates on a captured image of a detection target.SOLUTION: An image processing device 2 uses a distance sensor 4 to capture an image of a detection target 3, and obtains a distance image Dk in which each distance measured by the distance sensor 4 in pixel units is used as a pixel value. A central axis generation unit 10 generates, on the basis of the distance image Dk captured by the distance sensor 4, a central axis which passes through the center of gravity and is parallel to a direction vector in the distance image Dk. A plot unit 11 plots a position of each pixel in an orthogonal coordinate system, each axis of which is a distance to the central axis and a position on the central axis. An estimation unit 12 estimates tip coordinates on the distance image Dk corresponding to a tip portion P0 by performing function fittings on plot points obtained by the plot unit 11.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device and an image processing method. [Background technology]

[0002] Conventionally, as disclosed in Patent Document 1, an image recognition device is known that captures an image of an operator's palm with a camera and determines whether the operator's hand is right or left from the captured image. Patent Document 1 detects the center of gravity of the captured hand and detects the fingertip positions of the captured hand. Specifically, the image recognition device first calculates the center of gravity of each of the X and Y coordinates from a frame image from which the background has been removed. The image recognition device also performs binarization processing and contour extraction processing on the frame image from which the background has been removed to generate a contour image, and sets an X and Y coordinate system with the center of gravity as the origin for this contour image. Then, in this coordinate system, line segments where each position of the contour image intersects with the center of gravity are taken, and the maximum value among these is taken as the position of each of the five fingertips. The image recognition device determines whether the hand is right or left from these fingertip positions. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-204183 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, there is a need to capture a pointing gesture in which a specific finger of a hand is pointed at an object from the front with a camera and to detect the fingertip of the pointing gesture from the captured image. In particular, there has been a demand for the development of technology that can detect the fingertip with high accuracy. [Means for solving the problem]

[0005] An image processing device that solves the above-mentioned problem is a device that photographs a detection object using a distance sensor, acquires a distance image in which each distance measured by the distance sensor on a pixel-by-pixel basis is a pixel value, and calculates the position of the tip of the detection object in a coordinate system of the pixels of the distance image.The image processing device is equipped with a central axis generation unit that generates a central axis that passes through the center of gravity determined for each three-dimensional axis using multiple pixels that constitute the distance image and is parallel to a directional vector determined from vectors pointing from the most distal pixel among the multiple pixels that is closest to the distance sensor to each of the other pixels, a plotting unit that plots the position of each pixel in an orthogonal coordinate system whose axes are the distance relative to the central axis and the position on the central axis, and an estimation unit that estimates the tip coordinates on the distance image corresponding to the tip by performing function fitting on the plotted points obtained by the plotting unit.

[0006] An image processing method for solving the above problem involves photographing an object to be detected with a distance sensor, acquiring a distance image in which each distance measured by the distance sensor on a pixel-by-pixel basis has a pixel value, and calculating the position of the tip of the object to be detected in a coordinate system of the pixels of the distance image, and includes the steps of: calculating a center of gravity for each three-dimensional axis in multiple pixels constituting the distance image; calculating a direction vector from vectors pointing from a tip-end pixel among the multiple pixels that is closest to the distance sensor to each of the other pixels; generating a central axis that passes through the center of gravity and is parallel to the direction vector; plotting the distance to the central axis and the position of each pixel in an orthogonal coordinate system of the central axis; and estimating the tip coordinates on the distance image corresponding to the tip by performing function fitting on the plotted points obtained by the plotting. Effect of the Invention

[0007] The present invention can detect the tip coordinates on a captured image of a detection target with high accuracy. [Brief description of the drawings]

[0008] [Figure 1]FIG. 1 is an explanatory diagram illustrating a configuration of an image processing device according to an embodiment. [Diagram 2] FIG. 2 is a perspective view showing the resolution of a distance sensor. [Diagram 3] FIG. 13 is a schematic diagram showing an example of a distance image. [Figure 4] FIG. 13 is an explanatory diagram showing a method for extracting pixels within a tip region. [Diagram 5] FIG. 11 is an explanatory diagram showing a method for calculating a central axis. [Figure 6] FIG. 13 is an explanatory diagram showing how to determine plot points. [Figure 7] FIG. 1 is an explanatory diagram showing an overview of function fitting. [Figure 8] 13 is a flowchart showing a procedure for estimating a tip coordinate. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, one embodiment of the present disclosure will be described. (User operation detection device 1) As shown in Fig. 1, user operation detection device 1 detects a user operation using a part of the body by using an image processing device 2 that captures an image of the user. The user operation detection device 1 captures an image of a detection target 3, which is a part of the user's body, by the image processing device 2, and detects the user operation based on the image processing result of the image processing device 2. The detection target 3 in this example is the user's pointing hand with a certain fingertip raised. In this way, the user operation detection device 1 in this example detects a pointing operation by the user from an image captured by the image processing device 2.

[0010] (Image processing device 2) 1, the image processing device 2 includes a distance sensor 4 that captures an image of a detection target 3, and a calculation device 5 that identifies, by calculation, the positions of parts of the detection target 3 in a distance image Dk captured by the distance sensor 4. The image processing device 2 in this example captures an image of a hand from the front using the distance sensor 4. The calculation device 5 in this example calculates the tip part P0 of the image of the detection target 3, which is the fingertip P' in this example, in the distance image Dk captured by the distance sensor 4.

[0011] (Distance sensor 4) As shown in FIG. 2, the distance sensor 4 is, for example, a ToF (Time of flight) sensor. The ToF sensor measures the distance to the detection target 3 by measuring the time it takes for emitted light to be reflected and bounced back. The distance sensor 4 has a resolution of a predetermined number of pixels, in this example, 8×8 = 64 pixels. Thus, the distance sensor 4 in this example is a low-resolution ToF sensor. The distance sensor 4 has a directivity with only a certain predetermined range as an imaging area. The distance sensor 4 obtains three-dimensional information of the detection target 3 by imaging the detection target 3 and obtaining distance data for each pixel.

[0012] In this example, the left-right direction seen from the distance sensor 4 is the "X-axis", the height direction perpendicular to the X-axis is the "Y-axis", and the depth direction seen from the distance sensor 4 is the "Z-axis". (Distance image Dk) 3, the distance sensor 4 captures an image of the detection target 3 to obtain a distance image Dk as three-dimensional information. The distance image Dk has 64 pixels Pi with a resolution of 8×8, and obtains a pixel value Dpx for each pixel Pi according to the distance to the detection target 3. In this way, the distance sensor 4 obtains the distance image Dk in which each distance measured pixel by pixel is set as a pixel value Dpx.

[0013] The coordinate system of pixel Pi in distance image Dk is an XY plane coordinate system consisting of an X-axis, which is one axis of detection surface 7 (see FIG. 2) of distance sensor 4, and a Y-axis perpendicular to the X-axis. Therefore, the pixel position in the X-axis direction of distance sensor 4 becomes the X-axis coordinate of distance image Dk, and the pixel position in the Y-axis direction of distance sensor 4 becomes the Y-axis coordinate of distance image Dk.

[0014] The pixel value Dpx is a value corresponding to the Z coordinate in the Z-axis direction perpendicular to the XY plane. The pixel value Dpx is displayed, for example, in a different color on the image. Specifically, for pixels Pi whose distance between the detection target 3 and the distance sensor 4 is equal to or greater than a certain value, the pixel value Dpx is displayed in a color according to the distance. The display colors of the pixels Pi are, for example, "red," "pink," and "purple" in order of proximity to the distance sensor 4, and "blue" when the distance is less than a certain value. In this way, the coordinate in the Z-axis direction of the distance image Dk corresponds to the color of the pixel Pi.

[0015] As described above, the distance image Dk has X and Y coordinates represented by pixel coordinate positions, and the Z axis represented by the pixel value Dpx of pixel Pi. That is, the distance image Dk is displayed in a matrix, with the X and Y coordinates represented by the presence or absence of color, and the Z coordinate represented by the type of color.

[0016] (Coordinate position calculation function) 1, the image processing device 2 determines the position of the tip part P0 of the detection object 3 in the coordinate system of the pixel Pi of the distance image Dk acquired by the distance sensor 4. In this example, the tip part P0 is, for example, the tip P' of a user's finger pointing with his / her hand. The image processing device 2 determines the tip coordinate P (P is expressed as a vector: see FIG. 7) corresponding to the finger tip P' in the distance image Dk capturing the pointing hand captured by the distance sensor 4.

[0017] In this specification, coordinate symbols, centroid symbols, and vector symbols are simply indicated by alphabetic symbols, but in drawings and formulas, vector symbols are added for accurate notation. The image processing device 2 includes a tip region extraction unit 9 that extracts pixels Pi located within a tip region range L (see FIG. 4) that is set centered on a forefront pixel Pik (see FIG. 4) that is closest to the distance sensor 4, among the multiple pixels Pi that constitute the distance image Dk. The tip region extraction unit 9 is provided in the calculation device 5. The tip region extraction unit 9 sets a circular tip region range L of a predetermined range centered on the forefront pixel Pik, and extracts pixels Pi located within this tip region range L as tip region pixels Pit to be used for estimating the tip coordinate P.

[0018] The image processing device 2 includes a central axis generating unit 10, a plotting unit 11, and an estimation unit 12. The central axis generating unit 10, the plotting unit 11, and the estimation unit 12 are provided in the calculation device 5. The central axis generating unit 10, the plotting unit 11, and the estimation unit 12 estimate the finger tip P' reflected in the distance image Dk, i.e., the tip coordinate P on the image, by using, for example, pixels Pit within the tip region extracted by the tip region extraction unit 9.

[0019] 5, the central axis generating unit 10 generates the central axis C as a straight line passing through the center of gravity G and extending in the direction of the direction vector V. The central axis generating unit 10 generates the central axis C that passes through the center of gravity G determined for each three-dimensional axis using the multiple pixels Pi constituting the distance image Dk and is parallel to the direction vector V determined from vectors pointing from the forefront pixel Pik that is closest to the distance sensor 4 among the multiple pixels Pi to each of the other pixels Pir.

[0020] 6, the plot unit 11 plots the position of each pixel Pi in an orthogonal coordinate system having the distance X' from the central axis C and the position Z on the central axis C as the respective axes. In this way, the plot unit 11 plots the coordinate x of each pixel Pi in the coordinate system of the distance X' from the central axis C and the position Z on the central axis C. n A plot point Prt corresponding to the distance X' is obtained. The plot point Prt is obtained for each pixel Pit in the tip region. The axis of distance X' and the axis of position Z are mutually perpendicular.

[0021] 7, the estimation unit 12 estimates a tip coordinate P, which is a coordinate position corresponding to the tip part P0 in the distance image Dk. The estimation unit 12 estimates the tip coordinate P on the distance image Dk corresponding to the tip part P0 by performing function fitting on the plot point Prt obtained by the plotting unit 11. In this example, the estimation unit 12 estimates at least the Z-axis coordinate value of the tip coordinate P by function fitting.

[0022] Next, the operation of the image processing device 2 (image processing method) of this embodiment will be described. (Getting distance data) 8, in step 101, when the detection target 3 is photographed by the distance sensor 4, the calculation device 5 acquires distance data of the detection target 3 from the distance sensor 4. Specifically, the calculation device 5 acquires a distance image Dk of the detection target 3 photographed by the distance sensor 4.

[0023] (Extraction of tip region) As shown in FIG. 8, after acquiring distance image Dk, in step 102, tip region extraction unit 9 extracts pixels Pi that exist within tip region range L (tip region pixels Pit) from among pixels Pi in distance image Dk.

[0024] As shown in FIG. 4, the tip region extraction unit 9 first sets a front-end pixel Pik as a process for extracting pixels Pi (tip region pixels Pit) within the tip region range L. In this example, the tip region extraction unit 9, for example, selects the pixel Pik having the highest pixel value Dpx according to the distance from the detection target 3 among the multiple pixels Pi as the front-end pixel Pik. In this way, the front-end pixel Pik is the pixel Pi having the highest pixel value Dpx in the Z-axis direction. Then, the tip region extraction unit 9 selects a range of a certain distance (referred to as tip region r) centered on the front-end pixel Pik as the tip region range L. The tip region range L is, for example, a circular region centered on the front-end pixel Pik and with the tip region r as a radius. Then, the tip region extraction unit 9 extracts the pixels Pi that exist within the tip region range L as the tip region pixels Pit among the multiple pixels Pi.

[0025] (Generation of central axis C) As shown in FIG. 8, after extracting the pixels Pit in the tip region, in step 103, the medial axis generating unit 10 generates a medial axis C using the pixels Pit in the tip region.

[0026] As shown in FIG. 5, in order to generate the central axis C, the central axis generating unit 10 first calculates the center of gravity G by using the following equations (1) and (2). In this example, the center of gravity G is calculated by dividing the distance between the forefront pixel Pik and each of the other pixels Pir by a weight W n The center of gravity G is calculated by the weighted average of the three-dimensional coordinate system (G x ,G y ,G z ) is calculated as follows: x component of center of gravity G x is the center of gravity in the X-axis direction of the pixel group within the tip region range L. The y component G of the center of gravity G y is the center of gravity in the Y-axis direction of the pixel group within the tip region range L. The z component G of the center of gravity G z is the center of gravity in the Z-axis direction of the pixel group within the tip region range L.

[0027]

number

[0028]

number

[0029] In addition, "N" and "W" in formulas (1) and (2) n ", "x n ", "y n "," z n ", "L", "D n " is defined as follows: N: Number of pixels within tip region L W n : Weight for each pixel Pi x n ,y n ,z n : Coordinate value of each pixel Pi L: Upper limit of distance from the leading edge pixel Pik D n: The distance of each pixel Pi from the front-most pixel Pik (the front-most pixel is D n =0) Next, in order to generate the central axis C, the central axis generating unit 10 calculates the direction vector V using the following formula (3). Note that "v1 to v4" in formula (3) are vectors from the leading edge pixel Pik to the remaining pixels Pi within the leading edge region range L. The direction vector V is expressed in terms of a three-dimensional coordinate system (V x ,V y ,V z ) is calculated as follows.

[0030]

number

[0031] In this way, the direction vector V is a unit vector obtained by unitizing the resultant vector of the vectors pointing from the forefront pixel Pik to each of the other pixels Pir. Next, the central axis generating unit 10 calculates the central axis C using the center of gravity G and the direction vector V. The central axis generating unit 10 calculates the central axis C using the following equation (4). Note that "t" in equation (4) is a real number.

[0032]

number

[0033] As described above, the central axis generating unit 10 calculates the center of gravity G and the direction vector V using the pixels Pi within the tip region range L, and then generates the central axis C using the center of gravity G and the direction vector V. In this manner, the central axis generating unit 10 determines, as the central axis C, a straight line that passes through the center of gravity G and extends in the direction of the direction vector V.

[0034] (Calculation of plot point Prt) As shown in FIG. 8, after calculating the central axis C, in step 104, the plot unit 11 projects each pixel Pi within the tip region range L onto the coordinate plane to calculate a plot point Prt for each pixel Pi.

[0035] As shown in Fig. 6, plot unit 11 calculates plot points Prt for each pixel Pi by projecting each pixel Pi (in this example, a pixel Pit in the tip area) in tip area range L onto a plane whose coordinate system is "X'" which is the distance from center axis C and "Z" which is the position on the center axis. In this example, plot unit 11 calculates coordinates x of each pixel Pi by using the following equations (5) to (8). n The distance between the center axis C and the n '" and vector B, which is a vector from the center of gravity G to each pixel Pi. n Component "Z" in the direction of the central axis C n " and calculate the plot point Prt for each pixel Pi. Vector B n Component "Z" in the direction of the central axis C n " corresponds to position Z on the central axis C.

[0036]

number

[0037]

number

[0038]

number

[0039]

number

[0040] As can be seen from the above equation, distance X is calculated using Pythagoras' theorem. n ' and component Z n Then, the pixels Pit in the tip region are projected onto a plane having these as an orthogonal coordinate system to calculate plot points Prt. In this manner, the plot unit 11 calculates plot points Prt for each pixel Pi.

[0041] (Estimation of tip coordinates) 8, after the calculation of the plot points Prt, the estimation unit 12 estimates the tip coordinate P from the calculated plot points Prt using function fitting based on the least squares method in step 105. The reason why the least squares method is used in the function fitting is that the outline shape of the pointing finger resembles a quadratic parabola.

[0042] As shown in Fig. 7, the estimation unit 12 performs function fitting by the least squares method using the plot points Prt to estimate the shape of the fingertip. In this example, the estimation unit 12 estimates the shape of the fingertip by fitting the plot points Prt to a quadratic function "Z = aX' 2 +b". Fit the quadratic function "Z=aX' using the least squares method. 2 +b”, the estimation unit 12 calculates “a” and “b” using the following equations (9) and (10).

[0043]

number

[0044]

number

[0045] Then, the estimation unit 12 uses the obtained "b" to calculate the tip coordinate P according to the following formula (11).

[0046]

number

[0047] The calculation device 5 recognizes the tip coordinate P estimated by the estimation unit 12 as the tip of the finger reflected in the distance image Dk. This allows the calculation device 5 to acquire a location close to the tip of the finger as the position of the finger tip P' in the distance image Dk, rather than an erroneous location such as a location closer to the base of the finger.

[0048] Furthermore, the image processing device 2 executes the above-mentioned calculation process for determining the tip coordinate P of the finger tip P' at predetermined intervals or each time the distance image Dk changes. This allows the image processing device 2 to obtain the optimal finger tip coordinate P each time.

[0049] (Effects of the embodiment) According to the configuration of the above embodiment, the following effects can be obtained. (1) The image processing device 2 photographs the detection target 3 using the distance sensor 4, obtains a distance image Dk in which each distance measured by the distance sensor 4 in pixel units is set as a pixel value Dpx, and determines the position of the tip part P0 of the detection target 3 in the coordinate system of the pixel Pi of the distance image Dk. The central axis generating unit 10 included in the image processing device 2 generates a central axis C that passes through the center of gravity G determined for each three-dimensional axis using the multiple pixels Pi constituting the distance image Dk and is parallel to a direction vector V determined from vectors directed from the most distal pixel Pik closest to the distance sensor 4 among the multiple pixels Pi to each of the other pixels Pir. The plotting unit 11 included in the image processing device 2 plots the position of each pixel Pi in an orthogonal coordinate system whose axes are the distance X' to the central axis C and the position Z on the central axis C. The estimation unit 12 included in the image processing device 2 estimates the tip coordinate P on the distance image Dk corresponding to the tip part P0 by performing function fitting on the plot point Prt obtained by the plotting unit 11.

[0050] Incidentally, when calculating the tip coordinate P of a finger from the image of the distance image Dk, it is also possible to use a method of calculating the center of gravity of the coordinates for each of the three axes, X-axis, Y-axis, and Z-axis, as the tip position. In this case, however, a position near the base of the finger may be calculated as the tip coordinate P, which poses a problem of poor accuracy. On the other hand, according to the present configuration, the shape of the detection target 3 is estimated using function fitting, thereby calculating the tip coordinate P of the detection target 3 shown in the distance image Dk. Therefore, the tip coordinate P of the captured image of the detection target 3 can be detected with high accuracy.

[0051] (2) The coordinate system of pixel Pi of distance image Dk is an XY plane coordinate system consisting of an X axis, which is one axis of detection surface 7 of distance sensor 4, and a Y axis perpendicular to the X axis. The pixel value Dpx is a value corresponding to the Z coordinate of the Z axis perpendicular to the XY plane. The estimation unit 12 estimates at least the Z axis coordinate value of the tip coordinate P by function fitting. With this configuration, when determining the tip coordinate P of the detection object 3 reflected in distance image Dk, it is possible to accurately detect the position of the Z axis of the coordinate system. This makes it difficult for the base side of the tip part P0 to be estimated as the tip coordinate P.

[0052] (3) The image processing device 2 includes a tip region extraction unit 9 that extracts pixels Pi that are located within a tip region range L that is set around the forefront pixel Pik, from among the multiple pixels Pi that make up the distance image Dk. The central axis generation unit 10 generates a central axis C using the pixels Pit within the tip region extracted by the tip region extraction unit 9. This configuration makes it possible to generate the central axis C while excluding unnecessary pixels Pi. This makes it possible to accurately determine the central axis C, which in turn contributes to more accurate detection of the tip coordinate P.

[0053] (4) The center of gravity G is a weighted center of gravity calculated by a weighted average, with the weight being the distance between the forefront pixel Pik and each of the other pixels Pir. This configuration makes it possible to accurately determine the center of gravity G, which in turn contributes to more accurate detection of the tip coordinate P.

[0054] (5) The direction vector V is a unit vector obtained by unitizing the resultant vector of the vectors pointing from the forefront pixel Pik to each of the other pixels Pir. With this configuration, in the case of a unit vector, the vector value is "1", which simplifies the calculation for determining the central axis C.

[0055] (6) The estimation unit 12 estimates the tip coordinate P by function fitting using the least squares method. With this configuration, the tip coordinate P of the detection target 3 captured in the distance image Dk can be calculated with high accuracy using function fitting using the least squares method.

[0056] (Other embodiments) This embodiment can be modified as follows: This embodiment and the following modifications can be combined with each other to the extent that no technical contradiction occurs.

[0057] The center of gravity G is not limited to a weighted center of gravity, but may be any value calculated using a weighted average. The direction vector V is not limited to a unit vector, but may be a vector value itself.

[0058] The calculation of the tip coordinate P is not limited to using only pixels within the tip region L, but may use, for example, all pixels Pi. The function fitting is not limited to the least squares method. For example, a method other than the least squares method may be used depending on the shape of the detection target 3.

[0059] The distance sensor 4 is not limited to a Tof sensor, and may be any sensor capable of acquiring three-dimensional information of the detection target 3 being photographed. The detection target 3 is not limited to the user's hand, and may be changed to, for example, another member such as a workpiece moving along a factory line.

[0060] The image processing device 2 may be used in equipment other than a vehicle. The X and Y coordinates of the tip coordinate P are not limited to being calculated by function fitting, but may be calculated from the center of gravity. In other words, the Z coordinate of the tip coordinate P may be calculated by function fitting, and the X and Y coordinates may be obtained from the center of gravity.

[0061] The tip region extraction unit 9, the central axis generation unit 10, the plot unit 11, and the estimation unit 12 may be configured by [1] one or more processors operating according to a computer program (software), or [2] a combination of such a processor and one or more dedicated hardware circuits such as an application specific integrated circuit (ASIC) that executes at least some of the various processes. The processor includes a CPU and a memory such as a RAM and a ROM, and the memory stores program code or instructions configured to cause the CPU to execute the processes. The memory (computer-readable medium) includes any available medium that can be accessed by a general-purpose or dedicated computer. Alternatively, instead of a computer including the above processor, a processing circuit configured by one or more dedicated hardware circuits that execute all of the various processes may be used.

[0062] The tip region extraction unit 9, the central axis generation unit 10, the plot unit 11, and the estimation unit 12 may be configured from independent processors, or may be constructed from a processor that shares some of its functions. In this way, the tip region extraction unit 9, the central axis generation unit 10, the plot unit 11, and the estimation unit 12 are not limited to being independent functional blocks, and may be configured from one functional block, or may be configured from a functional block that shares some of its functions.

[0063] Although the present disclosure has been described with reference to the embodiment, it is understood that the present disclosure is not limited to the embodiment or structure. The present disclosure also includes various modifications and modifications within the scope of equivalents. In addition, various combinations and forms, as well as other combinations and forms including only one element, more than one element, or less than one element, are also within the scope and concept of the present disclosure. [Explanation of symbols]

[0064] 2...image processing device, 3...detection object, 4...distance sensor, 7...detection surface, 9...tip area extraction unit, 10...central axis generation unit, 11...plotting unit, 12...estimation unit, Dpx...pixel value, Dk...distance image, P0...tip portion, Pi...pixel, Pik...leading edge pixel, Pir...other pixels, G...center of gravity, V...directional vector, C...central axis, X'...distance relative to the central axis, Z...position on the central axis, Prt...plot point, P...tip coordinate, L...tip area range, Pit...pixel within the tip area.

Claims

1. 1. An image processing device that captures an image of a detection target using a distance sensor, obtains a distance image having pixel values ​​each of which represents a distance measured in pixel units by the distance sensor, and determines a position of a tip portion of the detection target in a pixel coordinate system of the distance image, a central axis generating unit that generates a central axis that passes through the center of gravity determined for each three-dimensional axis using a plurality of pixels that constitute the distance image and is parallel to a direction vector determined from vectors directed from a front-end pixel that is closest to the distance sensor among the plurality of pixels to each of the other pixels; a plotting unit that plots the position of each pixel in an orthogonal coordinate system having axes representing a distance from the central axis and a position on the central axis; an estimation unit that estimates tip coordinates on a distance image corresponding to the tip portion by performing function fitting on the plot points obtained by the plot unit.

2. a coordinate system of a pixel of the distance image is an XY plane coordinate system configured with an X axis, which is one axis of a detection surface of the distance sensor, and a Y axis perpendicular to the X axis; the pixel value is a value corresponding to a Z coordinate on a Z axis perpendicular to the XY plane, The image processing device according to claim 1 , wherein the estimation unit estimates at least the Z-axis coordinate value of the tip coordinate by the function fitting.

3. a tip region extraction unit that extracts pixels located within a tip region range set around the tip pixel from among the plurality of pixels constituting the distance image, The image processing device according to claim 1 , wherein the central axis generating section generates the central axis by using pixels in the tip region extracted by the tip region extracting section.

4. The image processing device according to claim 1 , wherein the centroid is a weighted centroid calculated by a weighted average in which a weight is a distance between the forefront pixel and each of the other pixels.

5. The image processing device according to claim 1 , wherein the direction vector is a unit vector obtained by unitizing a composite vector of the vectors directed from the forefront pixel to each of the other pixels.

6. The image processing device according to claim 1 , wherein the estimation unit estimates the tip coordinates by performing the function fitting using a least squares method.

7. 1. An image processing method for detecting a tip of a detection target by using a distance sensor, acquiring a distance image having pixel values ​​each of which represents a distance measured in pixel units by the distance sensor, and determining a position of a tip of the detection target in a pixel coordinate system of the distance image, comprising: determining a center of gravity for each three-dimensional axis in a plurality of pixels constituting the range image; determining a direction vector from vectors pointing from a front-end pixel that is closest to the distance sensor among the plurality of pixels to each of the other pixels; generating a central axis passing through the center of gravity and parallel to the direction vector; Plotting the distance to the central axis and the position of each pixel in a rectangular coordinate system relative to the central axis; and estimating tip coordinates on the distance image corresponding to the tip portion by performing function fitting on the plot points obtained by the plotting.

Images