Image Processing Device

The image processing device addresses the challenge of correcting distortion in oblique images of uneven surfaces by setting point groups and calculating a first average plane to convert the image into a directly facing orientation, enhancing evaluation accuracy.

JP7579095B2Active Publication Date: 2024-11-07HITACHI CONSTRUCTION MACHINERY CO LTD
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
JP2020154650
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Filing Date
2020-09-15
Publication Date
2024-11-07
Estimated Expiration
2040-09-15

AI Technical Summary

Technical Problem

Conventional image processing devices struggle to correct distortion in images of uneven object surfaces taken at oblique angles, especially when the object's front surface is not flat due to irregularities or foreign matter, making it difficult to measure orientation and correct distortion accurately.

Method used

An image processing device that sets multiple point groups on the target surface, calculates a first average plane based on distance measurements, and converts the image into a front-facing orientation using this plane to minimize the effects of local irregularities and foreign objects.

Benefits of technology

The device effectively converts obliquely taken images of uneven surfaces into directly facing images, allowing accurate evaluation of mechanical components by reducing the impact of surface irregularities and foreign objects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007579095000001
    Figure 0007579095000001
  • Figure 0007579095000002
    Figure 0007579095000002
  • Figure 0007579095000003
    Figure 0007579095000003
Patent Text Reader

Abstract

To provide an image processing device capable of converting an image obtained by diagonally photographing an object surface having irregularity of a subject into a front view image obtained by making the object surface face an imaging surface.SOLUTION: An image processing device 150 sets a first point group, a second point group and a third point group separated from one another in an object surface, and acquires from a distance sensor 120, a distance to each point of the first point group, a distance to each point of the second point group and a distance to each point of the third point group from an imaging device 110. Also, the image processing device 150 executes plane calculation processing for calculating a plane on the basis of one point selected from the first point group, one point selected from the second point group and one point selected from the third point group to a combination of all points to calculate a first plane group. Also, the image processing device 150 calculates a first average plane in which an average vector obtained by averaging unit normal vectors of the whole plane of the first plane group is determined as a normal vector and the total sum of distances to all points included in the first point group, the second point group and the third point group becomes minimum. Then, the image processing device 150 converts an image of the object surface into a front view image on the basis of the first average plane.SELECTED DRAWING: Figure 1
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to an image processing device. [Background technology]

[0002] Conventionally, there is known an image processing device that is used in digital still cameras and the like, and that corrects distortion caused by oblique shooting of an image of an object plane (see Patent Document 1 below). This conventional image processing device includes an imaging means for imaging the object plane, a plane measurement means for measuring the orientation of the object plane based on the imaging means, and an image distortion correction means for correcting distortion caused by oblique shooting of the image captured by the imaging means based on the measured orientation of the object plane (Patent Document, Claim 1). According to this conventional image processing device, even if an arbitrary object plane is captured from an arbitrary viewpoint and posture, image distortion can be easily corrected by measuring the plane (Patent Document, Paragraphs 0016 and 0077). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2000-307947 A Summary of the Invention [Problem to be solved by the invention]

[0004] For example, to evaluate the wear state of a mechanical part based on an image, a front image of the part facing the imaging surface is required. However, the front of the part is not necessarily flat, and may have unevenness due to the adhesion of foreign matter. When such a part is used as a subject, the conventional image processing device has difficulty in measuring the orientation of the mechanical part based on the imaging means, and may not be able to correct distortion caused by oblique shooting of the image captured by the imaging means.

[0005] The present disclosure provides an image processing device capable of converting an image of an object surface having unevenness captured obliquely into a frontal image in which the object surface is directly opposed to the imaging plane. [Means for solving the problem]

[0006] One aspect of the present disclosure is an image processing device that acquires an image of an object surface from an imaging device, acquires a distance between the imaging device and the object surface from a distance sensor, and converts the image of the object surface into a front-on image, the image processing device comprising: setting a first point group, a second point group, and a third point group spaced apart from each other on the object surface; acquiring a distance from the imaging device to each point of the first point group, a distance to each point of the second point group, and a distance to each point of the third point group from the distance sensor; calculating a first plane group by performing a plane calculation process for all combinations of points to calculate a plane based on one point selected from the first point group, one point selected from the second point group, and one point selected from the third point group; calculating a first average plane whose normal vector is an average vector of unit normal vectors of all planes in the first plane group and which has a minimum sum of distances between the first point group, the second point group, and all points included in the third point group; and converting the image of the object surface into the front-on image based on the first average plane. Effect of the Invention

[0007] According to the above aspect of the present disclosure, it is possible to provide an image processing device capable of converting an image of a subject's uneven surface captured from an oblique angle into a front-on image in which the subject surface is directly opposed to the imaging surface. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram illustrating an embodiment of an image processing device according to the present disclosure. [Diagram 2] 2 is a side view of a work machine showing an example of a processing target of the image processing device of FIG. 1; [Figure 3A] FIG. 2 is a flow chart showing an example of a processing flow by the image processing device of FIG. [Figure 3B]FIG. 2 is a flow chart showing an example of a processing flow by the image processing device of FIG. [Figure 4] FIG. 2 is an image diagram of a target surface and multiple point clouds displayed on the display device of the condition assessment device of FIG. 1. [Diagram 5] FIG. 2 is an image diagram showing an example of a mechanical part and a guide line displayed on the display device of FIG. 1. [Figure 6] 2 is a conceptual diagram of a first group of planes and unit normal vectors calculated by the image processing device of FIG. 1. [Figure 7] 2 is an explanatory diagram of a plane calculation process executed by the image processing device of FIG. 1; [Figure 8] 2 is a conceptual diagram of a first mean plane and a mean vector calculated by the image processing device of FIG. 1. [Figure 9] FIG. 2 is a conceptual diagram of the second plane group and average vector calculated by the image processing device of FIG. 1. [Figure 10] FIG. 2 is a conceptual diagram of the rotation process of the first and second mean planes by the image processing device of FIG. 1. [Figure 11] FIG. 2 is a conceptual diagram of the rotation process of the first and second mean planes by the image processing device of FIG. 1. [Figure 12] FIG. 2 is a conceptual diagram of the rotation process of the first and second mean planes by the image processing device of FIG. 1. [Figure 13] FIG. 2 is a conceptual diagram of the rotation process of the first and second mean planes by the image processing device of FIG. 1. [Figure 14] FIG. 2 is a conceptual diagram of a process for calculating a front-facing image by the image processing device of FIG. 1. [Figure 15] FIG. 2 is a conceptual diagram of a process for calculating a front-facing image by the image processing device of FIG. 1. [Figure 16] A plan view of a portion of a sprocket and a side view of two types of sprockets. [Figure 17] 2 is an explanatory diagram of secondary rotation of a frontal image by the image processing device of FIG. 1; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of an image processing device according to the present disclosure will be described with reference to the drawings.

[0010] 1 is a block diagram showing an embodiment of an image processing device 150 according to the present disclosure. The image processing device 150 of this embodiment constitutes, for example, a part of a condition evaluation device 100 for a machine component. The condition evaluation device 100 evaluates, for example, the condition of a component constituting a machine, such as wear, based on an image of the component. The condition evaluation device 100 includes, for example, an imaging device 110, a distance sensor 120, an input device 130, a display device 140, and the image processing device 150 of this embodiment.

[0011] The imaging device 110 is, for example, a digital camera, and includes an optical system including a lens, an electronic system including a light receiving element, and a housing for holding these. The imaging device 110 may be, for example, a mobile information terminal with an imaging function, such as a smartphone. The distance sensor 120 is, for example, an image sensor that measures the distance to an object using a time-of-flight (TOF) method. The distance sensor 120 includes, for example, a pulse-modulated light source, and measures the distances to multiple points on the object surface based on phase difference information between the timing of light emission from the light source and the timing of receiving the reflected light from the object by the image sensor.

[0012] Furthermore, the imaging device 110 or the distance sensor 120 visualizes a plurality of points on the target surface to which the distance is to be measured. Specifically, the imaging device 110 or the distance sensor 120 includes, for example, a visualization light source that irradiates a visible light laser onto a plurality of points on the target surface to which the distance is to be measured. This visualization light source may also serve as a distance measuring light source. Furthermore, the imaging device 110 or the distance sensor 120 may visualize a plurality of points on the target surface to which the distance is to be measured by another method. The imaging device 110 may include the distance sensor 120.

[0013] The input device 130 includes, for example, an operation button, an operation dial, or a touch sensor provided on the imaging device 110, a touch panel provided on the display device 140, or a keyboard or a microphone connected to the image processing device 150 separately from the imaging device 110 and the display device 140. An operator of the condition evaluation device 100 and the image processing device 150 inputs necessary information to the image processing device 150 via the input device 130, for example.

[0014] The display device 140 includes, for example, a viewfinder, a liquid crystal display device, or an organic EL display device provided in the imaging device 110, or a liquid crystal display device, an organic EL display device, or a mobile information terminal such as a smartphone connected to the image processing device 150 separately from the imaging device 110.

[0015] The image processing device 150 is, for example, a microcontroller including an input / output unit 151, a processing device 152 such as a central processing unit (CPU), a storage device 153 such as a RAM or a ROM, various control programs stored in the storage device 153, and a timer (not shown). The image processing device 150 realizes various processes and functions (described later) by executing the various control programs stored in the storage device 153 using the processing device 152.

[0016] The input / output unit 151 is connected to the imaging device 110, the distance sensor 120, the input device 130, and the display device 140 so as to be able to communicate with each other. The processing device 152 is connected to the input / output unit 151 so as to be able to communicate with each other, and transmits and receives control signals and information between the imaging device 110, the distance sensor 120, the input device 130, the display device 140, and the storage device 153 via the input / output unit 151. The storage device 153 stores, for example, various control programs and data, and stores processing results of the processing device 152 and information input from the imaging device 110, the distance sensor 120, and the input device 130.

[0017] Fig. 2 is a side view showing an example of a construction machine 200. An evaluation target of the condition evaluation device 100 of this embodiment shown in Fig. 1 is, for example, a part of the construction machine 200. Moreover, an image processing target of the image processing device 150 of this embodiment shown in Fig. 1 is, for example, a part of the construction machine 200. The construction machine 200 is, for example, a hydraulic excavator. Note that the construction machine 200 is not limited to a hydraulic excavator and may be another construction machine.

[0018] The work machine 200 includes, for example, a traveling device 210, a rotating body 220, and a front work implement 230. The rotating body 220 is provided on the traveling device 210 so as to be rotatable via a rotating mechanism including a hydraulic motor or an electric motor. The front work implement 230 includes, for example, a link mechanism in which a boom 231, an arm 232, and a bucket 233 are connected via joints. The front work implement 230 can perform desired work such as excavation work or leveling work by extending and retracting hydraulic cylinders 231S, 232S, and 233S using a hydraulic device.

[0019] The traveling device 210 is, for example, a crawler type traveling device including a sprocket 211, an idler 212, a link assembly 213, an upper roller 214, a lower roller 215, and a shoe plate 216. The sprocket 211 is connected to, for example, a hydraulic motor driven by a hydraulic device via a power transmission mechanism. The link assembly 213 is stretched across the sprocket 211, the idler 212, the upper roller 214, and the lower roller 215.

[0020] The multiple shoe plates 216 are attached to the outside of the annular link assembly 213, and the multiple shoe plates 216 located on the lower side come into contact with the ground. With this configuration, when the hydraulic motor rotates, the sprocket 211 rotates, and the annular link assembly 213, to which the multiple shoe plates 216 are attached on the outside, rotates around the idler 212, upper roller 214, and lower roller 215, causing the work machine 200 to travel.

[0021] For example, mechanical parts such as the sprocket 211, idler 212, link assembly 213, upper roller 214, and lower roller 215 of the traveling device 210 are subject to wear due to friction between the parts and soil that has gotten in between the parts. The image processing device 150 of this embodiment takes such mechanical parts as objects, captures the object surface of the object with the imaging device 110, and processes the image of the object surface to convert it into a frontal image.

[0022] Next, the operation of the image processing device 150 of this embodiment will be described with reference to Figures 3A and 3B. Figures 3A and 3B are flow diagrams showing an example of the flow of processing by the image processing device 150 of Figure 1.

[0023] For example, an operator of the condition assessment device 100 captures an image of a target surface of a sprocket 211 of a work machine 200 with the imaging device 110 and converts the image of the target surface into a frontal image with the image processing device 150, thereby assessing the condition, such as wear, of the sprocket 211. When the operator starts the condition assessment device 100, the image processing device 150 first executes process P1 for displaying and acquiring settings.

[0024] In this process P1, the image processing device 150 causes the display device 140 to display, for example, a threshold value to be used in process P13, which will be described later. The operator checks the threshold value displayed on the display device 140. This threshold value is, for example, a threshold value of the height of the unevenness with respect to a first mean plane H1, which will be described later. Furthermore, when the operator wishes to change the threshold value displayed on the display device 140, he or she operates the input device 130 to input a new threshold value. In this case, the image processing device 150 obtains the new threshold value input by the operator from the input device 130 and stores it in the storage device 153.

[0025] Also, in this process P1, the image processing device 150 causes the display device 140 to display, for example, whether or not a secondary rotation is required to rotate the frontal image of the target surface 211f of the sprocket 211 obtained in a process described below. When the operator wants to change whether or not the secondary rotation is required displayed on the display device 140, he or she operates the input device 130 to newly input whether or not the secondary rotation is required. In this case, the image processing device 150 obtains the information on whether or not the secondary rotation is required input by the operator from the input device 130 and stores it in the storage device 153.

[0026] Furthermore, in this process P1, the image processing device 150, for example, displays on the display device 140 whether or not automatic shooting is performed by the imaging device 110. When the operator wishes to change the whether or not automatic shooting is performed displayed on the display device 140, he or she operates the input device 130 to newly input whether or not automatic shooting is performed. In this case, the image processing device 150 acquires the whether or not automatic shooting is performed input by the operator from the input device 130 and stores it in the storage device 153.

[0027] Next, the image processing device 150 executes a process P2 for determining whether or not to perform a secondary rotation for rotating the front-view image of the target surface of the sprocket 211. If the presence or absence of secondary rotation stored in the storage device 153 is "YES," the image processing device 150 executes a process P3 for displaying guide lines.

[0028] In process P3, image processing device 150 causes display device 140, such as a viewfinder of imaging device 110, to display guide lines corresponding to the contour shape of sprocket 211, which is the part to be evaluated. The operator adjusts the attitude of imaging device 110 so that the guide lines displayed on display device 140 match the contour shape of sprocket 211, for example.

[0029] Next, the image processing device 150 executes a process P4 for determining whether or not automatic photography is performed by the imaging device 110. If the information on whether or not automatic photography is performed stored in the storage device 153 is "yes" (YES), the image processing device 150 executes a process P5 for determining whether or not the conditions for automatic photography are satisfied.

[0030] Fig. 4 is an image diagram of a target surface 211f of the sprocket 211 and a plurality of point groups on the target surface, displayed on a display device 140 such as a viewfinder of the imaging device 110. Fig. 5 is an image diagram showing an example of the sprocket 211 and guide lines GL displayed on the display device 140. For example, as shown in Fig. 4, the image processing device 150 sets a first point group G1, a second point group G2, and a third point group G3 that are spaced apart from each other on the target surface 211f of the sprocket 211 that is the evaluation target of the condition evaluation device 100.

[0031] The first point group G1, the second point group G2, and the third point group G3 each include a plurality of points p. The first point group G1, the second point group G2, and the third point group G3 may each include n points p, where n is a natural number equal to or greater than 2. The image processing device 150 can easily change the value of n, which is the number of points p, the position of each point group, and the position of the point p included in each point group, depending on the application.

[0032] The number of points p included in each of the first point group G1, the second point group G2, and the third point group G3 is not particularly limited, but may be, for example, 2 or more, 3 or more, 5 or more, or 10 or more. The distance between any two points p included in any one point group is closer than the distance between any one point p included in one of the two point groups and any one point p included in the other point group. In addition, the image processing device 150 sets the positions of the multiple points p included in each point group so that three points p selected one by one from each point group do not exist on a straight line.

[0033] The distance sensor 120 measures the distance to each point p included in each of the first point group G1, the second point group G2, and the third point group G3, and outputs the distance to the image processing device 150. In addition, the imaging device 110 or the distance sensor 120, for example, as described above, irradiates a visible light laser from a light source to visualize multiple points p on the target surface 211f to which the distance is to be measured.

[0034] In process P5 for determining whether the conditions for automatic photography are satisfied, image processing device 150 displays guide lines GL on display device 140, for example, as shown in the upper diagram of Fig. 5. The operator adjusts the orientation of imaging device 110 so that a reference point, such as the tip of a tooth of sprocket 211, is positioned on guide lines GL, for example, as shown in the lower diagram of Fig. 5. More specifically, image processing device 150 detects the outline of sprocket 211 on guide lines GL as a reference point, for example, based on distance information obtained from distance sensor 120.

[0035] In this case, in process P5, when the image processing device 150 determines that the number of reference points on the guide line GL has decreased from three or more to two, as shown in Fig. 5, it determines that the condition is satisfied (YES). Then, the image processing device 150 executes process P8, for example, to acquire the coordinates of the reference points of the sprocket 211 from the distance sensor 120 and to capture an image of the target surface 211f of the sprocket 211, which is the subject, with the imaging device 110.

[0036] On the other hand, in the process P2 for determining whether or not the secondary rotation is performed, if the presence or absence of the secondary rotation is "no" (NO), the image processing device 150 executes a process P6 for displaying a plurality of point clouds including the first point cloud G1, the second point cloud G2, and the third point cloud G3, for example. In this process P6, the image processing device 150 irradiates a visible light laser from a light source such as the imaging device 110 or the distance sensor 120, and displays a plurality of points p of each of the first point cloud G1, the second point cloud G2, and the third point cloud G3 on the target surface 211f on the display device 140. Furthermore, the image processing device 150 executes a process P7 for determining whether or not the operator has performed a photographing operation.

[0037] The operator performs a manual photographing operation, for example, by adjusting the orientation of the imaging device 110 so that the first point group G1, the second point group G2, and the third point group G3 displayed on the display device 140 are appropriately positioned on the target surface 211f, and by pressing the shutter button of the imaging device 110. When the image processing device 150 determines that the operator has performed an imaging operation (YES), it executes process P8 in which the imaging device 110 captures an image of the target surface 211f of the sprocket 211. When the image processing device 150 determines in process P7 that the operator has not performed an imaging operation (NO), it repeats the determination process P7.

[0038] Furthermore, in the process P4 for determining whether automatic photography is performed or not, if the determination is "no" (NO) in the process P4 for determining whether automatic photography is performed or not, the image processing device 150 executes a process P7 for determining whether an operator has performed a photography operation or not. The operator performs a manual photography operation by adjusting the orientation of the imaging device 110 so that the guide line GL displayed on the display device 140 such as the viewfinder of the imaging device 110 in the process P3 is positioned on the reference point of the tooth tip of the sprocket 211, for example.

[0039] If the image processing device 150 determines that the operator has performed an imaging operation (YES), it executes process P8 to capture an image of the target surface 211f of the sprocket 211 using the imaging device 110, and if it determines that the operator has not performed an imaging operation (NO), it repeats the determination process P7.

[0040] Next, the image processing device 150 executes a process P9 for displaying, as shown in Fig. 4, an image of the target surface 211f of the sprocket 211 captured in process P8, the first point group G1, the second point group G2, and the third point group G3 superimposed on the screen of the display device 140. Furthermore, the image processing device 150 executes a process P10 for acquiring, for example, the success or failure of the imaging.

[0041] In this process P10, the image processing device 150 causes the display device 140 to display a confirmation display such as characters, icons, images, etc. for confirming whether imaging has been successful. The operator visually confirms, for example, each point p of the first point group G1, the second point group G2, and the third point group G3 visualized on the display device 140, and the confirmation display. The operator inputs the success or failure of imaging to the input device 130, for example, by pressing a touch panel in accordance with the confirmation display on the display device 140.

[0042] More specifically, for example, when each point p is located on the target surface 211f of the sprocket 211, the operator inputs "Yes" to the input device 130, indicating success of imaging, and otherwise inputs "No" to the input device 130, indicating failure of imaging. The image processing device 150 acquires from the input device 130 the success or failure of imaging input by the operator via the input device 130.

[0043] Next, the image processing device 150 executes a process P11 for determining whether the imaging obtained from the input device 130 is successful or not. In this process P11, if the image processing device 150 determines that the imaging obtained in the process P10 is not successful (NO), the process returns to the above-mentioned process P2. On the other hand, in this process P11, if the image processing device 150 determines that the imaging obtained in the process P10 is successful (YES), the process executes a process P12 for calculating a first mean plane.

[0044] 6 is a conceptual diagram of the first plane group PG1 calculated in process P12 and the unit normal vectors NV1, NV2, . . . , NVn of the planes included in the first plane group PG1. In process P12, the image processing device 150 acquires, for example, the distance from the distance sensor 120 to each point p in the first point group G1 shown in FIG. 4, the distance to each point p in the second point group G2, and the distance to each point p in the third point group G3.

[0045] The image processing device 150, for example, sets an orthogonal coordinate system consisting of X-axis, Y-axis, and Z-axis with the distance sensor 120 as the origin, and calculates the coordinates of each point p. Furthermore, the image processing device 150 executes a plane calculation process for calculating a plane based on one point p selected from the first point group G1, one point p selected from the second point group G2, and one point p selected from the third point group G3, for all combinations of points p. This allows the image processing device 150 to calculate the first plane group PG1 as shown in FIG. 6.

[0046] 7 is an explanatory diagram of the plane calculation process executed by the image processing device 150 in the process P12. For example, the coordinates of an arbitrary point p1 included in the first point group G1 are (x1, y1, z1), the coordinates of an arbitrary point p2 included in the second point group G2 are (x2, y2, z2), and the coordinates of an arbitrary point p3 included in the third point group G3 are (x3, y3, z3). If the equation of the plane passing through these points p1, p2, and p3 is ax+by+cz+d=0, the unit normal vector NV of this plane is parallel to the vector V=(a, b, c).

[0047] Here, if vector α = (x, y, z) = (x2-x1, y2-y1, z2-z1) and vector α' = (x', y', z') = (x3-x1, y3-y1, z3-z1), their cross product is α×α' = (yz'-y'z, zx'-z'x, xy'-x'y). The cross product α×α' of vector α and vector α' is parallel to the unit normal vector NV of the above plane.

[0048] Therefore, if (a, b, c) = α × α', then d = -(ax1 + by1 + cz1) (∵ax1 + by1 + cz1 + d = 0). From the above, the equation of the above plane can be found, and the unit normal vector NV is (α × α') / |α × α'|. Note that -1 times this unit normal vector NV also becomes the unit normal vector on the opposite side of the above plane, but if the first point group G1, the second point group G2, and the third point group G3 are fixed, then n 3 All of the unit normal vectors NV are aligned on the same side with respect to the plane.

[0049] Next, the image processing device 150 executes a process P13 in which the unevenness of the target surface 211f of the sprocket 211 is displayed in the color of each point p. Fig. 8 is a conceptual diagram of the first mean plane H1 and the mean vector V1 calculated by the image processing device 150 in this process P13.

[0050] In this process P13, the image processing device 150 first calculates a first average plane H1. The first average plane H1 is a plane whose normal vector is an average vector V1 obtained by averaging the unit normal vectors NV of all planes in the first plane group PG1, and whose sum of distances between the first average plane H1 and all points p included in the first point group G1, the second point group G2, and the third point group G3 is the smallest. The number of all points p included in the first point group G1, the second point group G2, and the third point group G3 is, for example, 3n. If the formula of the first average plane H1 is ax+by+cz+d=0, a, b, and c can be obtained from the above average vector, and d can be easily calculated by the least absolute value method or the like.

[0051] Furthermore, in process P13, the image processing device 150 acquires a threshold value that is set in advance or that is newly set in process P1 and stored in the storage device 153. As described above, this threshold value is the threshold value of the height h of the unevenness with respect to the first mean plane H1. Based on the coordinates of each point p in the first point group G1, the second point group G2, and the third point group G3, the image processing device 150 recognizes, as a bulge point, a point p whose distance from the distance sensor 120 is closer than the first mean plane H1 and whose height h protruding from the first mean plane H1 is equal to or greater than the threshold value Th [mm].

[0052] Furthermore, the image processing device 150 recognizes, based on the coordinates of each point p of the first point group G1, the second point group G2, and the third point group G3, a point whose distance from the distance sensor 120 is farther than the first mean plane H1 and whose height h of the point depressed from the first mean plane H1 is equal to or greater than a threshold value Td [mm], as a depressed point. For example, the image processing device 150 causes the display device 140 to display the protruding points, the depressed points, and other points in different colors. For example, the image processing device 150 causes the display device 140 to display the heights at which the protruding points protrude from the first mean plane H1 and the heights at which the depressed points sink from the first mean plane H1.

[0053] Next, the image processing device 150 executes a process P14 for acquiring operation contents. In this process P14, the image processing device 150 causes the display device 140 to display, for example, images, characters, icons, or the like indicating options for the operation contents. The options for the operation contents are, for example, "take a picture again," "change the threshold value," and "proceed to the next process." The image processing device 150 acquires the operation contents input by the operator via the input device 130, for example, a touch panel.

[0054] For example, if the operator recognizes that the raised point displayed on the display device 140 in process P13 is due to a removable foreign object such as mud adhering to the target surface 211f of the sprocket 211, the operator can select the option of "retake image" after removing the foreign object. Also, if the target surface of the subject of the image capture device 110 is, for example, part of the surface of a large-diameter sphere that can be regarded as a pseudo-flat surface, it is possible that a point p on the target surface will be recognized as a raised point or a depressed point even if there are no bumps or depressions. In such a case, the operator can select the option of "proceed to next process" to use all points p on the target surface without excluding points p recognized as raised points or depressed points.

[0055] Next, the image processing device 150 executes a process P15 for determining whether or not to perform re-shooting. In this process P15, if the image processing device 150 determines that the operation content acquired in the above-mentioned process P14 is "re-shooting" (YES), the process returns to the above-mentioned process P2. Also, in this process P15, if the image processing device 150 determines that the operation content acquired in the above-mentioned process P14 is not "re-shooting" (NO), the process executes the next process P16.

[0056] In process P16, the image processing device 150 judges whether or not the operation content acquired in the above-mentioned process P14 is "changing the threshold value". If the image processing device 150 judges that the operation content is "changing the threshold value" (YES), it executes process P17 to acquire a threshold value, as in the above-mentioned process P1, and returns to the above-mentioned process P13. If the image processing device 150 judges that the operation content is not "changing the threshold value" (NO), it executes process P18 to acquire the presence or absence of exclusion.

[0057] In process P18, the image processing device 150, for example, displays "normal points" and "all points" as options on the display device 140. The operator selects an option via the input device 130, for example, by pressing a touch panel. The image processing device 150 acquires the option input by the operator from the input device 130.

[0058] Next, the image processing device 150 executes a process P19 to determine whether or not there is exclusion. If the option of "all points" is acquired in the above-mentioned process P18, the image processing device 150 determines in the process P19 that there is no exclusion (NO), and executes a process P20 to acquire the first mean plane H1 calculated in the above-mentioned process P12 and the mean vector V1 which is its normal vector. In this case, the image processing device 150 uses the first mean plane H1 in the process P22 to calculate the next rotation angle.

[0059] On the other hand, if the option "normal point" is acquired in the above-mentioned process P18, the image processing device 150 determines in process P19 that there is exclusion (YES), and executes process P21 for calculating the second mean plane H2.

[0060] Fig. 9 is a conceptual diagram of a process P21 for calculating the second plane group PG2. The second plane group PG2 is calculated in the same manner as the first plane group PG1 calculated in the above-mentioned process P12, except that points p recognized as protruding points or depressed points in the above-mentioned process P13 are excluded and other points p are used. By excluding protruding points and depressed points in this way, for example, a plane having a unit normal vector NVn whose angle is significantly different from the unit normal vectors NV1, NV2, and NV3 of the other planes in the first plane group PG1 shown in Fig. 6 can be excluded from the second plane group PG2 shown in Fig. 9.

[0061] Furthermore, in process P21, the image processing device 150 calculates a second mean plane H2 in the same manner as the first mean plane H1 calculated in the above-mentioned process P13. In this case, the image processing device 150 uses the second mean plane H2 in process P22 for calculating the next rotation angle. The second mean plane H2 is a plane whose normal vector is an average vector V2 obtained by averaging the unit normal vectors NV of all the planes in the second plane group PG2, and which has the smallest sum of distances to the remaining points p, excluding the protruding points and the depressed points, from all the points p in the first point group G1, the second point group G2, and the third point group G3.

[0062] Next, the image processing device 150 executes a process P22 of calculating a rotation angle for converting the first mean plane H1 or the second mean plane H2, which is not a front-facing image, into a front-facing image. Hereinafter, this process P22 will be described with reference to Figs. 10 to 14. Here, the front-facing image is, for example, an image facing the imaging surface of the imaging device 110. Also, the imaging surface is, for example, the light receiving surface of an imaging element, which is a surface perpendicular to the optical axis of the lens of the imaging device 110.

[0063] The image processing device 150, for example, determines a point (x', y', z') on the first mean plane H1 or the second mean plane H2 corresponding to all coordinates (x, y) of the image G captured by the imaging device 110 as shown in Fig. 10. When it is assumed that there is no distortion of the lens of the imaging device 110, this point (x', y', z') may be determined based on a vector from (x, y) toward the lens. Furthermore, the point (x', y', z') may be determined taking into account the distortion of the lens of the imaging device 110.

[0064] The first average plane H1 or the second average plane H2, which is not a frontal image, is expressed by the formula: x +b y +c z +d=0, and (a,b,c)≠(0,0,c). In this case, for example, as shown in Figure 11, the coordinate system is re-established as z←zd / c, and a,b,c are normalized to the equation: ax+by+cz+d=0 and a 2 +b 2 +c 2 There is no loss of generality in considering it as a plane passing through the origin represented as =1. Any point p on this plane is (c,0,-a) ⊥(a,b,c), (0,c,-b) ⊥(a,b,c), Using the parameters s and t, it is expressed by the following equation (1).

[0065] p(x,y)=s(c,0,-a)+t(0,c,-b) ···(1)

[0066] This constitutes the plane of the double triangle shown in FIG. If this point p is rotated around the y-axis by a rotation angle θ, where θ=arctan(a / c), then point p' expressed by the following equation (2) is obtained.

[0067] p'(x',y')=s(c·secθ,0,0) +t(b·sinθ,c,-b·cosθ) ···(2)

[0068] This point p' includes the x-axis (it becomes the x-axis when t = 0) and forms the plane of the double triangle shown in Figure 12. When this point p' is rotated around the x-axis by a rotation angle φ, as φ = arctan(b cosθ / c), we obtain point p" expressed by the following equation (3).

[0069] p"(x",y")=s(c·secθ,0,0) +t(b·sinθ,b·cosθ·sinφ+c·cosφ,-b·cosθ) =(s·D+t·a·b / D,t·c / D,0) ···(3)

[0070] In the formula (3), D = (a 2 +c 2 ) 1 / 2 Since this point p" coincides with the xy plane (z coordinate is always 0), it is a front-facing image and forms the double triangular plane shown in FIG. 13. As described above, in process P22, the image processing device 150 calculates the rotation angles θ and φ for converting the first mean plane H1 or the second mean plane H2 into a front-facing image by rotating the first mean plane H1 or the second mean plane H2 so as to be parallel to the imaging surface of the imaging device 110.

[0071] Next, the image processing device 150 executes a process P23 for calculating a front-facing image, as shown in Fig. 3B. In this process P23, the image processing device 150 obtains a point (x", y", z") corresponding to the total coordinates (x, y) of the image G captured by the imaging device 110 on the first mean plane H1 or the second mean plane H2 rotated so as to be parallel to the imaging surface of the imaging device 110, as shown in Fig. 14.

[0072] To convert the image of the target surface 211f of the sprocket 211, which is not a front-facing image, into a front-facing image, the color of the pixel at coordinates (x, y) is transferred to the pixel at coordinates (x", y"). The color of a pixel is generally represented by the numerical values ​​of each of the three primary colors (R, G, B). Even if x and y are positive values, this does not necessarily mean that x", y" will be positive values.

[0073] In the example shown in FIG. 15, for example, the area of ​​the pixel with coordinates (x, y) in the pre-conversion image corresponds to the area of ​​the coordinates (x", y") in the post-conversion image. This area of ​​coordinates (x", y") in the post-conversion image spans, for example, four pixels in the post-conversion image. In this case, the value obtained by multiplying the numerical values ​​of the three primary colors (R, G, B) of the pixel in the pre-conversion image by the percentage of the area that the pixel in the pre-conversion image overlaps with each pixel in the post-conversion image is added to the numerical values ​​of the three primary colors (R, G, B) of each pixel in the post-conversion image.

[0074] More specifically, in the example shown in FIG. 15, the area of ​​coordinates (x", y") in the converted image overlaps with ¼ of the area of ​​each of the four pixels in the converted image. Also, the numerical values ​​of the three primary colors of the pixel at coordinates (x, y) in the pre-conversion image are, for example, (R, G, B) = (128, 64, 192). In this case, (R, G, B) = (32, 16, 48), which is ¼ of the numerical values ​​of the three primary colors of the pixel at coordinates (x, y) in the pre-conversion image, is added to the numerical values ​​of the three primary colors (R, G, B) of each of the four pixels in the converted image. This makes it possible to determine the numerical values ​​of the three primary colors of the four pixels in the converted image.

[0075] As described above, in process P23 for calculating a frontal image, the image processing device 150 makes the numerical values ​​of the three primary colors of the pixel at coordinates (x, y) in the image before conversion correspond to the numerical values ​​of the three primary colors of the pixel in the image after conversion. This makes it possible to generate a frontal image in which the first mean plane H1 or the second mean plane H2 is directly opposed to the imaging plane of the imaging device 110 from the image of the target surface 211f of the sprocket 211 captured by the imaging device 110.

[0076] Next, the image processing device 150 executes a process P24 for determining whether or not a secondary rotation is to be performed, as shown in Fig. 3B, for example. In this process P24, if the presence or absence of secondary rotation stored in advance in the storage device or input in the above-mentioned process P1 is "no", the image processing device 150 determines that a secondary rotation is not to be performed (NO), and ends the process shown in Fig. 3A and Fig. 3B.

[0077] Furthermore, in process P24, if the presence or absence of secondary rotation, which is pre-stored in the storage device or input in the above-mentioned process P1, is "yes", the image processing device 150 determines that secondary rotation will be performed (YES), and executes process P25 to calculate a front-on image.

[0078] FIG. 16 is a plan view of a portion of sprocket 211 for explaining the necessity of secondary rotation, and a side view of two types of sprockets 211A and 211B.

[0079] The sprocket 211 of the work machine 200 shown in Fig. 2 may be, for example, one of two types of sprockets 211A and 211B as shown in Fig. 16. One of the sprockets, 211A, has, for example, a tooth width at the tooth tip 211t that is approximately equal to a tooth width at the tooth bottom 211b, while the other sprocket, 211B, has a tooth width at the tooth tip 211t that is narrower than the tooth width at the tooth bottom 211b.

[0080] For sprocket 211A, where the tooth width at tooth tip 211t is roughly equal to the tooth width at tooth bottom 211b, an image is taken using a plane perpendicular to the rotation axis as target surface 211f, and the image is converted into a front-on image. This allows the condition of the teeth of sprocket 211, which are most susceptible to wear, to be appropriately evaluated.

[0081] On the other hand, in the case of sprocket 211B, where the tooth width at tooth tip 211t is narrower than the tooth width at tooth bottom 211b, even if an image is taken with a plane perpendicular to the rotation axis as target surface 211f and the image is converted into a front-on image, the side surface of the tooth is inclined at a predetermined angle Ψ with respect to target surface 211f. Therefore, in this state, it is not possible to properly evaluate the condition of the teeth of sprocket 211B, which are most susceptible to wear.

[0082] Therefore, for sprocket 211B, where the tooth width of tooth tip 211t is narrower than the tooth width of tooth bottom 211b, it is necessary to convert the image of target surface 211f perpendicular to the rotation axis into a frontal image, and then perform a secondary rotation in which the frontal image is rotated by a predetermined angle Ψ. This makes it possible to obtain a frontal image of target surface 211f' by using the side of the tooth of sprocket 211B as a new target surface 211f'.

[0083] More specifically, the image of target surface 211f of sprocket 211B shown in the lower diagram of Fig. 5, captured in the above-mentioned process P8, is converted into a front-facing image as shown in the upper diagram of Fig. 17 through the above-mentioned processes P12 to P23. Here, as shown in Fig. 17, an orthogonal coordinate system is set having an x-axis with the right direction of the front-facing image as the positive direction, a y-axis with the upward direction of the front-facing image as the positive direction, and a z-axis perpendicular to the front-facing image.

[0084] In process P25, the image processing device 150 rotates the frontal image by a predetermined angle λ around the z-axis so that the guide line GL passing through the reference point of the tooth tip of the sprocket 211B shown in the upper diagram of Fig. 17 becomes parallel to the x-axis, thereby obtaining a frontal image of the target surface 211f as shown in the center of Fig. 17. The image processing device 150 further rotates this frontal image of the target surface 211f as shown in the center of Fig. 17 by a predetermined angle Ψ around the x-axis.

[0085] 17, a front-facing image after the secondary rotation can be obtained in which the side surface of the tooth of sprocket 211B is set as target surface 211f' and target surface 211f' faces the imaging surface of imaging device 110. With the above, process P25 ends, and image processing device 150 ends the processes shown in FIGS. 3A and 3B. The front-facing image may be stored in storage device 153.

[0086] The operation of the image processing device 150 of this embodiment will be described below.

[0087] For example, in order to evaluate the condition of a mechanical part based on an image, such as measuring the amount of wear of the sprocket 211 of the work machine 200, a front-facing image of the mechanical part is necessary from the viewpoint of improving the evaluation accuracy. However, the mechanical part may have unevenness on the target surface 211f, such as the sprocket 211, or foreign matter such as mud or oil may adhere to the target surface 211f. When such a mechanical part is used as a subject, the conventional image processing device may have difficulty in measuring the orientation of the mechanical part based on the imaging means, and may not be able to correct distortion caused by oblique shooting of the image captured by the imaging means.

[0088] In contrast, the image processing device 150 of this embodiment is a device that acquires an image of the target surface 211f from the imaging device 110 and acquires the distance from the target surface 211f from the distance sensor 120 to convert the image of the target surface 211f into a frontal image, and has the following functions. The image processing device 150 sets a first point group G1, a second point group G2, and a third point group G3 that are spaced apart from each other on the target surface 211f, and acquires the distance from the distance sensor 120 to each point p of the first point group G1, the distance to each point p of the second point group G2, and the distance to each point p of the third point group G3. The image processing device 150 also executes a plane calculation process for calculating a plane based on one point p selected from the first point group G1, one point p selected from the second point group G2, and one point p selected from the third point group G3 for all combinations of points to calculate a first plane group PG1. The image processing device 150 also calculates a first average plane H1 that has a minimum sum of distances to all points p included in the first point group G1, the second point group G2, and the third point group G3, and that uses an average vector V1 obtained by averaging unit normal vectors NV of all planes in the first plane group PG1 as a normal vector. The image processing device 150 then converts the image of the target surface 211f into a frontal image based on the first average plane H1.

[0089] With this configuration, the image processing device 150 of this embodiment can convert an image of the target surface 211f having unevenness of a machine part such as a sprocket 211, which is a subject, photographed obliquely into a front-facing image in which the target surface 211f is directly faced to the imaging plane of the imaging device 110. More specifically, as described above, the image processing device 150 of this embodiment calculates a first plane group PG1 by setting a first point group G1, a second point group G2, and a third point group G3 that are spaced apart from each other on the target surface 211f, and further calculates a first average plane H1 from the first plane group PG1. As a result, the influence of local unevenness or foreign matter on the target surface 211f is diluted, reduced, or removed, and a first average plane H1 that is substantially parallel to the target surface 211f can be obtained. By converting the image of the target surface 211f into a front-facing image based on this first average plane H1, an image of the target surface 211f photographed obliquely into a front-facing image can be converted.

[0090] In addition, the image processing device 150 of this embodiment sets n points in each of the first point group G1, the second point group G2, and the third point group G3, and calculates n points by a plane calculation process. 3 With this configuration, the image processing device 150 of this embodiment calculates a first plane group PG1 including planes p. Since the first point group G1, the second point group G2, and the third point group G3 each have an equal number of points p, the image processing device 150 of this embodiment can calculate a more average first plane group PG1.

[0091] Moreover, the image processing device 150 of this embodiment displays the image of the target surface 211f acquired from the imaging device 110 together with the first point group G1, the second point group G2, and the third point group G3 on the display device 140. With such a configuration, the image processing device 150 of this embodiment enables the operator to adjust the positions of the first point group G1, the second point group G2, and the third point group G3, and to determine whether imaging by the imaging device 110 has been successful.

[0092] Furthermore, the image processing device 150 of this embodiment displays points p whose distances to the first mean plane H1 are equal to or greater than a threshold in different colors from other points p on the display device 140. With this configuration, the image processing device 150 of this embodiment enables the operator to adjust the positions of the first point group G1, the second point group G2, and the third point group G3, and to determine whether or not imaging by the imaging device 110 has been successful.

[0093] Moreover, the image processing device 150 of this embodiment excludes points p whose distance to the first mean plane H1 is equal to or greater than a threshold value from the first point group G1, the second point group G2, and the third point group G3, and performs a plane calculation process for all combinations of the remaining points p to calculate the second plane group PG2. The image processing device 150 also calculates a second mean plane H2 that uses an average vector V2 obtained by averaging the unit normal vectors NV of all planes in the second plane group PG2 as a normal vector and minimizes the sum of distances to the remaining points p. Then, the image processing device 150 calculates the rotation angles θ, φ of the second mean plane H2 so that the normal vector of the second mean plane H2 is parallel to the normal vector of the image plane of the frontal image, and converts the image of the target surface 211f into a frontal image based on the rotation angles θ, φ. With this configuration, the image processing device 150 of this embodiment can convert the image of the target surface 211f into a frontal image by removing the effects of local unevenness and foreign matter on the target surface 211f.

[0094] Moreover, the image processing device 150 of this embodiment acquires a threshold value of the distance to the first mean plane H1 from the input device 130. With such a configuration, the image processing device 150 of this embodiment allows the operator to change the threshold value according to the situation.

[0095] Moreover, the image processing device 150 of this embodiment calculates the rotation angles θ, φ of the first mean plane H1 so that the normal vector of the first mean plane H1 is parallel to the normal vector of the image plane of the frontal image, and converts the image of the target surface 211f into a frontal image based on the rotation angles θ, φ. With this configuration, the image processing device 150 of this embodiment can calculate the first mean plane H1 using all the points p included in the first point group G1, the second point group G2, and the third point group G3, and further convert the image of the target surface 211f into a frontal image based on the first mean plane H1.

[0096] Moreover, the image processing device 150 of this embodiment obtains an image of the target surface 211f', which is an inclined surface inclined with respect to the target surface 211f, from the imaging device 110, and obtains the distance from the target surface 211f', which is an inclined surface, from the distance sensor 120 to convert the image of the target surface 211f' into a front-facing image. With this configuration, the image processing device 150 of this embodiment can obtain a front-facing image of an inclined surface inclined with respect to the target surface 211f, such as the side of a tooth of a sprocket 211B as shown in the lower diagram of Fig. 16, and can more accurately evaluate the state of wear, etc.

[0097] The mechanical part condition evaluation device 100 of this embodiment includes an imaging device 110, a distance sensor 120, and the image processing device 150 described above. With this configuration, the condition evaluation device 100 of this embodiment can achieve the same effect as the image processing device 150 described above, and can convert an image of the target surface 211f having the unevenness of the subject, photographed obliquely, into a frontal image in which the target surface is directly opposed to the imaging surface. In addition, when the mechanical part condition evaluation device 100 includes a storage device 153, as described above, it is possible to store the frontal image in the storage device 153, and when a frontal image is photographed thereafter, it is possible to calculate the degree of wear by comparing the difference with the past frontal image.

[0098] As described above, according to this embodiment, it is possible to provide an image processing device 150 that can convert an image of the uneven target surface 211f of a subject captured obliquely into a front-on image in which the target surface is directly opposed to the imaging surface of the imaging device 110.

[0099] The above describes in detail an embodiment of an image processing device according to the present disclosure using the drawings, but the specific configuration is not limited to this embodiment, and even if there are design changes, etc., within the scope that does not deviate from the gist of the present disclosure, they are included in the present disclosure. [Explanation of symbols]

[0100] 110 Imaging device 120 Distance Sensor 130 Input Device 140 Display device 150 Image Processing Device 211f Target surface 211f' Target surface (inclined surface) G1 First point group G2 2nd point group G3 3rd point group H1 1st mean plane H2 2nd mean plane NV unit normal vector NV1 unit normal vector NV2 unit normal vector NV3 unit normal vector NVn unit normal vector p point PG1 1st plane group PG2 2nd plane group V1 Mean Vector V2 Mean Vector θ Rotation angle φ rotation angle

Claims

1. 1. An image processing device that acquires an image of a target surface of an object from an imaging device, acquires a distance between the imaging device and the target surface from a distance sensor, and converts the image of the target surface into a frontal image, setting a first point group, a second point group, and a third point group on the target surface such that the first point group, the second point group, and the third point group are spaced apart from each other and each point selected from the first point group is not on the same line; n points (n is a natural number equal to or greater than 2) are set in each of the first point group, the second point group, and the third point group; acquiring, from the distance sensor, a distance from the imaging device to each point of the first point group, a distance from the imaging device to each point of the second point group, and a distance from the imaging device to each point of the third point group; calculating a first group of planes by executing a plane calculation process for calculating a plane based on one point selected from the first group of points, one point selected from the second group of points, and one point selected from the third group of points for all combinations of points; The first plane group including n 3 planes is calculated by the plane calculation process; calculating a first average plane that is a normal vector obtained by averaging unit normal vectors of all planes in the first plane group, and that minimizes the sum of distances between the first point group, the second point group, and all points included in the third point group; 2. An image processing device comprising: a first mean plane that converts an image of the target surface into the front-facing image based on the first mean plane.

2. The image processing apparatus according to claim 1 , further comprising a display device that displays the image of the target surface acquired from the imaging device together with the first point cloud, the second point cloud, and the third point cloud.

3. 3. The image processing apparatus according to claim 2, wherein points whose distance to said first mean plane is equal to or greater than a threshold value are displayed in different colors on said display device from other points.

4. excluding points whose distances to the first mean plane are equal to or greater than a threshold value from the first point group, the second point group, and the third point group, and performing the plane calculation process on all combinations of remaining points to calculate a second plane group; a second average plane having a minimum sum of distances to the remaining points is calculated by averaging the unit normal vectors of all the planes in the second group of planes as a normal vector; Calculating a rotation angle of the second mean plane so that the normal vector of the second mean plane is parallel to a normal vector of an image plane of the frontal image; 4. The image processing device according to claim 1, wherein the image of the target surface is converted into the front-facing image based on the rotation angle.

5. Calculating a rotation angle of the first mean plane so that the normal vector of the first mean plane is parallel to a normal vector of an image plane of the frontal image; The image processing apparatus according to claim 1 , wherein the image of the target surface is converted into the front-facing image based on the rotation angle.

Citation Information

Patent Citations

  • Image processor and image processing method

    JP2000307947A

  • Circumferential environment recognition device, autonomous mobile system using the same, and circumferential environment recognition method

    JP2014194729A

  • Image processing apparatus, image processing method and program

    JP2019192156A

  • Image-processing device, image-processing method, and image-processing program

    WO2014061372A1