Work machine

Abstract

To provide a work machine capable of detecting the position of a detection target with high accuracy.SOLUTION: An imaging device 4 captures an image of a detection target (27) so as to include a mark 71 whose position and orientation can be acquired externally. Acquisition means (11) acquires the position and orientation of the mark 71 relative to the imaging device 4 from the appearance of the mark 71 in the captured image. A storage device 13 stores information on the three-dimensional shape of the detection target (27). Calculation means (11) calculates the position and orientation of the three-dimensional shape of the detection target (27) based on the acquired position and orientation of the mark 71 and the stored information on the three-dimensional shape of the detection target (27).SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a work machine that detects the position of a detection target. [Background technology]

[0002] As disclosed in Patent Document 1, an image of a target (such as a dump truck) onto which a load (such as soil and sand) held by a work machine is dropped is obtained, and the position of a detection target (such as a loading platform) on the target shown in the image is identified based on a trained position identification model and the captured image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2020-35380 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, image recognition using a trained model lacks accuracy in identifying the location of a detected object.

[0005] An object of the present invention is to provide a position detection system capable of detecting the position of a detection target with high accuracy. [Means for solving the problem]

[0006] The present invention is a position detection system that detects the position of a detection target relative to a work machine, and is characterized in having: a mark that is provided on the detection target and whose position and attitude can be acquired externally; an imaging device that images the detection target so that the mark is included; acquisition means that acquires the position and attitude of the mark from the imaged mark; a storage device that stores information on the three-dimensional shape of the detection target; and calculation means that calculates the position and attitude of the detection target based on the acquired position and attitude of the mark and the stored information on the three-dimensional shape of the detection target. Effect of the Invention

[0007] According to the present invention, the position and orientation of the detection target are calculated based on the position and orientation of the mark acquired from the image of the mark and the three-dimensional shape information of the detection target stored in the storage device. This makes it possible to detect the position of the detection target with high accuracy. Therefore, for example, it is possible to assist an operator who operates a work machine so as to drop a transported object onto the detection target, or to automatically operate a work machine that operates to drop a transported object onto the detection target, with high accuracy. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a configuration diagram of a position detection system. [Diagram 2] FIG. 2 is a perspective view of the dump truck as viewed from the work machine side. [Diagram 3] FIG. 2 is a circuit diagram of the position detection system. [Figure 4] 13 is a flowchart of a position detection process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

[0010] (Configuration of location detection system) The position detection system according to this embodiment detects the position of a detection target relative to a work machine. As shown in Fig. 1, which is a configuration diagram of the position detection system 1, the position detection system 1 has a work machine 2, a dump truck 3, an imaging device 4, and a measuring device 5.

[0011] (Work machine configuration) 1, the work machine 2 is a machine that performs work using an attachment 30, and is, for example, a hydraulic excavator. The work machine 2 has a lower traveling body 21, an upper rotating body 22, a rotating device 24, the attachment 30, and a cylinder 40.

[0012] The lower traveling structure 21 is a portion that allows the work machine 2 to travel, and is equipped with, for example, crawlers. The upper rotating structure 22 is rotatably attached to the upper part of the lower traveling structure 21. A cab (operator's compartment) 23 is provided at the front part of the upper rotating structure 22. The rotating device 24 is capable of rotating the upper rotating structure 22.

[0013] The attachment 30 is attached to the upper rotating body 22 so as to be rotatable in the vertical direction. The attachment 30 includes a boom 31, an arm 32, and a bucket 33. The boom 31 is attached to the upper rotating body 22 so as to be rotatable in the vertical direction. The arm 32 is attached to the boom 31 so as to be rotatable in the vertical direction. The bucket 33 is attached to the arm 32 so as to be rotatable in the vertical direction.

[0014] The bucket 33 is a part that excavates, holds, drops, etc., the transported soil and sand. Note that the bucket 33 is one example of a tip attachment that is attached to the arm 32, and the tip attachment is not limited to this and may be a nibbler, a clamp arm, etc. Also, the transported material is not limited to soil and sand and may be rubble, scrap iron, gravel, etc.

[0015] The cylinder 40 is capable of hydraulically rotating the attachment 30. The cylinder 40 is a hydraulic telescopic cylinder. The cylinder 40 includes a boom cylinder 41, an arm cylinder 42, and a bucket cylinder 43.

[0016] The boom cylinder 41 drives and rotates the boom 31 relative to the upper rotating body 22. A base end of the boom cylinder 41 is rotatably attached to the upper rotating body 22. A tip end of the boom cylinder 41 is rotatably attached to the boom 31.

[0017] The arm cylinder 42 drives the arm 32 to rotate relative to the boom 31. A base end of the arm cylinder 42 is rotatably attached to the boom 31. A tip end of the arm cylinder 42 is rotatably attached to the arm 32.

[0018] The bucket cylinder 43 rotates the bucket 33 relative to the arm 32. A base end of the bucket cylinder 43 is rotatably attached to the arm 32. A tip end of the bucket cylinder 43 is rotatably attached to a link member 34 that is rotatably attached to the bucket 33.

[0019] The work machine 2 also has an angle sensor 52 and a tilt angle sensor 60 .

[0020] The angle sensor 52 detects the rotation angle of the upper rotating body 22 with respect to the lower traveling body 21. The angle sensor 52 is, for example, an encoder, a resolver, or a gyro sensor. In this embodiment, the rotation angle of the upper rotating body 22 when the front of the upper rotating body 22 coincides with the front of the lower traveling body 21 is set to 0°.

[0021] The inclination angle sensor 60 detects the attitude of the attachment 30. The inclination angle sensor 60 includes a boom inclination angle sensor 61, an arm inclination angle sensor 62, and a bucket inclination angle sensor 63.

[0022] The boom inclination angle sensor 61 is attached to the boom 31 and detects the attitude of the boom 31. The boom inclination angle sensor 61 is a sensor that acquires the inclination angle of the boom 31 with respect to the horizontal line, and is, for example, an inclination (acceleration) sensor. Note that the boom inclination angle sensor 61 may be a rotation angle sensor that detects the rotation angle of the boom foot pin (boom base end) or a stroke sensor that detects the stroke amount of the boom cylinder 41.

[0023] The arm inclination angle sensor 62 is attached to the arm 32 and detects the posture of the arm 32. The arm inclination angle sensor 62 is a sensor that obtains the inclination angle of the arm 32 with respect to the horizontal line, and is, for example, an inclination (acceleration) sensor. Note that the arm inclination angle sensor 62 may be a rotation angle sensor that detects the rotation angle of the arm connecting pin (base end of the arm) or a stroke sensor that detects the stroke amount of the arm cylinder 42.

[0024] The bucket inclination angle sensor 63 is attached to the link member 34 and detects the attitude of the bucket 33. The bucket inclination angle sensor 63 is a sensor that obtains the inclination angle of the bucket 33 with respect to the horizontal line, and is, for example, an inclination (acceleration) sensor. Note that the bucket inclination angle sensor 63 may be a rotation angle sensor that detects the rotation angle of the bucket connecting pin (bucket base end) or a stroke sensor that detects the stroke amount of the bucket cylinder 43.

[0025] (Dump truck configuration) As shown in FIG. 1, soil held by a work machine 2 is dumped onto a dump truck 3. The dump truck 3 has a cab 26 and a loading platform 27. The loading platform 27 is the part onto which soil is dumped, and is the detection target in this embodiment. The loading platform 27 has four side walls surrounding it. Of the four side walls, the side wall located furthest back as viewed from the work machine 2, and the surface facing the work machine 2, is defined as a back surface 28. In this embodiment, multiple types of dump trucks 3 enter and leave the work site. The detection target is not limited to the loading platform 27, and may be a soil pit, etc.

[0026] As shown in Fig. 2, which is a perspective view of the dump truck 3 viewed from the work machine 2 side, two marks 71 are provided on the back surface 28. The two marks 71 are arranged with a predetermined distance between them. The position and attitude of the mark 71 can be acquired from outside, and in this embodiment, the mark 71 is an AR marker. The mark 71 differs depending on the type of the loading platform 27 of the dump truck 3. The number of the mark 71 may be three or more, or may be one.

[0027] (Configuration of imaging device) As shown in Fig. 1, the imaging device 4 is attached to the work machine 2. The imaging device 4 may be installed in a location separate from the work machine 2. The imaging device 4 is a digital camera, and captures an image of the bed 27 of the dump truck 3 so as to include a mark 71 (see Fig. 2).

[0028] (Configuration of measuring device) As shown in Fig. 1, the measuring device 5 is attached to the work machine 2. The measuring device 5 may be installed at a location away from the work machine 2. The measuring device 5 is a lidar, and measures the distance to the bed 27 of the dump truck 3 by acquiring three-dimensional point cloud data. The measuring device 5 may be an ultrasonic sensor, a millimeter wave radar, a stereo camera, a distance image sensor, an infrared sensor, etc.

[0029] (Circuit configuration of position detection system) As shown in FIG. 3, which is a circuit diagram of the position detection system 1, the work machine 2 has a controller 11 and a storage device 13.

[0030] The controller 11 is capable of automatically operating the slewing device 24 and the attachment 30 based on the detection values ​​of the angle sensor 52 and the tilt angle sensor 60. In other words, the work machine 2 is automatically operated. Note that the work machine 2 may be configured to perform assisted operation to assist the operator who operates the work machine 2.

[0031] The storage device 13 stores information on the three-dimensional shape of the bed 27 of the dump truck 3. As described above, in this embodiment, a plurality of types of dump trucks 3 enter and leave the work site. Therefore, the storage device 13 stores information on the three-dimensional shape of each of the plurality of types of beds 27.

[0032] In this embodiment, the three-dimensional shape information is the coordinates of eight key points indicated by AH in FIG.

[0033] The controller (identification means) 11 identifies the type of the dump truck 3 from the mark 71 captured by the imaging device 4. That is, the controller 11 identifies the type of the loading platform 27.

[0034] Furthermore, the controller (acquisition means) 11 acquires the position and orientation of the mark 71 as marker information from the mark 71 captured by the imaging device 4. Here, the mark 71 is registered in advance in the controller 11 as an AR marker. The AR marker is a sign on which an image to be superimposed and displayed on a real space in AR (Augmented Reality) and identification information for specifying the position are recorded. In this embodiment, the mark 71 is recorded with identification information for specifying an image showing the three-dimensional coordinate system of the mark 71. The controller 11 acquires the orientation of the mark 71 by acquiring the three-dimensional coordinate system of the mark 71.

[0035] The controller (specific position calculation means) 11 calculates the position and attitude of the midpoint 72 as midpoint information from the positions of the acquired marks 71. Here, as shown in FIG. 2, the midpoint 72 is a specific position associated with the loading platform 27. Specifically, the midpoint 72 is the midpoint of the line connecting the two marks 71. Note that the specific position is not limited to the midpoint 72.

[0036] The controller 11 may perform spatial filtering using weighted averaging on the calculated midpoint information, thereby reducing noise and enhancing edges.

[0037] The controller (calculation means) 11 calculates the position and orientation of the platform 27 based on the calculated midpoint information and the information on the three-dimensional shape of the platform 27 stored in the storage device 13. That is, from the position and orientation of the midpoint 72, it specifies where in space each of the eight key points A to H is located. Here, the calculation of the position and orientation of the platform 27 uses information on the three-dimensional shape of the platform 27 of the type specified from among multiple types.

[0038] When there is only one mark 71 provided on the back surface 28, the position and orientation of the platform 27 are calculated based on the marker information of the mark 71 and the stored information on the three-dimensional shape of the platform 27. However, when there is only one mark 71, the calculated position and orientation of the platform 27 vary due to variation in the orientation (three-dimensional coordinate system) of the mark 71. Therefore, it is preferable to calculate the position and orientation of the platform 27 by calculating midpoint information from two or more marks 71.

[0039] In this way, the position and attitude of the bed 27 are calculated based on the position and attitude of the mark 71 acquired from the image of the mark 71 and the information on the three-dimensional shape of the bed 27 stored in the storage device 13. This makes it possible to detect the position of the bed 27 of the dump truck 3 with high accuracy. This makes it possible to assist the operator who operates the work machine 2 to dump earth and sand onto the bed 27, and to automatically operate the work machine 2 to operate to dump earth and sand onto the bed 27 with high accuracy.

[0040] Further, a plurality of marks 71 are provided on the loading platform 27, and the position and attitude of the midpoint 72 are calculated from the respective positions of the marks 71 obtained. Then, the position and attitude of the loading platform 27 are calculated based on the calculated position and attitude of the midpoint 72 and the stored information on the three-dimensional shape of the loading platform 27. When the position and attitude of the loading platform 27 are calculated using the position and attitude of one mark 71, the calculated position and attitude of the loading platform 27 vary due to the variation in the attitude of the mark 71. Therefore, the position and attitude of the midpoint 72 are calculated from the respective positions of the plurality of marks 71, and the position and attitude of the loading platform 27 are calculated using the position and attitude of the midpoint 72, thereby improving the accuracy of detecting the position of the loading platform 27 of the dump truck 3.

[0041] Further, the type of the loading platform 27 is identified from the captured image of the mark 71. Then, the position and posture of the loading platform 27 are calculated based on the information on the three-dimensional shape of the identified type of loading platform 27. This makes it possible to accurately detect the position of the loading platform 27 of the dump truck 3 even when multiple types of dump trucks 3 enter and leave the work site.

[0042] Here, when the controller (estimation means) 11 is unable to acquire the position of one of the two marks 71, it estimates the position and orientation of the midpoint 72 as midpoint information from the acquired position and orientation of the mark 71. Then, the controller 11 calculates the position and orientation of the loading platform 27 based on the estimated midpoint information and stored information on the three-dimensional shape of the loading platform 27.

[0043] In this way, when the positions of some of the multiple landmarks 71 cannot be acquired, the position and orientation of the midpoint 72 can be estimated, thereby improving robustness.

[0044] The controller 11 may perform spatial filtering using weighted averaging on the estimated midpoint information, thereby reducing noise and enhancing edges.

[0045] Furthermore, the controller (correction means) 11 corrects the position and attitude of the midpoint 72 based on the distance measured by the measuring device 5. Specifically, the rear surface 28 of the loading platform 27 is detected, and the position and attitude of the midpoint 72 are corrected based on this. This makes it possible to further improve the detection accuracy of the position of the loading platform 27 of the dump truck 3.

[0046] The controller 11 calibrates the relative relationship between the fields of view of the imaging device 4 and the measuring device 5. That is, the coordinate system of the imaging device 4 and the coordinate system of the measuring device 5 are matched. This makes it possible to accurately correct the position and orientation of the midpoint 72.

[0047] (Location detection system operation) Next, the operation of the position detection system 1 will be described with reference to FIG. 4 which is a flowchart of the position detection process.

[0048] First, the controller 11 of the work machine 2 causes the imaging device 4 to capture an image of the bed 27 of the dump truck 3 (step S1). At this time, the image of the bed 27 is captured so as to include the mark 71. Next, the controller 11 determines whether or not the mark 71 has been recognized (step S2). If it is determined in step S2 that the mark 71 has not been recognized (S2: NO), the process returns to step S1.

[0049] On the other hand, when it is determined in step S2 that the mark 71 has been recognized (S2: YES), the controller 11 identifies the type of the loading platform 27 of the dump truck 3 from the mark 71 (step S3). Then, the controller 11 acquires marker information (the position and the attitude of the mark 71) from the mark 71 (step S4).

[0050] Next, the controller 11 determines whether or not all the marker information has been acquired (step S5). That is, it determines whether or not the marker information has been acquired from each of the two marks 71. If it is determined in step S5 that all the marker information has been acquired (S5: YES), the controller 11 calculates midpoint information (the position and orientation of the midpoint 72) from the positions of each of the marks 71 (step S6).

[0051] Next, the controller 11 performs spatial filtering on the calculated midpoint information (step S7), and then corrects the attitude of the midpoint 72 in the yaw direction (step S8).

[0052] On the other hand, if it is determined in step S5 that all the marker information has not been acquired (S5: NO), the controller 11 estimates midpoint information from the acquired marker information (step S9), and then performs spatial filtering on the estimated midpoint information (step S10).

[0053] After step S8 or after step S10, the controller 11 causes the measuring device 5 to measure the distance to the platform 27 (step S11). Then, the controller 11 performs calibration to match the coordinate system of the imaging device 4 with the coordinate system of the measuring device 5 (step S12).

[0054] Next, the controller 11 determines whether or not the rear surface 28 of the loading platform 27 has been detected from the three-dimensional point cloud data acquired by the measuring device 5 (step S13). If it is determined in step S13 that the rear surface 28 of the loading platform 27 has been detected (S13: YES), the controller 11 corrects the midpoint information (step S14).

[0055] In step S13, if it is determined that the rear surface 28 of the loading platform 27 has not been detected (S13: NO), or after step S14, the controller 11 calculates the position and posture of the loading platform 27 as the coordinates of eight key points A to H based on the midpoint information and the information on the three-dimensional shape of the loading platform 27 stored in the storage device 13 (step S15).

[0056] Next, the controller 11 converts the coordinates of the eight key points A to H into the coordinate system of the work machine 2 (step S16). This allows the position of the bed 27 of the dump truck 3 to be detected with high accuracy. Thereafter, this flow ends.

[0057] (effect) As described above, according to the position detection system 1 of this embodiment, the position and attitude of the loading platform 27 are calculated based on the marker information (position and attitude of the mark 71) acquired from the image of the mark 71 and the information on the three-dimensional shape of the loading platform 27 stored in the storage device 13. This makes it possible to detect the position of the loading platform 27 of the dump truck 3 with high accuracy. Therefore, for example, it is possible to assist the operator who operates the work machine 2 so as to dump earth and sand onto the loading platform 27, or to automatically drive the work machine 2 operating to dump earth and sand onto the loading platform 27 with high accuracy.

[0058] Further, a plurality of marks 71 are provided on the loading platform 27, and midpoint information (position and attitude of the midpoint 72) is calculated from the acquired positions of the marks 71. Then, the position and attitude of the loading platform 27 are calculated based on the calculated midpoint information and the stored information on the three-dimensional shape of the loading platform 27. When the position and attitude of the loading platform 27 are calculated using the position and attitude of one mark 71, the calculated position and attitude of the loading platform 27 vary due to the variation in the attitude of the mark 71. Therefore, the position and attitude of the midpoint 72 are calculated from the positions of the plurality of marks 71, and the position and attitude of the loading platform 27 are calculated using the position and attitude of the midpoint 72, thereby improving the accuracy of detecting the position of the loading platform 27 of the dump truck 3.

[0059] Furthermore, when the positions of some of the multiple landmarks 71 cannot be acquired, the position and orientation of the midpoint 72 are estimated from the positions and orientation of the acquired landmarks 71. Then, the position and orientation of the platform 27 are calculated based on the estimated position and orientation of the midpoint 72 and stored information on the three-dimensional shape of the platform 27. By estimating the position and orientation of the midpoint 72 when the positions of some of the multiple landmarks 71 cannot be acquired, robustness can be improved.

[0060] Furthermore, the position and attitude of the midpoint 72 are corrected based on the measured distance to the loading platform 27. This makes it possible to further improve the detection accuracy of the position of the loading platform 27 of the dump truck 3.

[0061] Further, the type of the loading platform 27 is identified from the captured image of the mark 71. Then, the position and posture of the loading platform 27 are calculated based on the information on the three-dimensional shape of the identified type of loading platform 27. This makes it possible to accurately detect the position of the loading platform 27 of the dump truck 3 even when multiple types of dump trucks 3 enter and leave the work site.

[0062] Although the embodiments of the present invention have been described above, they are merely illustrative examples and do not limit the present invention, and the specific configurations and the like can be appropriately modified in design. Furthermore, the actions and effects described in the embodiments of the invention are merely a list of the most preferable actions and effects resulting from the present invention, and the actions and effects of the present invention are not limited to those described in the embodiments of the present invention.

[0063] For example, identification of the type of the bed 27 of the dump truck 3, acquisition of marker information, calculation, estimation, and correction of midpoint information, and calculation of the position and attitude of the bed 27 of the dump truck 3 may be performed by a server (not shown) rather than by the controller 11 of the work machine 2.

[0064] Moreover, one stereo camera may be used as both the imaging device 4 and the measuring device 5. In this case, it is not necessary to prepare the imaging device 4 and the measuring device 5 separately. [Explanation of symbols]

[0065] 1 Position Detection System 2. Work Machinery 3. Dump Truck 4. Imaging device 5. Measurement equipment 11 Controller (acquisition means, calculation means, specific position calculation means, estimation means, correction means, identification means) 13 Storage device 21 Undercarriage 22 Upper rotating body 23 Cab 24 Swivel 26 Cab 27 Cargo platform (detection target) 28 Back (including specific location corresponding area) (part of side wall) 30 Attachment 31 Boom 32 Arm 33 Bucket 34 Link member 40 Cylinder 41 Boom cylinder 42 Arm Cylinder 43 Bucket Cylinder 52 Angle Sensor 60 Inclination Sensor 61 Boom tilt angle sensor 62 Arm inclination angle sensor 63 Bucket tilt angle sensor 71 Landmark 72 Midpoint (specific position)

Claims

1. A lower running body; An upper rotating body rotatably attached to an upper portion of the lower traveling body; An attachment rotatably attached to the upper rotating body; an imaging device that images the detection target so as to include a mark provided on the detection target, the mark being capable of externally acquiring a position and a posture of the detection target; an acquisition means for acquiring a position and an orientation of the mark relative to the imaging device from an appearance of the mark in a captured image; A storage device that stores information on the three-dimensional shape of the detection target; a calculation means for calculating a position and an orientation of the three-dimensional shape of the detection target based on the acquired position and orientation of the mark and stored information on the three-dimensional shape of the detection target; A work machine comprising:

2. The detection target is a side wall surrounding a drop position where the transported object is dropped from the attachment.

2. A work machine according to claim 1.

3. A plurality of the marks are provided on the detection target, a specific position calculation means for calculating a position and an orientation of a specific position associated with the detection target from the acquired positions of each of the landmarks; The calculation means calculates a position and orientation of the three-dimensional shape of the detection target based on the calculated position and orientation of the specific position and stored information on the three-dimensional shape of the detection target.

3. A work machine according to claim 1 or 2.

4. The specific position is the midpoint of a line connecting the two landmarks.

4. A work machine according to claim 3.

5. an estimation means for estimating a position and orientation of the specific position from the acquired positions and orientations of the landmarks when the acquisition means is unable to acquire positions of some of the landmarks; The calculation means calculates a position and orientation of the three-dimensional shape of the detection target based on the estimated position and orientation of the specific position and stored information on the three-dimensional shape of the detection target.

5. A work machine according to claim 3 or 4.

6. a measuring device that measures position information of a specific position corresponding portion of the detection target that corresponds to the specific position; and a correction means for correcting the position of the specific position based on position information of the measured specific position corresponding portion. A working machine according to any one of claims 3 to 5.

7. The mark is different for each type of the detection target, the storage device stores information on the three-dimensional shape of each of a plurality of types of the detection targets; a determination unit that determines the type of the detection target from the image of the mark, The calculation means calculates a position and an orientation of the three-dimensional shape of the detection target based on information of the three-dimensional shape of the detection target of the specified type. A working machine according to any one of claims 1 to 6.

Images