Working machinery

Abstract

To provide a work machine capable of detecting the position of a detection target with high accuracy.SOLUTION: An imaging device 4 captures an image of a detection target (27) so as to include a mark 71 whose position and orientation can be acquired externally. Acquisition means (11) acquires the position and orientation of the mark 71 relative to the imaging device 4 from the appearance of the mark 71 in the captured image. A storage device 13 stores information on the three-dimensional shape of the detection target (27). Calculation means (11) calculates the position and orientation of the three-dimensional shape of the detection target (27) based on the acquired position and orientation of the mark 71 and the stored information on the three-dimensional shape of the detection target (27).SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a working machine that detects the position of a detection target.

Background Art

[0002] As disclosed in Patent Document 1, an imaging image of a dropping target (such as a dump truck) onto which a conveyed object (such as earth and sand) held by a working machine is dropped is acquired, and based on a position identification model, which is a learned model, and the imaging image, the position of a detection target (such as a loading platform) in the dropping target shown in the image is identified.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0007] According to the present invention, the position and orientation of the object to be detected are calculated based on the position and orientation of the marker obtained from the imaged marker and the information of the three-dimensional shape of the object to be detected stored in the storage device. This makes it possible to detect the position of the object with high accuracy. Therefore, for example, it is possible to assist an operator in operating a work machine to drop a transported object onto the object to be detected, or to automatically operate a work machine that operates to drop a transported object onto the object to be detected with high accuracy. [Brief explanation of the drawing]

[0008] [Figure 1] This is a diagram illustrating the configuration of the position detection system. [Figure 2] This is a perspective view of the dump truck from the perspective of the work machinery. [Figure 3] This is a circuit diagram of a position detection system. [Figure 4] This is a flowchart of the position detection process. [Modes for carrying out the invention]

[0009] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

[0010] (Configuration of the position detection system) The position detection system according to this embodiment detects the position of an object relative to a work machine. As shown in Figure 1, a configuration diagram of the position detection system 1, the position detection system 1 includes a work machine 2, a dump truck 3, an imaging device 4, and a measuring device 5.

[0011] (Configuration of the work machine) As shown in Figure 1, the work machine 2 is a machine that performs work with an attachment 30, and is, for example, a hydraulic excavator. The work machine 2 has a lower traveling body 21, an upper rotating body 22, a rotating device 24, an attachment 30, and a cylinder 40.

[0012] The lower traveling body 21 is the part on which the work machine 2 travels, and is equipped with, for example, crawler tracks. The upper slewing body 22 is rotatably mounted on top of the lower traveling body 21. A cab (operator's cabin) 23 is provided at the front of the upper slewing body 22. The slewing device 24 is capable of rotatable movement of the upper slewing body 22.

[0013] The attachment 30 is mounted on the upper slewing body 22 so as to be rotatable in the vertical direction. The attachment 30 comprises a boom 31, an arm 32, and a bucket 33. The boom 31 is mounted on the upper slewing body 22 so as to be rotatable in the vertical direction. The arm 32 is mounted on the boom 31 so as to be rotatable in the vertical direction. The bucket 33 is mounted on the arm 32 so as to be rotatable in the vertical direction.

[0014] The bucket 33 is the part that excavates, holds, and drops the soil and sand being transported. The bucket 33 is just one example of a tip attachment that can be attached to the arm 32; the tip attachment is not limited to this and may include a nibbler, clamp arm, etc. Furthermore, the transported material is not limited to soil and sand, but may include rubble, scrap metal, gravel, etc.

[0015] The cylinder 40 can rotate the attachment 30 hydraulically. The cylinder 40 is a hydraulic telescopic cylinder. The cylinder 40 includes a boom cylinder 41, an arm cylinder 42, and a bucket cylinder 43.

[0016] The boom cylinder 41 rotationally drives the boom 31 with respect to the upper slewing body 22. The base end portion of the boom cylinder 41 is rotatably attached to the upper slewing body 22. The tip end portion of the boom cylinder 41 is rotatably attached to the boom 31.

[0017] The arm cylinder 42 rotationally drives the arm 32 with respect to the boom 31. The base end portion of the arm cylinder 42 is rotatably attached to the boom 31. The tip end portion of the arm cylinder 42 is rotatably attached to the arm 32.

[0018] The bucket cylinder 43 rotationally drives the bucket 33 with respect to the arm 32. The base end portion of the bucket cylinder 43 is rotatably attached to the arm 32. The tip end portion of the bucket cylinder 43 is rotatably attached to a link member 34 that is rotatably attached to the bucket 33.

[0019] Further, the work machine 2 has an angle sensor 52 and an inclination angle sensor 60.

[0020] The angle sensor 52 detects the turning angle of the upper slewing body 22 with respect to the lower traveling body 21. The angle sensor 52 is, for example, an encoder, a resolver, or a gyro sensor. In the present embodiment, the turning angle of the upper slewing body 22 when the front of the upper slewing body 22 coincides with the front of the lower traveling body 21 is set to 0°.

[0021] The inclination angle sensor 60 detects the posture of the attachment 30. The inclination angle sensor 60 includes a boom inclination angle sensor 61, an arm inclination angle sensor 62, and a bucket inclination angle sensor 63.

[0022] The boom tilt angle sensor 61 is attached to the boom 31 and detects the attitude of the boom 31. The boom tilt angle sensor 61 is a sensor that acquires the tilt angle of the boom 31 with respect to the horizontal line, and is, for example, a tilt (acceleration) sensor. The boom tilt angle sensor 61 may also be a rotation angle sensor that detects the rotation angle of the boom foot pin (boom base) or a stroke sensor that detects the stroke amount of the boom cylinder 41.

[0023] The arm tilt angle sensor 62 is attached to the arm 32 and detects the posture of the arm 32. The arm tilt angle sensor 62 is a sensor that acquires the tilt angle of the arm 32 with respect to the horizontal line, and is, for example, a tilt (acceleration) sensor. The arm tilt angle sensor 62 may also be a rotation angle sensor that detects the rotation angle of the arm connecting pin (arm base end) or a stroke sensor that detects the stroke amount of the arm cylinder 42.

[0024] The bucket tilt angle sensor 63 is attached to the link member 34 and detects the attitude of the bucket 33. The bucket tilt angle sensor 63 is a sensor that acquires the tilt angle of the bucket 33 with respect to the horizontal line, and is, for example, a tilt (acceleration) sensor. The bucket tilt angle sensor 63 may also be a rotation angle sensor that detects the rotation angle of the bucket connecting pin (bucket base end) or a stroke sensor that detects the stroke amount of the bucket cylinder 43.

[0025] (Dump truck configuration) As shown in Figure 1, the dump truck 3 is used to drop soil and sand held by the work machine 2. The dump truck 3 has a driver's cab 26 and a cargo bed 27. The cargo bed 27 is the part into which the soil and sand are dropped and is the target of detection in this embodiment. The cargo bed 27 has four side walls surrounding it. Of the four side walls, the side wall located furthest inward from the perspective of the work machine 2 has a surface facing the work machine 2 that is designated as the rear surface 28. In this embodiment, multiple types of dump trucks 3 enter and exit the work site. Note that the target of detection is not limited to the cargo bed 27, but may also be a soil pit or other similar structure.

[0026] As shown in Figure 2, a perspective view of the dump truck 3 from the work machine 2 side, two markers 71 are provided on the rear surface 28. The two markers 71 are positioned at a predetermined distance apart. The position and orientation of the markers 71 can be acquired from the outside, and in this embodiment, they are AR markers. The markers 71 differ for each type of cargo bed 27 of the dump truck 3. Note that there may be three or more markers 71, or there may be just one.

[0027] (Configuration of the imaging device) As shown in Figure 1, the imaging device 4 is attached to the work machine 2. The imaging device 4 may also be installed at a location separate from the work machine 2. The imaging device 4 is a digital camera and captures images of the cargo bed 27 of the dump truck 3, including the marker 71 (see Figure 2).

[0028] (Configuration of the measuring device) As shown in Figure 1, the measuring device 5 is attached to the work machine 2. The measuring device 5 may also be installed at a location separate from the work machine 2. The measuring device 5 is a LiDAR (light-emitting radar) and measures the distance to the dump truck 3's cargo bed 27 by acquiring three-dimensional point cloud data. The measuring device 5 may also be an ultrasonic sensor, millimeter-wave radar, stereo camera, distance image sensor, infrared sensor, etc.

[0029] (Circuit configuration of the position detection system) As shown in Figure 3, which is a circuit diagram of the position detection system 1, the work machine 2 includes a controller 11 and a storage device 13.

[0030] The controller 11 can automatically operate the slewing device 24 and the attachment 30 based on the detection values ​​of the angle sensor 52 and the tilt angle sensor 60. In other words, the work machine 2 is operated automatically. Alternatively, an assist operation system may be configured to assist the operator operating the work machine 2.

[0031] The storage device 13 stores information about the three-dimensional shape of the cargo bed 27 of the dump truck 3. As described above, in this embodiment, multiple types of dump trucks 3 enter and exit the work site. Therefore, the storage device 13 stores information about the three-dimensional shape of each of the multiple types of cargo beds 27.

[0032] In this embodiment, the three-dimensional shape information is the coordinates of the eight key points shown as A to H in Figure 2.

[0033] The controller (identification means) 11 identifies the type of dump truck 3 from the marker 71 captured by the imaging device 4. In other words, the controller 11 identifies the type of cargo bed 27.

[0034] Furthermore, the controller (acquisition means) 11 acquires the position and orientation of the marker 71 as marker information from the marker 71 captured by the imaging device 4. Here, the marker 71 is pre-registered with the controller 11 as an AR marker. An AR marker is a sign that contains identification information for specifying an image to be superimposed on real space in AR (augmented reality) and its position. In this embodiment, the marker 71 contains identification information for specifying an image that shows the three-dimensional coordinate system of the marker 71. The controller 11 acquires the orientation of the marker 71 by acquiring the three-dimensional coordinate system of the marker 71.

[0035] The controller (specific position calculation means) 11 calculates the position and orientation of the midpoint 72 as midpoint information from the positions of each of the acquired markers 71. Here, as shown in Figure 2, the midpoint 72 is a specific position associated with the cargo bed 27. Specifically, the midpoint 72 is the midpoint of the straight line connecting the two markers 71. Note that the specific position is not limited to the midpoint 72.

[0036] The controller 11 may also apply weighted averaging spatial filtering to the calculated midpoint information. This can reduce noise or enhance edges.

[0037] The controller (calculation means) 11 calculates the position and orientation of the cargo bed 27 based on the calculated midpoint information and the three-dimensional shape information of the cargo bed 27 stored in the storage device 13. In other words, it identifies where each of the eight key points A to H is located in space from the position and orientation of the midpoint 72. Here, the calculation of the position and orientation of the cargo bed 27 uses three-dimensional shape information of a type of cargo bed 27 that has been identified from among several types.

[0038] When there is only one marker 71 on the rear surface 28, the position and orientation of the cargo bed 27 are calculated based on the marker information of this marker 71 and the stored information on the three-dimensional shape of the cargo bed 27. However, when there is only one marker 71, the calculated position and orientation of the cargo bed 27 will vary due to variations in the orientation (three-dimensional coordinate system) of the marker 71. Therefore, it is preferable to calculate the position and orientation of the cargo bed 27 by calculating midpoint information from two or more markers 71.

[0039] In this way, the position and orientation of the cargo bed 27 are calculated based on the position and orientation of the marker 71 obtained from the imaged marker 71 and the three-dimensional shape information of the cargo bed 27 stored in the storage device 13. This makes it possible to accurately detect the position of the cargo bed 27 of the dump truck 3. Therefore, it is possible to assist the operator in operating the work machine 2 to drop soil onto the cargo bed 27, or to automatically operate the work machine 2 that operates to drop soil onto the cargo bed 27 with high precision.

[0040] Furthermore, multiple markers 71 are provided on the cargo bed 27, and the position and orientation of the midpoint 72 are calculated from the position of each acquired marker 71. Then, the position and orientation of the cargo bed 27 are calculated based on the calculated position and orientation of the midpoint 72 and the stored information on the three-dimensional shape of the cargo bed 27. When the position and orientation of the cargo bed 27 are calculated using the position and orientation of a single marker 71, the calculated position and orientation of the cargo bed 27 will vary due to variations in the orientation of the marker 71. Therefore, by calculating the position and orientation of the midpoint 72 from the position of each of multiple markers 71, and then using the position and orientation of the midpoint 72 to calculate the position and orientation of the cargo bed 27, the accuracy of detecting the position of the cargo bed 27 of the dump truck 3 can be improved.

[0041] Furthermore, the type of cargo bed 27 is identified from the captured marker 71. Then, the position and orientation of the cargo bed 27 are calculated based on the three-dimensional shape information of the identified type of cargo bed 27. This allows for accurate detection of the position of the cargo bed 27 of the dump truck 3, even when multiple types of dump trucks 3 are entering and exiting the work site.

[0042] Here, if the controller (estimation means) 11 is unable to obtain the position of one of the two markers 71, it estimates the position and orientation of the midpoint 72 as midpoint information from the position and orientation of the obtained marker 71. Then, the controller 11 calculates the position and orientation of the cargo bed 27 based on the estimated midpoint information and the stored information on the three-dimensional shape of the cargo bed 27.

[0043] In this way, robustness can be improved by estimating the position and orientation of the midpoint 72 when the positions of some of the multiple markers 71 cannot be obtained.

[0044] Furthermore, the controller 11 may apply weighted averaging spatial filtering to the estimated midpoint information. This can reduce noise or enhance edges.

[0045] Furthermore, the controller (correction means) 11 corrects the position and orientation of the midpoint 72 based on the distance measured by the measuring device 5. Specifically, it detects the rear surface 28 of the cargo bed 27 and corrects the position and orientation of the midpoint 72 based on this. This further improves the accuracy of detecting the position of the cargo bed 27 of the dump truck 3.

[0046] Furthermore, the controller 11 calibrates the relative relationship of the fields of view between the imaging device 4 and the measuring device 5. In other words, it matches the coordinate system of the imaging device 4 with the coordinate system of the measuring device 5. This allows for accurate correction of the position and orientation of the midpoint 72.

[0047] (Operation of the position detection system) Next, we will explain the operation of the position detection system 1 using Figure 4, which is a flowchart of the position detection process.

[0048] First, the controller 11 of the work machine 2 causes the imaging device 4 to image the cargo bed 27 of the dump truck 3 (step S1). At this time, the cargo bed 27 is imaged so that the marker 71 is included. Next, the controller 11 determines whether or not the marker 71 has been recognized (step S2). If it is determined in step S2 that the marker 71 has not been recognized (S2: NO), the process returns to step S1.

[0049] On the other hand, if it is determined in step S2 that the marker 71 has been recognized (S2: YES), the controller 11 identifies the type of cargo bed 27 of the dump truck 3 from the marker 71 (step S3). Then, the controller 11 obtains marker information (position and orientation of the marker 71) from the marker 71 (step S4).

[0050] Next, the controller 11 determines whether or not all marker information has been acquired (step S5). That is, it determines whether or not marker information has been acquired from each of the two markers 71. If it is determined in step S5 that all marker information has been acquired (S5: YES), the controller 11 calculates midpoint information (position and orientation of midpoint 72) from the position of each marker 71 (step S6).

[0051] Next, the controller 11 applies spatial filtering to the calculated midpoint information (step S7). Then, the controller 11 corrects the yaw attitude of the midpoint 72 (step S8).

[0052] On the other hand, if it is determined in step S5 that not all marker information has been acquired (S5:NO), the controller 11 estimates the midpoint information from the acquired marker information (step S9). Then, the controller 11 applies spatial filtering to the estimated midpoint information (step S10).

[0053] After step S8, or after step S10, the controller 11 causes the measuring device 5 to measure the distance to the loading platform 27 (step S11). Then, the controller 11 performs calibration to match the coordinate system of the imaging device 4 with the coordinate system of the measuring device 5 (step S12).

[0054] Next, the controller 11 determines whether or not the rear surface 28 of the cargo bed 27 has been detected from the three-dimensional point cloud data acquired by the measuring device 5 (step S13). If the controller 11 determines in step S13 that the rear surface 28 of the cargo bed 27 has been detected (S13:YES), the controller 11 corrects the midpoint information (step S14).

[0055] If, in step S13, it is determined that the rear surface 28 of the cargo bed 27 has not been detected (S13: NO), or after step S14, the controller 11 calculates the position and orientation of the cargo bed 27 as coordinates of eight key points A to H based on the midpoint information and the three-dimensional shape information of the cargo bed 27 stored in the storage device 13 (step S15).

[0056] Next, the controller 11 transforms the coordinates of the eight key points A to H into the coordinate system of the work machine 2 (step S16). This allows for accurate detection of the position of the dump truck bed 27. After this, this flow is terminated.

[0057] (effect) As described above, according to the position detection system 1 of this embodiment, the position and orientation of the cargo bed 27 are calculated based on marker information (position and orientation of the marker 71) acquired from the imaged marker 71 and the three-dimensional shape information of the cargo bed 27 stored in the storage device 13. This makes it possible to accurately detect the position of the cargo bed 27 of the dump truck 3. Therefore, for example, it is possible to assist an operator operating the work machine 2 to drop soil onto the cargo bed 27, or to automatically operate the work machine 2 that operates to drop soil onto the cargo bed 27, with high accuracy.

[0058] Furthermore, multiple markers 71 are provided on the cargo bed 27, and midpoint information (position and orientation of midpoint 72) is calculated from the position of each acquired marker 71. Then, the position and orientation of the cargo bed 27 are calculated based on the calculated midpoint information and the stored three-dimensional shape information of the cargo bed 27. When the position and orientation of the cargo bed 27 are calculated using the position and orientation of a single marker 71, the calculated position and orientation of the cargo bed 27 will vary due to variations in the orientation of the marker 71. Therefore, by calculating the position and orientation of the midpoint 72 from the positions of multiple markers 71, and then using the position and orientation of the midpoint 72 to calculate the position and orientation of the cargo bed 27, the accuracy of detecting the position of the cargo bed 27 of the dump truck 3 can be improved.

[0059] Furthermore, if the positions of some of the multiple markers 71 cannot be obtained, the position and orientation of the midpoint 72 are estimated from the positions and orientations of the obtained markers 71. Then, the position and orientation of the cargo bed 27 are calculated based on the estimated position and orientation of the midpoint 72 and the stored information on the three-dimensional shape of the cargo bed 27. By estimating the position and orientation of the midpoint 72 when the positions of some of the multiple markers 71 cannot be obtained, robustness can be improved.

[0060] Furthermore, the position and orientation of the midpoint 72 are corrected based on the measured distance to the cargo bed 27. This further improves the accuracy of detecting the position of the cargo bed 27 of the dump truck 3.

[0061] Furthermore, the type of cargo bed 27 is identified from the captured marker 71. Then, the position and orientation of the cargo bed 27 are calculated based on the three-dimensional shape information of the identified type of cargo bed 27. This allows for accurate detection of the position of the cargo bed 27 of the dump truck 3, even when multiple types of dump trucks 3 are entering and exiting the work site.

[0062] Although embodiments of the present invention have been described above, these are merely illustrative examples and do not particularly limit the present invention. Specific configurations and other aspects can be modified as appropriate. Furthermore, the actions and effects described in the embodiments of the invention are merely a list of the most preferred actions and effects resulting from the present invention, and the actions and effects according to the present invention are not limited to those described in the embodiments.

[0063] For example, the identification of the type of cargo bed 27 of the dump truck 3, acquisition of marker information, calculation, estimation, and correction of midpoint information, and calculation of the position and orientation of the cargo bed 27 of the dump truck 3 may be performed by a server (not shown) rather than the controller 11 of the work machine 2.

[0064] Alternatively, a single stereo camera may be used as both the imaging device 4 and the measuring device 5. In this case, it is not necessary to prepare the imaging device 4 and the measuring device 5 separately. [Explanation of Symbols]

[0065] 1. Position detection system 2. Working Machines 3 Dump trucks 4. Imaging device 5. Measuring device 11. Controller (acquisition means, calculation means, specific position calculation means, estimation means, correction means, specific means) 13 Storage device 21 Lower running body 22 Upper rotating body 23 Cab 24 Swivel device 26 Driver's cab 27. Cargo bed (detection target) 28. Rear view (including specific location-specific areas) (part of the side wall) 30 Attachments 31 Boom 32 Arms 33 buckets 34 Link members 40 cylinders 41 Boom Cylinder 42 Arm Cylinder 43 Bucket Cylinder 52 Angle Sensor 60 Tilt Angle Sensor 61 Boom tilt angle sensor 62 Arm tilt angle sensor 63 Bucket tilt angle sensor 71 Landmark 72 Midpoint (specific position)

Claims

1. Lower running body and An upper slewing body is rotatably mounted on the upper part of the lower traveling body, An attachment rotatably mounted to the upper rotating body, An imaging device that images the target to be detected, including a marker placed on the target to be detected, the marker whose position and orientation can be acquired from the outside, An acquisition means for acquiring the position and orientation of the marker relative to the imaging device from the appearance of the marker in the captured image, A storage device that stores information about the three-dimensional shape of the object to be detected, A calculation means for calculating the position and orientation of the three-dimensional shape of the detection target based on the acquired position and orientation of the marker and the stored information on the three-dimensional shape of the detection target, A work machine characterized by having the following features.

2. The detection target is the side wall surrounding the drop position from which the transported object is dropped from the attachment. The work machine according to feature 1.

3. Multiple of the aforementioned markers are provided on the object to be detected, The system includes a specific position calculation means that calculates the position and orientation of a specific position associated with the detection target from the position of each of the acquired markers, The calculation means calculates the position and orientation of the three-dimensional shape of the detection target based on the calculated position and orientation of the specific location and the stored information on the three-dimensional shape of the detection target. The work machine according to feature 1 or 2.

4. The aforementioned specific location is the midpoint of the line connecting the two aforementioned markers. The work machine according to feature 3.

5. If the acquisition means fails to acquire the positions of some of the multiple markers, it includes an estimation means for estimating the position and orientation of the specific location from the acquired positions and orientations of the markers. The calculation means calculates the position and orientation of the three-dimensional shape of the detected object based on the estimated position and orientation of the specific location and the stored information on the three-dimensional shape of the detected object. The work machine according to feature 3 or 4.

6. A measuring device that measures the location information of the detected target, specifically the location information of the part corresponding to the specific location that corresponds to the specific location, The system includes a correction means for correcting the position of the specific location based on the measured position information of the part corresponding to the specific location. A work machine according to any one of claims 3 to 5.

7. The marker differs for each type of object to be detected. The storage device stores information on the three-dimensional shape of each of the multiple types of detection targets. The system includes a means for identifying the type of the object to be detected from the imaged marker, The calculation means calculates the position and orientation of the three-dimensional shape of the detected object based on the information of the three-dimensional shape of the detected object of a specified type. A work machine according to any one of claims 1 to 6.

Images