Work machine and method for estimating the position of the work machine

Abstract

The position of the work equipment is estimated. The work machine includes a work implement including a bucket, an imaging device that images the work implement, and a controller. The controller acquires the area of ​​the bucket in an image 61 that includes the work implement and is captured by the imaging device. The controller estimates the position of the tip of the bucket using the acquired area of ​​the bucket and additional information.

Description

[Technical Field]

[0001] FIELD OF THE DISCLOSURE The present disclosure relates to a work machine and a method for estimating the position of the work machine. [Background technology]

[0002] Japanese Patent Publication No. 2019-214835 (Patent Document 1) discloses that an image of a work machine captured by an imaging device is acquired, and an estimated position of the work machine is calculated by estimating the position of the work machine from the captured image using a trained position estimation model. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-214835 A Summary of the Invention [Problem to be solved by the invention]

[0004] This disclosure proposes a new technique for estimating the position of a work machine. [Means for solving the problem]

[0005] A work machine according to one aspect of the present disclosure includes a work implement including a bucket, an imaging device that images the work implement, and a controller. The controller acquires an area of ​​the bucket in an image captured by the imaging device from an image including the work implement. The controller estimates a tip position of the bucket using the acquired area of ​​the bucket and accompanying information.

[0006] A method for estimating the position of a work machine according to one aspect of the present disclosure includes the following steps: a first step is a step of acquiring a captured image of the work machine including a bucket; a second step is a step of acquiring the area of ​​the bucket in the captured image; and a third step is a step of estimating the position of the tip of the bucket using the acquired area of ​​the bucket and accompanying information. [Effects of the Invention]

[0007] According to the present disclosure, a new technique for estimating the position of a work machine can be provided. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating a configuration of a hydraulic excavator. [Figure 2] FIG. 1 is a block diagram showing a schematic configuration of a system including a hydraulic excavator. [Figure 3] 3A to 3C are diagrams illustrating an operation mode of the hydraulic excavator in the first embodiment. [Figure 4] FIG. 3 is a block diagram showing a process for estimating the tip position of a bucket according to the first embodiment. [Figure 5] Schematic diagram of a trained model. [Figure 6] FIG. 2 is a schematic diagram showing an area in which a bucket is captured in a captured image. [Figure 7] FIG. 1 is a schematic diagram showing a response surface. [Figure 8] 10A to 10C are diagrams illustrating an operation mode of the hydraulic excavator in the second embodiment. [Figure 9] FIG. 10 is a block diagram showing a process for estimating the tip position of a bucket according to a second embodiment. [Figure 10] FIG. 10 is a schematic diagram showing an area in a captured image in which a bucket tooth is captured. [Figure 11] 10A to 10C are diagrams illustrating an operation mode of the hydraulic excavator in the third embodiment. [Figure 12] FIG. 10 is a block diagram showing a process for estimating the tip position of the bucket according to a third embodiment. [Figure 13] FIG. 10 is a schematic diagram showing an area in a captured image in which a part of a bucket is captured. [Figure 14] FIG. 10 is a schematic diagram showing an area in which an arm is captured in a captured image. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, the embodiments will be described with reference to the drawings. In the following description, the same parts and components are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed descriptions thereof will not be repeated. In the drawings, configurations may be omitted or simplified for the sake of convenience. It is also intended from the beginning that any configurations may be extracted from the embodiments and arbitrarily combined.

[0010] <Work machine configuration> Fig. 1 is a diagram schematically illustrating the configuration of a hydraulic excavator 100 as an example of a work machine according to an embodiment of the present disclosure. As shown in Fig. 1, the hydraulic excavator 100 of this embodiment has a main body 1 and a hydraulically operated work implement 2. The main body 1 has a revolving unit 3 and a traveling unit 5.

[0011] The running body 5 has a pair of crawler tracks 5Cr and a traveling motor 5M. The hydraulic excavator 100 is capable of traveling by rotation of the crawler tracks 5Cr. The traveling motor 5M is provided as a drive source for the running body 5. The traveling motor 5M is a hydraulic motor that is hydraulically operated. The running body 5 may have wheels (tires).

[0012] The rotating body 3 is disposed on the running body 5 and is supported by the running body 5. The rotating body 3 can be rotated relative to the running body 5 about a rotation axis RX by a rotation motor (not shown). The rotation axis RX is the rotation center of the rotating body 3. The rotation motor is a hydraulic motor operated by hydraulic pressure. The rotation axis RX is a virtual straight line that is the rotation center of the rotating body 3. Note that the traveling motor 5M or the rotation motor may be an electric motor.

[0013] The rotating unit 3 has a driver's cab 4. Inside the driver's cab 4, there is provided a driver's seat 4S where an operator sits. The operator (crew member) sits in the driver's cab 4 and can operate the work equipment 2, rotate the rotating unit 3 relative to the traveling unit 5, and travel the hydraulic excavator 100 using the traveling unit 5. The rotating unit 3 has an exterior cover 9. The exterior cover 9 covers the machine room. The hydraulic excavator 100 may be remotely operated.

[0014] The work implement 2 is supported by the rotating unit 3. The work implement 2 is attached to the rotating unit 3 so as to be movable relative to the rotating unit 3. The work implement 2 has a boom 6, an arm 7, and a bucket 8. The work implement 2 further has a boom cylinder 10, an arm cylinder 11, and a bucket cylinder 12.

[0015] Boom 6 is rotatably connected to main body 1. Specifically, the base end of boom 6 is rotatably connected to revolving unit 3 with boom foot pin 13 as a fulcrum. Arm 7 is rotatably connected to boom 6. Specifically, the base end of arm 7 is rotatably connected to the tip of boom 6 with boom top pin 14 as a fulcrum. Bucket 8 is rotatably connected to arm 7. Specifically, the base end of bucket 8 is rotatably connected to the tip of arm 7 with arm top pin 15 as a fulcrum.

[0016] A plurality of bucket teeth 8T are attached to the bucket 8. The bucket teeth 8T have two stops (cutting edges) 8TT at their tips. The two stops 8TT form the tip of the bucket 8. The tip position of the bucket 8 is the position of the two stops 8TT.

[0017] One end of the boom cylinder 10 is connected to the revolving unit 3, and the other end is connected to the boom 6. The boom 6 can move relative to the main body 1 by the boom cylinder 10. By the operation of the boom cylinder 10, the boom 6 can rotate up and down relative to the revolving unit 3, with the boom foot pin 13 as a fulcrum.

[0018] One end of the arm cylinder 11 is connected to the boom 6, and the other end is connected to the arm 7. The arm 7 can move relative to the boom 6 by the arm cylinder 11. By operation of the arm cylinder 11, the arm 7 can rotate up and down or back and forth relative to the boom 6, with the boom top pin 14 as a fulcrum.

[0019] One end of the bucket cylinder 12 is connected to the arm 7, and the other end is connected to a bucket link 17. The bucket 8 can move relative to the arm 7 by the bucket cylinder 12. By the operation of the bucket cylinder 12, the bucket 8 can rotate up and down relative to the arm 7, with the arm top pin 15 as a fulcrum.

[0020] Each of the boom cylinder 10, the arm cylinder 11 and the bucket cylinder 12 is a hydraulic cylinder that is driven by hydraulic pressure, but may be another actuator such as an electric cylinder.

[0021] The hydraulic excavator 100 further has an imaging device 20. The imaging device 20 is attached to, for example, the cab 4. The imaging device 20 is attached to the top surface of the cab 4. The imaging device 20 may also be attached to the front surface of the cab 4. The imaging device 20 may also be disposed inside the cab 4. The imaging device 20 may also be disposed at the viewpoint of an operator seated in the cab 4S, in which case it becomes possible to obtain an image that is close to what the operator sees.

[0022] The imaging device 20 captures an image of an object. The imaging device 20 acquires an optical image of the object and detects the outer shape of the object. The imaging device includes an imaging element such as a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The imaging device 20 may be a monocular camera or a stereo camera.

[0023] The optical axis of the imaging device 20 extends forward and downward from the imaging device 20. The imaging target of the imaging device 20 includes the work implement 2. The imaging device 20 captures an image of the work implement 2, and a captured image including the work implement 2 is obtained. The imaging target of the imaging device 20 includes the terrain or objects at the work site in front of the hydraulic excavator 100. The imaging range of the imaging device 20 includes the work implement 2 and the terrain in front of the hydraulic excavator 100. A portion of the work implement 2 is included within the angle of view of the imaging device 20.

[0024] The position and orientation sensor 21 is, for example, a GNSS (Global Navigation Satellite Systems) receiver. The position and orientation sensor 21 includes two GNSS receivers 21a and 21b. Each of the two GNSS receivers 21a and 21b is installed at a different position on the rotating unit 3. Each of the GNSS receivers 21a and 21b receives a satellite positioning signal from a satellite, the satellite positioning signal indicating the position of the rotating unit 3 in the global coordinate system. The position and orientation sensor 21 outputs the received satellite positioning signal indicating the position of the rotating unit 3 in the global coordinate system.

[0025] <System configuration> Fig. 2 is a block diagram showing a schematic configuration of a system including a hydraulic excavator 100. The system according to the embodiment is a system for determining the position of the work implement 2, in particular the position of the tip of the bucket 8 (the position of the two stops 8TT). The system according to the embodiment includes the hydraulic excavator 100 as an example of a work machine described with reference to Fig. 1, and a control device 200 shown in Fig. 2. The control device 200 may be mounted on the hydraulic excavator 100.

[0026] 1, a global coordinate calculation unit 23, an IMU (Inertial Measurement Unit) 24, an operation device 25, a directional control valve 34, a pressure sensor 36, and a controller 102. The imaging device 20 outputs an image of the imaging target to the controller 102.

[0027] The position and orientation sensor 21 outputs the received radio waves (satellite positioning signals) to the global coordinate calculation unit 23. The global coordinate calculation unit 23 detects the installation position of the position and orientation sensor 21 in a global coordinate system. The global coordinate system is a three-dimensional coordinate system based on a reference position installed in the work area. The reference position may be the position of the tip of a reference stake set in the work area.

[0028] The IMU 24 is provided on the rotating unit 3. The IMU 24 is disposed, for example, on a frame below the cab 4. The IMU 24 may be disposed to the side (right or left) of the rotation axis RX of the rotating unit 3. The IMU 24 measures the acceleration of the rotating unit 3 in the front-to-rear, left-to-right, and up-to-down directions, and the angular velocity of the rotating unit 3 about the front-to-rear, left-to-right, and up-to-down directions.

[0029] The engine 31 is a diesel engine. The amount of fuel injected into the engine 31 is controlled by the controller 102, thereby controlling the output of the engine 31.

[0030] The hydraulic pump 33 is connected to the engine 31. The rotational driving force of the engine 31 is transmitted to the hydraulic pump 33, thereby driving the hydraulic pump 33. The hydraulic pump 33 is a variable displacement hydraulic pump that has a swash plate and changes the discharge capacity by changing the tilt angle of the swash plate. The hydraulic oil discharged from the hydraulic pump 33 is reduced to a constant pressure by a pressure reducing valve and supplied to the directional control valve 34.

[0031] The directional control valve 34 is a spool-type valve that switches the direction of hydraulic oil flow by moving a rod-shaped spool. The amount of hydraulic oil supplied to the hydraulic actuator 40 is adjusted by the axial movement of the spool. The directional control valve 34 is provided with a spool stroke sensor that detects the movement distance of the spool (spool stroke). The hydraulic actuator 40 includes the boom cylinder 10, arm cylinder 11, bucket cylinder 12, travel motor 5M, and a swing motor (not shown) shown in FIG. 1.

[0032] In this example, the oil supplied to the hydraulic actuator 40 to operate the hydraulic actuator 40 is referred to as hydraulic oil. Also, the oil supplied to the directional control valve 34 to operate the directional control valve 34 is referred to as pilot oil. Also, the pressure of the pilot oil is referred to as pilot oil pressure.

[0033] The operating device 25 is located in the operator's cab 4. The operator operates the operating device 25. The operating device 25 has a first operating lever 25R and a second operating lever 25L. The boom 6 and bucket 8 are operated by the first operating lever 25R located on the right side of the operator's seat 4S. The arm 7 and rotating body 3 are operated by the second operating lever 25L located on the left side of the operator's seat 4S. In this example, the operating device 25 is a pilot hydraulic operating device. Pilot oil that is delivered from a hydraulic pump 33 and reduced in pressure by a pressure reducing valve is supplied to the operating device 25.

[0034] The operating device 25 and the directional control valve 34 are connected via a pilot oil passage 450. When the operating device 25 is operated, a pilot oil pressure corresponding to the operation of the operating device 25 is supplied to the directional control valve 34 via the pilot oil passage 450. This adjusts the position of the spool of the directional control valve 34. A pressure sensor 36 is arranged in the pilot oil passage 450. The pressure sensor 36 detects the pilot oil pressure. The detection result of the pressure sensor 36 is output to the controller 102.

[0035] The control device 200 is configured so that the hydraulic pump 33 is driven by the engine 31, and the hydraulic oil discharged from the hydraulic pump 33 is supplied to various hydraulic actuators 40 via a direction control valve 34 in response to the operation of the control device 25 by the operator. By adjusting the flow direction and flow rate of the hydraulic oil supplied to the hydraulic actuators 40, the operation of the work implement 2, the rotation of the revolving unit 3, and the traveling operation of the traveling unit 5 are controlled.

[0036] <Estimation of Tip Position of Bucket 8: First Embodiment> FIG. 3 is a diagram showing an operating mode in the first embodiment of the hydraulic excavator 100, the object of which is to estimate the tip position of the bucket 8. In the operating mode in the first embodiment, the hydraulic excavator 100 is placed on flat ground G. The bucket 8 is away from the ground G. The boom 6 rotates relative to the revolving unit 3, and the arm 7 rotates relative to the boom 6. The bucket 8 does not rotate relative to the arm 7. The relative position of the bucket 8 to the arm 7 is kept constant. The length of the bucket cylinder 12 is constant.

[0037] Fig. 4 is a block diagram showing a process for estimating the tip position of the bucket 8 according to the first embodiment. The controller 102 acquires a captured image 61 captured by the imaging device 20 from the imaging device 20. The captured image 61 shown in Fig. 4 includes the terrain ahead of the hydraulic excavator 100 and the work implement 2. The captured image 61 also includes the bucket 8, a portion of the arm 7, and a portion of the boom 6.

[0038] The controller 102 may acquire in real time the captured images 61 captured by the imaging device 20. The controller 102 may acquire the captured images 61 captured by the imaging device 20 at a predetermined time or at predetermined intervals.

[0039] The controller 102 has a trained model 70A for calculating the area of ​​the bucket 8 in the captured image 61. FIG. 5 is a schematic diagram of the trained model 70A. The trained model 70A includes the neural network shown in FIG. 5. The neural network includes an input layer 71, an intermediate layer (hidden layer) 72, and an output layer 73. The intermediate layer 72 is multi-layered. The input layer 71, the intermediate layer 72, and the output layer 73 each have one or more units (neurons). The number of units in the input layer 71, the intermediate layer 72, and the output layer 73 can be set as appropriate.

[0040] Units in adjacent layers are connected, and each connection has a weight. A bias is set for each unit. A threshold is set for each unit. The output value of each unit is determined by whether the sum of the products of the input values ​​to each unit and the weights plus the bias exceeds the threshold.

[0041] The trained model 70A has been trained to determine the area of ​​the bucket 8 in the captured image 61 and the position of the bucket 8 in the captured image 61 from the captured image 61. Parameters of the trained model 70A obtained by training are stored in the controller 102. The parameters of the trained model 70A include, for example, the number of layers of the neural network, the number of units in each layer, the connection relationships between the units, the connection weights between each unit, the bias associated with each unit, and the threshold value for each unit.

[0042] The controller 102 inputs the captured image 61 captured by the imaging device 20 to the input layer 71. The output layer 73 outputs output values ​​indicating the area of ​​the bucket 8 in the captured image 61 and the position of the bucket 8 in the captured image 61. For example, the controller 102 uses the captured image 61 as an input to the input layer 71 to perform a calculation process of forward propagation of the neural network of the trained model 70A. As a result, the controller 102 obtains, as output values ​​output from the output layer 73 of the neural network, an estimated bucket area obtained by estimating the area of ​​the bucket 8 in the captured image 61 and an in-image bucket position obtained by estimating the position of the bucket 8 in the captured image 61.

[0043] 6 is a schematic diagram showing the area in which the bucket 8 appears in the captured image 61. A Cartesian coordinate system is set in the captured image 61. The left-right direction in FIG. 6 is set as the X direction, and the up-down direction in FIG. 6 is set as the Y direction. If the captured image 61 is a rectangular image, the direction in which one of the two orthogonal sides that form the vertices of the rectangle extends may be set as the X direction, and the direction in which the other side extends may be set as the Y direction.

[0044] Controller 102 imports captured image 61 captured by imaging device 20 and obtains the area and position of bucket 8 in captured image 61 through image recognition using trained model 70A. By comparing the obtained area and position of bucket 8 with the original captured image 61, controller 102 determines the area in which bucket 8 appears in captured image 61, as shown in Fig. 6. The in-image bucket position, which is the position of bucket 8 in captured image 61, is represented by the position in the X direction (X coordinate) and the position in the Y direction (Y coordinate) of the area in which bucket 8 appears in captured image 61.

[0045] The trained model 70A has undergone a learning process using a training dataset so that, when a captured image 61 is input, the trained model 70A outputs an estimated bucket area obtained by estimating the area of ​​the bucket 8 in the captured image 61 and an in-image bucket position obtained by estimating the position of the bucket 8 in the captured image 61. The trained model 70A is generated by a learning process using a training dataset. The training dataset includes multiple pieces of training data labeled with the estimated bucket area obtained by estimating the area of ​​the bucket 8 in the captured image 61 and the in-image bucket position obtained by estimating the position of the bucket 8 in the captured image 61 for the captured image 61.

[0046] Alternatively, the controller 102 may use the trained model 70A to extract the area in the captured image 61 in which the bucket 8 appears ( FIG. 6 ). The trained model 70A may use the captured image 61 as input to the input layer 71 and output the area in the captured image 61 in which the bucket 8 appears from the output layer 73. Outside the trained model 70A, the controller 102 may extract the number of pixels in the area in which the bucket 8 appears and calculate the number of extracted pixels to obtain an estimated bucket area that estimates the area of ​​the bucket 8 in the captured image 61. The controller 102 may also obtain the in-image bucket position, which is the position of the bucket 8 in the captured image 61, by calculating the X coordinate and Y coordinate of the area in which the bucket 8 appears.

[0047] Next, the controller 102 estimates the tip position of the bucket 8 based on a map showing the correspondence relationship between the area of ​​the bucket 8 in the captured image 61 and the tip position of the bucket 8. FIG. 7 is a schematic diagram showing a response surface 80. The controller 102 has the response surface 80 shown in FIG. 7. The response surface 80 corresponds to an example of a map showing the correspondence relationship between the area of ​​the bucket 8 and the tip position of the bucket 8. The response surface 80 is created by experimentally obtaining the tip position of the bucket 8 relative to the area of ​​the bucket 8 in the captured image 61, and turning this data into a map, which is stored in advance in the controller 102.

[0048] Response surface 80 shown in FIG. 7 indicates the correspondence relationship between the estimated bucket area and the in-image bucket position and the tip position of bucket 8, and is a map for converting the estimated bucket area and the in-image bucket position into the tip position of bucket 8. Controller 102 applies the acquired estimated bucket area and in-image bucket position to response surface 80 shown in FIG. 7 to determine the tip position of bucket 8. Controller 102 acquires the tip position of bucket 8 based on response surface 80 shown in FIG. 7 using the estimated bucket area and in-image bucket position as parameters. The in-image bucket position corresponds to an example of incidental information in the first embodiment. The in-image bucket position can also be considered incidental information related to work machine 2.

[0049] Meanwhile, the controller 102 estimates a target position, which is a position to which the tip position of the bucket 8 should be moved, from the captured image 61. The controller 102 may estimate the target position from the topography, ground conditions, and object conditions in the captured image. The controller 102 may also estimate the target position from target information according to the work content, such as data related to the final construction surface. The controller 102 may estimate the target position by combining the captured image 61 and target information according to the work content.

[0050] The controller 102 calculates the difference between the estimated tip position of the bucket 8 and the target position. The controller 102 further has a PID controller 90 as an example of a difference cancellation controller. The controller 102 can determine the amount of correction for the calculated difference between the calculated estimated tip position of the bucket 8 and the target position by combining three operations: proportional operation, integral operation, and derivative operation. The stability of the control can be improved by appropriately adjusting the parameters of the proportional operation, integral operation, and derivative operation.

[0051] A control is incorporated that corrects the difference between the tip position of the bucket 8 estimated using the area of ​​the bucket 8 and the target position of the bucket 8, and a work implement operation command, which is a command signal obtained by correcting the difference between the estimated tip position of the bucket 8 and the target position, is output to the directional control valve 34 (FIG. 2). The spool of the directional control valve 34 moves in accordance with the work implement operation command, causing the hydraulic actuator 40 to operate. This makes it possible to eliminate the difference between the estimated tip position of the bucket 8 and the target position, and to move the tip of the bucket 8 closer to the target position. It becomes possible to move the tip of the bucket 8 to the intended position. This makes it possible to cause the hydraulic excavator 100 to perform the intended operation.

[0052] The difference cancellation controller is not limited to PID control, and fuzzy control or other techniques may also be applied.

[0053] <Estimation of Tip Position of Bucket 8: Second Embodiment> FIG. 8 is a diagram showing an operating mode in the second embodiment of the hydraulic excavator 100, the object of which is to estimate the tip position of the bucket 8. In the operating mode in the second embodiment, the hydraulic excavator 100 is placed on flat ground G. The bucket 8 is away from the ground G. The boom 6 rotates relative to the revolving unit 3, and the arm 7 rotates relative to the boom 6. The bucket 8 rotates relative to the arm 7 around the arm top pin 15. The relative position of the bucket 8 with respect to the arm 7 is not constant, but rather fluctuates as appropriate.

[0054] FIG. 9 is a block diagram showing a process for estimating the tip position of the bucket 8 according to the second embodiment. The trained model 70A shown in FIG. 9 has been trained to determine, from a captured image 61, the estimated bucket area and the bucket position within the image, as well as the area and position in the captured image 61 of an accessory attached to the bucket 8. The accessory is, for example, a bucket tooth 8T. The position of a tooth stop 8TT at the tip of the bucket tooth 8T is the tip position of the bucket 8. The accessory is not limited to a bucket tooth 8T, and may be, for example, a side cutter.

[0055] The trained model 70A has a neural network similar to that shown in Fig. 5. The controller 102 inputs the captured image 61 to the input layer 71. The controller 102 obtains, as output values ​​from the output layer 73, an estimated bucket area, a bucket position within the image, an estimated tooth area obtained by estimating the area of ​​the bucket tooth 8T within the captured image 61, and a tooth position within the image obtained by estimating the position of the bucket tooth 8T within the captured image 61.

[0056] 10 is a schematic diagram showing the area in which the bucket tooth 8T appears in the captured image 61. The controller 102 imports the captured image 61 and obtains the area and position of the bucket tooth 8T in the captured image 61 through image recognition using the trained model 70A. By comparing the obtained area and position of the bucket tooth 8T with the original captured image 61, the controller 102 determines the area in which the bucket tooth 8T appears in the captured image 61, as shown in FIG. The in-image tooth position, which is the position of the bucket tooth 8T in the captured image 61, is represented by the position in the X direction (X coordinate) and the position in the Y direction (Y coordinate) of the area in which the bucket tooth 8T appears in the captured image 61.

[0057] Next, the controller 102 estimates the tip position of the bucket 8 based on a map indicating the correspondence between the estimated bucket area, bucket position in the image, estimated tooth area, and tooth position in the image, and the tip position of the bucket 8. The controller 102 applies the acquired estimated bucket area, bucket position in the image, estimated tooth area, and tooth position in the image to the map to acquire the tip position of the bucket 8. The bucket position in the image, estimated tooth area, and tooth position in the image correspond to an example of incidental information in the second embodiment. The bucket position in the image, estimated tooth area, and tooth position in the image can also be considered incidental information related to the work machine 2. The subsequent processing is the same as in the first embodiment, so a repeated description will not be given here.

[0058] The rotation direction of bucket 8 relative to arm 7 can be determined from information on the area and position of accessories attached to bucket 8 in captured image 61. By using this as additional information, the tip position of bucket 8 can be estimated with higher accuracy.

[0059] <Estimation of Tip Position of Bucket 8: Third Embodiment> 11 is a diagram showing an operating mode in the third embodiment of the hydraulic excavator 100, the object of which is to estimate the tip position of the bucket 8. In the operating mode in the third embodiment, the hydraulic excavator 100 is performing work to excavate the ground G, and part of the ground G is raised. The bucket 8 is in contact with the ground G. The boom 6 rotates relative to the revolving unit 3, and the arm 7 rotates relative to the boom 6. The bucket 8 rotates relative to the arm 7. The relative position of the bucket 8 to the arm 7 is not constant, but rather fluctuates as appropriate. The soil excavated by the bucket 8 is interposed between the bucket 8 and the imaging device 20 attached to the revolving unit 3.

[0060] Fig. 12 is a block diagram showing a process for estimating the tip position of the bucket 8 according to the third embodiment. The controller 102 acquires a captured image 62 captured by the imaging device 20 from the imaging device 20. The captured image 62 shown in Fig. 12 includes the terrain ahead of the hydraulic excavator 100 and the work implement 2. The captured image 62 also includes a part of the bucket 8, a part of the arm 7, and a part of the boom 6. A part of the bucket 8 is hidden by a raised portion of the ground G. The bucket tooth 8T is not included in the captured image 62. Occlusion has occurred, where a part of the bucket 8 is hidden in the ground.

[0061] Trained model 70A for determining the area of ​​bucket 8 in captured image 62 has been trained to determine, from captured image 62, an estimated bucket area, the bucket position within the image, and the area and position in captured image 62 of an accessory (bucket tooth 8T) attached to bucket 8. Note that since captured image 62 shown in Fig. 12 does not include bucket tooth 8T, the area of ​​bucket tooth 8T in captured image 62 is set to zero, and the position of bucket tooth 8T in captured image 62 is considered to be non-existent.

[0062] The controller 102 has a trained model 70B for calculating the area of ​​the arm 7 in the captured image 62. The trained model 70B has a neural network similar to that shown in FIG. 5. The trained model 70B has been trained to calculate the area of ​​the arm 7 in the captured image 62 and the position of the arm 7 in the captured image 62 from the captured image 62. The controller 102 inputs the captured image 62 to the input layer 71 of the trained model 70B. The controller 102 acquires, as output values ​​output from the output layer 73 of the trained model 70B, an estimated arm area obtained by estimating the area of ​​the arm 7 in the captured image 62 and an in-image arm position obtained by estimating the position of the arm 7 in the captured image 62.

[0063] The trained model 70B has undergone a learning process using a training dataset so that, when a captured image 62 is input, the trained model 70B outputs an estimated arm area obtained by estimating the area of ​​the arm 7 in the captured image 62 and an in-image arm position obtained by estimating the position of the arm 7 in the captured image 62. The trained model 70B has been generated by a learning process using a training dataset. The training dataset includes a plurality of training data labeled with an estimated arm area obtained by estimating the area of ​​the arm 7 in the captured image 62 and an in-image arm position obtained by estimating the position of the arm 7 in the captured image 62 for the captured image 62.

[0064] 13 is a schematic diagram showing an area in which a portion of the bucket 8 is captured in the captured image 62. The controller 102 imports the captured image 62 captured by the imaging device 20 and obtains the area of ​​a portion of the bucket 8 included in the captured image 62 and the position of that portion of the bucket 8 through image recognition using the trained model 70A. By comparing the area and position of the acquired portion of the bucket 8 with the original captured image 62, the controller 102 determines the area in the captured image 62 in which a portion of the bucket 8 is captured, as shown in FIG. The in-image bucket position, which is the position of the bucket 8 in the captured image 62, is represented by the position in the X direction (X coordinate) and the position in the Y direction (Y coordinate) of the area in which the bucket 8 is captured in the captured image 62.

[0065] 14 is a schematic diagram showing the area in which the arm 7 appears in the captured image 62. The controller 102 imports the captured image 62 and obtains the area and position of the arm 7 in the captured image 62 by image recognition using the trained model 70B. By comparing the obtained area and position of the arm 7 with the original captured image 62, the controller 102 determines the area in which the arm 7 appears in the captured image 62, as shown in FIG. 14. The in-image arm position, which is the position of the arm 7 in the captured image 62, is represented by the position in the X direction (X coordinate) and the position in the Y direction (Y coordinate) of the area in which the arm 7 appears in the captured image 62.

[0066] Next, the controller 102 estimates the tip position of the bucket 8 based on a map indicating the correspondence between the estimated bucket area, bucket position in the image, estimated tooth area, tooth position in the image, estimated arm area, and arm position in the image, and the tip position of the bucket 8. The controller 102 applies the acquired estimated bucket area, bucket position in the image, estimated tooth area, tooth position in the image, estimated arm area, and arm position in the image to the map to acquire the tip position of the bucket 8. The bucket position in the image, estimated tooth area, tooth position in the image, estimated arm area, and arm position in the image correspond to an example of incidental information in the third embodiment. The bucket position in the image, estimated tooth area, tooth position in the image, estimated arm area, and arm position in the image can also be considered incidental information related to the work machine 2. The subsequent processing is the same as in the first embodiment, so a repeated description will not be given here.

[0067] In the explanation of the embodiment so far, an example has been described in which the tip position of bucket 8 is estimated using information obtained from an image of work machine 2 captured by imaging device 20. The additional information used together with the area of ​​bucket 8 to estimate the tip position of bucket 8 may include the shape of the ground surface G. The image of ground surface G in front of hydraulic excavator 100 is captured by imaging device 20, and the shape of the ground surface G can be recognized from changes in the area of ​​the ground surface G in the captured image.

[0068] The incidental information may also include detection information from sensors attached to the hydraulic excavator 100. For example, it can be determined that the bucket 8 is in contact with the ground G from the detection results of the pressure of the hydraulic oil supplied to the hydraulic cylinder that drives the work implement 2. The incidental information may also include the hydraulic cylinder length obtained from the detection results of a cylinder stroke sensor, the coordinates of the center of gravity of the arm 7, the coordinates of the center of gravity of the boom 6, and the like, and by using these as incidental information, the position of the tip of the bucket 8 can be estimated with greater accuracy. The hydraulic cylinder length, the coordinates of the center of gravity of the arm 7, and the coordinates of the center of gravity of the boom 6 can also be considered incidental information related to the work implement 2.

[0069] The additional information may also include the details of the work being performed by the hydraulic excavator 100. For example, based on information that plowing work is being performed in which the bucket teeth 8T is moved horizontally relative to the ground G to level the ground, it is possible to determine that the bucket teeth 8T are located on the ground G. In this case, the area of ​​the bucket 8 in the captured image 61 can be used to estimate where on the ground G the tooth stop 8TT is located.

[0070] In the description of the embodiments, a hydraulic excavator 100 is given as an example of a work machine, but the work machine is not limited to the hydraulic excavator 100 and may be an electric excavator, a hybrid excavator, or the like. The actuators of the work machine may not be hydraulic, but may be electric, pneumatic, or the like. The concept of the present disclosure may also be applied to other types of work machines, such as wheel loaders.

[0071] <Action and effect> The characteristic configuration and effects of this embodiment are summarized as follows.

[0072] 4, the controller 102 acquires the area of ​​the bucket 8 in the captured image 61 from the captured image 61 including the work implement 2, which is captured by the imaging device 20. The controller 102 estimates the position of the tip of the bucket 8 using the acquired area of ​​the bucket 8 and the accompanying information.

[0073] When an operator in the cab 4 manually operates the hydraulic excavator 100, the operator visually grasps the posture of the work implement 2 and performs the operation. Similar to the operator's vision, the configuration for estimating the tip position of the bucket 8 can be simplified by estimating information about the tip of the bucket 8 from information obtained from an image of the bucket 8. Even in cases where the state of the excavation target changes, such as during excavation work, the trajectory for moving the tip position of the bucket 8 can be determined in accordance with the site conditions, just as when the operator operates the excavator. When assisting the operator's operation with partial automatic control, estimating the tip position of the bucket 8 using information obtained by simulating the operator's vision makes it possible to coordinate the operator operation with the automatic control without causing the operator to feel uncomfortable.

[0074] 4 and 5, the controller 102 may have a trained model 70A for determining the area of ​​the bucket 8, and may obtain an output that estimates the area of ​​the bucket 8 by inputting a captured image 61 into the trained model 70A. The area of ​​the bucket 8 can be estimated with high accuracy by previously training a model using the captured image 61 of the bucket 8 and the area of ​​the bucket 8 in the captured image 61 as training data, and then inputting the captured image 61 captured on-site into the trained model 70A.

[0075] 4 and 7, the controller 102 may have a map indicating the correspondence relationship between the area of ​​the bucket 8 and the tip position of the bucket 8, and may estimate the tip position of the bucket 8 based on this map. A map indicating the correspondence relationship between the area of ​​the bucket 8 and the tip position of the bucket 8 is experimentally created in advance and stored in the controller 102, and the estimated area of ​​the bucket 8 is applied to this map, thereby making it possible to estimate the tip position of the bucket 8 with high accuracy.

[0076] 4 and 7, the controller 102 may have a map showing the correspondence between the area and incidental information of the bucket 8 and the tip position of the bucket 8. The incidental information is added to a map for determining the tip position of the bucket 8, and a map showing the correspondence that indicates the tip position of the bucket 8 when the area of ​​the bucket 8 and the incidental information are combined is created in advance and stored in the controller 102. By applying the estimated area of ​​the bucket 8 and the incidental information to the map, the tip position of the bucket 8 can be estimated with higher accuracy.

[0077] 4 and 6, the position of bucket 8 in captured image 61 may be included in the supplementary information. Controller 102 can accurately estimate the tip position of bucket 8 using the acquired area of ​​bucket 8 and the position of bucket 8 in captured image 61. Because the tip position of bucket 8 can be estimated based on information obtained from captured image 61 captured by imaging device 20 without using detection results from various sensors, the configuration for estimating the tip position of bucket 8 can be simplified.

[0078] 9 and 10, the area of ​​an accessory attached to bucket 8 in captured image 61 may be included in the supplementary information. Controller 102 can accurately estimate the tip position of bucket 8 using the acquired area of ​​bucket 8 and the area of ​​the accessory. Because the tip position of bucket 8 can be estimated based on information obtained from captured image 61 captured by imaging device 20 without using detection results from various sensors, the configuration for estimating the tip position of bucket 8 can be simplified.

[0079] 9 and 10, the positions of accessories attached to bucket 8 in captured image 61 may be included in the supplementary information. Controller 102 can accurately estimate the tip position of bucket 8 using the acquired area of ​​bucket 8 and the positions of the accessories. Because the tip position of bucket 8 can be estimated based on information obtained from captured image 61 captured by imaging device 20 without using detection results from various sensors, the configuration for estimating the tip position of bucket 8 can be simplified.

[0080] 9 and 10, the attachment attached to the bucket 8 has bucket teeth 8T, the bucket teeth 8T have tooth stops 8TT at their tips, and the tip position of the bucket 8 may be the position of the tooth stops 8TT. The controller 102 can accurately estimate the position of the tooth stops 8TT using the acquired area of ​​the bucket 8 and the area and / or position of the bucket teeth 8T.

[0081] 12 and 14, the area of ​​arm 7 to which bucket 8 is connected in captured image 62 may be included in the supplementary information. Controller 102 can accurately estimate the tip position of bucket 8 using the acquired area of ​​bucket 8 and the area of ​​arm 7 in captured image 62. Because the tip position of bucket 8 can be estimated based on information obtained from captured image 62 captured by imaging device 20 without using detection results from various sensors, the configuration for estimating the tip position of bucket 8 can be simplified.

[0082] 12 and 14, the position of arm 7 to which bucket 8 is connected in captured image 62 may be included in the supplementary information. Controller 102 can accurately estimate the tip position of bucket 8 using the acquired area of ​​bucket 8 and the position of arm 7 in captured image 62. Because the tip position of bucket 8 can be estimated based on information obtained from captured image 62 captured by imaging device 20 without using detection results from various sensors, the configuration for estimating the tip position of bucket 8 can be simplified.

[0083] <Additional Notes> The above description includes the following additional features.

[0084] (Appendix 1) a work implement including a bucket; an imaging device that images the work machine; a controller that acquires an area of ​​the bucket in an image captured by the imaging device from an image including the work machine, and estimates a tip position of the bucket using the acquired area of ​​the bucket and accompanying information.

[0085] (Appendix 2) 2. The work machine of claim 1, wherein the controller has a trained model for determining the area of ​​the bucket, and obtains an output that estimates the area of ​​the bucket by inputting the captured image into the trained model.

[0086] (Appendix 3) The work machine according to claim 1 or 2, wherein the controller has a map indicating a correspondence relationship between an area of ​​the bucket and the tip position, and estimates the tip position based on the map.

[0087] (Appendix 4) 4. The work machine according to claim 3, wherein the map indicates a correspondence relationship between the area of ​​the bucket and the auxiliary information, and the tip position.

[0088] (Appendix 5) 5. The work machine according to claim 1, wherein the additional information includes a position of the bucket in the captured image.

[0089] (Appendix 6) the work machine includes an attachment attached to the bucket; 6. The work machine according to any one of Supplementary Note 1 to Supplementary Note 5, wherein the supplementary information includes an area of ​​the accessory part in the captured image.

[0090] (Appendix 7) The work machine according to claim 6, wherein the supplementary information includes the position of the accessory part in the captured image.

[0091] (Appendix 8) The accessory includes a bucket tooth, and the bucket tooth has a cutting edge at a tip thereof. The work machine according to claim 6 or 7, wherein the tip position of the bucket is the position of the cutting edge.

[0092] (Appendix 9) The work machine further includes an arm to which the bucket is connected, The work machine according to any one of Supplementary Note 1 to Supplementary Note 8, wherein the supplementary information includes an area of ​​the arm in the captured image.

[0093] (Appendix 10) 10. The work machine according to claim 9, wherein the supplementary information includes the position of the arm in the captured image.

[0094] (Appendix 11) acquiring a captured image of a work implement including a bucket; acquiring an area of ​​the bucket in the captured image; and estimating the position of the tip of the bucket using the acquired area of ​​the bucket and additional information.

[0095] (Appendix 12) 12. The method of claim 11, wherein the step of acquiring the area includes inputting the captured image into a trained model for determining the area of ​​the bucket to obtain an output that estimates the area of ​​the bucket.

[0096] (Appendix 13) 13. The method of claim 11 or 12, wherein the estimating step includes estimating the tip position based on a map indicating a correspondence relationship between an area of ​​the bucket and the tip position.

[0097] (Appendix 14) 14. The method according to claim 13, wherein the map indicates a correspondence between the area of ​​the bucket, the additional information, and the tip position.

[0098] (Appendix 15) 15. The method according to any one of claims 11 to 14, wherein the additional information includes a position of the bucket in the captured image.

[0099] (Appendix 16) the work machine includes an attachment attached to the bucket; 16. The method according to any one of claims 11 to 15, wherein the additional information includes an area of ​​the accessory part in the captured image.

[0100] (Appendix 17) 17. The method of claim 16, wherein the additional information includes the position of the accessory part in the captured image.

[0101] (Appendix 18) The accessory includes a bucket tooth, and the bucket tooth has a cutting edge at a tip thereof. 18. The method of claim 16 or 17, wherein the tip position of the bucket is the position of the cutting edge.

[0102] (Appendix 19) The work machine further includes an arm to which the bucket is connected, 19. The method according to any one of appendices 11 to 18, wherein the additional information includes an area of ​​the arm in the captured image.

[0103] (Appendix 20) 20. The method of claim 19, wherein the additional information includes the position of the arm in the captured image.

[0104] The embodiments disclosed herein are to be considered in all respects as illustrative and not restrictive. The scope of the present invention is defined by the claims, not by the above description, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0105] 1 main body, 2 work equipment, 3 rotating body, 4 operator's cab, 5 traveling body, 5M traveling motor, 6 boom, 7 arm, 8 bucket, 8T bucket tooth, 8TT two-stop (cutting tip), 9 exterior cover, 10 boom cylinder, 11 arm cylinder, 12 bucket cylinder, 20 imaging device, 21 position and orientation sensor, 25 operation device, 31 engine, 33 hydraulic pump, 34 directional control valve, 36 pressure sensor, 40 hydraulic actuator, 61, 62 captured image, 70A, 70B trained model, 71 input layer, 72 intermediate layer, 73 output layer, 80 response surface, 90 PID controller, 100 hydraulic excavator, 102 computer, 200 control device, G ground, RX rotation axis.

Claims

1. a work implement including a bucket; an imaging device that images the work machine; a controller that acquires an area of ​​the bucket in an image captured by the imaging device from an image including the work machine, and estimates a tip position of the bucket using the acquired area of ​​the bucket and accompanying information.

2. 2. The work machine according to claim 1, wherein the controller has a trained model for determining the area of ​​the bucket, and obtains an output that estimates the area of ​​the bucket by inputting the captured image into the trained model.

3. The work machine according to claim 1 , wherein the controller has a map indicating a correspondence relationship between an area of ​​the bucket and the tip position, and estimates the tip position based on the map.

4. The work machine according to claim 3 , wherein the map indicates a correspondence relationship between the area of ​​the bucket, the additional information, and the tip position.

5. The work machine according to claim 1 , wherein the additional information includes a position of the bucket in the captured image.

6. the work machine includes an attachment attached to the bucket; The work machine according to claim 1 , wherein the additional information includes an area of ​​the accessory part in the captured image.

7. The work machine according to claim 6 , wherein the supplementary information includes a position of the accessory part in the captured image.

8. The accessory includes a bucket tooth, and the bucket tooth has a cutting edge at a tip thereof. The work machine according to claim 6 or 7, wherein the tip position of the bucket is the position of the cutting edge.

9. The work machine further includes an arm to which the bucket is connected, The work machine according to claim 1 , wherein the additional information includes an area of ​​the arm in the captured image.

10. The work machine according to claim 9 , wherein the supplementary information includes a position of the arm in the captured image.

11. acquiring a captured image of a work implement including a bucket; acquiring an area of ​​the bucket in the captured image; and estimating the position of the tip of the bucket using the acquired area of ​​the bucket and additional information.

12. The method of claim 11 , wherein the step of obtaining the area includes obtaining an output that estimates the area of ​​the bucket by inputting the captured image into a trained model for determining the area of ​​the bucket.

13. The method of claim 11 , wherein the estimating step includes estimating the tip position based on a map indicating a correspondence between an area of ​​the bucket and the tip position.

14. The method according to claim 13 , wherein the map indicates a correspondence relationship between the area of ​​the bucket and the additional information and the tip position.

15. The method of claim 11 , wherein the extrinsic information includes a position of the bucket in the captured image.

16. the work machine includes an attachment attached to the bucket; The method according to claim 11 , wherein the additional information includes an area of ​​the accessory part in the captured image.

17. The method of claim 16 , wherein the additional information includes a position of the attachment in the captured image.

18. The accessory includes a bucket tooth, and the bucket tooth has a cutting edge at a tip thereof.

18. The method according to claim 16 or claim 17, wherein the tip position of the bucket is the position of the cutting edge.

19. The work machine further includes an arm to which the bucket is connected, The method according to claim 11 , wherein the additional information includes an area of ​​the arm in the captured image.

20. The method according to claim 19 , wherein the additional information includes a position of the arm in the captured image.

Images