Working machinery

Abstract

To provide a work machine that can appropriately control a loading work onto a transport machine by verifying validity of a recognition result of the transport machine.SOLUTION: A hydraulic shovel 1 comprises: an outside measuring device 70 that measures a surrounding environment of the hydraulic shovel 1; an information processing device 54 that recognizes a dump truck 200 present around the hydraulic shovel 1 based on measurement results of the outside measuring device 70; a control device 40 that controls an operation of a vehicle body of the hydraulic shovel 1 based on the recognition result of the information processing device 54; and a truck information acquisition device 56 that acquires a position and vehicle grade information of the dump truck 200 through external communication. The information processing device 54 corrects a recognition result of the dump truck 200 based on the information acquired by the truck information acquisition device 56. The control device 40 controls an operation of the vehicle body of the hydraulic shovel 1 based on the corrected recognition result of the dump truck 200.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present invention relates to a working machine.

Background Art

[0002] There is known an articulated working machine (such as a hydraulic excavator) having a front working device (such as attachments such as a boom, an arm, and a bucket) driven by a hydraulic actuator. This type of working machine performs a loading operation of loading an object such as excavated earth and sand onto a transport machine (such as a dump truck) which is a loading machine.

[0003] When performing the loading operation, if the front working device is swiveled at a position lower than the transport machine in terms of height (for example, the height of the bucket), there is a possibility that the front working device may collide with the transport machine. Therefore, there is a need for a function to assist the operation of the operator of the working machine performing the loading operation and a technology for the working machine to automatically perform the loading operation. When the working machine performs the loading operation semi-automatically or fully automatically (hereinafter collectively referred to as "automatically"), it is necessary for the working machine to accurately recognize the position and attitude of the transport machine.

[0004] As a conventional technique for recognizing a transport machine, for example, there is the technique described in Patent Document 1. Patent Document 1 discloses an image processing system including a data acquisition unit that acquires an imaging image in which a loading / unloading target of a transported object of a working machine is imaged, a region specification unit that specifies a region including the loading / unloading target from the imaging image, and a loading / unloading target specification unit that specifies at least one predetermined surface of the loading / unloading target from the region including the loading / unloading target.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] The technology disclosed in Patent Document 1 uses a machine learning-based image recognition system to detect the position and orientation of a transport machine from images captured by a camera attached to the work machine, in order to identify the transport machine. However, with a machine learning-based recognition system, it is difficult to guarantee the validity and accuracy of the recognition results for sites or transport machines from which training data has not been collected.

[0007] Furthermore, if a third-party system is incorporated as the recognition system for the transport machinery, the development company of the work machinery may not be able to disclose the details of the recognition system's algorithm. It is not easy for the development company of the work machinery to verify the recognition results of the recognition system. It is difficult for the development company of the work machinery to determine in advance whether it is possible to properly control the loading operation without collisions with the transport machinery using the recognition results of the recognition system.

[0008] The present invention has been made in view of the above, and aims to provide a work machine that can appropriately control the loading operation onto a transport machine by accurately recognizing the position and orientation of the transport machine in various situations. [Means for solving the problem]

[0009] To solve the above problems, the present invention provides a work machine for loading an object onto a transport machine, comprising: an external environment measuring device for measuring the surrounding environment of the work machine's body; an information processing device for recognizing the transport machine present around the body based on the measurement results of the external environment measuring device; a control device for controlling the operation of the body based on the recognition results of the information processing device; and a transport machine information acquisition device for acquiring the position and vehicle class information of the transport machine from the outside, wherein the information processing device corrects the recognition result of the transport machine based on the position and vehicle class information of the transport machine acquired by the transport machine information acquisition device, and the control device controls the operation of the body based on the corrected recognition result of the transport machine. [Effects of the Invention]

[0010] According to the present invention, it is possible to provide a work machine that can appropriately control the loading operation onto a transport machine by accurately recognizing the position and orientation of the transport machine in various situations. Other issues, configurations, and effects will be clarified by the following description of the embodiments. [Brief explanation of the drawing]

[0011] [Figure 1] A schematic side view showing the external configuration of a hydraulic excavator, which is an example of a work machine according to Embodiment 1. [Figure 2] A block diagram showing the configuration of the hydraulic system and control system mounted on the hydraulic excavator shown in Figure 1. [Figure 3] Figure 2 illustrates an example of the operation of a hydraulic excavator. [Figure 4] Figure 2 shows a flowchart illustrating an example of the operation of a hydraulic excavator. [Figure 5] A block diagram illustrating the functional configuration of the information processing device shown in Figure 2. [Figure 6] A side view of the coordinate systems set for a hydraulic excavator. [Figure 7] Figure 6 shows each coordinate system viewed from above. [Figure 8] A diagram showing the coordinate system of a dump truck as viewed from the side. [Figure 9] Figure 8 shows the vehicle coordinate system as viewed from above. [Figure 10] A flowchart illustrating an example of the recognition result verification process. [Figure 11] A diagram illustrating the degree of overlap between the recognition region and the estimation region. [Figure 12] This diagram shows a table that defines the criteria for determining the validity of recognition results and the method for correcting the recognition area for each task performed by a hydraulic excavator. [Figure 13] A diagram illustrating the method for correcting the recognition area when acquiring a loading task. [Figure 14] A diagram illustrating the method for correcting the recognition area when acquiring data during a reaching task. [Figure 15] A diagram showing an example of a display of a display device when the accuracy of the recognition result is below a threshold value. [Figure 16] A block diagram for explaining the functional configuration of the information processing apparatus according to Embodiment 2. [Figure 17] A diagram for explaining the processing of the track attitude estimation unit shown in FIG. 16. [Figure 18] A block diagram for explaining the functional configuration of the information processing apparatus according to Embodiment 3.

Mode for Carrying Out the Invention

[0012] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In each embodiment, components denoted by the same reference numerals have the same functions in each embodiment, and the description thereof will be omitted unless otherwise specified.

[0013] The working machine of the present embodiment is a working machine that performs a loading operation of loading an object such as excavated earth and sand onto a transport machine that is a loading machine. Hereinafter, a hydraulic excavator equipped with a bucket as a working machine and a dump truck as a transport machine will be described as an example. However, the working machine of the present embodiment may be a hydraulic excavator equipped with an attachment other than a bucket, or may be a working machine other than a hydraulic excavator. The transport machine of the present embodiment may be a transport machine other than a dump truck.

[0014] [Embodiment 1] The hydraulic excavator 1 according to Embodiment 1 will be described with reference to FIGS. 1 to 15.

[0015] <External Configuration of Hydraulic Excavator> FIG. 1 is a side view schematically showing the external configuration of a hydraulic excavator 1 which is an example of the working machine according to Embodiment 1.

[0016] The hydraulic excavator 1 includes an articulated front working device 2 that holds an object and rotates in the vertical or front-rear direction, and a machine body 3 on which the front working device 2 is mounted.

[0017] The machine body 3 comprises a lower traveling body 5 which is driven by a right-travel hydraulic motor 4a and a left-travel hydraulic motor 4b provided on the right and left sides of the lower traveling body 5, and an upper slewing body 7 which is attached to the upper part of the lower traveling body 5 via a slewing device and rotates by a slewing hydraulic motor 6 of the slewing device. In this embodiment, the right-travel hydraulic motor 4a and the left-travel hydraulic motor 4b are collectively referred to as "travel hydraulic motors 4a, 4b".

[0018] The front working device 2 is a multi-jointed working device composed of multiple front members attached to the front of the upper slewing body 7. The upper slewing body 7 rotates with the front working device 2 mounted on it. The front working device 2 includes a boom 8 connected to the front of the upper slewing body 7 so as to be rotatable in the vertical direction, an arm 9 connected to the tip of the boom 8 so as to be rotatable in the vertical direction, and a bucket 10 connected to the tip of the arm 9 so as to be rotatable in the vertical direction.

[0019] The boom 8 is connected to the upper slewing body 7 by a boom pin 8a and rotates by the extension and retraction of the boom cylinder 11. The arm 9 is connected to the tip of the boom 8 by an arm pin 9a and rotates by the extension and retraction of the arm cylinder 12. The bucket 10 is connected to the tip of the arm 9 by a bucket pin 10a and a bucket link 16 and rotates by the extension and retraction of the bucket cylinder 13.

[0020] A boom angle sensor 14 is attached to the boom pin 8a to detect the rotation angle of the boom 8. An arm angle sensor 15 is attached to the arm pin 9a to detect the rotation angle of the arm 9. A bucket angle sensor 17 is attached to the bucket link 16 to detect the rotation angle of the bucket 10.

[0021] The rotation angles of the boom 8, arm 9, and bucket 10 may also be obtained by detecting the angles of the boom 8, arm 9, and bucket 10 with respect to a reference plane such as the horizontal plane using an inertial measuring device and converting them into rotation angles. Alternatively, the rotation angles of the boom 8, arm 9, and bucket 10 may also be obtained by detecting the strokes of the boom cylinder 11, arm cylinder 12, and bucket cylinder 13 using a stroke sensor and converting them into rotation angles.

[0022] The upper slewing body 7 is equipped with a tilt angle sensor 18 that detects the tilt angle of the machine body 3 with respect to a reference plane such as a horizontal plane. The slewing device between the lower traveling body 5 and the upper slewing body 7 is equipped with a slewing angle sensor 19 that detects the slewing angle, which is the relative angle of the upper slewing body 7 with respect to the lower traveling body 5. The upper slewing body 7 is equipped with an angular velocity sensor (not shown) that detects the angular velocity of the upper slewing body 7. In this embodiment, the boom angle sensor 14, arm angle sensor 15, bucket angle sensor 17, tilt angle sensor 18, and slewing angle sensor 19 are collectively referred to as the "attitude detection device 53". The attitude detection device 53 detects the rotation angles of the front working device 2 and the slewing angle of the upper slewing body 7, etc.

[0023] A display device 55, consisting of a touchscreen or the like capable of receiving operator input, is installed in the operator's cab located in the upper slewing body 7. Furthermore, an operating device for operating multiple hydraulic actuators 4a, 4b, 6, 11, 12, and 13 is installed in the operator's cab located in the upper slewing body 7. Specifically, the operating device comprises a right travel lever 23a for operating the right travel hydraulic motor 4a, a left travel lever 23b for operating the left travel hydraulic motor 4b, a right operating lever 22a for operating the boom cylinder 11 and bucket cylinder 13, and a left operating lever 22b for operating the arm cylinder 12 and slewing hydraulic motor 6. In this embodiment, the right travel lever 23a, left travel lever 23b, right operating lever 22a, and left operating lever 22b are collectively referred to as "operating levers 22 and 23."

[0024] Furthermore, an external environment measuring device 70 is attached to the upper rotating body 7 to measure the surrounding environment of the hydraulic excavator 1. The external environment measuring device 70 measures the depth (distance to the object) of objects present around the hydraulic excavator 1. As a result of the measurement, the external environment measuring device 70 acquires depth information of the object. The external environment measuring device 70 may be, for example, a LiDAR (Light Detection And Ranging) or a stereo camera. Multiple external environment measuring devices 70 may be attached to the hydraulic excavator 1.

[0025] Furthermore, a positioning device 60 is attached to the upper rotating body 7 to measure the position and orientation of the hydraulic excavator 1 at the site. The positioning device 60 acquires positioning information, including the position and orientation of the hydraulic excavator 1 at the site, as measurement results. The positioning device 60 may be, for example, a GNSS receiver or a surveying instrument such as a TS (Total Station). The positioning device 60 is not limited to these, and may be, for example, a camera fixed at the site. In this case, the positioning device 60 may measure the position and orientation of the hydraulic excavator 1 at the site based on information calculated from the detected image of the hydraulic excavator 1. In this case, it is desirable to have at least two positioning devices 60 in order to accurately measure the orientation of the hydraulic excavator 1.

[0026] Furthermore, the upper rotating body 7 is equipped with a truck information acquisition device 56 that acquires truck information, including the position, posture, and vehicle size information of the dump truck 200, from an external source. The vehicle size information includes information on the type and dimensions of the dump truck 200. The truck information acquisition device 56 may be a communication terminal that performs wireless communication with a dump truck 200 location and dispatch management system, such as an FMS (Fleet Management System). The truck information acquisition device 56 can acquire truck information by receiving truck information of the dump truck 200 transmitted from the dump truck 200 location and dispatch management system. The truck information acquisition device 56 may be configured as software that implements the functions of the truck information acquisition device 56 and incorporated into the information processing device 54.

[0027] Many dump truck location and dispatch management systems, such as FMS, have an adjustment function that matches the location information of the dump truck 200, acquired from the dump truck 200's GNSS receiver, to a route on a map. The truck information acquisition device 56 can acquire information about the amount of position adjustment of the dump truck 200 based on this adjustment function, along with the truck information of the dump truck 200, from the location and dispatch management system.

[0028] Furthermore, the upper rotating body 7 is equipped with a task acquisition device 58 that acquires a task indicating the next operation (or work) to be performed by the hydraulic excavator 1. The task acquisition device 58 may be, for example, a communication terminal that communicates wirelessly with a management device that manages the work site. The task acquisition device 58 can acquire a task by receiving a task from the management device for the hydraulic excavator 1. The task acquisition device 58 may be configured as a communication terminal shared with the track information acquisition device 56. The task acquisition device 58 may be configured as software that implements the functions of the task acquisition device 58 and be incorporated into the information processing device 54. Alternatively, the task acquisition device 58 may acquire a task by automatically determining the task from the operation of the operation levers 22 and 23 by the operator. The task acquisition device 58 may be an input terminal that can be operated by the operator and may acquire a task by accepting input from the operator. In this embodiment, the method of acquiring a task is not particularly limited.

[0029] <Internal structure of a hydraulic excavator> Figure 2 is a block diagram showing the configuration of the hydraulic system and control system mounted on the hydraulic excavator 1 shown in Figure 1.

[0030] The engine 103, which is the prime mover mounted on the upper slewing body 7, drives the hydraulic pump 102 and the pilot pump 104. The control device 40 controls the rotational movement of the front work device 2, the travel movement of the lower traveling body 5, and the slewing movement of the upper slewing body 7 (i.e., the movement of the body of the hydraulic excavator 1 composed of these) according to the operation information (operation amount and direction) of the operating levers 22 and 23 by the operator. Specifically, the control device 40 detects the operation information (operation amount and direction) of the operating levers 22 and 23 by the operator using sensors 52a to 52f such as rotary encoders or potentiometers, and outputs control commands to electromagnetic proportional valves 47a to 47l according to the detected operation information. The electromagnetic proportional valves 47a to 47l are located on the pilot line 100 and operate when a control command is input from the control device 40, outputting pilot pressure to the flow control valve 101 and operating the flow control valve 101.

[0031] The flow control valve 101 controls the pressure oil supplied from the hydraulic pump 102 to the swing hydraulic motor 6, arm cylinder 12, boom cylinder 11, bucket cylinder 13, travel right hydraulic motor 4a, and travel left hydraulic motor 4b, respectively, according to the pilot pressure from the solenoid proportional valves 47a to 47l. The solenoid proportional valves 47a and 47b output pilot pressure to the flow control valve 101 to control the pressure oil supplied to the swing hydraulic motor 6. The solenoid proportional valves 47c and 47d output pilot pressure to the flow control valve 101 to control the pressure oil supplied to the arm cylinder 12. The solenoid proportional valves 47e and 47f output pilot pressure to the flow control valve 101 to control the pressure oil supplied to the boom cylinder 11. The solenoid proportional valves 47g and 47h output pilot pressure to the flow control valve 101 to control the pressure oil supplied to the bucket cylinder 13. The solenoid proportional valves 47i and 47j output pilot pressure to the flow control valve 101 to control the pressurized oil supplied to the right travel hydraulic motor 4a. The solenoid proportional valves 47k and 47l output pilot pressure to the flow control valve 101 to control the pressurized oil supplied to the left travel hydraulic motor 4b.

[0032] The boom cylinder 11, arm cylinder 12, and bucket cylinder 13 extend and retract using the supplied pressurized oil, respectively, to rotate the boom 8, arm 9, and bucket 10. This changes the position and orientation of the bucket 10. The slewing hydraulic motor 6 rotates using the supplied pressurized oil to slewing the upper slewing body 7. The travel right hydraulic motor 4a and travel left hydraulic motor 4b rotate using the supplied pressurized oil to move the lower traveling body 5.

[0033] Furthermore, the control device 40 can automatically (semi-automatically or fully automatically) control the operation of the hydraulic excavator 1's body (rotation of the front work device 2, travel of the lower travel body 5, and rotation of the upper slewing body 7) based on the recognition results of the information processing device 54. The information processing device 54 recognizes dump trucks 200 present around the hydraulic excavator 1 based on the measurement results of the external environment measurement device 70. The information processing device 54 is composed of a computer in which a CPU (Central Processing Unit) 73, RAM (Random Access Memory) 72, ROM (Read Only Memory) 71, and an external I / F (Interface) 74 are connected to each other by a bus 75. The control device 40, display device 55, external environment measurement device 70, positioning device 60, attitude detection device 53, task acquisition device 58, truck information acquisition device 56, and storage device 57 (e.g., hard disk drive or large-capacity flash memory) are connected to the external I / F 74.

[0034] <Overview of Hydraulic Excavator Operation> Figure 3 illustrates an example of the operation of the hydraulic excavator 1 shown in Figure 2.

[0035] The dump truck 200 stops at a predetermined position where the hydraulic excavator 1 can load the object. The task acquisition device 58 of the hydraulic excavator 1 acquires the loading task as the next task, which is to load the excavated soil and other objects onto the dump truck 200. When the loading task is acquired, first the external measurement device 70 of the hydraulic excavator 1 measures the vessel of the dump truck 200. Next, the information processing device 54 of the hydraulic excavator 1 calculates the recognition area and estimation area described later for the presence area 210 of the dump truck 200. The information processing device 54 of the hydraulic excavator 1 calculates the position of the bucket 10 when loading the object from the bucket 10 onto the vessel of the dump truck 200 (hereinafter also referred to as the "loading position"). Then, the control device 40 of the hydraulic excavator 1 outputs a control command to move the bucket 10 to the loading position so that the front working device 2 does not collide with the dump truck 200, based on the calculated recognition area, estimation area and loading position of the dump truck 200. This allows the hydraulic excavator 1 to automatically control the loading operation properly without colliding with the dump truck 200.

[0036] In this embodiment, as shown in Figure 3, the dump truck 200 is assumed to stop at a predetermined position where the hydraulic excavator 1 can load the object. Specifically, in this embodiment, the position of the dump truck 200 at the work site is managed by a dump truck 200 position and dispatch management system such as FMS. However, the method of controlling the dump truck 200 until it stops at the predetermined position is not particularly limited. For example, the dump truck 200 may stop at the predetermined position by the operation of the dump truck 200 operator. In this embodiment, the operator of the hydraulic excavator 1 may visually confirm that the dump truck 200 has stopped at the predetermined position, or the information processing device 54 may recognize it based on the measurement results of the external environment measuring device 70.

[0037] In Figure 3, the external environment measuring device 70 is mounted facing the left side of the hydraulic excavator 1, but it may also be mounted facing the front of the hydraulic excavator 1. Furthermore, the hydraulic excavator 1 may be further equipped with an external environment measuring device 70 mounted facing the right side or rear of the hydraulic excavator 1.

[0038] Figure 4 is a flowchart showing an example of the operation of the hydraulic excavator 1 shown in Figure 2.

[0039] In step S111, the task acquisition device 58 of the hydraulic excavator 1 acquires an excavation task to excavate the ground to be excavated.

[0040] In step S112, the control device 40 of the hydraulic excavator 1 controls the rotational movement of the front work device 2 according to the acquired excavation task, causing the hydraulic excavator 1 to perform an excavation operation.

[0041] In step S113, the task acquisition device 58 of the hydraulic excavator 1 acquires a loading task to load excavated soil and other materials onto the dump truck 200.

[0042] In step S114, the control device 40 of the hydraulic excavator 1 controls the rotational movement of the front working device 2 and the rotational movement of the upper slewing body 7 according to the acquired loading task, causing the hydraulic excavator 1 to perform a loading operation.

[0043] In step S115, the control device 40 of the hydraulic excavator 1 determines whether or not to finish loading onto the dump truck 200. The operator of the hydraulic excavator 1 may, after visual confirmation, notify the control device 40 that they have finished loading onto the dump truck 200 by operating specific operating levers 22 and 23 or by inputting information into the display device 55. The control device 40 may determine that loading onto the dump truck 200 has been finished based on the notification of these operations from the operator. Alternatively, the control device 40 may determine that loading onto the dump truck 200 has been finished based on the determination result of a load determination device that measures the load of the objects loaded onto the dump truck 200 and determines whether the load is appropriate. If it is determined that loading onto the dump truck 200 has been finished, the control device 40 proceeds to step S118. If it is determined that loading onto the dump truck 200 has not been finished, the control device 40 proceeds to step S116.

[0044] In step S116, the task acquisition device 58 of the hydraulic excavator 1 acquires a reaching task to move the front work device 2 to the ground to be excavated next.

[0045] In step S117, the control device 40 of the hydraulic excavator 1 controls the rotational movement of the front work device 2, the travel movement of the lower travel body 5, and the rotational movement of the upper slewing body 7 to cause the hydraulic excavator 1 to perform a reaching operation according to the acquired reaching task. After that, the control device 40 proceeds to step S111.

[0046] In step S118, the control device 40 of the hydraulic excavator 1 determines whether or not to terminate the work. The control device 40 may determine that the work should be terminated when the task acquisition device 58 receives a work completion notification. Alternatively, the control device 40 may determine that the work should be terminated when the operator or manager stops the engine 103 of the hydraulic excavator 1. If it is determined that the work should be terminated, the control device 40 interrupts the operation of the hydraulic excavator 1 and terminates the work. If it is determined that the work should not be terminated, the control device 40 proceeds to step S116. The hydraulic excavator 1 continues the reaching, digging, and loading operations.

[0047] The tasks and operations of hydraulic excavator 1 are not limited to those shown in Figure 4, and may include tasks and operations not shown in Figure 4, such as moving or leveling.

[0048] During the operation of the hydraulic excavator 1, when the task acquisition device 58 acquires each task, information such as the position, orientation, and shape of the dump truck 200 may be required. In this case, each time the task acquisition device 58 acquires a task, the information processing device 54 recognizes the dump truck 200 based on the measurement results of the external environment measurement device 70. The information processing device 54 then verifies whether the recognition result of the dump truck 200 is not valid to the extent that it cannot be corrected, or whether the recognition result of the dump truck 200 is valid to the extent that it can be corrected (hereinafter referred to as "verifying validity"). If the recognition result of the dump truck 200 is valid, the information processing device 54 corrects the recognition result.

[0049] In this embodiment, the process of verifying the validity of the recognition result of the dump truck 200 and correcting the recognition result is also referred to as the "recognition result verification process." The functions of the information processing device 54 that performs the recognition result verification process will be described below.

[0050] <Functional Configuration of Information Processing Devices> Figure 5 is a block diagram illustrating the functional configuration of the information processing device 54 shown in Figure 2. Figure 6 is a side view of the coordinate systems 300 to 500 set on the hydraulic excavator 1. Figure 7 is a top view of the coordinate systems 300 to 500 shown in Figure 6.

[0051] The information processing device 54 includes an attitude calculation unit 81, a coordinate transformation unit 82, a positioning information calculation unit 83, a task acquisition unit 80, a recognition unit 84, a region estimation unit 85, a verification unit 86, and a correction unit 87.

[0052] The information processing device 54 is pre-set with the vehicle body coordinate system 400 shown in Figures 6 and 7 as a reference coordinate system for identifying the position and orientation of the components of the hydraulic excavator 1. In this embodiment, the vehicle body coordinate system 400 of the hydraulic excavator 1 is defined as a right-handed coordinate system with the origin being the point on the pivot centerline 120 of the upper rotating body 7 where the lower traveling body 5 and the ground G are in contact. The vehicle body coordinate system 400 of the hydraulic excavator 1 defines the forward direction of the lower traveling body 5 as the positive direction of the X axis. The vehicle body coordinate system 400 of the hydraulic excavator 1 defines the direction in which the pivot centerline 120 extends upward as the positive direction of the Z axis. The vehicle body coordinate system 400 of the hydraulic excavator 1 is orthogonal to the X axis and Z axis, respectively, with the left side defined as the positive direction of the Y axis. Furthermore, in the vehicle coordinate system 400 of the hydraulic excavator 1, the slewing angle θsw of the upper slewing body 7 is defined as 0 degrees when the front working device 2 is parallel to the X axis.

[0053] Furthermore, in this embodiment, a sensor coordinate system 300, as shown in Figures 6 and 7, is defined as the reference coordinate system for the external measurement device 70. In this embodiment, a field coordinate system 500, as shown in Figures 6 and 7, is defined as the reference coordinate system for the field.

[0054] <Posture calculation section> The attitude calculation unit 81 calculates the attitude of the components of the hydraulic excavator 1 in the vehicle coordinate system 400 of the hydraulic excavator 1 from the detection signals of the attitude detection device 53. Specifically, the attitude calculation unit 81 calculates the rotation angle θbm of the boom 8 with respect to the X axis from the detection signal of the rotation angle of the boom 8 output from the boom angle sensor 14. The attitude calculation unit 81 calculates the rotation angle θam of the arm 9 with respect to the boom 8 from the detection signal of the rotation angle of the arm 9 output from the arm angle sensor 15. The attitude calculation unit 81 calculates the rotation angle θbk of the bucket 10 with respect to the arm 9 from the detection signal of the rotation angle of the bucket 10 output from the bucket angle sensor 17. The attitude calculation unit 81 calculates the rotation angle θsw of the upper slewing body 7 with respect to the X axis (lower traveling body 5) from the detection signal of the slewing angle of the upper slewing body 7 output from the slewing angle sensor 19. Furthermore, the attitude calculation unit 81 calculates the rotational velocity ωsw of the upper rotating body 7 from the rotational angle θsw of the upper rotating body 7.

[0055] Furthermore, the attitude calculation unit 81 calculates the tilt angle of the machine body 3 (lower traveling body 5) relative to the reference plane DP from the tilt angle detection signal of the machine body 3 output from the tilt angle sensor 18. The reference plane DP is, for example, a horizontal plane perpendicular to the direction of gravity. The tilt angle includes the rotation angle θp about the Y axis and the rotation angle θr about the X axis.

[0056] <Coordinate Transformation Section> The coordinate transformation unit 82 uses the attitude information of the hydraulic excavator 1 output from the attitude calculation unit 81 to transform the coordinate system representing the depth information acquired by the external measurement device 70 from the sensor coordinate system 300 to the body coordinate system 400 of the hydraulic excavator 1. The depth information acquired by the external measurement device 70 is given as a set of three-dimensional point data (i.e., point cloud data) represented in the sensor coordinate system 300.

[0057] To convert point data Ps(Xs,Ys,Zs) in sensor coordinate system 300 to point data Pv(Xv,Yv,Zv) in the body coordinate system 400 of hydraulic excavator 1, for example, the following equations (Equations 1) to (Equations 3) can be used.

[0058]

number

number

number

[0059] In equations (1) to (3) above, Rsv is the rotation matrix from the sensor coordinate system 300 to the vehicle coordinate system 400. αs, βs, and γs are the angles formed by each axis of the external measurement device 70 (each axis of the sensor coordinate system 300) in the vehicle coordinate system 400. If the external measurement device 70 is fixed to the hydraulic excavator 1, these angles can be obtained, for example, by pre-measuring the posture of the external measurement device 70 in the vehicle coordinate system 400 and pre-saving it in the storage device 57. Alternatively, if the external measurement device 70 changes its posture relative to the hydraulic excavator 1 while performing measurements, a posture measurement sensor may be attached to the external measurement device 70, and the coordinate transformation matrix may be calculated using the angles detected by the posture measurement sensor. θsw is the rotation angle of the upper rotating body 7 and is output from the posture calculation unit 81.

[0060] In equations (1) to (3) above, Tsv is the translation vector from the origin of the vehicle coordinate system 400 to the origin of the sensor coordinate system 300. Lsx, Lsy, and Lsz are equal to the origin coordinates of the sensor coordinate system 300 as viewed from the vehicle coordinate system 400. The mounting position of the external environment measuring device 70 is often fixed to the hydraulic excavator 1. In that case, the mounting position of the external environment measuring device 70 to the hydraulic excavator 1 should be measured in advance and stored in the storage device 57 beforehand.

[0061] <Positioning Information Processing Unit> The positioning information calculation unit 83 calculates the position of the origin of the vehicle body coordinate system 400 in the site coordinate system 500 of the hydraulic excavator 1, and the direction θdir of the front work device 2 in the site coordinate system 500, from the positioning information acquired by the positioning device 60.

[0062] <Task Acquisition Section> The task acquisition unit 80 acquires the next task to be performed by the hydraulic excavator 1 from the task acquisition device 58. In this embodiment, the tasks performed by the hydraulic excavator 1 include an excavation task, a loading task, and a reaching task. The excavation task is a task that specifies the excavation operation in which the hydraulic excavator 1 excavates the ground to be excavated and holds the target material such as soil in the bucket 10. The loading task is a task that specifies the loading operation in which the hydraulic excavator 1 moves the bucket 10 from the state in which the excavation operation has been completed to above the vessel of the dump truck 200 and releases the target material from the bucket 10 into the vessel. The reaching task is a task that specifies the reaching operation in which the hydraulic excavator 1 moves the bucket 10 from the state in which the loading operation has been completed to the position of the next ground to be excavated. The tasks performed by the hydraulic excavator 1 are not limited to these, and may include, for example, a leveling task which specifies a leveling operation in which the hydraulic excavator 1 levels the surrounding ground while the dump truck 200 is parked at a predetermined position on site, or a leveling task which specifies a movement operation when the hydraulic excavator 1 changes the ground to be excavated.

[0063] <Recognition part> The recognition unit 84 recognizes the dump truck 200 from the measurement results of the external environment measurement device 70. Specifically, the recognition unit 84 recognizes the position, orientation, and shape of the dump truck 200 using point cloud data acquired as measurement results from the external environment measurement device 70 and converted to the vehicle body coordinate system 400 by the coordinate transformation unit 82. The recognition unit 84 then calculates the recognition region, which is the area in which the recognized dump truck 200 exists. In other words, the recognition region is the recognition result of the recognition unit 84 regarding the area 210 in which the dump truck 200 exists.

[0064] The method for calculating the recognition area of ​​the dump truck 200 by the recognition unit 84 is, for example, to store a 3D mesh model of the dump truck 200, which has been measured in advance, in the storage device 57. Then, the position, orientation, and shape are compared between the point cloud data converted to the vehicle coordinate system 400 obtained from the coordinate transformation unit 82 and the 3D mesh model. As a result, the recognition unit 84 can calculate the position, orientation, and shape of the target dump truck 200, and thus calculate the recognition area of ​​the dump truck 200. However, the method for calculating the recognition area of ​​the dump truck 200 is not limited to this, and for example, the recognition area of ​​the dump truck 200 may be calculated by processing to extract a specific plane of the dump truck 200 from point cloud data obtained from the external measurement device 70. Alternatively, a neural network that calculates the recognition area of ​​the dump truck 200 from point cloud data including the dump truck 200 may be constructed in advance, and the calculation may be performed using a classifier that has been trained with machine learning to obtain the features of the dump truck 200 to be recognized.

[0065] In this embodiment, the position, orientation, and presence region 210 of the dump truck 200 are defined as follows.

[0066] Figure 8 shows the body coordinate system 600 of dump truck 200 viewed from the side. Figure 9 shows the body coordinate system 600 shown in Figure 8 viewed from above.

[0067] In this embodiment, the body coordinate system 600 of the dump truck 200 has the center of the rear axle of the dump truck 200 as its origin, with the direction from the rear wheels to the front wheels as the X-axis, the direction in which the rear axle extends as the Y-axis, and the height direction of the dump truck 200 as the Z-axis. The position of the dump truck 200 is defined as the origin Pd(Xd,Yd,Zd) of the body coordinate system 600. The attitude of the dump truck 200 is defined as the angle θd(θroll,θpitch,θyaw) formed between each axis of the site coordinate system 500 and each axis of the body coordinate system 600 of the dump truck 200. The existence region 210 of the dump truck 200 is defined as the rectangular region Sd(Pd1~Pd4) with vertices at the four corner points Pd1~Pd4 of the dump truck 200's vessel. The existence region 210 of the dump truck 200 is not limited to this, and may be, for example, a hexahedron (e.g., a rectangular prism or cube) that encompasses the entire dump truck 200. In this embodiment, the recognition region of the dump truck 200, which is the recognition result of the recognition unit 84 regarding the existence region 210 of the dump truck 200, is denoted as Sdr(Pdr1~Pdr4).

[0068] <Area estimation part> The area estimation unit 85 calculates an estimated area, which is the area where the dump truck 200 is estimated to be located, based on the location and vehicle size information (truck information) of the dump truck 200 acquired by the truck information acquisition device 56. In other words, the estimated area is the estimation result of the area estimation unit 85 for the area 210 where the dump truck 200 is located. The estimated area is calculated in order to verify the validity of the recognition result of the dump truck 200 by comparing it with the recognition area of ​​the dump truck 200 calculated by the recognition unit 84.

[0069] The method for calculating the estimated area of ​​the dump truck 200 by the area estimation unit 85 involves, for example, pre-storing a method for calculating the estimated area of ​​the dump truck 200 from truck information in the storage device 57 for each vehicle type. Then, using the truck information acquired by the truck information acquisition device 56 and the calculation method corresponding to that truck information, the estimated area of ​​the target dump truck 200 in the site coordinate system 500 is calculated and converted to the vehicle body coordinate system 400 of the hydraulic excavator 1. As a result, the area estimation unit 85 can calculate the estimated area of ​​the dump truck 200 that can be compared with the recognized area of ​​the dump truck 200 calculated by the recognition unit 84. In this embodiment, the estimated area of ​​the dump truck 200, which is the estimation result of the area estimation unit 85 for the existence area 210 of the dump truck 200, is denoted as Sde(Pde1~Pde4).

[0070] The conversion from the site coordinate system 500 to the vehicle coordinate system 400 of hydraulic excavator 1 can be performed using the following formulas. For example, the following formulas (Equations 4) to (6) can be used to convert point data Pv(Xv,Yv,Zv) in the vehicle coordinate system 400 of hydraulic excavator 1 to point data Pg(Xg,Yg,Zg) in the site coordinate system 500.

[0071]

number

number

number

[0072] In equations (4) to (6) above, Rvg is the rotation matrix from the vehicle coordinate system 400 to the field coordinate system 500. θr, θp, and θy are the angles formed by each axis of the vehicle coordinate system 400 in the field coordinate system 500. These angles can be calculated using the values ​​calculated by the attitude calculation unit 81, the rotation angle, the tilt angle of the machine body 3, and the direction calculated by the positioning information calculation unit 83.

[0073] In equations (4) to (6) above, Tvg is the translation vector from the origin of the field coordinate system 500 to the origin of the vehicle coordinate system 400. x0, y0, z0 are equal to the origin coordinates of the vehicle coordinate system 400 as viewed from the field coordinate system 500. Tvg can use the calculation result of the positioning information calculation unit 83.

[0074] Furthermore, the region estimation unit 85 can adjust the position of the dump truck 200 included in the truck information using the position adjustment amount of the dump truck 200 acquired simultaneously with the truck information by the truck information acquisition device 56. Then, the region estimation unit 85 can calculate the estimated region of the dump truck 200 from the adjusted position of the dump truck 200.

[0075] <Verification Department> The verification unit 86 verifies the validity of the recognition result of the dump truck 200 based on the recognition area calculated by the recognition unit 84 and the estimated area calculated by the area estimation unit 85. Specifically, the verification unit 86 calculates the accuracy of the recognition result of the dump truck 200 by comparing the recognition area with the estimated area, and verifies the validity of the recognition result of the dump truck 200 according to the calculated accuracy. In detail, the verification unit 86 calculates the degree of overlap, which indicates the degree of overlap between the recognition area and the estimated area, as the accuracy of the recognition result of the dump truck 200. Then, if the calculated degree of overlap is higher than a threshold, the verification unit 86 determines that the recognition result of the dump truck 200 is valid (correctable). In this case, the verification unit 86 can change the threshold of the degree of overlap, which is the criterion for determining the validity of the recognition result, according to the task acquired by the task acquisition unit 80.

[0076] <Correction Section> The correction unit 87 corrects the recognition area calculated by the recognition unit 84 based on the estimated area calculated by the area estimation unit 85. Specifically, the correction unit 87 compares the recognition area calculated by the recognition unit 84 with the estimated area calculated by the area estimation unit 85 and corrects the recognition area calculated by the recognition unit 84. In this way, the correction unit 87 corrects the recognition result of the dump truck 200. At this time, the correction unit 87 can change the method of correcting the recognition area (recognition result) calculated by the recognition unit 84 according to the task acquired by the task acquisition unit 80.

[0077] The information processing device 54 uses the verification unit 86 and the correction unit 87 to verify the validity of the recognition result of the dump truck 200 and performs recognition result verification processing to correct the recognition result. The recognition result verification processing will be described in detail below.

[0078] <Recognition result verification process> Figure 10 is a flowchart illustrating an example of the recognition result verification process. Figure 11 is a diagram illustrating the degree of overlap between the recognition area and the estimation area. Figure 12 is a table that defines the criteria for determining the validity of the recognition result and the method for correcting the recognition area for each task of the hydraulic excavator 1. Figure 13 is a diagram illustrating the method for correcting the recognition area when acquiring a loading task. Figure 14 is a diagram illustrating the method for correcting the recognition area when acquiring a reaching task. Figure 15 is a diagram illustrating an example of the display on the display device 55 when the accuracy of the recognition result is below a threshold.

[0079] In step S121, the verification unit 86 of the information processing device 54 acquires the task acquired by the task acquisition unit 80.

[0080] In step S122, the verification unit 86 of the information processing device 54 acquires the recognition area calculated by the recognition unit 84.

[0081] In step S123, the verification unit 86 of the information processing device 54 acquires the estimated region calculated by the region estimation unit 85.

[0082] In step S124, the verification unit 86 of the information processing device 54 calculates the accuracy of the recognition result of the dump truck 200. As shown in Figure 11, in this embodiment, the verification unit 86 calculates the degree of overlap Acover, which indicates the degree of overlap between the recognition area calculated by the recognition unit 84 and the estimated area calculated by the area estimation unit 85, as the accuracy of the recognition result of the dump truck 200. The degree of overlap Acover is given by the following (Equation 7).

[0083]

number

[0084] In equation (7) above, Sde represents the area of ​​the estimated region calculated by the region estimation unit 85. Scover represents the area of ​​the region where the recognized region (Sdr) calculated by the recognition unit 84 and the estimated region (Sde) calculated by the region estimation unit 85 overlap. Scover can be reduced to, for example, the problem of finding the area of ​​the intersection region of two convex polygons, and can therefore be solved as a numerical computation problem.

[0085] In step S125, the verification unit 86 of the information processing device 54 determines whether the calculated degree of polymerization Acover is higher than the threshold Ath. Based on this, the verification unit 86 verifies the validity of the recognition result of the dump truck 200. If the calculated degree of polymerization Acover is higher than the threshold Ath, the verification unit 86 determines that the recognition result of the dump truck 200 is valid (correctable) and proceeds to step S126. If the calculated degree of polymerization Acover is less than or equal to the threshold Ath, the verification unit 86 determines that the recognition result of the dump truck 200 is not valid (not correctable) and proceeds to step S127.

[0086] Here, as shown in Figure 12, the verification unit 86 can change the threshold Ath of the degree of polymerization Acover, which is the criterion for determining the validity of the recognition result, according to the task acquired by the task acquisition unit 80. The table shown in Figure 12 is stored in the storage device 57 in advance. As shown in column 1201 of the table shown in Figure 12, if the task acquired by the task acquisition unit 80 is a loading task, the verification unit 86 sets Ath1 as the threshold Ath. If the task acquired by the task acquisition unit 80 is a reaching task, the verification unit 86 sets Ath2 as the threshold Ath. It is desirable that Ath1 and Ath2 be determined by taking into account the calculation accuracy of each area of ​​the area estimation unit 85 and the recognition unit 84 during the functional verification performed in advance, the presence or absence of an object in the bucket 10 at the time of task acquisition, the distance between the front work device 2 and the dump truck 200, etc.

[0087] In the loading operation specified by the loading task, the hydraulic excavator 1 starts with an object in the bucket 10 and a large distance between the front working device 2 and the dump truck 200. As a result, the damage caused when the front working device 2 collides with the dump truck 200 is considered to be greater during the loading operation than during the reaching operation. Therefore, it is desirable that the threshold Ath1 set when the loading task is acquired is the same value as or greater than the threshold Ath2 set when the reaching task is acquired.

[0088] In step S126, the correction unit 87 of the information processing device 54 corrects the recognition result of the dump truck 200. Specifically, the correction unit 87 corrects the recognition area calculated by the recognition unit 84. The correction unit 87 then outputs to the control device 40 the determination result that the recognition result of the dump truck 200 is valid (correctable), the accuracy (degree of overlap) of the recognition result, and the corrected recognition area. Furthermore, the correction unit 87 may display this information output to the control device 40 on the display device 55 to notify the operator, or transmit it to the management device that manages the work site to notify the manager. After that, the information processing device 54 terminates the recognition result verification process shown in Figure 10.

[0089] Here, as shown in Figure 12, the correction unit 87 can change the correction method of the recognition area calculated by the recognition unit 84 according to the task acquired by the task acquisition unit 80. As shown in column 1202 of the table in Figure 12, if the task acquired by the task acquisition unit 80 is a loading task, the correction unit 87 corrects the recognition area calculated by the recognition unit 84 to become an area (Sdc) that encompasses the recognition area (Sdr) and the estimation area (Sde), as shown in Figure 13. The area that encompasses the recognition area (Sdr) and the estimation area (Sde) is the area (Sdc) that encompasses the OR area occupied by the recognition area (Sdr) or the estimation area (Sde). In the example in Figure 13, the correction unit 87 corrects the recognition area (Sdr) calculated by the recognition unit 84 to a rectangular area (Sdc) that encompasses the recognition area (Sdr) and the estimation area (Sde) and has the smallest area.

[0090] In the loading operation specified by the loading task, the hydraulic excavator 1 moves the bucket 10 from the ground outside the dump truck 200 towards the top of the vessel inside the dump truck 200 to the loading position. Therefore, if the recognition area calculated by the recognition unit 84 is corrected to encompass both the recognition area and the estimated area, the risk of the front work device 2, which moves from the outside to the inside of the dump truck 200 during the loading operation, colliding with the dump truck 200 is reduced. Thus, if the task acquired by the task acquisition unit 80 is a loading task, the correction unit 87 corrects the recognition area calculated by the recognition unit 84 to encompass both the recognition area and the estimated area.

[0091] The correction unit 87 can calculate the position of the corrected recognition area in the height direction (Z-axis direction) as follows: In other words, in order to avoid a collision between the front work device 2 and the dump truck 200, the correction unit 87 can use the maximum value of the Z-axis component of each vertex coordinate of the recognition area and the estimation area as the position of the corrected recognition area in the height direction.

[0092] Furthermore, the correction unit 87 can calculate the corrected orientation vdc of the recognition region using the orientation vector vdr of the recognition region and the orientation vector vde of the estimation region, for example, from (Equation 8) to (Equation 9) below.

[0093]

number

number

[0094] The Δdr (Δxdr, Δydr, Δzdr) used in the calculation of the gain Gk in (Equation 9) above is information regarding the calculation accuracy of the recognition area in the recognition unit 84. For example, Δdr can be the measurement accuracy of the external measuring device 70 and the calculation accuracy of the recognition unit 84, which have been measured in advance. Also, if the recognition algorithm of the recognition unit 84 uses a classifier that employs machine learning, Δdr may be the recognition accuracy information output by that classifier.

[0095] The Δde(Δxde, Δyde, Δzde) used in the calculation of the gain Gk in (Equation 9) above is information regarding the calculation accuracy of the estimated region in the region estimation unit 85. As Δde, the position adjustment amount of the dump truck 200, which can be acquired simultaneously with the truck information by the truck information acquisition device 56, can be used. That is, the correction unit 87 can correct the recognition region calculated by the recognition unit 84 using the position adjustment amount of the dump truck 200 acquired by the truck information acquisition device 56. If the truck information acquisition device 56 cannot acquire the position adjustment amount, the result of a prior investigation of the error in the positioning information acquired by the positioning device 60 at the site of the hydraulic excavator 1 can be used as Δde.

[0096] Furthermore, the correction unit 87 calculates the loading position Pdc(Xdc, Ydc, Zdc) of the bucket 10. Specifically, the correction unit 87 calculates, for example, the central position Pdr0 of the Bessel in the recognition region calculated by the recognition unit 84 and the central position Pde0 of the Bessel in the estimated region calculated by the region estimation unit 85. Then, the correction unit 87 can use the central position Pdr0 of the Bessel in the recognition region and the central position Pde0 of the Bessel in the estimated region to perform the calculation, for example, from (Equation 10) below.

[0097]

number

[0098] In equation (10) above, the gain Gk is given by equation (9) above, which includes Δde. That is, the correction unit 87 can calculate the loading position of the bucket 10 based on the position adjustment amount of the dump truck 200 acquired by the truck information acquisition device 56, the recognition area and the estimation area. The correction unit 87 outputs the calculated loading position of the bucket 10 to the control device 40.

[0099] On the other hand, if the task acquired by the task acquisition unit 80 is a reaching task, the correction unit 87 corrects the recognition region (Sdr) calculated by the recognition unit 84 so that it becomes a region (Sdc) that is contained within the region where the recognition region (Sdr) and the estimation region (Sde) overlap, as shown in Figure 14. The region (Sdc) contained within the region where the recognition region (Sdr) and the estimation region (Sde) overlap is a region (Sdc) contained within the AND region occupied by both the recognition region (Sdr) and the estimation region (Sde). In the example in Figure 14, the correction unit 87 corrects the recognition region (Sdr) calculated by the recognition unit 84 to a rectangular region (Sdc) that contains the recognition region (Sdr) and the estimation region (Sde) and has the largest area.

[0100] In a reaching operation specified by a reaching task, the hydraulic excavator 1 moves the bucket 10 from a loading position above the vessel inside the dump truck 200 to the ground outside the dump truck 200. Therefore, if the recognition area calculated by the recognition unit 84 is corrected to an area encompassed by the overlapping area of ​​the recognition area and the estimated area, the risk of the front work device 2, which moves from the inside to the outside of the dump truck 200 during the reaching operation, colliding with the dump truck 200 is reduced. Thus, if the task acquired by the task acquisition unit 80 is a reaching task, the correction unit 87 corrects the recognition area calculated by the recognition unit 84 to an area encompassed by the overlapping area of ​​the recognition area and the estimated area.

[0101] Furthermore, the correction unit 87 can use the maximum value of the Z-axis component of each vertex coordinate of the recognition area and the estimation area as the position of the corrected recognition area in the height direction (Z-axis direction), similar to when a loading task is acquired. Also, the correction unit 87 can calculate the orientation of the corrected recognition area from the above equations (8) to (9), similar to when a loading task is acquired.

[0102] In step S127, the verification unit 86 of the information processing device 54 outputs to the control device 40 the determination result that the recognition result of the dump truck 200 is not valid (cannot be corrected), the accuracy (degree of overlap) of the recognition result, the recognition area, and the estimation area. Furthermore, the verification unit 86 displays this information output to the control device 40 on the display device 55 to notify the operator, or transmits it to the management device that manages the work site to notify the manager. At this time, the verification unit 86 confirms with the user, such as the operator or manager, whether or not it is OK to continue the operation of the hydraulic excavator 1.

[0103] For example, as shown in Figure 15, the verification unit 86 displays a screen 551 on the display device 55 that notifies the user that the recognition result of the dump truck 200 is invalid (cannot be corrected) and asks the user whether or not to continue the operation of the hydraulic excavator 1. Screen 551 includes a message 552 that notifies the user that the recognition result of the dump truck 200 is invalid (cannot be corrected) and asks the user whether or not to continue the operation of the hydraulic excavator 1. Screen 551 includes a button 553 for the user to input a confirmation result that the operation of the hydraulic excavator 1 will continue. Screen 551 includes a button 554 for the user to input a confirmation result that the operation of the hydraulic excavator 1 will not continue. Screen 551 includes a button 555 for the user to input that the hydraulic excavator 1 will be switched from automatic control to manual operation by the operator. Furthermore, screen 551 displays the calculation results for the recognition area (Sdr) and estimation area (Sde) of the dump truck 200, informing the user of the positional relationship between the dump truck 200 and the hydraulic excavator 1.

[0104] When the verification unit 86 receives the confirmation result of whether or not to continue from the user, it outputs the confirmation result to the control device 40. After that, the information processing device 54 terminates the recognition result verification process shown in Figure 10.

[0105] <Control of the control device> The control device 40 controls the operation of the hydraulic excavator 1's body (for example, the rotation of the front work device 2, the travel of the lower travel body 5, and the rotation of the upper slewing body 7) based on the results of verifying the validity of the dump truck 200 recognition result by the information processing device 54.

[0106] If the recognition result is valid (correctable), the control device 40 controls the operation of the hydraulic excavator 1 based on the task acquired by the task acquisition device 58 and the corrected recognition area output from the information processing device 54. Specifically, if a loading task is acquired, the control device 40 controls the loading operation of the hydraulic excavator 1 so that the front working device 2 moves along a trajectory that prevents it from colliding with the dump truck 200 until the bucket 10 reaches the loading position from the ground. If a reaching task is acquired, the control device 40 controls the reaching operation of the hydraulic excavator 1 so that the front working device 2 moves along a trajectory that prevents it from colliding with the dump truck 200 until the bucket 10 reaches the ground of the next excavation target from the loading position.

[0107] Furthermore, if the recognition result is not valid (cannot be corrected), the control device 40 controls the operation of the hydraulic excavator 1 according to the confirmation result of whether or not the operation of the hydraulic excavator 1 can be continued, when the information processing device 54 outputs the result of the confirmation.

[0108] <Effects and Effects> As described above, the hydraulic excavator 1 of Embodiment 1 is a work machine that loads objects onto a dump truck 200. The hydraulic excavator 1 is equipped with an external environment measuring device 70 that measures the surrounding environment of the hydraulic excavator 1's body. The hydraulic excavator 1 is equipped with an information processing device 54 that recognizes dump trucks 200 present around the hydraulic excavator 1's body based on the measurement results of the external environment measuring device 70. The hydraulic excavator 1 is equipped with a control device 40 that controls the operation of the hydraulic excavator 1's body based on the recognition results of the information processing device 54. The hydraulic excavator 1 is equipped with a truck information acquisition device 56 that acquires the position and vehicle size information of the dump truck 200 from the outside. The information processing device 54 corrects the recognition result of the dump truck 200 based on the position and vehicle size information of the dump truck 200 acquired by the truck information acquisition device 56. The control device 40 controls the operation of the hydraulic excavator 1's body based on the corrected recognition result of the dump truck 200.

[0109] As a result, the hydraulic excavator 1 of Embodiment 1 can control loading operations on the dump truck 200 using the accurate recognition results of the dump truck 200 in various situations. Therefore, Embodiment 1 provides a work machine that can appropriately control loading operations on transport machinery by accurately recognizing the position and orientation of the transport machinery in various situations.

[0110] Conventionally, when a hydraulic excavator 1 performs a loading operation onto a dump truck 200, it has been difficult to verify the recognition result of the dump truck 200 from the measurement results of the external environment measurement device 70. In particular, with recognition systems using machine learning, it is difficult to appropriately evaluate the uncertainty of the recognition result for sites or dump trucks 200 from which training data has not been collected. Furthermore, if the developer of the hydraulic excavator 1 and the developer of the recognition system are different companies, the developer of the hydraulic excavator 1 may not be able to disclose the details of the recognition system's algorithm. It is difficult for the developer of the hydraulic excavator 1 to conduct a sufficient risk assessment in advance to determine whether it is possible to appropriately control the loading operation without colliding with the dump truck 200 using the recognition results of the recognition system.

[0111] In contrast, the hydraulic excavator 1 of Embodiment 1 can verify the validity of the recognition result of the dump truck 200 based on information from a third party acquired by the truck information acquisition device 56, and correct the recognition result. Then, the hydraulic excavator 1 of Embodiment 1 can appropriately control loading operations, etc., based on the corrected recognition result. In other words, the hydraulic excavator 1 of Embodiment 1 can control the operation of the excavator body based on accurate recognition results such as the position and orientation of the dump truck 200. Therefore, the hydraulic excavator 1 of Embodiment 1 can reduce the risk of collision with the dump truck 200 and control loading and reaching operations, thereby improving the safety and productivity of loading operations. Thus, Embodiment 1 provides a hydraulic excavator 1 that can appropriately control loading operations onto the dump truck 200 by accurately recognizing the position and orientation of the dump truck 200 in various situations.

[0112] Furthermore, the information processing device 54 includes a recognition unit 84 that recognizes the dump truck 200 from the measurement results of the external environment measuring device 70 and calculates a recognition area which is the area in which the recognized dump truck 200 is located. The information processing device 54 also includes an area estimation unit 85 that calculates an estimated area which is the area in which the dump truck 200 is estimated to be located, based on the position and vehicle size information of the dump truck 200 acquired by the truck information acquisition device 56. The information processing device 54 also includes a correction unit 87 that corrects the recognition area based on the estimated area.

[0113] As a result, the hydraulic excavator 1 of Embodiment 1 can correct the recognition result based on a relatively simple indicator, namely the area where the dump truck 200 is located. Therefore, the hydraulic excavator 1 of Embodiment 1 can accurately and easily recognize the position and orientation of the dump truck 200. The hydraulic excavator 1 of Embodiment 1 can further improve the safety and productivity of loading operations. Thus, Embodiment 1 makes it possible to easily provide a hydraulic excavator 1 that can appropriately control loading operations onto the dump truck 200 by accurately recognizing the position and orientation of the dump truck 200 in various situations.

[0114] Furthermore, the information processing device 54 (verification unit 86) calculates the accuracy of the recognition result of the dump truck 200 by comparing the recognition area with the estimation area. The correction unit 87 corrects the recognition area according to the calculated accuracy.

[0115] As a result, the hydraulic excavator 1 of Embodiment 1 can quantitatively verify the validity of the recognition result of the dump truck 200 and correct the recognition area accurately and precisely. The hydraulic excavator 1 of Embodiment 1 can further improve the safety and productivity of loading operations. Therefore, according to Embodiment 1, it is possible to provide a hydraulic excavator 1 that can accurately and precisely recognize the position and orientation of the dump truck 200 in various situations, thereby enabling more appropriate control of the loading operation onto the dump truck 200.

[0116] Furthermore, the information processing device 54 (verification unit 86) calculates the degree of overlap, which indicates the degree of overlap between the recognition area and the estimation area, as the accuracy of the recognition result. The correction unit 87 corrects the recognition area according to the calculated degree of overlap.

[0117] As a result, the hydraulic excavator 1 of Embodiment 1 can calculate the accuracy of the recognition result based on a clear standard, and can correct the recognition area more accurately and precisely. The hydraulic excavator 1 of Embodiment 1 can further improve the safety and productivity of loading operations. Therefore, according to Embodiment 1, it is possible to provide a hydraulic excavator 1 that can more accurately and precisely recognize the position and orientation of the dump truck 200 in various situations, thereby enabling more appropriate control of the loading operation onto the dump truck 200.

[0118] Furthermore, the truck information acquisition device 56 acquires information regarding the amount of position adjustment of the dump truck 200. The region estimation unit 85 calculates the estimated region using the amount of position adjustment acquired by the truck information acquisition device 56.

[0119] As a result, the hydraulic excavator 1 of Embodiment 1 can calculate the estimation area using values ​​that are more realistic than the error in the position information of the dump truck 200 obtained through prior verification, etc. Therefore, the hydraulic excavator 1 of Embodiment 1 can correct the recognition area more accurately. The hydraulic excavator 1 of Embodiment 1 can further improve the safety and productivity of loading operations. Thus, according to Embodiment 1, it is possible to provide a hydraulic excavator 1 that can more accurately recognize the position and orientation of the dump truck 200 in various situations, thereby enabling more appropriate control of the loading operation onto the dump truck 200.

[0120] Furthermore, the information processing device 54 further includes a task acquisition unit 80 that acquires the next task to be performed by the hydraulic excavator 1. The correction unit 87 changes the correction method of the recognition area according to the acquired task.

[0121] As a result, the hydraulic excavator 1 of Embodiment 1 can be corrected to the optimal recognition area for each acquired task, so that the hydraulic excavator 1 can perform the optimal operation for each task. The hydraulic excavator 1 of Embodiment 1 can further improve the safety and productivity of loading operations. Therefore, according to Embodiment 1, it is possible to provide a hydraulic excavator 1 that can more accurately recognize the position and orientation of the dump truck 200 in various situations, thereby enabling more appropriate control of the loading operation onto the dump truck 200.

[0122] Furthermore, when a loading task is acquired in which the object is loaded onto the dump truck 200, the correction unit 87 corrects the recognition area so that it becomes an area that encompasses the recognition area and the estimation area.

[0123] As a result, in the hydraulic excavator 1 of Embodiment 1, the risk of the front working device 2, which moves from the outside to the inside of the dump truck 200 during loading operations, colliding with the dump truck 200 can be further reduced. The hydraulic excavator 1 of Embodiment 1 can further improve the safety and productivity of loading operations. Therefore, according to Embodiment 1, it is possible to provide a hydraulic excavator 1 that can more appropriately control the loading operation onto the dump truck 200 by accurately recognizing the position and orientation of the dump truck 200 in various situations.

[0124] Furthermore, the hydraulic excavator 1 is further equipped with a bucket 10 for holding the object. The truck information acquisition device 56 further acquires the position adjustment amount of the dump truck 200. When a loading task to load the object onto the dump truck 200 is acquired, the correction unit 87 calculates the position of the bucket 10 when loading the object onto the dump truck 200, based on the position adjustment amount acquired by the truck information acquisition device 56, the recognition area and the estimation area. The control device 40 controls the loading operation of the hydraulic excavator 1 based on the position of the bucket 10 calculated by the correction unit 87.

[0125] As a result, the hydraulic excavator 1 of Embodiment 1 can set the control target value of the control device 40, which is the loading position of the bucket 10, to a more reasonable value. Therefore, the hydraulic excavator 1 of Embodiment 1 can further improve the safety and productivity of loading operations. Thus, Embodiment 1 provides a hydraulic excavator 1 that can accurately recognize the position and orientation of the dump truck 200 in various situations, thereby enabling more appropriate control of the loading operation onto the dump truck 200.

[0126] Furthermore, if a reaching task is acquired after the completion of the loading task of loading the object onto the dump truck 200, the correction unit 87 corrects the recognition area so that it is included in the area where the recognition area and the estimation area overlap.

[0127] As a result, the hydraulic excavator 1 of Embodiment 1 can further reduce the risk of the front working device 2, which moves from the inside to the outside of the dump truck 200 during reaching operations, colliding with the dump truck 200. Therefore, the hydraulic excavator 1 of Embodiment 1 can further improve the safety and productivity of loading operations. Thus, Embodiment 1 provides a hydraulic excavator 1 that can more appropriately control the loading operation onto the dump truck 200 by accurately recognizing the position and orientation of the dump truck 200 in various situations.

[0128] Furthermore, if the information processing device 54 (verification unit 86) determines that the recognition result of the dump truck 200 is uncorrectable, it notifies the user of this determination and displays a screen 551 on the display device 55 to ask the user whether the operation of the hydraulic excavator 1 can continue. When the information processing device 54 (verification unit 86) receives the confirmation result from the user, it outputs the confirmation result to the control device 40. The control device 40 controls the operation of the hydraulic excavator 1 according to the confirmation result.

[0129] The hydraulic excavator 1 may determine that the recognition result of the dump truck 200 cannot be corrected due to the influence of the surrounding site environment, such as sand and dust, or sensor malfunctions. Even in such cases, the hydraulic excavator 1 of Embodiment 1 can continue to be operated under appropriate judgment by seeking instructions from the operator or manager. Therefore, Embodiment 1 provides a hydraulic excavator 1 that can appropriately control the loading operation onto the dump truck 200 even when the recognition result of the dump truck 200 cannot be corrected.

[0130] [Embodiment 2] The working machine of Embodiment 2 will be described using Figures 16 and 17. The same configuration and operation as in Embodiment 1 will not be described in the working machine of Embodiment 2.

[0131] Figure 16 is a block diagram illustrating the functional configuration of the information processing device 54 of Embodiment 2. Figure 17 is a diagram illustrating the processing of the track attitude estimation unit 88 shown in Figure 16.

[0132] The truck information acquisition device 56 of Embodiment 1 acquires truck information including the position, orientation, and vehicle size information of the dump truck 200. However, depending on the dump truck 200's position and dispatch management system, while the dump truck 200's two-dimensional position and orientation on a map may be managed, the orientation of the dump truck 200 may not be managed.

[0133] Therefore, the hydraulic excavator 1 of Embodiment 2 estimates the attitude of the dump truck 200 from the terrain information surrounding the hydraulic excavator 1. Specifically, as shown in Figure 16, the hydraulic excavator 1 of Embodiment 2 is equipped with a terrain information acquisition device 59 that acquires terrain information surrounding the hydraulic excavator 1. The information processing device 54 of Embodiment 2 is equipped with a truck attitude estimation unit 88 that estimates the attitude of the dump truck 200 based on the terrain information acquired by the terrain information acquisition device 59. The area estimation unit 85 of Embodiment 2 calculates the estimated area using the attitude of the dump truck 200 estimated by the truck attitude estimation unit 88.

[0134] The terrain information acquisition device 59 may be attached to the upper rotating body 7 of the hydraulic excavator 1. The terrain information acquisition device 59 may, for example, share sensors with the external measurement device 70 to measure the terrain around the hydraulic excavator 1, and acquire terrain information from the measurement results in the information processing device 54. Alternatively, the terrain information acquisition device 59 may have a wireless communication function and acquire terrain information by receiving terrain information transmitted from an external system that measures the terrain at the site. In this embodiment, the method of acquiring terrain information is not particularly limited.

[0135] In this embodiment, the terrain information is provided as point cloud data. However, the terrain information is not limited to point cloud data; for example, it may be grid-format data in which the area around the hydraulic excavator 1 is divided into a grid, and height information is stored for each grid cell.

[0136] The truck attitude estimation unit 88 acquires the position, orientation, and vehicle size information of the dump truck 200 included in the truck information acquired by the truck information acquisition device 56, and the terrain information acquired by the terrain information acquisition device 59. Based on this acquired information, the truck attitude estimation unit 88 extracts the terrain information within the area 210 where the dump truck 200 is located. From the extracted terrain information, the truck attitude estimation unit 88 performs a plane estimation of the terrain within the area 210 where the dump truck 200 is located. For the plane estimation, for example, least squares approximation can be used. Then, the truck attitude estimation unit 88 estimates the attitude of the dump truck 200 by calculating the inclination angle of the plane obtained by the plane estimation. Specifically, the truck attitude estimation unit 88 calculates the angles θroll and θpitch that indicate the attitude of the dump truck 200 from the normal vector nt of the plane obtained by the plane estimation and the angles formed by the Y axis and Z axis of the site coordinate system 500, respectively.

[0137] In Embodiment 2, the area estimation unit 85 calculates the estimated area of ​​the dump truck 200 based on the position, orientation, and vehicle size information of the dump truck 200 acquired by the truck information acquisition device 56, and the attitude of the dump truck 200 estimated by the truck attitude estimation unit 88. The method for calculating the estimated area is the same as in Embodiment 1.

[0138] As described above, the hydraulic excavator 1 of Embodiment 2 further includes a terrain information acquisition device 59. The information processing device 54 of Embodiment 2 includes a truck attitude estimation unit 88 that estimates the attitude of the dump truck 200 based on the terrain information acquired by the terrain information acquisition device 59. The area estimation unit 85 of Embodiment 2 calculates the estimated area using the attitude of the dump truck 200 estimated by the truck attitude estimation unit 88.

[0139] As a result, the hydraulic excavator 1 of Embodiment 2 can accurately correct its recognition area even when the dump truck 200's position and dispatch management system does not manage the dump truck 200's orientation. Therefore, Embodiment 2 provides a hydraulic excavator 1 that can accurately recognize the position and orientation of the dump truck 200 in various situations, thereby enabling appropriate control of the loading operation onto the dump truck 200.

[0140] [Embodiment 3] The hydraulic excavator 1 of Embodiment 3 will be described using Figure 18. The same configuration and operation as in Embodiment 1 will not be described in this description.

[0141] Figure 18 is a block diagram illustrating the functional configuration of the information processing device 54 of Embodiment 3.

[0142] In Embodiment 3, the information processing device 54 recognizes the dump truck 200 using only the point cloud data of the dump truck 200 and its surroundings from the point cloud data acquired by the external measurement device 70.

[0143] Specifically, the information processing device 54 of Embodiment 3 further includes a filter unit 89 that extracts point cloud data within a predetermined region, including the estimated region calculated by the region estimation unit 85, from the point cloud data acquired by the external measurement device 70. The recognition unit 84 of Embodiment 3 recognizes the dump truck 200 from the point cloud data extracted by the filter unit 89 and calculates the recognition region of the dump truck 200. The method for calculating the recognition region is the same as in Embodiment 1.

[0144] If the point cloud data acquired by the external measurement device 70 includes data other than that of the dump truck 200 (such as other work machinery or the ground), it can generally lead to a decrease in the accuracy of the recognition process and an increase in processing time. The hydraulic excavator 1 of Embodiment 3 can remove point cloud data that is a disturbance using the estimated region calculated by the region estimation unit 85. As a result, the hydraulic excavator 1 of Embodiment 3 can improve the accuracy of the recognition process and shorten the processing time. Therefore, according to Embodiment 3, it is possible to provide a hydraulic excavator 1 that can accurately and quickly recognize the position and orientation of the dump truck 200 in various situations, thereby enabling appropriate and rapid control of the loading operation onto the dump truck 200.

[0145] [others] It should be noted that the present invention is not limited to the embodiments described above, and various modifications are included. For example, the embodiments described above are described in detail to make the present invention easier to understand, and are not necessarily limited to those having all the configurations described. Furthermore, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. In addition, it is possible to add, delete, or replace parts of the configuration of each embodiment with other configurations.

[0146] Furthermore, each of the above configurations, functions, processing units, and processing means may be implemented in hardware, either partially or entirely, by designing them, for example, using integrated circuits. Alternatively, each of the above configurations and functions may be implemented in software by having the processor interpret and execute programs that realize each function. Information such as programs, tapes, and files that realize each function can be stored in memory, recording devices such as hard disks and SSDs (solid state drives), or recording media such as IC cards, SD cards, and DVDs.

[0147] Furthermore, the control lines and information lines shown are those deemed necessary for explanatory purposes, and not all control lines and information lines are necessarily shown in the actual product. In reality, it is safe to assume that almost all components are interconnected. [Explanation of symbols]

[0148] 1...Hydraulic excavator (working machine), 54...Information processing device, 55...Display device, 56...Truck information acquisition device (transportation machine information acquisition device), 59...Terrain information acquisition device, 70...External environment measurement device, 80...Task acquisition unit, 84...Recognition unit, 85...Area estimation unit, 86...Verification unit, 87...Correction unit, 88...Truck posture estimation unit, 89...Filter unit, 200...Dump truck (transportation machine), Sde...Estimated area, Sdr...Recognition area

Claims

1. A machine that loads objects onto a transport machine, An external environment measuring device for measuring the surrounding environment of the vehicle body of the aforementioned work machine, An information processing device that recognizes the transport machine located around the vehicle body based on the measurement results of the external measuring device, and calculates a recognition area which is the region in which the recognized transport machine is located. A control device that controls the operation of the vehicle body based on the recognition result of the information processing device, The system includes a transport machine information acquisition device that acquires the position and vehicle class information of the transport machine from an external source, The information processing device corrects the recognition area, which is the recognition result of the transport machine, based on the position and vehicle size information of the transport machine acquired by the transport machine information acquisition device. The control device controls the operation of the vehicle body based on the corrected recognition area of ​​the transport machine. A work machine characterized by the following features.

2. The aforementioned information processing device is A recognition unit that recognizes the transport machine from the measurement results of the external measuring device and calculates the recognition area which is the region in which the recognized transport machine is located, A region estimation unit calculates an estimated region, which is the region in which the transport machine is estimated to exist, based on the position and vehicle size information of the transport machine acquired by the transport machine information acquisition device. The system includes a correction unit that corrects the recognition area based on the estimated area. The work machine according to feature 1.

3. The information processing device calculates the accuracy of the recognition result by comparing the recognition area with the estimation area. The correction unit corrects the recognition area according to the calculated accuracy. The work machine according to feature 2.

4. The information processing device calculates the degree of overlap, which indicates the degree of overlap between the recognition area and the estimation area, as the accuracy. The correction unit corrects the recognition area according to the calculated degree of polymerization. The work machine according to feature 3.

5. The transport machine information acquisition device further acquires information regarding the position adjustment amount, which is the adjustment amount used to match the position information of the transport machine to a route on a map. The region estimation unit calculates the estimated region using the position adjustment amount obtained by the transport machine information acquisition device. The work machine according to feature 2.

6. The information processing device further comprises a task acquisition unit that acquires the next task to be performed by the work machine, The correction unit changes the correction method for the recognition area according to the acquired task. The work machine according to feature 2.

7. When a loading task is acquired in which the object is loaded onto the transport machine, the correction unit corrects the recognition area so that it becomes an area that includes the recognition area and the estimation area. The work machine according to feature 6.

8. The bucket further comprises the object being held, The transport machine information acquisition device further acquires information regarding the position adjustment amount, which is the adjustment amount used to match the position information of the transport machine to a route on a map. When a loading task is acquired to load the object onto the transport machine, the correction unit calculates the position of the bucket when loading the object onto the transport machine based on the position adjustment amount acquired by the transport machine information acquisition device, the recognition area, and the estimation area. The control device controls the loading operation of the object onto the transport machine based on the position of the bucket calculated by the correction unit. The work machine according to feature 6.

9. When a reaching task is acquired after the completion of a loading task in which the object is loaded onto the transport machine, the correction unit corrects the recognition area so that it is included in the area where the recognition area and the estimation area overlap. The work machine according to feature 6.

10. The aforementioned information processing device is If the recognition result of the transport machine is determined to be uncorrectable, the user is notified of the determination that the recognition result is uncorrectable, and a screen is displayed on the display device to confirm with the user whether the operation of the vehicle body can continue. When the user provides the result of the confirmation of whether or not to continue, the confirmation result is output to the control device. The control device controls the operation of the vehicle body according to the confirmation result. The work machine according to feature 2.

11. The machine further comprises a terrain information acquisition device that acquires terrain information around the work machine, The information processing device further comprises a posture estimation unit that estimates the posture of the transport machine based on the terrain information acquired by the terrain information acquisition device, The region estimation unit calculates the estimated region using the posture of the transport machine estimated by the posture estimation unit. The work machine according to feature 2.

12. The external environment measurement device acquires point cloud data of the surrounding environment as a result of the measurement, The information processing device further comprises a filter unit that extracts point cloud data within a predetermined region including the estimation region from the point cloud data acquired by the external measurement device, The recognition unit recognizes the transport machine from the point cloud data extracted by the filter unit and calculates the recognition area. The work machine according to feature 2.

Images