Transport vehicle recognition system and work machine

Abstract

To provide a transport vehicle recognition system which can highly accurately recognize the position and the attitude of a transport vehicle.SOLUTION: A transport vehicle recognition system includes an information processor, recognizes the position and the attitude of a transport vehicle, using point group data around a work machine, and outputs the recognition result to a control device of the work machine. The information processor acquires the point group data from a measurement device, acquires information on the attitude of the work machine from an attitude detection device of the work machine, and performs recognition processing of the transport vehicle. The information processor constructs a recognition model of the transport vehicle, on the basis of the point group data, and combination data in which the position and the attitude of the transport vehicle when the point group data is acquired are associated with each other. The information processor recognizes the position and the attitude of the transport vehicle on the basis of the point group data and the attitude of the work machine if the recognition model is not constructed, and recognizes the position and the attitude of the transport vehicle on the basis of the recognition model and the point group data if the recognition model is constructed.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to a transport vehicle recognition system that recognizes transport vehicles around a work machine such as a hydraulic excavator, and to a work machine equipped with a transport vehicle recognition system. [Background technology]

[0002] Hydraulic excavators and other work machines equipped with articulated working devices are known. These work machines are used for loading work in which excavated material, such as soil and sand, is loaded onto a transport vehicle, such as a dump truck. The loading work includes a transport operation in which the rotating body is rotated relative to the traveling body to transport the excavated material to the transport vehicle, and a discharge operation (e.g., soil dumping operation) in which the working device is operated to discharge the transported material onto the transport vehicle.

[0003] During loading operations, if the rotating body rotates while the height of the work equipment (for example, the height of the tip of the bucket) is lower than the tray (vessel) of the transport vehicle, there is a possibility that the work equipment will interfere with the transport vehicle during the transport operation. Therefore, there is a demand for a function to assist the operator in operating the work machine during loading operations, as well as technology that allows the work machine to perform loading operations automatically. In order to perform loading operations on a work machine automatically or semi-automatically, it is necessary to recognize the position and attitude (angle) of the transport vehicle relative to the work machine.

[0004] An example of a conventional technology for recognizing a transport vehicle that is the target of a work machine's loading operation is the technology described in Patent Document 1. Patent Document 1 discloses an image processing system that identifies an area including the tray (vessel) of a transport vehicle from an image that shows the tray, and identifies at least one surface of the tray from the identified area. This image processing system receives an input image and identifies the area including the tray based on a segmentation model, which is a trained model that outputs an output image in which the values ​​of multiple pixels each take a value that represents the type of object that appears in the pixel of the input image that corresponds to the input pixel, and on the captured image. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2020-126363 Summary of the Invention [Problem to be solved by the invention]

[0006] In systems that use trained models, such as the image processing system described in Patent Document 1, a measuring device is generally installed at the work site in advance, data from the work site is collected using the measuring device for a predetermined period of time, and only the correct data is extracted from the collected data to create a model. However, this work involves manual labor and is costly. Furthermore, when a new vehicle model, i.e., a transport vehicle equipped with a tray not registered in the trained model, is incorporated into the work due to a change in the work site, the accuracy of recognizing the position and orientation of the transport vehicle decreases. As a result, there is a risk of a decrease in the work efficiency of the work machine.

[0007] An object of the present invention is to provide a transport vehicle recognition system that can reduce the work required to create a recognition model and recognize the position and orientation of a transport vehicle with high accuracy. [Means for solving the problem]

[0008] A transport vehicle recognition system according to one aspect of the present invention includes an information processing device that recognizes the position and attitude of a transport vehicle around a work machine using point cloud data that is a measurement result of the surroundings of the work machine acquired by a measuring device, and outputs the results of the recognition process for the transport vehicle performed by the information processing device to a control device of the work machine. The information processing device acquires the point cloud data from the measuring device and acquires information about the attitude of the work machine from an attitude detection device of the work machine to perform recognition process for the transport vehicle, accumulates combination data that associates the point cloud data with the position and attitude of the transport vehicle at the time the point cloud data was acquired, and constructs a recognition model for the transport vehicle based on the accumulated combination data. In the recognition process for the transport vehicle, if a recognition model corresponding to the transport vehicle to be worked on by the work machine has not been constructed, recognition process is performed in a first recognition mode that recognizes the position and attitude of the transport vehicle based on the point cloud data and the attitude of the work machine, and if a recognition model corresponding to the transport vehicle to be worked on by the work machine has been constructed, recognition process is performed in a second recognition mode that recognizes the position and attitude of the transport vehicle based on the recognition model and the point cloud data. [Effects of the Invention]

[0009] According to the present invention, a recognition model is created based on the recognition results of the position and attitude of the transport vehicle and the measurement results of the measuring device, so the work required to create the recognition model can be reduced. Furthermore, according to the present invention, even if a recognition model of the transport vehicle does not exist at the start of work, it is possible to create a recognition model by collecting information about the transport vehicle while the work machine is working, and after the recognition model is created, it can be used to recognize the position and attitude of the transport vehicle with high accuracy. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a side view of a hydraulic excavator, which is an example of a work machine according to this embodiment. [Figure 2]FIG. 2 is a block diagram showing the main parts of the hydraulic system and control system of the hydraulic excavator shown in FIG. 1 together with the related configuration. [Figure 3] FIG. 3 is a plan view showing an example of work performed by the hydraulic excavator. [Figure 4] FIG. 4 is a side view showing an example of a hydraulic excavator in operation. [Figure 5] FIG. 5 is a side view showing the reference coordinate system together with the hydraulic excavator. [Figure 6] FIG. 6 is a plan view showing the reference coordinate system together with the hydraulic excavator. [Figure 7] FIG. 7 is a flowchart showing an example of the flow of processing in loading control executed by the control device. [Figure 8] FIG. 8 is a functional block diagram of the information processing device according to the first embodiment. [Figure 9] FIG. 9 is a flowchart showing an example of the flow of processing including a mode setting process and a recognition process executed by the information processing device according to the first embodiment. [Figure 10] FIG. 10 is a side view showing a vehicle coordinate system, which is a reference coordinate system based on the delivery vehicle, together with the delivery vehicle. [Figure 11] FIG. 11 is a plan view showing the vehicle coordinate system together with the transport vehicle. [Figure 12] FIG. 12 is a flowchart showing an example of the flow of the tray recognition process that is executed when the learning mode is set. [Figure 13] FIG. 13 is a flowchart showing an example of the flow of a recognition result acquisition process executed by an information processing device. [Figure 14] FIG. 14 is a flowchart showing an example of the flow of a recognition model generation process executed by an information processing device. [Figure 15] FIG. 15 is a diagram illustrating a specific example of the process of determining whether construction of a recognition model is complete. [Figure 16] FIG. 16 is a diagram showing an example of a display screen of the monitor. [Figure 17]FIG. 17 is a diagram showing the relationship between the transport vehicle positioned at the loading position Pa, the transport vehicle positioned at the loading position Pb, and the measurement range of the measuring device. [Figure 18] FIG. 18 is a diagram showing an example of the operation of the hydraulic excavator when the control device executes loading control. [Figure 19] FIG. 19 is a functional block diagram of an information processing device according to the second embodiment. [Figure 20] FIG. 20 is a flowchart showing an example of the flow of processing including a mode setting process and a recognition process executed by an information processing device according to the second embodiment. [Figure 21] FIG. 21 is a functional block diagram of a server according to the third embodiment. [Figure 22] FIG. 22 is a diagram showing an example of a measurement surface selection screen. [Figure 23] FIG. 23 is a diagram showing an example of a selection screen for a placement pattern of transport vehicles relative to a hydraulic excavator. [Figure 24] FIG. 24 is a diagram showing a measurement plane table that defines the relationship between the arrangement pattern and the selected measurement plane. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] First Embodiment -Work machinery- FIG. 1 is a side view of a hydraulic excavator 1, which is an example of a work machine according to this embodiment. In this specification, the left side in FIG. 1 is the front of the hydraulic excavator. Furthermore, although FIG. 1 exemplarily illustrates a large hydraulic excavator, the present invention is also applicable to medium-sized or smaller hydraulic excavators, or other types of work machines having working implements.

[0013] In FIG. 1, a hydraulic excavator 1 (work machine) performs excavation work at a work site, excavating an excavation target such as natural ground, and loading work of loading excavated material such as earth and sand onto a transport vehicle 200 (see FIG. 3) such as a dump truck. The loading work includes a transporting operation and a discharging operation. The transporting operation is an operation of transporting the excavated material (carried material) scooped into the bucket 10 to above the transport vehicle 200 by rotating the rotating body 7. The discharging (earth dumping) operation is an operation of discharging the carried material from the bucket 10 to the transport vehicle 200. Note that in the first embodiment, an example will be described in which a plurality of transport vehicles 200 are performing transport work at a work site, but all of them are the same model. In other words, each of the plurality of transport vehicles 200 has the same external shape.

[0014] The hydraulic excavator 1 comprises a vehicle body 3 and an articulated working device (front working machine) 2 provided on the vehicle body 3. The vehicle body 3 includes a traveling body 5 and a rotating body 7. The traveling body 5 is the lower structure of the hydraulic excavator 1 and is provided with left and right crawler-type traveling devices 5a. The traveling devices 5a are driven by traveling motors 4 (hydraulic motors). The traveling body 5 travels driven by the traveling motors 4 of the traveling devices 5a. The rotating body 7 is attached to the traveling body 5 via a rotating device (not shown) so as to be able to rotate. The rotating body 7 is driven by a rotating motor 6 (hydraulic motor) of the rotating device and rotates relative to the traveling body 5. A driver's cab 21 is provided on the rotating body 7. The rotating body 7 is also equipped with a hydraulic system and a control system for the hydraulic excavator 1, which will be described later.

[0015] The work device 2 is attached to the front of the revolving unit 7 and revolves together with the revolving unit 7 relative to the travelling unit 5. The work device 2 includes a boom 8, an arm 9, and a bucket 10. The boom 8 is rotatably connected to the front of the revolving unit 7 via a boom pin 8a (see FIG. 5 ). The boom 8 is driven in the up and down direction by the extension and retraction movement of a boom cylinder 11. The arm 9 is rotatably connected to the tip of the boom 8 via an arm pin 9a. The arm 9 is driven in the cloud direction and the dump direction by the extension and retraction movement of an arm cylinder 12. The bucket 10 is rotatably connected to the tip of the arm 9 via a bucket pin 10a. The bucket 10 is driven in the cloud direction and the dump direction by the extension and retraction movement of a bucket cylinder 13. The boom 8, arm 9, and bucket 10 are driven members (front members) driven by hydraulic cylinders (11, 12, 13).

[0016] A boom angle sensor 14 (see FIG. 5) that detects the angle of the boom 8 relative to the revolving structure 7 is attached to the boom pin 8a. An arm angle sensor 15 that detects the angle of the arm 9 relative to the boom 8 is attached to the arm pin 9a. A bucket angle sensor 17 that detects the angle of the bucket 10 relative to the arm 9 is attached to the bucket pin 10a. These angle sensors are configured, for example, by potentiometers.

[0017] The angles of the boom 8, arm 9, and bucket 10 may be obtained by detecting accelerations acting on the boom 8, arm 9, and bucket 10 using an inertial measurement unit (IMU) and converting these detected values. Alternatively, the strokes of the boom cylinder 11, arm cylinder 12, and bucket cylinder 13 may be detected by a stroke sensor, and the angles of the boom 8, arm 9, and bucket 10 may be obtained by converting these detected values.

[0018] An inclination angle sensor 18 is attached to the rotating unit 7, which detects the inclination angle of the rotating unit 7 with respect to a reference plane such as a horizontal plane. A rotation angle sensor 19 is attached to the rotation device that connects the running unit 5 and the rotating unit 7, which detects the rotation angle, which is the relative angle of the rotating unit 7 with respect to the running unit 5. An angular velocity sensor 20 is attached to the rotating unit 7, which detects the angular velocity of the rotating unit 7.

[0019] The revolving unit 7 is provided with a measuring device 70 that measures objects around the revolving unit 7 (in this embodiment, on the left side of the revolving unit 7). The measuring device 70 outputs point cloud data that represents the results of the measurement. The measuring device 70 is a distance measurement sensor that measures the distance (depth) to objects that exist around the hydraulic excavator 1. The measuring device 70 may be any device that is capable of outputting point cloud data, and for example, a LiDAR (Light Detection And Ranging) or a stereo camera may be used. One or more measuring devices 70 are attached to the hydraulic excavator 1. A suitable arrangement of the measuring device 70 will be described later.

[0020] -Hydraulic system and control system- Figure 2 is a block diagram showing the main parts of the hydraulic system and control system of the hydraulic excavator 1 shown in Figure 1 together with the related configuration. As shown in Figure 2, the hydraulic excavator 1 has a prime mover 103, a hydraulic system, and a control system. The hydraulic system has a main pump 102, a pilot pump 104, solenoid proportional valves 47a to 47l, and a flow control valve 101. The control system has a control device 40 and an information processing device 54, as well as an attitude detection device 53, a touch sensor 56, a transport amount calculation device 80, a measuring device 70, and sensors 52a to 52f of the operation devices 22 and 23.

[0021] The prime mover 103 is, for example, an engine (or may be an electric motor), and drives hydraulic pumps, namely, a main pump 102 and a pilot pump 104. The control device 40 controls the operations of the work implement 2, the revolving unit 7, and the running unit 5 in response to operation signals input from the operation devices 22, 23. In other words, the operation devices 22, 23 are instruction devices that instruct the operations of the work implement 2, the revolving unit 7, and the running unit 5.

[0022] The operating devices 22, 23 are provided inside the operator's cab 21 of the revolving body 7. The operating device 22 has a right operating lever 22a and a left operating lever 22b. The operating device 23 has a right traveling lever 23a and a left traveling lever 23b. The right traveling lever 23a is used to operate the traveling motor 4 of the right traveling device. The left traveling lever 23b is used to operate the traveling motor 4 of the left traveling device. The right operating lever 22a is used in common to operate the boom cylinder 11 and the bucket cylinder 13. The left operating lever 22b is used in common to operate the arm cylinder 12 and the swing motor 6. The operating devices 22, 23 are of an electric lever type, and their sensors 52a to 52f (e.g., rotary encoders, potentiometers) detect the amount and direction of operation of the levers by the operator and output operation signals corresponding to the amount and direction of operation.

[0023] The control device 40 calculates a command signal based on operation signals input from sensors 52a to 52f in response to the operator's operation of the operating devices 22, 23, and outputs the command signal to the electromagnetic proportional valves 47a to 47l. The electromagnetic proportional valves 47a to 47l operate in response to the command signal from the control device 40, reduce the pressure of the pressure oil supplied from the pilot pump 104 via the pilot line 100, and output a pilot pressure to the flow control valve 101.

[0024] The electromagnetic proportional valves 47a and 47b output pilot pressure related to the operation of the swing motor 6. The electromagnetic proportional valves 47c and 47d output pilot pressure related to the operation of the arm cylinder 12. The electromagnetic proportional valves 47e and 47f output pilot pressure related to the operation of the boom cylinder 11. The electromagnetic proportional valves 47g and 47h output pilot pressure related to the operation of the bucket cylinder 13. The electromagnetic proportional valves 47i and 47j output pilot pressure related to the operation of the right traveling motor 4. The electromagnetic proportional valves 47k and 47l output pilot pressure related to the operation of the left traveling motor 4.

[0025] The flow control valve 101 has directional control valves (not shown) corresponding to the swing motor 6, arm cylinder 12, boom cylinder 11, bucket cylinder 13, and left and right traveling motors 4. Each directional control valve operates by receiving pilot pressure input from the corresponding one of the electromagnetic proportional valves 47a to 47l in its pressure-receiving chamber. Each directional control valve controls the direction and flow rate of pressure oil supplied from the main pump 102 and supplies it to the corresponding hydraulic actuator. This causes the swing motor 6, arm cylinder 12, boom cylinder 11, bucket cylinder 13, and left and right traveling motors 4 to operate in response to operation of the operation devices 22, 23 by the operator. Therefore, operation of the operation devices 22, 23 changes the position and angle of the bucket 10, causes the swing unit 7 to swing, and causes the traveling unit 5 to travel.

[0026] The information processing device 54 is a computer and has the function of executing recognition processing (described below) of the transport vehicle 200 based on the point cloud data output by the measuring device 70. The information processing device 54 has a ROM (Read Only Memory) 71, a RAM (Random Access Memory) 72, a CPU (Central Processing Unit) 73, an I / F (Interface) 74, a storage device 57, etc. The storage device 57 is, for example, a hard disk drive or a large-capacity flash memory. These pieces of hardware work together to run software and realize multiple functions. The information processing device 54 may be configured as one computer or multiple computers.

[0027] The ROM 71, RAM 72, CPU 73, and I / F 74 are connected via a bus 75. The ROM 71 stores programs capable of executing various calculations. That is, the ROM 71 is a storage medium (storage device) from which the programs for implementing the functions of this embodiment can be read. The RAM 72 is a storage medium (storage device) that temporarily stores the calculation results by the CPU 73 and signals input from the I / F 74. The CPU 73 loads the programs stored in the ROM 71 into the RAM 72 and performs calculations. It performs predetermined calculations on data acquired from the I / F 74, ROM 71, and RAM 72 in accordance with the programs. The input section of the I / F 74 converts signals input from various devices (such as the measuring device 70, the attitude detection device 53, the transportation amount calculation device 80, the touch sensor 56, and the control device 40) into data that can be calculated by the CPU 73. The output section of the I / F 74 generates output signals corresponding to the calculation results by the CPU 73 and outputs the signals to various devices (such as the control device 40 and the monitor 55). The hardware configuration of the control device 40 is similar to that of the information processing device 54, and therefore a description thereof will be omitted.

[0028] The information processing device 54 is connected to the control device 40, monitor 55, touch sensor 56, measuring device 70, attitude detection device 53, and transport amount calculation device 80 via an I / F 74. The monitor 55 is a display device that displays a display image showing operation information of the hydraulic excavator 1 on a display screen based on a display control signal from the information processing device 54. The monitor 55 is, for example, a liquid crystal display monitor, and is arranged in a position inside the cab 21 that is easily visible to the operator sitting in the driver's seat. The monitor 55 is a touch panel monitor having a touch sensor 56 provided on the display screen. The touch sensor 56 is an input device that inputs an input signal to the information processing device 54 in response to an operation by the operator. The attitude detection device 53 is configured to include a boom angle sensor 14, an arm angle sensor 15, a bucket angle sensor 17, a tilt angle sensor 18, a swing angle sensor 19, and an angular velocity sensor 20. The attitude detection device 53 detects the attitude of the work device 2 (e.g., the angle of the boom 8) and the attitude of the rotating body 7 (e.g., the rotation angle of the rotating body 7 relative to the running body 5, and the inclination angle of the rotating body 7 from the horizontal plane), and outputs a signal representing the detection result to the information processing device 54.

[0029] The transport amount calculation device 80 is a device that calculates the weight of the soil and sand in the bucket 10 of the hydraulic excavator 1. The transport amount calculation device 80 includes a pressure sensor that detects the pressure of the arm cylinder 12, a pressure sensor that detects the pressure of the boom cylinder 11, a pressure sensor that detects the pressure of the bucket cylinder 13, and a weight calculation device. The weight calculation device is mounted on the revolving bed 7. The weight calculation device calculates the weight of the soil and sand (excavated material) held in the bucket 10 by excavating the soil and sand, based on the cylinder pressure detected by the multiple pressure sensors and the attitude of the work implement 2 detected by the attitude detection device 53. The weight calculation device calculates the load weight, which is the total weight of the material loaded onto the transport vehicle 200 by repeated loading operations. The weight calculation device calculates the load weight by adding up the calculated weights of the soil and sand (excavated material) held in the bucket 10 each time the bucket 10 carrying the excavated material transports it from the excavation point to the discharge point. A signal indicating the timing of the addition is input from the information processing device 54. When the weight of the transported goods reaches or exceeds a specified amount and the loading operation onto the transport vehicle 200 is completed, the calculated value of the load weight is reset (to 0). A signal indicating the timing to reset the load weight is input from the information processing device 54. Note that the information processing device 54 may have the function of the weight calculation device.

[0030] -Work example- 3 and 4 are a plan view and a side view showing an example of an operation of the hydraulic excavator 1. FIGS. 3 and 4 show a loading operation in which the hydraulic excavator 1 excavates earth and sand, transports the excavated earth and sand, and loads it onto a tray T of the transport vehicle 200. The transport vehicle 200 stops at a loading position prior to the loading operation. The loading position is a position where the hydraulic excavator 1 can load transported materials. The loading position is set, for example, within an area within the maximum swing radius from the swing center line 120 of the swing unit 7 (see FIG. 5 ) and equal to or greater than the swing radius of the rear end of the swing unit 7. While the transport vehicle 200 is parked at the loading position, the information processing device 54 of the hydraulic excavator 1 executes a recognition process to recognize the position and posture of the transport vehicle 200 around the hydraulic excavator 1, using point cloud data that is the measurement result of the surroundings of the hydraulic excavator 1 acquired by the measuring device 70. The information processing device 54 outputs the result of the recognition process for the transport vehicle 200 to the control device 40 of the hydraulic excavator 1. Based on the results of the recognition processing by the information processing device 54, the control device 40 of the hydraulic excavator 1 controls the operation of the hydraulic excavator 1 (e.g., the work device 2) so that the work device 2 etc. does not collide with the tray T, for example, during the loading operation of transported materials onto the transport vehicle 200. Note that the results of the recognition processing can be applied to interference prevention control between the transport vehicle 200 and the hydraulic excavator 1 in various operations, not just during loading operations. The results of the recognition processing can also be applied to interference avoidance control between the tray T and the work device 2 when the bucket 10 of the hydraulic excavator 1 is moved to the next excavation point after loading operations, for example.

[0031] -Measurement equipment layout- 3 and 4 are assumed, it is preferable to install the measuring device 70 so that it can acquire point cloud data of the area to the left of the revolving body 7 during excavation, for example. In other words, it is preferable for the measuring device 70 to have a measurement range to the left of the revolving body 7. However, the installation position of the measuring device 70 is not limited to this example, and for example, the measuring device 70 can be attached to a position where it can measure the front of the revolving body 7.

[0032] In the recognition process of the transport vehicle 200, the position and posture of the transport vehicle 200 are recognized based on point cloud data obtained from the measuring device 70. In this embodiment, the recognition process method differs before a recognition model, which will be described later, is generated and after the recognition model is generated. In the recognition process before the recognition model is generated, the information processing device 54 extracts two surfaces, one on the left and one on the right, of the tray T of the transport vehicle 200 based on the point cloud data obtained from the measuring device 70. Generally, the inner wall surfaces on both the left and right sides of the tray T of the transport vehicle 200 are formed as flat surfaces or smoothly curved surfaces. Therefore, the position and posture of the tray T can be recognized by extracting the two surfaces, one on the left and one on the right, of the tray T. The measuring device 70 is installed so that the inner wall surfaces of the tray T of the transport vehicle 200 can be measured.

[0033] When the transport vehicle 200 is parked at a low position relative to the hydraulic excavator 1 as in the examples of Figures 3 and 4, it is desirable to install the measuring device 70 so that it can measure the transport vehicle 200 from an oblique bird's-eye view. In this embodiment, the measuring device 70 is installed at the left end of the upper part of the cab 21, with its optical axis facing obliquely downward so as to measure the distance to an object present in the left area of ​​the revolving unit 7. θsh in Figure 3 represents the horizontal angle of the measurement range of the measuring device 70. θsv in Figure 4 represents the vertical angle of the measurement range of the measuring device 70. Lsr in Figure 3 represents the distance from the hydraulic excavator 1 (measuring device 70) to the measurement range of the measuring device 70.

[0034] -Coordinate system- FIG. 5 is a side view showing the reference coordinate system together with the hydraulic excavator 1. FIG. 6 is a plan view showing the reference coordinate system together with the hydraulic excavator 1. The ROM 71 of the information processing device 54 stores a vehicle body coordinate system 400, a sensor coordinate system 300, and a site coordinate system 500 shown in FIGS. 5 and 6. The vehicle body coordinate system 400 is a reference coordinate system that specifies the positions and angles (attitude) of the components of the hydraulic excavator 1 (for example, the bucket 10). The vehicle body coordinate system 400 is a local coordinate system of the vehicle body 3 that uses the traveling body 5 as a reference, and is set in advance.

[0035] In this embodiment, the vehicle body coordinate system 400 is defined as an XYZ right-handed Cartesian coordinate system with its origin at the intersection of the turning center line 120, which is the rotation axis of the revolving unit 7, and the contact surface (the bottom surface that contacts the ground GL) of the running unit 5. In the vehicle body coordinate system 400, the forward direction of the running unit 5 is defined as the positive direction of the X axis, the direction (upward) from the running unit 5 toward the revolving unit 7 along the turning center line 120 is defined as the positive direction of the Z axis, and the left side of the running unit 5 is defined as the positive direction of the Y axis. In the vehicle body coordinate system 400, the turning angle θsw (see FIG. 6) of the revolving unit 7 is defined as 0 degrees when the working device 2 faces the positive X axis direction and the center line of the working device 2 (the dashed dotted line in FIG. 6) is parallel to the X axis of the vehicle body coordinate system 400.

[0036] The sensor coordinate system 300 is an xyz right-handed Cartesian coordinate system based on the measuring device 70. In the sensor coordinate system 300, the forward direction of the revolving unit 7 is the positive direction of the x-axis, the left direction of the revolving unit 7 is the positive direction of the y-axis, and the direction from the running unit 5 toward the revolving unit 7 (upward) is the positive direction of the z-axis. The site coordinate system 500 is an X'Y'Z' right-handed Cartesian coordinate system based on the work site. In the site coordinate system 500, the upward vertical direction is the positive direction of the Z'-axis, any direction parallel to the horizontal direction is the positive direction of the X'-axis, and one of the directions perpendicular to the X'-axis and Z'-axis is the positive direction of the Y'-axis.

[0037] -Posture calculation- The ROM 71 of the information processing device 54 stores geometric information used to calculate the positions and angles (postures) of the components of the hydraulic excavator 1 (for example, the bucket 10). This geometric information includes, for example, the boom length Lbm, the arm length Lam, and the bucket length Lbk. The boom length Lbm is the length of the boom 8 and corresponds to the distance between the centers of the boom pin 8a and the arm pin 9a. The arm length Lam is the length of the arm 9 and corresponds to the distance between the centers of the arm pin 9a and the bucket pin 10a. The bucket length Lbk is the length of the bucket 10 and corresponds to the distance from the center of the bucket pin 10a to the tip of the bucket.

[0038] The information processing device 54 acquires, from the attitude detection device 53, information (signals) representing the angle of the boom 8, the angle of the arm 9, the angle of the bucket 10, the swing angle of the rotating unit 7 relative to the running unit 5, and the tilt angle of the rotating unit 7 from a reference plane, as information relating to the attitude of the hydraulic excavator 1. Based on the acquired detection results of the attitude detection device 53, the information processing device 54 calculates the positions and angles (attitude) of the components of the hydraulic excavator 1 in the vehicle body coordinate system 400. For example, the information processing device 54 calculates the angle θbm of the boom 8 relative to the X-axis based on the detection result of the boom angle sensor 14. The information processing device 54 calculates the angle θam of the arm 9 relative to the boom 8 based on the detection result of the arm angle sensor 15. The information processing device 54 calculates the angle θbk of the bucket 10 relative to the arm 9 based on the detection result of the bucket angle sensor 17. The information processing device 54 calculates the swing angle θsw of the rotating unit 7 relative to the running unit 5 based on the detection result of the swing angle sensor 19. The information processing device 54 calculates the tilt angle θg of the rotating body relative to the reference plane DP based on the detection result of the tilt angle sensor 18. The reference plane DP is, for example, a horizontal plane perpendicular to the direction of gravity (vertical direction). The information processing device 54 calculates the position of the tip of the bucket 10 based on the boom angle θbm, arm angle θam, bucket angle θbk, and geometric information (Lbm, Lam, Lbk). The calculation results of the information processing device 54 are output to the control device 40 and are used for control such as loading control by the control device 40.

[0039] - Coordinate conversion - The information processing device 54 uses the position and angle data of each part of the vehicle body obtained by the attitude calculation to coordinate convert the output point cloud of the measurement device 70 from values ​​in the sensor coordinate system 300 to values ​​in the vehicle body coordinate system 400. The point cloud output by the measurement device 70 is a collection of three-dimensional point data Ps (Xps, Yps, Zps) represented by the sensor coordinate system 300. The point data (Xps, Yps, Zps) in the sensor coordinate system 300 is converted into point data Pv (Xpv, Ypv, Zpv) in the vehicle body coordinate system 400 using the following (Equations 1) to (Equations 3).

[0040]

number

[0041]

number

[0042]

number

[0043] Here, Rsv is a rotation matrix that converts coordinates in the sensor coordinate system 300 into coordinates in the vehicle body coordinate system 400, and αs, βs, and γs in the rotation matrix Rsv are tilt angles of the measuring device 70 with respect to the x, y, and z axes in the vehicle body coordinate system 400. When the measuring device 70 is fixed to the hydraulic excavator 1, αs, βs, and γs can be obtained, for example, by measuring the position and angle of the measuring device 70 in the vehicle body coordinate system 400 and can be stored in advance in the ROM 71 or the storage device 57. θsw is the rotation angle of the rotating unit 7, and is obtained by attitude calculation.

[0044] Tsv is a translation vector whose starting point is the origin of the vehicle body coordinate system 400 and whose ending point is the origin of the sensor coordinate system 300. The components Lsx, Lsy, and Lsz of Tsv are equal to the origin coordinates of the sensor coordinate system 300 in the vehicle body coordinate system 400. If the measuring device 70 is fixed, the origin position of the sensor coordinate system 300 in the vehicle body coordinate system 400 is immovable, and therefore Tsv(Lsx, Lsy, Lsz) can be measured in advance and stored in the ROM 71 or the storage device 57 in advance.

[0045] -Basic operation of hydraulic excavators- Fig. 7 is a flowchart showing an example of the flow of processing in loading control executed by the control device 40. The processing shown in the flowchart of Fig. 7 is started, for example, when the ignition switch is turned on, and is repeatedly executed at a predetermined control period.

[0046] (Step S101) In step S101, the control device 40 executes a process for confirming whether the transport vehicle 200 is stopped. In the process for confirming whether the transport vehicle 200 is stopped, the control device 40 determines whether the transport vehicle 200 is stopped at a position where transported goods can be loaded (loading position). For example, when the control device 40 receives a transport vehicle stop notification signal from the vehicle allocation management system of the transport vehicle 200, the control device 40 determines that the transport vehicle 200 is stopped at the loading position. When the control device 40 does not receive a transport vehicle stop notification signal, the control device 40 determines that the transport vehicle 200 is not stopped at the loading position. In this case, a communication device (not shown) for wireless communication with a server of the vehicle allocation management system is mounted on the hydraulic excavator 1. An example of a vehicle allocation management system is an Fleet Management System (FMS) used at a work site.

[0047] The control device 40 may determine that the transport vehicle 200 is stopped at the loading position when a vehicle stop notification signal is input from a vehicle stop confirmation switch (not shown) in response to operation of the vehicle stop confirmation switch by the operator. In this case, the operator visually confirms that the transport vehicle 200 is stopped at the loading position and operates the vehicle stop confirmation switch. In step S101, when the control device 40 determines that the transport vehicle 200 is stopped at the loading position, that is, when it confirms that the transport vehicle 200 is stopped at the loading position, it notifies the information processing device 54 of this, and the process proceeds to step S102.

[0048] (Step S102) In step S102, the control device 40 executes a process for detecting the start of loading work. In the process for detecting the start of loading work, the control device 40 determines whether or not an instruction to start loading work has been issued. When loading control is performed automatically, that is, when the operator does not operate the work implement 2 and the revolving unit 7 for loading work, the control device 40 determines that an instruction to start loading work has been issued when a loading start signal is received from the dispatch management system of the transport vehicle 200. The control device 40 determines that an instruction to start loading work has not been issued while the loading start signal has not been received. Note that the control device 40 may also determine that an instruction to start loading work has been issued when a loading start signal is input from a loading start switch (not shown) in response to operation of the loading start switch by the operator.

[0049] When loading control is performed semi-automatically, i.e., when the control device 40 assists the operator in operating the work device 2 and the rotating unit 7, the control device 40 determines that a command to start loading work has been issued when, for example, the sensors 52a to 52f detect operation of the operating devices 22, 23. For example, when loading work starts, characteristic operations are performed, such as the rotating unit 7 rotating toward the transport vehicle 200 and the boom 8 starting to rise. For this reason, the start of loading work can also be detected from the operation of the operating devices 22, 23, based on data previously obtained by deep learning of the operation of the hydraulic excavator 1 associated with the start of loading work. When it is determined in step S102 that a command to start loading has been issued, i.e., when the start of loading work has been detected, the process proceeds to step S103.

[0050] (Step S103) In step S103, the control device 40 executes a process for acquiring the position of the tray T, etc. In the process for acquiring the position of the tray T, etc., the control device 40 acquires the current position and attitude (angle) of the tray T of the transport vehicle 200 onto which the transported goods are loaded, as well as the currently set recognition mode, from the information processing device 54. Specifically, the control device 40 outputs a request command for the position of the tray T, etc., to the information processing device 54. Upon acquiring the request command, the information processing device 54 executes various processes for recognizing the transport vehicle 200 (see steps S114 and S115 in FIG. 9). The information processing device 54 recognizes the position and attitude (angle) of the transport vehicle 200 through a recognition process for the transport vehicle 200. The information processing device 54 outputs the recognized position and attitude of the tray T and the recognition mode (see step S114 in FIG. 9) that was set prior to the recognition process (see step S115 in FIG. 9) to the control device 40. Details of the recognition process for the transport vehicle 200 and the recognition mode setting process will be described later.

[0051] The position and attitude (angle) of the tray T output from the information processing device 54 to the control device 40 are specified by the vehicle body coordinate system 400 of the hydraulic excavator 1. For example, coordinates of the vehicle body coordinate system 400 representing the four corners of the upper end of the tray T (points Pd1 to Pd4 in FIG. 11) are output from the information processing device 54 to the control device 40. In addition, the information processing device 54 outputs the position coordinates of the origin of the vehicle coordinate system 600 (see FIGS. 10 and 11) of the transport vehicle 200 in the vehicle body coordinate system 400 of the hydraulic excavator 1 to the control device 40 as position information of the transport vehicle 200. Furthermore, the information processing device 54 outputs to the control device 40, as posture information of the transport vehicle 200, information such as the angle (azimuth angle) formed between the X″ axis of the vehicle coordinate system 600 (see FIGS. 10 and 11) of the transport vehicle 200 and the X axis of the vehicle body coordinate system 400 of the hydraulic excavator 1. In step S103, when the control device 40 acquires the position and posture (angle) of the tray T, the processing proceeds to step S104.

[0052] (Step S104) In step S104, the control device 40 starts loading control. Through loading control, excavation work and loading work are repeatedly performed by the hydraulic excavator 1. In step S104, the control device 40 controls the operation of at least one of the rotating body 7 and the work implement 2 (boom 8, arm 9, and bucket 10) based on the position and attitude (angle) of the tray T acquired in step S103. As an example, when a swing operation signal is input, not only the swing but also the boom raising operation according to the swing speed is automatically or semi-automatically controlled even if the boom raising operation is not performed (or even if the operation amount is smaller than appropriate). Through this loading control, the work implement 2 (particularly the bucket 10) moves between the digging position and the discharge position (soil dumping position) without colliding with the tray T.

[0053] (Step S105) When loading control is started in step S104, processing in step S105 is started. In step S105, the control device 40 executes processing to determine whether the loading operation is complete. In the processing to determine whether the loading operation is complete, the control device 40 determines whether the loading operation completion condition has been met. When loading control is performed under automatic control, the control device 40 determines that the loading operation completion condition has been met when a loading completion signal has been received from the vehicle dispatch management system of the transport vehicle 200. The control device 40 determines that the loading operation completion condition has not been met while the loading completion signal has not been received. Note that the control device 40 may also determine that the loading operation completion condition has been met when a loading completion signal is input from a loading completion switch (not shown) in response to operation of the loading completion switch by an operator. For example, the operator visually checks the loading status of the transported goods on the tray T, and operates the loading completion switch when the operator wishes to end the loading operation on that tray T. In addition, the loading completion signal may be input to the control device 40 from an input device such as an operation switch provided on the operation devices 22, 23 or a touch sensor 56 provided on the monitor 55, when the operator performs a specific operation.

[0054] The control device 40 may determine that the loading operation termination condition is met when the total weight of the transported goods loaded onto the transport vehicle 200 through the loading operation (the load weight of the transport vehicle 200) reaches a specified value. In this case, the control device 40 acquires the total weight of the transported goods calculated by the transport amount calculation device 80 via the information processing device 54. The transport vehicle 200 may be equipped with a load weight measuring device (e.g., a load cell). In this case, the control device 40 may receive from the transport vehicle 200 the measurement result of the load weight measured by the measuring device, and determine that the loading operation termination condition is met when the load weight reaches a specified value.

[0055] If the control device 40 determines that the termination condition for the loading operation on the stopped transporter vehicle 200 is not met, the control device 40 returns the process to step S102 and repeats a series of processes related to the loading operation on the currently stopped transporter vehicle 200. In other words, the excavation operation and the loading operation are repeatedly performed by the hydraulic excavator 1. If it is determined in step S105 that the termination condition for the loading operation is met, the process proceeds to step S106.

[0056] (Step S106) In step S106, the control device 40 executes a work completion determination process. In the work completion determination process, the control device 40 determines whether work by the hydraulic excavator 1 has been completed, for example, based on a signal that is input (or stopped from being input) when an operator (or an administrator in the case of an unmanned vehicle) performs an operation to stop the prime mover 103 of the hydraulic excavator 1 (hereinafter also referred to as a stop operation). If it is determined that no stop operation has been performed and operation is continuing, the control device 40 returns the process to step S101 and waits until the next transport vehicle 200 stops at the loading position. If a stop operation has been performed, the control device 40 executes a predetermined termination process including a process to stop the prime mover 103, and ends the process shown in the flowchart of FIG. 7.

[0057] 7 is included in the result of the recognition process of the transport vehicle 200 executed by the information processing device 54. In other words, the control device 40 controls the operation of the revolving body 7 and the work implement 2 of the hydraulic excavator 1 based on the result of the recognition process of the transport vehicle 200 executed by the information processing device 54. The recognition process of the transport vehicle 200 will be described in detail below.

[0058] -Functions of the information processing device (transport vehicle recognition system)- FIG. 8 is a functional block diagram of the information processing device 54. The information processing device 54 constitutes a transport vehicle recognition system 110 that recognizes transport vehicles 200 around the hydraulic excavator 1 using point cloud data around the hydraulic excavator 1 acquired by the hydraulic excavator 1. The transport vehicle recognition system 110 only needs to include at least the information processing device 54, and may further include various devices connected to the information processing device 54. As shown in FIG. 8, the information processing device 54 executes programs stored in the ROM 71 to function as a mode switching unit 81, a recognition unit 82, a recognition result acquisition unit 83, a recognition result storage unit 85, a model generation unit 84, and a model storage unit 86. The functions of the mode switching unit 81, the recognition unit 82, the recognition result acquisition unit 83, and the model generation unit 84 are mainly performed by the CPU 73. The functions of the recognition result storage unit 85 and the model storage unit 86 are mainly performed by the storage device 57, the ROM 71, or the RAM 72.

[0059] The information processing device 54 acquires point cloud data from the measuring device 70 and acquires information about the attitude of the hydraulic excavator 1 from the attitude detection device 53. The information processing device 54 has functions as a first processing device 131 (a mode switching unit 81, a recognition unit 82, and a recognition result acquisition unit 83) that execute a recognition process for recognizing the relative position and attitude of the haulage vehicle 200 with respect to the hydraulic excavator 1 based on the point cloud data output by the measuring device 70 and the attitude of the revolving unit 7 detected by the attitude detection device 53, and a mode switching process for switching the mode of the recognition process. The information processing device 54 also has functions as a second processing device 132 (a recognition result storage unit 85, a model generation unit 84, and a model storage unit 86) that executes a model generation process for building a recognition model of the haulage vehicle 200 based on the accumulated combination data, which associates the point cloud data used for recognizing the haulage vehicle 200 with the position and attitude of the haulage vehicle 200 when the point cloud data was acquired.

[0060] The mode switching unit 81 switches the recognition processing mode between a learning mode (first recognition mode) and a model use mode (second recognition mode) based on information (data) stored in the model storage unit 86. The recognition unit 82 executes recognition processing according to the recognition mode. FIG. 9 is a flowchart showing an example of a processing flow including a mode setting processing and a recognition processing executed by the information processing device 54. The processing shown in the flowchart of FIG. 9 is started, for example, when the control device 40 confirms that the haulage vehicle 200 has stopped (see S101 in FIG. 7), and is repeatedly executed at a predetermined control period. In other words, the processing shown in the flowchart of FIG. 9 is executed every time an empty haulage vehicle 200 is placed at a loading position.

[0061] (Step S111) In step S111, the mode switching unit 81 determines whether or not a recognition model is stored in the model storage unit 86. If a recognition model is stored in the model storage unit 86, the mode switching unit 81 proceeds to step S112. If a recognition model is not stored in the model storage unit 86, the mode switching unit 81 proceeds to step S113.

[0062] (Step S112) In step S112, the mode switching unit 81 sets the recognition mode to the model use mode, and the process proceeds to step S114.

[0063] (Step S113) In step S113, the mode switching unit 81 sets the recognition mode to the learning mode, and the process proceeds to step S114.

[0064] (Step S114) In step S114, the mode switching unit 81 notifies the control device 40 of the set recognition mode in response to a request command from the control device 40, and the process proceeds to step S115.

[0065] (Step S115) In step S115, the recognition unit 82 executes a recognition process of the transport vehicle 200 including the tray T, i.e., a recognition process of the position and attitude (angle) of the transport vehicle 200. The recognition unit 82 recognizes the position and attitude of the transport vehicle 200 based on the point cloud data output by the measuring device 70. The recognition process of the transport vehicle 200 is performed using a recognition algorithm according to the set recognition mode.

[0066] - Recognition processing using a recognition model (model use mode) - When the model use mode is set, the recognition unit 82 performs a recognition process of the transport vehicle 200 including the tray T using a recognition model. The recognition model refers to a recognition algorithm and parameters for recognizing the position and posture of a specific vehicle type of transport vehicle 200 based on point cloud data output from the measurement device 70. The recognition model according to this embodiment includes point cloud data that is a 3D model of the external shape of the vehicle type of the transport vehicle 200 that performs work at the work site, and a vehicle coordinate system 600 of the transport vehicle 200 that is associated with the 3D model (see FIGS. 10 and 11). A method for generating the point cloud data that constitutes the recognition model will be described in detail below.

[0067] The recognition unit 82 converts the coordinates of the point cloud data output from the measuring device 70 from coordinates in the sensor coordinate system 300 to coordinates in the vehicle body coordinate system 400, using the positions and angles (postures) of each part of the hydraulic excavator 1 (such as the measuring device 70) in the vehicle body coordinate system 400. The parameters used for the coordinate conversion (the tilt angles of the measuring device 70 with respect to the x, y, and z axes in the vehicle body coordinate system 400, and the position of the measuring device 70) are stored in advance in the ROM 71 or the storage device 57.

[0068] The recognition unit 82 recognizes the position and attitude (angle) of the transport vehicle 200 in the vehicle body coordinate system 400 of the hydraulic excavator 1 by performing well-known point cloud data matching between the point cloud data acquired from the measuring device 70 and the point cloud data constituting the recognition model. As the point cloud data matching algorithm, for example, an ICP algorithm (Iterative Closest Point) or an NDT algorithm (Normal Distributions Transform) can be adopted.

[0069] In response to a request command from the control device 40, the recognition unit 82 outputs the position and orientation of the tray T in the vehicle body coordinate system 400 included in the result of the recognition processing to the control device 40. The recognition unit 82 may also display the relative position and orientation of the transport vehicle 200 with respect to the hydraulic excavator 1 on the monitor 55 (see FIG. 16).

[0070] -Processing that occurs when learning mode is enabled- When the learning mode is set, the recognition unit 82 performs recognition processing without using a recognition model (see Figure 12), the recognition result acquisition unit 83 performs recognition result acquisition processing (see Figure 13), and the model generation unit 84 performs recognition model generation processing (see Figure 14).

[0071] -Recognition processing without using a recognition model- The recognition unit 82 performs recognition processing of the tray T of the transport vehicle 200 without using a recognition model. In other words, the recognition unit 82 performs recognition processing of the tray T of the transport vehicle 200 using a general-purpose algorithm that is not dependent on a specific vehicle type. When the learning mode is set, the position and attitude (angle) of the transport vehicle 200 are specifically acquired according to the following principle.

[0072] Fig. 10 is a side view showing a vehicle coordinate system 600, which is a reference coordinate system based on the transporter vehicle 200, together with the transporter vehicle 200, and Fig. 11 is a plan view showing the vehicle coordinate system 600 together with the transporter vehicle 200. As shown in Figs. 10 and 11, the vehicle coordinate system 600 of the transporter vehicle 200 to be loaded with earth and sand is defined as an X", Y", Z" right-handed Cartesian coordinate system with its origin at the center of the vehicle width on the center axis of the rear wheels of the transporter vehicle 200. In the vehicle coordinate system 600, the direction from the origin toward the front of the vehicle is the positive direction of the X" axis, the direction from the origin toward the left along the wheel axis is the positive direction of the Y" axis, and the direction perpendicular to the X" axis and Y" axis toward the top of the vehicle from the origin is the positive direction of the Z" axis.

[0073] In this embodiment, the position and attitude (angle) of the tray T of the transport vehicle 200 is identified based on the coordinates of the vehicle body coordinate system 400 of four points Pd1, Pd2, Pd3, and Pd4 located at the four corners of the tray T of the transport vehicle 200 in a plan view. In the recognition process (S115), the points Pd1 to Pd4 are extracted from the output data (point cloud data) of the measuring device 70 converted into the vehicle body coordinate system 400 of the hydraulic excavator 1, and the coordinates of these points Pd1 to Pd4 in the vehicle body coordinate system 400 are obtained as data on the position and attitude (angle) of the tray T of the transport vehicle 200.

[0074] 12 is a flowchart showing an example of the flow of the tray recognition process executed when the learning mode is set. In the tray recognition process, two surfaces (e.g., inner wall surfaces on the left and right sides of the tray T of the transport vehicle 200) are extracted and feature points (Pd1 to Pd4) of the tray T are output as follows:

[0075] (Step S121) In the case where the learning mode is set, the process of step S121 is executed when the recognition process (S115) of the haulage vehicle 200 is started. In step S121, the recognition unit 82 acquires point cloud data cluster data of the haulage vehicle 200.

[0076] The method for extracting point cloud data cluster data of the transport vehicle 200 from the point cloud data output by the measurement device 70 is a common technique. For example, the recognition unit 82 calculates the distance between each point in the point cloud data, extracts a set (cluster) of points whose distance to adjacent points is less than a predetermined distance, and recognizes a cluster with a number of points greater than the predetermined value as a surrounding object. In this case, the recognition unit 82 removes points estimated to be ground distance measurement points from the point cloud data, and uses the remaining data to recognize the surrounding object. For example, the recognition unit 82 extracts a plane corresponding to the ground from the point cloud data, and removes point cloud data included in the extracted plane, estimating that the ground distance measurement points are ground distance measurement points. The recognition unit 82 then calculates the distance between each remaining point and its adjacent point (point-to-point distance) and extracts a set of points whose point-to-point distance is less than a predetermined value as a point cloud data cluster representing the same object. For example, the Euclidean Cluster Extraction algorithm can be applied to extract point cloud data clusters to be detected as objects. Furthermore, for example, the RANSAC (RANdom SAmple Consensus) algorithm can be applied to extract the ground surface.

[0077] The recognition unit 82 calculates the smallest cube that encloses the detected cluster of point cloud data, and if one side of the cube is within a predetermined range (for example, 5 m or more and 10 m or less), determines that the point cloud data cluster enclosed by the cube is data representing the delivery vehicle 200. Note that the determination of the delivery vehicle 200 is not limited to a determination based on the size of the point cloud data cluster, but may also be determined based on whether or not a characteristic geometric shape of the delivery vehicle 200 (for example, the circular shape of the tires) is extracted.

[0078] (Step S122) In the next step S122, the recognition unit 82 calculates the normal vector of each point in the point cloud data cluster. The normal vector of each point can be obtained by a general method, for example, by fitting a plane to the local point cloud data using the least squares method, and then determining a vector perpendicular to the plane as the normal vector.

[0079] (Step S123) In the next step S123, the recognition unit 82 extracts point cloud data clusters whose distances to adjacent points are equal to or less than a predetermined value and whose normal vectors are equal to or less than a predetermined value (grouping process). For example, a Region Growing algorithm can be applied to the grouping process using normals.

[0080] (Step S124) In the next step S124, the recognition unit 82 calculates an approximate plane for each of the grouped point cloud data clusters and calculates a normal vector of the approximate plane for each point cloud data cluster. For example, the RANSAC algorithm can be applied to calculate the approximate plane.

[0081] (Step S125) In the next step S125, the recognition unit 82 removes unnecessary point cloud data clusters, specifically, point cloud data clusters whose normal vectors calculated in step S124 are parallel to the gravity direction vector. For example, the recognition unit 82 calculates the dot product of the normal vector and the Z-axis direction vector of the vehicle body coordinate system 400, and determines whether the normal vector is parallel to the gravity direction vector based on whether the calculated dot product is equal to or greater than a preset threshold. If the pitch angle θpitch (see FIG. 10), roll angle θroll (see FIG. 10), and yaw angle θyaw (see FIG. 11) of the transporter vehicle 200 can be obtained, the dot product may be calculated taking these pitch angles and roll angles into account.

[0082] (Step S126) In the next step S126, the recognition unit 82 determines whether or not there are any point cloud data clusters that form a pair (e.g., parallel approximate planes) as the left and right inner wall surfaces of the tray T, from the point cloud data clusters that form the approximate planes calculated in step S124. For example, for each point cloud data cluster extracted in step S124, the combination in which the absolute value of the dot product of the normal vectors of each approximate plane is equal to or greater than a preset threshold and the dot product is maximum is calculated as the two left and right surfaces of the tray T.

[0083] If it is determined in step S126 that two surfaces forming a pair have not been extracted (for example, if there is no pair of approximate planes whose absolute value of the dot product is equal to or greater than the threshold), the process proceeds to step S127. If it is determined in step S126 that two surfaces forming a pair have been extracted (for example, if there is a pair of approximate planes whose absolute value of the dot product is equal to or greater than the threshold), the process proceeds to step S128.

[0084] (Step S127) In step S127, a signal indicating that the recognition of the tray T has failed is output to the monitor 55 and the control device 40. The monitor 55 displays an image indicating that the recognition of the tray T has failed on the display screen. In this case, the control device 40 does not shift the process from the process of acquiring the position, etc. of the tray T (step S103 in FIG. 7) to the loading control (step S104 in FIG. 7), but returns the process to the process of detecting the start of the loading work (step S102 in FIG. 7).

[0085] (Step S128) In step S128, the recognition unit 82 calculates the coordinates of the end points of the four corners of the tray T (points Pd1 to Pd4 in FIGS. 10 and 11) in the vehicle body coordinate system 400. The recognition unit 82, for example, extracts two point cloud data (point cloud data arranged in a line) that form the upper end surface of each point cloud data cluster that constitutes the point cloud data, and extracts both ends (start point and end point) of these two point cloud data as points Pd1 to Pd4. Note that, since point cloud data that represents the shape of the body of the transport vehicle 200 to which the tray T is attached has also been acquired, the recognition unit 82 can recognize the front and rear of the tray T (the front and rear of the transport vehicle 200). In addition, the recognition unit 82 can calculate the attitude of the transport vehicle 200 (the angle between the X-axis of the vehicle coordinate system 600 and the X-axis of the vehicle body coordinate system 400) and the like from the end points of the four corners of the tray T. In this way, the recognition unit 82 recognizes the position and attitude of the transport vehicle 200 and the shape of the tray T.

[0086] (Step S129) In the next step S129, the recognition unit 82 outputs the recognition result of the transport vehicle 200, including the position, posture, and shape of the tray T, to the control device 40. As a result, the control device 40 shifts the processing from the process of acquiring the position, etc. of the tray T (step S103 in FIG. 7) to loading control (step S104 in FIG. 7). In step S129, the recognition unit 82 outputs a signal indicating that the recognition of the tray T has been successful to the monitor 55. The monitor 55 displays an image indicating that the recognition of the tray T has been successful on the display screen. The recognition unit 82 may also display the relative position and posture of the tray T of the transport vehicle 200 with respect to the hydraulic excavator 1 on the monitor 55 (see FIG. 16).

[0087] When the process of either step S127 or S129 is completed, the process shown in the flowchart of FIG. 12 ends.

[0088] - Acquisition of recognition results - The recognition result acquisition unit 83 acquires the recognition result of the transporter vehicle 200 by the recognition unit 82. Fig. 13 is a flowchart showing an example of the flow of a recognition result acquisition process executed by the information processing device 54. The process shown in the flowchart in Fig. 13 is started, for example, when a learning mode is set (see step S113 in Fig. 9), and is repeatedly executed at a predetermined control period.

[0089] (Step S131) In step S131, the recognition result acquisition unit 83 determines whether or not the recognition of the tray T of the transporter vehicle 200 has been successful. If a signal indicating that the recognition of the tray T has been successful is output from the recognition unit 82 (see step S129 in FIG. 12), the recognition result acquisition unit 83 determines that the recognition of the tray T has been successful, and proceeds to step S132. If a signal indicating that the recognition of the tray T has been successful is not output from the recognition unit 82, the recognition result acquisition unit 83 determines that the recognition of the tray T has not been successful. The recognition result acquisition unit 83 repeatedly executes the processing of step S131 until an affirmative determination is made.

[0090] (Step S132) In step S132, the recognition result acquisition unit 83 acquires loading information of the transport vehicle 200. For example, the recognition result acquisition unit 83 acquires the loaded weight (total weight of transported goods) calculated by the transported amount calculation device 80 as the loading information of the transport vehicle 200. As described above, the transported amount calculation device 80 calculates the loaded weight (total weight of transported goods) by accumulating the weight of the transported goods held in the bucket 10 each time the bucket 10 moves from the excavation position to the discharge position. The loaded weight is reset to 0 (zero), for example, when the loading operation into the transport vehicle 200 is completed (Yes in step S105 of FIG. 7). When it is confirmed that the transport vehicle 200 has stopped at the loading position, there are no transported goods on the tray T. For this reason, the loaded weight as the loading information acquired in step S132 is usually 0 (zero).

[0091] The recognition result acquisition unit 83 may acquire the total weight (load weight) of the transported goods as the loading information from the transport vehicle 200 by wireless communication. In this case, the transport vehicle 200 is provided with a load weight measuring instrument and a communication device that wirelessly transmits the load weight.

[0092] In the next step S133, the recognition result acquisition unit 83 determines whether the transport vehicle 200 is in an empty state or a loaded state based on the loading information acquired in step S132. For example, if the loaded weight is equal to or less than a weight threshold, the recognition result acquisition unit 83 determines that the transport vehicle 200 is in an empty state, and if the loaded weight is greater than the weight threshold, the recognition result acquisition unit 83 determines that the transport vehicle 200 is in a loaded state. The weight threshold is a threshold (e.g., 0 kg) for determining whether the transport vehicle 200 is in an empty state, and is stored in advance in the ROM 71, etc. If it is determined that the transport vehicle 200 is in an empty state, the process proceeds to step S134. If it is determined that the transport vehicle 200 is in a loaded state, the process shown in the flowchart of FIG. 13 is terminated without executing the recognition result storage process (S134).

[0093] In step S134, the recognition result acquisition unit 83 acquires the recognition result by the recognition unit 82 and stores it in the recognition result storage unit 85. The recognition result acquisition unit 83 generates combination data that associates the vehicle type of the haulage vehicle 200, the recognition result of the haulage vehicle 200, and the point cloud data used to obtain the recognition result, and stores the combination data in the recognition result storage unit 85. The point cloud data stored in the recognition result storage unit 85 is coordinate data in the vehicle body coordinate system 400 of a point cloud that represents the shape of the haulage vehicle 200 in an unloaded state. The point cloud data includes not only the shape of the tray T but also all of the point cloud data that represents the haulage vehicle 200 (i.e., the point cloud cluster of the haulage vehicle 200). The recognition result of the haulage vehicle 200 stored in the recognition result storage unit 85 is coordinate data that represents the position and attitude (orientation) in the vehicle body coordinate system 400 of the hydraulic excavator 1 at the time the point cloud data was acquired.

[0094] The vehicle type of the haulage vehicle 200 may be acquired from the vehicle dispatching system or by an input operation by the operator. In this embodiment, since only one type of haulage vehicle 200 is operated at the work site, it is not necessary to acquire vehicle type information. In this case, the recognition result storage unit 85 stores combination data that associates point cloud data of the shape of the unladen haulage vehicle 200 parked at the loading position with the position and posture of the haulage vehicle 200 when the point cloud data was acquired.

[0095] -Generating recognition models- As loading work onto the unladen transport vehicle 200 is repeatedly performed, the recognition result storage unit 85 accumulates combination data (hereinafter also simply referred to as combination data) of the vehicle type, recognition results, and point cloud data. Based on the combination data accumulated in the recognition result storage unit 85, the model generation unit 84 generates, as a recognition model, a 3D model that represents the external shape of the vehicle type corresponding to the transport vehicle 200 in operation. Note that the accumulated combination data does not include point cloud data of the shape of the loaded transport vehicle 200. In other words, the recognition model is generated based only on the point cloud data of the shape of the unladen transport vehicle 200. As a result, a highly accurate recognition model is created.

[0096] Fig. 14 is a flowchart showing an example of the flow of recognition model generation processing executed by the information processing device 54. The processing shown in the flowchart in Fig. 14 is executed, for example, when the recognition result acquisition unit 83 performs processing to save combined data of the recognition result and point cloud data in the recognition result storage unit 85 (see step S134 in Fig. 13), and is repeatedly executed at a predetermined control period. Note that the recognition model generation processing is not limited to being executed every time the recognition result storage unit 85 is updated. For example, the recognition model generation processing may be executed when the recognition result storage unit 85 has been updated a predetermined number of times. The recognition model generation processing may be executed after the hydraulic excavator 1 has operated for a certain period of time and the combined data has been accumulated.

[0097] (Step S141) In step S141, the model generation unit 84 acquires, from the recognition result storage unit 85, combination data of the recognition result and the point cloud data.

[0098] (Step S142) In the next step S142, the model generation unit 84 creates a recognition model specific to the vehicle type using the combination data acquired in step S141. The model generation unit 84 converts the coordinates of the point cloud data of the shape of the haulage vehicle 200 into coordinates in the vehicle coordinate system 600 of the haulage vehicle 200 based on the position and attitude of the haulage vehicle 200 in the vehicle body coordinate system 400 of the hydraulic excavator 1. The model generation unit 84 creates a 3D model representing the external shape of the haulage vehicle 200 by overlaying multiple point cloud data acquired in time series. The point cloud data of the 3D model is stored as coordinates in the vehicle coordinate system 600 of the haulage vehicle 200.

[0099] (Step S143) In the next step S143, the model generation unit 84 executes a process for determining whether construction of a recognition model is complete. The model generation unit 84 determines whether construction of a recognition model is complete based on the density of point cloud data (hereinafter also referred to as point density) of the shape of the transport vehicle 200 within a predetermined designated area of ​​the 3D model created in step S142.

[0100] A specific example of the process for determining completion of construction of a recognition model will be described with reference to FIG. 15. As shown in FIG. 15, the model generation unit 84 calculates the smallest rectangular parallelepiped that encloses the created 3D model and divides each side into N parts. This division process is also called grid processing. The smallest rectangular parallelepiped has, for example, four sides (first sides) extending in the front-to-rear direction (X"-axis direction) of the transporter vehicle 200, four sides (second sides) extending in the left-to-right direction (Y"-axis direction) of the transporter vehicle 200, and four sides (third sides) extending in the up-down direction (Z"-axis direction) of the transporter vehicle 200.

[0101] The model generation unit 84 calculates the point density of each grid that constitutes the outer surface of the rectangular parallelepiped. For example, the model generation unit 84 performs grid processing to partition the top surface of the rectangular parallelepiped into a lattice pattern. As a result, the top surface is divided into an N x N grid G(m, n). The grid width Gx in the X"-axis direction is the value obtained by dividing the length Ld of the first side by the number of divisions N (Gx = Ld / N). The grid width Gy in the Y"-axis direction is the value obtained by dividing the length Wd of the second side by the number of divisions N (Gy = Wd / N). m is an integer (m = 1 to N) that specifies the position of the grid in the X"-axis direction. n is an integer (n = 1 to N) that specifies the position of the grid in the Y"-axis direction.

[0102] The model generation unit 84 sets the grid group at the top of the rectangular parallelepiped, i.e., the grid group having the surface (top surface) projected from above the rectangular parallelepiped, as the designated region. The model generation unit 84 calculates the density ρ(m,n) of point cloud data present in each grid G(m,n) constituting the designated region. The model generation unit 84 calculates the number of grids G(m,n) (hereinafter also referred to as the number of high-density grids) Ng where the density ρ(m,n) of point cloud data exceeds a density threshold ρth. The model generation unit 84 calculates the ratio α of the number of high-density grids to the total number of grids in the designated region by dividing the number of high-density grids Ng by the total number of grids (N×N) in the designated region (α=Ng / (N×N)).

[0103] The model generation unit 84 determines whether the proportion α of high-density grids is equal to or greater than a proportion threshold αth. If the model generation unit 84 determines that the proportion α of high-density grids is less than the proportion threshold αth, it determines that construction of the recognition model is not complete. If the model generation unit 84 determines that the proportion α of high-density grids is equal to or greater than the proportion threshold αth, it determines that construction of the recognition model is complete.

[0104] The density threshold ρth and the proportion threshold αth are stored in the storage device 57 or the ROM 71. It is desirable that the density threshold ρth is sufficiently larger than the density of the point cloud data obtained by the measurement device 70 in one detection of the transport vehicle 200 within its measurement range. Note that grids with an extremely low point density compared to other grids may be excluded from the specified region. In this case, the proportion threshold αth is preferably as close to 100% as possible, and is desirably at least 80% or more.

[0105] The model construction completion determination process is not limited to the above-described method. For example, similar processing may be performed not only on the surface of the rectangular parallelepiped viewed from above (top surface), but also on the surfaces of the rectangular parallelepiped viewed from the left and right (left and right sides) and the surfaces of the rectangular parallelepiped viewed from the front and rear (front and rear). Here, it is difficult for the measurement device 70 to measure the bottom end of the transport vehicle 200. For this reason, it is preferable to exclude the surfaces of the rectangular parallelepiped viewed from below (bottom surface) from the surfaces constituting the designated area used to determine the completion of model construction (hereinafter also referred to as measurement surfaces). When the top, left, and right sides, as well as the front and rear surfaces of the rectangular parallelepiped are set as measurement surfaces, it is determined that construction of the recognition model is complete when the proportion α of high-density grids is equal to or greater than the proportion threshold αth on all of the multiple measurement surfaces. In other words, it is determined that construction of the recognition model is not complete when the proportion α of high-density grids on at least one of the multiple measurement surfaces is less than the proportion threshold αth. It is also possible to set the proportion threshold αth to a different value for each measurement surface, or to set the same value for each measurement surface. The model generation unit 84 may determine that model construction is complete when point cloud data is measured at a point density equal to or greater than a predetermined density around the entire periphery of the transport vehicle 200 (front, left side, back, and right side).

[0106] The transport vehicle 200 is usually parked with its back facing the hydraulic excavator 1. For this reason, the surface (front) of the rectangular parallelepiped seen from the front does not need to be set as the measurement surface used to determine whether model construction is complete.

[0107] Depending on the work site, there may be little change in the relative position and relative angle between the hydraulic excavator 1 and the transport vehicle 200. In this case, the model generation unit 84 may calculate the point density only in the direction that includes the surface of the transport vehicle 200 that can be measured from the hydraulic excavator 1, and then determine whether model construction is complete. For example, if work is performed only in the arrangement pattern shown in Fig. 3, the model construction determination is performed using the top surface, right side surface, and back surface of the rectangular parallelepiped as the measurement surfaces. By limiting the measurement surfaces and performing the model construction determination, the time required to construct a recognition model can be shortened.

[0108] In the above-described method, the model generation unit 84 surrounds the transporter vehicle 200 with a rectangular parallelepiped. However, the 3D model of the transporter vehicle 200 may be surrounded by a hemisphere, a sphere, a cylinder, or the like. In the above-described method, the model generation unit 84 determines whether model construction is complete based on the density of point cloud data. However, the model generation unit 84 may determine whether model construction is complete by comparing the number of samples (number of combination data) of the recognition results stored in the recognition result storage unit 85 with a threshold value. If the number of samples is less than the threshold value, the model generation unit 84 determines that construction of the recognition model is not complete. If the number of samples is equal to or greater than the threshold value, the model generation unit 84 determines that construction of the recognition model is complete. Alternatively, the model generation unit 84 may determine whether model construction is complete by comparing the total number of point cloud data (total number of point cloud data) stored in the recognition result storage unit 85 with a threshold value. If the total number of point cloud data is less than the threshold value, the model generation unit 84 determines that construction of the recognition model is not complete. If the total number of point cloud data is equal to or greater than the threshold, the model generation unit 84 determines that construction of the recognition model is complete.

[0109] The model generation unit 84 may combine the determination using the density of the point cloud data, the total number of point cloud data, and the number of combination data. For example, the model generation unit 84 may determine that a recognition model has been constructed when the proportion of high-density grids in the specified region is equal to or greater than a threshold, the number of samples of the recognition result is equal to or greater than a threshold, and the total number of point cloud data is equal to or greater than a threshold.

[0110] 14, if it is determined that the construction of the recognition model is complete, the process proceeds to step S144. If it is determined that the construction of the recognition model is not complete, the process shown in the flowchart of FIG. 14 ends without executing the storage process of the recognition model (step S144) described below.

[0111] (Step S144) In step S144, the model generation unit 84 stores the 3D model created in step S142 in the model storage unit 86 as a vehicle-specific recognition model. The recognition model is stored in association with the vehicle type (label information). The point cloud data of the recognition model is stored as coordinates in the vehicle coordinate system 600 of the haulage vehicle 200. In other words, the recognition model is composed of the point cloud data constituting the 3D model and the vehicle coordinate system 600 of the haulage vehicle 200 associated with the 3D model. When the storage process of the recognition model (step S144) is completed, the process shown in the flowchart of FIG. 14 ends.

[0112] In this way, the information processing device 54 determines whether or not construction of the recognition model is complete based on at least one of the number of point cloud data (total number of point cloud data), the number of combination data (number of samples), and the density of point cloud data of the shape of the transport vehicle 200 within a pre-specified designated area (for example, all grids constituting the top level of a rectangular parallelepiped), and if it determines that construction of the recognition model is complete, it retains the constructed recognition model.

[0113] FIG. 16 is a diagram showing an example of the display screen of the monitor 55. As shown in FIG. 16, the recognition unit 82 controls the monitor 55 to display an image showing the positional relationship between the hydraulic excavator 1 and the transport vehicle 200 on the display screen. The recognition unit 82 also displays the currently set recognition mode on the display screen of the monitor 55. The monitor 55 also displays the progress rate until the construction of the recognition model is completed. For example, the progress rate may be a value obtained by dividing the proportion α of the number of high-density grids by a proportion threshold αth. The proportion α of the number of high-density grids and the proportion threshold αth may be displayed on the monitor 55 as information representing the progress rate. Displaying the recognition mode and progress rate on the monitor 55 makes it possible to inform the operator of the construction status of the recognition model and the current recognition mode. The information processing device 54 may transmit the recognition mode and progress rate to a server of the vehicle dispatch management system and notify the administrator via a display device connected to the server.

[0114] -Specific examples of operations and processing- Specific examples of the operation and processing of the hydraulic excavator 1 according to this embodiment will be described with reference to Figures 17 and 18. Figure 17 is a diagram showing the relationship between the transport vehicle 200 positioned at loading position Pa, the transport vehicle 200 positioned at loading position Pb, and the measurement range of the measuring device 70.

[0115] When the hydraulic excavator 1 performs work at a work site for the first time, a recognition model corresponding to the transport vehicle 200 that is the target of loading work by the hydraulic excavator 1 has not been constructed. In other words, no recognition model is stored in the model storage unit 86. In this case, the information processing device 54 sets the recognition processing mode to a learning mode (first recognition mode) that does not use a recognition model (No in step S111, S113 in FIG. 9). As shown in FIG. 17, when the transport vehicle 200 moves to the loading position Pa, the information processing device 54 executes recognition processing of the transport vehicle 200 (step S115 in FIG. 9 and FIG. 12), and outputs data on the position and posture of the tray T to the control device 40 (see step S129 in FIG. 12). The control device 40 executes loading control based on the acquired data on the position and posture of the tray T (steps S103 and S104 in FIG. 7).

[0116] When the loaded weight of the transport vehicle 200 exceeds a specified amount, the excavation and loading operations are interrupted (Yes in step S105, No in S106 in FIG. 7). When the loaded transport vehicle 200 leaves the loading position Pa and another unladen transport vehicle 200 moves to the loading position Pa, the information processing device 54 and the control device 40 execute similar processing. At the work site, multiple transport vehicles 200 are performing transport operations. The multiple transport vehicles 200 are of the same model, and have the same shapes of their bodies and trays T. By repeatedly executing the recognition processing of the transport vehicle 200, combined data in which the point cloud data of the unladen transport vehicle 200 and the position and posture of the transport vehicle 200 are associated with each other is accumulated in the information processing device 54 (FIG. 13).

[0117] The information processing device 54 creates a 3D model based on the accumulated combination data (steps S141 and S142 in FIG. 14). The point cloud data constituting the 3D model is converted into coordinates of the vehicle coordinate system 600 of the transport vehicle 200 based on the position and posture of the transport vehicle 200. When sufficient point cloud data has been accumulated, the 3D model is stored in the model storage unit 86 as a recognition model (Yes in step S143 and S144 in FIG. 14). The recognition model includes not only the shape of the tray T but also various shape features of the transport vehicle 200 such as the shapes of the tires, the cab 21, etc.

[0118] In this way, when a recognition model corresponding to the transport vehicle 200 that is the target of loading work by the hydraulic excavator 1 is constructed, the recognition model is stored in the model storage unit 86. In this case, the information processing device 54 switches the recognition processing mode from a learning mode (first recognition mode) to a model use mode (second recognition mode) that uses the recognition model (Yes in step S111, S112 in FIG. 9). The information processing device 54 collects shape data of the transport vehicle 200 in operation at the site where the hydraulic excavator 1 is operated, creates a recognition model specific to the vehicle type, and switches the recognition mode from the learning mode to the model use mode. As a result, the position and posture of the transport vehicle 200 are recognized with high accuracy.

[0119] Furthermore, in the recognition process in the learning mode, in order to properly recognize the overall shape of the tray T, it was necessary to position the transport vehicle 200 so that the entire tray T was within the measurement range 91 of the measuring device 70. For this reason, as shown in FIG. 17 , the loading position Pa of the transport vehicle 200 is set near the center of the measurement range 91 of the measuring device 70.

[0120] In contrast, in the recognition processing in the model use mode, the transport vehicle 200 can be recognized from part of the overall shape characteristics of the transport vehicle 200. The information processing device 54 can recognize the position and posture of the transport vehicle 200 from, for example, part of the tray T, the tires, the shape of the cab 21, etc. Therefore, the information processing device 54 can recognize the position and posture of the tray T of the transport vehicle 200 as long as part of the transport vehicle 200 is within the measurement range 91 of the measuring device 70. Therefore, after the recognition model is created, even if the transport vehicle 200 is placed at a loading position Pb where the entire tray T does not fit within the measurement range of the measuring device 70, it is possible to properly recognize the overall shape of the tray T and perform loading work. In other words, the degree of freedom in setting the loading position is improved, and work efficiency can be improved.

[0121] Figure 18 is a diagram showing an example of the operation of the hydraulic excavator 1 when the control device 40 executes loading control. (a) is a diagram showing the transport vehicle 200 as viewed from behind, and schematically shows the movement trajectory of the bucket 10. (b) is a diagram showing the transport vehicle 200 as viewed from above. Figure 18 schematically shows the operation of the hydraulic excavator 1 as it excavates the natural ground 90, moves the bucket 10 holding the excavated material to the center of the tray T, and discharges (discharges) the material into the tray T.

[0122] The recognition accuracy of the haulage vehicle 200 when the model use mode is set is higher than the recognition accuracy of the haulage vehicle 200 when the learning mode is set. Therefore, as shown in FIG. 18, for example, in loading control, the height direction margin h2 when the model use mode is set can be made smaller than the height direction margin h1 when the learning mode is set (h2

[0123] After the recognition model is created, the height margin during loading work can be reduced, thereby improving the work efficiency of the hydraulic excavator 1 and the transport vehicle 200. In other words, productivity at the work site is improved. When the hydraulic excavator 1 is controlled automatically, the operating speed of the hydraulic excavator 1 after the recognition model is created may be made faster than the operating speed before the recognition model was created. When the hydraulic excavator 1 is controlled semi-automatically, the upper limit of the operating speed of the hydraulic excavator 1 after the recognition model is created may be made higher than the upper limit of the operating speed before the recognition model was created. By increasing the operating speed after the recognition model is created, productivity at the work site can be improved.

[0124] Generally, the shapes of the cab 21 and the tray T vary greatly depending on the manufacturer and model of the transport vehicle 200. Therefore, when using a recognition algorithm that features these shapes, the recognition model and the model of the actual transport vehicle 200 must match. For shape data of the transport vehicle 200, the manufacturer has CAD (Computer Aided Design) data or the like. Therefore, it is conceivable to obtain CAD data and create a recognition model. However, at a work site, the body of the transport vehicle 200 and the tray T may be manufactured by different manufacturers. In this case, it may be difficult to obtain CAD data corresponding to the transport vehicle 200 operating at the work site.

[0125] ​ In contrast, in this embodiment, point cloud data of the transport vehicle 200 can be accumulated and a recognition model can be created in parallel with work at the work site, so an appropriate recognition model can be created efficiently even when CAD data is not available. Furthermore, in this embodiment, when creating a recognition model, there is no need to perform manual work such as comparing data collected at the work site with correct data (teaching data). This reduces the amount of work required to create a recognition model.

[0126] According to this embodiment, the robustness of the recognition of the transport vehicle 200 is improved, and by applying the control function of the hydraulic excavator 1, the productivity of transport work at the work site can be improved.

[0127] According to the above-described embodiment, the following advantageous effects are achieved.

[0128] (1) The transport vehicle recognition system 110 includes an information processing device 54 that recognizes the position and posture of the transport vehicle 200 around the hydraulic excavator (work machine) 1 by using point cloud data, which is a measurement result of the surroundings of the hydraulic excavator (work machine) 1, acquired by a measuring device 70. The result of the recognition processing of the transport vehicle 200 by the information processing device 54 is output to the control device 40 of the hydraulic excavator 1. The hydraulic excavator 1 includes a traveling body 5, a revolving body 7 that is rotatably attached to the traveling body 5, and a work implement 2 that is attached to the revolving body 7. The hydraulic excavator 1 includes a measuring device 70 that is provided on the revolving body 7 and measures objects around the revolving body 7 and outputs point cloud data representing the measurement results. The hydraulic excavator 1 includes an attitude detection device 53 that detects the attitude of the revolving body 7 (the revolving angle with respect to the traveling body 5). The hydraulic excavator 1 includes a control device 40 that controls the operation of the hydraulic excavator 1 based on the result of the recognition processing by the information processing device 54.

[0129] The information processing device 54 is provided in the hydraulic excavator 1. The information processing device 54 functions as a first processing device 131 that performs recognition processing to recognize the position and attitude of the haulage vehicle 200 and the shape of the tray T in a vehicle body coordinate system 400 based on the running body 5 of the hydraulic excavator 1, based on the point cloud data output by the measuring device 70 and the attitude of the revolving unit 7 detected by the attitude detection device 53. The information processing device 54 functions as a second processing device 132 that accumulates combination data that associates the point cloud data used to recognize the haulage vehicle 200 with the position and attitude of the haulage vehicle 200 when the point cloud data was acquired, and constructs a recognition model of the haulage vehicle 200 based on the accumulated combination data.

[0130] The information processing device 54 as the first processing device 131 acquires point cloud data from the measuring device 70 and acquires information about the attitude of the hydraulic excavator 1 from the attitude detection device 53 to perform recognition processing of the haulage vehicle 200. In the recognition processing of the haulage vehicle 200, if a recognition model corresponding to the haulage vehicle 200 that is the target of work by the hydraulic excavator 1 has not been constructed, the information processing device 54 as the first processing device 131 sets the recognition processing mode to a learning mode (first recognition mode) that does not use a recognition model. In the recognition processing of the haulage vehicle 200, if a recognition model corresponding to the haulage vehicle 200 that is the target of work by the hydraulic excavator 1 has been constructed, the information processing device 54 as the first processing device 131 switches the recognition processing mode from the learning mode (first recognition mode) to a model use mode (second recognition mode) that uses a recognition model. In the recognition processing in the learning mode, the information processing device 54 as the first processing device 131 recognizes the position and attitude of the haulage vehicle 200 based on the point cloud data and the attitude of the hydraulic excavator 1. The information processing device 54 as the first processing device 131 recognizes the position and posture of the transport vehicle 200 based on the recognition model and point cloud data in the recognition process in the model use mode.

[0131] In this embodiment, an example has been described in which the information processing device 54 has the functions of the first processing device 131 and the functions of the second processing device 132, but the first processing device 131 and the second processing device 132 may be provided separately.

[0132] According to this embodiment, a recognition model is created based on the recognition results of the position and attitude of the haulage vehicle 200 and the measurement results of the measuring device 70, so manual work is not required and the work required to create the recognition model can be reduced. Furthermore, according to this embodiment, even if a recognition model of the haulage vehicle 200 does not exist at the start of work, it is possible to create a recognition model by collecting information about the haulage vehicle 200 while the hydraulic excavator 1 is working, and after the recognition model is created, it is possible to recognize the position and attitude of the haulage vehicle 200 with high accuracy using the recognition model. Since the position and attitude of the haulage vehicle 200 can be recognized with high accuracy, work efficiency can be improved.

[0133] (2) The information processing device 54 as the second processing device 132 determines whether construction of the recognition model is complete based on at least one of the density of point cloud data of the shape of the haulage vehicle 200 within a predetermined designated area, the number of point cloud data, and the number of combination data. If the information processing device 54 as the second processing device 132 determines that construction of the recognition model is complete, it retains the constructed recognition model.

[0134] The designated area is designated based on the positional relationship between the transport vehicle 200 and the hydraulic excavator 1 during loading work. The designated area corresponds to, for example, all the grids that make up the top level of a rectangular parallelepiped (see FIG. 15). In this way, by designating the designated area taking into consideration the work form and the positional relationship between the transport vehicle 200 and the hydraulic excavator 1, it is possible to shorten the time until construction of the recognition model is completed. Reducing the time until construction of the recognition model is completed leads to improved productivity at the work site.

[0135] (3) The information processing device 54 as the first processing device 131 determines whether the haulage vehicle 200 is in an unladen state. When the information processing device 54 as the first processing device 131 determines that the haulage vehicle 200 is in an unladen state, it generates combination data that associates point cloud data of the shape of the haulage vehicle 200 in an unladen state with the position and posture of the haulage vehicle 200 when the point cloud data was acquired. The information processing device 54 as the second processing device 132 accumulates the above-mentioned combination data generated by the information processing device 54 as the first processing device 131, i.e., combination data that associates point cloud data of the shape of the haulage vehicle 200 in an unladen state with the position and posture of the haulage vehicle 200 when the point cloud data was acquired. The accumulated combination data does not include point cloud data of the shape of the haulage vehicle 200 in a loaded state.

[0136] In this configuration, point cloud data is collected when the vehicle is empty, i.e., not carrying any load such as earth and sand, and a recognition model is constructed based on the collected point cloud data. This makes it possible to construct an accurate recognition model that represents the external shape of the transport vehicle 200. By constructing the recognition model with high accuracy, it is possible to perform recognition processing using the recognition model with high accuracy.

[0137] (4) The information processing device 54 serving as the first processing device 131 displays the set recognition processing mode on the monitor (display device) 55 (see FIG. 16 ). In this configuration, the recognition processing mode is notified to the operator. Therefore, when a change in the behavior of the hydraulic excavator 1 occurs due to a decrease in the margin or an increase in the speed of the hydraulic actuator as a result of switching from the learning mode to the model use mode, the reason for the change in the behavior of the hydraulic excavator 1 can be notified to the operator. Furthermore, the information processing device 54 may transmit mode information to the vehicle dispatching management system and cause a display device included in the vehicle dispatching management system to display the set recognition processing mode. In this configuration, the recognition processing mode is notified to the administrator. Switching from the learning mode to the model use mode reduces the loading operation time onto the haulage vehicle 200. Therefore, when the administrator confirms that the learning mode has been switched to the model use mode, the administrator can improve productivity at the work site by changing the vehicle dispatch plan, for example, by increasing the number of haulage vehicles 200 dispatched per given time. In this embodiment, not only the recognition mode but also the progress rate until the construction of the recognition model is completed is displayed on the display device, which enables the operator or manager to predict the timing at which the behavior of the hydraulic excavator 1 will change.

[0138] (5) When the model use mode (second recognition mode) is set, the control device 40 reduces the margin for avoiding interference between the working device 2 and the transporter vehicle 200 compared to when the learning mode (first recognition mode) is set (see FIG. 18). This configuration can improve the efficiency of loading operations.

[0139] When the hydraulic excavator 1 is moved to another work site to perform work and the external shape of the transport vehicle 200 being used at that work site does not correspond to the recognition model stored in the model storage unit 86, the data in the model storage unit 86 is erased. Similarly, when the transport vehicle 200 being used at the work site is changed and a new vehicle type, i.e., a transport vehicle 200 with an external shape that does not correspond to the recognition model, is incorporated into the work, the data in the model storage unit 86 is erased. As a result, the recognition model generation process is executed again.

[0140] <Modification of the first embodiment> In the above embodiment, an example has been described in which the process shown in the flowchart of Fig. 9 is executed every time an empty transport vehicle 200 is placed at a loading position. However, the process shown in the flowchart of Fig. 9 may be executed only when an empty transport vehicle 200 is placed at a loading position for the first time in a day (i.e., only at the start of work), and when an empty transport vehicle 200 is placed at a loading position later during the same day, the processes of steps S111 to S114 in Fig. 9 may be omitted and only the process of step S115 may be executed.

[0141] Second Embodiment A delivery vehicle recognition system 210 according to a second embodiment of the present invention will be described with reference to Figures 19 and 20. Components that are the same as or equivalent to those described in the first embodiment are given the same reference symbols, and differences will be mainly described. In the first embodiment, an example was described in which only delivery vehicles 200 of a predetermined model are performing delivery work at a work site. In contrast, in the second embodiment, the multiple delivery vehicles 200 performing delivery work at a work site include delivery vehicles 200 of different models.

[0142] Fig. 19 is a functional block diagram of an information processing device 254 according to the second embodiment. As shown in Fig. 19, the transport vehicle recognition system 210 according to the second embodiment includes a vehicle type acquisition device 290 in addition to the same configuration as in the first embodiment. The vehicle type acquisition device 290 acquires information related to the vehicle type of the transport vehicle 200 to be loaded. The vehicle type acquisition device 290 acquires information including the vehicle types of the transport vehicles 200 around the hydraulic excavator 1, for example, through wireless communication with a vehicle dispatch management system such as an FMS.

[0143] The vehicle type acquisition device 290 may be an input device that outputs the vehicle type to the information processing device 254 in response to an input operation of the vehicle type by the operator of the hydraulic excavator 1 when the transport vehicle 200 stops at the loading position, for example. The input device is, for example, a touch sensor 56 (see FIG. 2) provided on the display screen of the monitor 55.

[0144] The vehicle type acquisition device 290 may include a photographing device such as a camera, and an image processing device that identifies the vehicle type by image recognition of an image such as a two-dimensional code photographed by the photographing device. The two-dimensional code is created according to the vehicle type. The two-dimensional code is formed on the side of the tray T of the transport vehicle 200, for example. The photographing device is provided on the revolving body 7, photographs the transport vehicle 200 around the revolving body 7, and outputs the photographed image to the image processing device.

[0145] In the second embodiment, a plurality of vehicle types of transport vehicles 200 are operated at a work site. For this reason, the information processing device 254 needs to recognize the vehicle type of the transport vehicle 200 and set an appropriate recognition model every time the transport vehicle 200 stops at the loading position. The information processing device 254 acquires the vehicle type of the transport vehicle 200 stopped around the hydraulic excavator 1 (loading position) from the vehicle type acquisition device 290, searches the model storage unit 86 for a vehicle type-specific recognition model corresponding to the stopped transport vehicle 200, and determines whether or not the recognition model is stored in the model storage unit 86.

[0146] Fig. 20 is a flowchart similar to Fig. 9, showing an example of a processing flow including a mode setting process and a recognition process executed by the information processing device 254 according to the second embodiment. In the flowchart of Fig. 20, the process of step S211 is executed instead of the process of step S111 in the flowchart of Fig. 9, and the process of step S210 is executed before the process of S211.

[0147] (Step S210) In step S210, the mode switching unit 281 acquires the vehicle type of the transport vehicle 200 from the vehicle type acquisition device 290. When the control device 40 confirms that the transport vehicle 200 is stopped (see S101 in FIG. 7), the vehicle type acquisition device 290 acquires the vehicle type and outputs it to the mode switching unit 281.

[0148] (Step S211) In step S211, the mode switching unit 281 determines whether or not a recognition model corresponding to the vehicle model acquired in step S210 is stored in the model storage unit 86. If a recognition model corresponding to the vehicle model is stored in the model storage unit 86, the mode switching unit 281 proceeds to step S112 and sets a model usage mode for using that recognition model. If a recognition model corresponding to the vehicle model is not stored in the model storage unit 86, the mode switching unit 281 proceeds to step S113.

[0149] In this way, the information processing device 254 according to the second embodiment checks whether a recognition model (vehicle-specific recognition model) corresponding to the vehicle type of the transport vehicle 200 being used at the work site where the hydraulic excavator 1 is operating exists in the model storage unit 86. If a corresponding vehicle-specific recognition model exists, the information processing device 254 sets the model usage mode and executes recognition processing using the vehicle-specific recognition model. On the other hand, if a corresponding vehicle-specific recognition model does not exist, the information processing device 254 sets the learning mode and executes recognition processing without using a recognition model.

[0150] The information processing device 254 executes the processing shown in the flowchart of Fig. 20 every time the transport vehicle 200 that is the target of the loading operation stops around the hydraulic excavator 1 (loading position). When the learning mode is set by the processing of step S113 of Fig. 20, the recognition processing shown in the flowchart of Fig. 12 is executed, as in the first embodiment. Furthermore, in step S134 of Fig. 13, the recognition result acquisition unit 83 stores in the recognition result storage unit 85 combination data that associates the vehicle type of the transport vehicle 200 acquired by the processing of step S210 of Fig. 20, point cloud data representing the shape of the transport vehicle 200, and the position and attitude of the transport vehicle 200 at the time the point cloud data was acquired (recognition result of the transport vehicle 200).

[0151] When the execution conditions for the recognition model generation process are met, for example, by updating the recognition result storage unit 85, the process shown in the flowchart of FIG. 14 is executed, as in the first embodiment. In step S142 of FIG. 14, the model generation unit 84 creates a recognition model specific to the specified vehicle model using the combination data of the specified vehicle model acquired in step S141. As a method for creating the model, for example, a 3D model representing the external shape of the transport vehicle 200 of the specified vehicle model is created by overlaying multiple point cloud data of the specified vehicle model acquired in time series. Prior to the overlay process, the point cloud data is converted into coordinates of the vehicle coordinate system 600 of the transport vehicle 200. In step S144, the model generation unit 84 stores the 3D model created in step S142 in the model storage unit 86 as a recognition model specific to the specified vehicle model. The recognition model is stored in association with the specified vehicle model (label information) acquired in the process of step S210 of FIG. 20.

[0152] In this way, the model generation unit 84 creates point cloud data that is a 3D model of the external shape of each vehicle model as a recognition model by overlaying point cloud data for each vehicle model based on the combination data for each vehicle model stored in the recognition result storage unit 85, and stores the point cloud data in the model storage unit 86.

[0153] As described above, in the second embodiment, the information processing device 254 serving as the second processing device 232 accumulates combination data for each vehicle type. The information processing device 254 (second processing device 232) constructs a recognition model of the haulage vehicle 200 for each vehicle type based on the accumulated combination data for each vehicle type. The information processing device 254 serving as the first processing device 231 acquires the vehicle type of the haulage vehicle 200 to be worked on by the hydraulic excavator 1 (i.e., the haulage vehicle 200 positioned at the loading position) from the vehicle type acquisition device 290. The information processing device 254 (first processing device 231) determines whether a recognition model corresponding to the acquired vehicle type is held in the information processing device 254 (second processing device 232). If the information processing device 254 (first processing device 231) determines that a recognition model corresponding to the acquired vehicle type is held in the information processing device 254 (second processing device 232), it sets the mode of the recognition process to a model use mode (second recognition mode) that uses the recognition model corresponding to the vehicle type.

[0154] In the second embodiment, when multiple types of transport vehicles 200 are in operation at a work site, the recognition accuracy and robustness of the transport vehicles 200 can be improved. As in the first embodiment, a recognition model can be created by continuing work, so manual work for creating the recognition model is not required and the amount of work can be reduced. Furthermore, by performing recognition processing using the recognition model, the position and posture of the transport vehicle 200 can be recognized with high accuracy, so by reducing the margin, the efficiency of the loading work of the hydraulic excavator 1 can be improved and productivity can be improved.

[0155] <Third embodiment> A transport vehicle recognition system 310 according to a third embodiment of the present invention will be described with reference to Fig. 21. Components that are the same as or equivalent to those described in the second embodiment are given the same reference symbols, and differences will be mainly described. In the first and second embodiments, an example was described in which the transport vehicle recognition system 310 is mounted on a hydraulic excavator 1. In contrast, in the third embodiment, some of the functions of the transport vehicle recognition system 310 are realized by a server 354B.

[0156] In the third embodiment, the information processing device 354A mounted on the hydraulic excavator 1 has the same functions as the information processing device 254 described in the second embodiment, as a first processing device 331 that executes a recognition process for recognizing the relative position and attitude of the haulage vehicle 200 with respect to the traveling body 5 of the hydraulic excavator 1 based on the point cloud data output by the measuring device 70 and the attitude of the revolving unit 7 detected by the attitude detection device 53, and a mode switching process for switching the recognition process mode. In addition, the server 354B has the functions of a second processing device 332 that accumulates combination data that associates the point cloud data used to recognize the haulage vehicle 200 with the position and attitude of the haulage vehicle 200 when the point cloud data was acquired, and executes a model generation process for constructing a recognition model of the haulage vehicle 200 based on the accumulated combination data.

[0157] The server 354B collects combination data for each vehicle type from the multiple hydraulic excavators 1 and generates a recognition model for each vehicle type. The recognition model generated by the server 354B is transmitted to the multiple hydraulic excavators 1. In other words, in the third embodiment, the recognition model generated based on the recognition results of the transport vehicle 200 by the multiple hydraulic excavators 1 is shared by the multiple hydraulic excavators 1.

[0158] 21 is a functional block diagram of a server 354B. The server 354B is provided in a management facility that is installed in a location away from the work site. The management facility is provided with a management device 350 that includes the server 354B and a communication device 360 ​​connected to the server 354B. The server 354B has a model generation unit 384, a recognition result storage unit 385, and a model storage unit 386. The model generation unit 384, the recognition result storage unit 385, and the model storage unit 386 have functions similar to those of the model generation unit 84, the recognition result storage unit 85, and the model storage unit 86 of the second embodiment.

[0159] The server 354B is equipped with a communication device 360. The hydraulic excavator 1 is equipped with a communication device 60. The communication devices 360, 60 are configured to be able to perform two-way communication via a communication line NT of a wide area network. In other words, the server 354B and the information processing device 354A of the hydraulic excavator 1 can send and receive information (data) via the communication line NT. The communication line NT is a mobile phone communication network (mobile communication network) deployed by a mobile phone carrier or the like, the Internet, or the like.

[0160] For example, the hydraulic excavator 1A working at work site A, the hydraulic excavator 1B working at work site B, and the hydraulic excavator 1C working at work site C each transmit the recognition result of the transport vehicle 200 to the server 354B via the communication device 60. That is, in step S134 of Fig. 13 , the information processing device 354A temporarily stores the recognition result and executes processing to transmit the stored recognition result to the server 354B.

[0161] For example, when combination data for a predetermined vehicle type is transmitted from each of the hydraulic excavators 1A to 1C to the server 354B, the communication device 360 ​​stores the acquired combination data for the predetermined vehicle type in the recognition result storage unit 385. The model generation unit 384 constructs a recognition model by superimposing the point cloud data of the combination data for the predetermined vehicle type stored in the recognition result storage unit 385. In this embodiment, the recognition model is constructed based on data from multiple hydraulic excavators 1, so the time required to complete construction of the recognition model can be shortened compared to when the recognition model is constructed based on data from a single hydraulic excavator 1. The model generation unit 384 stores the constructed recognition model in the model storage unit 386.

[0162] When the information processing device 354A confirms that the transport vehicle 200 has stopped at the loading position, it transmits a request command for the vehicle type to the server 354B. In response to the request command from the hydraulic excavator 1, the server 354B transmits the vehicle type of the constructed recognition model stored in the model storage unit 386 to the hydraulic excavator 1. When the vehicle type of the vehicle stopped at the loading position matches the vehicle type acquired from the server 354B, the information processing device 354A determines that the recognition model is stored in the server 354B and sets the model usage mode. When the model usage mode is set, the information processing device 354A transmits a request command for a recognition model corresponding to the vehicle type of the transport vehicle 200 stopped at the loading position to the server 354B. In response to the request command from the hydraulic excavator 1, the server 354B transmits the constructed recognition model stored in the model storage unit 386 and the vehicle type of the recognition model to the hydraulic excavator 1. The information processing device 354A stores the received recognition model and executes recognition processing using the stored recognition model.

[0163] As described above, the haulage vehicle recognition system 310 according to the third embodiment includes an information processing device 354A (first processing device 331) provided in the hydraulic excavator 1, and a server 354B (second processing device 332) provided outside the hydraulic excavator 1. The server 354B collects and stores combination data for each vehicle type from the information processing devices 354A provided in each of the multiple hydraulic excavators 1. The server 354B constructs a recognition model for the haulage vehicle 200 based on the stored combination data.

[0164] In the third embodiment, the recognition results of the transport vehicles 200 at a plurality of work sites are collected by the server 354B, and the vehicle-specific recognition models created by the server 354B are shared among the hydraulic excavators 1 at a plurality of work sites. Therefore, the time required to create a recognition model for each vehicle type can be reduced, and the recognition results can be used to automatically perform the loading operation of the hydraulic excavator 1 onto the transport vehicle 200, thereby improving the productivity of the entire work site.

[0165] For example, when a transport vehicle 200 of vehicle type A is operated for the first time at work site A, it may have already been operated at other work sites B and C, and a recognition model for that vehicle type A may have been created. In this case, even when the hydraulic excavator 1 at work site A is performing loading work on the transport vehicle 200 of vehicle type A for the first time, it can perform recognition processing using a recognition model specific to the vehicle type.

[0166] The following modified examples are also within the scope of the present invention, and it is possible to combine the configuration shown in the modified example with the configuration described in the above embodiment, to combine the configurations described in the different embodiments above, or to combine the configurations described in the different modified examples below.

[0167] <Variation 1> The operator may be allowed to select a measurement surface (designated area) of the haulage vehicle 200 to be used in the process of determining completion of construction of a recognition model (step S143 in FIG. 14). For example, in the haulage vehicle recognition system 110 according to the first embodiment, when the ignition switch is turned on, the information processing device 54 causes the monitor 55 to display a screen indicating whether or not selection of a measurement surface is required. The monitor 55 is a touch panel monitor having a touch sensor 56 (see FIG. 2) provided on the display screen. When the operator operates the touch sensor 56 of the monitor 55 to start selection of a measurement surface, the information processing device 54 causes the monitor 55 to display a measurement surface selection screen 450.

[0168] Fig. 22 is a diagram showing an example of a measurement surface selection screen 450. As shown in Fig. 22, the measurement surface selection screen 450 displays a first selection area 451, a second selection area 452, a third selection area 453, a fourth selection area 454, a fifth selection area 455, and a sixth selection area 456 as operation areas for selecting a measurement surface. The measurement surface selection screen 450 displays a guide image 459 showing a rectangular parallelepiped surrounding the transporter vehicle 200 and the names of each surface constituting the rectangular parallelepiped.

[0169] When the first selection area 451 is operated, the information processing device 54 selects the "front" of the haulage vehicle 200 as the measurement surface. When the second selection area 452 is operated, the information processing device 54 selects the "right side" of the haulage vehicle 200 as the measurement surface. When the third selection area 453 is operated, the information processing device 54 selects the "left side" of the haulage vehicle 200 as the measurement surface. When the fourth selection area 454 is operated, the information processing device 54 selects the "rear" of the haulage vehicle 200 as the measurement surface. When the fifth selection area 455 is operated, the information processing device 54 selects the "top" of the haulage vehicle 200 as the measurement surface. When the sixth selection area 456 is operated, the information processing device 54 selects the "bottom" of the haulage vehicle 200 as the measurement surface.

[0170] The information processing device 54 executes a process for determining whether construction of a recognition model is complete (step S143 in FIG. 14) for the selected measurement surface (designated region). If multiple measurement surfaces are selected, the information processing device 54 determines that construction of a recognition model is complete when the proportion α of high-density grids becomes equal to or greater than the proportion threshold αth for all of the selected measurement surfaces.

[0171] As described above, the transport vehicle recognition system 110 according to the present modified example 1 includes a touch sensor 56 (input device) that inputs a measurement surface (designated area) used in the process of determining whether construction of a recognition model has been completed. The information processing device 54 (second processing device 132) determines whether construction of a recognition model has been completed based on the density of point cloud data of the shape of the transport vehicle 200 within the designated area input by the touch sensor 56.

[0172] Depending on the work site, the relative positions of the transport vehicle 200 and the hydraulic excavator 1 may be constant when loading work is performed. In this case, by limiting the measurement surface (designated area) as in this modified example, the time required to create a recognition model can be reduced. This modified example can also be applied to the second and third embodiments.

[0173] <Variation 2> In the above-described first modification, an example has been described in which the operator selects measurement surfaces one by one. However, the operator may select an arrangement pattern of the haulage vehicle 200 relative to the hydraulic excavator 1, and the information processing device 54 may select a measurement surface according to the selected arrangement pattern. For example, in the haulage vehicle recognition system 110 according to the first embodiment, when the ignition switch is turned on, the information processing device 54 displays on the monitor 55 a screen indicating whether or not to select an arrangement pattern. The monitor 55 is a touch panel monitor having a touch sensor 56 (see FIG. 2 ) provided on the display screen. When the operator operates the touch sensor 56 of the monitor 55 to start selecting an arrangement pattern, the information processing device 54 displays on the monitor 55 an arrangement pattern selection screen 550.

[0174] FIG. 23 is a diagram showing an example of a selection screen 550 for selecting a placement pattern of the transport vehicle 200 relative to the hydraulic excavator 1. As shown in FIG. 23, the placement pattern selection screen 550 displays images 554a, 554b, and 554c representing a plurality of placement patterns of the transport vehicle 200 relative to the hydraulic excavator 1. The selection screen 550 displays a message 553 prompting the user to select one placement pattern from the plurality of placement patterns. The selection screen 550 displays a left-side selection area 552L that is selected when the transport vehicle 200 is placed on the left side of the hydraulic excavator 1, and a right-side selection area 552R that is selected when the transport vehicle 200 is placed on the right side of the hydraulic excavator 1. The selection screen 550 displays a first selection area 551a, a second selection area 551b, and a third selection area 551c as operation areas for selecting a placement pattern.

[0175] FIG. 24 is a diagram showing a measurement surface table, which is a data table that defines the relationship between the arrangement pattern and the selected measurement surface. The information processing device 54 stores the measurement surface table. When the left selection area 552L and the first selection area 551a are selected, the information processing device 54 sets an arrangement pattern AL. The information processing device 54 refers to the measurement surface table and selects a measurement surface (right side, back surface, and top surface) corresponding to the arrangement pattern AL. When the left selection area 552L and the second selection area 551b are selected, the information processing device 54 sets an arrangement pattern BL. The information processing device 54 refers to the measurement surface table and selects a measurement surface (front surface, left side surface, and top surface) corresponding to the arrangement pattern BL. When the left selection area 552L and the third selection area 551c are selected, the information processing device 54 sets an arrangement pattern CL. The information processing device 54 refers to the measurement surface table and selects a measurement surface (left side surface, back surface, and top surface) corresponding to the arrangement pattern CL.

[0176] When the right selection area 552R is selected and the first selection area 551a is selected, the information processing device 54 sets an arrangement pattern AR. The information processing device 54 refers to the measurement surface table and selects measurement surfaces (left side, back, and top) corresponding to the arrangement pattern AR. When the right selection area 552R is selected and the second selection area 551b is selected, the information processing device 54 sets an arrangement pattern BR. The information processing device 54 refers to the measurement surface table and selects measurement surfaces (front, right side, and top) corresponding to the arrangement pattern BR. When the right selection area 552R is selected and the third selection area 551c is selected, the information processing device 54 sets an arrangement pattern CR. The information processing device 54 refers to the measurement surface table and selects measurement surfaces (right side, back, and top) corresponding to the arrangement pattern CR.

[0177] The information processing device 54 executes a process for determining whether construction of a recognition model is complete (step S143 in FIG. 14) for the selected measurement surface (designated region). If multiple measurement surfaces are selected, the information processing device 54 determines that construction of a recognition model is complete when the proportion α of high-density grids becomes equal to or greater than the proportion threshold αth for all of the selected measurement surfaces.

[0178] As described above, the transport vehicle recognition system 110 according to the second modification includes a touch sensor 56 (input device) for inputting the arrangement pattern of the transport vehicle 200 relative to the hydraulic excavator 1. The information processing device 54 stores a measurement plane table (data table) that defines the relationship between the arrangement pattern and the designated area (see FIG. 24). The information processing device 54 refers to the measurement plane table and determines the measurement plane (designated area) based on the arrangement pattern input by the touch sensor 56.

[0179] Depending on the work site, the relative positioning of the transport vehicle 200 and the hydraulic excavator 1 may be constant when loading work is performed. In this case, by limiting the positioning pattern as in this modified example, the time required to create a recognition model can be reduced. This modified example can also be applied to the second and third embodiments. When applied to the third embodiment, the measurement plane table only needs to be stored in at least one of the information processing device 354A and the server 354B. Furthermore, the process of determining the measurement plane (designated area) using the measurement plane table may be performed by either the information processing device 354A or the server 354B.

[0180] <Variation 3> The method of recognition processing (recognition processing when the learning mode is set) before the recognition model is constructed is not limited to the example described in the above embodiment. When the transport vehicle 200 and the hydraulic excavator 1 are each equipped with a position and attitude detection device capable of detecting the position and attitude (orientation) of the transport vehicle, the hydraulic excavator 1 may acquire the position, attitude, and vehicle type of the transport vehicle 200 from the transport vehicle 200. The position and attitude of the vehicle are determined, for example, by using an antenna for multiple GNSS (Global Navigation Satellite Systems) (hereinafter referred to as the GNSS antenna) and a positioning calculation device that calculates the vehicle's position expressed in real coordinates in three-dimensional space and the azimuth angle, which is the angle from a reference orientation, based on satellite signals (GNSS radio waves) from multiple positioning satellites received by the GNSS antenna. The vehicle's position is expressed, for example, by position coordinates in a geographic coordinate system (global coordinate system).

[0181] In this modified example, the information processing device 54 converts the position and attitude (orientation) of the transport vehicle 200 in the geographic coordinate system into the position and attitude in the vehicle body coordinate system 400 of the hydraulic excavator 1 (traveling body 5) based on the position and attitude (orientation) of the hydraulic excavator 1 (swinging unit 7) in the geographic coordinate system and the swing angle θsw of the swinging unit 7. The information processing device 54 stores in the recognition result storage unit 85 combination data that associates the position and attitude of the transport vehicle 200 in the vehicle body coordinate system 400 of the hydraulic excavator 1 with the vehicle type acquired from the transport vehicle 200 and the measurement results (point cloud data) of the measuring device 70.

[0182] According to this modification, it is possible to obtain the same effects as those of the above embodiment. Note that the transport vehicle 200 may be equipped with a tray T that is different from the tray T that is normally installed in the vehicle model. Even in such a case, a recognition model of the transport vehicle 200 is appropriately generated, so that highly accurate work can be performed.

[0183] <Variation 4> The method for creating a vehicle-specific recognition model is not limited to the method described in the above embodiment. For example, a classifier constructed by machine learning using the obtained recognition result of the transport vehicle 200 and the output data of the measuring device 70 at that time as training data may be used as the vehicle-specific recognition model.

[0184] <Variation 5> In the above embodiment, an example has been described in which the operator operates each part of the hydraulic excavator 1 from within the operator's cab 21, but the present invention may also be applied to a remotely controlled hydraulic excavator 1. Furthermore, the present invention is applicable not only to hydraulic excavators 1 that are manually operated, but also to hydraulic excavators 1 in which some or all of the operations are performed automatically.

[0185] <Variation 6> In the above embodiment, an example has been described in which the work machine is a hydraulic excavator 1, but the present invention is not limited to this. The present invention can be applied to various work machines that perform work in cooperation with a transport vehicle 200. For example, the present invention may be applied to a work machine such as a wheel loader.

[0186] <Variation 7> In the above embodiment, an example has been described in which the recognition model is configured with point cloud data. However, the recognition model may be configured with mesh data. The mesh data has a data structure such as a triangular mesh model or a quadrilateral mesh model. The mesh data includes information on the coordinates of multiple nodes, the edges connecting the nodes, and the constituent surfaces surrounded by the multiple edges. Furthermore, the recognition model may be configured with an implicit function surface.

[0187] <Variation 8> In the above embodiments, the case where at least a portion of the haulage vehicle recognition systems 110, 210, 310 are installed on the work machine has been described. However, the configuration of the haulage vehicle recognition system is not limited to this. For example, the haulage vehicle recognition system may be configured to acquire various information from the work machine or other external sources, recognize haulage vehicles around the work machine, and output the results to the outside. For example, the information processing device 54 in the first embodiment, the information processing device 254 in the second embodiment, and the information processing device 354A in the third embodiment may be installed outside the work machine, for example, on a server 354B.

[0188] In other words, the transport vehicle recognition system includes an information processing device that recognizes the position and attitude of the transport vehicle around the work machine using point cloud data, which is the measurement result of the area around the work machine acquired by a measuring device, and outputs the results of the transport vehicle recognition processing by the information processing device to the work machine's control device, wherein the information processing device acquires point cloud data from the measuring device and acquires information about the attitude from the work machine's attitude detection device to perform transport vehicle recognition processing, accumulates combination data that corresponds the point cloud data with the position and attitude of the transport vehicle at the time the point cloud data was acquired, and constructs a recognition model of the transport vehicle based on the accumulated combination data, and in the transport vehicle recognition processing, if a recognition model corresponding to the transport vehicle that will be the target of work by the work machine has not been constructed, performs recognition processing in a first recognition mode that recognizes the position and attitude of the transport vehicle based on the point cloud data and the attitude of the work machine, and if a recognition model corresponding to the transport vehicle that will be the target of work by the work machine has been constructed, performs recognition processing in a second recognition mode that recognizes the position and attitude of the transport vehicle based on the recognition model and point cloud data.

[0189] <Additional Notes> Although the embodiments of the present invention have been described above, these embodiments merely illustrate some of the application examples of the present invention, and the technical scope of the present invention is not intended to be limited to the specific configurations of the above embodiments. The present invention is not limited to those having all of the configurations described in the above embodiments, and also includes those in which some of the configurations are omitted. Furthermore, the above-described configurations, functions, etc. may be realized in part or in whole by designing, for example, an integrated circuit. Furthermore, the above-described configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. [Explanation of symbols]

[0190] 1...hydraulic excavator (work machine), 2...working device, 3...vehicle body, 5...traveling body, 7...swivel body, 8...boom, 9...arm, 10...bucket, 11...boom cylinder (hydraulic cylinder), 12...arm cylinder (hydraulic cylinder), 13...bucket cylinder (hydraulic cylinder), 18...tilt angle sensor (attitude sensor), 19...swivel angle sensor (attitude sensor), 21...operator's cab, 22, 23...operation device, 40...control device, 53...attitude detection device, 54...information processing device, 55...monitor (display device), 56...touch sensor (input device), 57...storage device, 60...communication device, 70...measuring device, 80...transportation amount calculation device, 81...mode switching unit, 82...recognition unit, 83...recognition result acquisition unit, 84...model generation unit, 85...recognition result storage unit, 86...model storage unit, 110...transportation vehicle recognition system, 131...first processing device, 132...second processing device, 2 00...transport vehicle, 210...transport vehicle recognition system, 231...first processing device, 232...second processing device, 254...information processing device, 281...mode switching unit, 290...vehicle type acquisition device, 300...sensor coordinate system, 310...transport vehicle recognition system, 331...first processing device, 332...second processing device, 350...management device, 354A...information processing device, 354B...server, 360...communication device, 384...model generation unit, 385... Recognition result storage unit, 386...model storage unit, 400...vehicle body coordinate system (hydraulic excavator reference coordinate system), 450...measurement surface (designated area) selection screen, 451-456...first to sixth selection areas, 500...site coordinate system, 550...arrangement pattern selection screen, 551a-551c...first to third selection areas, 552L...left side selection area, 552R...right side selection area, 600...vehicle coordinate system (transport vehicle reference coordinate system), h1, h2...margins

Claims

1. A transport vehicle recognition system including an information processing device that recognizes the position and attitude of transport vehicles around a work machine using point cloud data that is a measurement result of the surroundings of the work machine acquired by a measurement device, and that outputs the results of recognition processing of the transport vehicles by the information processing device to a control device of the work machine, The information processing device includes: acquiring the point cloud data from the measurement device and acquiring information about the attitude of the work machine from the attitude detection device of the work machine, and performing recognition processing of the transport vehicle; accumulating combination data in which the point cloud data is associated with the position and posture of the transport vehicle when the point cloud data was acquired, and constructing a recognition model of the transport vehicle based on the accumulated combination data; In the recognition process of the transport vehicle, If the recognition model corresponding to the transport vehicle that is the target of work by the work machine has not been constructed, a recognition process is performed in a first recognition mode that recognizes the position and attitude of the transport vehicle based on the point cloud data and the attitude of the work machine; When the recognition model corresponding to the transport vehicle that is the target of work by the work machine is constructed, a recognition process is performed in a second recognition mode in which the position and attitude of the transport vehicle are recognized based on the recognition model and the point cloud data. A transport vehicle recognition system.

2. 2. The transport vehicle recognition system according to claim 1, The information processing device includes a first processing device provided on the work machine and performing recognition processing of the haulage vehicle, and a second processing device that constructs a recognition model of the haulage vehicle. A transport vehicle recognition system.

3. 3. The transport vehicle recognition system according to claim 2, The second processing device is determining whether construction of the recognition model is complete based on at least one of the density of the point cloud data of the shape of the transport vehicle within a pre-specified designated area, the number of the point cloud data, and the number of the combination data; If it is determined that the construction of the recognition model is completed, the constructed recognition model is retained. A transport vehicle recognition system.

4. 3. The transport vehicle recognition system according to claim 2, The first processing device is determining whether the transport vehicle is in an empty state; When it is determined that the transport vehicle is in an unladen state, combination data is generated that associates the point cloud data of the shape of the transport vehicle in the unladen state with the position and attitude of the transport vehicle when the point cloud data was acquired; The second processing device is storing the combined data generated by the first processing device; The accumulated combination data does not include the point cloud data of the shape of the transport vehicle in a loaded state. A transport vehicle recognition system.

5. 3. The transport vehicle recognition system according to claim 2, the first processing device causes a display device to display the set mode of the recognition processing; A transport vehicle recognition system.

6. 3. The transport vehicle recognition system according to claim 2, The second processing device is Accumulating the combination data for each vehicle model, constructing a recognition model of the transport vehicle for each vehicle type based on the stored combination data for each vehicle type; The first processing device is The vehicle type of the transport vehicle that is the target of work by the work machine is acquired; determining whether the recognition model corresponding to the acquired vehicle model is stored in the second processing device; when it is determined that the recognition model corresponding to the acquired vehicle type is stored in the second processing device, the mode of the recognition process is set to the second recognition mode that uses the recognition model corresponding to the vehicle type. A transport vehicle recognition system.

7. 7. The transport vehicle recognition system according to claim 6, the first processing device is configured by an information processing device provided on the work machine, the second processing device is configured by a server provided outside the work machine, the second processing device collects and stores the combination data for each vehicle type from the first processing device provided in each of the plurality of work machines, A transport vehicle recognition system.

8. 4. The transport vehicle recognition system according to claim 3, an input device for inputting the designated area; the second processing device determines whether construction of the recognition model is completed based on the density of the point cloud data of the shape of the transport vehicle within the specified area input by the input device. A transport vehicle recognition system.

9. 4. The transport vehicle recognition system according to claim 3, an input device for inputting an arrangement pattern of the transport vehicle relative to the work machine, a data table defining a relationship between the arrangement pattern and the designated area is stored in the first processing device or the second processing device; the first processing device or the second processing device refers to the data table and determines the designated area based on the arrangement pattern input by the input device. A transport vehicle recognition system.

10. A work machine equipped with the transport vehicle recognition system according to claim 1, When the second recognition mode is set, the control device reduces a margin for avoiding interference between a work implement of the work machine and the transport vehicle compared to when the first recognition mode is set. A work machine characterized by:

Images