Remote control system for industrial machinery and method for remote control of industrial machinery

Abstract

To prevent a decline in work efficiency. [Solution] The remote control system for the work machine includes a processor. The processor acquires captured images showing the work site of the remotely controlled work machine captured by an imaging device, and displays teaching images related to the working area of ​​the work machine, specified by a management device that manages the work site, together with the captured images on a display device located outside the work machine.

Description

Technical Field

[0001] The present disclosure relates to a remote operation system for a work machine and a method for remotely operating a work machine.

Background Art

[0002] In the technical field related to a remote operation system for a work machine, a remote operation system as disclosed in Patent Document 1 is known. In the remote operation system, a work site of a work machine is imaged by an imaging device. An imaging image of the work site imaged by the imaging device is displayed on a display device arranged in a remote operation room. An operator in the remote operation room remotely operates the work machine while checking the imaging image of the work site displayed on the display device.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The work machine may work in an operation area designated at the work site. When it becomes difficult for the operator in the remote operation room to recognize the operation area, the work efficiency of the work machine may decrease.

[0005] The present disclosure aims to suppress a decrease in work efficiency.

Means for Solving the Problems

[0006] This disclosure provides a remote control system for a work machine. The remote control system for a work machine includes a processor. The processor acquires captured images showing the work site of the remotely controlled work machine captured by an imaging device, and causes a teaching image relating to the working area of ​​the work machine, specified by a management device that manages the work site, to be displayed together with the captured images on a display device located outside the work machine. [Effects of the Invention]

[0007] According to this disclosure, the decline in work efficiency is suppressed. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a schematic diagram showing a work site according to the first embodiment. [Figure 2] Figure 2 is a schematic diagram showing the management system and remote control system according to the first embodiment. [Figure 3] Figure 3 is a schematic side view showing a bulldozer according to the first embodiment. [Figure 4] Figure 4 is a schematic diagram showing the soil removal area according to the first embodiment. [Figure 5] Figure 5 is a hardware configuration diagram showing a remote controller according to the first embodiment. [Figure 6] Figure 6 is a block diagram showing the management system and remote control system according to the first embodiment. [Figure 7] Figure 7 is a diagram illustrating the driving data according to the first embodiment. [Figure 8] Figure 8 is a diagram illustrating the divided area according to the first embodiment. [Figure 9] Figure 9 is a diagram illustrating the operating area of ​​a dump truck according to the first embodiment. [Figure 10] Figure 10 is a diagram illustrating the bulldozer operating area according to the first embodiment. [Figure 11] Figure 11 is a diagram illustrating a virtual wall according to the first embodiment. [Figure 12] FIG. 12 is a diagram for explaining a bulldozer operation area and a dump truck operation area according to the first embodiment. [Figure 13] FIG. 13 is a diagram for explaining a bulldozer operation area and a dump truck operation area according to the first embodiment. [Figure 14] FIG. 14 is a diagram showing a display device according to the first embodiment. [Figure 15] FIG. 15 is a flowchart showing a method of displaying a captured image and an instruction image according to the first embodiment. [Figure 16] FIG. 16 is a diagram showing a display device according to the second embodiment. [Figure 17] FIG. 17 is a diagram showing a display device according to the third embodiment. [Figure 18] FIG. 18 is a diagram showing a display device according to the fourth embodiment. [Figure 19] FIG. 19 is a diagram showing a display device according to the fifth embodiment. [Figure 20] FIG. 20 is a diagram schematically showing a waste dumping area according to the sixth embodiment.

MODE FOR CARRYING OUT THE INVENTION

[0009] Hereinafter, embodiments according to the present disclosure will be described with reference to the drawings, but the present disclosure is not limited to the embodiments. The components of the embodiments described below can be combined as appropriate. Also, some components may not be used.

[0010] [First Embodiment] The first embodiment will be described.

[0011] <Work site> FIG. 1 is a diagram schematically showing a work site 1 according to the first embodiment. As the work site 1, a mine or a quarry is exemplified. A mine refers to a place or establishment for mining minerals. A quarry refers to a place or establishment for mining stones. Examples of mines include metal mines for mining metals, non-metal mines for mining limestone, and coal mines for mining coal.

[0012] At the work site 1, a plurality of work machines operate. Examples of work machines include a transport machine that performs a transport operation for transporting a load, a loading machine that performs a loading operation for loading a load onto the transport machine, and an excavation machine that performs an excavation operation for excavating at least a part of the work site 1. The excavation machine can perform one or both of an embankment forming operation for forming an embankment at the work site 1 and a leveling operation for leveling the ground of the work site 1.

[0013] In the embodiment, a dump truck 2 which is an example of a transport machine, a shovel 3 which is an example of a loading machine, and a bulldozer 4 and a motor grader 5 which are examples of excavation machines operate at the work site 1. The dump truck 2 has a dump body. The dump truck 2 can perform a transport operation by loading a load onto the dump body. The shovel 3 has a working device. The shovel 3 can perform a loading operation and an excavation operation using the working device. Each of the bulldozer 4 and the motor grader 5 has a working device. Each of the bulldozer 4 and the motor grader 5 can perform a leveling operation using the working device.

[0014] A loading area 6, a waste dump area 7, and a transport path 8 are provided at the work site 1. Each of the loading area 6, the waste dump area 7, and the transport path 8 is a work area where a work machine operates.

[0015] The loading area 6 refers to a work area where a loading operation for loading a load onto the dump truck 2 is performed. Examples of the load include excavated materials excavated in the loading area 6. The shovel 3 performs an excavation operation and a loading operation in the loading area 6.

[0016] The soil removal area 7 is the work area where the soil removal operation is carried out, in which the dump truck 2 unloads its cargo. The bulldozer 4 performs embankment formation and leveling work in the soil removal area 7. The bulldozer 4 may also perform embankment formation and leveling work in the loading area 6.

[0017] The transport path 8 refers to the road on which the dump truck 2 travels. The transport path 8 leads to the loading area 6 and the soil removal area 7, respectively. The transport path 8 is provided to connect at least the loading area 6 and the soil removal area 7. The dump truck 2 traveling towards at least one of the loading area 6 and the soil removal area 7 travels along the transport path 8. The dump truck 2 travels along the transport path 8, for example, by going back and forth between the loading area 6 and the soil removal area 7. The motor grader 5 performs leveling work on the transport path 8.

[0018] Each of the dump truck 2, shovel 3, bulldozer 4, and motor grader 5 may be either an unmanned or manned work machine. An unmanned work machine is a work machine that performs work without operator operation. A manned work machine is a work machine that performs work based on operator operation. In this embodiment, the dump truck 2 is an unmanned work machine. The dump truck 2 travels around the work site 1 unmanned. Each of the shovel 3, bulldozer 4, and motor grader 5 is a manned work machine.

[0019] <Management system and remote control system> Figure 2 is a schematic diagram showing the management system 10 and remote control system 19 according to the first embodiment. The management system 10 manages the work site 1. The management system 10 manages the dump truck 2, excavator 3, bulldozer 4, and motor grader 5, respectively. In Figure 2, the dump truck 2 and bulldozer 4 are shown as work machines. In the following description, for the sake of simplicity, an example will be given in which the management system 10 manages the dump truck 2 and bulldozer 4.

[0020] The management system 10 comprises a management device 11 and a communication system 12. The management device 11 includes a computer. The management device 11 is located outside the dump truck 2 and the bulldozer 4. The management device 11 is installed in the control facility 13 of the work site 1. The management device 11 manages the work site 1. The management device 11 manages at least the dump truck 2 and the bulldozer 4. The communication system 12 includes at least one of the following: the internet, a mobile phone network, a satellite network, and a local area network (LAN).

[0021] The dump truck 2 comprises a body 14, a running gear 15, a dump body 16, a controller 17, and a wireless communication device 12A. The body 14 includes a body frame. The body 14 is supported by the running gear 15. The running gear 15 supports the body 14 and drives. The running gear 15 includes wheels, tires mounted on the wheels, an engine, a brake system, and a steering system. The dump body 16 is the component into which the cargo is loaded. The dump body 16 is supported by the body 14. The dump body 16 performs dumping and lowering operations. Dumping operation refers to the operation of moving the dump body 16 away from the body 14 and tilting it in the dumping direction. Lowering operation refers to the operation of moving the dump body 16 closer to the body 14. When loading work is performed, the dump body 16 performs a lowering operation. When soil removal work is performed, the dump body 16 performs a dumping operation. The controller 17 includes a computer. The wireless communication device 12A is connected to the controller 17.

[0022] The bulldozer 4 has a controller 18 and a wireless communication device 12B. The controller 18 includes a computer. The wireless communication device 12B is connected to the controller 18.

[0023] The communication system 12 includes a wireless communication device 12A connected to the controller 17 of the dump truck 2, a wireless communication device 12B connected to the controller 18 of the bulldozer 4, and a wireless communication device 12C connected to the management device 11. The management device 11 and the controller 17 of the dump truck 2 communicate wirelessly via the communication system 12. The management device 11 and the controller 18 of the bulldozer 4 communicate wirelessly via the communication system 12.

[0024] In this embodiment, the bulldozer 4 is remotely controlled by a remote control system 19. At least a portion of the remote control system 19 is located in a remote control room 20. The remote control room 20 is located outside the bulldozer 4. The remote control room 20 is installed in a remote location away from the work site 1. The remote control system 19 comprises a remote control device 21, a display device 22, an input device 53, and a remote controller 23.

[0025] The remote control device 21 is located in the remote control room 20. The remote control device 21 is operated by an operator in the remote control room 20. When the remote control device 21 is operated, an operation signal is generated to operate the bulldozer 4. The operator can operate the remote control device 21 while seated in the driver's seat 24.

[0026] The display device 22 is located in the remote control room 20. The display device 22 displays captured images of the work site 1. The display device 22 includes a flat panel display such as a liquid crystal display (LCD) or an organic electroluminescence display (OELD). The display device 22 may also be a curved display or may include a projector. The operator operates the remote control device 21 while checking the captured images of the work site 1 displayed on the display device 22. The bulldozer 4 is remotely controlled by the remote control device 21.

[0027] The input device 53 is located in the remote control room 20. The input device 53 generates input data when operated by an operator. Examples of input devices 53 include a touch panel, buttons, and a computer keyboard.

[0028] The remote controller 23 is located in the remote control room 20. The remote controller 23 includes a computer. The remote controller 23 and the controller 18 of the bulldozer 4 communicate via a communication system 25. The communication system 25 includes at least one of the following: the internet, a mobile phone network, a satellite network, and a local area network (LAN).

[0029] The management device 11 and the remote controller 23 communicate via the communication system 25. Note that communication system 12 and communication system 25 may be separate communication systems. At least a portion of communication system 12 and communication system 25 may be the same communication system.

[0030] <Bulldozer> Figure 3 is a schematic side view showing a bulldozer 4 according to the first embodiment. As shown in Figure 3, the bulldozer 4 comprises a vehicle body 26, a traveling device 27, an excavating workpiece 28, a ripper workpiece 29, a position sensor 30, an attitude sensor 31, and an imaging device 32.

[0031] The vehicle body 26 supports an engine (not shown). The engine is the power source for the bulldozer 4. The running gear 27 supports the vehicle body 26 and moves. The running gear 27 has a pair of tracks 33. The bulldozer 4 moves as the tracks 33 rotate.

[0032] The excavation work machine 28 performs excavation work, embankment work, soil pushing work, embankment formation work, and land leveling work. The excavation work machine 28 is connected to the vehicle body 26. At least a portion of the excavation work machine 28 is positioned in front of the vehicle body 26. The excavation work machine 28 has an excavation blade 34, a lift frame 35, a tilt cylinder 36, and a lift cylinder 37.

[0033] The drilling blade 34 is positioned in front of the vehicle body 26. The drilling blade 34 has a cutting edge 34A. The lift frame 35 supports the drilling blade 34. One end of the lift frame 35 is connected to the rear of the drilling blade 34 via a pivot mechanism. The other end of the lift frame 35 is connected to the side of the traveling device 27 via a pivot mechanism. The lift frame 35 includes a first lift frame 35 connected to the left side of the rear of the drilling blade 34 and a second lift frame 35 connected to the right side of the rear of the drilling blade 34.

[0034] The tilt cylinder 36 and the lift cylinder 37 each operate the drilling blade 34. The tilt cylinder 36 is driven to tilt the drilling blade 34. The lift cylinder 37 is driven to move the drilling blade 34 up and down. One end of the tilt cylinder 36 is connected to the back of the drilling blade 34 via a pivot mechanism. The other end of the tilt cylinder 36 is connected to the upper surface of the lift frame 35. The tilt angle of the drilling blade 34 changes as the tilt cylinder 36 extends and retracts. One end of the lift cylinder 37 is connected to the back of the drilling blade 34 via a pivot mechanism. The middle part of the lift cylinder 37 is connected to the side of the vehicle body 26. The drilling blade 34 moves up and down as the lift cylinder 37 extends and retracts.

[0035] The ripper implement 29 performs ripping operations, such as cutting or crushing the workpiece. For example, it can be used for crushing rocks. The ripper implement 29 is connected to the vehicle body 26. At least a portion of the ripper implement 29 is positioned at the rear of the vehicle body 26. The ripper implement 29 has a shank 38, a ripper arm 39, a tilt cylinder 40, a lift cylinder 41, and a beam 42.

[0036] The shank 38 is located at the rear of the vehicle body 26. The shank 38 has a ripper point 38A. The ripper point 38A is located at the tip (lower end) of the shank 38, including the cutting edge position of the ripper. The ripper arm 39 supports the shank 38. The ripper arm 39 connects the vehicle body 26 and the shank 38. One end of the ripper arm 39 is connected to the rear of the vehicle body 26 via a pivot mechanism. The other end of the ripper arm 39 is connected to a beam 42. The beam 42 is rotatably connected to the ripper arm 39. The shank 38 is connected to the ripper arm 39 via the beam 42.

[0037] The tilt cylinder 40 and the lift cylinder 41 each operate the shank 38. The tilt cylinder 40 and the lift cylinder 41 are each connected to the vehicle body 26. The tilt cylinder 40 drives the shank 38 to tilt. The lift cylinder 37 drives the shank 38 to move up and down. One end of the tilt cylinder 40 is connected to the beam 42 via a pivot mechanism. The other end of the tilt cylinder 36 is connected to the rear of the vehicle body 26. As the tilt cylinder 40 extends and retracts, the tilt angle of the shank 38 changes. The tilt cylinder 40 moves the ripper point 38A in the front-rear direction. One end of the lift cylinder 41 is connected to the beam 42 via a pivot mechanism. The other end of the lift cylinder 41 is connected to the rear of the vehicle body 26. As the lift cylinder 41 extends and retracts, the shank 38 moves in the up-and-down direction. The lift cylinder 41 moves the shank 38, which includes the ripper point 38A, in the vertical direction.

[0038] The ripper implement 29 pierces the workpiece with its ripper point 38A. While the traveling device 27 is stopped, the ripper point 38A is brought into contact with the workpiece, and as the traveling device 27 moves with the ripper point 38A embedded in the workpiece, the workpiece is cut or crushed. While the traveling device 27 is moving, the ripper implement 29 is moved in the up, down, forward, and backward directions, bringing it into contact with the workpiece, and cutting or crushing occurs as the device moves with the workpiece embedded in it.

[0039] The position sensor 30 detects the position of the bulldozer 4. The position sensor 30 is located on the vehicle body 26. The position sensor 30 detects the position of the bulldozer 4 using the Global Navigation Satellite System (GNSS). The Global Navigation Satellite System includes the Global Positioning System (GPS). The Global Navigation Satellite System detects the position in a global coordinate system defined by latitude, longitude, and altitude coordinate data. A global coordinate system refers to a coordinate system fixed to the Earth. The position sensor 30 includes a GNSS receiver located on the vehicle body 26. The position sensor 30 detects the position of the bulldozer 4 in the global coordinate system.

[0040] The attitude sensor 31 detects the attitude of the bulldozer 4. The attitude sensor 31 detects the attitude of the bulldozer 4 at least when the position sensor 30 has detected the position of the bulldozer 4. The attitude sensor 31 is positioned on the vehicle body 26. The attitude of the bulldozer 4 includes the tilt of the vehicle body 26. The attitude of the bulldozer 4 includes the tilt angle of the vehicle body 26 with respect to the horizontal plane. An inertial measurement unit (IMU) is exemplified as the attitude sensor 31. The attitude sensor 31 is capable of detecting the tilt angle of the vehicle body 26 with respect to the horizontal plane. The attitude of the bulldozer 4 includes the bearing of the bulldozer 4. The bearing of the bulldozer 4 includes the bearing angle with respect to a reference bearing. The attitude sensor 31 is capable of detecting the bearing of the bulldozer 4.

[0041] The imaging device 32 captures images of the target object. The imaging device 32 is positioned on the vehicle body 26. The target object of the imaging device 32 includes the work site 1. The target object of the imaging device 32 includes the ground of the work site 1 surrounding the bulldozer 4 and objects surrounding the bulldozer 4. The images of the target object captured by the imaging device 32 are displayed on the display device 22 in the remote control room 20. An RGB camera is given as an example of the imaging device 32. However, the imaging device 32 may also be an RGB-D camera or a hyperspectral camera.

[0042] <Soil Disposal Area> Figure 4 is a schematic diagram showing the soil removal area 7 according to the first embodiment. The soil removal area 7 is an example of a work area where a dump truck 2 and a bulldozer 4 work at the work site 1. The dump truck 2 can travel through the soil removal area 7. The ground in the soil removal area 7 is substantially flat. The dump truck 2 performs soil removal work in the soil removal area 7. A sloping cliff 43 exists around the soil removal area 7. The sloping cliff 43 is located outside the edge 44 of the soil removal area 7. The sloping cliff 43 is a prohibited area where the dump truck 2 and the bulldozer 4 are prohibited from working at the work site 1. The dump truck 2 and the bulldozer 4 cannot travel on the sloping cliff 43. The soil removal area 7 and the sloping cliff 43 are separated by the edge 44 of the soil removal area 7. In this embodiment, a bank 45 is formed on the edge 44. The bank 45 protrudes upward from the ground of the soil removal area 7 near the edge 44.

[0043] During the soil removal operation, the dump truck 2 approaches the edge 44 of the soil removal area 7 while reversing. The controller 17 of the dump truck 2 starts the dumping operation of the dump body 16 after the dump truck 2 has approached the edge 44 of the soil removal area 7. As the dump truck 2 approaches the edge 44 while reversing, the dump body 16 starts the dumping operation, causing the load loaded on the dump body 16 to be removed down the cliff 43.

[0044] The height of the embankment 45 is set so as not to obstruct the soil removal work of the dump truck 2, and so as not to make it difficult for the dump truck 2 to drive over the embankment 45. The height of the embankment 45 is, for example, lower than the radius of the tires of the dump truck 2. Since the embankment 45 is formed on the edge 44, the dump truck 2 performing soil removal work is prevented from going outside the soil removal area 7.

[0045] Bulldozer 4 performs embankment formation work to create an embankment 45 in the soil removal area 7. Bulldozer 4 also performs leveling work to flatten the ground in the soil removal area 7. For example, unevenness may be formed on the ground in the soil removal area 7 due to the passage of dump truck 2 or the dropping of cargo from the dump body 16. Bulldozer 4 performs leveling work to make the ground in the soil removal area 7 flat.

[0046] <Computer> Figure 5 is a hardware configuration diagram showing a remote controller 23 according to the first embodiment. The remote controller 23 includes a computer. The management device 11 includes a processor 47 such as a CPU (Central Processing Unit), a main memory 48 including non-volatile memory such as ROM (Read Only Memory) and volatile memory such as RAM (Random Access Memory), a storage 49, an input / output interface 50 including input / output circuits, and a communication interface 51 including communication circuits. The functions of the management device 11 are stored in the storage 49 as a computer program 52. The processor 47 reads the computer program 52 from the storage 49, loads it into the main memory 48, and executes processing according to the computer program 52. The computer program 52 may be distributed to the management device 11 via a network.

[0047] Similar to the remote controller 23, the management device 11, the controller 17 of the dump truck 2, and the controller 18 of the bulldozer 4 each include a computer. The management device 11, the controller 17 of the dump truck 2, and the controller 18 of the bulldozer 4 each also have a processor, main memory, storage for storing computer programs, an input / output interface, and a communication interface.

[0048] Figure 6 is a block diagram showing the management system 10 and remote control system 19 according to the first embodiment.

[0049] The dump truck 2 includes a running gear 15, a dump body 16, a position sensor 55, a compass sensor 56, and a speed sensor 57.

[0050] The position sensor 55 detects the position of the dump truck 2. The position sensor 55 includes a GNSS receiver located on the dump truck 2. The position sensor 55 detects the position of the dump truck 2 in a global coordinate system.

[0051] The orientation sensor 56 detects the orientation of the dump truck 2. The orientation of the dump truck 2 includes the azimuth angle relative to a reference azimuth. The reference azimuth is, for example, north. An inertial measurement unit (IMU) is exemplified as the orientation sensor 56. The orientation sensor 56 may also include a calculator that calculates the orientation from position data detected by two GNSS antennas installed on the dump truck 2. The calculator can calculate the orientation from a vector connecting the two GNSS antennas. The orientation sensor 56 may also detect the tilt angle of the dump truck 2 with respect to the horizontal plane. If the orientation sensor 56 is an inertial measurement unit (IMU), it can detect the tilt angle of the dump truck 2 with respect to the horizontal plane.

[0052] The speed sensor 57 detects the travel speed of the dump truck 2. The speed sensor 57 detects the travel speed of the dump truck 2 by, for example, detecting the rotational speed of the drive shaft connected to the wheels of the running gear 15.

[0053] The processor 47 of the remote controller 23 has multiple functional units. The functional units of the processor 47 of the remote controller 23 include an operation signal transmission unit 65, an input data transmission unit 66, an image acquisition unit 67, an operating area data acquisition unit 68, and a display control unit 69.

[0054] The operation signal transmission unit 65 transmits the operation signal generated by the operation of the remote control device 21 to the controller 18 of the bulldozer 4. Based on the operation signal transmitted from the operation signal transmission unit 65, the controller 18 controls at least one of the traveling device 27, the excavating work machine 28, and the ripper work machine 29.

[0055] The input data transmission unit 66 transmits the input data generated by the operation of the input device 53 to the management device 11.

[0056] The image acquisition unit 67 acquires an image showing the work site 1 captured by the imaging device 32. The image acquisition unit 67 also acquires detection data from the position sensor 30 and the attitude sensor 31 when the image is captured by the imaging device 32. The position of an object on the image is defined in the camera coordinate system set in the imaging device 32. Based on the detection data from the position sensor 30 and the attitude sensor 31, the image acquisition unit 67 can convert the position in the global coordinate system to the position on the image in the camera coordinate system.

[0057] The operating area data acquisition unit 68 acquires operating area data relating to the operating area of ​​the bulldozer 4 specified by the control device 11. The control device 11 sets the operating area at the work site 1. The operating area is the area in which the bulldozer 4 should operate.

[0058] The display control unit 69 causes the captured image captured by the imaging device 32 to be displayed on the display device 22. The display control unit 69 also causes the teaching image related to the operating area to be displayed on the display device 22 together with the captured image.

[0059] The processor 47 of the management device 11 has multiple functional units. The functional units of the processor 47 of the management device 11 include a notification data generation unit 58, a soil discharge point setting unit 59, a travel data generation unit 60, a divided area setting unit 61, an operating area designation unit 62, and a virtual wall setting unit 63.

[0060] The notification data generation unit 58 generates notification data to be sent to the operator. The notification data includes display data to be displayed on the display device 22. The notification data generation unit 58 generates display data to be displayed on the display device 22 located outside the bulldozer 4. The notification data generated by the notification data generation unit 58 is transmitted to the remote controller 23.

[0061] The soil discharge point setting unit 59 sets the soil discharge point 72 in the soil discharge area 7. The soil discharge point 72 is the position where the dump truck 2 performs soil discharge work to unload its cargo.

[0062] In this embodiment, the work site 1 is surveyed in advance. Based on the survey data, two-dimensional data representing the two-dimensional shape of the terrain of the work site 1 is calculated. The two-dimensional data represents the two-dimensional shape of the terrain on a predetermined plane substantially parallel to the ground of the work site 1. The two-dimensional data may be defined by latitude and longitude coordinate data. The survey includes detecting the two-dimensional shape of the work site using position sensors, including a GNSS receiver. For example, a survey vehicle equipped with position sensors travels along the edge of the work area, and the two-dimensional data of the work site 1 is calculated based on the detection data from the position sensors. The two-dimensional data of the work site may be calculated by the management device 11 or by a computer separate from the management device 11.

[0063] Furthermore, 3D data showing the 3D shape of the terrain at work site 1 may be calculated based on the survey data. The survey may include detecting the 3D shape of the work site using a 3D sensor. A 3D sensor is a sensor capable of detecting the 3D shape of an object to be detected. Examples of 3D sensors include laser sensors (LiDAR: Light Detection and Ranging) that detect the 3D shape of an object by emitting laser light, and stereo cameras. For example, if an aircraft such as a drone equipped with a 3D sensor flies over work site 1, and the 3D sensor mounted on the drone detects work site 1, 3D data of work site 1 may be calculated based on the detection data from the 3D sensor. The 3D data of the work site may be calculated by the management device 11, or by a computer other than the management device 11.

[0064] The soil removal point setting unit 59 sets soil removal points 72 in the soil removal area 7 of the work site 1 based on the 3D data of the soil removal area 7. The position of the soil removal points 72 is defined in the global coordinate system. The soil removal point setting unit 59 sets multiple soil removal points 72 in the soil removal area 7.

[0065] The driving data generation unit 60 generates driving data indicating the driving conditions of the dump truck 2 in the work area. The driving data generation unit 60 generates driving data for the dump truck 2 in at least the soil discharge area 7. The driving data generation unit 60 generates driving data for the dump truck 2 based on the soil discharge point 72 set by the soil discharge point setting unit 59. The driving conditions for the dump truck 2 include the target position of the dump truck 2, the target direction of the dump truck 2, the target driving speed of the dump truck 2, and the target turning radius of the dump truck 2.

[0066] Figure 7 is a diagram illustrating the driving data according to the first embodiment. Figure 7 shows the driving data set in the soil removal area 7. The driving data defines the driving conditions of the dump truck 2. The driving data for the dump truck 2 includes the driving point 70, the driving path 79, the target position of the dump truck 2, the target direction of the dump truck 2, and the target driving speed of the dump truck 2. The driving data generation unit 60 generates at least the driving path 79 of the dump truck 2.

[0067] Multiple travel points 70 are set in the soil removal area 7. Travel points 70 define the target position of the dump truck 2. The target position of the dump truck 2 is defined in the global coordinate system. For each of the multiple travel points 70, the target bearing and target travel speed of the dump truck 2 are set. The multiple travel points 70 are set at intervals. The intervals between travel points 70 may be uniform or uneven.

[0068] The target position of dump truck 2 refers to the target position of dump truck 2 when it passes through the travel point 70. The target direction of dump truck 2 refers to the target direction of dump truck 2 when it passes through the travel point 70. The target travel speed of dump truck 2 refers to the target travel speed of dump truck 2 when it passes through the travel point 70. The travel path 79 refers to a virtual line indicating the target travel route of dump truck 2. The travel path 79 is defined by a trajectory that passes through multiple travel points 70. The travel data generation unit 60 generates the travel path 79 so as to include the soil discharge point 72.

[0069] In the example shown in Figure 7, of the multiple travel points 70, one travel point 70 is set as an entry point 70S, and one travel point 70 is set as an exit point 70E. In addition, of the multiple travel points 70, at least one travel point 70 is set as a switchback point 71, and at least one travel point 70 is set as an earth removal point 72. The switchback point 71 is the position where the dump truck 2 performs a switchback. A switchback is the action in which the forward-moving dump truck 2 makes a sharp change in direction and moves in reverse.

[0070] The soil removal point 72 may be set near the edge 44 of the soil removal area 7. As described with reference to Figure 4, a mound 45 is formed at the edge 44. The soil removal point 72 may be set at the lower end of the mound 45 on the soil removal area 7 side. The soil removal point 72 may be set on the slope of the mound 45 on the soil removal area 7 side. As described with reference to Figure 4, if the soil removal work is carried out so that a natural bedrock 46 is formed on the ground of the soil removal area 7, the soil removal point 72 may be set at any location in the soil removal area 7 away from the edge 44.

[0071] The location of the travel point 70, which includes the switchback point 71 and the soil removal point 72, and the location of the travel path 79, which indicates the travel route of the dump truck 2, are defined in the global coordinate system.

[0072] The driving data generated in the driving data generation unit 60 is transmitted to the dump truck 2. The controller 17 of the dump truck 2 controls the driving device 15 based on the driving data. The controller 17 controls the driving device 15 so that the dump truck 2 travels according to the driving path 79, based on the detection data from the position sensor 55 and the detection data from the orientation sensor 56. Specifically, the controller 17 controls the driving device 15 so that the deviation between the detected position of the dump truck 2 detected by the position sensor 55 and the target position of the dump truck 2 set at the driving point 70 is small when passing through the driving point 70. The controller 17 controls the driving device 15 so that the deviation between the detected orientation of the dump truck 2 detected by the orientation sensor 56 and the target orientation of the dump truck 2 set at the driving point 70 is small when passing through the driving point 70. The controller 17 controls the driving device 15 so that the dump truck 2 travels at the target driving speed, based on the detection data from the speed sensor 57. In other words, the controller 17 controls the driving device 15 so that the deviation between the detected driving speed of the dump truck 2 detected by the speed sensor 57 when passing the driving point 70 and the target driving speed of the dump truck 2 set at the driving point 70 becomes small.

[0073] In the example shown in Figure 7, dump truck 2 enters the soil removal area 7 from the transport path 8 while moving forward. After entering the soil removal area 7 from the transport path 8, dump truck 2 passes the entry point 70S and then enters the switchback point 71 while moving forward. After entering the switchback point 71, dump truck 2 performs a switchback and then enters the soil removal point 72 while moving backward. After entering the soil removal point 72, dump truck 2 performs soil removal work at the soil removal point 72. With dump truck 2 positioned at the soil removal point 72, the dump body 16 performs a dumping operation, and the load is discharged from dump truck 2 at the soil removal point 72. Having completed the soil removal work, dump truck 2 moves forward towards the exit point 70E to leave the soil removal point 72. After passing the exit point 70E, dump truck 2 leaves the soil removal area 7 and returns to the transport path 8.

[0074] The division area setting unit 61 sets up multiple division areas 73 in the excavation area 7 where the bulldozer 4 and dump truck 2 work. The positions of the division areas 73 are defined in the global coordinate system.

[0075] Figure 8 is a diagram illustrating a divided area 73 according to an embodiment. As shown in Figure 8, the soil removal point setting unit 59 sets a plurality of soil removal points 72 in the soil removal area 7. In the example shown in Figure 8, the plurality of soil removal points 72 are set along the edge 44 of the soil removal area 7. In the example shown in Figure 8, soil removal work by the dump truck 2 has not yet been performed. In the following description, soil removal points 72 where soil removal work by the dump truck 2 has not yet been performed will be appropriately referred to as un-soil-removed points 721, and soil removal points 72 where soil removal work by the dump truck 2 has already been performed will be appropriately referred to as completed points 722.

[0076] The division area setting unit 61 sets multiple division areas 73 in the soil removal area 7. The division area setting unit 61 sets the division areas 73 so that the multiple division areas 73 do not overlap with each other. The division area setting unit 61 sets the division areas 73 so that soil removal points 72 are located inside the division areas 73. The division area setting unit 61 sets the division areas 73 so that multiple soil removal points 72 that are adjacent to each other are located inside the division areas 73.

[0077] In the example shown in Figure 8, the division area setting unit 61 sets three division areas 73 in the soil removal area 7. The division area 73 includes a first division area 731, a second division area 732, and a third division area 733. The first division area 731, the second division area 732, and the third division area 733 are set so as not to overlap with each other. The second division area 732 is set next to one side of the first division area 731, and the third division area 733 is set next to the other side of the first division area 731. In the example shown in Figure 8, the first division area 731 includes six adjacent soil removal points 72. The second division area 732 includes seven adjacent soil removal points 72. The third division area 733 includes five adjacent soil removal points 72.

[0078] In the example shown in Figure 8, the outline of the divided area 73 is a rectangle. However, the outline of the divided area 73 is not limited to a rectangle. The outline of the divided area 73 may be any polygon, such as a hexagon or an octagon. The outline of the divided area 73 may also be a circle or an ellipse.

[0079] The division area setting unit 61 determines the size of the division area 73 based on the size of the soil removal area 7. As described above, the work site 1 is surveyed in advance. The size of the soil removal area 7 is calculated based on the survey data. The division area 73 is set inside the soil removal area 7. The size of the division area 73 is smaller than the size of the soil removal area 7. The division area setting unit 61 sets the division area 73 so that the division area 73 is smaller than the soil removal area 7. The division area setting unit 61 sets the division area 73 so that the first division area 731, the second division area 732, and the third division area 733 do not extend beyond the soil removal area 7.

[0080] Each of the bulldozer 4 and dump truck 2 operates within the divided area 73. The divided area setting unit 61 determines the size of the divided area 73 so that each of the bulldozer 4 and dump truck 2 can operate smoothly within the divided area 73. The divided area setting unit 61 determines the size of the divided area 73 so that when each of the bulldozer 4 and dump truck 2 operates within the divided area 73, they are prevented from extending outside the divided area 73. The divided area setting unit 61 determines the size of the divided area 73 based on the vehicle data of the bulldozer 4 and dump truck 2. The vehicle data includes at least one of the external dimensions of the bulldozer 4 and dump truck 2, the minimum turning radius of the bulldozer 4 and dump truck 2, and the external dimensions of the work equipment of the bulldozer 4.

[0081] The external dimensions of the bulldozer 4 and dump truck 2 are known data derived from the specification data of the bulldozer 4 and dump truck 2. The external dimensions of the bulldozer 4 and dump truck 2 include the external dimensions on a predetermined plane parallel to the ground of the soil removal area 7. If the ground of the soil removal area 7 is substantially parallel to a horizontal plane, the external dimensions of the bulldozer 4 and dump truck 2 include the external dimensions in the horizontal direction. The external dimensions of the bulldozer 4 and dump truck 2 may also include the external dimensions in the height direction perpendicular to the ground of the soil removal area 7. The division area setting unit 61 sets the division area 73 such that the larger the external dimensions of the bulldozer 4 and dump truck 2, the larger the division area 73 becomes.

[0082] The minimum turning radius of a work machine refers to the radius of the circle traced by the outermost part of the work machine when it travels across the ground of the work area with the steering angle at its maximum. The minimum turning radius of the bulldozer 4 and dump truck 2 is known data derived from the specifications data of the bulldozer 4 and dump truck 2. The division area setting unit 61 sets the division area 73 such that the larger the turning radius of the bulldozer 4 and dump truck 2, the larger the division area 73 becomes.

[0083] The working equipment of the bulldozer 4 includes an excavating working equipment 28 and a ripper working equipment 29. The external dimensions of the working equipment of the bulldozer 4 are known data derived from the specifications data of the bulldozer 4. The external dimensions of the working equipment of the bulldozer 4 include the external dimensions on a predetermined plane parallel to the ground of the soil removal area 7. The external dimensions of the working equipment of the bulldozer 4 include the external dimensions in the width direction of the bulldozer 4. The external dimensions of the working equipment of the bulldozer 4 are, for example, the width of the excavating blade 34. The external dimensions of the working equipment of the bulldozer 4 may also include the external dimensions in the height direction perpendicular to the ground of the soil removal area 7. The division area setting unit 61 sets the division area 73 such that the larger the external dimensions of the working equipment of the bulldozer 4, the larger the division area 73 becomes.

[0084] The work area designation unit 62 designates the division area 73 in which the bulldozer 4 will work from among the multiple division areas 73. The work area designation unit 62 also designates the division area 73 in which the dump truck 2 will work from among the multiple division areas 73. The division area 73 in which the bulldozer 4 will work and the division area 73 in which the dump truck 2 will work are different.

[0085] In the following description, the sectioned area 73 designated for bulldozer 4 to work in will be referred to as the bulldozer work area 74 as appropriate. The sectioned area 73 designated for dump truck 2 to work in will be referred to as the dump truck work area 75 as appropriate. In the bulldozer work area 74, bulldozer 4 works, and dump truck 2 does not. In the dump truck work area 75, dump truck 2 works, and bulldozer 4 does not.

[0086] Figure 9 is a diagram illustrating the dump truck operating area 75 according to the first embodiment. The operating area designation unit 62 designates a divided area 73 that includes an un-excavated soil point 721 from among a plurality of divided areas 73 as the dump truck operating area 75. The driving data generation unit 60 generates a driving path 79 so that soil excavation work is performed at the un-excavated soil point 721 in the dump truck operating area 75. The driving data generation unit 60 generates a driving path 79 so that at least a portion of the driving path 79 is located inside the dump truck operating area 75. The dump truck 2 performs soil excavation work at each of the plurality of un-excavated soil points 721 inside the dump truck operating area 75.

[0087] In the example shown in Figure 9, the first divided area 731 is designated as the dump truck operating area 75. The operating area designation unit 62 designates the first divided area 731 as the dump truck operating area 75 while the bulldozer 4 is located outside the first divided area 731.

[0088] After the soil removal work is completed, the controller 17 of the dump truck 2 transmits a soil removal completion signal to the management device 11, indicating that the soil removal work is finished. The soil removal point setting unit 59 can recognize that the unremoved soil points 721 have changed to soil removed soil points 722 by receiving the soil removal completion signal. Figure 9 shows the state in which the soil removal work of the dump truck 2 is progressing in the first divided area 731, and two of the six unremoved soil points 721 in the first divided area 731 have changed to soil removed soil points 722.

[0089] Figure 10 is a diagram illustrating the bulldozer operating area 74 according to the first embodiment. After all of the un-excavated soil points 721 in the first divided area 731, which is designated as the dump truck operating area 75, have changed to soil-excavated soil points 722, the operating area designation unit 62 designates the first divided area 731 as the bulldozer operating area 74. That is, the operating area designation unit 62 designates the first divided area 731, where the soil-excavation work of the dump truck 2 has been completed, as the bulldozer operating area 74 where the bulldozer 4 will work. The operating area designation unit 62 also designates a divided area 73, separate from the first divided area 731, as the dump truck operating area 75 where the dump truck 2 will work. In the example shown in Figure 10, the operating area designation unit 62 designates the second divided area 732, which includes the un-excavated soil points 721, as the dump truck operating area 75.

[0090] The notification data generation unit 58 transmits notification data to the remote controller 23 indicating that the first divided area 731 has been designated as the bulldozer operating area 74. The display control unit 69 displays display data on the display device 22 indicating that the first divided area 731 has been designated as the bulldozer operating area 74.

[0091] The work of bulldozer 4 includes leveling the ground in section 73 after the soil removal work has been completed. The operator operates the remote control device 21 so that bulldozer 4 enters section 731, which has been designated as the bulldozer operating area 74, in order to perform leveling work on the ground in section 731 after the soil removal work by dump truck 2 has been completed.

[0092] The virtual wall setting unit 63 sets a virtual wall 76 in the first divided area 731 (bulldozer operating area 74) when the bulldozer 4 is inside the first divided area 731 (bulldozer operating area 74) and the dump truck 2 is outside the first divided area 731 (bulldozer operating area 74). The virtual wall setting unit 63 sets the virtual wall 76 in the first divided area 731 after the dump truck 2 has left the first divided area 731 and the bulldozer 4 has entered the first divided area 731. The bulldozer 4 performs leveling work in the first divided area 731 inside the virtual wall 76.

[0093] Figure 11 is a diagram illustrating the virtual wall 76 according to the first embodiment. The virtual wall 76 is a virtual wall set at the boundary between the first divided area 731 and the area outside the first divided area 731 in order to restrict the bulldozer 4 from advancing from the first divided area 731 (bulldozer operating area 74) and the dump truck 2 from entering the first divided area 731 (bulldozer operating area 74). As shown in Figure 11, the virtual wall setting unit 63 sets the virtual wall 76 along the outline of the first divided area 731.

[0094] The virtual wall 76 restricts the operating range of the bulldozer 4. The virtual wall 76 is set to surround the bulldozer 4. The virtual wall 76 is set in the global coordinate system. The virtual wall 76 is set to the south, north, east, west, above, and below the bulldozer 4. The virtual wall 76 may be set to at least one of the south, north, east, west, above, and below the bulldozer 4. The virtual wall 76 does not move even if the bulldozer 4 moves within the work site 1. The virtual wall 76 is fixed within the work site 1. The area outside the virtual wall 76 is an avoidance area where the bulldozer 4 should avoid entering. When the bulldozer 4 approaches the virtual wall 76, the operation of at least one of the traveling device 27, the excavating work machine 28, and the ripper work machine 29 is restricted, or an alarm is output in the remote control room 20. In other words, if there is a possibility that at least a part of the bulldozer 4 may go outside the virtual wall 76, the operation of the bulldozer 4 will be restricted or an alarm will be issued to prevent the bulldozer 4 from going outside the virtual wall 76.

[0095] There may be obstacles at the work site 1 that could hinder the operation of the bulldozer 4. Examples of obstacles include dump trucks 2 traveling in the soil removal area 7, structures above the bulldozer 4, cliffs or holes in the work site 1, and buildings. If at least a part of the bulldozer 4 moves outside the virtual wall 76, the work efficiency of the bulldozer 4 may decrease. By setting a virtual wall 76 at the boundary between the bulldozer operating area 74 where the bulldozer 4 works and the avoidance area, the bulldozer 4 is prevented from moving beyond the virtual wall 76 into the avoidance area. This prevents a decrease in the work efficiency of the bulldozer 4. An example of a virtual wall is disclosed in Japanese Patent Application Publication No. 2024-034408.

[0096] Figure 12 is a diagram illustrating the bulldozer operating area 74 and the dump truck operating area 75 according to the first embodiment. As shown in Figure 12, the first divided area 731 is designated as the bulldozer operating area 74, and the second divided area 732 is designated as the dump truck operating area 75. A virtual wall 76 is set in the first divided area 731 (bulldozer operating area 74).

[0097] After a virtual wall 76 is set in the first divided area 731, the driving data generation unit 60 generates a driving path 79 so that soil removal work is performed at the unremoved soil points 721 in the second divided area 732. The driving data generation unit 60 generates the driving path 79 so that at least a portion of the driving path 79 is located inside the second divided area 732 (dump truck operating area 75). The driving data generation unit 60 generates the driving path 79 so that the dump truck 2 does not enter the first divided area 731 (bulldozer operating area 74). The driving data generation unit 60 generates the driving path 79 so that the driving path 79 does not go inside the virtual wall 76. The dump truck 2 performs soil removal work at each of the multiple unremoved soil points 721 inside the second divided area 732 (dump truck operating area 75).

[0098] After the soil removal work is completed, the controller 17 of the dump truck 2 transmits a soil removal completion signal to the management device 11, indicating that the soil removal work is finished. Upon receiving the soil removal completion signal, the soil removal point setting unit 59 can recognize that the unremoved soil point 721 has changed to the removed soil point 722.

[0099] After completing the leveling work in the first divided area 731, the operator of bulldozer 4 operates the input device 53 to notify the control device 11 that the leveling work in the first divided area 731 has been completed. When the input device 53 is operated, input data is generated indicating that the leveling work of bulldozer 4 has been completed. The input data transmission unit 66 of the remote controller 23 transmits the input data generated by the input device 53 to the control device 11.

[0100] The virtual wall setting unit 63 acquires input data indicating that the bulldozer 4 has finished leveling the ground in the first divided area 731. After acquiring the input data indicating that the bulldozer 4 has finished leveling the ground, the virtual wall setting unit 63 determines that the dump truck 2 has finished its soil removal work in the second divided area 732 and has left the second divided area 732, and then cancels the setting of the virtual wall 76. The virtual wall setting unit 63 can determine whether the soil removal work of the dump truck 2 in the second divided area 732 has been completed based on the soil removal completion signal from the controller 17 of the dump truck 2. The virtual wall setting unit 63 can determine whether the dump truck 2 has left the second divided area 732 based on the detection data from the position sensor 55 of the dump truck 2.

[0101] Figure 13 is a diagram illustrating the bulldozer operating area 74 and the dump truck operating area 75 according to the first embodiment. As shown in Figure 13, the virtual wall setting unit 63 releases the setting of the virtual wall 76 in the first divided area 731 after the bulldozer 4 has finished leveling the ground in the first divided area 731, the dump truck 2 has finished removing soil in the second divided area 732, and the dump truck 2 has left the second divided area 732. By releasing the setting of the virtual wall 76, the bulldozer 4 can move outside the first divided area 731.

[0102] After the setting of the virtual wall 76 is released, the operating area designation unit 62 designates the second divided area 732 as the bulldozer operating area 74 where the bulldozer 4 will perform leveling work. The operating area designation unit 62 designates a divided area 73 from a different divided area 73 than the first divided area 731 and the second divided area 732 where the dump truck 2 will perform soil removal work. In this embodiment, the third divided area 733 is designated as the dump truck operating area 75.

[0103] The notification data generation unit 58 sends notification data to the remote controller 23 indicating that the setting of the virtual wall 76 in the first divided area 731 has been canceled and that the second divided area 732 has been designated as the bulldozer operating area 74. The display control unit 69 causes the display device 22 to display display data indicating that the setting of the virtual wall 76 in the first divided area 731 has been canceled and that the second divided area 732 has been designated as the bulldozer operating area 74.

[0104] Figure 14 shows a display device 22 according to the first embodiment. As shown in Figure 14, the display control unit 69 causes the display device 22 to display the captured image 80 of the work site 1 captured by the imaging device 32. In the example shown in Figure 14, the captured image 80 displayed on the display device 22 includes a part of the body 26 of the bulldozer 4, the soil removal area 7, and the edge 44 of the soil removal area 7.

[0105] The display control unit 69 generates a reference image 82 showing at least one of the dimensions of the bulldozer 4 and the distance from the bulldozer 4. In one embodiment, the display control unit 69 generates a reference image 82 showing the width of the excavation blade 34 and the distance from the bulldozer 4. The display control unit 69 can generate the reference image 82 based, for example, on the specifications data of the bulldozer 4. The specifications data of the bulldozer 4 is known data.

[0106] The reference image 82 is superimposed on the captured image 80. The reference image 82 shows the width of the excavation blade 34 and the distance from the bulldozer 4. The reference image 82 includes a blade reference line 821 showing the width of the excavation blade 34 and a distance reference line 822 showing the distance from the bulldozer 4. The blade reference line 821 is displayed extending forward from the left and right ends of the excavation blade 34, respectively. The left blade reference line 821 and the right blade reference line 821 are parallel. The distance reference line 822 is displayed connecting the front end of the left blade reference line 821 and the front end of the right blade reference line 821. The distance reference line 822 is displayed at a specified distance (e.g., 20m) forward from the front end of the vehicle body 26.

[0107] In this embodiment, the display control unit 69 causes the display device 22 to display a teaching image 90 relating to the bulldozer operating area 74 of the bulldozer 4. The display control unit 69 displays the teaching image 90 together with the captured image 80 on the display device 22. As shown in Figure 14, the display control unit 69 superimposes the teaching image 90 onto the captured image 80.

[0108] As explained with reference to Figure 13, after completing the leveling work in the first divided area 731, the bulldozer 4 moves outside the first divided area 731. The display control unit 69 causes the teaching image 90 to be displayed on the display device 22 when the bulldozer 4 moves outside at least the first divided area 731 (bulldozer operating area 74).

[0109] The control device 11 manages multiple bulldozer operating areas 74 of the bulldozer 4. As explained with reference to Figure 13, after the bulldozer 4 has finished leveling the ground in the first divided area 731, the operating area designation unit 62 designates the second divided area 732 as the bulldozer operating area 74 where the bulldozer 4 will perform leveling work. The bulldozer 4 then performs leveling work in the second divided area 732 after the first divided area 731.

[0110] The operating area data acquisition unit 68 acquires operating area data from the management device 11 indicating the bulldozer operating area 74 in which the bulldozer 4 will perform land leveling work next after completing land leveling work in the first divided area 731. The display control unit 69 causes the display device 22 to display a teaching image 90 indicating that the bulldozer operating area 74 in which land leveling work will be performed next after the first divided area 731 is the second divided area 732. The teaching image 90 is a guidance image that teaches the divided area 73 in which the bulldozer 4 will perform land leveling work next.

[0111] In the example shown in Figure 14, the bulldozer 4, having completed the leveling work in the first divided area 731, is located in the first divided area 731. When the bulldozer 4 completes the leveling work in the first divided area 731 (bulldozer operating area 74), the display control unit 69 causes the display device 22 to display a teaching image 90 indicating the second divided area 732, where the leveling work will be performed next.

[0112] The teaching image 90 includes a position image 91 that shows the location of the second divided area 732. The position image 91 is a graphic image that shows a specific location in the second divided area 732. The specific location is, for example, the center of the second divided area 732. In the example shown in Figure 14, the position image 91 is a star-shaped image displayed at the center of the second divided area 732. In the example shown in Figure 14, the display control unit 69 causes the display device 22 to display an excavation point image 72V that shows the excavation point 72 of the second divided area 732. The specific location where the position image 91 is displayed may be the excavation point 72 or a vicinity of the excavation point 72.

[0113] The teaching image 90 includes a compass image 92 indicating the direction from the bulldozer in the first divided area 731 to the second divided area 732. The compass image 92 is a graphic image indicating the direction of travel of the bulldozer 4. In the example shown in Figure 14, the compass image 92 is an arrow image extending from the front of the bulldozer 4's body 26 toward the center of the second divided area 732.

[0114] The display device 22 shows a teaching image 90, allowing the operator to recognize the bulldozer operating area 74 where work should be performed next. The operator operates the remote control device 21 to have the bulldozer 4 enter the second divided area 732, which has been designated as the bulldozer operating area 74, in order to perform leveling work to flatten the ground in the second divided area 732 after the soil removal work has been completed.

[0115] After the bulldozer 4 enters the second divided area 732, the display control unit 69 terminates the display of the teaching image 90.

[0116] The display control unit 69 displays the teaching image 90 on the display device 22 at a predetermined display timing. When the display control unit 69 guides the bulldozer 4 to move from the first divided area 731 to the second divided area 732, it may display the teaching image 90 on the display device 22 when the leveling work of the bulldozer 4 in the first divided area 731 is completed. As described above, the operator operates the input device 53 after the leveling work in the first divided area 731 is completed. The display control unit 69 may display the teaching image 90 on the display device 22 when input data is generated by the operation of the input device 53. The display control unit 69 may also display the teaching image 90 on the display device 22 before the second divided area 732 is designated as the bulldozer operating area 74. The display control unit 69 may display the teaching image 90 on the display device 22 when the second divided area 732 is designated as the bulldozer operating area 74 and the leveling work of the bulldozer 4 in the first divided area 731 is completed.

[0117] The display control unit 69 may display the teaching image 90 on the display device 22 when the leveling work of the bulldozer 4 in the first divided area 731 is completed and the bulldozer 4 is in a state where it can move from the first divided area 731 to the second divided area 732. The state in which the bulldozer 4 is in a state where it can move from the first divided area 731 to the second divided area 732 includes the state in which the soil removal work of the dump truck 2 in the second divided area 732 is completed.

[0118] The display control unit 69 may display the teaching image 90 on the display device 22 after the bulldozer 4 has finished leveling the ground in the first divided area 731 and the bulldozer 4 has moved outside the first divided area 731.

[0119] As explained with reference to Figure 13, in this embodiment, the state in which the bulldozer 4 can move from the first divided area 731 to the second divided area 732 includes the state in which the bulldozer 4 has finished leveling the ground in the first divided area 731, the dump truck 2 has finished removing soil in the second divided area 732, the dump truck 2 has left the second divided area 732, and the setting of the virtual wall 76 in the first divided area 731 has been released.

[0120] <Display method> Figure 15 is a flowchart illustrating the display method of the captured image 80 and the teaching image 90 according to the first embodiment. The image acquisition unit 67 acquires the captured image 80 of the work site 1 captured by the imaging device 32 (step S1). The display control unit 69 causes the captured image 80 to be displayed on the display device 22 (step S2).

[0121] The operating area data acquisition unit 68 determines whether the second divided area 732 has been designated as the bulldozer operating area 74 (step S3).

[0122] When the leveling work of the bulldozer 4 in the first divided area 731 is completed, the operating area designation unit 62 of the control device 11 designates the second divided area 732 as the bulldozer operating area 74. The notification data generation unit 58 sends notification data to the remote controller 23 indicating that the second divided area 732 has been designated as the bulldozer operating area 74. The operating area data acquisition unit 68 can determine whether or not the second divided area 732 has been designated as the bulldozer operating area 74 based on the notification data from the control device 11.

[0123] In step S3, if it is determined that the second divided area 732 has been designated as the bulldozer operating area 74 (step S3: Yes), the operating area data acquisition unit 68 acquires operating area data indicating that the bulldozer operating area 74, which will be leveled after the first divided area 731, is the second divided area 732 (step S4).

[0124] The display control unit 69 determines whether it is the display timing to display the teaching image 90 after the operating area data acquisition unit 68 has acquired the operating area data (step S5).

[0125] In step S5, if it is determined that it is time to display the teaching image 90 (step S5: Yes), the display control unit 69 superimposes the teaching image 90 onto the captured image 80 (step S6).

[0126] In step S3, if it is determined that the bulldozer operating area 74 is not specified (step S3: No), the display control unit 69 displays the captured image 80 on the display device 22 and does not display the teaching image 90. In step S5, if it is determined that it is not the timing to display the teaching image 90 (step S5: No), the display control unit 69 displays the captured image 80 on the display device 22 and does not display the teaching image 90.

[0127] The display control unit 69 determines whether or not to terminate the display of the captured image 80 on the display device 22 (step S7). If it is determined in step S7 to continue displaying the captured image 80 (step S7: No), the process returns to step S1. If it is determined in step S7 to terminate the display of the captured image 80 (step S7: Yes), the display of the captured image 80 is terminated.

[0128] <Effects> As described above, the processor 47 of the remote controller 23 includes an image acquisition unit 67 that acquires an image 80 showing an image of the work site 1 of the remotely operated bulldozer 4 captured by the imaging device 32, and a display control unit 69 that causes a teaching image 90 relating to the bulldozer operating area 74, which is the operating area of ​​the bulldozer 4 designated by the management device 11 that manages the work site 1, to be displayed together with the image 80 on a display device 22 located outside the bulldozer 4.

[0129] According to this embodiment, the teaching image 90 relating to the bulldozer operating area 74 is displayed on the display device 22 together with the captured image 80, so that the operator in the remote control room 20 can recognize the bulldozer operating area 74. As a result, a decrease in the work efficiency of the bulldozer 4 is suppressed.

[0130] The control device 11 manages multiple divided areas 73 that may be designated as the bulldozer operating area 74. The display control unit 69 displays a teaching image 90 that shows the second divided area 732, which will be designated as the bulldozer operating area 74 after the first divided area 731. By checking the teaching image 90 displayed on the display device 22, the operator can recognize the second divided area 732, which should be leveled after the first divided area 731.

[0131] The teaching image 90 includes a position image 91 indicating the location of the second division area 732, which is designated as the bulldozer operating area 74 after the first division area 731. The operator can recognize the location of the second division area 732 by checking the position image 91 displayed on the display device 22. Since the position image 91 is superimposed on the captured image 80, the operator can recognize the relative position between the bulldozer 4 and the second division area 732 based on the relative positions of the captured image 80 and the position image 91 of the bulldozer 4 displayed on the display device 22.

[0132] The teaching image 90 includes a compass image 92 that shows the direction from the bulldozer 4 located in the first divided area 731 to the second divided area 732, which is designated as the bulldozer operating area 74 after the first divided area 731. By checking the compass image 92 displayed on the display device 22, the operator can recognize the direction in which the bulldozer 4 should move.

[0133] The display control unit 69 starts displaying the teaching image 90 at any display timing and ends displaying the teaching image 90 at any end timing. For example, the display control unit 69 starts displaying the teaching image 90 when the bulldozer 4 finishes work in the first divided area and ends displaying the teaching image 90 after the bulldozer 4 enters the second divided area 732. Since the teaching image 90 is displayed only for the period for which it is necessary, the operator is less likely to feel annoyed.

[0134] The display control unit 69 may also display the teaching image 90 when the bulldozer 4 has finished its work in the first divided area 731 and has transitioned to a state where the bulldozer 4 can move from the first divided area 731 to the second divided area 732. When the bulldozer 4 is in the first divided area 731, the dump truck 2 performs soil removal work in the second divided area 732. The state in which the bulldozer 4 can move from the first divided area 731 to the second divided area 732 may include the state in which the soil removal work of the dump truck 2 in the second divided area 732 has been completed. After the soil removal work of the dump truck 2 is completed and the dump truck 2 has left the second divided area 732, the operator has the bulldozer 4 enter the second divided area 732 based on the teaching image 90 displayed on the display device 22. This prevents contact between the dump truck 2 and the bulldozer 4.

[0135] The work of bulldozer 4 in the first divided area 731 includes leveling the ground in the first divided area 731. The ground in the soil removal area 7 is leveled by bulldozer 4.

[0136] [Second Embodiment] A second embodiment will now be described. In the following description, components that are the same as or equivalent to those in the first embodiment described above will be denoted by the same reference numerals, and the descriptions of those components will be simplified or omitted.

[0137] Figure 16 shows a display device 22 according to the second embodiment. As shown in Figure 16, leveling work is performed by a bulldozer 4 in the first divided area 731. In the example shown in Figure 16, the third divided area 733 is designated as the bulldozer operating area 74 after the first divided area 731. The display control unit 69 causes the display device 22 to display a position image 91 indicating the position of the third divided area 733.

[0138] In this embodiment, the teaching image 90 includes a path image 93 showing the travel path from the bulldozer 4 located in the first divided area 731 to the third divided area 733.

[0139] As described above, 3D data of the work site 1 is calculated in advance based on survey data. The management device 11 manages the 3D data of the work site 1. The operating area data acquisition unit 68 can generate a travel path for the bulldozer 4 from the first divided area 731 to the third divided area 733 based on the 3D data of the soil removal area 7. The operating area data acquisition unit 68 can generate the travel path for the bulldozer 4 based on an existing pathfinding algorithm, such as the A* search algorithm. The display control unit 69 generates a path image 93 along the travel path of the bulldozer 4. The path image 93 is a line-shaped image along the travel path of the bulldozer 4. The display control unit 69 overlays the path image 93 on the captured image 80. The path image 93 is displayed on the display device 22 so as to connect the body 26 of the bulldozer 4 and the position image 91 in the captured image 80.

[0140] The control device 11 manages the location of obstacles 77 that obstruct the movement of the bulldozer 4 at the work site 1. The operating area data acquisition unit 68 generates a travel path for the bulldozer 4 from the bulldozer 4 in the first divided area 731 to the third divided area 733, based on the three-dimensional data of the excavation area 7 including the obstacles 77. The display control unit 69 generates a path image 93 that bypasses the obstacles 77. The path image 93 is displayed on the display device 22 so as to bypass the obstacles 77 in the captured image 80.

[0141] As described above, according to this embodiment, the path image 93 is displayed on the display device 22 together with the captured image 80, so that the operator in the remote control room 20 can recognize the travel path of the bulldozer 4. Therefore, a decrease in the work efficiency of the bulldozer 4 is suppressed.

[0142] [Third Embodiment] A third embodiment will now be described. In the following description, components that are the same as or equivalent to those in the first embodiment described above will be denoted by the same reference numerals, and the descriptions of those components will be simplified or omitted.

[0143] Figure 17 shows a display device 22 according to the third embodiment. As shown in Figure 17, the display control unit 69 may display an overhead image 7V of the soil removal area 7 on the display device 22. As described above, the work site 1 is surveyed and 3D data of the work site 1 is calculated. The notification data generation unit 58 can generate an overhead image 7V of the soil removal area 7 based on the 3D data of the work site 1. The display control unit 69 displays the overhead image 7V generated by the notification data generation unit 58 on the display device 22.

[0144] The display control unit 69 causes the display device 22 to display a bulldozer image 4V representing the model of the bulldozer 4. The bulldozer image 4V is an image that mimics the bulldozer 4 as seen from above. When the bulldozer 4 travels in the excavation area 7, the bulldozer image 4V moves on the display screen of the display device 22 in accordance with the movement of the bulldozer 4. The display control unit 69 can move the bulldozer image 4V on the display screen of the display device 22 based on the detection data from the position sensor 30 and attitude sensor 31 of the bulldozer 4.

[0145] The display control unit 69 causes the display device 22 to display a first line image 731V showing the outline of the first divided area 731, a second line image 732V showing the outline of the second divided area 732, and a third line image 733V showing the outline of the third divided area 733.

[0146] The display control unit 69 causes the display device 22 to display a bulldozer operating area image 74V, which shows the bulldozer operating area 74, and a dump truck operating area image 75V, which shows the dump truck operating area 75. In the example shown in Figure 17, the bulldozer operating area image 74V is a pattern image that fills in the inside of the first line image 731V and the inside of the second line image 732V, respectively. The dump truck operating area image 75V is a pattern image that fills in the inside of the third line image 733V. The display format of the bulldozer operating area image 74V and the display format of the dump truck operating area image 75V are different. By checking the display device 22, the operator can recognize that the first divided area 731 and the second divided area 732 have been designated as the bulldozer operating area 74, and that the third divided area 733 has been designated as the dump truck operating area 75.

[0147] The display control unit 69 causes the display device 22 to display an un-excavated point image 721V indicating an un-excavated point 721 and an excavated point image 722V indicating an excavated point 722. The display format of the un-excavated point image 721V and the excavated point image 722V are different. As described above, after the excavation work of the dump truck 2 is completed, the controller 17 of the dump truck 2 transmits an excavation completion signal to the management device 11 indicating that the excavation work is completed. Upon receiving the excavation completion signal, the notification data generation unit 58 can generate display data to be displayed on the display device 22 so that the un-excavated points 721 and the excavated points 722 are displayed in different formats. By checking the display data displayed on the display device 22, the operator can confirm the location and number of un-excavated points 721 and the location and number of excavated points 722.

[0148] The display control unit 69 causes the display device 22 to display a teaching image 90 relating to the second division area 732, which is designated as the bulldozer operating area 74 after the first division area 731, as a teaching image 90. This teaching image 90 includes a position image 91 indicating the location of the second division area 732 and an orientation image 92 indicating the direction from the bulldozer 4 in the first division area 731 to the second division area 732. The position image 91 is displayed at the center of the rectangular second line image 732V. The orientation image 92 is displayed on the display device 22 so as to extend from the first line image 731V, which represents the first division area 731, to the second line image 732V, which represents the second division area 732.

[0149] As described above, in this embodiment as well, the teaching image 90 is displayed on the display device 22 together with the overhead image 7V, so that the operator in the remote control room 20 can recognize the bulldozer operating area 74. Therefore, a decrease in the work efficiency of the bulldozer 4 is suppressed.

[0150] [Fourth Embodiment] A fourth embodiment will now be described. In the following description, components that are the same as or equivalent to those in the first embodiment described above will be denoted by the same reference numerals, and the descriptions of those components will be simplified or omitted.

[0151] Figure 18 shows a display device 22 according to the fourth embodiment. In this embodiment, a plurality of display devices 22 are arranged side by side in the remote control room 20. In the example shown in Figure 18, the display device 22 includes a first display device 22A and a second display device 22B arranged next to the first display device 22A.

[0152] In this embodiment, the display control unit 69 causes the first display device 22A to display the captured image 80 and the second display device 22B to display the teaching image 90. In the example shown in Figure 18, the captured image 80 displayed on the first display device 22A includes a part of the body 26 of the bulldozer 4, the dump truck 2, the soil discharge area 7, and the edge 44 of the soil discharge area 7. The teaching image 90 displayed on the second display device 22B includes a position image 91 and an orientation image 92. The second display device 22B displays an overhead view of the soil discharge area 7. The teaching image 90 is superimposed on the overhead view of the soil discharge area 7.

[0153] [Fifth Embodiment] A fifth embodiment will now be described. In the following description, components that are the same as or equivalent to those in the first embodiment described above will be denoted by the same reference numerals, and the descriptions of those components will be simplified or omitted.

[0154] Figure 19 shows a display device 22 according to the fifth embodiment. As shown in Figure 19, the display control unit 69 may generate a window image in which the teaching image 90 is superimposed on an overhead view image of the soil removal area 7, and then superimpose the generated window image onto the captured image 80. Superimposition display refers to a display method in which a window image, which is a secondary screen, is superimposed on a part of the captured image 80, which is the main screen.

[0155] [Sixth Embodiment] A sixth embodiment will now be described. In the following description, components that are the same as or equivalent to those in the first embodiment described above will be denoted by the same reference numerals, and the descriptions of those components will be simplified or omitted.

[0156] Figure 20 is a schematic diagram showing the soil removal area 7 according to the sixth embodiment. As shown in Figure 20, the dump truck 2 may perform soil removal work such that multiple piles 46 are formed on the ground of the soil removal area 7. The multiple piles 46 are formed regularly at intervals in the soil removal area 7. The bulldozer 4 can perform leveling work to smooth the multiple piles 46 formed on the ground of the soil removal area 7. Even when multiple piles 46 are formed on the ground of the soil removal area 7, the display control unit 69 can display the teaching image 90 relating to the bulldozer operating area 74 together with the captured image 80 on the display device 22.

[0157] [Other embodiments] In the embodiment described above, the reference image 82 is superimposed on the captured image 80. The reference image 82 does not need to be displayed.

[0158] In the above embodiment, a virtual wall 76 is set in the bulldozer operating area 74. The virtual wall 76 does not need to be set.

[0159] In the above embodiment, the reference image 82 does not need to be displayed on the display device 22.

[0160] In the embodiment described above, the remotely controlled work machine is a bulldozer 4. The remotely controlled work machine may be other work machines such as an excavator, wheel loader, and motor grader.

[0161] In the above-described embodiment, at least a portion of the functional parts of the remote controller 23 may be provided in the management device 11. At least a portion of the functional parts of the management device 11 may be provided in the remote controller 23. For example, at least one of the notification data generation unit 58, the soil discharge point setting unit 59, the travel data generation unit 60, the divided area setting unit 61, the operating area designation unit 62, and the virtual wall setting unit 63 may be provided in the remote controller 23.

[0162] In the embodiments described above, each of the functional units of the management device 11 may be configured by a separate computer (hardware). Each of the multiple functional units of the remote controller 23 may be configured by a separate computer (hardware). The multiple functional units of the management device 11 and the multiple functional units of the remote controller 23 may be configured by a single computer (hardware). [Explanation of symbols]

[0163] 1...Work site, 2...Dump truck, 3...Excavator, 4...Bulldozer (working machine), 4V...Bulldozer image, 5...Motor grader, 6...Loading area, 7...Soil removal area, 7V...Overhead view image, 8...Transport route, 10...Management system, 11...Management device, 12...Communication system, 12A...Wireless communication device, 12B...Wireless communication device, 12C...Wireless communication device, 13...Control facility, 14...Vehicle body, 15...Running gear, 16...Dump body, 17...Controller, 18...Controller, 19...Remote control system, 20...Remote control room, 21...Remote control device, 22...Display device, 22A...First display device 22B...Second display device, 23...Remote controller, 24...Driver's seat, 25...Communication system, 26...Vehicle body, 27...Running gear, 28...Excavating work equipment, 29...Ripper work equipment, 30...Position sensor, 31...Attitude sensor, 32...Imaging device, 33...Track, 34...Excavating blade, 34A...Cutting edge, 35...Lift frame, 36...Tilt cylinder, 37...Lift cylinder, 38...Shank, 38A...Ripper point, 39...Ripper arm, 40...Tilt cylinder, 41...Lift cylinder, 42...Beam, 43...Downhill cliff, 44...Edge, 45...Embankment, 46...Ground, 47...Pro 48...Main memory, 49...Storage, 50...Input / output interface, 51...Communication interface, 52...Computer program, 53...Input device, 55...Position sensor, 56...Direction sensor, 57...Speed ​​sensor, 58...Notification data generation unit, 59...Soil removal point setting unit, 60...Travel data generation unit, 61...Division area setting unit, 62...Operation area specification unit, 63...Virtual wall setting unit, 65...Operation signal transmission unit, 66...Input data transmission unit, 67...Image acquisition unit, 68...Operation area data acquisition unit, 69...Display control unit, 70...Travel point, 70E...Departure point, 70S...Move Entry point, 71... Switchback point, 72... Soil removal point, 72V... Soil removal point image, 73... Divided area, 74... Bulldozer operating area, 74V... Bulldozer operating area image, 75... Dump truck operating area, 75V... Dump truck operating area image, 76... Virtual wall, 79... Driving path, 80... Captured image, 82... Reference image, 90... Teaching image, 91... Position image, 92... Orientation image, 93... Path image, 721... Unremoved point, 721V... Unremoved point image, 722... Soil removal completed point, 722V... Soil removal completed point image, 731... First divided area, 731V... First line image, 732... Second divided area,732V…Second line image, 733…Third divided area, 733V…Third line image, 821…Blade guideline, 822…Distance guideline.

Claims

1. Equipped with a processor, The aforementioned processor, The imaging device acquires images showing the work site of a remotely operated work machine, The management device for managing the work site displays the teaching image relating to the operating area of ​​the work machine, as specified by the management device, together with the captured image, on a display device located outside the work machine. Remote control system for industrial machinery.

2. The aforementioned management device manages multiple operating areas, The aforementioned processor, The machine, having completed work in the first operating area, displays the teaching image indicating the second operating area where it will next perform work. A remote control system for a work machine according to claim 1.

3. The teaching image includes a position image showing the location of the second operating area. A remote control system for a work machine according to claim 2.

4. The teaching image includes an orientation image showing the direction from the work machine located in the first operating area to the second operating area. A remote control system for a work machine according to claim 2.

5. The teaching image includes a path image showing the travel path from the work machine located in the first operating area to the second operating area. A remote control system for a work machine according to claim 2.

6. The management device manages the location of obstacles present at the work site that obstruct the movement of the work machine, The aforementioned path image is generated to bypass the obstacle, A remote control system for a work machine according to claim 5.

7. The aforementioned processor, When the work of the work machine in the first operating area is completed, the teaching image is displayed. A remote control system for a work machine according to claim 2.

8. The aforementioned processor, The teaching image is displayed when the work of the machine in the first operating area is completed and the machine is in a state where it can move to the second operating area. A remote control system for a work machine according to claim 2.

9. In the aforementioned second operating area, an unmanned vehicle performs soil removal work. The state in which the work machine can be moved to the second operating area includes the state in which the soil removal work of the unmanned vehicle in the second operating area has been completed. A remote control system for a work machine according to claim 8.

10. The work of the work machine in the first working area includes either or both of the following: embankment formation work to form an embankment in the first working area and leveling work to level the ground in the first working area. A remote control system for a work machine according to claim 9.

11. The aforementioned processor, The teaching image is displayed after the work of the machine in the first operating area is completed and the machine has moved outside the first operating area. A remote control system for a work machine according to claim 2.

12. The aforementioned control device is Multiple divided areas are set up in the aforementioned work site, Select the operating area from a plurality of the aforementioned divided areas. A remote control system for a work machine according to claim 1.

13. The aforementioned processor, The teaching image is superimposed onto the captured image. A remote control system for a work machine according to claim 1.

14. In a remote control room where a remote control device for remotely operating the aforementioned work machine is located, a plurality of the aforementioned display devices are arranged side by side. The aforementioned processor, The captured image is displayed on the first display device, and the teaching image is displayed on the second display device. A remote control system for a work machine according to claim 1.

15. To acquire images showing the work site of a remotely operated work machine captured by an imaging device, This includes displaying, together with the captured image, a teaching image relating to the operating area of ​​the work machine, as specified by the management device for managing the work site, on a display device located outside the work machine, A method for remotely controlling industrial machinery.

16. The aforementioned management device manages multiple operating areas, The machine, having completed work in the first operating area, displays the teaching image indicating the second operating area where it will next perform work. A method for remotely operating a work machine according to claim 15.

17. The teaching image includes a position image showing the location of the second operating area. A method for remotely operating a work machine according to claim 16.

18. The teaching image includes an orientation image showing the direction from the work machine located in the first operating area to the second operating area. A method for remotely operating a work machine according to claim 16.

19. The teaching image includes a path image showing the travel path from the work machine located in the first operating area to the second operating area. A method for remotely operating a work machine according to claim 16.

20. The management device manages the location of obstacles present at the work site that obstruct the movement of the work machine, The aforementioned path image is generated to bypass the obstacle, A method for remotely operating a work machine according to claim 19.

Images