Remote operation assistance system, remote operation device for work vehicle, and remote operation assistance method

Abstract

This remote operation assistance system assists remote operation of a work vehicle. The work vehicle loads excavated material excavated by a bucket of a work machine into a loading container having a bottom part and a side wall part extending upward from the bottom part. The remote operation assistance system acquires orientation information indicating the orientation of the work machine. The remote operation assistance system acquires position information indicating the three-dimensional relative position of the side wall part with respect to the work vehicle. On the basis of the orientation information and the position information, the remote operation assistance system displays, on a display installed at a location where the remote operation is performed, an image indicating the relative positional relationship between the bucket and the side wall part when the work vehicle and the loading container are viewed from the side of the work machine.

Description

Remote operation support system, remote operation device for work vehicle, and remote operation support method

[0001] The present disclosure relates to a remote operation support system, a remote operation device for a work vehicle, and a remote operation support method.

[0002] The operator of the work vehicle loads the material excavated by the work machine into a loading container (for example, the vessel of a dump truck). As an example of a system for supporting such an operation, U.S. Patent Application Publication No. 2022 / 0127816A1 (Patent Document 1) discloses a container loading support system for a work vehicle.

[0003] The container loading assistance system assists an operator operating a work vehicle from within the cab of the work vehicle. In the container loading assistance system, a controller drives the work vehicle when a command to move the work vehicle is received. The controller determines the distance to the container, the ground speed, and the position of the boom or implement. When a command to raise the boom is received, the controller raises the boom while traveling. The controller determines whether the distal end reaches a threshold height before the work vehicle reaches the container. The controller activates an indicator if the distal end does not reach the threshold height within that time.

[0004] US Patent Application Publication No. 2022 / 0127816A1

[0005] When an operator operates a work vehicle remotely, it is more difficult for the operator to grasp the relative positional relationship between the bucket of the work machine and the wall of the loading container, such as the side panel of the vessel, compared to when the operator operates the work vehicle from within the cab of the work vehicle as in Patent Document 1.

[0006] The present disclosure provides a remote operation support system, a remote operation device for a work vehicle, and a remote operation support method that make it easy to grasp the relative positional relationship between a bucket and a wall portion of a loading container when remotely operating a work vehicle.

[0007] A remote operation assistance system according to one aspect of the present disclosure assists in remote operation of a work vehicle. The work vehicle loads material excavated by a bucket of a work implement into a loading container having a bottom and a sidewall extending upward from the bottom. The remote operation assistance system acquires attitude information indicating the attitude of the work implement. The remote operation assistance system acquires first position information indicating the three-dimensional position of the sidewall relative to the work vehicle. Based on the attitude information and the first position information, the remote operation assistance system displays, on a display installed at a location where remote operation is performed, a first image indicating the relative positional relationship between the bucket and the sidewall when the work vehicle and the loading container are viewed from the side of the work implement,

[0008] According to the above configuration, when the work vehicle is remotely operated, it is easy to grasp the relative positional relationship between the bucket and the wall of the loading container.

[0009] 10 is a diagram showing the overall configuration of a remote control system. It is a schematic diagram showing the travel route of a wheel loader and the configuration of a dump truck. It is a block diagram showing the general configuration of a control system for a wheel loader. It is a functional block diagram showing the functional configuration of a support system. It is a diagram showing an example of an image displayed on the display of a remote control device when the distance from the reference position of the wheel loader to the side plate is longer than a threshold. It is a diagram showing an example of an image displayed on the display of a remote control device when the distance from the reference position of the wheel loader to the side plate is equal to or shorter than a threshold. It is a diagram showing another example of an image displayed on the display of a remote control device when the distance from the reference position of the wheel loader to the side plate is equal to or shorter than a threshold. It is a flow chart showing the flow of processing executed in the support system. It is a flow chart showing the flow of processing executed in the support system after the processing shown in FIG.

[0010] Hereinafter, the embodiments will be described with reference to the drawings. In the following description, the same parts and components are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed descriptions thereof will not be repeated. It is also intended from the beginning that any configuration may be extracted from the embodiments and arbitrarily combined.

[0011] In this embodiment, a wheel loader will be described as an example of a work vehicle. However, the work vehicle is not limited to a wheel loader. The work vehicle may be any vehicle equipped with a bucket. The work vehicle is not limited to a vehicle that travels on wheels. The work vehicle may be a vehicle that travels on tracks, such as a hydraulic excavator.

[0012] <A. Overall Configuration of Remote Control System 1000> Fig. 1 is a diagram showing the overall configuration of a remote control system 1000. As shown in Fig. 1, the remote control system 1000 includes a wheel loader 1 and a remote control device 200.

[0013] (Wheel Loader 1) The body of the wheel loader 1 mainly comprises a body frame 2, a work implement 3, a traveling device 4, and a cab 5. The work implement 3, the traveling device 4, and the cab 5 are attached to the body frame 2 of the wheel loader 1.

[0014] The traveling device 4 causes the vehicle body of the wheel loader 1 to travel. The traveling device 4 includes traveling wheels 4 a, 4 b. The wheel loader 1 can be self-propelled by driving the traveling wheels 4 a, 4 b to rotate. The wheel loader 1 can perform desired work using the work implement 3.

[0015] In this specification, the direction in which the wheel loader 1 travels straight ahead is referred to as the fore-and-aft direction of the wheel loader 1. In the fore-and-aft direction of the wheel loader 1, the side on which the work implement 3 is arranged relative to the body frame 2 is referred to as the front direction. The side opposite the front direction is referred to as the rear direction. The left-and-right direction of the wheel loader 1 is the direction perpendicular to the fore-and-aft direction when the wheel loader 1 is viewed from above on flat ground. The right and left sides of the left-and-right direction when looking forward are the right direction and the left direction, respectively. The up-and-down direction of the wheel loader 1 is the direction perpendicular to the plane defined by the fore-and-aft direction and the left-and-right direction. In the up-and-down direction, the side with ground is the bottom side, and the side with sky is the top side.

[0016] The vehicle body frame 2 includes a front frame 2a and a rear frame 2b. The front frame 2a is disposed in front of the rear frame 2b. The front frame 2a and the rear frame 2b are attached to each other by a center pin 10 so that they can move laterally relative to each other. In this example, the center pin 10 includes two center pins 10A and 10B. The center pin 10A is located above (directly above) the center pin 10B.

[0017] In the remote operation system 1000, the position of the center pin 10 (for example, a predetermined position on the axis Ax of the center pin 10A) is set as the reference position of the wheel loader 1 when remote control is performed. However, the reference position is not limited to this, and may be any specific position on the wheel loader 1. It is preferable that the reference position is a position on the wheel loader 1 that is different from the position of the work implement 3, such as the position of the center pin 10. In other words, it is preferable that the reference position is a position on the wheel loader 1 other than the work implement 3. For example, the reference position may be a position on the vehicle body where the work implement 3 is attached.

[0018] A pair of left and right steering cylinders 11 are attached across the front frame 2a and the rear frame 2b. The steering cylinders 11 are hydraulic cylinders. The steering cylinders 11 are extended and retracted by hydraulic oil from a steering pump (not shown), thereby changing the direction of travel of the wheel loader 1 left and right. The front frame 2a and rear frame 2b are connected so as to be able to bend about the axis Ax, and together they form an articulated body frame 2.

[0019] A work implement 3 and a pair of running wheels (front wheels) 4a are attached to the front frame 2a. The work implement 3 is attached to the front of the body of the wheel loader 1. The work implement 3 is supported by the body of the wheel loader 1. Specifically, the work implement 3 is rotatably supported by the body frame 2, more particularly by the front frame 2a. The work implement 3 is disposed in front of the body frame 2.

[0020] The work implement 3 includes a boom 14. The base end of the boom 14 is rotatably attached to the front frame 2a by a boom pin 9. The boom 14 includes a left boom member 14L and a right boom member (not shown). The left boom member 14L and the right boom member are joined by a joining member extending in the left-right direction so as not to move relative to each other, forming the boom 14 of an integral structure. The boom pin 9 includes a pair of left and right boom pins, a left boom pin 9L and a right boom pin (not shown). The boom 14 is rotatable relative to the front frame 2a around the left boom pin 9L and the right boom pin as rotation centers. The left boom pin 9L and the right boom pin support the work implement 3 rotatably relative to the body frame 2.

[0021] The work implement 3 includes a bucket 6. The bucket 6 is disposed at the tip of the work implement 3. The bucket 6 is a work tool for excavating and loading. The cutting edge 6a is the tip of the bucket 6. The back surface 6b is part of the outer surface of the bucket 6. The back surface 6b is formed as a flat surface. The back surface 6b extends rearward from the cutting edge 6a. The bucket 6 is rotatably attached to the boom 14 by a bucket pin 17 located at the tip of the boom 14. The bucket 6 has a left boom attachment portion to which the left boom member 14L is attached, and a right boom attachment portion to which a right boom member (not shown) is attached.

[0022] The work implement 3 further includes a bell crank 18 and a link 15. The bell crank 18 is rotatably supported at its approximate center on the boom 14 by a support pin 18a located approximately in the longitudinal center of the boom 14. The link 15 is connected to a connecting pin 18c provided at the lower end (tip) of the bell crank 18. The link 15 connects the bell crank 18 and the bucket 6. The bell crank 18 and the link 15 are disposed between the left boom member 14L and the right boom member in the left-right direction.

[0023] The front frame 2a and the boom 14 are connected by a pair of boom cylinders 16. The boom cylinders 16 are hydraulic cylinders. The boom cylinders 16 rotate the boom 14 up and down around the boom pin 9. The base end of the boom cylinder 16 is attached to the front frame 2a. The tip of the boom cylinder 16 is attached to the boom 14. The boom cylinder 16 is a hydraulic actuator that moves the boom 14 up and down relative to the front frame 2a. As the boom 14 moves up and down, the bucket 6 attached to the tip of the boom 14 also moves up and down.

[0024] The bucket cylinder 19 connects the bell crank 18 and the front frame 2a. The base end of the bucket cylinder 19 is attached to the front frame 2a. The tip of the bucket cylinder 19 is attached to a connecting pin 18b provided at the upper end (base end) of the bell crank 18. The bucket cylinder 19 is a hydraulic actuator that rotates the bucket 6 up and down relative to the boom 14. The bucket cylinder 19 is an implement cylinder that drives the bucket 6. The bucket cylinder 19 drives the bucket 6 to rotate around the bucket pin 17. The bucket 6 is configured to be movable relative to the boom 14. The bucket 6 is configured to be movable relative to the front frame 2a.

[0025] The boom cylinder 16 and the bucket cylinder 19 constitute a work implement actuator that drives the work implement 3 .

[0026] A cab 5 in which an operator can ride and a pair of running wheels (rear wheels) 4b are attached to the rear frame 2b. The box-shaped cab 5 is disposed behind the boom 14. The cab 5 is mounted on the rear frame 2b. The cab 5 is placed on the body frame 2.

[0027] The cab 5 is provided with a camera 112. The camera 112 is disposed, for example, on the ceiling of the cab 5. The camera 112 is mounted, for example, on an upper portion inside the cab 5. The camera 112 is disposed in the front portion of the cab 5. The camera 112 is attached to the cab 5 facing forward, and is capable of capturing an image of an object in front of the cab 5. The camera 112 periodically captures an image of the object. Therefore, a video is obtained by the camera 112.

[0028] The cab 5 is provided with a LiDAR (Light Detection And Ranging) 111, which is an example of a perception device. The LiDAR 111 is arranged, for example, on the ceiling of the cab 5. The LiDAR 111 is mounted, for example, on the top surface of the cab 5. The LiDAR 111 is arranged in the front of the cab 5. The LiDAR 111 is attached to the cab 5 facing forward, and is able to acquire information about the area in front of the cab 5. In this way, the LiDAR 111 is mounted on the wheel loader 1, and senses the area in front of the wheel loader 1. It should be noted that the number of LiDARs 111 is not limited to one.

[0029] The wheel loader 1 further includes a Global Navigation Satellite System (GNSS) as an example of a position information acquisition device. The wheel loader 1 acquires position information of the current position of the wheel loader 1 via the GNSS.

[0030] (Remote operation device 200) The remote operation device 200 includes a communication device 201, an assistance system 202, and a display 203. As described above, the display 203 is installed at a location where remote operation is performed. The assistance system 202 is an example of a "remote operation assistance system" of the present disclosure. The assistance system 202 is connected to the communication device 201 and the display 203.

[0031] The remote control device 200 remotely controls the wheel loader 1. The remote control device 200 is a device (not shown) that simulates the operation device and machine monitor inside the cab 5. An operator operating the remote control device 200 can remotely control the wheel loader 1 by operating the simulated operation device (not shown). For example, the engine 21 (see FIG. 3) of the wheel loader 1 is started or stopped by remote control. The wheel loader 1 travels by remote control. The wheel loader 1 is stopped while traveling by remote control. The work implement 3 is driven by remote control.

[0032] More specifically, the operator remotely controls the wheel loader 1 while visually checking the information displayed on the display 203. The remote control device 200 uses the communication device 201 to perform two-way communication with the wheel loader 1.

[0033] The support system 202 supports the operator in remotely operating the wheel loader 1. The support system 202 displays support information on the display 203. Details of the support information will be described later.

[0034] <B. Travel Route> FIG. 2 is a schematic diagram showing the travel route of the wheel loader 1 and the configuration of the dump truck.

[0035] The excavation target 400 shown in Fig. 2 is a pile of excavated material to be excavated by the bucket 6 of the wheel loader 1. The excavation target 400 is located in area P2. Note that the excavation target 400 may also be a pile of excavated material accumulated in an excavated material accumulation area such as a stockyard surrounded by walls on three sides. The excavation target 400 may also be a pile of excavated material formed on a vacant lot. The excavated material is, for example, earth and sand.

[0036] The dump truck 300 is parked in a parking area P3, which is a predetermined loading location (position) for the excavation target 400. Information about the parking area P3 is set in advance. The information about the parking area P3 is stored in the assistance system 202.

[0037] Although the dump truck 300 basically stops in a stopping area that complies with the rules of the work site, the dump truck 300 may stop in an arbitrary stopping area that is different from the stopping area that complies with the rules of the work site at the discretion of the operator who operates the dump truck 300 or at the instruction of a control center that monitors the work site. In this case, it is necessary to update the information about the stopping area stored in the assistance system 202.

[0038] The dump truck 300 has a vessel 301 for loading excavated material. The vessel 301 is an example of a "loading container" in the present disclosure. The vessel 301 includes a bottom 310 and a side plate 320 extending upward from the bottom 310. The side plate 320 has a side plate 320L and a side plate 320R. The side plate 320L is the left rear side plate as seen from the driver seated in the cab of the dump truck 300. The side plate 320R is the right rear side plate as seen from the driver. The side plate 320L has a top portion 321L. The side plate 320R has a top portion 321R. The side plates 320L, 320R are an example of a "side wall portion extending upward from a bottom portion" in the present disclosure.

[0039] The side plate 320L and the side plate 320R face each other in the vehicle width direction of the dump truck 300. The side plate 320L and the side plate 320R extend in the overall length direction of the dump truck 300.

[0040] The wheel loader 1 moves forward from area P1, which is the work start position, along path (i) to just before area P2 of the excavation target 400. The wheel loader 1 excavates the excavation target 400. Next, the wheel loader 1 moves backward from just before area P2 to area P1 along path (ii).

[0041] Thereafter, the wheel loader 1 performs a dump approach. Specifically, while raising the work implement 3, the wheel loader 1 advances along the path (iii) to just before the stopping area P3 of the dump truck 300. Next, the wheel loader 1 discharges the excavated material in the bucket 6 into the vessel 301.

[0042] The side plate 320 can prevent the excavated material placed on the vessel 301 from falling to the ground. During the dump approach, of the side plate 320L and the side plate 320R, the side plate 320L is positioned closer to the wheel loader 1.

[0043] As described above, the wheel loader 1 loads the excavated material excavated by the bucket 6 of the work implement 3 into a vessel 301 located in front of the wheel loader 1 and having a bottom 310 and side panels 320 extending upward from the bottom 310.

[0044] <C. Control System of Wheel Loader 1> FIG. 3 is a block diagram showing the schematic configuration of the control system of the wheel loader 1.

[0045] As shown in FIG. 3 , the wheel loader 1 includes an operating device 8, a machine monitor 51, a vehicle body controller 50, electromagnetic proportional control valves 35, 36, a main valve 32, a boom cylinder 16, a bucket cylinder 19, a work implement pump 13, an engine 21, a transmission 23, and an axle 25.

[0046] The operation device 8 includes an accelerator pedal 41 and a work machine operation lever 42. During remote control, the operation device 8 is not operated by the operator.

[0047] The vehicle body controller 50 includes an engine controller 60, a transmission (T / M) controller 70, and a work machine controller 80. The vehicle body controller 50 is generally implemented by reading various programs using a CPU (Central Processing Unit). The vehicle body controller 50 has memory (not shown). The memory functions as a work memory and stores various programs for implementing the functions of the wheel loader 1.

[0048] The wheel loader 1 further includes an articulation angle sensor 121 , a vehicle speed sensor 122 , a boom angle sensor 123 , a bucket angle sensor 124 , and a boom cylinder pressure sensor 125 .

[0049] The articulation angle sensor 121 detects the articulation angle, which is the angle between the front frame 2 a and the rear frame 2 b, and generates a signal of the detected articulation angle. The articulation angle sensor 121 outputs the signal of the articulation angle to the vehicle body controller 50.

[0050] The vehicle speed sensor 122 detects the travel speed of the wheel loader 1 caused by the traveling device 4, for example, by detecting the rotational speed of the output shaft of the transmission 23, and generates a signal of the detected vehicle speed. The vehicle speed sensor 122 outputs the vehicle speed signal to the vehicle body controller 50.

[0051] The boom angle sensor 123 is configured, for example, by a rotary encoder provided on the boom pin 9, which is the attachment portion of the boom 14 to the body frame 2. The boom angle sensor 123 detects the angle (boom angle) of the boom 14 with respect to the horizontal direction, and generates a signal of the detected angle of the boom 14. The boom angle sensor 123 outputs the signal of the angle of the boom 14 to the body controller 50.

[0052] Bucket angle sensor 124 is configured, for example, by a rotary encoder provided on support pin 18a, which is the rotation axis of bell crank 18. Bucket angle sensor 124 detects the angle of bell crank 18 with respect to boom 14 (bell crank angle) and generates a signal of the detected angle of bell crank 18. Bucket angle sensor 124 outputs the signal of the angle of bell crank 18 to vehicle body controller 50. Vehicle body controller 50 calculates the angle of bucket 6 with respect to boom 14 (bucket angle) from the detected angle of bell crank 18.

[0053] The boom angle sensor 123 and the bucket angle sensor 124 are work implement attitude sensors that detect the attitude of the work implement 3. The boom angle sensor 123 may be a stroke sensor arranged in the boom cylinder 16. The bucket angle sensor 124 may be a potentiometer or a proximity switch attached to the bucket pin 17, or may be a stroke sensor arranged in the bucket cylinder 19. An inertial measurement unit (IMU) may be used instead of the boom angle sensor 123 and the bucket angle sensor 124.

[0054] Boom cylinder pressure sensor 125 detects the pressure on the bottom side of boom cylinder 16 (boom bottom pressure) and generates a signal of the detected boom bottom pressure. The boom bottom pressure increases when bucket 6 is loaded and decreases when bucket 6 is empty. Boom cylinder pressure sensor 125 outputs the boom bottom pressure signal to vehicle body controller 50.

[0055] The wheel loader 1 further includes a remote operation controller 90 and a communication device 150. The remote operation controller 90 is communicatively connected to the vehicle body controller 50, the communication device 150, the LiDAR 111, and the camera 112. The remote operation controller 90 functions as an interface between the remote operation device 200 and the vehicle body controller 50.

[0056] (Transmission of information to remote operation device 200) The remote operation controller 90 uses the communication device 150 to send various pieces of information used for remote operation to the remote operation device 200. This information can be displayed on the display 203 of the remote operation device 200. The operator operating the remote operation device 200 remotely operates the wheel loader 1 while referring to this information. Examples of this information will be described below.

[0057] When the remote operation controller 90 acquires information about the area in front of the cab 5 acquired by the LiDAR 111, it sends the information about the area in front of the cab 5 to the remote operation device 200 via the communication device 150. The acquisition and transmission of information about the area in front of the cab 5 is performed at a predetermined cycle. The information about the area in front of the cab 5 is point cloud data. The point cloud data is data that indicates three-dimensional position information of an object to be scanned in front of the cab 5, with the LiDAR 111 as the reference. The point cloud data includes position information for each of a plurality of positioning points.

[0058] When the remote control controller 90 acquires an image of the subject captured by the camera 112, it transmits the image of the subject to the remote control device 200 via the communication device 150. The image of the subject and the transmission of the image of the subject are performed at a predetermined cycle. Note that this cycle may be the same as or different from the cycle for acquiring information about the area in front of the cab 5.

[0059] The remote operation controller 90 acquires from the vehicle body controller 50 the same information that the vehicle body controller 50 outputs to the machine monitor 51. The remote operation controller 90 sends the acquired information to the remote operation device 200 via the communication device 150.

[0060] The remote operation controller 90 acquires information on the articulation angle, vehicle speed, angle of the boom 14, angle of the bucket 6 relative to the boom 14 (bucket angle), and boom bottom pressure from the vehicle body controller 50. The remote operation controller 90 transmits this information to the remote operation device 200 via the communication device 150.

[0061] (Receiving a remote operation command from the remote operation device 200) The remote operation controller 90 acquires a remote operation command sent from the remote operation device 200 via the communication device 150. The remote operation command is generated by an operator operating the remote operation device 200. The remote operation controller 90 sends the acquired remote operation command to the vehicle body controller 50.

[0062] The remote operation controller 90 notifies the vehicle body controller 50 of information (signal) indicating the operation amount of the accelerator pedal (not shown) of the remote operation device 200. More specifically, the information indicating the operation amount of the accelerator pedal of the remote operation device 200 is notified to the transmission (T / M) controller 70. The information is also notified to the engine controller 60.

[0063] The remote operation controller 90 notifies the vehicle body controller 50 of information (signal) indicating the amount of operation of a work implement control lever (not shown) of the remote operation device 200. In more detail, the information indicating the amount of operation of the work implement control lever of the remote operation device 200 is notified to the work implement controller 80. The information is also notified to the engine controller 60.

[0064] (Operation Based on Remote Operation Command) The engine controller 60 controls the output of the engine 21. The engine controller 60 controls the output of the engine 21 based on information indicating the amount of operation of the accelerator pedal of the remote operation device 200 and information indicating the amount of operation of the work implement operation lever of the remote operation device 200. The engine 21 is a drive source that generates drive force for driving the work implement 3 and the traveling device 4.

[0065] The transmission controller 70 controls the transmission 23. The transmission controller 70 controls the operation of the transmission 23 based on information indicating the amount of accelerator pedal operation of the remote control device 200. The transmission 23 changes the driving force generated by the engine 21 to an appropriate torque and rotational speed. The driving force changed by the transmission 23 is transmitted to the axle 25. The driving force is transmitted from the axle 25 to the running wheels 4a, 4b (FIG. 1).

[0066] The work machine controller 80 controls the operation of the work machine 3. The work machine controller 80 controls the attitude of the work machine 3. The work machine controller 80 controls the operation of the work machine 3 based on information indicating the amount of operation of a work machine operation lever of the remote control device 200. The work machine controller 80 sets the attitude of the work machine 3 to a position corresponding to the amount of operation of the work machine operation lever.

[0067] The work machine controller 80 controls the electromagnetic proportional control valves 35, 36 based on the amount of operation of the work machine operation lever of the remote operation device 200. The electromagnetic proportional control valve 35 switches the main valve 32 so that the bucket cylinder 19 retracts and moves the bucket 6 in the dump direction (the direction in which the cutting edge of the bucket 6 lowers). The electromagnetic proportional control valve 35 also switches the main valve 32 so that the bucket cylinder 19 extends and moves the bucket 6 in the tilt direction (the direction in which the cutting edge of the bucket 6 rises). The electromagnetic proportional control valve 36 switches the main valve 32 so that the boom cylinder 16 retracts and moves the boom 14 lower. The electromagnetic proportional control valve 36 also switches the main valve 32 so that the boom cylinder 16 extends and moves the boom 14 up.

[0068] More specifically, a portion of the driving force of the engine 21 is transmitted to the work implement pump 13. The work implement pump 13 is driven by the engine 21 and operates the work implement 3 by the hydraulic oil that it discharges. The work implement 3 is driven by the hydraulic oil from the work implement pump 13. The hydraulic oil discharged from the work implement pump 13 is supplied to the boom cylinder 16 and the bucket cylinder 19 via a main valve 32. The boom cylinder 16 receives the supply of hydraulic oil and extends and contracts, thereby raising and lowering the boom 14. The bucket cylinder 19 receives the supply of hydraulic oil and extends and contracts, thereby rotating the bucket 6 up and down.

[0069] <D. Configuration of the Support System 202> As described above, the remote operation device 200 includes the support system 202 and the display 203. The support system 202 supports the operator's remote operation. The support system 202 supports the remote operation by displaying a support image on the display 203. The following description focuses on the display of the support image.

[0070] Fig. 4 is a functional block diagram showing the functional configuration of the support system 202. As shown in Fig. 4, the support system 202 includes an imaging data acquisition unit 221, a posture identification unit 222, a position information acquisition unit 223, a determination unit 224, an object image generation unit 225, and a display control unit 226. The determination unit 224 includes a distance calculation unit 2241. The object image generation unit 225 includes a loader object generation unit 2251 and a side panel object generation unit 2252.

[0071] The posture identification unit 222 stores work machine shape information D1 and work machine mounting position information D2. The work machine shape information D1 indicates the shape of the work machine 3 of the wheel loader 1. The work machine shape information D1 stores the geometric dimensions of the work machine 3. The work machine shape information D1 includes various dimensional data of multiple members included in the work machine 3.

[0072] For example, with respect to the boom 14, the work machine shape information D1 includes the distance between the through hole into which the boom pin 9 is inserted and the through hole into which the bucket pin 17 is inserted. For example, with respect to the bell crank 18, the work machine shape information D1 includes the distance between the through hole into which the connecting pin 18b is inserted and the through hole into which the support pin 18a is inserted, and the distance between the through hole into which the support pin 18a is inserted and the through hole into which the connecting pin 18c is inserted.

[0073] The work implement attachment position information D2 is data that indicates the attachment position of the boom 14 (position of the boom pin 9) relative to the above-mentioned reference position (center pin 10A) when the wheel loader 1 is not articulated.

[0074] The position information acquisition unit 223 includes an ICP (Iterative Closest Point) matching unit 2231. The ICP matching unit 2231 stores an ICP algorithm (not shown) as an algorithm used for shape matching, and dump shape information D3. Note that the method used for matching is not limited to ICP.

[0075] The dump truck shape information D3 indicates the shape of the dump truck 300. More specifically, the dump truck shape information D3 includes shape information of the vessel 301. The dump truck shape information D3 includes information indicating the position (range) of the vessel 301 in the dump truck 300. The dump truck shape information D3 includes shape information of the side plates 320. The dump truck shape information D3 includes information indicating the position (range) of the side plates 320 in the vessel 301. In this way, the dump truck shape information D3 includes information that identifies the position of the side plates 320 in the dump truck 300 (more specifically, the vessel 301). The dump truck shape information D3 is an example of "shape information" in the present disclosure.

[0076] The imaging data acquisition unit 221 periodically receives imaging data (image data) captured by the camera 112 of the wheel loader 1. The imaging data acquisition unit 221 periodically receives imaging data of a subject in front of the wheel loader 1. The imaging data acquisition unit 221 sends the acquired imaging data to the display control unit 226.

[0077] The posture identification unit 222 acquires posture information indicating the posture of the work machine 3 by identifying the posture of the work machine 3. Specifically, the posture identification unit 222 periodically acquires sensing results from the sensors 121 to 125 (FIG. 3) from the wheel loader 1. The posture identification unit 222 identifies the posture of the work machine 3 based on the sensing results, the work machine shape information D1, and the work machine mounting position information D2. In this way, the posture identification unit 222 can acquire posture information indicating the posture of the work machine 3. The posture identification unit 222 sends the posture information to the object image generation unit 225. Specifically, the posture information is sent to the loader object generation unit 2251.

[0078] 2, the position information acquisition unit 223 acquires position information (hereinafter also referred to as "relative position information of the side plate 320L") indicating the three-dimensional relative position of the side plate 320L with respect to the wheel loader 1. The relative position information of the side plate 320L is an example of "first position information" in the present disclosure.

[0079] In detail, the position information acquisition unit 223 acquires position information indicating the three-dimensional relative position of the area in front of the wheel loader 1 with respect to the LiDAR 111. Specifically, the position information acquisition unit 223 periodically acquires point cloud data acquired by the LiDAR 111 from the wheel loader 1. The point cloud data is an example of "second position information" in the present disclosure.

[0080] The ICP matching unit 2231 of the position information acquisition unit 223 matches the acquired point cloud data with the dump truck shape information D3 ( FIG. 4 ) using an ICP algorithm while the dump truck 300 is parked in a predetermined parking area P3. If the matching is successful, the position information acquisition unit 223 extracts point cloud data indicating the dump truck 300 from the acquired point cloud data.

[0081] The position information acquisition unit 223 calculates the orientation and position of the dump truck 300 within the stopping area of ​​the dump truck 300 based on the point cloud data indicating the dump truck 300. The position information acquisition unit 223 calculates the orientation and position of the dump truck 300 with respect to the stopping area. Note that this orientation is also the orientation of the vessel 301. The position information acquisition unit 223 sends the acquired orientation and position of the dump truck 300 to the determination unit 224.

[0082] 2, the determination unit 224 determines whether or not to generate an object image of the side panel 320L. The determination unit 224 makes this determination based on the distance calculated by the distance calculation unit 2241.

[0083] The determination unit 224 calculates in real time the positional relationship between the wheel loader 1 and the stopping area of ​​the dump truck 300, based on the position information of the wheel loader 1 acquired by the GNSS 113. The distance calculation unit 2241 periodically calculates the separation distance between the reference position of the wheel loader 1 (in this example, the center pin 10A) and the side plate 320L (the front side plate of the two side plates 320L, 320R) of the dump truck 300, based on the information on the orientation and position of the dump truck 300 from the position information acquisition unit 223 and the calculated positional relationship.

[0084] If the calculated distance is equal to or less than a threshold value, the determination unit 224 sends a generation command to the object image generation unit 225 to generate an object image of the side plate 320, together with the relative position information of the side plate. The threshold value is a distance at which the bucket 6 and the side plate 320L are sufficiently separated from each other. For example, 8 m can be used as the threshold value.

[0085] The object image generation unit 225 generates an object image to be displayed on the display 203. Below, the explanation will be divided into a case where the object image generation unit 225 has not received a generation command to generate an object image of the side plate 320 from the determination unit 224 and a case where the object image generation unit 225 has received the generation command. In other words, the explanation will be divided into a case where the distance (separation distance) from the reference position of the wheel loader 1 to the side plate 320L is longer than a threshold value and a case where the distance is equal to or shorter than the threshold value.

[0086] (When the Separation Distance is Longer than the Threshold) When the distance from the reference position of the wheel loader 1 to the side plate 320L is longer than the threshold, the object image generating unit 225 performs the following process.

[0087] The loader object generation unit 2251 of the object image generation unit 225 generates object image data (hereinafter also referred to as “loader object data DR”) that indicates the state of the wheel loader 1 when viewed in a direction that provides a side view of the work machine 3, based on the posture information sent from the posture identification unit 222.

[0088] In this example, the loader object data DR is image data showing a part of the wheel loader 1. The loader object data DR includes an object image of the bucket 6. The loader object data DR includes an object image of the work machine 3. The loader object data DR reflects the posture of the work machine 3. The loader object data DR includes image data showing other object images such as the wheels 4a.

[0089] The loader object generating unit 2251 periodically generates loader object data DR. The object image generating unit 225 periodically sends the generated loader object data DR to the display control unit 226.

[0090] The display control unit 226 generates image data including the loader object data DR and the imaging data acquired from the imaging data acquisition unit 221. In this example, the display control unit 226 periodically generates composite image data in which part of the imaging data acquired from the imaging data acquisition unit 221 is replaced with the loader object data DR.

[0091] The display control unit 226 displays a composite image based on the generated composite image data on the display 203. A moving image based on a plurality of successive composite images is displayed on the display 203.

[0092] Figure 5 is a diagram showing an example of an image displayed on the display 203 of the remote control device 200 when the distance from the reference position of the wheel loader 1 to the side plate 320L is longer than the threshold value. Figure 5 is an image displayed on the display 203 when the wheel loader 1 is located at position Q1 in Figure 2.

[0093] As shown in Fig. 5, the display control unit 226 (Fig. 4) displays a composite image G1 on the display 203. The composite image G1 includes an image G11 obtained by capturing an image with the camera 112. An object image B1 of the wheel loader 1 is displayed in the display region R in the lower right corner of the composite image G1. The image data representing the object image B1 is the loader object data DR described above.

[0094] In this example, the object image B1 is an image showing a portion of the wheel loader 1 on the side of the work implement 3. However, the object image B1 is not limited to this. The object image B1 may show the entire wheel loader 1. The object image B1 may show only the work implement 3. The object image B1 may show only the bucket 6. The display region R may be the lower left corner of the composite image G1, for example.

[0095] In this case, the operator performing remote operation can visually check the composite image G1 to grasp the condition in front of the wheel loader 1 and the posture of the work implement 3 when viewed from the side on a single screen.

[0096] (When the Separation Distance is Less than or Equal to the Threshold) When the distance from the reference position of the wheel loader 1 to the side plate 320L is less than or equal to the threshold, the object image generating unit 225 performs the following process.

[0097] The side panel object generation unit 2252 of the object image generation unit 225 stores in advance shape information of the vessel 301 of the dump truck 300. The shape information of the vessel 301 includes shape information on the length and height of the side panels 320L, 320R, and information on the separation distance between the side panels 320L and 320R.

[0098] Based on the posture information and the relative position information of the side plates 320L, the object image generation unit 225 generates image data that indicates the relative positional relationship between the wheel loader 1 and the side plates 320L when the wheel loader 1 and the dump truck 300 (more specifically, the vessel 301) are viewed in a direction that corresponds to a side view of the work implement 3. In this example, the object image generation unit 225 further uses shape information of the vessel 301 to generate image data (hereinafter also referred to as "relative position image data DT") that indicates the relative positional relationship between the wheel loader 1 and the left and right side plates 320L, 320R when the wheel loader 1 and the dump truck 300 are viewed in a direction that corresponds to a side view of the work implement 3.

[0099] The relative position image data DT includes image data of an object image indicating the state of the wheel loader 1 (the above-mentioned loader object data DR), and image data of an object image indicating the side panel 320 (hereinafter also referred to as "side panel object data DA"). The loader object data DR is generated by the loader object generation unit 2251, as described above. The side panel object data DA is generated by the side panel object generation unit 2252.

[0100] Specifically, the relative position image data DT includes image data of an object image indicating the state of the bucket 6 and image data of an object image indicating the side panel 320. More specifically, the relative position image data DT includes image data of an object image indicating the state of the work implement 3 and image data of an object image indicating the side panel 320. In this example, the relative position image data DT includes image data indicating other object images such as the wheels 4a.

[0101] The object image generating unit 225 periodically generates the relative position image data DT and periodically sends the relative position image data DT to the display control unit 226.

[0102] The display control unit 226 generates image data including the imaging data acquired from the imaging data acquisition unit 221 and the relative position image data DT. In this example, the display control unit 226 periodically generates composite image data in which part of the imaging data acquired from the imaging data acquisition unit 221 is replaced with the relative position image data DT.

[0103] The display control unit 226 displays a composite image based on the generated composite image data on the display 203. A moving image based on a plurality of successive composite images is displayed on the display 203.

[0104] Figures 6 and 7 are diagrams showing examples of images displayed on the display 203 of the remote control device 200 when the distance from the reference position of the wheel loader 1 to the side plate 320L is equal to or less than a threshold. Figure 6 is an image displayed on the display 203 when the wheel loader 1 is located at position Q2 in Figure 2. Figure 7 is an image displayed on the display 203 when the wheel loader 1 is located at position Q3 in Figure 2.

[0105] As shown in Fig. 6, the display control unit 226 (Fig. 4) displays a composite image G2 on the display 203. The composite image G2 includes an image G21 obtained by imaging using the camera 112. An image G22 based on the relative position image data DT is displayed in the display region R in the lower right corner of the composite image G2. The image G22 is an image that shows the relative positional relationship between the work implement 3 (in particular, the bucket 6) and the side panel 320 when the wheel loader 1 and the dump truck 300 (more specifically, the vessel 301) are viewed in a direction that provides a side view of the work implement 3.

[0106] In this example, image G22 includes an object image B1 of the wheel loader 1 and an object image B320 showing the side panel 320. Object image B1 includes an object image B3 of the work implement 3. Object image B3 includes an object image B6 of the bucket 6. Object image B320 has an object image B320L showing the side panel 320L and an object image B320R showing the side panel 320R. The image data showing object image B320 is the side panel object data DA described above.

[0107] The separation distance between the object image B1 of the wheel loader 1 and the object image B320L showing the side plate 320L is based on the distance calculated by the distance calculation unit 2241. The distance between the object image B320L showing the side plate 320L and the object image B320R showing the side plate 320R is based on the dump truck shape information D3.

[0108] Therefore, by viewing the composite image G2, the operator performing remote operation can grasp, on a single screen, the state in front of the wheel loader 1 and the posture of the work implement 3 in a side view of the work implement 3. Furthermore, the operator can easily grasp the relative positional relationship between the bucket 6 and the side plate 320L of the vessel 301 from the content displayed in the display area R. In particular, the operator can easily grasp the positional relationship between the bucket 6 and the top 321L of the nearer side plate 320L. Therefore, it is possible to quickly avoid the bucket 6 from colliding with the side plate 320L during a dump approach.

[0109] As shown in Fig. 7, the display control unit 226 (Fig. 4) displays a composite image G3 on the display 203. The composite image G3 includes an image G31 obtained by imaging using the camera 112. An image G32 based on the relative position image data DT is displayed in the display region R in the lower right corner of the composite image G3. The image G32 is an image that shows the relative positional relationship between the work implement 3 (in particular, the bucket 6) and the side panel 320 when the wheel loader 1 and the dump truck 300 (more specifically, the vessel 301) are viewed in a direction that provides a side view of the work implement 3.

[0110] In this example, the image G32, like the image G22 shown in FIG. 6, includes an object image B1 of the wheel loader 1, an object image B320L showing the side plate 320L, and an object image B320R showing the side plate 320R.

[0111] Therefore, by viewing the composite image G3, the operator performing remote operation can grasp, on a single screen, the state of the front of the wheel loader 1 and the posture of the work implement 3 in a side view of the work implement 3. Furthermore, the operator can easily grasp the relative positional relationship between the bucket 6 and the side plate 320L of the vessel 301 from the content displayed in the display area R. In particular, the operator can easily grasp the positional relationship between the bucket 6, the lower end of the boom 14, and the top 321L of the front side plate 320L. Furthermore, the operator can easily determine whether the bucket 6 is in an appropriate position for discharging the excavated material in the bucket 6.

[0112] <E. Processing Flow> Figures 8 and 9 are flow diagrams showing the flow of processing executed in the support system 202. Figure 8 shows the flow of processing in a preparation stage that is carried out with the wheel loader 1 stopped. The processing in this preparation stage is typically carried out before the wheel loader 1 is caused to travel towards the excavation target 400 in order to excavate the excavation target 400. Figure 9 shows the flow of processing that is carried out after preparation is complete.

[0113] 8 and 9 are typically executed by a processor in the assistance system 202. Specifically, the processes are realized by the processor executing a program stored in a memory. The processor is not limited to a processor dedicated to the assistance system 202, and may be a processor of the remote operation device 200.

[0114] As shown in Fig. 8, in step S1, the support system 202 periodically acquires point cloud data by sensing using the LiDAR 111 while the wheel loader 1 is stopped. This process is executed by the position information acquisition unit 223 shown in Fig. 4. When the operator performs a predetermined remote operation, acquisition of point cloud data begins.

[0115] In step S2, the assistance system 202 matches the acquired point cloud data with the dump truck shape information D3 ( FIG. 4 ) using the ICP algorithm while the dump truck 300 is parked in a predetermined parking area P3. This process is executed by the position information acquisition unit 223 (more specifically, the ICP matching unit 2231).

[0116] If the matching is successful (YES in step S3), in step S4, the support system 202 extracts point cloud data indicating the dump truck 300 from the acquired point cloud data. In step S5, the support system 202 calculates the orientation and position of the dump truck 300 within the stopping area of ​​the dump truck 300. In detail, the support system 202 calculates the orientation and position of the dump truck 300 with respect to the stopping area. These processes are executed by the position information acquisition unit 223.

[0117] When the above preparatory processing is completed, the process proceeds to the execution of the processing shown in Fig. 9. The support system 202 may start the processing shown in Fig. 9 based on remote operation by an operator, or may start it automatically.

[0118] As shown in Fig. 9, in step S11, the assistance system 202 periodically acquires the posture of the work implement 3. This process is executed by the posture identification unit 222 shown in Fig. 4. In step S12, the assistance system 202 periodically generates loader object data DR when the wheel loader 1 is viewed in a direction that provides a side view of the work implement 3. This process is executed by the loader object generation unit 2251 shown in Fig. 4.

[0119] In step S13, the support system 202 calculates in real time the positional relationship between the wheel loader 1 and the stopping area of ​​the dump truck 300, based on the position information of the wheel loader 1 acquired by the GNSS 113. This process is executed by the determination unit 224 shown in FIG.

[0120] In step S14, the support system 202 periodically calculates the separation distance between the reference position of the wheel loader 1 (in this example, the center pin 10A) and the side plate 320L (the front side plate of the two side plates 320L, 320R) of the dump truck 300, based on the information about the orientation and position of the dump truck 300 calculated in step S5 of Fig. 8 and the calculated positional relationship. This process is executed by the distance calculation unit 2241 in the determination unit 224 of Fig. 4.

[0121] If the calculated separation distance is equal to or less than the threshold value (YES in step S15), in step S16, the support system 202 periodically generates side panel object data DA when the dump truck (vessel) is viewed in a direction that corresponds to a side view of the work implement 3. This process is executed by the side panel object generation unit 2252 in FIG. 4 .

[0122] In step S17, the support system 202 generates relative position image data DT including the loader object data DR and the side panel object data DA. This process is executed by the object image generation unit 225.

[0123] In step S18, the support system 202 displays an image based on the relative position image data DT on the display 203, together with an image based on the image data captured by the camera 112. This process is executed by the display control unit 226. By displaying an image based on the relative position image data DT (for example, image G22 in FIG. 6 and image G32 in FIG. 7 ), the operator can grasp the relative positional relationship between the wheel loader 1 (more specifically, the work implement 3) and the side panel 320 on the display 203.

[0124] If the calculated separation distance is equal to or less than the threshold value (NO in step S15), in step S19, an image based on the loader object data DR (for example, object image B1 in FIG. 5 ) is displayed on the display 203 together with an image based on the image data captured by the camera 112. This process is executed by the display control unit 226. By displaying the image based on the loader object data DR, the operator can recognize that the distance to the dump truck 300 (more specifically, the vessel 301) is longer than the threshold value.

[0125] In the above description, the side panel object data DA is generated on the condition that the separation distance is equal to or less than a threshold value, but this is not limiting. The side panel object data DA may be generated even if the separation distance is longer than the threshold value.

[0126] Alternatively, if the separation distance is longer than the threshold value, the support system 202 may display on the display 203 only an image based on image data obtained by capturing an image using the camera 112 instead of step S19, and omit displaying an image based on the loader object data DR.

[0127] <F. Summary> Some of the processes executed by the assistance system 202 and their advantages can be summarized as follows.

[0128] (1) The assistance system 202 assists in the remote operation of the wheel loader 1. The assistance system 202 acquires attitude information indicating the attitude of the work implement 3. The assistance system 202 acquires first position information indicating the three-dimensional relative position of the side plate 320 with respect to the wheel loader 1. Based on the attitude information and the first position information, the assistance system 202 displays on the display 203 installed at the location where remote operation is performed a first image indicating the relative positional relationship between at least the work implement 3 (particularly the bucket 6) and the side plate 320 when the wheel loader 1 and the side plate 320 are viewed in a direction that provides a side view of the work implement 3. Note that the first position information includes the relative position information of the side plate 320L described above.

[0129] With this configuration, the operator performing remote control can easily grasp the relative positional relationship between the bucket 6 and the side plate 320L of the vessel 301. Therefore, it is possible to avoid the bucket 6 accidentally colliding with the side plate 320L during loading work.

[0130] (2) The assistance system 202 stores shape information indicating the shape of the vessel 301 and an ICP algorithm used for shape matching. The assistance system 202 acquires, from the LiDAR 111 that is mounted on the wheel loader 1 and senses the area in front of the wheel loader 1, second position information that indicates the three-dimensional relative position of that area with respect to the LiDAR 111. The assistance system 202 acquires information about the orientation and position of the vessel 301 by matching the second position information with the shape information using the ICP algorithm. The assistance system 202 acquires first position information based on the orientation and position of the vessel 301 and the current position of the wheel loader 1.

[0131] With this configuration, the support system 202 can acquire first position information that indicates the three-dimensional relative position of the side plate 320 with respect to the wheel loader 1 .

[0132] (3) The first image includes an object image B3 ( FIGS. 6 and 7 ) representing the work implement 3 and an object image B320 ( FIGS. 6 and 7 ) representing the side panel 320. With this configuration, the operator can view the object image B3 representing the work implement 3 and the object image B320 representing the side panel 320 simultaneously on the display 203.

[0133] (4) The support system 202 displays the first image on the display 203 together with a second image showing the front of the wheel loader 1 captured by the camera 112 attached to the wheel loader 1. With this configuration, the operator can grasp the state of the front of the wheel loader 1 and the posture of the bucket 6 in a side view of the work implement 3 on a single screen.

[0134] (5) The support system 202 calculates the distance from the reference position of the wheel loader 1 to the side plate 320 (in this example, the front side plate 320L) based on the first position information. The support system 202 displays the first image on the display 203 on the condition that the calculated distance is equal to or less than a predetermined threshold value.

[0135] When the wheel loader 1 is away from the dump truck 300 (when the distance is longer than the threshold), the operator does not need to pay particular attention to the dump truck 300 (particularly the side plate 320 of the vessel 301). On the other hand, when the wheel loader 1 approaches the dump truck 300, the operator needs to pay attention to the side plate 320. Therefore, on the condition that the calculated distance is equal to or less than a predetermined threshold, by displaying a first image on the display 203 that indicates the relative positional relationship between the work implement 3 (particularly the bucket 6) and the side plate 320, it is possible to support the operator's operation at a timing that is convenient for the operator.

[0136] (6) The reference position is a position on the wheel loader 1 that is different from the position of the work implement 3. If the reference position is a position on the work implement 3 (for example, the position of the cutting edge 6a of the bucket 6), then while the wheel loader 1 is moving forward, after the measured distance falls below the threshold, it is possible that the measured distance will become greater than the threshold due to the rise of the boom 14 and / or the rotation of the bucket 6 about the bucket pin 17. In such cases, a display fluttering phenomenon occurs, where the first image described above is sometimes displayed and sometimes not displayed.

[0137] However, by setting the reference position to a position on the wheel loader 1 that is different from the position of the work implement 3, the measured distance from the wheel loader 1 to the side plate 320 is not affected by changes in the attitude of the work implement 3. Therefore, during a dump approach, it is possible to prevent the fluttering phenomenon described above from occurring at a position where the separation distance between the wheel loader 1 and the side plate 320 is close to the threshold value.

[0138] <G. Modifications> (1) In the above, an example has been described in which the remote control device 200 includes the assistance system 202, but the present invention is not limited to this. Some of the functions of the assistance system 202 may be executed by the wheel loader 1.

[0139] For example, the attitude identification unit 222 in the support system 202 may be provided in the wheel loader 1. Furthermore, the object image generation unit 225 may be provided in the wheel loader 1. The position information acquisition unit 223 in the support system 202 may be provided in the wheel loader 1. Furthermore, the determination unit 224 may be provided in the wheel loader 1. In this way, the support system 202 may be configured by the wheel loader 1 and the remote control device 200.

[0140] (2) In the above, the side panel 320 portion is extracted from the sensing results of the LiDAR 111 by performing matching using the ICP algorithm. However, matching is not always necessary.

[0141] For example, the assistance system may generate object image data showing the side panel 320 of the vessel 301 from the above-mentioned dump shape information D3 and the sensing results (point cloud data) from the LiDAR 111, and display the generated image data on the display 203.

[0142] Alternatively, the assistance system may display the sensing results (point cloud data) obtained by the LiDAR 111 on the display 203. In this case, the operator determines the location of the side panel 320 from the displayed point cloud data.

[0143] (3) As shown in Figures 5 to 7, an object image of the work implement 3 is shown in the display area R, but this is not necessarily limited to this. It is sufficient that an object image of at least the bucket 6 is displayed in the display area R.

[0144] (4) It is not necessarily necessary to display the images (e.g., images G11, G21, G31) captured by the camera 112, and only the object image B1 may be displayed on the display 203. Furthermore, instead of displaying the image captured by the camera 112 and the object image B1 on a single display 203, each may be displayed on a separate display.

[0145] (5) The loading vessel may be any vessel into which excavated material is loaded from above. The loading vessel may be a hopper. The loading vessel may be a container. The container may be installed on a vehicle or placed directly on the ground.

[0146] (6) In the above description, after successful matching by ICP matching, the separation distance between the reference position of the wheel loader 1 and the side plate 320L of the dump truck 300 is calculated using the GNSS 113 without using ICP matching. However, this is not limited to this. If the calculation processing speed in the assistance system 202 is sufficiently fast, the assistance system 202 may calculate the separation distance by constantly performing ICP matching without using the position information of the GNSS 113.

[0147] The embodiments disclosed herein are merely examples and are not limited to the above. The scope of the present invention is defined by the claims, and it is intended to include all modifications within the meaning and scope of the claims.

[0148] 1 Wheel loader, 2 Body frame, 2a Front frame, 2b Rear frame, 3 Work implement, 4 Traveling device, 5 Cab, 6 Bucket, 6a Cutting edge, 6b Rear face, 8 Operating device, 9 Boom pin, 10, 10A, 10B Center pin, 11 Steering cylinder, 13 Work implement pump, 14 Boom, 15 Link, 16 Boom cylinder, 17 Bucket pin, 18 Bell crank, 19 Bucket cylinder, 25 Axle, 32 Main valve, 35, 36 Electromagnetic proportional control valve, 41 Accelerator pedal, 42 Work implement operating lever, 50 Body controller, 90 Remote operation controller, 112 Camera, 121 Articulate angle sensor, 122 Vehicle speed sensor, 123 Boom angle sensor, 124 Bucket angle sensor, 125 Boom cylinder pressure sensor, 150, 201 Communication device, 200 Remote operation device, 202 support system, 203 display, 300 dump truck, 301 vessel, 310 bottom, 320, 320L, 320R side plate, 321L, 321R top, 400 excavation target, 1000 remote operation system, B1, B320L, B320R object image, R display area.

Claims

1. A remote operation support system that supports the remote operation of a work vehicle, wherein the work vehicle loads material excavated by a bucket of a work machine into a loading container having a bottom and a side wall extending upward from the bottom, the remote operation support system acquires attitude information that indicates the attitude of the work machine, acquires first position information that indicates the three-dimensional relative position of the side wall with respect to the work vehicle, and displays, based on the attitude information and the first position information, a first image that indicates the relative positional relationship between the bucket and the side wall when the work vehicle and the loading container are viewed in a direction that provides a side view of the work machine, on a display installed at a location where the remote operation is performed.

2. A remote operation assistance system that assists in the remote operation of a work vehicle, wherein the work vehicle loads material excavated by a bucket of a work machine into a loading container having a bottom and side walls extending upward from the bottom, the remote operation assistance system stores shape information that indicates the shape of the loading container and an algorithm used for shape matching, acquires attitude information that indicates the attitude of the work machine, acquires first position information that indicates the three-dimensional relative position of the side walls with respect to the work vehicle, acquires second position information from a perception device that is mounted on the work vehicle and senses an area in front of the work vehicle, the second position information indicating the three-dimensional relative position of the area with respect to the perception device, and acquires information on the orientation and position of the loading container by matching the second position information with the shape information using the algorithm, the first position information being acquired based on the orientation and position of the loading container and the current position of the work vehicle, A remote operation support system that displays, based on the posture information and the first position information, a first image showing the relative positional relationship between the bucket and the side wall portion when the work vehicle and the loading container are viewed in a direction that is a side view of the work machine, on a display installed at a location where the remote operation is performed.

3. The remote operation support system according to claim 2, wherein the perception device is a LiDAR (Light Detection And Ranging) and the algorithm is an ICP (Iterative Closest Point) algorithm.

4. A remote operation support system as described in claim 1 or 2, wherein the first image includes an object image representing the work machine and an object image representing the side wall portion.

5. A remote operation support system as described in claim 1 or 2, wherein the first image is displayed on the display together with a second image showing the front of the work vehicle captured by a camera attached to the work vehicle.

6. The remote operation support system according to claim 1 or 2, wherein the remote operation support system calculates the distance from the reference position of the work vehicle to the side wall portion based on the first position information, and displays the first image on the display when the calculated distance is equal to or less than a predetermined threshold value.

7. A remote operation support system according to claim 6, wherein the reference position is a position different from the position of the work implement of the work vehicle.

8. The remote operation support system according to claim 1 or 2, wherein the loading container is a vessel of a dump truck, and the side wall portion is a side plate.

9. A remote control device for a work vehicle, comprising the remote control support system according to claim 1 or 2 and the display.

10. A remote operation assistance method for assisting in the remote operation of a work vehicle, comprising the steps of: acquiring, from the work vehicle, attitude information indicating the attitude of the work implement of the work vehicle; acquiring first position information indicating the three-dimensional relative position of a side wall of a loading container into which material excavated by the bucket of the work implement is loaded, relative to the work vehicle; and displaying, based on the attitude information and the first position information, a first image indicating the relative positional relationship between the bucket and the side wall when the work vehicle and the loading container are viewed in a direction that provides a side view of the work implement, on a display installed at a location where the remote operation is performed.

11. The remote operation assistance method according to claim 10, wherein the first image includes an object image representing the work machine and an object image representing the side wall portion.

12. A remote operation assistance method as described in claim 10, wherein in the step of displaying the first image on the display, the first image is displayed on the display together with a second image showing the front of the work vehicle captured by a camera attached to the work vehicle.

13. The remote operation assistance method according to any one of claims 10 to 12, further comprising a step of calculating the distance from a reference position of the work vehicle to the side wall portion based on the first position information, and in the step of displaying the first image on the display, the first image is displayed on the display on the condition that the calculated distance is equal to or less than a predetermined threshold value.

14. A remote operation assistance method according to claim 13, wherein the reference position is a position different from the position of the work implement of the work vehicle.

15. The remote operation assistance method according to any one of claims 10 to 12, wherein the loading container is a vessel of a dump truck, and the side wall portion is a side plate.

Images