Work support device, work support system, work machine, remote operation device, program, and work support method

Abstract

A work support device (10) is a device for supporting work in which an object (90) held by a tip-end attachment (55) of a work machine (40) is released from the tip-end attachment (55) and is disposed in a release area (80) present in front of the work machine (40). The work support device (10) is provided with a controller (11) that causes a display device (23) to display an index image (G1) that includes: a release area range image (G2), which is a virtual image serving as an index indicating the range of the release area (80) at least in the longitudinal direction; and an attachment marker (G3) which is a virtual image representing the relative position of the tip-end attachment (55) with respect to the release area (80).

Description

Work support device, work support system, work machine, remote control device, program, and work support method

[0001] The present disclosure relates to a technique for assisting work performed by a work machine.

[0002] Patent Document 1 discloses a display control device and a remote control device that aim to provide appropriate graphic displays for work support during a series of operations of a work machine. Patent Document 2 discloses an image display system that aims to suppress a decrease in work efficiency when working using a work machine equipped with a work implement having a work tool.

[0003] The display control device in Patent Document 1 superimposes on a display a graphic that highlights the position of the bottom end of the bucket and a graphic that highlights the position of the top end of the container. While the graphic display provided by this display control device may help the operator grasp the relative position of the bucket to the container in the up-down direction, it does not help the operator grasp the relative position of the bucket to the container in the front-to-back direction. Furthermore, the image display system in Patent Document 2 does not take into consideration helping the operator grasp the relative position of the bucket to the container in the front-to-back direction.

[0004] JP 2023-120743 A Patent No. 6777375 A

[0005] The present disclosure aims to provide a technology that can assist an operator in understanding the relative position in the front-to-rear direction of a tip attachment with respect to an open area of ​​a container or the like.

[0006] The work assistance device in the present disclosure is a device for assisting in the work of releasing an object held on a tip attachment of a work machine from the tip attachment and placing the object in a release area located in front of the work machine, and the work assistance device is equipped with a controller that causes a display device to display an index image that includes a release area range image, which is a virtual image that serves as an indicator representing the range of the release area in at least the forward / backward direction, and an attachment marker, which is a virtual image that represents the relative position of the tip attachment with respect to the release area.

[0007] The work assistance system according to the present disclosure includes the work assistance device, the work machine, and a remote control device having the display device and for remotely operating the work machine.

[0008] The work machine of the present disclosure includes a work machine body, an attitude detection device that acquires attitude information related to the attitude of the work machine body, an imaging device that acquires a site image that is an image related to the work site where the release area is located, and a machine controller that outputs the attitude information and the site image to the work support device.

[0009] The remote control device in the present disclosure includes a remote control lever that receives operations for remotely operating the work machine, and the display device that displays the index image including the release area range image and the attachment marker in response to a command (index image display command) output by the work support device.

[0010] The program disclosed herein is a program for assisting in the task of releasing an object held on a tip attachment of a work machine from the tip attachment and placing the object in a release area located in front of the work machine, and causes a computer to realize the function of displaying on a display device an index image that includes a release area range image, which is a virtual image that serves as an indicator representing the range of the release area in at least the forward / backward direction, and an attachment marker, which is a virtual image that represents the relative position of the tip attachment with respect to the release area.

[0011] The work assistance method disclosed herein is a method for assisting in the work of releasing an object held on a tip attachment of a work machine from the tip attachment and placing the object in a release area located in front of the work machine, and includes displaying on a display device an index image that includes a release area range image, which is a virtual image that serves as an indicator representing the range of the release area in at least the forward / backward direction, and an attachment marker, which is a virtual image that represents the relative position of the tip attachment with respect to the release area.

[0012] 1 is a block diagram showing a work assistance system according to an embodiment. FIG. 1 is a side view showing a work machine according to an embodiment. FIG. 2 is a view showing a remote control device according to an embodiment. FIG. 3 is a view showing an example of a display screen displayed on the display device of the remote control device. FIG. 4 is a view showing another example of a display screen displayed on the display device of the remote control device. FIG. 5 is a view showing yet another example of a display screen displayed on the display device of the remote control device. FIG. 6 is a view showing the positional relationship between the work machine and a container of the vehicle, and an example of an index image. FIG. 7 is a view showing the positional relationship between the work machine and a container of the vehicle, and another example of an index image. FIG. 8 is a view showing the positional relationship between the work machine and a container of the vehicle, and another example of an index image. FIG. 9 is a view showing yet another example of a display screen displayed on the display device of the remote control device. FIG. 10 is a view showing yet another example of a display screen displayed on the display device of the remote control device. FIG. 11 is a view showing yet another example of a display screen displayed on the display device of the remote control device. FIG. 1 is a diagram for explaining yet another example of the positional relationship between the work machine and the vehicle's container, and an index image. FIG. 2 is a diagram for explaining yet another example of the display screen displayed on the display device of the remote control device. FIG. 3 is a diagram for explaining yet another example of the positional relationship between the work machine and the vehicle's container, and an index image. FIG. 4 is a diagram for explaining yet another example of the positional relationship between the work machine and the vehicle's container, and an index image. FIG. 5 is a diagram for explaining yet another example of the display screen displayed on the display device of the remote control device. FIG. 6 is a diagram for explaining yet another example of the display screen displayed on the display device of the remote control device. FIG. 7 is a diagram for explaining yet another example of the display screen displayed on the display device of the remote control device. FIG. 8 is a block diagram showing a work assistance system according to a modified example of an embodiment.

[0013] An embodiment of the present disclosure will be described with reference to the drawings. Fig. 1 is a block diagram showing a work support system 100 according to an embodiment. Fig. 2 is a side view showing a work machine 40 according to an embodiment. Fig. 3 is a diagram showing a remote control device 20 according to an embodiment.

[0014] The work assistance system 100 comprises a work assistance device 10, a work machine 40, and a remote control device 20. The work assistance device 10, the work machine 40, and the remote control device 20 are capable of transmitting and receiving data to and from each other via communication using various networks 200 such as the Internet or a mobile phone network.

[0015] Fig. 4 is a diagram showing an example of a display screen displayed on the display device 23 of the remote control device 20. Fig. 5 is a diagram showing another example of a display screen displayed on the display device 23 of the remote control device. Fig. 6 is a diagram showing yet another example of a display screen displayed on the display device 23 of the remote control device 20. Fig. 7 is a diagram for explaining the positional relationship between the work machine 40 and the container 80 of the vehicle 70, and an example of an index image G1. Fig. 4 shows the early stage of earth removal work described below, showing a state in which there is almost no earth 90 loaded in the container 80. Fig. 5 shows the middle or final stage of the earth removal work, showing a state in which a large amount of earth 90 has been loaded in the container 80.

[0016] The front-to-rear direction in the drawing represents the positional relationship between the work machine 40 and the vehicle 70. The container 80 (bed) of the vehicle 70 is disposed in front of the work machine 40. That is, the front is the horizontal direction from the work machine 40 toward the container 80. Specifically, for example, the front may be the horizontal direction from the upper rotating body 51 of the work machine 40 toward the container 80. The left-to-right direction in the drawing is a horizontal direction perpendicular to the front-to-rear direction. The container 80 of the vehicle 70 is the bed of the vehicle 70 that transports soil and sand 90. The vehicle 70 has a vehicle cab 71. The container 80 may have, for example, a rectangular shape in a plan view. The container 80 may have, for example, a substantially rectangular parallelepiped shape having a bottom 81, a right wall 83, a left wall 84, a front wall 85, and a rear wall 86. The bottom 81 of the container 80 has a bottom surface 82 (the upper surface of the bottom 81). The top of the container 80 is open. That is, the container 80 has an opening 87 at its top.

[0017] The container 80 is not limited to the loading platform of the vehicle 70, but may be, for example, a facility such as a box (a box that cannot move on its own) installed at a work site, or may be a part of the facility. The part of the facility may be, for example, a portion from which soil and sand are discharged from the facility, or a portion from which soil and sand are input into the facility. Furthermore, the open area in the present disclosure is not limited to the container 80, but may also be a designated soil discharge area at a work site. Specifically, the soil discharge area may be, for example, a construction target area, which is an area at a work site where soil and sand are supplied.

[0018] 4, 5 and 7, the work support device 10 is a device for supporting work of releasing soil 90 (an example of an object) held in a bucket 55 of a work machine 40 from the bucket 55 and placing the soil 90 into a container 80 arranged in front of the work machine 40. The work support device 10 according to this embodiment has the following first characteristic.

[0019] [First Feature] The work assistance device 10 includes a controller 11, which outputs an index image display command to cause the display device 23 to display an index image G1 including a container range image, which is a virtual image serving as an index representing the range of the container 80 at least in the front-to-rear direction, and an attachment marker G3, which is a virtual image representing the relative position of the bucket 55 with respect to the container 80 (step S106 in FIG. 16 , described later). The container range image may be, for example, the virtual plane image G2 shown in FIGS. 6 and 7, or the virtual frame image G5 in the modified examples shown in FIGS. 22 to 24, described later. The container range image is an example of an open area range image in the present disclosure.

[0020] In this work support device 10, when the controller 11 outputs the index image display command, an index image G1 including the container range image (e.g., virtual plane image G2) and an attachment marker G3 is displayed on the display device 23 of the remote control device 20. The index image G1 displayed on the display device 23 makes it easier for the operator operating the remote control device 20 to grasp the relative position of the bucket 55 in the fore-and-aft direction with respect to the container 80. Therefore, in the earth removal work of releasing soil 90 from the bucket 55 and placing the soil 90 into the container 80 (an example of a releasing work), the operator can appropriately adjust the relative position of the bucket 55 in the fore-and-aft direction with respect to the container 80 while viewing the index image G1 displayed on the display device 23.

[0021] The work support device 10 according to this embodiment further includes the following second to ninth features.

[0022] 6 and 7 , the controller 11 preferably generates, as the container range image, a virtual plane image G2, which is an image of a virtual plane that represents the range of the container 80 in the front-to-rear and left-to-right directions. In a work assistance device 10 having this second feature, an operator operating the remote control device 20 can easily grasp not only the relative position of the bucket 55 with respect to the container 80 in the front-to-rear direction but also the relative position in the left-to-right direction, based on an index image G1 including the virtual plane image G2 displayed on the display device 23 of the remote control device 20. Therefore, during the soil removal work, the operator can appropriately adjust the relative position of the bucket 55 with respect to the container 80 in the front-to-rear direction and the left-to-right direction while viewing the index image G1 displayed on the display device 23.

[0023] 6 and 7 , the controller 11 preferably generates, as the container range image, a virtual opening area image G21 (an example of a virtual surface image G2), which is an image of a virtual surface that represents an area corresponding to the opening 87 of the container 80. In a work assistance device 10 having this third feature, the operator can easily grasp the relative position in the front-to-rear direction and the relative position in the left-to-right direction of the bucket 55 with respect to the opening 87 of the container 80, based on the index image G1 that includes the virtual opening area image G21 and that is displayed on the display device 23. Therefore, during the soil removal work, the operator can appropriately adjust the relative position in the front-to-rear direction and the relative position in the left-to-right direction of the bucket 55 with respect to the container 80 while viewing the index image G1 displayed on the display device 23.

[0024] 11 , 12 , 13 , and 14 , the controller 11 may generate a virtual bottom image G22 (an example of a virtual surface image G2) as the container range image. This virtual bottom image G22 represents an area corresponding to the bottom surface 82 of the container 80. In this work support device 10, the operator can easily grasp the relative positions of the bucket 55 in the front-to-rear direction and the left-to-right direction relative to the bottom surface 82 of the container 80 by viewing the index image G1 including the virtual bottom image G22 displayed on the display device 23. Therefore, during the soil removal operation, the operator can appropriately adjust the relative positions of the bucket 55 in the front-to-rear direction and the left-to-right direction relative to the container 80 while viewing the index image G1 displayed on the display device 23. Note that FIG. 12 illustrates the initial stage of the soil removal operation, showing a state in which the container 80 contains almost no soil 90. FIG. 14 illustrates the middle or final stage of the soil removal operation, showing a state in which the container 80 contains a large amount of soil 90.

[0025] [Fourth Feature] As shown in Figures 7 and 8, when a predetermined portion of the bucket 55 is located at a position corresponding to the container 80, the controller 11 preferably generates, as the index image G1, an image in which the attachment marker G3 is located at a position corresponding to the virtual surface image G2. The predetermined portion may be, for example, the tip 55a of the bucket 55 as shown in Figure 7. Alternatively, the predetermined portion may be, for example, the front end 55b, which is the foremost portion of the bucket 55, as shown in Figure 8. The front end 55b of the bucket 55 is a part that changes as the bucket 55 rotates. The tip 55a may be the center of the bucket 55 in the width direction. The front end 55b may be the center of the bucket 55 in the width direction.

[0026] Specifically, in work assistance device 10 having this fourth feature, for example, when the predetermined portion (tip portion 55a or front end portion 55b) of bucket 55 is inside or directly above container 80, controller 11 may generate as index image G1 an image in which attachment marker G3 is located directly below bucket 55 at a position on virtual plane image G2, or may generate as index image G1 an image in which attachment marker G3 is located directly below bucket 55 at a position spaced above virtual plane image G2. In other words, when the predetermined portion is located at a position corresponding to container 80, this may mean when the predetermined portion is inside container 80, or when the predetermined portion is located directly above container 80 (a position spaced above container 80).

[0027] 9 , when the predetermined portion is shifted in the front-rear direction or the left-right direction with respect to the container 80, the controller 11 preferably generates, as the index image G1, an image in which the attachment marker G3 is positioned at a position shifted in the front-rear direction or the left-right direction with respect to the virtual plane image G2. Specifically, for example, when the predetermined portion is shifted in the front-rear direction or the left-right direction with respect to the container 80, the controller 11 may generate, as the index image G1, an image in which the attachment marker G3 is positioned at a position corresponding to the ground around the container 80, as shown in FIG.

[0028] 10 , when a portion of the soil 90 contained in the container 80 (e.g., an upper portion 91 of the soil 90) is located above the height of the virtual opening region image G21 (an example of the virtual surface image G2), the controller 11 may generate, as the index image G1, an image in which the region corresponding to the upper portion 91 of the soil 90 is omitted from the virtual surface image G2. In the work support device 10 having this sixth feature, the upper portion 91 of the soil 90 is not covered by the virtual surface image G2, making it easier for the operator to grasp the status of the soil 90, such as the position of the soil 90 in the container 80 and the uneven distribution of the soil 90.

[0029] 11 and 12 , the controller 11 may generate, as the container range image, a virtual bottom image G22 (an example of a virtual surface image G2), which is an image of a virtual surface that represents an area corresponding to the bottom surface 82 of the container 80. In this case, the controller 11 may generate, as the index image G1, an image in which the attachment marker G3 is located directly below the bucket 55 and at a position on the virtual bottom surface image G22, or may generate, as the index image G1, an image in which the attachment marker G3 is located directly below the bucket 55 and at a position spaced above the virtual bottom surface image G22, as shown in FIG. 13 . In the work assistance device 10 having the seventh feature, the attachment marker G3 is displayed on the display device 23 at a position directly below the bucket 55 and corresponding to the virtual bottom surface image G22 (a position on the virtual bottom surface image G22 or a position spaced above the virtual bottom surface image G22). Therefore, during the soil removal work, the operator can easily grasp the relative position of the bucket 55 in the front-to-rear direction and the left-to-right direction with respect to the container 80.

[0030] 12 and 13 , the attachment marker G3 is a first attachment marker G3 that is positioned directly below the bucket 55 and spaced above the virtual bottom image G22, and the controller 11 preferably generates, as the index image G1, an image that includes the virtual bottom image G22, the first attachment marker G3, and a second attachment marker G4 that is a virtual image that is positioned directly below the bucket 55 and on the virtual bottom image G22. In the work assistance device 10 having this eighth feature, the first attachment marker G3 is positioned on the display device 23 at a position spaced above the virtual bottom image G22, and the second attachment marker G4 is positioned on the virtual bottom image G22. Therefore, during the soil removal work, the operator operating the remote control device 20 can more easily grasp the relative position of the bucket 55 in the front-to-rear direction and the relative position in the left-to-right direction with respect to the bottom 82 of the container 80.

[0031] 15 , the controller 11 may generate, as the index image G1, an image including a virtual opening area image G21 (an example of the virtual surface image G2), an attachment marker G3, and a cab rear wall image G6, which is a virtual image representing an area corresponding to the rear wall 72 of the vehicle cab 71. In the work assistance device 10 having this ninth feature, during the earth removal work, the operator operating the remote control device 20 can easily grasp not only the relative position of the bucket 55 with respect to the container 80, but also the relative position of the bucket 55 with respect to the rear wall 72 of the vehicle cab 71. In the work assistance device 10 having this ninth feature, it is more preferable that the cab rear wall image G6 be displayed on the display device 23 in a different display mode from the virtual opening area image G21. In this case, during the earth removal work, the operator can easily distinguish between the container 80 (the loading platform) and the rear wall 72 of the vehicle cab 71. The different display modes may be, for example, different colors or different shades.

[0032] The area corresponding to the rear wall 72 of the vehicle driver's cab 71 may be the area of ​​the rear wall 72 that separates the space of the vehicle driver's cab 71 from the rear. In addition, if the vehicle 70 has a so-called shrine gate portion such as a guard frame disposed directly behind the vehicle driver's cab 71, the area corresponding to the rear wall 72 of the vehicle driver's cab 71 may be the area surrounded by the outer edge of the shrine gate portion.

[0033] The controller 11 may also generate an image including a container rear wall image as the index image G1. The container rear wall image is a virtual image that represents the area of ​​the rear wall 86 of the container 80.

[0034] The above are the main features of the work assistance device 10 according to this embodiment. Below, a specific example of a work assistance system 100 including the work assistance device 10, a work machine 40, and a remote control device 20 will be described, but the work assistance device, work machine, and remote control device, as well as the work assistance system including these, in this disclosure are not limited to the following specific example.

[0035] 1, the work assistance device 10 may be equipped with a communication device 13, the work machine 40 may be equipped with a communication device 48, and the remote control device 20 may be equipped with a communication device 24. The work assistance device 10, the work machine 40, and the remote control device 20 can send and receive data to and from each other by communication using various networks 200 such as the Internet or a mobile phone network.

[0036] The work assistance device 10 may be an information terminal such as a tablet computer (a so-called tablet), a smartphone, a laptop personal computer, or a desktop personal computer. The work assistance device 10 may be a management device such as a server for managing work performed by a work machine 40. The work assistance device 10 may be a computer in a cloud service provided as a service over a network such as the Internet. The work assistance device 10 may include multiple information terminals.

[0037] The work support device 10 may be provided with a database 12 that stores various data such as dimensional information (specification data) of the work machine 40 and information regarding the range of the container 80 .

[0038] 1 and 2 , work machine 40 includes a work machine main body, an attitude detection device 44, an imaging device 43, and a machine controller 41. The work machine main body includes a lower running structure 50 including a traveling device, an upper rotating structure 51 supported on the lower running structure 50 so as to be rotatable relative to the lower running structure 50 about a vertically extending rotation axis, and a work device 52 supported on the upper rotating structure 51. The work machine 40 according to this embodiment is a hydraulic excavator.

[0039] The upper rotating body 51 includes a rotating frame and a cab 51A supported by the rotating frame. The rotating frame is a frame supported by the lower traveling body 50 so as to be rotatable, and constitutes a base portion of the upper rotating body 51.

[0040] The work device 52 includes a boom 53 that is attached to the upper rotating body 51 so as to be able to rise and fall, an arm 54 that is rotatably attached to the boom 53, and a bucket 55 that is rotatably attached to the arm 54. The bucket 55 is an example of a tip attachment. The tip attachment is not limited to the bucket 55, and may be, for example, another tip attachment such as a grapple, a fork, or a lifting magnet.

[0041] The cab 51A is located, for example, at the front left portion of the revolving frame. A driver's seat, an operating lever 42, and the like are arranged inside the cab 51A. The operating lever 42 receives various operations by the operator, such as boom operation for raising and lowering the boom 53, arm operation for rotating the arm 54, bucket operation for rotating the bucket 55, swing operation for swinging the upper revolving body 51 relative to the undercarriage 50, and travel operation for traveling the undercarriage 50. The operating lever 42 is used when the operator operates the work machine 40 from inside the cab 51A. The operating lever 42 is not used when the work machine 40 is remotely operated. An outer wall defining a machinery room is arranged behind the cab 51A. Various equipment is arranged in the machinery room, including power equipment such as an engine, a battery, and a generator, hydraulic equipment such as a hydraulic pump, and electrical equipment. The hydraulic pump is driven by the power equipment.

[0042] The work machine 40 is equipped with a plurality of actuators. Each of the plurality of actuators is operated by receiving a supply of hydraulic oil discharged from the hydraulic pump. The plurality of actuators includes a boom cylinder 56 for raising and lowering the boom 53, an arm cylinder 57 for rotating the arm 54, a bucket cylinder 58 for rotating the bucket 55, a swing motor 59 for rotating the upper swing structure 51 relative to the lower traveling structure 50, and a travel motor (not shown) for traveling the lower traveling structure 50.

[0043] The attitude detection device 44 acquires attitude information related to the attitude of the work machine main body. The attitude detection device 44 may include a plurality of work implement attitude detectors. As shown in Fig. 2, the plurality of work implement attitude detectors may include a boom attitude detector 44A, an arm attitude detector 44B, and a tip attachment attitude detector 44C. The attitude detection device 44 may further include an inclination detector 44D and a swing angle detector 44E.

[0044] The boom attitude detector 44A may be a sensor that detects the attitude of the boom 53, or may be a sensor that detects the state of the boom cylinder 56 that correlates with the attitude of the boom 53. The arm attitude detector 44B may be a sensor that detects the attitude of the arm 54, or may be a sensor that detects the state of the arm cylinder 57 that correlates with the attitude of the arm 54. The tip attachment attitude detector 44C may be a sensor that detects the attitude of the tip attachment 55, or may be a sensor that detects the state of the tip attachment cylinder 58 that correlates with the attitude of the tip attachment 55. The inclination detector 44D may be a sensor that detects the inclination of the upper rotating body 51 with respect to the horizontal plane. The swing angle detector 44E may be a sensor that detects the relative angle of the upper rotating body 51 with respect to the undercarriage 50, or may be a sensor that detects the orientation of the upper rotating body 51 in the global coordinate system.

[0045] Each of the multiple attitude detectors may include, for example, an inertial measurement unit (IMU), a sensor that detects the degree of extension and contraction of the cylinder (e.g., a stroke sensor), or other sensors.

[0046] The tilt detector 44D may include an inertial measurement unit (IMU). The tilt detector 44D may include a device that calculates the tilt of the upper rotating body 51 based on position information acquired using a satellite positioning system such as a Global Navigation Satellite System (GNSS). In this case, the tilt detector 44D may include multiple antennas, such as multiple GNSS antennas. The tilt detector 44D may also include other sensors.

[0047] The rotation angle detector 44E may include a potentiometer. The rotation angle detector 44E may include a device that calculates the relative angle of the upper rotating body 51 or the orientation of the upper rotating body 51 based on position information acquired using the satellite positioning system. In this case, the rotation angle detector 44E may include multiple antennas, such as multiple GNSS antennas. The rotation angle detector 44E may also include other sensors.

[0048] The attitude information acquired by the attitude detection device 44 is input to the machine controller 41. The machine controller 41 transmits the input attitude information to the controller 11 of the work support device 10 via the communication device 48, the network 200, and the communication device 13. The controller 11 can calculate the attitude of the work machine 40 using the received attitude information.

[0049] The imaging device 43 acquires site images, which are images related to the work site. A vehicle 70 equipped with a container 80 (loading platform) is disposed at the work site. The imaging device 43 is a device that captures the site images for an operator who remotely operates the work machine 40 using the remote control device 20. The imaging device 43 may be disposed, for example, inside the cab 51A or outside the cab 51A. The imaging device 43 is preferably disposed, for example, in a position close to the eye level of a typical operator.

[0050] The work machine 40 may be equipped with a distance measuring device 45. The distance measuring device 45 acquires three-dimensional data (e.g., point cloud data) related to the work site and inputs the acquired three-dimensional data to the machine controller 41. The distance measuring device 45 is configured to acquire three-dimensional data (e.g., point cloud data) of objects present at the work site, such as the ground, obstacles, and vehicle 70. Specific examples are as follows:

[0051] The ranging device 45 may be a ranging sensor that acquires distance information regarding the distance to an object by irradiating light such as a laser beam. The ranging sensor may be, for example, a LiDAR (Light Detection and Ranging). The ranging device 45 may also be a stereo camera, a total station, or another sensor capable of acquiring three-dimensional data. The ranging device 45 may also be a combination of two or more of these devices. The ranging sensor may be attached to the work machine body of the work machine 40, may be installed at the work site where the work machine 40 is located, or may be attached to a wirelessly controlled unmanned aerial vehicle (a so-called drone).

[0052] When a ranging sensor is installed at a work site or attached to a drone, the results of distance measurement by the ranging sensor are input from the ranging sensor to the machine controller 41 via wired or wireless communication. Specifically, for example, a total station installed at the work site may measure the distance between a predetermined point on the container 80 and a predetermined point on the work implement 52 of the work machine 40, and the results of distance measurement may be input from the total station to the machine controller 41. Alternatively, a LiDAR or stereo camera installed at the work site, or a LiDAR or stereo camera attached to the drone, may acquire three-dimensional data of the distance measurement range including the work machine 40 and the container 80, and the acquired three-dimensional data may be input from the LiDAR or stereo camera to the machine controller 41.

[0053] If the imaging device 43 has both the function of acquiring site images and the function of acquiring three-dimensional data (for example, point cloud data), the distance measuring device 45 can be omitted from the work machine 40. The remote control device 20 may also be equipped with a setting interface 29, and the range of the container 80 on the vehicle 70 may be input into the setting interface 29. In this case, the distance measuring device 45 can be omitted from the work machine 40. The setting interface 29 may set the range (for example, coordinates) of the container 80 at the work site based on an input operation performed by a person involved in the work, such as an operator. The person involved in the work may perform the input operation while looking at a remote output interface (for example, a display device 23 such as a display).

[0054] Specifically, for example, if the container 80 is rectangular as shown in FIG. 4 , the controller 11 may set the extent of the container 80 by identifying the positions of three or more of the four corners of the rectangle. The position of a corner may be identified, for example, by the operator operating the operation lever 22 to position the tip of the bucket 55 at a position corresponding to the corner, and then performing an operation (coordinate identification operation) on the setting interface 29. The controller 11 can calculate the position of the tip of the bucket 55 based on the attitude information input from the attitude detection device 44. The position of a corner may also be identified, for example, by a worker inputting the coordinates of the corner into the setting interface 29 of the remote operation device 20.

[0055] The remote control device 20 is a device for remotely operating the work machine 40. The remote control device 20 is equipped with a remote control lever 22 that receives operations for remotely operating the work machine 40, a display device 23, and a seat 28. The remote control device 20 is placed in a location remote from the work machine 40, and is a device for remotely operating the work machine 40.

[0056] The display device 23 may be a display that displays a site image of the work site where the container 80 (cargo bed) of the vehicle 70 is placed, as shown in Fig. 4, for example. The controller 11 causes the display device 23 to display an index image G1 superimposed on the site image of the work site where the container 80 (cargo bed) of the vehicle 70 is placed, as shown in Fig. 6, for example. As shown in Fig. 3, the remote control device 20 may further include, in addition to the display device 23, a second display device 25 or a third display device 26, for example. The remote control device 20 may also further include a speaker 27 for outputting sounds of the work site.

[0057] Each of the various positions required for control for remote operation, such as the position of the imaging device 43, the position of the ranging device 45, the range of the container 80, and the position of the work machine 40 (for example, the position of the bucket 55), may be information represented by coordinates in a coordinate system based on the imaging device 43 (imaging device coordinate system), may be information represented by coordinates in a coordinate system based on the work machine 40 (machine coordinate system), may be information represented by coordinates in a coordinate system based on a specific position at the work site (site coordinate system), or may be information represented by coordinates in a global coordinate system. The controller 11 may be configured to be able to convert position information in any of these coordinate systems into position information in another coordinate system.

[0058] In the following specific example, the controller 11 is configured to convert point cloud data input from the distance measuring device 45 into point cloud data in a predetermined reference coordinate system. The controller 11 may also be configured to convert on-site image data input from the imaging device 43 into data in a predetermined reference coordinate system. The reference coordinate system may be the machine coordinate system, the on-site coordinate system, the global coordinate system, or another coordinate system.

[0059] The origin of the reference coordinate system may be set at a specific position relative to the work machine 40, for example. Specifically, the origin of the reference coordinate system may be, for example, any position on the swing axis. More specifically, the origin of the reference coordinate system may be the intersection of the swing axis and the ground. Furthermore, the origin of the reference coordinate system may be, for example, on the swing axis and at a position between the lower traveling body 50 and the upper swing body 51.

[0060] In this embodiment, the reference coordinate system is a three-dimensional coordinate system. In this case, the reference coordinate system may be, for example, an orthogonal coordinate system defined by an x-axis parallel to the left-right direction, a y-axis parallel to the front-rear direction, and a z-axis parallel to the vertical direction. However, the reference coordinate system is not limited to the above specific example, and various other aspects can be adopted.

[0061] The controller 11 of the work support device 10 includes a computer including a processing unit and a memory. The controller 11 has various functions, such as an image processing function for generating an index image G1. The processing unit executes a program stored in the memory, thereby realizing each of the various functions of the controller 11.

[0062] The controller 11 may determine the range of the container 80 of the vehicle 70 based on the point cloud data input from the distance measuring device 45 .

[0063] The remote controller 21 of the remote operation device 20 includes a computer including an arithmetic processing unit and a memory. The remote controller 21 controls the operation of the remote operation device 20, such as the operation of the display device 23. The remote controller 21 realizes various functions by the arithmetic processing unit executing programs stored in the memory.

[0064] The machine controller 41 of the work machine 40 is equipped with a computer including an arithmetic processing unit and memory. The machine controller 41 controls the operation of the work machine 40. The machine controller 41 outputs the posture information and the site image to the work support device 10. The machine controller 41 realizes each of its various functions by having the arithmetic processing unit execute programs stored in memory.

[0065] FIG. 16 is a flowchart showing an example of calculation processing performed by the controller 11 of the work support device 10, the remote controller 21 of the remote operation device 20, and the machine controller 41 of the work machine 40.

[0066] The remote controller 21 of the remote operation device 20 determines whether or not the input interface of the remote operation device 20 has received a designation operation (step S201 in FIG. 16 ). The designation operation may be, for example, a preset operation indicating the start of remote operation of the work machine 40. The designation operation may also be an operation given to the input interface by the operator. If the input interface has received the designation operation (YES in step S201), the remote controller 21 transmits an environment confirmation request to the work machine 40 to request a site image of the work site (step S202).

[0067] When the machine controller 41 of the work machine 40 receives the environment confirmation request (step C40), it acquires information about the site image captured by the imaging device 43 and information about the three-dimensional data acquired by the distance measuring device 45, and transmits the acquired site image and three-dimensional data to the work support device 10 (step S401). The machine controller 41 acquires posture information detected by the posture detection device 44, and transmits the acquired posture information to the work support device 10 (step S402).

[0068] The controller 11 of the work support device 10 receives the work site image and the three-dimensional data transmitted from the work machine 40 (step C11). The controller 11 transmits information related to the received work site image to the remote control device 20 (step S101).

[0069] The remote controller 21 of the remote operation device 20 receives the site image transmitted by the controller 11 of the work support device 10 (step C21).

[0070] The remote controller 21 controls the operation of the display device 23 so that the received site image is displayed on the display device 23 (step S203). That is, the display device 23 displays the site image as shown in FIG.

[0071] The controller 11 of the work support device 10 receives the attitude information transmitted by the work machine 40 (step C12).

[0072] The controller 11 of the work support device 10 recognizes the range of the container 80 based on the three-dimensional data (e.g., point cloud data) of the container 80 acquired by, for example, the distance measuring device 45 (step S102). Note that if the database 12 stores information about the range of the container 80, the controller 11 may read the information about the range of the container 80 from the database 12 and identify the range of the container 80 based on the read information (step S102).

[0073] Based on the posture information input from the posture detection device 44 and the range of the container, the controller 11 determines whether the specified part of the bucket 55 (tip 55a or front end 55b) is in a position corresponding to the container 80 (specifically, for example, whether the specified part is within the range of the container 80) (step S103).

[0074] If the specified portion of the bucket 55 is at a position corresponding to the container 80 (YES in step S103), the controller 11 generates an image as an index image G1 in which the attachment marker G3 is positioned at a position corresponding to the virtual surface image G2, as shown in Figure 7 or Figure 8 (step S105).

[0075] If the specified part of the bucket 55 is shifted in the forward / backward or left / right direction relative to the container 80 (NO in step S103), the controller 11 generates an image as an index image G1 in which the attachment marker G3 is positioned at a position shifted in the forward / backward or left / right direction relative to the virtual surface image G2, as shown in Figure 9 (step S104).

[0076] The controller 11 outputs an index image display command to cause the display device 23 to display the index image G1 including the virtual surface image G2 and the attachment marker G3 (step S106).

[0077] The remote controller 21 of the remote operation device 20 receives the index image display command transmitted by the controller 11 of the work support device 10 (step C22).

[0078] In response to the received index image display command, the remote controller 21 of the remote operation device 20 causes the display device 23 to display an index image G1 superimposed on a work site image where the container 80 (cargo bed) of the vehicle 70 is placed, for example, as shown in Fig. 6 (step S204). That is, the display device 23 displays the index image G1 corresponding to the index image display command superimposed on the work site image.

[0079] Furthermore, the remote controller 21 of the remote operation device 20 transmits to the work machine 40 a remote operation command, which is a command corresponding to the operation (remote operation) by the operator received by the remote operation lever 22 (step S205).

[0080] When the machine controller 41 of the work machine 40 receives the remote operation command transmitted from the remote operation device 20 (step C41), it controls the control objects so that the work implement 52 and / or the upper rotating body 51 operate in accordance with the remote operation command (step S403). The control objects may include, for example, the boom cylinder 56, the arm cylinder 57, the bucket cylinder 58, the swing motor 59, and a control valve that hydraulically controls the operation of the travel motor. In this case, the machine controller 41 can cause the work machine 40 to operate in accordance with the remote operation command by inputting a command in accordance with the remote operation command to the control valve.

[0081] [Modifications] Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments and includes, for example, the following modifications.

[0082] (A) Regarding the attachment marker In the above embodiment, the attachment marker G3 has a conical or triangular pyramid shape with the apex located at the bottom, but it may also have a shape that projects the outline of the bucket 55 onto the virtual surface image G2 directly below the bucket 55, as shown in Figures 17 and 18.

[0083] 19, the index image G1 may include the container range image such as a virtual plane image G2, an attachment marker G3, and a movable range image that is a virtual image that indicates the relative position of the movable range of the work device 52 with respect to the container 80. The attachment marker G3 shown in Fig. 19 is an image in which the outline of the bucket 55 is projected onto the container range image directly below the bucket.

[0084] 19 , the movable range image may include a forward movable range marker G8 and a rearward movable range marker G9. The forward movable range marker G8 is an image projected onto the container range image of the range of movement of the work device 52 that is forward of the attachment marker G3 and that can be reached by the tip of the bucket 55. The rearward movable range marker G9 is an image projected onto the container range image of the range of movement of the work device 52 that is rearward of the attachment marker G3 and that can be reached by the tip of the bucket 55.

[0085] Also, as shown in Figure 20, when the specified part of the bucket 55 is in a position corresponding to the container 80, the controller 11 of the work support device 10 may generate an image as an index image G1 in which the attachment marker G3 is placed on the surface of the soil 90 directly below the bucket 55.

[0086] (B) Regarding the Remote Control Device In the above embodiment, the Internet and a mobile phone network are exemplified as networks for communication between the work assistance device 10 and the remote control device 20. However, the work assistance device 10 and the remote control device 20 may be configured to communicate via a network such as a wired LAN (Local Area Network) or a CAN (Controller Area Network). The remote control device 20 and the work assistance device 10 do not necessarily have to be configured as separate devices, and may be integrated.

[0087] The remote operation device 20 may be provided with the work assistance device 10. In this case, the remote controller 21 of the remote operation device 20 combines the functions of the remote controller 21 of the remote operation device 20 described in the above embodiment and the functions of the controller 11 of the work assistance device 10, thereby constituting a controller in the present disclosure. Specifically, the remote controller 21 of the remote operation device 20 may perform not only the arithmetic processing by the remote controller 21 in the flowchart shown in FIG. 16 , but also the arithmetic processing by the controller 11 of the work assistance device 10 in the flowchart shown in FIG. 16 .

[0088] (C) Regarding the Release Area Range Image In the above embodiment, the controller 11 of the work support device 10 generates, as the container range image, the virtual opening region image G21 shown in Figures 6 and 7 or the virtual bottom surface image G22 shown in Figures 11 to 13, but is not limited to these. For example, as shown in Figure 21, the controller 11 may generate, as the container range image, a virtual surface image G2 that represents the range of the container 80 in the front-rear and left-right directions and that passes through the top of the soil 90.

[0089] 22 , for example, the controller 11 of the work support device 10 may generate, as the container range image, a virtual frame image G5 that is an image of a virtual frame that represents the range of the container 80 in the front-to-rear direction and the left-to-right direction. In this modification, the index image G1 including the virtual frame image G5 displayed on the display device 23 makes it easier for the operator to grasp not only the relative position of the bucket 55 with respect to the container 80 in the front-to-rear direction but also the relative position in the left-to-right direction. Therefore, during the soil removal work, the operator can appropriately adjust the relative position of the bucket 55 with respect to the container 80 in the front-to-rear direction and the left-to-right direction while looking at the index image G1 displayed on the display device 23. Specifically, this is as follows.

[0090] Fig. 22 is a diagram showing yet another example of a display screen displayed on the display device 23 of the remote control device 20. In this modification, the controller 11 generates an image including a virtual frame image G5 as an open area range image and an attachment marker G3 as an index image G1, and outputs an index image display command for displaying the index image G1 on the display device 23, causing the display device 23 to display an image in accordance with the index image display command, as shown in Fig. 22. The image shown in Fig. 22 is a view of the container 80 as seen from directly above.

[0091] The attachment marker G3 may include a marker point image G31 and an auxiliary line G32, as shown in the left diagram (A) and the center diagram (B) of FIG.

[0092] The marker point image G31 is an image that represents the relative position of the tip attachment 55 with respect to the container 80, and may be an image that represents a position directly below the predetermined portion of the bucket 55 (the tip portion 55a or the front end portion 55b). The auxiliary line G32 may include a front-to-rear line that passes through a position directly below the predetermined portion of the bucket 55 and is parallel to the front-to-rear direction, and a left-to-right line that passes through a position directly below the predetermined portion of the bucket 55 and is parallel to the left-to-right direction. The front-to-rear line may be a line that extends from the front end of the container 80 to the rear end of the container 80, and the left-to-right line may be a line that extends from the left end of the container 80 to the right end of the container 80. Furthermore, as shown in the right diagram (C) of FIG. 22 , the attachment marker G3 may include the auxiliary line G32 but not the marker point image G31.

[0093] 23 is a diagram showing yet another example of a display screen displayed on the display device 23 of the remote control device 20. In this modification, the controller 11 generates an image including a virtual frame image G5 as an open area range image and an attachment marker G3 as an index image G1, and outputs an index image display command to cause the display device 23 to display the index image G1, and the display device 23 thereby displays an image in accordance with the index image display command, as shown in FIG.

[0094] The attachment marker G3 includes an auxiliary line G32. The auxiliary line G32 may include a longitudinal line that passes directly below the predetermined portion of the bucket 55 and is parallel to the longitudinal direction, and an arc that passes directly below the predetermined portion of the bucket 55 and has its center at the center of rotation of the upper rotating body 51. The arc may be a curve that extends from the left end of the container 80 to the right end of the container 80.

[0095] Fig. 24 is a diagram showing yet another example of the display screen displayed on the display device 23 of the remote control device 20. In this modification, the controller 11 generates an image including a virtual plane image G2 or a virtual frame image G5 as an open area range image and an attachment marker G3 as an index image G1, and outputs an index image display command to cause the display device 23 to display the index image G1, causing the display device 23 to display an image in accordance with the index image display command, as shown in the left diagram (A) in Fig. 24, for example.

[0096] In this modification, the controller 11 changes the display mode of the index image G1 on the display device 23 based on the distance between the end of the container 80 and the working device 52 (e.g., the bucket 55). Specifically, when the distance between the end of the container 80 and the working device 52 becomes equal to or less than a predetermined threshold Th, the controller 11 may change the display mode of the index image G1 on the display device 23, for example, as shown in the upper diagram (B), the center diagram (C), or the lower diagram (D) in FIG. 24 . In the specific example of FIG. 24 , the controller 11 determines whether or not to change the display mode based on the distance between the front end of the container 80 and the working device 52 (e.g., the bucket 55). However, the controller 11 may also determine whether or not to change the display mode based on the distance between the rear end, right end, or left end of the container 80 and the working device 52.

[0097] In the specific example shown in the upper diagram (B) of Figure 24, when the distance between the front end of the container 80 and the bucket 55 of the work device 52 becomes equal to or less than a predetermined threshold value Th, the controller 11 changes the display mode of the end (e.g., front end G2A) of the virtual surface image G2 or the virtual frame image G5 of the index image G1.

[0098] In a specific example shown in the central figure (C) in Figure 24, when the distance between the front end of the container 80 and the bucket 55 of the work device 52 becomes equal to or less than a predetermined threshold value Th, the controller 11 changes the overall display mode of the virtual surface image G2 of the index image G1, or changes the display mode inside the virtual frame image G5.

[0099] In a specific example shown in the lower figure (D) of Figure 24, when the distance between the front end of the container 80 and the bucket 55 of the work device 52 becomes less than a predetermined threshold value Th, the controller 11 changes the display mode of the attachment marker G3 in the index image G1.

[0100] In the modification shown in FIG. 24, the change in the display mode of the image may be, for example, a change in the color of the image, or may be causing at least a part of the image to blink.

[0101] However, the change in the display mode is not limited to the specific example shown in FIG.

[0102] FIG. 25 is a diagram showing yet another example of a display screen displayed on the display device 23 of the remote control device 20. In the modified example of FIG. 25 , similar to the left diagram (A) and the center diagram (B) of FIG. 22 , the attachment marker G3 includes a marker point image G31 and an auxiliary line G32. The auxiliary line G32 includes a front-rear straight line that passes directly below the predetermined portion of the bucket 55 and is parallel to the front-rear direction, and a left-right straight line that passes directly below the predetermined portion of the bucket 55 and is parallel to the left-right direction. Each of the front-rear straight line and the left-right straight line may be positioned so as to pass through the lower end of the marker point image G31. The front-rear straight line may be a straight line extending from the front end of the container 80 to the rear end of the container 80, and the left-right straight line may be a straight line extending from the left end of the container 80 to the right end of the container 80.

[0103] In this modification, the controller 11 may change the display mode of the cab rear wall image G6 on the display device 23 based on the distance between the rear wall 72 of the vehicle cab 71 and the working device 52 (e.g., the bucket 55), and may cause the display device 23 to display an alert to notify that the distance between the rear wall 72 of the vehicle cab 71 and the working device 52 has become small. Specifically, when the distance between the rear wall 72 of the vehicle cab 71 and the working device 52 becomes equal to or smaller than a predetermined threshold, the controller 11 may change the display mode of the cab rear wall image G6 on the display device 23, and may cause the display device 23 to display the alert.

[0104] The change in the display mode of the cab rear wall image G6 may be, for example, a change in color of the cab rear wall image G6 or may be blinking at least a part of the cab rear wall image G6. The alert display may include, for example, text or a mark in a conspicuous color.

[0105] (D) Regarding the work support device In the above embodiment, an example was given of a case where an operator remotely controls the work machine 40 using the remote control device 20, but the work support device 10 in the present disclosure is not limited to remote control, and may also be applied to a case where an operator gets on board the work machine 40 and operates the control lever 42.

[0106] In the above embodiment, the controller 11 of the work support device 10 transmits the on-site image to the remote controller 21 of the remote operation device 20 in step S101 of FIG. 16 , but the on-site image may also be transmitted to the remote controller 21 of the remote operation device 20 when outputting the index image display command in step S106, for example.

[0107] In the above embodiment, the controller 11 of the work assistance device 10 transmits the work site image to the remote controller 21 of the remote operation device 20 in step S101, transmits the index image display command to the remote controller 21 of the remote operation device 20 in step S106, and the remote controller 21 of the remote operation device 20 controls the display device 23 based on the received index image display command to display the index image G1 superimposed on the work site image of the work site where the container 80 (bed) of the vehicle 70 is placed. However, this is not necessarily limited to the above embodiment. For example, in step 106, the controller 11 of the work assistance device 10 may generate an index image G1 in accordance with the work site image and the index image display command, further generate a superimposed image by superimposing the index image G1 on the work site image, and transmit the generated superimposed image to the remote controller 21 of the remote operation device 20, and the remote controller 21 of the remote operation device 20 may display the received superimposed image on the display device 23.

[0108] (E) Work Support Device for Simulator In the above embodiment, an example was given of a case in which an operator remotely controls an actual work machine 40 located at a work site in real space using a remote control device 20, but the work support device in the present disclosure may also be applied to a simulator configured so that an operator controls a virtual work machine located at a work site constructed in a virtual space. Specifically, this is as follows.

[0109] 26 is a block diagram showing an example of a work assistance system 300 in which the work assistance device 10 is applied to the simulator described above. The work assistance system 300 includes the work assistance device 10, a remote control device 20, and an operation simulator device 60.

[0110] The operation simulator device 60 places a virtual work site in a virtual space created by the operation simulator device 60, and places a virtual work machine 40A at the work site. The operation simulator device 60 displays images of the work site and work machine 40A in the virtual space on the display device 23 of the remote control device 20. The operator can operate the work machine 40A at the work site in the virtual space by operating the remote control lever 22 of the remote control device 20.

[0111] The operation simulator device 60 includes a computer and communications equipment. The computer of the operation simulator device 60 performs calculations to operate the work machine 40A in a virtual space. The virtual space is a virtual three-dimensional space, and the work machine 40A, which is a three-dimensional model simulating the work machine 40 in real space, is placed within the virtual space. The work site constructed in the virtual space is a simulation of the work site in real space, and includes, for example, a three-dimensionally modeled ground surface, which is configured so that it can be excavated by the work machine 40A. A virtual vehicle simulating the vehicle 70 in real space, i.e., a three-dimensionally modeled vehicle (e.g., a vehicle equipped with a loading platform for loading earth and sand), may be placed at the work site in the virtual space. Furthermore, objects simulating devices, equipment, and other objects present at the work site in real space, i.e., three-dimensionally modeled objects, may be placed at the work site in the virtual space. The communications equipment of the operation simulator device 60 is configured to be able to communicate with external devices such as the work assistance device 10 and the remote operation device 20 via the network 200. The network 200 may be a network such as the Internet or a mobile phone network, or may be a network such as a wired LAN or CAN. The communication device of the operation simulator device 60 receives information transmitted from the external device and inputs the information to the computer of the operation simulator device 60. The communication device of the operation simulator device 60 may transmit the results of calculations performed by the computer to the external device.

[0112] A work machine 40A modeled three-dimensionally in virtual space includes components that simulate the undercarriage 50, upper revolving body 51, and work implement 52 of the work machine 40 in real space shown in Figure 2. The computer of the operation simulator device 60 may perform calculations so that the work machine 40A in virtual space simulates the operation of various components such as the control lever 42 and imaging device 43 that are included in the work machine 40 in real space shown in Figure 1. In this case, the work machine 40A in virtual space performs operations that simulate inputs received by the work machine 40 in real space and outputs corresponding to these inputs. For example, as shown in Figure 26, the work machine 40A in virtual space may include various modeled components, namely, a machine controller 41A, control lever 42A, imaging device 43A, a group of sensors (attitude detection device 44, distance measurement device 45A), and communication equipment 48A.

[0113] Specifically, when the operator operates the remote control lever 22 of the remote control device 20, a remote control command is transmitted from the remote control device 20 to the operation simulator device 60, and the computer of the operation simulator device 60 performs calculations regarding the operation of the work machine 40A in virtual space in accordance with the received remote operation command. For example, the modeled imaging device 43A acquires images of a work site constructed in virtual space. The modeled distance measuring device 45A acquires three-dimensional data of the work site constructed in virtual space. The modeled attitude detection device 44 acquires attitude information regarding the attitude of the work machine 40A. For each of the modeled imaging device 43A and the group of sensors, device operating characteristics such as detection range and response speed, as well as output information such as data output format and output voltage range, may be modeled. The computer of the operation simulator device 60 may perform calculations to simulate the modeled machine controller 41A transmitting the operating characteristics and output information to an external device via the modeled communication device 48A. The communication device of the operation simulator device 60 may transmit the calculation result to at least one of the work assistance device 10 and the remote operation device 20 via the network 200 .

[0114] The work support device 10 and remote control device 20 of the work support system 300 shown in Fig. 26 are similar to the work support device 10 and remote control device 20 of the work support system 100 shown in Fig. 1. In the work support system 300 shown in Fig. 26, the computer of the operation simulator device 60 transmits information about the work site and the operation of the work machine 40A in the virtual space constructed in the operation simulator device 60 to the work support device 10, and the controller 11 of the work support device 10 displays, based on the received information, an index image G1 including a release area range image such as a virtual plane image G2 and an attachment marker G3 on the display device 23 of the remote control device 20, as shown in Figs. 6 to 15, for example.

[0115] In the work support system 300 equipped with the operation simulator device 60, the operator can appropriately adjust the relative position of the bucket with respect to an open area such as a container existing in the virtual space while looking at the index image G1 displayed on the display device 23.

[0116] In the specific example shown in Fig. 26 , the work support system 300 is configured such that the operation simulator device 60, the work support device 10, and the remote control device 20 communicate with each other via a network 200, but the work support system for a simulator is not limited to the specific example shown in Fig. 26 . In the work support system for a simulator, the work support device 10 may be mounted on the operation simulator device 60 or on the remote control device 20. Furthermore, if the operation simulator device 60 is equipped with an operation lever and a display device, the remote control device 20 can be omitted from the work support system. Furthermore, in the specific example shown in Fig. 26 , the operation simulator device 60, the work support device 10, and the remote control device 20 are configured as separate devices, but they may also be configured as an integrated device.

[0117] (F) Regarding the Work Machine The work machine 40 according to this embodiment is a shovel, but the work machine in this disclosure is not limited to a shovel and may be another work machine such as a crane or a bulldozer.

[0118] In the above embodiment, the Internet and a mobile phone network are exemplified as networks for communication between the work assistance device 10 and the work machine 40, but the work assistance device 10 and the work machine 40 may be configured to communicate via a network such as a wired LAN (Local Area Network) or CAN (Controller Area Network). The work assistance device 10 may be mounted on the work machine 40.

[0119] The work machine 40 may be equipped with a work support device 10. In this case, the machine controller 41 of the work machine 40 may combine the functions of the machine controller 41 of the work machine 40 described in the above embodiment and the functions of the controller 11 of the work support device 10. Specifically, the machine controller 41 of the work machine 40 may perform not only the arithmetic processing by the machine controller 41 in the flowchart shown in Figure 16, but also the arithmetic processing by the controller 11 of the work support device 10 in the flowchart shown in Figure 16.

[0120] (G) The present disclosure includes a program according to the following modified example. The program according to this modified example is a program for assisting in the work of releasing soil 90 held in the bucket 55 of a work machine 40 from the bucket 55 and pouring the soil 90 into a container 80 located in front of the work machine 40, and causes a computer to implement the function of displaying, on the display device 23, an index image G1 that includes a container range image, which is a virtual image that serves as an index indicating the range of the container 80 in at least the fore-and-aft direction, and an attachment marker G3, which is a virtual image that indicates the relative position of the bucket 55 with respect to the container 80. This program may be provided via communications means (for example, a network such as the Internet), or may be provided stored on various recording media (for example, non-transitory tangible recording media).

[0121] (H) The present disclosure includes a work assistance method according to the following modified example. This work assistance method according to this modified example is a method for assisting in the work of releasing soil 90 held in the bucket 55 of a work machine 40 from the bucket 55 and pouring the soil 90 into a container 80 located in front of the work machine 40, and includes displaying on the display device 23 an index image G1 that includes a container range image, which is a virtual image that serves as an index indicating the range of the container 80 in at least the front-to-rear direction, and an attachment marker G3, which is a virtual image that indicates the relative position of the bucket 55 with respect to the container 80.

[0122] As described above, the present disclosure provides a technique that can assist an operator in grasping the relative position in the front-to-rear direction of a tip attachment with respect to an open area of ​​a container, etc. The present disclosure includes the following first to fifteenth aspects.

[0123] A work assistance device according to a first aspect is a device for assisting in the work of releasing an object held on a tip attachment of a work machine from the tip attachment and placing the object in a release area located in front of the work machine, and the work assistance device is equipped with a controller that causes a display device to display an index image that includes a release area range image, which is a virtual image that serves as an index indicating the range of the release area in at least the forward / backward direction, and an attachment marker, which is a virtual image that indicates the relative position of the tip attachment with respect to the release area.

[0124] In the work assistance device according to the first aspect, the controller causes the display device to display the index image including the released area range image and the attachment marker, making it easier for the operator to grasp the relative position of the end attachment in the fore-and-aft direction with respect to the released area from the index image displayed on the display device. Therefore, in the operation of releasing the object from the end attachment and placing the object in the released area (releasing operation), the operator can appropriately adjust the relative position of the end attachment in the fore-and-aft direction with respect to the released area while viewing the index image displayed on the display device.

[0125] A work assistance device according to a second aspect preferably includes the following configuration in addition to the work assistance device according to the first aspect. That is, in the work assistance device according to the second aspect, the controller preferably generates, as the released area range image, a virtual surface image that is an image of a virtual surface representing the extent of the released area in the front-rear direction and the left-right direction, or a virtual frame image that is an image of a virtual frame representing the extent of the released area in the front-rear direction and the left-right direction. In this second aspect, the indicator image including the virtual surface image or the virtual frame image displayed on the display device makes it easy for the operator to grasp not only the relative position of the end attachment with respect to the released area in the front-rear direction but also the relative position in the left-right direction. Therefore, during the release work, the operator can appropriately adjust the relative position of the end attachment with respect to the released area in the front-rear direction and the left-right direction while viewing the indicator image displayed on the display device.

[0126] A work assistance device according to a third aspect preferably includes the following configuration in addition to the work assistance device according to the first aspect. That is, in the work assistance device according to the third aspect, the release area is preferably a container disposed in front of the work machine, and the controller preferably generates, as the release area range image, a virtual surface image (virtual opening area image) that is an image of a virtual surface representing an area corresponding to the opening of the container or a virtual surface image (virtual bottom image) that is an image of a virtual surface representing an area corresponding to the bottom of the container. In this third aspect, the indicator image including the virtual opening area image or the virtual bottom image displayed on the display device makes it easier for the operator to grasp the relative positions in the front-to-rear direction and the left-to-right direction of the tip attachment with respect to the opening of the container or the bottom of the container. Therefore, during the release work, the operator can appropriately adjust the relative positions in the front-to-rear direction and the left-to-right direction of the tip attachment with respect to the container while viewing the indicator image displayed on the display device.

[0127] A work assistance device according to a fourth aspect preferably includes the work assistance device according to the second or third aspect further including the following configuration. That is, in the work assistance device according to the fourth aspect, when a predetermined portion of the end attachment is located in a position corresponding to the released area, the controller preferably generates, as the index image, an image in which the attachment marker is located in a position corresponding to the virtual surface image. Specifically, for example, when the predetermined portion of the end attachment is inside or directly above the released area, the controller may generate, as the index image, an image in which the attachment marker is located directly below the end attachment at a position on the virtual surface image, or may generate, as the index image, an image in which the attachment marker is located directly below the end attachment at a position spaced above the virtual surface image. Note that when the predetermined portion is located in a position corresponding to the released area, the predetermined portion may be located on the released area or may be located directly above the released area (a position spaced above the released area).

[0128] A work assistance device according to a fifth aspect preferably includes the following configuration in addition to the work assistance device according to the fourth aspect. That is, in the work assistance device according to the fifth aspect, when the predetermined portion is shifted in the front-rear direction or the left-right direction with respect to the released area, the controller preferably generates, as the indicator image, an image in which the attachment marker is positioned at a position shifted in the front-rear direction or the left-right direction with respect to the virtual plane image. Specifically, for example, when the predetermined portion is shifted in the front-rear direction or the left-right direction with respect to the released area, the controller may generate, as the indicator image, an image in which the attachment marker is positioned at a position corresponding to the ground surface around the released area.

[0129] A work assistance device according to a sixth aspect may be the work assistance device according to any one of the second to fifth aspects, further including the following configuration. That is, in the work assistance device according to the sixth aspect, the open area is a container placed in front of the work machine, and when a portion of the object contained in the container is located higher than the height of the virtual plane image, the controller may generate, as the indicator image, an image in which an area of ​​the virtual plane image corresponding to the portion of the object is omitted. In this sixth aspect, because the portion of the object does not overlap with the virtual plane image, the operator can easily grasp the status of the object, such as the position of the object in the container and the bias of the object.

[0130] A work assistance device according to a seventh aspect may be the work assistance device according to any one of the first to sixth aspects, further including the following configuration. That is, in the work assistance device according to the seventh aspect, the release area is a container disposed in front of the work machine, and the controller may generate, as the release area range image, a virtual surface image (virtual bottom surface image) that is an image of a virtual surface representing an area corresponding to a bottom surface of the container, and generate, as the index image, an image in which the attachment marker is positioned directly below the tip attachment and at a position on the virtual surface image or a position spaced above from the virtual surface image. In this seventh aspect, the attachment marker is displayed on the display device directly below the tip attachment and at a position corresponding to the virtual bottom surface image (a position on the virtual bottom surface image or a position spaced above from the virtual bottom surface image), making it easier for the operator to grasp the relative positions of the tip attachment in the front-to-rear direction and the left-to-right direction with respect to the container during the release work.

[0131] A work assistance device according to an eighth aspect preferably includes the following configuration in addition to the work assistance device according to the seventh aspect. That is, in the work assistance device according to the eighth aspect, it is preferable that the attachment marker is a first attachment marker located directly below the tip attachment and at a distance above the virtual surface image, and the controller generates, as the index image, an image including the release area range image, the first attachment marker, and a second attachment marker that is a virtual image located directly below the tip attachment and at a position on the virtual surface image. In this eighth aspect, since the first attachment marker is located at a distance above the virtual bottom surface image and the second attachment marker is located at a position on the virtual bottom surface image on the display device, during the release operation, the operator can more easily grasp the relative position of the tip attachment in the front-to-rear direction and the relative position in the left-to-right direction with respect to the bottom surface of the container.

[0132] A work assistance device according to a ninth aspect may further include the following configuration in the work assistance device according to any one of the first to eighth aspects. That is, in the work assistance device according to the ninth aspect, the released area may be a loading platform of a vehicle transporting the object, the vehicle may have a vehicle cab, and the controller may generate, as the index image, an image including the released area range image, the attachment marker, and a cab rear wall image that is a virtual image representing an area corresponding to the rear wall of the vehicle cab. In this ninth aspect, during the release operation, the operator can easily grasp not only the relative position of the end attachment with respect to the released area, but also the relative position of the end attachment with respect to the rear wall of the vehicle cab. In this ninth aspect, it is more preferable that the cab rear wall image be displayed on the display device in a display mode different from that of the released area range image. In this case, during the release operation, the operator can easily distinguish between the loading platform as the released area and the rear wall of the vehicle cab.

[0133] A work assistance system according to a tenth aspect includes a work assistance device according to any one of the first to ninth aspects, the work machine, and a remote control device having the display device and for remotely operating the work machine.

[0134] The work machine according to the eleventh aspect comprises a work machine body, an attitude detection device that acquires attitude information relating to the attitude of the work machine body, an imaging device that acquires a site image that is an image relating to the work site where the release area is present, and a machine controller that outputs the attitude information and the site image to a work support device according to any one of the first to ninth aspects.

[0135] The remote control device according to the twelfth aspect includes a remote control lever that receives operations for remotely operating the work machine, and the display device that displays the index image including the release area range image and the attachment marker in response to a command (index image display command) output by a work support device according to any one of the first to ninth aspects.

[0136] The remote control device according to the thirteenth aspect preferably further comprises the following configuration in addition to the remote control device according to the twelfth aspect: That is, in the remote control device according to the thirteenth aspect, it is preferable that the controller causes the display device to display the index image superimposed on a site image of the work site where the released area exists.

[0137] A program according to a fourteenth aspect is a program for assisting in the task of releasing an object held on a tip attachment of a work machine from the tip attachment and placing the object in a release area located in front of the work machine, and causes a computer to realize the function of displaying on a display device an index image that includes a release area range image, which is a virtual image that serves as an indicator representing the range of the release area in at least the forward / backward direction, and an attachment marker, which is a virtual image that represents the relative position of the tip attachment with respect to the release area.

[0138] A work assistance method according to a fifteenth aspect is a method for assisting in the work of releasing an object held on a tip attachment of a work machine from the tip attachment and placing the object in a release area located in front of the work machine, and includes displaying on a display device an index image that includes a release area range image, which is a virtual image that serves as an indicator representing the range of the release area in at least the forward / backward direction, and an attachment marker, which is a virtual image that represents the relative position of the tip attachment with respect to the release area.

Claims

1. A work support device for supporting the task of releasing an object held on a tip attachment of a work machine from the tip attachment and placing the object in a release area located in front of the work machine, the work support device comprising a controller that causes a display device to display an index image including a release area range image, which is a virtual image that serves as an index showing the range of the release area in at least the forward / backward direction, and an attachment marker, which is a virtual image that shows the relative position of the tip attachment with respect to the release area.

2. The work support device of claim 1, wherein the controller generates, as the released area range image, a virtual surface image that is an image of a virtual surface that represents the range of the released area in the forward / backward and left / right directions, or a virtual frame image that is an image of a virtual frame that represents the range of the released area in the forward / backward and left / right directions.

3. The work support device according to claim 1, wherein the release area is a container located in front of the work machine, and the controller generates, as the release area range image, a virtual surface image that is an image of a virtual surface representing an area corresponding to the opening of the container or a virtual surface image that is an image of a virtual surface representing an area corresponding to the bottom surface of the container.

4. A work support device as described in claim 2 or 3, wherein when a predetermined portion of the tip attachment is in a position corresponding to the release area, the controller generates an image as the index image in which the attachment marker is positioned in a position corresponding to the virtual surface image.

5. The work support device of claim 4, wherein when the specified portion is shifted in the forward / backward or left / right direction relative to the released area, the controller generates as the index image an image in which the attachment marker is positioned at a position shifted in the forward / backward or left / right direction relative to the virtual surface image.

6. A work support device as claimed in any one of claims 2 to 5, wherein the release area is a container placed in front of the work machine, and when a part of the object contained in the container is located above the height of the virtual plane image, the controller generates as the index image an image in which an area corresponding to the part of the object in the virtual plane image is omitted.

7. A work support device as described in any one of claims 1 to 6, wherein the release area is a container placed in front of the work machine, and the controller generates as the release area range image a virtual surface image that is an image of a virtual surface that represents an area corresponding to the bottom surface of the container, and generates as the indicator image an image in which the attachment marker is positioned directly below the tip attachment at a position on the virtual surface image or at a position above and spaced apart from the virtual surface image.

8. The work support device described in claim 7, wherein the attachment marker is a first attachment marker located directly below the tip attachment and above the virtual surface image, and the controller generates as the index image an image including the release area range image, the first attachment marker, and a second attachment marker which is a virtual image located directly below the tip attachment and at a position on the virtual surface image.

9. A work assistance device as described in any one of claims 1 to 8, wherein the release area is a loading platform of a vehicle that transports the object, the vehicle has a vehicle cab, and the controller generates, as the index image, an image including the release area range image, the attachment marker, and a cab rear wall image that is a virtual image representing an area corresponding to the rear wall of the vehicle cab.

10. A work support system comprising: the work support device according to any one of claims 1 to 9; the work machine; and a remote control device having the display device and for remotely controlling the work machine.

11. A work machine comprising: a work machine body; an attitude detection device that acquires attitude information relating to the attitude of the work machine body; an imaging device that acquires a site image that is an image relating to the work site where the release area is present; and a machine controller that outputs the attitude information and the site image to a work support device described in any one of claims 1 to 9.

12. A remote control device comprising: a remote control lever that receives operations for remotely operating the work machine; and a display device that displays the index image including the release area range image and the attachment marker in response to a command output by a work support device described in any one of claims 1 to 9.

13. The remote control device according to claim 12, wherein the controller causes the display device to display the index image superimposed on a site image of the work site where the release area exists.

14. A program for assisting in the task of releasing an object held on a tip attachment of a work machine from the tip attachment and placing the object in a release area located in front of the work machine, the program enabling a computer to realize the function of displaying on a display device an index image including a release area range image, which is a virtual image that serves as an indicator showing the range of the release area at least in the forward and backward directions, and an attachment marker, which is a virtual image that shows the relative position of the tip attachment with respect to the release area.

15. A work support method for supporting the task of releasing an object held on a tip attachment of a work machine from the tip attachment and placing the object in a release area located in front of the work machine, the work support method including displaying on a display device an index image including a release area range image, which is a virtual image that serves as an index indicating the range of the release area in at least the forward / backward direction, and an attachment marker, which is a virtual image that indicates the relative position of the tip attachment with respect to the release area.

Images