Work support device, work support system, work machine, remote operation device, program, and work support method

Abstract

To provide a technique capable of supporting an operator to grasp a relative position in a longitudinal direction of a tip attachment with respect to a release area such as a container.SOLUTION: The work support device 10 is a device for supporting work of releasing an object 90 held by a tip attachment 55 of a working machine 40 from the tip attachment 55 and disposing the object 90 in a release area 80 present in front of the working machine 40. The work support device 10 includes the controller 11 that causes the display unit 23 to display the indicator image G2 including the release area range image G3, which is an imaginary image serving as an indicator indicating the range of the release area 80 at least in the front-rear direction, and the attachment marker G1, which is an imaginary image indicating the relative position of the tip attachment 55 with respect to the release area 80.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present disclosure relates to a technique for assisting work performed by a work machine. [Background technology]

[0002] Patent Document 1 discloses a display control device and a remote control device that aim to provide appropriate graphic displays for work support during a series of operations of a work machine.Patent Document 2 discloses an image display system that aims to suppress a decrease in work efficiency when working using a work machine equipped with a work implement having a work tool. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-120743 [Patent Document 2] Patent No. 6777375 Summary of the Invention [Problem to be solved by the invention]

[0004] The display control device in Patent Document 1 superimposes on a display a graphic that highlights the position of the bottom end of the bucket and a graphic that highlights the position of the top end of the container. The graphic display provided by this display control device may help the operator grasp the relative position of the bucket to the container in the up-down direction, but does not help the operator grasp the relative position of the bucket to the container in the front-to-back direction. Furthermore, the image display system in Patent Document 2 does not take into consideration helping the operator grasp the relative position of the bucket to the container in the front-to-back direction.

[0005] The present disclosure aims to provide a technology that can assist an operator in understanding the relative position in the front-to-rear direction of a tip attachment with respect to an open area of ​​a container or the like. [Means for solving the problem]

[0006] A work assistance device according to a first aspect is a device for assisting in the work of releasing an object held on a tip attachment of a work machine from the tip attachment and placing the object in a release area located in front of the work machine, and the work assistance device is equipped with a controller that causes a display device to display an index image that includes a release area range image, which is a virtual image that serves as an indicator representing the range of the release area in at least the forward / backward direction, and an attachment marker, which is a virtual image that represents the relative position of the tip attachment with respect to the release area.

[0007] In the work assistance device according to the first aspect, the controller causes the display device to display the index image including the released area range image and the attachment marker, making it easier for the operator to grasp the relative position of the end attachment in the front-to-rear direction with respect to the released area from the index image displayed on the display device. Therefore, in the work of releasing the object from the end attachment and placing the object in the released area (release work), the operator can appropriately adjust the relative position of the end attachment in the front-to-rear direction with respect to the released area while viewing the index image displayed on the display device.

[0008] A work assistance device according to a second aspect preferably includes the following configuration in addition to the work assistance device according to the first aspect. That is, in the work assistance device according to the second aspect, the controller preferably generates, as the released area range image, a virtual surface image that is an image of a virtual surface representing the range of the released area in the front-rear direction and the left-right direction, or a virtual frame image that is an image of a virtual frame representing the range of the released area in the front-rear direction and the left-right direction. In this second aspect, the indicator image including the virtual surface image or the virtual frame image displayed on the display device makes it easy for the operator to grasp not only the relative position of the end attachment with respect to the released area in the front-rear direction but also the relative position in the left-right direction. Therefore, during the release work, the operator can appropriately adjust the relative position of the end attachment with respect to the released area in the front-rear direction and the left-right direction while viewing the indicator image displayed on the display device.

[0009] A work assistance device according to a third aspect preferably includes the following configuration in addition to the work assistance device according to the first aspect. That is, in the work assistance device according to the third aspect, the release area is preferably a container disposed in front of the work machine, and the controller preferably generates, as the release area range image, a virtual surface image (virtual opening area image) that is an image of a virtual surface representing an area corresponding to the opening of the container or a virtual surface image (virtual bottom image) that is an image of a virtual surface representing an area corresponding to the bottom of the container. In this third aspect, the indicator image including the virtual opening area image or the virtual bottom image displayed on the display device makes it easier for the operator to grasp the relative positions in the front-to-rear direction and the left-to-right direction of the tip attachment with respect to the opening of the container or the bottom of the container. Therefore, during the release work, the operator can appropriately adjust the relative positions in the front-to-rear direction and the left-to-right direction of the tip attachment with respect to the container while viewing the indicator image displayed on the display device.

[0010] A work assistance device according to a fourth aspect is preferably the work assistance device according to the second or third aspect further including the following configuration. That is, in the work assistance device according to the fourth aspect, when a predetermined portion of the end attachment is in a position corresponding to the released area, the controller preferably generates, as the index image, an image in which the attachment marker is located at a position corresponding to the virtual surface image. Specifically, for example, when the predetermined portion of the end attachment is inside or directly above the released area, the controller may generate, as the index image, an image in which the attachment marker is located directly below the end attachment at a position on the virtual surface image, or may generate, as the index image, an image in which the attachment marker is located directly below the end attachment at a position spaced above the virtual surface image. Note that when the predetermined portion is located at a position corresponding to the released area, the predetermined portion may be located on the released area, or may be located directly above the released area (a position spaced above the released area).

[0011] A work assistance device according to a fifth aspect preferably includes the following configuration in addition to the work assistance device according to the fourth aspect. That is, in the work assistance device according to the fifth aspect, when the predetermined portion is shifted in the front-rear direction or the left-right direction with respect to the released area, the controller preferably generates, as the index image, an image in which the attachment marker is positioned at a position shifted in the front-rear direction or the left-right direction with respect to the virtual plane image. Specifically, for example, when the predetermined portion is shifted in the front-rear direction or the left-right direction with respect to the released area, the controller may generate, as the index image, an image in which the attachment marker is positioned at a position corresponding to the ground around the released area.

[0012] A work assistance device according to a sixth aspect may be the work assistance device according to any one of the second to fifth aspects, further including the following configuration. That is, in the work assistance device according to the sixth aspect, the open area is a container placed in front of the work machine, and when a portion of the object contained in the container is located higher than the height of the virtual plane image, the controller may generate, as the index image, an image in which an area corresponding to the portion of the object is omitted from the virtual plane image. In this sixth aspect, since the portion of the object does not overlap with the virtual plane image, the operator can easily grasp the status of the object, such as the position of the object in the container and the bias of the object.

[0013] A work assistance device according to a seventh aspect may be the work assistance device according to any one of the first to sixth aspects, further including the following configuration. That is, in the work assistance device according to the seventh aspect, the release area is a container placed in front of the work machine, and the controller may generate, as the release area range image, a virtual surface image (virtual bottom surface image) that is an image of a virtual surface that represents an area corresponding to the bottom surface of the container, and generate, as the indicator image, an image in which the attachment marker is positioned directly below the tip attachment and at a position on the virtual surface image or a position spaced above from the virtual surface image. In this seventh aspect, the attachment marker is displayed on the display device directly below the tip attachment and at a position corresponding to the virtual bottom surface image (a position on the virtual bottom surface image or a position spaced above from the virtual bottom surface image), making it easier for the operator to grasp the relative position of the tip attachment in the front-to-rear direction and the relative position in the left-to-right direction with respect to the container during the release work.

[0014] The work support device according to an eighth aspect preferably includes the following configuration in addition to the work support device according to the seventh aspect. That is, in the work support device according to the eighth aspect, it is preferable that the attachment marker is a first attachment marker located directly below the tip attachment and at a distance above the virtual surface image, and the controller generates, as the index image, an image including the release area range image, the first attachment marker, and a second attachment marker that is a virtual image located directly below the tip attachment and at a position on the virtual surface image. In this eighth aspect, since the first attachment marker is located at a distance above the virtual bottom surface image and the second attachment marker is located at a position on the virtual bottom surface image on the display device, during the release operation, the operator can more easily grasp the relative position of the tip attachment in the front-to-rear direction and the relative position in the left-to-right direction with respect to the bottom surface of the container.

[0015] A work assistance device according to a ninth aspect may be the work assistance device according to any one of the first to eighth aspects, further including the following configuration. That is, in the work assistance device according to the ninth aspect, the released area may be a loading platform of a vehicle transporting the object, the vehicle may have a vehicle cab, and the controller may generate, as the index image, an image including the released area range image, the attachment marker, and a cab rear wall image that is a virtual image representing an area corresponding to the rear wall of the vehicle cab. In this ninth aspect, during the release operation, the operator can easily grasp not only the relative position of the end attachment with respect to the released area, but also the relative position of the end attachment with respect to the rear wall of the vehicle cab. In this ninth aspect, it is more preferable that the cab rear wall image be displayed on the display device in a display mode different from that of the released area range image. In this case, during the release operation, the operator can easily distinguish between the loading platform as the released area and the rear wall of the vehicle cab.

[0016] A work assistance system according to a tenth aspect includes a work assistance device according to any one of the first to ninth aspects, the work machine, and a remote control device having the display device and for remotely operating the work machine.

[0017] The work machine according to the eleventh aspect comprises a work machine body, an attitude detection device that acquires attitude information relating to the attitude of the work machine body, an imaging device that acquires a site image that is an image relating to the work site where the release area is located, and a machine controller that outputs the attitude information and the site image to a work support device according to any one of the first to ninth aspects.

[0018] The remote control device of the twelfth aspect comprises a remote control lever that receives operations for remotely operating the work machine, and the display device that displays the index image including the release area range image and the attachment marker in response to a command (index image display command) output by the work support device of any one of the first to ninth aspects.

[0019] A remote control device according to a thirteenth aspect is preferably the remote control device according to the twelfth aspect further including the following configuration: That is, in the remote control device according to the thirteenth aspect, it is preferable that the controller causes the display device to display the index image superimposed on a site image of the work site where the released area exists.

[0020] A program according to a fourteenth aspect is a program for assisting in the task of releasing an object held on a tip attachment of a work machine from the tip attachment and placing the object in a release area located in front of the work machine, and causes a computer to realize the function of displaying on a display device an index image that includes a release area range image, which is a virtual image that serves as an indicator representing the range of the release area in at least the forward / backward direction, and an attachment marker, which is a virtual image that represents the relative position of the tip attachment with respect to the release area.

[0021] A work assistance method according to a fifteenth aspect is a method for assisting in the work of releasing an object held on a tip attachment of a work machine from the tip attachment and placing the object in a release area located in front of the work machine, and includes displaying on a display device an index image that includes a release area range image, which is a virtual image that serves as an indicator representing the range of the release area in at least the forward / backward direction, and an attachment marker, which is a virtual image that represents the relative position of the tip attachment with respect to the release area. [Effects of the Invention]

[0022] According to the present disclosure, a technique is provided that can assist an operator in grasping the relative position in the front-to-rear direction of a tip attachment with respect to an open area of ​​a container or the like. [Brief explanation of the drawings]

[0023] [Figure 1] 1 is a block diagram showing a work support system according to an embodiment; [Figure 2] 1 is a side view showing a work machine according to an embodiment. [Figure 3] 1 is a diagram illustrating a remote control device according to an embodiment. [Figure 4] FIG. 4 is a diagram showing an example of a display screen displayed on a display device of the remote control device. [Figure 5] 10A and 10B are diagrams illustrating other examples of the display screen displayed on the display device of the remote control device. [Figure 6] 10 is a diagram showing still another example of a display screen displayed on the display device of the remote control device. FIG. [Figure 7] 10A and 10B are diagrams for explaining the positional relationship between the work machine and the container of the vehicle, and an example of an index image. [Figure 8] 10A and 10B are diagrams for explaining the positional relationship between the work machine and the container of the vehicle, and other examples of index images. [Figure 9] 10A and 10B are diagrams for explaining still another example of the positional relationship between the work machine and the container of the vehicle, and an index image. [Figure 10] 10 is a diagram showing still another example of a display screen displayed on the display device of the remote control device. FIG. [Figure 11] 10 is a diagram showing still another example of a display screen displayed on the display device of the remote control device. FIG. [Figure 12] 10 is a diagram showing still another example of a display screen displayed on the display device of the remote control device. FIG. [Figure 13] 10A and 10B are diagrams for explaining still another example of the positional relationship between the work machine and the container of the vehicle, and an index image. [Figure 14] 10 is a diagram showing still another example of a display screen displayed on the display device of the remote control device. FIG. [Figure 15] 10 is a diagram showing still another example of a display screen displayed on the display device of the remote control device. FIG. [Figure 16] 5 is a flowchart illustrating an example of a calculation process performed by a controller of the work assistance device according to the embodiment. [Figure 17] 10 is a diagram showing still another example of a display screen displayed on the display device of the remote control device. FIG. [Figure 18] 10A and 10B are diagrams for explaining still another example of the positional relationship between the work machine and the container of the vehicle, and an index image. [Figure 19] 10 is a diagram showing still another example of a display screen displayed on the display device of the remote control device. FIG. [Figure 20] 10A and 10B are diagrams for explaining still another example of the positional relationship between the work machine and the container of the vehicle, and an index image. [Figure 21] 10A and 10B are diagrams for explaining still another example of the positional relationship between the work machine and the container of the vehicle, and an index image. [Figure 22] 10 is a diagram showing still another example of a display screen displayed on the display device of the remote control device. FIG. [Figure 23] 10 is a diagram showing still another example of a display screen displayed on the display device of the remote control device. FIG. [Figure 24] 10 is a diagram showing still another example of a display screen displayed on the display device of the remote control device. FIG. [Figure 25] 10 is a diagram showing still another example of a display screen displayed on the display device of the remote control device. FIG. [Figure 26] FIG. 10 is a block diagram showing a work support system according to a modified example of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0024] An embodiment of the present disclosure will be described with reference to the drawings. Fig. 1 is a block diagram showing a work support system 100 according to an embodiment. Fig. 2 is a side view showing a work machine 40 according to an embodiment. Fig. 3 is a diagram showing a remote control device 20 according to an embodiment.

[0025] The work support system 100 comprises a work support device 10, a work machine 40, and a remote control device 20. The work support device 10, the work machine 40, and the remote control device 20 can send and receive data to and from each other via communication using various networks 200 such as the Internet or a mobile phone network.

[0026] Fig. 4 is a diagram showing an example of a display screen displayed on the display device 23 of the remote control device 20. Fig. 5 is a diagram showing another example of a display screen displayed on the display device 23 of the remote control device. Fig. 6 is a diagram showing yet another example of a display screen displayed on the display device 23 of the remote control device 20. Fig. 7 is a diagram for explaining the positional relationship between the work machine 40 and the container 80 of the vehicle 70, and an example of an index image G1. Fig. 4 shows the early stage of earth removal work, which will be described later, and shows a state in which the container 80 has almost no earth 90 loaded therein. Fig. 5 shows the middle or final stage of the earth removal work, and shows a state in which a large amount of earth 90 has been loaded in the container 80.

[0027] The front-to-rear direction in the drawing represents the positional relationship between the work machine 40 and the vehicle 70. The container 80 (bed) of the vehicle 70 is disposed in front of the work machine 40. In other words, the front is the horizontal direction from the work machine 40 toward the container 80. The left-to-right direction in the drawing is the horizontal direction perpendicular to the front-to-rear direction. The container 80 of the vehicle 70 is the bed of the vehicle 70 that transports earth and sand 90. The vehicle 70 has a vehicle cab 71. The container 80 may have, for example, a rectangular shape in a plan view. The container 80 may have, for example, a substantially rectangular parallelepiped shape having a bottom 81, a right wall 83, a left wall 84, a front wall 85, and a rear wall 86. The bottom 81 of the container 80 has a bottom surface 82. The top of the container 80 is open. In other words, the container 80 has an opening 87 at its top.

[0028] The container 80 is not limited to the loading platform of the vehicle 70, but may be, for example, a facility such as a box (a box that cannot move on its own) installed at a work site, or may be a part of the facility. The part of the facility may be, for example, a portion from which soil and sand is discharged from the facility, or a portion from which soil and sand is input into the facility. Furthermore, the open area in the present disclosure is not limited to the container 80, but may also be a designated soil discharge area at a work site. Specifically, the soil discharge area may be, for example, a construction target area, which is an area at a work site where soil and sand is supplied.

[0029] 4, 5 and 7, the work support device 10 is a device for supporting work of releasing earth and sand 90 (an example of an object) held in a bucket 55 of a work machine 40 from the bucket 55 and putting the earth and sand 90 into a container 80 arranged in front of the work machine 40. The work support device 10 according to this embodiment has the following first characteristic.

[0030] [First feature] The work support device 10 includes a controller 11, which outputs an index image display command to cause the display device 23 to display an index image G1 including a container range image, which is a virtual image that serves as an index representing the range of the container 80 at least in the front-to-rear direction, and an attachment marker G3, which is a virtual image that represents the relative position of the bucket 55 with respect to the container 80 (step S106 in FIG. 10, which will be described later). The container range image may be, for example, the virtual plane image G2 shown in FIGS. 6 and 7, or may be a virtual frame image G5 in the modified examples shown in FIGS. 22 to 24, which will be described later. The container range image is an example of an open area range image in the present disclosure.

[0031] In this work support device 10, when the controller 11 outputs the index image display command, an index image G1 including the container range image (e.g., virtual plane image G2) and an attachment marker G3 is displayed on the display device 23 of the remote control device 20. The index image G1 displayed on the display device 23 makes it easier for the operator operating the remote control device 20 to grasp the relative position of the bucket 55 in the front-to-rear direction with respect to the container 80. Therefore, in the earth removal work of releasing earth 90 from the bucket 55 and putting the earth 90 into the container 80 (an example of a releasing work), the operator can appropriately adjust the relative position of the bucket 55 in the front-to-rear direction with respect to the container 80 while looking at the index image G1 displayed on the display device 23.

[0032] The work support device 10 according to this embodiment further includes the following second to ninth features.

[0033] [Second feature] 6 and 7, the controller 11 preferably generates, as the container range image, a virtual plane image G2, which is an image of a virtual plane that represents the range of the container 80 in the front-to-back and left-to-right directions. In a work assistance device 10 having this second feature, the operator operating the remote control device 20 can easily grasp not only the relative position of the bucket 55 with respect to the container 80 in the front-to-back direction but also the relative position in the left-to-right direction, from the index image G1, which includes the virtual plane image G2, displayed on the display device 23 of the remote control device 20. Therefore, during the soil removal work, the operator can appropriately adjust the relative position of the bucket 55 with respect to the container 80 in the front-to-back direction and the left-to-right direction while looking at the index image G1 displayed on the display device 23.

[0034] [Third feature] 6 and 7, the controller 11 preferably generates, as the container range image, a virtual opening area image G21 (an example of a virtual surface image G2), which is an image of a virtual surface that represents an area corresponding to the opening 87 of the container 80. In the work support device 10 having this third feature, the operator can easily grasp the relative position in the front-to-rear direction and the relative position in the left-to-right direction of the bucket 55 with respect to the opening 87 of the container 80, by using the index image G1 that includes the virtual opening area image G21 displayed on the display device 23. Therefore, during the soil discharge work, the operator can appropriately adjust the relative position in the front-to-rear direction and the relative position in the left-to-right direction of the bucket 55 with respect to the container 80 while looking at the index image G1 displayed on the display device 23.

[0035] 12, 13, and 14, the controller 11 may generate, as the container range image, a virtual bottom image G22 (an example of a virtual surface image G2), which is an image of a virtual surface representing an area corresponding to the bottom surface 82 of the container 80. In this work support device 10, the operator can easily grasp the relative position of the bucket 55 in the front-to-rear direction and the left-to-right direction with respect to the bottom surface 82 of the container 80 by using the index image G1 including the virtual bottom image G22 displayed on the display device 23. Therefore, during the earth removal work, the operator can appropriately adjust the relative position of the bucket 55 in the front-to-rear direction and the left-to-right direction with respect to the container 80 while viewing the index image G1 displayed on the display device 23. Note that FIG. 12 shows the initial stage of the earth removal work, showing a state in which almost no soil 90 has been loaded in the container 80. FIG. 14 shows the middle or final stage of the earth removal work, showing a state in which a large amount of soil 90 has been loaded in the container 80.

[0036] [Fourth feature] As shown in Figures 7 and 8, when a predetermined portion of the bucket 55 is at a position corresponding to the container 80, the controller 11 preferably generates, as the index image G1, an image in which the attachment marker G3 is positioned at a position corresponding to the virtual surface image G2. The predetermined portion may be, for example, the tip 55a of the bucket 55 as shown in Figure 7. The predetermined portion may also be, for example, the front end 55b, which is the foremost portion of the bucket 55, as shown in Figure 8. The front end 55b of the bucket 55 is a part that changes as the bucket 55 rotates. The tip 55a may be the center of the bucket 55 in the width direction. The front end 55b may be the center of the bucket 55 in the width direction.

[0037] Specifically, in work assistance device 10 having this fourth feature, for example, when the predetermined portion of bucket 55 (tip portion 55a or front end portion 55b) is inside or directly above container 80, controller 11 may generate as index image G1 an image in which attachment marker G3 is located at a position directly below bucket 55 and at a position on virtual plane image G2, or may generate as index image G1 an image in which attachment marker G3 is located at a position directly below bucket 55 and at a spaced apart upper position relative to virtual plane image G2. In other words, when the predetermined portion is located at a position corresponding to container 80, this may mean when the predetermined portion is inside container 80, or may mean when the predetermined portion is located directly above container 80 (a position spaced apart above container 80).

[0038] [5th ​​feature] 9, when the predetermined portion is shifted in the front-rear direction or the left-right direction with respect to the container 80, the controller 11 preferably generates, as the index image G1, an image in which the attachment marker G3 is positioned at a position shifted in the front-rear direction or the left-right direction with respect to the virtual plane image G2. Specifically, for example, when the predetermined portion is shifted in the front-rear direction or the left-right direction with respect to the container 80, the controller 11 may generate, as the index image G1, an image in which the attachment marker G3 is positioned at a position corresponding to the ground around the container 80, as shown in FIG.

[0039] [6th feature] 10, when a portion of the soil 90 contained in the container 80 (for example, an upper portion 91 of the soil 90) is located above the height of the virtual opening region image G21 (an example of the virtual surface image G2), the controller 11 may generate, as the index image G1, an image in which the region corresponding to the upper portion 91 of the soil 90 is omitted from the virtual surface image G2. In the work support device 10 having this sixth feature, the upper portion 91 of the soil 90 is not covered by the virtual surface image G2, so that the operator can easily grasp the state of the soil 90, such as the position of the soil 90 in the container 80 and the uneven distribution of the soil 90.

[0040] [7th feature] 11 and 12, the controller 11 may generate, as the container range image, a virtual bottom image G22 (an example of a virtual surface image G2), which is an image of a virtual surface that represents an area corresponding to the bottom surface 82 of the container 80. In this case, the controller 11 may generate, as the index image G1, an image in which the attachment marker G3 is positioned directly below the bucket 55 and at a position on the virtual bottom surface image G22, or may generate, as the index image G1, an image in which the attachment marker G3 is positioned directly below the bucket 55 and at a position spaced above the virtual bottom surface image G22, as shown in FIG. 13. In the work assistance device 10 having the seventh feature, the attachment marker G3 is displayed on the display device 23 at a position directly below the bucket 55 and corresponding to the virtual bottom surface image G22 (a position on the virtual bottom surface image G22 or a position spaced above the virtual bottom surface image G22). Therefore, during the soil removal work, the operator can easily grasp the relative position of the bucket 55 in the front-to-rear direction and the left-to-right direction with respect to the container 80.

[0041] [8th feature] 12 and 13, the attachment marker G3 is a first attachment marker G3 that is positioned directly below the bucket 55 and spaced above the virtual bottom image G22, and the controller 11 preferably generates, as the index image G1, an image that includes the virtual bottom image G22, the first attachment marker G3, and a second attachment marker G4 that is a virtual image that is positioned directly below the bucket 55 and on the virtual bottom image G22. In the work support device 10 having this eighth feature, the first attachment marker G3 is positioned on the display device 23 at a position spaced above the virtual bottom image G22, and the second attachment marker G4 is positioned on the virtual bottom image G22, so that during the soil removal work, the operator operating the remote control device 20 can more easily grasp the relative position of the bucket 55 in the front-to-rear direction and the relative position in the left-to-right direction with respect to the bottom surface 82 of the container 80.

[0042] [9th feature] As shown in FIG. 15 , the controller 11 may generate, as the index image G1, an image including a virtual opening area image G21 (an example of the virtual surface image G2), an attachment marker G3, and a cab rear wall image G6, which is a virtual image representing an area corresponding to the rear wall 72 of the vehicle cab 71. In the work assistance device 10 having this ninth feature, during the earth-discharging operation, the operator operating the remote control device 20 can easily grasp not only the relative position of the bucket 55 with respect to the container 80, but also the relative position of the bucket 55 with respect to the rear wall 72 of the vehicle cab 71. In the work assistance device 10 having this ninth feature, it is more preferable that the cab rear wall image G6 be displayed on the display device 23 in a display manner different from that of the virtual opening area image G21. In this case, during the earth-discharging operation, the operator can easily distinguish between the container 80 (the loading platform) and the rear wall 72 of the vehicle cab 71. The different display manner may be, for example, different colors or different shadings.

[0043] The area corresponding to the rear wall 72 of the vehicle cab 71 may be the area of ​​the rear wall 72 that separates the space of the vehicle cab 71 from the rear. In addition, if the vehicle 70 has a so-called shrine gate portion such as a guard frame disposed directly behind the vehicle cab 71, the area corresponding to the rear wall 72 of the vehicle cab 71 may be the area surrounded by the outer edge of the shrine gate portion.

[0044] The controller 11 may also generate an image including a container rear wall image as the index image G1. The container rear wall image is a virtual image that represents the area of ​​the rear wall 86 of the container 80.

[0045] The main features of the work assistance device 10 according to this embodiment are as described above. Below, a specific example of a work assistance system 100 including the work assistance device 10, a work machine 40, and a remote control device 20 will be described, but the work assistance device, work machine, and remote control device, as well as the work assistance system including these, in this disclosure are not limited to the following specific example.

[0046] 1, the work assistance device 10 may be equipped with a communication device 13, the work machine 40 may be equipped with a communication device 48, and the remote control device 20 may be equipped with a communication device 24. The work assistance device 10, the work machine 40, and the remote control device 20 can send and receive data to and from each other through communication using various networks 200 such as the Internet or a mobile phone network.

[0047] The work support device 10 may be an information terminal such as a tablet computer (a so-called tablet), a smartphone, a laptop personal computer, or a desktop personal computer. The work support device 10 may be a management device such as a server for managing work performed by a work machine 40. The work support device 10 may be a computer in a cloud service provided as a service over a network such as the Internet. The work support device 10 may include multiple information terminals.

[0048] The work support device 10 may be provided with a database 12 that stores various data such as dimensional information (specification data) of the work machine 40, information relating to the range of the container 80, and the like.

[0049] 1 and 2, work machine 40 includes a work machine main body, an attitude detection device 44, an imaging device 43, and a machine controller 41. The work machine main body includes a lower running structure 50 including a traveling device, an upper rotating structure 51 supported on lower running structure 50 so as to be rotatable relative to lower running structure 50 about a vertically extending rotation axis, and a work device 52 supported on upper rotating structure 51. Work machine 40 according to this embodiment is a hydraulic excavator.

[0050] The upper rotating body 51 includes a rotating frame and a cab 51A supported by the rotating frame. The rotating frame is a frame supported by the lower traveling body 50 so as to be rotatable, and forms a base portion of the upper rotating body 51.

[0051] The work device 52 includes a boom 53 that is attached to the upper rotating body 51 so as to be able to rise and fall, an arm 54 that is attached to the boom 53 so as to be able to rotate, and a bucket 55 that is attached to the arm 54 so as to be able to rotate. The bucket 55 is an example of a tip attachment. The tip attachment is not limited to the bucket 55, and may be, for example, another tip attachment such as a grapple, a fork, or a lifting magnet.

[0052] The operator's cab 51A is located, for example, at the front left portion of the revolving frame. Inside the operator's cab 51A, a driver's seat, an operation lever 42, and the like are arranged. The operation lever 42 receives various operations by the operator, such as boom operation for raising and lowering the boom 53, arm operation for rotating the arm 54, bucket operation for rotating the bucket 55, swing operation for swinging the upper revolving body 51 relative to the undercarriage 50, and travel operation for traveling the undercarriage 50. The operation lever 42 is used when the operator operates the work machine 40 from inside the operator's cab 51A. The operation lever 42 is not used when the work machine 40 is remotely operated. An outer wall defining a machinery room is arranged behind the operator's cab 51A. Various equipment is arranged in the machinery room, including power equipment such as an engine, battery, and generator, hydraulic equipment such as a hydraulic pump, and electrical equipment. The hydraulic pump is driven by the power equipment.

[0053] The work machine 40 is equipped with a plurality of actuators. Each of the plurality of actuators is operated by receiving a supply of hydraulic oil discharged from the hydraulic pump. The plurality of actuators includes a boom cylinder 56 for raising and lowering the boom 53, an arm cylinder 57 for rotating the arm 54, a bucket cylinder 58 for rotating the bucket 55, a swing motor 59 for rotating the upper swing structure 51 relative to the lower traveling structure 50, and a travel motor (not shown) for traveling the lower traveling structure 50.

[0054] The attitude detection device 44 acquires attitude information related to the attitude of the work machine main body. The attitude detection device 44 may include a plurality of work implement attitude detectors. As shown in Fig. 2, the plurality of work implement attitude detectors may include a boom attitude detector 44A, an arm attitude detector 44B, and a tip attachment attitude detector 44C. The attitude detection device 44 may further include an inclination detector 44D and a swing angle detector 44E.

[0055] The boom attitude detector 44A may be a sensor that detects the attitude of the boom 53, or may be a sensor that detects the state of the boom cylinder 56 that correlates with the attitude of the boom 53. The arm attitude detector 44B may be a sensor that detects the attitude of the arm 54, or may be a sensor that detects the state of the arm cylinder 57 that correlates with the attitude of the arm 54. The tip attachment attitude detector 44C may be a sensor that detects the attitude of the tip attachment 6, or may be a sensor that detects the state of the tip attachment cylinder 58 that correlates with the attitude of the tip attachment 55. The inclination detector 44D may be a sensor that detects the inclination of the upper rotating body 51 with respect to the horizontal plane. The swing angle detector 44E may be a sensor that detects the relative angle of the upper rotating body 51 with respect to the lower running body 50, or may be a sensor that detects the orientation of the upper rotating body 51 in the global coordinate system.

[0056] Each of the multiple attitude detectors may include, for example, an inertial measurement unit (IMU), may include a sensor that detects the degree of extension and contraction of the cylinder (for example, a stroke sensor), or may include other sensors.

[0057] The tilt detector 44D may include an inertial measurement unit (IMU). The tilt detector 44D may include a device that calculates the tilt of the upper rotating body 51 based on position information acquired using a satellite positioning system such as a global navigation satellite system (GNSS). In this case, the tilt detector 44D may include multiple antennas, such as multiple GNSS antennas. The tilt detector 44D may also include other sensors.

[0058] The rotation angle detector 44E may include a potentiometer. The rotation angle detector 44E may include a device that calculates the relative angle of the upper rotating body 51 or the orientation of the upper rotating body 51 based on position information acquired using the satellite positioning system. In this case, the rotation angle detector 44E may include multiple antennas, such as multiple GNSS antennas. The rotation angle detector 44E may also include other sensors.

[0059] The attitude information acquired by the attitude detection device 44 is input to the machine controller 41. The machine controller 41 transmits the input attitude information to the controller 11 of the work support device 10 via the communication device 48, the network 200, and the communication device 13. The controller 11 can calculate the attitude of the work machine 40 using the received attitude information.

[0060] The imaging device 43 acquires site images, which are images relating to the work site. A vehicle 70 equipped with a container 80 (loading platform) is placed at the work site. The imaging device 43 is a device that captures the site images for an operator who remotely operates the work machine 40 using the remote control device 20. The imaging device 43 may be placed, for example, inside the cab 51A or outside the cab 51A. The imaging device 43 is preferably placed, for example, at a position close to the eye level of a typical operator.

[0061] The work machine 40 may be equipped with a distance measuring device 45. The distance measuring device 45 acquires three-dimensional data (e.g., point cloud data) relating to the work site and inputs the acquired three-dimensional data to the machine controller 41. The distance measuring device 45 is configured to acquire three-dimensional data (e.g., point cloud data) of objects present at the work site, such as the ground, obstacles, and vehicle 70. Specific examples are as follows:

[0062] Distance measuring device 45 may be a distance measuring sensor that acquires distance information regarding the distance to an object by irradiating light such as laser light. The distance measuring sensor may be, for example, LiDAR (Light Detection and Ranging). Distance measuring device 45 may also be a stereo camera, a total station, or other sensor capable of acquiring three-dimensional data. Distance measuring device 45 may also be a combination of two or more of these devices. The distance measuring sensor may be attached to the work machine body of work machine 40, may be installed at the work site where work machine 40 is located, or may be attached to a wirelessly controlled unmanned aerial vehicle (a so-called drone).

[0063] When a ranging sensor is installed at the work site or attached to the drone, the results of distance measurement by the ranging sensor are input from the ranging sensor to the machine controller 41 via wired or wireless communication. Specifically, for example, a total station installed at the work site may measure the distance between a predetermined point on the container 80 and a predetermined point on the work implement 52 of the work machine 40, and the results of distance measurement may be input from the total station to the machine controller 41. Alternatively, a LiDAR or stereo camera installed at the work site, or a LiDAR or stereo camera attached to the drone, may acquire three-dimensional data of the distance measurement range including the work machine 40 and the container 80, and the acquired three-dimensional data may be input from the LiDAR or stereo camera to the machine controller 41.

[0064] Note that if the imaging device 43 has both the function of acquiring site images and the function of acquiring three-dimensional data (for example, point cloud data), the distance measuring device 45 can be omitted from the work machine 40. Furthermore, the remote control device 20 may be provided with a setting interface 29, and the range of the container 80 on the vehicle 70 may be input into the setting interface 29. In this case, the distance measuring device 45 can be omitted from the work machine 40. The setting interface 29 may set the range (for example, coordinates) of the container 80 at the work site based on an input operation performed by a person involved in the work, such as an operator. The person involved in the work may perform the input operation while looking at a remote output interface (for example, a display device 23 such as a display).

[0065] Specifically, for example, if the container 80 is rectangular as shown in FIG. 4 , the controller 11 may set the extent of the container 80 by identifying the positions of three or more of the four corners of the rectangle. The position of a corner may be identified, for example, by the operator operating the operation lever 22 to position the tip of the bucket 55 at a position corresponding to the corner, and then performing an operation (coordinate identification operation) on the setting interface 29. The controller 11 can calculate the position of the tip of the bucket 55 based on the attitude information input from the attitude detection device 44. The position of a corner may also be identified, for example, by a worker inputting the coordinates of the corner into the setting interface 29 of the remote operation device 20.

[0066] The remote control device 20 is a device for remotely operating the work machine 40. The remote control device 20 is equipped with a remote control lever 22 that receives operations for remotely operating the work machine 40, a display device 23, and a seat 28. The remote control device 20 is placed in a location remote from the work machine 40, and is a device for remotely operating the work machine 40.

[0067] The display device 23 may be a display that displays a site image of the work site where the container 80 (loading platform) of the vehicle 70 is placed, for example, as shown in Fig. 4. The controller 11 causes the display device 23 to display an index image G1 superimposed on the site image of the work site where the container 80 (loading platform) of the vehicle 70 is placed, for example, as shown in Fig. 6. As shown in Fig. 3, the remote control device 20 may further include, in addition to the display device 23, for example, a second display device 25 or a third display device 26. The remote control device 20 may also further include a speaker 27 for outputting sounds of the work site.

[0068] Each of the various positions required for control for remote operation, such as the position of the imaging device 43, the position of the ranging device 45, the range of the container 80, and the position of the work machine 40 (for example, the position of the bucket 55), may be information represented by coordinates in a coordinate system based on the imaging device 43 (imaging device coordinate system), may be information represented by coordinates in a coordinate system based on the work machine 40 (machine coordinate system), may be information represented by coordinates in a coordinate system based on a specific position at the work site (site coordinate system), or may be information represented by coordinates in a global coordinate system. The controller 11 may be configured to be able to convert position information in any of these coordinate systems into position information in another coordinate system.

[0069] In the following specific example, the controller 11 is configured to convert point cloud data input from the distance measuring device 45 into point cloud data in a predefined reference coordinate system. The controller 11 may also be configured to convert on-site image data input from the imaging device 43 into data in a predefined reference coordinate system. The reference coordinate system may be the machine coordinate system, the on-site coordinate system, the global coordinate system, or another coordinate system.

[0070] The origin of the reference coordinate system may be set at a specific position relative to the work machine 40, for example. Specifically, the origin of the reference coordinate system may be, for example, any position on the swing axis. More specifically, the origin of the reference coordinate system may be the intersection of the swing axis and the ground. Furthermore, the origin of the reference coordinate system may be, for example, on the swing axis and at a position between the lower traveling body 50 and the upper swing body 51.

[0071] In this embodiment, the reference coordinate system is a three-dimensional coordinate system. In this case, the reference coordinate system may be, for example, an orthogonal coordinate system defined by an x-axis parallel to the left-right direction, a y-axis parallel to the front-rear direction, and a z-axis parallel to the vertical direction. However, the reference coordinate system is not limited to the above specific example, and various other aspects can be adopted.

[0072] The controller 11 of the work support device 10 includes a computer including a processing unit and a memory. The controller 11 has various functions, such as an image processing function for generating an index image G1. The controller 11 realizes each of the various functions by the processing unit executing a program stored in the memory.

[0073] The controller 11 may determine the range of the container 80 of the vehicle 70 based on the point cloud data input from the distance measuring device 45.

[0074] The remote controller 21 of the remote operation device 20 includes a computer including an arithmetic processing unit and a memory. The remote controller 21 controls the operation of the remote operation device 20, such as the operation of the display device 23. The remote controller 21 realizes various functions by having the arithmetic processing unit execute programs stored in the memory.

[0075] The machine controller 41 of the work machine 40 is equipped with a computer including an arithmetic processing unit and memory. The machine controller 41 controls the operation of the work machine 40. The machine controller 41 outputs the posture information and the site image to the work support device 10. The machine controller 41 realizes each of its various functions by having the arithmetic processing unit execute programs stored in memory.

[0076] FIG. 16 is a flowchart showing an example of calculation processing performed by the controller 11 of the work support device 10.

[0077] The remote controller 21 of the remote operation device 20 determines whether or not the input interface of the remote operation device 20 has received a command operation (step S201 in FIG. 16). The command operation may be, for example, a preset operation indicating the start of remote operation of the work machine 40. The command operation may also be an operation given by the operator to the input interface. If the input interface has received the command operation (YES in step S201), the remote controller 21 transmits an environment confirmation request to the work machine 40 to request a site image of the work site (step S202).

[0078] When the machine controller 41 of the work machine 40 receives the environment confirmation request (step C40), it acquires information about the site image captured by the imaging device 43 and information about the three-dimensional data acquired by the distance measuring device 45, and transmits the acquired site image and three-dimensional data to the work support device 10 (step S401). The machine controller 41 acquires posture information detected by the posture detection device 44, and transmits the acquired posture information to the work support device 10 (step S402).

[0079] The controller 11 of the work support device 10 receives the site image and the three-dimensional data transmitted from the work machine 40 (step C11). The controller 11 transmits information related to the received site image to the remote operation device 20 (step S101).

[0080] The remote controller 21 of the remote operation device 20 receives the site image transmitted by the controller 11 of the work support device 10 (step C21).

[0081] The remote controller 21 controls the operation of the display device 23 so that the received site image is displayed on the display device 23 (step S203). That is, the display device 23 displays the site image as shown in, for example, FIG.

[0082] The controller 11 of the work support device 10 receives the attitude information transmitted by the work machine 40 (step C12).

[0083] The controller 11 of the work support device 10 recognizes the range of the container 80 based on the three-dimensional data (e.g., point cloud data) of the container 80 acquired by, for example, the distance measuring device 45 (step S102). Note that if the database 12 stores information about the range of the container 80, the controller 11 may read the information about the range of the container 80 from the database 12 and identify the range of the container 80 based on the read information (step S102).

[0084] Based on the posture information input from the posture detection device 44 and the range of the container, the controller 11 determines whether the specified part of the bucket 55 (tip 55a or front end 55b) is in a position corresponding to the container 80 (specifically, for example, whether the specified part is within the range of the container 80) (step S103).

[0085] When the specified portion of the bucket 55 is at a position corresponding to the container 80 (YES in step S103), the controller 11 generates an image as an index image G1 in which the attachment marker G3 is positioned at a position corresponding to the virtual surface image G2, as shown in Figure 7 or Figure 8 (step S105).

[0086] If the specified portion of the bucket 55 is shifted in the forward / backward or left / right direction relative to the container 80 (NO in step S103), the controller 11 generates an image as an index image G1 in which the attachment marker G3 is positioned at a position shifted in the forward / backward or left / right direction relative to the virtual surface image G2, as shown in Figure 9 (step S104).

[0087] The controller 11 outputs an index image display command to cause the display device 23 to display the index image G1 including the virtual plane image G2 and the attachment marker G3 (step S106).

[0088] The remote controller 21 of the remote operation device 20 receives the index image display command transmitted by the controller 11 of the work support device 10 (step C22).

[0089] In response to the received index image display command, the remote controller 21 of the remote operation device 20 causes the display device 23 to display the index image G1 superimposed on a work site image where the container 80 (bed) of the vehicle 70 is placed, for example, as shown in Fig. 6 (step S204). That is, the display device 23 displays the index image G1 in response to the index image display command superimposed on the work site image.

[0090] Furthermore, the remote controller 21 of the remote operation device 20 transmits to the work machine 40 a remote operation command, which is a command corresponding to the operation (remote operation) by the operator received by the remote operation lever 22 (step S205).

[0091] When the machine controller 41 of the work machine 40 receives the remote operation command transmitted from the remote operation device 20 (step C41), it controls the control objects so that the work implement 52 and / or the upper rotating body 51 operate in accordance with the remote operation command (step S403). The control objects may include, for example, the boom cylinder 56, the arm cylinder 57, the bucket cylinder 58, the swing motor 59, and a control valve that hydraulically controls the operation of the travel motor. In this case, the machine controller 41 can cause the work machine 40 to operate in accordance with the remote operation command by inputting a command in accordance with the remote operation command to the control valve.

[0092] [Variations] Although the embodiments of the present disclosure have been described above, the present disclosure is not limited to the above-described embodiments and includes, for example, the following modified examples.

[0093] (A) About attachment markers In the above embodiment, the attachment marker G3 has the shape of a cone or a triangular pyramid with the apex located at the bottom, but it may also have a shape that projects the outline of the bucket 55 onto the virtual surface image G2 directly below the bucket 55, as shown in Figures 17 and 18.

[0094] 19, the index image G1 may include the container range image such as a virtual plane image G2, an attachment marker G3, and a movable range image which is a virtual image that indicates the relative position of the movable range of the working device 52 with respect to the container 80. The attachment marker G3 shown in Fig. 19 is an image in which the outline of the bucket 55 is projected onto the container range image directly below the bucket.

[0095] The movable range image may include, for example, a front movable range marker G8 and a rear movable range marker G9 as shown in Fig. 19. The front movable range marker G8 is an image projected onto the container range image of the range of movement of the work device 52 that is forward of the attachment marker G3 and that the tip of the bucket 55 can reach. The rear movable range marker G9 is an image projected onto the container range image of the range of movement of the work device 52 that is rear of the attachment marker G3 and that the tip of the bucket 55 can reach.

[0096] Furthermore, as shown in FIG. 20, when the specified portion of the bucket 55 is in a position corresponding to the container 80, the controller 11 of the work support device 10 may generate an image as an index image G1 in which the attachment marker G3 is placed on the surface of the soil 90 directly below the bucket 55.

[0097] (B) Remote control devices In the above embodiment, the Internet and a mobile phone network are exemplified as networks for communication between the work assistance device 10 and the remote control device 20, but the work assistance device 10 and the remote control device 20 may be configured to communicate via a network such as a wired LAN (Local Area Network) or a CAN (Controller Area Network). The remote control device 20 and the work assistance device 10 do not necessarily have to be configured as separate devices, and may be integrated.

[0098] The remote operation device 20 may be equipped with the work assistance device 10. In this case, the remote controller 21 of the remote operation device 20 combines the functions of the remote controller 21 of the remote operation device 20 described in the above embodiment and the functions of the controller 11 of the work assistance device 10, and constitutes a controller in the present disclosure. Specifically, the remote controller 21 of the remote operation device 20 may perform not only the arithmetic processing by the remote controller 21 in the flowchart shown in FIG. 16 but also the arithmetic processing by the controller 11 of the work assistance device 10 in the flowchart shown in FIG. 16.

[0099] (C) About the image of the open area In the above embodiment, the controller 11 of the work support device 10 generates, as the container range image, a virtual opening area image G21 shown in Fig. 6 and Fig. 7 or a virtual bottom surface image G22 shown in Fig. 11 to Fig. 13, but is not limited to these. For example, as shown in Fig. 21, the controller 11 may generate, as the container range image, a virtual surface image G2 that represents the range of the container 80 in the front-rear and left-right directions and that passes through the top of the soil 90.

[0100] Furthermore, the controller 11 of the work support device 10 may generate, as the container range image, a virtual frame image G5 that is an image of a virtual frame that represents the range of the container 80 in the front-to-back direction and the left-to-right direction, as shown in Fig. 22, for example. In this modification, the operator can easily grasp not only the relative position of the bucket 55 with respect to the container 80 in the front-to-back direction but also the relative position in the left-to-right direction, from the index image G1 that includes the virtual frame image displayed on the display device 23. Therefore, during the soil discharge work, the operator can appropriately adjust the relative position of the bucket 55 with respect to the container 80 in the front-to-back direction and the left-to-right direction while looking at the index image G1 displayed on the display device 23. Specifically, this is as follows.

[0101] Fig. 22 is a diagram showing yet another example of a display screen displayed on the display device 23 of the remote control device 20. In this modification, the controller 11 generates an image including a virtual frame image G5 as an open area range image and an attachment marker G3 as an index image G1, and outputs an index image display command to display the index image G1 on the display device 23, causing the display device 23 to display an image in accordance with the index image display command, as shown in Fig. 22. The image shown in Fig. 22 is a view of the container 80 as seen from directly above.

[0102] The attachment marker G3 may include a marker point image G31 and an auxiliary line G32, as shown in the left and center diagrams of FIG.

[0103] The marker point image G31 is an image that represents the relative position of the tip attachment 55 with respect to the container 80, and may be an image that represents the position directly below the predetermined portion of the bucket 55 (the tip portion 55a or the front end portion 55b). The auxiliary line G32 may include a front-to-rear line that passes through a position directly below the predetermined portion of the bucket 55 and is parallel to the front-to-rear direction, and a left-to-right line that passes through a position directly below the predetermined portion of the bucket 55 and is parallel to the left-to-right direction. The front-to-rear line may be a line that extends from the front end of the container 80 to the rear end of the container 80, and the left-to-right line may be a line that extends from the left end of the container 80 to the right end of the container 80. Furthermore, as shown in the right diagram in FIG. 22, the attachment marker G3 may include the auxiliary line G32 but not the marker point image G31.

[0104] 23 is a diagram showing yet another example of a display screen displayed on the display device 23 of the remote control device 20. In this modification, the controller 11 generates an image including a virtual frame image G5 as an open area range image and an attachment marker G3 as an index image G1, and outputs an index image display command to cause the display device 23 to display the index image G1, so that the display device 23 displays an image in accordance with the index image display command, as shown in FIG.

[0105] The attachment marker G3 includes an auxiliary line G32. The auxiliary line G32 may include a longitudinal line that passes directly below the predetermined portion of the bucket 55 and is parallel to the longitudinal direction, and an arc that passes directly below the predetermined portion of the bucket 55 and has its center at the center of rotation of the upper rotating body 51. The arc may be a curve that extends from the left end of the container 80 to the right end of the container 80.

[0106] Fig. 24 is a diagram showing yet another example of a display screen displayed on the display device 23 of the remote control device 20. In this modification, the controller 11 generates an image including a virtual plane image G2 or a virtual frame image G5 as an open area range image and an attachment marker G3 as an index image G1, and outputs an index image display command to cause the display device 23 to display the index image G1, whereby the display device 23 displays an image in accordance with the index image display command, as shown in the left diagram (A) in Fig. 24, for example.

[0107] In this modification, the controller 11 changes the display mode of the index image G1 on the display device 23 based on the distance between the end of the container 80 and the working device 52 (e.g., the bucket 55). Specifically, when the distance between the end of the container 80 and the working device 52 becomes equal to or less than a predetermined threshold Th, the controller 11 may change the display mode of the index image G1 on the display device 23, for example, as shown in the upper diagram (B), the center diagram (C), or the lower diagram (D) in FIG. 24. In the specific example of FIG. 24, the controller 11 determines whether or not to change the display mode based on the distance between the front end of the container 80 and the working device 52 (e.g., the bucket 55). However, the controller 11 may also determine whether or not to change the display mode based on the distance between the rear end, right end, or left end of the container 80 and the working device 52.

[0108] In the specific example shown in the upper diagram (B) of Figure 24, when the distance between the front end of the container 80 and the bucket 55 of the work device 52 becomes less than a predetermined threshold value Th, the controller 11 changes the display mode of the end (e.g., front end G2A) of the virtual surface image G2 or the virtual frame image G5 of the index image G1.

[0109] In a specific example shown in the central figure (C) in Figure 24, when the distance between the front end of the container 80 and the bucket 55 of the work device 52 becomes less than a predetermined threshold value Th, the controller 11 changes the overall display mode of the virtual surface image G2 of the index image G1, or changes the display mode inside the virtual frame image G5.

[0110] In a specific example shown in the lower figure (D) of Figure 24, when the distance between the front end of the container 80 and the bucket 55 of the work device 52 becomes less than a predetermined threshold value Th, the controller 11 changes the display mode of the attachment marker G3 in the index image G1.

[0111] In the modification shown in FIG. 24, the change in the display mode of the image may be, for example, a change in the color of the image, or may be causing at least a part of the image to blink.

[0112] However, the change in the display mode is not limited to the specific example shown in FIG.

[0113] FIG. 25 is a diagram showing yet another example of a display screen displayed on the display device 23 of the remote control device 20. In the modified example of FIG. 25 , as in the left and center diagrams of FIG. 22 , the attachment marker G3 includes a marker point image G31 and an auxiliary line G32. The auxiliary line G32 includes a front-rear straight line that passes directly below the predetermined portion of the bucket 55 and is parallel to the front-rear direction, and a left-right straight line that passes directly below the predetermined portion of the bucket 55 and is parallel to the left-right direction. Each of the front-rear straight line and the left-right straight line may be positioned so as to pass through the lower end of the marker point image G31. The front-rear straight line may be a straight line extending from the front end of the container 80 to the rear end of the container 80, and the left-right straight line may be a straight line extending from the left end of the container 80 to the right end of the container 80.

[0114] In this modification, the controller 11 may change the display mode of the cab rear wall image G6 on the display device 23 based on the distance between the rear wall 72 of the vehicle cab 71 and the working device 52 (e.g., bucket 55), and may cause the display device 23 to display an alert to notify that the distance between the rear wall 72 of the vehicle cab 71 and the working device 52 has become small. Specifically, when the distance between the rear wall 72 of the vehicle cab 71 and the working device 52 becomes equal to or less than a predetermined threshold, the controller 11 may change the display mode of the cab rear wall image G6 on the display device 23, and may cause the display device 23 to display the alert.

[0115] The change in the display mode of the cab rear wall image G6 may be, for example, a change in color of the cab rear wall image G6 or may be caused to blink at least a part of the cab rear wall image G6. The alert display may include, for example, text or a mark in an easily noticeable color.

[0116] (D) Work support devices In the above embodiment, an example is given of an operator remotely operating the work machine 40 using the remote control device 20, but the work support device 10 in the present disclosure is not limited to remote operation and may also be applied to a case where an operator is on board the work machine 40 and operates the operating lever 42.

[0117] In the above embodiment, the controller 11 of the work support device 10 transmits the site image to the remote controller 21 of the remote operation device 20 in step S101 of FIG. 16, but the site image may also be transmitted to the remote controller 21 of the remote operation device 20 when outputting the index image display command in step S106, for example.

[0118] In the above embodiment, the controller 11 of the work support device 10 transmits the work site image to the remote controller 21 of the remote operation device 20 in step S101, transmits the index image display command to the remote controller 21 of the remote operation device 20 in step S106, and the remote controller 21 of the remote operation device 20 controls the display device 23 based on the received index image display command to display the index image G1 superimposed on the work site image of the work site where the container 80 (bed) of the vehicle 70 is placed. However, this is not necessarily limited to the above embodiment. For example, in step 106, the controller 11 of the work support device 10 may generate an index image G1 in accordance with the work site image and the index image display command, further generate a superimposed image by superimposing the index image G1 on the work site image, transmit the generated superimposed image to the remote controller 21 of the remote operation device 20, and the remote controller 21 of the remote operation device 20 may display the received superimposed image on the display device 23.

[0119] (E) Work support devices for simulators In the above embodiment, an example has been given of an operator remotely operating a real work machine 40 located at a work site in real space using a remote control device 20, but the work support device of the present disclosure may also be applied to a simulator configured so that an operator operates a virtual work machine located at a work site constructed in a virtual space. Specifically, as described below.

[0120] 26 is a block diagram showing an example of a work assistance system 300 in which the work assistance device 10 is applied to the simulator described above. The work assistance system 300 includes the work assistance device 10, a remote control device 20, and an operation simulator device 60.

[0121] The operation simulator device 60 places a virtual work site in a virtual space created by the operation simulator device 60, and places a virtual work machine 40A at the work site. The operation simulator device 60 displays images of the work site and work machine 40A in the virtual space on the display device 23 of the remote control device 20. The operator can operate the work machine 40A at the work site in the virtual space by operating the remote control lever 22 of the remote control device 20.

[0122] The operation simulator device 60 includes a computer and a communication device. The computer of the operation simulator device 60 performs calculations to operate the work machine 40A in a virtual space. The virtual space is a virtual three-dimensional space, and the work machine 40A, which is a three-dimensional model simulating the work machine 40 in real space, is placed within the virtual space. The work site constructed in the virtual space is a simulation of the work site in real space, and includes, for example, a three-dimensionally modeled ground surface, which is configured so that it can be excavated by the work machine 40A. A virtual vehicle simulating the vehicle 70 in real space, i.e., a three-dimensionally modeled vehicle (for example, a vehicle equipped with a loading platform for loading earth and sand), may be placed at the work site in the virtual space. Furthermore, objects simulating devices, equipment, and other objects present at the work site in real space, i.e., three-dimensionally modeled objects, may be placed at the work site in the virtual space. The communication device of the operation simulator device 60 is configured to be able to communicate with external devices such as the work support device 10 and the remote operation device 20 via a network 200. The network 200 may be a network such as the Internet or a mobile phone network, or may be a network such as a wired LAN or CAN. The communication device of the operation simulator device 60 receives information transmitted from the external device and inputs the information to the computer of the operation simulator device 60. The communication device of the operation simulator device 60 transmits the calculation results of the computer to the external device.

[0123] A work machine 40A modeled three-dimensionally in virtual space includes components that simulate the undercarriage 50, upper revolving body 51, and work implement 52 of the work machine 40 in real space shown in FIG. 2. The computer of the operation simulator device 60 may perform calculations so that the work machine 40A in virtual space simulates the operation of various components such as the control lever 42 and imaging device 43 that are included in the work machine 40 in real space shown in FIG. 1. In this case, the work machine 40A in virtual space performs operations that simulate inputs received by the work machine 40 in real space and outputs corresponding to these inputs. For example, as shown in FIG. 26, the work machine 40A in virtual space may include various modeled components, namely, a machine controller 41A, a control lever 42A, an imaging device 43A, a group of sensors (posture detection device 44A, distance measurement device 45A), and communication equipment 48A.

[0124] Specifically, when the operator operates the remote control lever 22 of the remote control device 20, a remote control command is transmitted from the remote control device 20 to the operation simulator device 60, and the computer of the operation simulator device 60 performs calculations regarding the operation of the work machine 40A in virtual space in response to the received remote operation command. For example, a modeled imaging device 43A acquires images of a work site constructed in virtual space. A modeled distance measuring device 45A acquires three-dimensional data of the work site constructed in virtual space. A modeled attitude detection device 44A acquires attitude information regarding the attitude of the work machine 40A. For each of the modeled imaging device 43A and the group of sensors, device operating characteristics such as detection range and response speed, and output information such as data output format and output voltage range may be modeled. The computer of the operation simulator device 60 may perform calculations to simulate the modeled machine controller 41A transmitting the operating characteristics and output information to an external device via a modeled communication device 48A. The communication device of the operation simulator device 60 transmits the calculation result to at least one of the work support device 10 and the remote operation device 20 via the network 200.

[0125] The work support device 10 and remote control device 20 of the work support system 300 shown in Fig. 26 are similar to the work support device 10 and remote control device 20 of the work support system 100 shown in Fig. 1. In the work support system 300 shown in Fig. 26, the computer of the operation simulator device 60 transmits information about the work site and the operation of the work machine 40A in the virtual space constructed in the operation simulator device 60 to the work support device 10, and the controller 11 of the work support device 10 displays, based on the received information, an index image G1 including an open area range image such as a virtual plane image G2 and an attachment marker G3 on the display device 23 of the remote control device 20, for example, as shown in Figs. 6 to 15.

[0126] In the work support system 300 equipped with the operation simulator device 60, the operator can appropriately adjust the relative position of the bucket with respect to an open area such as a container existing in the virtual space while looking at the index image G1 displayed on the display device 23.

[0127] In the specific example shown in FIG. 26 , the work support system 300 is configured such that the operation simulator device 60, the work support device 10, and the remote control device 20 communicate with each other via a network 200, but the work support system for a simulator is not limited to the specific example shown in FIG. 26 . In the work support system for a simulator, the work support device 10 may be mounted on the operation simulator device 60 or on the remote control device 20. Furthermore, if the operation simulator device 60 is equipped with an operation lever and a display device, the remote control device 20 can be omitted from the work support system. Furthermore, in the specific example shown in FIG. 26 , the operation simulator device 60, the work support device 10, and the remote control device 20 are configured as separate devices, but they may also be configured as an integrated device.

[0128] (F) About work machines The work machine 40 according to this embodiment is a shovel, but the work machine in this disclosure is not limited to a shovel and may be another work machine such as a crane or a bulldozer.

[0129] In the above embodiment, the Internet and a mobile phone network are exemplified as networks for communication between the work assistance device 10 and the work machine 40, but the work assistance device 10 and the work machine 40 may be configured to communicate via a network such as a wired LAN (Local Area Network) or CAN (Controller Area Network). The work assistance device 10 may be mounted on the work machine 40.

[0130] The work machine 40 may be equipped with a work support device 10. In this case, the machine controller 41 of the work machine 40 may combine the functions of the machine controller 41 of the work machine 40 described in the above embodiment and the functions of the controller 11 of the work support device 10. Specifically, the machine controller 41 of the work machine 40 may perform not only the arithmetic processing by the machine controller 41 in the flowchart shown in FIG. 16, but also the arithmetic processing by the controller 11 of the work support device 10 in the flowchart shown in FIG.

[0131] (G) The present disclosure includes a program according to the following modified example. The program according to this modified example is a program for assisting in the work of releasing soil 90 held in the bucket 55 of the work machine 40 from the bucket 55 and placing the soil 90 into a container 80 located in front of the work machine 40, and causes a computer to realize the function of displaying, on the display device 23, an index image G1 that includes a container range image, which is a virtual image that serves as an index indicating the range of the container 80 in at least the fore-and-aft direction, and an attachment marker G3, which is a virtual image that indicates the relative position of the bucket 55 with respect to the container 80. This program may be provided via communications means (for example, a network such as the Internet), or may be provided stored on various recording media (for example, non-transitory tangible recording media).

[0132] (H) The present disclosure includes a work assistance method according to the following modified example. This work assistance method according to the modified example is a method for assisting in the work of releasing soil 90 held in the bucket 55 of the work machine 40 from the bucket 55 and putting the soil 90 into a container 80 located in front of the work machine 40, and includes displaying on the display device 23 an index image G1 that includes a container range image, which is a virtual image that serves as an index indicating the range of the container 80 in at least the front-to-rear direction, and an attachment marker G3, which is a virtual image that indicates the relative position of the bucket 55 with respect to the container 80. [Explanation of symbols]

[0133] 10 Work support device 11 Controller 20 Remote Control Device 23 Display device 40 Work Machinery 41 Machine Controller 42 Operating lever 43 Imaging device 44 Attitude detection device 45 Ranging device 50 Undercarriage 51 Upper rotating body 51A Driver's Cab 52 Work equipment 55 Bucket (Example of tip attachment) 70 vehicles 71 Vehicle cab 72 Rear wall of the vehicle cab 80 containers 82 bottom 87 Opening 90 Soil and sand (example of object) 91 Part of soil and sand (an example of a part of an object) 100 Work Support System G1 Index Image G2 Virtual surface image (example of open area range image) G21 Virtual opening area image (example of virtual surface image) G22 Virtual bottom image (example of virtual surface image) G3 Attachment Marker (Example of the first attachment marker) G4 Attachment Marker (Example of a second attachment marker) G5 Virtual frame image (example of open area range image) G6 rear wall image of the driver's cab

Claims

1. 1. A work support device for supporting a work of releasing an object held on a tip attachment of a work machine from the tip attachment and placing the object in a release area present in front of the work machine, A work support device comprising a controller that causes a display device to display an index image including a release area range image, which is a virtual image that serves as an index representing the range of the release area at least in the forward and backward directions, and an attachment marker, which is a virtual image that represents the relative position of the tip attachment with respect to the release area.

2. 2. The work assistance device according to claim 1, wherein the controller generates, as the released area range image, a virtual surface image that is an image of a virtual surface that represents the range of the released area in the forward / backward direction and the left / right direction, or a virtual frame image that is an image of a virtual frame that represents the range of the released area in the forward / backward direction and the left / right direction.

3. the release area is a container located in front of the work machine; The work support device according to claim 1, wherein the controller generates, as the released area range image, a virtual surface image that is an image of a virtual surface representing an area corresponding to the opening of the container or a virtual surface image that is an image of a virtual surface representing an area corresponding to the bottom surface of the container.

4. 3. The work support device according to claim 2, wherein when a predetermined portion of the end attachment is in a position corresponding to the release area, the controller generates, as the index image, an image in which the attachment marker is positioned in a position corresponding to the virtual surface image.

5. 5. The work support device according to claim 4, wherein, when the predetermined portion is shifted in the front-to-rear direction or the left-to-right direction with respect to the released area, the controller generates, as the index image, an image in which the attachment marker is positioned at a position shifted in the front-to-rear direction or the left-to-right direction with respect to the virtual surface image.

6. the release area is a container located in front of the work machine; 3. The work support device according to claim 2, wherein when a portion of the object contained in the container is located above a height of the virtual plane image, the controller generates, as the index image, an image in which an area corresponding to the portion of the object in the virtual plane image is omitted.

7. the release area is a container located in front of the work machine; The controller generating a virtual surface image that is an image of a virtual surface that represents an area corresponding to the bottom surface of the container as the released area range image; The work support device according to claim 1 , wherein the index image is generated as an image in which the attachment marker is positioned directly below the end attachment and on the virtual surface image or at a position above and spaced apart from the virtual surface image.

8. the attachment marker is a first attachment marker that is placed directly below the tip attachment and at a position spaced above the virtual plane image, 8. The work support device according to claim 7, wherein the controller generates, as the index image, an image including the released area range image, the first attachment marker, and a second attachment marker that is a virtual image positioned at a position directly below the end attachment on the virtual plane image.

9. the release area is a loading platform of a vehicle that transports the object, the vehicle has a vehicle cab; 2. The work assistance device according to claim 1, wherein the controller generates, as the index image, an image including the released area range image, the attachment marker, and a cab rear wall image that is a virtual image representing an area corresponding to a rear wall of the vehicle cab.

10. A work support device according to any one of claims 1 to 9; the work machine; a remote control device having the display device and for remotely controlling the work machine;

11. A work machine body; a posture detection device for acquiring posture information relating to the posture of the work machine body; an imaging device for acquiring site images that are images relating to a work site where the open area exists; A work machine comprising: a machine controller that outputs the posture information and the site image to the work support device according to any one of claims 1 to 9.

12. a remote control lever that receives operation for remotely controlling the work machine; 10. A remote control device comprising: the display device that displays the index image including the released area range image and the attachment marker in response to a command output by the work support device according to any one of claims 1 to 9.

13. The remote operation device according to claim 12 , wherein the controller causes the display device to display the index image superimposed on a site image of the work site where the released area exists.

14. 1. A program for assisting an operation of releasing an object held on a tip attachment of a work machine from the tip attachment and placing the object in a release area present in front of the work machine, A program for causing a computer to realize the function of displaying on a display device an index image including an open area range image, which is a virtual image that serves as an indicator representing the range of the open area at least in the forward and backward directions, and an attachment marker, which is a virtual image that represents the relative position of the tip attachment with respect to the open area.

15. 1. A work assistance method for assisting an operation of releasing an object held on a tip attachment of a work machine from the tip attachment and placing the object in a release area present in front of the work machine, comprising: A work support method including displaying on a display device an index image including an open area range image, which is a virtual image that serves as an indicator representing the range of the open area at least in the forward and backward directions, and an attachment marker, which is a virtual image that represents the relative position of the tip attachment with respect to the open area.

Images