Pilot assistance device and work machine
The maneuvering support device with a transparent display enhances crane operation efficiency by superimposing virtual images of boom movements and surroundings, addressing the inefficiencies caused by instructor reliance and limited camera visibility.
Patent Information
- Application Number
- JP2023517209
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2021-04-27
- Filing Date
- 2022-03-30
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2042-03-30
AI Technical Summary
Crane operators face inefficiencies due to reliance on instructor instructions and limited visibility from boom cameras, leading to prolonged cargo movement times and insufficient information for efficient operation.
A maneuvering support device with a transparent display worn by the operator, superimposing virtual images of the crane's boom movements and surroundings onto the real-world view, providing enhanced situational awareness and operational guidance.
Improves crane operation efficiency by allowing operators to visualize critical crane positions and trajectories directly in their field of view, reducing reliance on instructor instructions and enhancing maneuvering precision.
Smart Images

Figure 0007593487000001 
Figure 0007593487000002 
Figure 0007593487000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a steering assistance device for a work machine and a work machine. [Background technology]
[0002] Conventionally, when transporting cargo using a work machine such as a mobile crane, the operator of the mobile crane operates multiple operating tools, such as a slewing operating tool, a hoisting operating tool, and an extension operating tool for operating the mobile crane's boom, and a winch operating tool for hoisting or lowering the cargo.
[0003] In addition, the operator must operate the mobile crane while simultaneously operating multiple operating tools so that the load does not come into contact with structures at the work site. Therefore, the operator obtains information about the load's location and information about features from riggers, etc., to supplement information about areas and times that the operator cannot see. In other words, the operator operates the crane based on information from his or her own eyes and information from an external instructor.
[0004] In addition, the mobile crane has a boom camera or the like to obtain information on areas that are difficult for the operator to recognize from the operator's seat. The mobile crane can obtain images of the cargo being transported and surrounding structures from above using the boom camera. For example, see Patent Document 1.
[0005] The crane described in Patent Document 1 uses a boom camera attached to the tip of the boom to capture images of the work site and images of the cargo suspended from the hook, and also displays, superimposed on the images, at least one of a reference line centered on the cargo, a scale, a directed line indicating the direction of cargo movement, and a trajectory in which the cargo can turn. The crane operator operates the crane by comprehensively assessing his own visual information, information from an instructor, and image information from the boom camera. [Prior art documents] [Patent documents]
[0006] [Patent Document 1] JP 2019-156538 A Summary of the Invention [Problem to be solved by the invention]
[0007] A crane operator cannot operate a crane at his / her own pace because he / she operates the crane according to the instructions and timing of the instructions from the instructor. In addition, if the operator cannot understand the intention of the instructor or if the instructions from the instructor are not fully conveyed due to poor communication, it may take a long time to move the load. On the other hand, the image from the boom camera has a limited shooting direction and shooting range. Therefore, the crane operator cannot obtain enough information required to efficiently move the load from the image from the boom camera.
[0008] An object of the present invention is to provide a steering assistance device and a working machine that can improve the working efficiency of the working machine. [Means for solving the problem]
[0009] One aspect of the steering assistance device according to the present invention is to A steering assistance device for assisting in steering a work machine having a boom that can be rotated, raised, lowered, and extended, comprising: With a transparent display A setting unit that sets a reference plane on the scenery seen through the display; The display device is equipped with a control unit that displays a virtual image for assisting in the operation of the work machine on the display so that the virtual image overlaps with a reference plane displayed on the display. When implementing a piloting assistance device as described above, the piloting assistance device may preferably be configured to be wearable on the user's head. Furthermore, the setting unit may set the reference plane to the scenery that the user sees through the display when using the piloting assistance device. Furthermore, the reference plane may be the ground surface in the scene the user sees through the display during use, or a plane arbitrarily selected by the user in the scene the user sees through the display. Furthermore, the control unit generates a virtual image seen from the user's viewpoint, and converts the generated virtual image into So that it overlaps with the user's field of vision Display on the display. When implementing the above-mentioned steering assistance device, specifically, The virtual image is an image showing a target position of a reference member moving with the boom; An image showing a trajectory of the reference member when it moves from a target position in accordance with at least one boom movement selected from a rotation, a lifting, and a telescopic movement; and It may include an image showing a line parallel to the vertical direction passing through the current position of a reference member that moves with the boom. Effect of the Invention
[0010] According to the present invention, it is possible to provide a steering assistance device and a work machine that can improve the work efficiency of the work machine. [Brief description of the drawings]
[0011] [Figure 1] FIG. 1 is a perspective view of a transmission type display device in a working machine operation assistance device according to one embodiment of the present invention worn by a user. [Diagram 2] FIG. 2 is a perspective view of a transmission type display device in a working machine operation assistance device according to one embodiment of the present invention. [Diagram 3] FIG. 3 is a block diagram of a working machine operation assistance device according to one embodiment of the present invention. [Figure 4] FIG. 4 is a schematic diagram of a virtual image displayed on a transmissive display device in a working machine operation assistance device according to one embodiment of the present invention. [Diagram 5] FIG. 5 is a schematic diagram showing a state in which the reference plane of a virtual image displayed on a transmissive display device in a working machine operation assistance device according to one embodiment of the present invention is changed. [Figure 6] FIG. 6 is a side view of a crane to which a working machine steering assistance device according to one embodiment of the present invention is connected. [Figure 7] FIG. 7 is a control block diagram of a crane to which a working machine operation assistance device according to one embodiment of the present invention is connected. [Figure 8] FIG. 8 is a block diagram illustrating a process performed when the operation assist device for a work machine according to one embodiment of the present invention generates information required for displaying the first virtual image and the second virtual image. [Figure 9] FIG. 9 is a schematic diagram showing a state in which a working machine driving assistance device according to one embodiment of the present invention is displaying a first virtual image, a second virtual image, a third virtual image, a fifth virtual image, and a seventh virtual image. [Figure 10] FIG. 10 is a block diagram for generating information required for the operation assist device of the work machine according to one embodiment of the present invention to display the third virtual image. [Figure 11] FIG. 11 is a block diagram for generating information required for the operation assist device of the work machine according to one embodiment of the present invention to display the fourth virtual image. [Figure 12] FIG. 12 is a schematic diagram showing a state in which the working machine driving assistance device according to one embodiment of the present invention displays the third virtual image, the fourth virtual image, the fifth virtual image, and the seventh virtual image. [Figure 13] FIG. 13 is a block diagram for generating information required for the operation assist device of the work machine according to one embodiment of the present invention to display the fifth virtual image. [Figure 14] FIG. 14 is a block diagram for generating information required for the operation assist device of the work machine according to one embodiment of the present invention to display the sixth virtual image. [Figure 15] FIG. 15 is a schematic diagram showing a state in which the working machine driving assistance device according to one embodiment of the present invention displays the first virtual image, the second virtual image, the sixth virtual image, and the seventh virtual image. [Figure 16] FIG. 16 is a block diagram for generating information required for the operation assist device of the work machine according to one embodiment of the present invention to display the seventh virtual image. [Figure 17] FIG. 17 is a schematic diagram showing a state in which the operation assist device for a work machine according to an embodiment of the present invention displays a modified example of the first virtual image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0012] Hereinafter, a working machine steering assistance device 1 according to one embodiment of the present invention will be described with reference to Figures 1 to 5. Figure 1 is a perspective view of a transmissive display device in a working machine steering assistance device according to one embodiment of the present invention worn by a user.
[0013] Fig. 2 is a perspective view of the transmissive display device 2 in the work machine steering support device 1. Fig. 3 is a block diagram of the work machine steering support device 1. Fig. 4 is a schematic diagram showing a virtual image Im displayed in the work machine steering support device 1.
[0014] 5 is a schematic diagram in which the reference plane S1 for displaying the virtual image Im is changed in the driving assistance device 1 for the work machine. In this embodiment, a rough terrain crane (hereinafter simply referred to as "crane") will be described as the work machine.
[0015] However, the work machine may be any work machine having a boom that can be raised and lowered via a swivel base, such as an all-terrain crane, a truck crane, or an aerial work platform.
[0016] 1 and 2, a driving assistance device 1 for a crane 11 (see FIG. 6), which is a work machine, is a system that uses mixed reality to assist in the driving of the crane 11. The driving assistance device 1 has a transmissive display device 2, a communication device 8, and an image processing device 9.
[0017] The transmissive display device 2 is a display device that allows a user to visually recognize the background of the transmissive display device 2 that overlaps with the display screen of the transmissive display device 2 as seen from the user's perspective. The transmissive display device 2 is a wearable terminal that is worn by the user on the body.
[0018] In this embodiment, the transmissive display device 2 will be described as a glasses-type wearable terminal. The transmissive display device 2 is worn by a user, that is, an operator of a crane 11. The transmissive display device 2 includes a frame 3, a transmissive display 4, a GNSS receiver 5 for the display device, a camera 6 for the display device, and a control device 7 for the display device.
[0019] The frame 3 supports a transmissive display 4, a display device GNSS receiver 5, a display device camera 6, and a display device control device 7. The frame 3 is configured as a glasses-type frame. The frame 3 is worn on the user's head. The frame 3 has a bridge supported by the user's nose and a pair of temples supported by both ears.
[0020] The transmissive display 4 is a display with a transmittance of approximately 50% or more. In other words, the transmissive display 4 is a transparent display that allows the user to see the scenery through the transmissive display 4. The transmissive display 4 is composed of a liquid crystal display, an organic EL display, an inorganic EL display, or the like that does not have a backlight, a polarizing filter, or a color filter.
[0021] In other words, the transmissive display 4 does not have any components such as a backlight or a filter that cover the back surface of the display screen facing the user. Therefore, the transmissive display 4 allows the background overlapping with the transmissive display 4 to be seen through from the user's viewpoint. In addition, the transmissive display 4 can display a virtual image Im, which is an image generated by the function of the transmissive display 4, on the display screen.
[0022] In this specification, a virtual image is an image or video other than the scenery viewed by the user on the transmissive display device, and refers to a line, figure, image, video, three-dimensional image, or three-dimensional video displayed by the transmissive display device. In other words, a virtual image is an image that can be displayed without discomfort on the scenery viewed by the user by converting an image of an object that does not actually exist into an image of a shape seen from the user's viewpoint. The virtual image is displayed superimposed on the scenery viewed by the user on the transmissive display device. The virtual image is also an image for assisting the user in operating a work machine.
[0023] In this specification, the term "viewpoint" refers to a standpoint from which an object is viewed, and the term "user's viewpoint" refers to the position of the user relative to an object when the user is viewing the object.
[0024] The transmissive display 4 transmits light incident from the back surface of a portion where the virtual image Im is not displayed, and blocks at least a part of the light incident from the back surface of a portion where the virtual image Im is displayed.
[0025] Furthermore, the transmissive display 4 displays a virtual image Im in the light-shielded portion. At this time, the virtual image Im seen from the user's viewpoint is displayed on the transmissive display 4. As a result, the transmissive display 4 allows the user to visually recognize a state in which the virtual image Im is superimposed on the scenery Ls seen from the user's viewpoint.
[0026] The transmissive display 4 is provided on the frame 3. The transmissive display 4 is positioned so as to cover the visual field of the right eye and the visual field of the left eye of a user wearing the frame 3. The transmissive display 4 is configured to be capable of displaying a virtual image Im for the right eye and a virtual image Im for the left eye as parallax images on the display screen.
[0027] This allows the transmissive display 4 to display a stereoscopic virtual image Im in almost the entire visual field of the user. Note that a technique for displaying a virtual image Im seen from the user's viewpoint in the visual field of the user in the transmissive display device 2 is well known.
[0028] The GNSS receiver 5 for the display device is a receiver that constitutes a global navigation satellite system. The GNSS receiver 5 for the display device receives distance measurement radio waves from satellites and detects the absolute coordinates of the GNSS receiver 5 for the display device, such as latitude, longitude, altitude, and direction.
[0029] The GNSS receiver 5 for the display device is provided on the frame 3 worn by the user. The GNSS receiver 5 for the display device detects information Ifa relating to the position of the user (pilot).
[0030] The information Ifa on the user's position includes the latitude, longitude, altitude, and direction of the transmissive display device 2. That is, the information Ifa on the user's position is information on the position of the transmissive display device 2.
[0031] In this embodiment, the GNSS receiver 5 for the display device is provided on the frame 3, but may be located in any position where it can detect the information Ifa relating to the user's position. For example, the GNSS receiver 5 for the display device may be attached to the body of the user who is wearing the transmissive display device 2.
[0032] The display device camera 6 captures an image in the direction of the line of sight of a user wearing the transmissive display device 2. The display device camera 6 is provided on the frame 3. The display device camera 6 is configured to capture an image in a range approximately equal to the visual field range of the user from a position approximately equal to the viewpoint of the user.
[0033] In this embodiment, the display device camera 6 is provided on the frame 3, but it may be located in any position that allows it to capture an image of the user's visual field from the user's viewpoint.
[0034] 3, the display device control device 7 corresponds to an example of a control unit, and is a control device that displays a virtual image Im on the transmissive display 4. The display device control device 7 actually includes a CPU, a ROM, a RAM, a HDD, etc., which are connected via a bus. Alternatively, the display device control device 7 may be configured with a one-chip LSI or the like.
[0035] The display device control device 7 is connected to the transmissive display 4, the display device GNSS receiver 5, and the display device camera 6. The display device control device 7 is connected to an image processing device 9 via a communication device 8. The display device control device 7 has information Ifd about a reference plane S1. In this embodiment, the reference plane S1 is set on the earth's surface.
[0036] The display device control device 7 can acquire information Ifa about the position of the user from the display device GNSS receiver 5. The display device control device 7 can acquire an image P captured by the display device camera 6 from the display device camera 6.
[0037] Furthermore, the display device control device 7 can acquire information If related to the virtual image Im from the image processing device 9. The display device control device 7 can generate a virtual image Im based on the acquired information If related to the virtual image Im. Furthermore, the display device control device 7 can cause the transmissive display 4 to display the virtual image Im.
[0038] In this specification, information about a virtual image means information necessary for displaying the virtual image on the display screen of the transmissive display device as an image on an arbitrary reference plane seen from the viewpoint of a user of the transmissive display device. The information about the virtual image includes the coordinates of the virtual image, the display state, etc.
[0039] 3 and 4, the transmissive display device 2 can display a virtual image Im on the display screen of the transmissive display 4 while being worn on the head of a user. Furthermore, the transmissive display device 2 can acquire information Ifa about the user's position, which is necessary for displaying the virtual image Im on the transmissive display 4, by the GNSS receiver 5 for the display device.
[0040] The transmissive display device 2 can generate a virtual image Im viewed from the user's viewpoint on the display screen of the transmissive display 4, through which the scenery Ls is transmitted, based on information If about the virtual image Im and information Ifa about the user's position.
[0041] Furthermore, the transmissive display device 2 can display the virtual image Im on the display screen of the transmissive display 4. This allows the user of the transmissive display device 2 to visually recognize the virtual image Im in a state where it is superimposed on the landscape Ls.
[0042] 3, the communication device 8 is a device that transmits and receives information among the transmissive display device 2, the image processing device 9, and the crane control device 30. The communication device 8 is provided in each of the display device control device 7 of the transmissive display device 2 and the image processing device 9.
[0043] The communication device 8 transmits and receives information between the crane control device 30 and the image processing device 9. In addition, the communication device 8 transmits and receives information between the display device control device 7 and the image processing device 9.
[0044] In this specification, the communication device means a device that transmits and receives information between the image processing device and the display device control device, between the image processing device and the input device, and between the image processing device and the work machine control device. The communication device may be configured integrally with the display device control device and the image processing device.
[0045] The image processing device 9 is a control device that generates information If about a virtual image Im to be displayed on the transmissive display device 2. The image processing device 9 is substantially composed of a CPU, a ROM, a RAM, a HDD, etc., connected via a bus. Alternatively, the image processing device 9 may be configured with a one-chip LSI or the like.
[0046] In this embodiment, the image processing device 9 is provided in the transmissive display device 2. The image processing device 9 stores various programs and data for controlling the operation of the transmissive display device 2 and the like and for processing image data. The image processing device 9 may be configured integrally with the display device control device 7 or the crane control device 30.
[0047] The image processing device 9 is connected to the display device control device 7 of the transmissive display device 2 via the communication device 8. The image processing device 9 acquires information Ifa about the user's position detected by the display device GNSS receiver 5 from the display device control device 7 via the communication device 8. The image processing device 9 also acquires the image P captured by the display device camera 6 from the display device control device 7 via the communication device 8.
[0048] Furthermore, the image processing device 9 acquires information Ifb about the position of the crane 11 from the crane control device 30 via the communication device 8. The information Ifb about the position of the crane 11 includes the latitude, longitude, altitude, and vehicle direction of the location of the rotation center of the crane 11. Furthermore, the image processing device 9 transmits information Ife about the target position Lt of the main hook 20a to the crane control device 30 via the communication device 8.
[0049] Furthermore, the image processing device 9 acquires information Ifc about the attitude of the crane 11 from the crane control device 30 via the communication device 8. The information Ifc about the attitude of the crane 11 includes the slewing angle of the crane 11, the boom hoisting angle, the boom length, the payout length of the main wire rope 24, the payout length of the sub-wire rope 26, and the operation direction and operation amount of each operating tool of the crane 11. In other words, the information Ifc about the attitude of the crane 11 includes, for example, information about the current position Lp of the main hook 20a of the crane 11, which is the reference member.
[0050] Moreover, the image processing device 9 acquires information Ifd about the reference plane S1 from the display device control device 7 via the communication device 8. The information Ifd about the reference plane S1 includes the position and inclination of the reference plane S1 (see FIG. 4) that serves as a reference for the position at which the virtual image Im is displayed as seen from the user's viewpoint.
[0051] The image processing device 9 calculates a plurality of reference coordinates C based on the acquired information Ifc about the attitude of the crane 11 and information Ifd about the reference plane S1. The reference coordinates C are coordinates of a virtual image Im to be displayed on the reference plane S1. The image processing device 9 corresponds to an example of a setting unit that sets the reference plane S1 on the scenery seen through the transmissive display 4.
[0052] The image processing device 9 generates information If about a virtual image Im based on the acquired information Ifa about the user's position, information Ifb about the crane 11's position, and information Ifd about the reference plane S1. The information If about the virtual image Im is information for displaying a plurality of reference coordinates C as a virtual image Im on the reference plane S1 as seen from the user's viewpoint on the display screen of the transmissive display 4. The image processing device 9 transmits the generated information If about the virtual image Im about the crane to the display device control device 7.
[0053] 3 and 4, the transmissive display device 2 acquires information If about a virtual image Im of the crane via the communication device 8. When the reference plane S1 is the ground surface, the virtual image Im seen from the user's viewpoint is displayed on the display screen of the transmissive display 4 overlapping with the ground surface viewed by the user through the transmissive display 4.
[0054] As shown in Figure 5, when the image processing device 9 detects a movement of a part of the user's body or a movement of a specific marker from the image P captured by the display device camera 6, it recognizes it as an input movement to input information If related to the virtual image Im.
[0055] For example, when the image processing device 9 detects an action of a user's hand grabbing and moving the reference plane S1 seen from the user's viewpoint, it changes the reference plane S1 to a new reference plane S2 having a new position and inclination according to the direction and amount of movement of the hand.
[0056] Similarly, when the image processing device 9 detects, for example, a movement of the user's hand indicating an arbitrary position as seen from the user's viewpoint, it changes the position where the virtual image Im is displayed according to the direction and amount of movement of the hand. In this way, the display device camera 6 has a function as an input device for inputting information If about the virtual image Im.
[0057] In this specification, the input device means a device that inputs information for displaying a virtual image to the image processing device. The input device may be configured integrally with the image processing device and the transmissive display device. The input device may be an input by a movement of a part of the body of a user or a movement of a marker within the field of view of the user wearing the transmissive display device.
[0058] The steering assistance device 1 configured in this manner acquires information Ifc about the attitude of the crane 11 from the crane control device 30 via the communication device 8. Furthermore, the steering assistance device 1 acquires information Ifa about the position of the user and information Ifd about the reference plane S1 via the communication device 8.
[0059] The steering assistance device 1 generates information If about the virtual image Im based on each piece of information acquired by the image processing device 9. The steering assistance device 1 displays the virtual image Im on the transmissive display device 2 based on the information If about the virtual image Im. The steering assistance device 1 displays the virtual image Im showing various information about the crane 11 superimposed on the scenery Ls viewed by the user, thereby suppressing the influence that the information acquired from the instructor has on the steering and assisting the user in steering the crane 11 so that the crane 11 can be efficiently operated.
[0060] A crane 11, which is a working machine according to one embodiment of the present invention, will be described below with reference to Figures 6 and 7. Figure 6 shows the overall configuration of the crane 11 to which the steering assistance device 1 is connected. Figure 7 shows the control configuration of the crane 11 to which the steering assistance device 1 is connected. Note that, although a rough terrain crane will be described in this embodiment, any working machine provided with a boom that can be raised and lowered via a swivel base, such as an all-terrain crane, a truck crane, or an aerial work vehicle, may be used.
[0061] 6 and 7, the crane 11 is a mobile crane that can be moved to any location. The crane 11 includes a vehicle 12, a crane device 16, a boom camera 19b that is an imaging device, and a crane control device 30 (see FIG. 7).
[0062] The vehicle 12 is a traveling body that transports the crane device 16. The vehicle 12 has a plurality of wheels 13. The vehicle 12 travels using an engine 14 as a power source. The vehicle 12 has outriggers 15. The outriggers 15 have hydraulically extendable extension beams on both sides in the width direction of the vehicle 12, and hydraulic jack cylinders that can be extended in a direction perpendicular to the ground.
[0063] The crane apparatus 16 is a work device that hoists a load W by a wire rope. The crane apparatus 16 includes a swivel base 17, a boom 19, a main hook block 20, a sub-hook block 21, a hoisting hydraulic cylinder 22, a main winch 23, a main wire rope 24, a sub-winch 25, a sub-wire rope 26, a cabin 27, and the like.
[0064] The swivel base 17 is a rotating device that enables the crane device 16 to rotate. The swivel base 17 is provided on the frame of the vehicle 12 via an annular bearing. The swivel base 17 is configured to be freely rotatable around the center of the annular bearing.
[0065] The swivel base 17 has a hydraulic motor 17a for swivel, which is an actuator. The swivel base 17 is configured to be swivelable in one direction and the other direction by the hydraulic motor 17a for swivel.
[0066] The crane GNSS receiver 18 (see FIG. 7) is a receiver that constitutes the Global Navigation Satellite System.
[0067] The crane GNSS receiver 18 receives ranging radio waves from satellites and detects the absolute coordinates of the crane GNSS receiver 18, such as latitude, longitude, altitude, and direction. The crane GNSS receiver 18 is provided on the swivel base 17.
[0068] The crane GNSS receiver 18 detects information Ifb about the position of the crane 11. The information Ifb about the position of the crane 11 includes the latitude, longitude, altitude, direction, and the like of the crane 11.
[0069] The boom 19 is a movable support that supports the main wire rope 24 and the sub wire rope 26. The boom 19 is made up of a plurality of boom members. The base end of the base boom member of the boom 19 is swingably mounted on the rotating base 17 approximately at the center of the rotating base 17.
[0070] The boom 19 has telescopic hydraulic cylinders 19a (see FIG. 7) which are actuators that extend and retract each boom member, and a hoisting hydraulic cylinder 22 that raises and lowers the boom members. The boom 19 is extended and retracted in the axial direction by the telescopic hydraulic cylinders 19a.
[0071] The boom 19 also has a boom camera 19b (see FIG. 7) which is an imaging device. The boom 19 also has a jib 19c which is an extension member. The image captured by the boom camera 19b is displayed on a display device of the crane 11 or the like.
[0072] The main hook block 20 and the sub-hook block 21 are components that suspend a load W. The main hook block 20 has a plurality of hook sheaves around which the main wire rope 24 is wound, and a main hook 20a that suspends the load W. The sub-hook block 21 has a sub-hook 21a that suspends the load W.
[0073] The hoisting hydraulic cylinder 22 is an actuator that raises and lowers the boom 19 and maintains the posture of the boom 19. The end of the cylinder portion of the hoisting hydraulic cylinder 22 is connected to the swivel base 17 so that it can be raised and lowered freely. The end of the rod portion of the hoisting hydraulic cylinder 22 is connected to the base boom member of the boom 19 so that it can swing freely.
[0074] The main winch 23 is a device that reels in (winds up) and reels out (winds down) the main wire rope 24. The sub winch 25 is a device that reels in (winds up) and reels out (winds down) the sub wire rope 26.
[0075] The main winch 23 is rotated by a main hydraulic motor (not shown) which serves as an actuator for a main drum around which the main wire rope 24 is wound. The sub winch 25 is rotated by a sub hydraulic motor (not shown) which serves as an actuator for a sub drum around which the sub wire rope 26 is wound.
[0076] The cabin 27 is a housing that covers the operator's seat. The cabin 27 is mounted on the rotating base 17. The cabin 27 is provided with an operator's seat (not shown). The operator's seat is provided with a travel operation tool 28 for operating the vehicle 12 to travel, a rotation operation tool 29a for operating the rotation hydraulic motor 17a of the crane apparatus 16, an extension operation tool 29b for operating the extension cylinder, a hoisting operation tool 29c for operating the hoisting hydraulic cylinder 22, a main operation tool 29d for operating the main winch 23, and a sub operation tool 29e for operating the sub winch 25, and other crane apparatus operation tools 29 (see FIG. 2).
[0077] As shown in Fig. 7, the crane controller 30 controls each actuator of the crane 11. The crane controller 30 is provided inside the cabin 27. In reality, the crane controller 30 includes a CPU, a ROM, a RAM, a HDD, etc., which are connected via a bus. The crane controller 30 may be configured from a one-chip LSI or the like.
[0078] The crane control device 30 stores various programs and data for controlling the operation of each actuator and switching valve, etc., and for processing image data. The crane control device 30 has a communication device 30a. The communication device 30a transmits and receives information to and from the image processing device 9 of the maneuvering assistance device 1.
[0079] The crane control device 30 is connected to the crane GNSS receiver 18. The crane control device 30 can acquire information Ifb about the position of the crane 11. The crane control device 30 is connected to a traveling operation tool 28 and a crane equipment operation tool 29.
[0080] The crane control device 30 can obtain operation signals of the traveling operation tool 28 and the crane device operation tool 29. The crane control device 30 generates a control signal for the vehicle 12 based on the operation signal generated by operating the traveling operation tool 28.
[0081] Similarly, the crane controller 30 generates a control signal for the crane apparatus 16 based on an operation signal generated by operating the crane apparatus operating tool 29. The crane controller 30 can transmit the generated control signal to each actuator of the crane apparatus 16.
[0082] The crane control device 30 is connected to the image processing device 9 of the steering assistance device 1 via the communication device 30a. The crane control device 30 transmits information Ifb about the position of the crane 11, information Ifc about the attitude of the crane 11, and information Ifh about the path of the main hook 20a to the image processing device 9. The crane control device 30 can obtain information Ife about the target position Lt of the main hook 20a from the image processing device 9 via the communication device 30a.
[0083] The crane 11 thus configured can move the vehicle 12 to any desired position by operating the traveling operation tool 28. Also, the crane 11 can transport a load W to any desired position by rotating, raising and lowering the boom 19 by operating the crane device operation tool 29. Also, the crane 11 can hoist or lower the load W with the main winch 23 or the like by operating the crane device operation tool 29.
[0084] Next, the operation assistance provided to the operator of the crane 11 by the operation assistance device 1 will be described with reference to Figs.
[0085] Fig. 8 shows a state in which the operation assistance device 1 of the crane 11 acquires information necessary for displaying the first virtual image Im1 and the second virtual image Im2. Fig. 9 shows a state in which the operation assistance device 1 of the crane 11 displays the first virtual image Im1, the second virtual image Im2, the third virtual image Im3, the fifth virtual image Im5, and the seventh virtual image Im7.
[0086] Fig. 10 shows how the operation assistance device 1 of the crane 11 acquires information necessary for displaying the third virtual image Im3. Fig. 11 shows how the operation assistance device 1 of the crane 11 acquires information necessary for displaying the fourth virtual image Im4.
[0087] Fig. 12 shows a state in which the operation assistance device 1 of the crane 11 displays the third virtual image Im3, the fourth virtual image Im4, the fifth virtual image Im5, and the seventh virtual image Im7. Fig. 13 shows a state in which the operation assistance device 1 of the crane 11 acquires information necessary for displaying the fifth virtual image Im5.
[0088] Fig. 14 shows the state in which the operation assistance device 1 of the crane 11 acquires information required for displaying the sixth virtual image Im6. Fig. 15 shows the state in which the operation assistance device 1 of the crane 11 displays the first virtual image Im1, the second virtual image Im2, the sixth virtual image Im6, and the seventh virtual image Im7. Fig. 16 shows the state in which the operation assistance device 1 of the crane 11 acquires information required for displaying the seventh virtual image Im7.
[0089] The operator of the crane 11 is a user of the transmissive display device 2. The operator of the crane 11 operates the crane 11 while wearing the transmissive display device 2 on his / her head. The operator of the crane 11 also operates the crane 11 while seated in a cabin 27 (see FIG. 6 ) of the crane 11. In this embodiment, the reference member is the main hook 20a of the crane 11. The reference member may also be a load W transported by the crane 11.
[0090] <Display of trajectory of main hook 20a at current position Lp> As shown in FIGS. 8 and 9, the steering assistance device 1 displays, on the transmissive display device 2, a trajectory of the main hook 20a, which is a reference member, at a current position Lp, as a first virtual image Im1.
[0091] In this specification, a trajectory means a figure in which a set of coordinates through which a reference member of a work machine passes by operation of each operating tool of the work machine is expressed by a continuous line. Also, the trajectory includes a figure in which a set of coordinates through which a reference member of a work machine passes by operation of each operating tool is converted into coordinates on an arbitrary reference plane and expressed by a continuous line.
[0092] Based on the acquired information Ifc about the attitude of the crane 11 and information Ifd about the reference plane S1, the image processing device 9 calculates multiple first reference coordinates C1 for displaying on the reference plane S1 the current position Lp of the main hook 20a based on the tip of the boom 19 and the trajectory from the current position Lp of the main hook 20a moved by the crane 11 when the crane equipment operating tool 29 (see Figure 7) of the crane 11 is operated (hereinafter simply referred to as "the trajectory from the current position Lp of the main hook 20a").
[0093] Based on the acquired information Ifa regarding the position of the operator (user) and information Ifb regarding the position of the crane 11, the image processing device 9 generates information for displaying multiple first reference coordinates C1 on the display screen of the transmissive display 4 as a first virtual image Im1 on the reference plane S1 viewed from the operator's viewpoint.
[0094] That is, the image processing device 9 generates information about the current position Lp of the main hook 20a on the reference plane S1 seen from the operator's viewpoint and the trajectory of the main hook 20a from the current position Lp as information If1 about the first virtual image Im1 of the crane 11. The image processing device 9 transmits the generated information If1 about the first virtual image Im1 of the crane 11 to the display device control device 7.
[0095] The display device control device 7 acquires information If1 related to the first virtual image Im1 of the crane 11 via the communication device 8. The display device control device 7 displays, on the display screen of the transmissive display 4, the current position Lp of the main hook 20a on the reference plane S1 seen from the operator's viewpoint, which is the first virtual image Im1, and a trajectory from the current position Lp of the main hook 20a.
[0096] When the reference plane S1 is the ground surface, the display device control device 7 displays the current position Lp of the main hook 20a on the ground surface as seen from the pilot's viewpoint and the trajectory from the current position Lp of the main hook 20a on the ground surface as seen from the pilot's viewpoint as a first virtual image Im1 on the display screen of the transparent display 4 overlapping with the ground surface viewed by the pilot through the transparent display 4.
[0097] Furthermore, the first virtual image Im1 may be regarded as an image projected onto the reference plane S1 of the trajectory along which the main hook 20a moves when the pilot operates the turning operation tool 29a from a state in which the main hook 20a is located at the current position Lp.
[0098] The first virtual image Im1 may be an image projected onto the reference plane S1 of the trajectory along which the main hook 20a moves when the pilot operates the telescopic operating device 29b or the elevation operating device 29c from a state in which the main hook 20a is located at the current position Lp.
[0099] Alternatively, the first virtual image Im1 may be an image projected onto the reference plane S1 of the trajectory along which the main hook 20a moves when the pilot simultaneously operates at least two operating tools selected from the turning operating tool 29a, the extension operating tool 29b, and the elevation operating tool 29c from a state in which the main hook 20a is located at the current position Lp.
[0100] The operator recognizes that the current position Lp of the main hook 20a and the trajectory from the current position Lp of the main hook 20a are displayed superimposed on the ground surface of the work site, which is the actual scenery Ls, through the transmission type display device 2.
[0101] Even if the pilot cannot see the main hook 20a from the cockpit, the pilot can visually recognize the current position Lp of the main hook 20a and the trajectory from the current position Lp of the main hook 20a on the ground surface.
[0102] The piloting assistance device 1 superimposes and displays the first virtual image Im1 within the pilot's field of vision, thereby providing the pilot with information for understanding the current position Lp of the main hook 20a and the positional relationship between the main hook 20a and features.
[0103] Furthermore, the operation assistance device 1 displays within the operator's field of vision the trajectory of the main hook 20a from any direction and position desired by the operator. Therefore, even if the operator receives insufficient instructions from the person in charge of slinging, the operator can supplement the information necessary to operate the crane 11 with the trajectory of the main hook 20a displayed within the operator's field of vision. This allows the operator to efficiently operate the crane 11 by suppressing the effect that the information obtained from the person in charge of slinging has on the operation.
[0104] <Trajectory display of main hook 20a at target position Lt> As shown in FIGS. 8 and 9, the steering assistance device 1 may display, on the transmissive display device 2, a trajectory of the main hook 20a at a target position Lt with respect to the tip of the boom 19 as a reference, as a second virtual image Im2.
[0105] The image processing device 9 acquires an image of the operator's hand or a marker from the operator's viewpoint by the display device camera 6 having the function of an input device. The image processing device 9 judges whether the acquired image is information Ife about the target position Lt of the main hook 20a from the operator's hand movement or the type of marker in the acquired image.
[0106] When the image processing device 9 determines that the acquired image is information Ife regarding the target position Lt of the main hook 20a, it identifies the target position Lt of the main hook 20a from the shape, position and movement direction of the operator's hand in the image or the type, position and movement direction of the marker, etc.
[0107] The image processing device 9 calculates a plurality of second reference coordinates C2 based on the acquired information Ifc on the attitude of the crane 11, information Ifd on the reference plane S1, and information Ife on the target position Lt of the main hook 20a.
[0108] The multiple second reference coordinates C2 are coordinates for displaying, on the reference plane S1, the target position Lt of the main hook 20a and the trajectory from the target position Lt of the main hook 20a moved by the crane 11 when the crane equipment operating tool 29 (see Figure 7) of the crane 11 is operated (hereinafter simply referred to as "the trajectory from the target position Lt of the main hook 20a").
[0109] Based on the acquired information Ifa regarding the operator's position and information Ifb regarding the position of the crane 11, the image processing device 9 generates information for displaying multiple second reference coordinates C2 on the display screen of the transmissive display 4 as a second virtual image Im2 on the reference plane S1 viewed from the operator's viewpoint.
[0110] In other words, the image processing device 9 generates information regarding the target position Lt of the main hook 20a on the reference plane S1 as seen from the operator's viewpoint and the trajectory of the main hook 20a from the target position Lt as information If2 regarding the second virtual image Im2 of the crane 11.
[0111] The image processing device 9 transmits information If2 relating to the generated second virtual image Im2 of the crane 11 to the display device control device 7.
[0112] The display device control device 7 acquires information If2 related to the second virtual image Im2 of the crane 11 via the communication device 8. The display device control device 7 displays the second virtual image Im2 on the display screen of the transmissive display 4 based on the information If2 related to the second virtual image Im2 of the crane 11. Here, the second virtual image Im2 is an image related to the target position Lt of the main hook 20a on the reference plane S1 as viewed from the operator's viewpoint and the trajectory of the main hook 20a from the target position Lt.
[0113] The operator recognizes that the target position Lt of the main hook 20a and the trajectory from the target position Lt of the main hook 20a are displayed superimposed on the ground surface of the work site, which is the actual scenery Ls, through the transmission type display device 2.
[0114] When the first virtual image Im1, which is the trajectory from the current position Lp of the main hook 20a, and the second virtual image Im2, which is the trajectory from the target position Lt of the main hook 20a, are both displayed, the pilot can visually recognize on the ground surface the positional relationship between the current position Lp of the main hook 20a and the target position Lt, and the positional relationship between the target position Lt of the main hook 20a and features on the ground, even if the main hook 20a cannot be seen from the cockpit.
[0115] Therefore, even if the instructions from the instructor are insufficient, the operator can supplement the information required to move the main hook 20a to the target position Lt by the trajectory of the main hook 20a at the target position Lt displayed within the operator's field of vision. This allows the operator to suppress the influence of the information obtained from the instructor on the operation, and efficiently operate the work machine.
[0116] <Display of perpendicular line at current position Lp of main hook 20a> As shown in Figures 9 and 10, the steering assistance device 1 may display a perpendicular line (i.e., a line parallel to the vertical direction) passing through the current position Lp of the main hook 20a or the display position as a third virtual image Im3 on the transmissive display device 2.
[0117] Based on the acquired information Ifc regarding the posture of the crane 11 and information Ifd regarding the reference plane S1, the image processing device 9 calculates multiple first reference coordinates C1 for displaying the current position Lp of the main hook 20a and the trajectory from the current position Lp of the main hook 20a on the reference plane S1.
[0118] Furthermore, the image processing device 9 calculates a plurality of third reference coordinates C3 for displaying a perpendicular line passing through the current position Lp of the main hook 20a, based on the calculated first reference coordinates C1.
[0119] Based on the acquired information Ifa regarding the operator's position and information Ifb regarding the position of the crane 11, the image processing device 9 generates information for displaying multiple third reference coordinates C3 as a third virtual image Im3 viewed from the operator's viewpoint on the display screen of the transmissive display 4.
[0120] That is, the image processing device 9 generates information about a perpendicular line passing through the current position Lp of the main hook 20a as seen from the operator's viewpoint as information If3 about the third virtual image Im3 of the crane 11.
[0121] The image processing device 9 transmits information If3 relating to the generated third virtual image Im3 of the crane 11 to the display device control device 7. Note that the information If3 relating to the third virtual image Im3 of the crane 11 may include information about a perpendicular line passing through the target position Lt of the main hook 20a.
[0122] The display device control device 7 acquires information If3 related to the third virtual image Im3 of the crane 11 via the communication device 8. The display device control device 7 displays the third virtual image Im3 on the display screen of the transmissive display 4 based on the information If3 related to the third virtual image Im3 of the crane 11. Here, the third virtual image Im3 is an image related to a perpendicular line passing through the current position Lp of the main hook 20a as seen from the operator's viewpoint.
[0123] The operator recognizes that the perpendicular line indicating the current position Lp of the main hook 20a is displayed superimposed on the actual scenery Ls by the transmission-type display device 2.
[0124] When a perpendicular line from the current position Lp of the main hook 20a, which is the third virtual image Im3, and a perpendicular line from the target position Lt of the main hook 20a, which is the second virtual image Im2, are both displayed, the pilot can recognize the positional relationship between the current position Lp of the main hook 20a and the target position Lt even if the main hook 20a is not visible from the cockpit.
[0125] In addition, the operator can visually confirm the positional relationship between the vertical trajectory of the main hook 20a and features at the current position Lp or the target position Lt of the main hook 20a. Therefore, even if the instructions from the instructor are insufficient, the operator can suppress the influence of the information obtained from the instructor on the operation, and can efficiently operate the crane 11.
[0126] <Display of the trajectory of the main hook 20a on the new reference plane S2> As shown in Figures 11 and 12, the steering assistance device 1 may display at least one of the trajectory at the current position Lp of the main hook 20a and the trajectory at the target position Lt on an arbitrarily specified new reference plane S2 as a fourth virtual image Im4 on the transparent display device 2.
[0127] The image processing device 9 acquires an image P of the operator's hand or marker from the operator's viewpoint by the display device camera 6 having the function of an input device. The image processing device 9 determines whether the acquired image P is information Iff about a new reference plane S2 from the operator's hand movement or the type of marker.
[0128] When the image processing device 9 determines that the acquired image is information Iff about the new reference plane S2, it identifies the position and inclination of the new reference plane S2 from the shape, position and movement direction of the pilot's hand or the type, position and movement direction of the marker in the image P.
[0129] The image processing device 9 calculates a plurality of fourth reference coordinates C4 based on the information Ifc about the posture of the crane 11 and the information Iff about the new reference plane S2. The fourth reference coordinates C4 are coordinates for displaying the current position Lp of the main hook 20a and the trajectory from the current position Lp of the main hook 20a on the new reference plane S2.
[0130] Based on the acquired information Ifa regarding the operator's position and information Ifb regarding the position of the crane 11, the image processing device 9 generates information for displaying multiple fourth reference coordinates C4 on the display screen of the transmissive display 4 as a fourth virtual image Im4 on a new reference plane S2 viewed from the operator's viewpoint.
[0131] In other words, the image processing device 9 generates information regarding the current position Lp of the main hook 20a on the new reference plane S2 as seen from the operator's viewpoint and the trajectory of the main hook 20a from the current position Lp as information If4 regarding the fourth virtual image Im4 of the crane 11.
[0132] The image processing device 9 transmits information If4 relating to the generated fourth virtual image Im4 of the crane 11 to the display device control device 7.
[0133] The information If4 about the fourth virtual image Im4 of the crane 11 may include information about the target position Lt of the main hook 20a and the trajectory of the main hook 20a from the target position Lt.
[0134] The display device control device 7 acquires information If4 related to the fourth virtual image Im4 of the crane 11 via the communication device 8. The display device control device 7 displays the fourth virtual image Im4 on the display screen of the transmissive display 4 based on the information If4 related to the fourth virtual image Im4 of the crane 11. Here, the fourth virtual image Im4 is an image related to the current position Lp of the main hook 20a on the new reference plane S2 as seen from the operator's viewpoint and the trajectory from the current position Lp of the main hook 20a.
[0135] The operator perceives through the transparent display device 2 that the current position Lp of the main hook 20a and the trajectory from the current position Lp of the main hook 20a are superimposed on a new reference plane S2 that has been arbitrarily set within the work site, which is the actual scenery Ls.
[0136] When the fourth virtual image Im4 displays both the trajectory from the current position Lp of the main hook 20a and the trajectory from the target position Lt of the main hook 20a, the pilot can visually recognize, on the new reference plane S2, the positional relationship between the current position Lp of the main hook 20a and the target position Lt, and the positional relationship between the main hook 20a and features at the target position Lt, even if the pilot cannot see the main hook 20a from the cockpit.
[0137] Therefore, even if the instructions from the instructor are insufficient, the operator can supplement the information required to move the main hook 20a to the target position Lt by the trajectory at the current position Lp and the target position Lt of the main hook 20a on the new reference plane S2 displayed within the operator's field of vision. This allows the operator to efficiently operate the crane 11 while suppressing the influence of the information obtained from the instructor on the operation.
[0138] <Display of the movement direction of the main hook 20a> As shown in FIGS. 12 and 13, the steering assistance device 1 may display, on the transmissive display device 2, a vector diagram indicating the moving direction of the main hook 20a as a fifth virtual image Im5.
[0139] As described above, the image processing device 9 generates information If1 related to the first virtual image Im1 of the crane 11. Furthermore, the image processing device 9 acquires information Ifg related to the moving direction and moving speed of the main hook 20a from the crane control device 30 via the communication device 8.
[0140] The image processing device 9 calculates a plurality of fifth reference coordinates C5 based on the information Ifc on the attitude of the crane 11, the information Ifd on the reference plane S1, and the information Ifg on the moving direction and moving speed of the main hook 20a. The fifth reference coordinates C5 are coordinates for displaying a vector diagram representing the moving direction and moving speed of the main hook 20a on the reference plane S1.
[0141] Based on the acquired information Ifa regarding the operator's position and information Ifb regarding the position of the crane 11, the image processing device 9 generates information for displaying multiple fifth reference coordinates C5 as a fifth virtual image Im5 viewed from the operator's viewpoint on the display screen of the transmissive display 4.
[0142] That is, the image processing device 9 generates information about a vector diagram indicating the moving direction of the main hook 20a from the current position Lp as seen from the operator's viewpoint as information If5 about the fifth virtual image Im5 of the crane 11.
[0143] The image processing device 9 transmits information If1 related to the generated first virtual image Im1 of the crane 11 and information If5 related to the generated fifth virtual image Im5 of the crane 11 to the display device control device 7.
[0144] The display device control device 7 obtains information If1 related to a first virtual image Im1 of the crane 11 and information If5 related to a fifth virtual image Im5 of the crane 11 via the communication device 8.
[0145] The display device control device 7 displays, on the display screen of the transmissive display 4, the current position Lp of the main hook 20a as seen from the operator's viewpoint, which is the first virtual image Im1, based on information If1 related to the first virtual image Im1 of the crane 11.
[0146] In addition, the display device control device 7 displays, on the display screen of the transparent display 4, a vector diagram showing the direction of movement of the main hook 20a from the current position Lp as seen from the operator's viewpoint, which is the fifth virtual image Im5, based on information If5 regarding the fifth virtual image Im5 of the crane 11.
[0147] The operator recognizes that the direction of movement of the main hook 20a from the current position Lp due to the operation of the crane operation tool 29 is displayed superimposed on the actual scenery Ls on the transmission type display device 2.
[0148] When the current position Lp of the main hook 20a, which is the first virtual image Im1, and the target position Lt of the main hook 20a, which is the second virtual image Im2, are both displayed, the operator can recognize the positional relationship between the moving direction of the main hook 20a from the current position Lp by operating the crane apparatus operating tool 29 and the target position Lt, even in a state where the main hook 20a cannot be seen from the operator's seat. Therefore, even if the instructions from the instructor are insufficient, the operator can suppress the influence of the information obtained from the instructor on the operation, and can efficiently operate the crane 11.
[0149] <Route display of main hook 20a> As shown in FIGS. 14 and 15, the steering assistance device 1 may display, on the transmissive display device 2, a path from the current position Lp of the main hook 20a to the target position Lt as a sixth virtual image Im6.
[0150] The image processing device 9 acquires information Ife about the attitude of the crane 11, including information about the current position Lp of the main hook 20a. Furthermore, the image processing device 9 acquires information Ife about the target position Lt of the main hook 20a by the display device camera 6. The image processing device 9 transmits the information Ife about the target position Lt of the main hook 20a to the crane control device 30 via the communication device 8.
[0151] The crane control device 30 acquires information Ife about the target position Lt of the main hook 20a. The crane control device 30 calculates a path of the main hook 20a that minimizes the pump flow rate of each actuator when the main hook 20a is moved from the current position Lp to the target position Lt of the main hook 20a.
[0152] The crane control device 30 transmits information Ifh relating to the calculated path of the main hook 20a to the image processing device 9 via the communication device 8. Note that a method for calculating the path that minimizes the pump flow rate of each actuator when the main hook 20a is moved from the current position Lp to the target position Lt is a known technique.
[0153] The image processing device 9 generates information If1 related to the first virtual image Im1 of the crane 11 and information If2 related to the second virtual image Im2 of the crane 11 as described above.
[0154] Furthermore, the image processing device 9 acquires information Ifh relating to the path of the main hook 20a. The image processing device 9 calculates a plurality of sixth reference coordinates C6 based on the acquired information Ifc relating to the attitude of the crane 11 and the information Ifh relating to the path of the main hook 20a. The sixth reference coordinates C6 are coordinates for displaying the path from the current position Lp of the main hook 20a to the target position Lt.
[0155] Based on the acquired information Ifa regarding the operator's position and information Ifb regarding the position of the crane 11, the image processing device 9 generates information for displaying multiple sixth reference coordinates C6 as a sixth virtual image Im6 viewed from the operator's viewpoint on the display screen of the transmissive display 4.
[0156] That is, the image processing device 9 generates information about the path from the current position Lp of the main hook 20a to the target position Lt as viewed from the operator's viewpoint, as information If6 about the sixth virtual image Im6 of the crane 11.
[0157] The image processing device 9 transmits information If1 regarding the first virtual image Im1 of the crane 11, information If2 regarding the second virtual image Im2 of the crane 11, and information If6 regarding the sixth virtual image Im6 of the crane 11 to the display device control device 7.
[0158] The display device control device 7 obtains information If1 related to a first virtual image Im1 of the crane 11, information If2 related to a second virtual image Im2 of the crane 11, and information If6 related to a sixth virtual image Im6 of the crane 11 via the communication device 8.
[0159] The display device control device 7 displays a sixth virtual image Im6 on the display screen of the transmissive display 4, together with the first virtual image Im1 and the second virtual image Im2, based on information If6 about the sixth virtual image Im6 of the crane 11. Here, the sixth virtual image Im6 is an image relating to the path from the current position Lp of the main hook 20a to the target position Lt as viewed from the operator's viewpoint.
[0160] The operator recognizes that the path from the current position Lp of the main hook 20a to the target position Lt is displayed on the transmission type display device 2 in a superimposed manner on the ground surface of the work site, which is the actual scenery Ls.
[0161] The operator can visually confirm in advance the positional relationship between the current position Lp of the main hook 20a and the target position Lt, and the positional relationship between the path of the main hook 20a and features. Therefore, even if the operator does not receive sufficient instructions from the instructor, the operator can supplement the information required to move the main hook 20a from the current position Lp to the target position Lt by the path of the main hook 20a displayed within the operator's field of vision. This allows the operator to efficiently operate the crane 11 while suppressing the effect of the information obtained from the instructor on the operation.
[0162] <Indicator display> As shown in FIGS. 15 and 16, the operation assistance device 1 may display various indicators related to the operation of the crane 11 on the transmissive display device 2 as a seventh virtual image Im7.
[0163] The image processing device 9 has information Ifi about a specific area A of the crane 11 that displays only a specific virtual image Im as seen from the operator's viewpoint. The image processing device 9 obtains information Ifj about the operating state of the crane 11 from the crane control device 30 via the communication device 8.
[0164] Based on information Ifi regarding the specific area A of the crane 11 and information Ifj regarding the operating state of the crane 11, the image processing device 9 calculates multiple seventh reference coordinates C7 for displaying an engine tachometer, an oil pressure gauge, an oil sound gauge, various warning lights, etc., which indicate the operating state of the crane 11, in the specific area A.
[0165] The image processing device 9 generates information for displaying the plurality of seventh reference coordinates C7 on the display screen of the transmissive display 4 as a seventh virtual image Im7 viewed from the pilot's viewpoint.
[0166] In other words, the image processing device 9 generates information If7 regarding the seventh virtual image Im7 of the crane 11, which information is related to the engine tachometer, oil pressure gauge, oil sound gauge, various warning lights, etc., which indicate the operating status of the crane and are displayed in the specific area A seen from the operator's perspective.
[0167] The image processing device 9 transmits information If7 relating to the generated seventh virtual image Im7 of the crane 11 to the display device control device 7.
[0168] The display device control device 7 acquires information If7 related to the seventh virtual image Im7 of the crane 11 via the communication device 8. The display device control device 7 displays the seventh virtual image Im7 on a specific area A of the transmissive display 4 based on the information If7 related to the seventh virtual image Im7 of the crane 11. Here, the seventh virtual image Im7 is an image related to an engine tachometer, an oil pressure gauge, an oil sound gauge, and various warning lights that indicate the operating state of the crane as seen from the operator's viewpoint.
[0169] The operator perceives that the engine tachometer, oil pressure gauge, oil sound gauge, various warning lights, etc., which indicate the operating status of the crane 11, are superimposed and displayed on the actual landscape Ls through the transmissive display device 2.
[0170] Therefore, the operator can visually check indicators relating to the operating status of the crane 11 without moving his / her line of sight to the operation panel in the cabin 27, thereby enabling the operator to operate the crane 11 efficiently.
[0171] <Trajectory display when using combined operations> As shown in FIG. 17, the maneuvering assistance device 1 may display, on the transmissive display device 2, a trajectory in the case where a plurality of crane operating tools 29 are operated in combination, as a modified example of the first virtual image Im1.
[0172] When the image processing device 9 acquires multiple operation signals of multiple operating tools among the crane equipment operating tools 29, it generates information for displaying the trajectory of the main hook 20a from the current position Lp when each part moves at a speed based on the acquired multiple operation signals.
[0173] The display device control device 7 displays, in accordance with the generated information, the trajectory of the main hook 20a from the current position Lp when each part moves at a speed based on multiple operation signals, as a modified example of the first virtual image Im1.
[0174] The first virtual image Im1, which is a trajectory from the current position Lp of the main hook 20a, can be changed to an arbitrary trajectory by operating a plurality of the operating tools 29 for the crane apparatus.
[0175] The first virtual image Im1 changes from a circular arc trajectory passing through the current position Lp to an elliptical trajectory by, for example, a combined operation of the turning operation tool 29a and the raising and lowering operation tool 29c of the boom 19. Therefore, when the operator operates the crane operation tool 29 to cause the first virtual image Im1 to pass through the target position Lt, the operator can move the main hook 20a to the target position Lt by maintaining the current operation amount of the crane operation tool 29.
[0176] <Other embodiments> In each of the above-mentioned embodiments, the operating assistance device 1 of the crane 11 displays any one of the first virtual image Im1, the second virtual image Im2, the third virtual image Im3, the fourth virtual image Im4, the fifth virtual image Im5, the sixth virtual image Im6, and the seventh virtual image Im7 on the transparent display device 2.
[0177] However, the operation assistance device 1 of the crane 11 may display at least one of the first virtual image Im1 to the seventh virtual image Im7 selected arbitrarily by the user of the transmissive display device 2. Furthermore, the operation assistance device 1 of the crane 11 is not limited to the virtual images displayed in the above-described embodiment, and may display any virtual image that assists in the operation of the work machine.
[0178] In the above-described embodiment, the transmissive display device 2 of the operation assistance device 1 is worn by an operator who operates the crane 11 inside the cabin 27 of the crane 11. However, the transmissive display device 2 may be worn by an operator who remotely operates the crane 11 from outside the crane 11.
[0179] For example, when an operator remotely operating the crane 11 near the cargo W to be transported is wearing the transparent display device 2, the operator can more accurately visually recognize the current position Lp of the main hook 20a relative to the position of the cargo W by displaying the first virtual image Im1 and the second virtual image Im2 on the transparent display device 2.
[0180] In each of the above-described embodiments, the transmissive display device 2 of the operation assistance device 1 for the crane 11 is worn by the operator of the crane 11. However, the transmissive display device 2 may be worn by a worker other than the operator of the crane 11.
[0181] For example, if the slinger is wearing the transparent display device 2, the slinger can more accurately convey the current position Lp of the main hook 20a relative to the cargo W, etc. to the operator of the crane 11 by displaying the first virtual image Im1 and the third virtual image Im3 on the transparent display device 2.
[0182] In each of the above-described embodiments, the transmissive display device 2 of the operation assistance device 1 for the crane 11 is worn by the operator of the crane 11. However, the transmissive display device 2 may be worn by at least one of the operator of the crane 11 and a worker other than the operator of the crane 11.
[0183] For example, if the operator and slinger of the crane 11 are each wearing a see-through display device 2, the operation assistance device 1 of the crane 11 causes each of the see-through display devices 2 to display the same or different types of virtual images Im based on the same information Ifc about the attitude of the crane 11. As a result, the operator and slinger of the crane 11 visually recognize the virtual images Im based on the same information Ifc about the attitude of the crane 11 within the field of view at each position, and can therefore more accurately share the position of the main hook 20a relative to the cargo W, etc.
[0184] In the above-described embodiment, the steering assistance device 1 is configured as a steering assistance device 1 for the crane 11. However, the steering assistance device may be configured as a steering assistance device for an aerial work vehicle that is a work machine.
[0185] The steering assistance device for the aerial work vehicle acquires, via the communication device, information regarding the position of the user, information regarding the position of the aerial work vehicle, information regarding the attitude of the aerial work vehicle, and information regarding a reference plane including the position and attitude of a reference plane serving as a reference for displaying a virtual image. The image processing device of the steering assistance device for the aerial work vehicle generates information regarding a virtual image for displaying, on a display screen of a transparent display device, a current position and a trajectory from the current position of a bucket, which is a reference member moved by the aerial work vehicle, as a virtual image on an arbitrary reference plane as viewed from the viewpoint of the operator of the aerial work vehicle. The transparent display device acquires information regarding the virtual image for the aerial work vehicle and displays the virtual image on the display screen.
[0186] In each of the above-described embodiments, the operation assistance device 1 of the crane 11 displays at least one of the first virtual image Im1 to the seventh virtual image Im7 using the transmission-type display device 2.
[0187] However, the operation assistance device 1 of the crane 11 may display at least one of the first virtual image Im1 to the seventh virtual image Im7 on an image captured by the camera. At least one of the first virtual image Im1 to the seventh virtual image Im7 can be operated on a display screen such as a touch panel.
[0188] In addition, in each of the above-described embodiments, the transmissive display device 2 acquires information Ifa about the user's position, which is necessary for displaying the virtual image Im on the transmissive display 4, by the GNSS receiver 5 for the display device.
[0189] However, the transmissive display device may calculate the position of the transmissive display device by recognizing a marker located at a specific position. The transmissive display device calculates the position of the transmissive display device by recognizing a marker located in the cabin of a crane, for example. The transmissive display device also calculates the position using an inertial measurement unit (IMU) based on the calculated position.
[0190] In each of the above-described embodiments, the transmissive display device 2 is a wearable terminal that is worn by the user. However, the transmissive display device may be any display device that allows the user to see a virtual image seen from the user's viewpoint by superimposing it on a display screen that allows the user to see the scenery seen from the user's viewpoint. The transmissive display device may be, for example, a head-up display that uses the windshield of the cabin as a display screen to display a virtual image seen from the viewpoint of a pilot seated in the cockpit.
[0191] The above-mentioned embodiment is merely a representative embodiment, and various modifications can be made without departing from the gist of the embodiment. Of course, the present invention can be embodied in various other forms, and the scope of the present invention is defined by the claims, and includes the equivalent meanings of the claims and all modifications within the scope of the claims.
[0192] The disclosures of the specification, drawings, and abstract contained in Japanese application No. 2021-075393, filed on April 27, 2021, are incorporated herein by reference in their entirety. [Industrial Applicability]
[0193] The present invention can be applied to various working machines having a boom that can be rotated, raised, lowered, and extended. [Explanation of symbols]
[0194] 1 Pilot assistance device 2 Transparent display device 3 Frame 4. See-through display 5 GNSS receiver for display device 6. Cameras for display devices 7 Display device control device 8. Communications Equipment 9 Image Processing Device 11. Crane 20a Main hook (reference member) 30 Crane control device Lp Current position Lt target position S1 reference plane S2 New Reference Plane Ifa Information about the user's location Ifb Crane location information Ifc Crane position information Ifd Reference Plane Information Ife Information on the target position of the main hook Iff Information on new datum Ifg Information on the direction and speed of movement of the main hook Ifh Main Hook Route Information Ifi Crane Specific Area Information Information on the operating status of Ifj cranes Im1 First virtual image C1 First reference coordinate If1 Information about the first virtual image of the crane
Claims
1. A steering assistance device configured to be wearable on a user's head and for assisting in the operation of a work machine having a boom that can be rotated, raised, lowered, and extended, comprising: With a transparent display A setting unit that sets a reference plane to a scene that the user sees through the display when using the piloting assistance device; A control unit that displays a virtual image for assisting in the operation of the work machine on the display so as to overlap with the reference plane displayed on the display, the reference plane is a ground surface in the scenery viewed by the user through the display during the use, or a plane arbitrarily selected by the user in the scenery viewed by the user through the display, the control unit generates the virtual image seen from the viewpoint of the user, and displays the generated virtual image on the display so as to be superimposed within the field of view of the user; The virtual image is an image showing a target position of a reference member moving with the boom; an image showing a trajectory of the reference member when the reference member moves from the target position in accordance with at least one operation of the boom selected from a rotation, a lifting, and a telescopic movement; and An image showing a line parallel to the vertical direction passing through a current position of a reference member moving together with the boom; Pilot assistance device.
2. The operation assistance device according to claim 1, wherein the virtual image is an image showing a current position of a reference member that moves together with the boom, and an image showing a trajectory that the reference member follows as it moves from the current position in conjunction with at least one movement of the boom selected from rotation, elevation, and extension.
3. The piloting assistance device according to claim 1 , wherein the virtual image includes an image showing a line that passes through a target position of a reference member that moves together with the boom and is parallel to the vertical direction.
4. The piloting assistance device according to claim 1 , wherein the virtual image includes an image of a vector indicating a moving speed and a moving direction of a reference member that moves together with the boom.
5. The steering assistance device according to claim 1 , wherein the control unit controls the display on the display based on a hand movement of a user of the steering assistance device captured by a camera.
6. The piloting assistance device according to claim 1 , further comprising a frame that supports the display and is wearable by a user of the piloting assistance device.
7. The piloting assistance device according to claim 6 , wherein the frame has a pair of temples supported on the user's ears.
8. A boom that can be rotated, raised, lowered, and extended; The steering assistance device according to claim 1, Work machine.
Citation Information
Patent Citations
Camera system and mechanical apparatus
JP2008312004A
Operation confirmation device
JP2014144850A
Article storage auxiliary device and system using ar / vr
JP2015124023A
Feeder arrangement support system and feeder arrangement support method
JP2018056252A
Remote operation terminal and work vehicle
JP2019156538A
