Steering assistance system for work vehicle and work vehicle having steering assistance system

Abstract

To provide an operation support system capable of operating a crane efficiently while suppressing an influence of information acquired from an instructor on the operation and a work vehicle including the operation support system.SOLUTION: An operation support system 1 includes a transmission type display device 2 superimposing a background Ls and a virtual image Im to be visually recognized. A position information acquisition device generates information Ifd about a selected position Sp on a display screen displaying a first virtual image Im1, which is arbitrary selected by a user. An image processing device 9 acquires the information Ifd about the selected position Sp and calculates the selected position Sp selected by the user. The image processing device 9 generates information Iff about a moving direction and a moving speed of a main hook 20a based on a positional relation between the first virtual image Im1 displayed on the display screen and the selected position Sp and transmits the information Iff about the moving direction and the moving speed of the main hook 20a to a control device through a communication device 8.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to a steering assistance system for a work vehicle and a work vehicle having a steering assistance system. [Background technology]

[0002] Conventionally, when transporting a load using a mobile crane, a work vehicle, the operator of the mobile crane operates multiple operating tools, such as a slewing tool, a hoisting tool, an extension tool, and a winch tool for lifting or lowering the load, all of which are used to operate the mobile crane's boom. The operator must also operate the mobile crane while simultaneously operating these multiple operating tools, while ensuring that the load does not come into contact with structures or other objects at the work site. Therefore, the operator obtains information about the load's location and features from riggers or other personnel to supplement information about areas and times that the operator cannot see. In other words, the operator operates the mobile crane based on visual information and information from an external instructor.

[0003] Mobile cranes also have a boom camera or the like to obtain information about areas that are difficult for the operator to see from the operator's seat. The boom camera can be used to obtain images of the cargo being transported and surrounding structures from above. For example, see Patent Document 1.

[0004] The crane described in Patent Document 1 uses a boom camera attached to the tip of the boom to capture images of the work site and images of the load suspended from the hook, and superimposes on the images at least one of a reference line centered on the load, a scale, a directed line indicating the direction of load movement, and a trajectory in which the load can turn. The crane operator operates the crane by comprehensively assessing his or her own visual information, information from an instructor, and image information from the boom camera. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2019-156538 Summary of the Invention [Problem to be solved by the invention]

[0006] Because crane operators operate the crane in accordance with the instructions and timing of the instructor, they cannot operate the crane at their own pace. Furthermore, if the operator cannot understand the instructor's intentions or if the instructor's instructions are not fully conveyed due to poor communication, it can be difficult to move the load in the desired direction or location. Meanwhile, images from a boom camera are limited in their capture direction and range. Therefore, the crane operator cannot obtain sufficient information from the images from the boom camera to move the load to the desired location.

[0007] An object of the present invention is to provide a working vehicle steering assistance system that allows the working vehicle to be steered by intuitive operation from any position, and a working vehicle equipped with the steering assistance system. [Means for solving the problem]

[0008] The present inventors have studied a steering assistance system for a work vehicle that can move a reference member in a desired direction at a desired speed by intuitive operation from any position, and the configuration of a work vehicle equipped with a steering assistance system. As a result of extensive research, the present inventors have come up with the following configuration.

[0009] The work vehicle steering assistance system of the present invention comprises a transparent display device that displays a virtual image seen from the user's perspective on a display screen that allows the background to be seen through, a position information acquisition device that generates information regarding a selected position that the user has arbitrarily selected on the display screen as seen from the user's perspective, an image processing device that generates information regarding the virtual image, acquires information regarding the selected position, and calculates the selected position, and a communication device that sends and receives information between the work vehicle control device, the image processing device, and the transparent display device.

[0010] The image processing device calculates the current position of a reference member moved by the work vehicle based on information about the position of the work vehicle and information about the attitude of the work vehicle acquired via the communication device, and generates information about a first virtual image for displaying on the display screen a first virtual image for moving the reference member as viewed from the user's point of view based on information about the user's position and the current position of the reference member acquired via the communication device.

[0011] The transmission-type display device displays the first virtual image on the display screen based on information about the first virtual image acquired via the communication device, and the position information acquisition device generates information about the selected position on the display screen on which the first virtual image is displayed.

[0012] The image processing device acquires information about the selected position, calculates the selected position, generates information about a moving direction of the reference member based on a positional relationship between the first virtual image displayed on the display screen and the selected position, and transmits the information about the moving direction of the reference member to the control device via the communication device.

[0013] The above-described piloting assistance system displays a guide display for moving a reference member on a background (scenery) viewed by a user of a transmissive display device as a first virtual image seen from the user's viewpoint. In other words, the piloting assistance system displays graphical information, textual information, etc., superimposed on a background within the user's field of view, allowing a user at any position to move the reference member in any direction. The piloting assistance system also generates information regarding the movement direction of the reference member based on the positional relationship between an arbitrary selected position within the field of view selected by the user of the transmissive display device and the first virtual image. The user of the transmissive display device can generate a control signal for the work vehicle by selecting the desired direction, position, etc., of the reference member using the first virtual image as a reference, using methods such as eye tracking and hand tracking. This allows the work vehicle to be piloted in any direction from any position through intuitive operations.

[0014] The work vehicle steering assistance system of the present invention preferably includes the following configuration: the image processing device generates information related to the moving speed of the reference member based on the positional relationship between the first virtual image displayed on the display screen and the selected position, and transmits the information related to the moving speed of the reference member to the control device via the communication device.

[0015] In the above-described configuration, the steering assistance system displays, superimposed on a background in the user's field of view, graphic information, text information, etc., for a user at any position to move a reference member in any direction, as well as graphic information, text information, etc., for moving the reference member at any speed. The steering assistance system also generates information about the movement direction of the reference member and information about the movement speed of the reference member based on the positional relationship between an arbitrary selected position in the field of view selected by the user of the transmissive display device and the first virtual image. This allows the work vehicle to be steered in any direction and at any speed from any position through intuitive operation.

[0016] The work vehicle steering assistance system of the present invention preferably includes the following configuration: when the selected position overlaps a part of the first virtual image as viewed from the user's viewpoint, the image processing device generates information regarding the moving direction and moving speed of the reference member based on the position of the first virtual image where the selected position overlaps.

[0017] In the above configuration, the piloting assistance system generates information about the movement direction and movement speed of the reference member based on a portion of the first virtual image selected by a user of the transmission-type display device. The user of the transmission-type display device can generate a control signal for the work vehicle by selecting a portion of the first virtual image indicating the desired direction, position, etc. of the reference member by eye tracking, hand tracking, or other methods. This allows the work vehicle to be piloted intuitively from any position.

[0018] The work vehicle steering assistance system preferably includes the following configuration: when the selected position overlaps a part of the first virtual image as viewed from the user's viewpoint, the image processing device generates information regarding the moving direction and moving speed of the reference member based on the position of the first virtual image where the selected position overlaps.

[0019] In the above configuration, the operation assistance system generates information regarding the movement direction and movement speed of the reference member based on the distance and direction from the reference member in a portion of the first virtual image selected by a user of the transmissive display device. The user of the transmissive display device can generate a control signal for the work vehicle by specifying the distance and direction from the reference member using methods such as eye tracking and hand tracking. This allows the work vehicle to be operated intuitively from any position.

[0020] Furthermore, a work vehicle steering assistance system according to one embodiment of the present invention preferably includes the following configuration. The work vehicle steering assistance system has an input device for inputting information about a virtual image. The image processing device acquires information about the target position of the reference member input from the input device, and generates information about a second virtual image for displaying the target position of the reference member as seen from the user's viewpoint on the display screen as a second virtual image. The transmission-type display device acquires information about the second virtual image via the communication device, and displays the second virtual image as seen from the user's viewpoint on the display screen.

[0021] In the above-described configuration, the steering assistance system displays information for understanding the positional relationship between the current position of the reference member and the target position, and the positional relationship between the target position of the reference member and features, superimposed within the user's field of view. The user of the transmissive display device can generate a control signal for the work vehicle by using methods such as eye tracking and hand tracking to determine the direction and position to which the reference member should be moved, while referring to the distance and direction from the current position of the reference member to the target position. This allows the work vehicle to be steered intuitively from any position.

[0022] Furthermore, it is preferable that the work vehicle steering assistance system according to one embodiment of the present invention includes the following configuration: the image processing device generates information related to a third virtual image for displaying a perpendicular line passing through the reference member as seen from the user's viewpoint on the display screen as a third virtual image, and the transmission-type display device acquires information related to the third virtual image via the communication device and displays the third virtual image as seen from the user's viewpoint on the display screen.

[0023] In the above configuration, the steering assistance system uses a transmission-type display device to display a perpendicular line indicating the current position of the reference member as a third virtual image on the background viewed by the user. The perpendicular line indicating the current position of the reference member displayed in the user's field of view can provide the user with information necessary to move the reference member of the work vehicle. This allows the user to operate the work vehicle from any position through intuitive operations.

[0024] From another viewpoint, the work vehicle steering assistance system according to the present invention preferably includes the following configuration: the image processing device generates information related to a fourth virtual image for displaying a vector diagram representing the movement direction and movement speed of the reference member as seen from the user's viewpoint on the display screen as a fourth virtual image, and the transmission-type display device acquires information related to the fourth virtual image via the communication device and displays the fourth virtual image as seen from the user's viewpoint on the display screen.

[0025] In the above configuration, the steering assistance system uses the transmissive display device to display a vector diagram indicating the movement direction of the reference member as a fourth virtual image viewed from the user's viewpoint on the background viewed by the user. In other words, the steering assistance system displays the direction in which the reference member will move based on the position on the display screen selected by the user. The user can supplement the information necessary to move the reference member using the vector diagram of the reference member displayed in their field of view. This allows the user to operate the work vehicle intuitively from any position.

[0026] From another perspective, it is preferable that the work vehicle steering assistance system according to the present invention includes the following configuration: the position information acquisition device generates information regarding the selected position from the position of the user's hand or the position of the user's gaze on the display screen as seen from the user's viewpoint; and the image processing device is capable of performing hand tracking to calculate the selected position selected by the user based on the position of the user's hand included in the acquired information regarding the selected position, or eye tracking to calculate the selected position selected by the user based on the position of the user's gaze included in the acquired information regarding the selected position.

[0027] In the above configuration, the driving assistance system moves the reference member by hand tracking or eye tracking based on the user's body movements on the display screen on which the first virtual image is displayed, thereby enabling the work vehicle to be driven by an intuitive operation from any position.

[0028] A work vehicle according to one embodiment of the present invention includes any one of the above-described operator assistance systems.

[0029] A work vehicle equipped with one of the above-described steering assistance systems is operated from any position by a user wearing the transmissive display device. The user inputs the direction and speed of movement of the reference member from the display screen while information for moving the reference member is displayed on the display screen superimposed on a background from any position. This allows the work vehicle to be operated intuitively from any position.

[0030] The terminology used in this specification is used for the purpose of defining particular embodiments only, and is not intended to limit the invention.

[0031] As used herein, "and / or" includes all combinations of one or more of the associated listed members.

[0032] In this specification, the use of "including," "comprising," or "having" and variations thereof identify the presence of stated features, steps, elements, components, and / or equivalents thereof, but may include one or more of the steps, operations, elements, components, and / or groups thereof.

[0033] As used herein, the terms "attached," "connected," "coupled," and / or their equivalents are used broadly to encompass both "direct and indirect" attachments, connections, and couplings. Furthermore, "connected" and "coupled" are not restricted to physical or mechanical connections or couplings, but can include direct or indirect connections or couplings.

[0034] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0035] Terms defined in commonly used dictionaries should be construed to have a meaning consistent with the relevant art and meaning in the context of this disclosure, and should not be construed in an idealized or overly formal sense unless expressly defined herein.

[0036] It is understood that in describing the present invention, several techniques and processes are disclosed, each of which has distinct advantages and can be used in conjunction with one or more, or in some cases all, of the other disclosed techniques.

[0037] Thus, for the sake of clarity, the description of the present invention refrains from unnecessarily repeating every possible combination of the individual steps, but the specification and claims should be read with the understanding that all such combinations are within the scope of the present invention.

[0038] In this specification, embodiments of a work vehicle steering assistance system and a work vehicle equipped with a steering assistance system according to the present invention will be described.

[0039] In the following description, numerous specific examples are set forth to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without these specific examples.

[0040] Accordingly, the following disclosure is to be considered as illustrative of the present invention and is not intended to limit the invention to the specific embodiments illustrated by the following drawings or description.

[0041] [Virtual image] In this specification, a virtual image is an image or video other than the background viewed by a user in a transmissive display device, and refers to a line, figure, image, video, 3D image, or 3D video displayed by the transmissive display device. In other words, a virtual image is an image that can be displayed naturally on the background viewed by a user by converting an image of an object that does not actually exist into an image of a shape seen from the user's viewpoint. The virtual image is displayed superimposed on the background viewed by a user in a transmissive display device.

[0042] [perspective] In this specification, the term "viewpoint" refers to a standpoint from which an object is viewed. The term "user's viewpoint" refers to the user's position relative to an object when the user is viewing the object.

[0043] [Information about virtual images] In this specification, the information about the virtual image means information necessary for displaying the virtual image on the display screen of the transmissive display device as an image on an arbitrary reference plane seen from the viewpoint of the user of the transmissive display device, including the coordinates of the virtual image, the display state, etc.

[0044] [Communication Device] In this specification, the communication device refers to a device that transmits and receives information between an image processing device and a display device control device, an image processing device and an input device, and an image processing device and a control device of a work vehicle. The communication device may be configured integrally with the display device control device and the image processing device. [Effects of the Invention]

[0045] According to one embodiment of the present invention, a steering assistance system and a work vehicle equipped with the steering assistance system can be steered by intuitive operation from any position. [Brief explanation of the drawings]

[0046] [Figure 1] FIG. 1 is a perspective view showing a transmission-type display device in a working vehicle steering assistance system according to one embodiment of the present invention worn by a user. [Figure 2] FIG. 2 is a perspective view of a transmission type display device in a working vehicle steering assistance system according to one embodiment of the present invention. [Figure 3] FIG. 3 shows a control block diagram of a working vehicle steering assistance system according to one embodiment of the present invention. [Figure 4] FIG. 4 shows an image captured by a display device camera in a work vehicle steering assistance system according to one embodiment of the present invention. [Figure 5] FIG. 5 shows a side view of a crane to which a working vehicle operation assistance system according to one embodiment of the present invention is connected. [Figure 6] FIG. 6 shows a control block diagram of a crane to which a working vehicle operation assistance system according to one embodiment of the present invention is connected. [Figure 7] FIG. 7 is a block diagram showing how the maneuvering assistance system for a work vehicle according to one embodiment of the present invention generates information necessary for displaying the first virtual image, the second virtual image, and the third virtual image. [Figure 8]FIG. 8 is a schematic diagram showing a state in which the steering assistance system for a work vehicle according to one embodiment of the present invention displays the first virtual image, the second virtual image, and the third virtual image on the transmission display device. [Figure 9] FIG. 9 is a schematic diagram showing a state in which the main hook moves when a user selects a portion of the first virtual image on the transmissive display device in a work vehicle steering assistance system according to one embodiment of the present invention. [Figure 10] FIG. 10 is a block diagram showing how the steering assist system for a work vehicle according to one embodiment of the present invention generates information necessary for displaying the fourth virtual image. [Figure 11] FIG. 11 is a schematic diagram showing a state in which the working vehicle steering assistance system according to one embodiment of the present invention is displaying the fourth virtual image. DETAILED DESCRIPTION OF THE INVENTION

[0047] A working vehicle operation assistance system according to one embodiment of the present invention will be described below with reference to FIGS. 1 to 4. FIG. 1 is a perspective view of a user wearing a transparent display device 2 in a working assistance system 1 for a crane 11 according to one embodiment of the present invention. FIG. 2 is a perspective view of the transparent display device 2 in the working assistance system 1 for a crane 11. FIG. 3 is a control block diagram of the working assistance system 1 for a crane 11. FIG. 4 shows an image P acquired by a display device camera 6 in the working assistance system 1 for a crane 11. In this embodiment, a rough terrain crane (hereinafter simply referred to as a "crane") will be described as the working vehicle. However, the working vehicle may be any working vehicle equipped with a boom that can be raised or lowered via a swivel, such as an all-terrain crane, a truck crane, or an aerial work platform. In the following description, the "user" refers to a person wearing the transparent display device 2.

[0048] 1 and 2, a driving assistance system 1 for a crane 11 (see FIG. 6), which is a work vehicle, is a system that uses mixed reality to assist in the driving of the crane 11. The driving assistance system 1 has a transmissive display device 2, a communication device 8, and an image processing device 9.

[0049] The transmissive display device 2 is a display device that allows the user to view a background Ls that overlaps with the transmissive display 4 of the transmissive display device 2 when viewed from the user's perspective. The transmissive display device 2 is a wearable terminal that is worn by the user on the body. In this embodiment, the transmissive display device 2 is described as a glasses-type wearable terminal. The transmissive display device 2 is worn by the operator of the crane 11, who is the user. The transmissive display device 2 has a frame 3, a transmissive display 4, a display device GNSS receiver 5, a display device camera 6, and a display device control device 7.

[0050] The frame 3 supports a transmissive display 4, a display device GNSS receiver 5, a display device camera 6, and a display device control device 7. The frame 3 is configured as a glasses-type frame. The frame 3 is worn on the user's head. The frame 3 is supported by the user's nose and both ears.

[0051] The transmissive display 4 is a display with a transmittance of approximately 50% or more. The transmissive display 4 is composed of a liquid crystal display, an organic EL display, an inorganic EL display, or the like that does not have a backlight, a polarizing filter, or a color filter. In other words, the transmissive display 4 does not have components such as a backlight or a filter that cover the back of the display screen that faces the user. Therefore, the transmissive display 4 allows the background that overlaps with the transmissive display 4 to be seen from the user's perspective. In addition, the transmissive display 4 can display a virtual image Im, which is an image generated by the function of the transmissive display 4, on the display screen.

[0052] The transmissive display 4 transmits light incident from the back surface in a portion where the virtual image Im is not displayed. The transmissive display 4 also blocks at least a portion of the light incident from the back surface in a portion where the virtual image Im is displayed. The transmissive display 4 then displays the virtual image Im in the blocked portion. The transmissive display 4 displays the virtual image Im as seen from the user's viewpoint. This allows the user to visually recognize the virtual image Im superimposed on a background Ls as seen from the user's viewpoint. The transmissive display 4 is provided on the frame 3. The transmissive display 4 is positioned to cover the visual fields of the right and left eyes of a user wearing the frame 3. The transmissive display 4 is configured to display a virtual image Im for the right eye and a virtual image Im for the left eye as parallax images on the display screen. This allows the transmissive display 4 to display a three-dimensional virtual image Im in almost the entire visual field of the user. It should be noted that the technology for displaying the virtual image Im seen from the user's viewpoint in the user's field of view in the transmission type display device 2 is well known.

[0053] The GNSS receiver 5 for the display device is a receiver that constitutes a global navigation satellite system. The GNSS receiver 5 for the display device receives ranging radio waves from satellites and detects the absolute coordinates of the GNSS receiver 5 for the display device, such as latitude, longitude, altitude, and direction. The GNSS receiver 5 for the display device is provided on the frame 3 worn by the user. The GNSS receiver 5 for the display device detects information Ifa about the position of the user (pilot). The information Ifa about the user's position includes the latitude, longitude, altitude, and direction of the transmissive display device 2. In other words, the information Ifa about the user's position includes information about the position of the transmissive display device 2 and the direction of the background Ls that the transmissive display 4 transmits. In this embodiment, the GNSS receiver 5 for the display device is provided on the frame 3, but it may be located in any position where it can detect the information Ifa about the user's position. For example, the GNSS receiver 5 for the display device may be worn on the body of the user wearing the transmissive display device 2.

[0054] The display device camera 6 captures an image in the direction of the line of sight of a user wearing the transmissive display device 2. The display device camera 6 is provided on the frame 3. The display device camera 6 is configured to capture an image P in a range approximately equal to the user's visual field from a position approximately equal to the user's viewpoint. In this embodiment, the display device camera 6 is provided on the frame 3, but it may be located in any position that allows it to capture an image of the user's visual field from the user's viewpoint.

[0055] The display device camera 6 is a position information acquisition device that generates information Ifd about a selection position Sp arbitrarily selected by a user on a display screen seen from the user's viewpoint. The display device camera 6 captures the movement of a part of the user's body, such as a hand, or a specific marker. That is, the display device camera 6 generates an image P that includes the information Ifd about the selection position Sp, such as the part of the user's body or the specific marker. In this way, the display device camera 6 generates an image P that includes information Ifd about the selection position Sp arbitrarily selected by the user on the transmissive display 4 on which the virtual image Im is displayed.

[0056] As shown in FIG. 3 , the display device control device 7 is a control device that displays a virtual image Im on the transmissive display 4. The display device control device 7 actually includes a CPU, a ROM, a RAM, a HDD, etc., connected via a bus. Alternatively, the display device control device 7 may be configured with a one-chip LSI or the like. The display device control device 7 is connected to the transmissive display 4, the display device GNSS receiver 5, and the display device camera 6. The display device control device 7 is connected to an image processing device 9 via a communication device 8. The display device control device 7 continuously transmits images P acquired from the display device camera 6 to the image processing device 9 via the communication device 8 at unit time intervals.

[0057] The display device control device 7 can acquire information Ifa about the user's position from the display device GNSS receiver 5. The display device control device 7 can acquire an image P from the display device camera 6. The display device control device 7 can also acquire information If about a virtual image Im from the image processing device 9. The display device control device 7 can generate a virtual image Im based on the acquired information If about the virtual image Im. The display device control device 7 can also display the virtual image Im on the transmissive display 4.

[0058] The transmissive display device 2, while worn on the user's head, can display a virtual image Im on the display unit of the transmissive display 4 (hereinafter simply referred to as "transmissive display 4"). The transmissive display device 2 can also acquire, by the display device GNSS receiver 5, information Ifa about the user's position, which is necessary for displaying the virtual image Im on the transmissive display 4. The transmissive display device 2 can generate a virtual image Im seen from the user's viewpoint on the transmissive display 4 through which the background Ls is transmitted, based on the information If about the virtual image Im and the information Ifa about the user's position. This allows the user to view an image in which the virtual image Im is superimposed on the background Ls through the transmissive display device 2.

[0059] The communication device 8 is a device that transmits and receives information between the transmissive display device 2, the image processing device 9, and the crane control device 30 of the crane 11, which is a work vehicle. The communication device 8 is provided in both the display device control device 7 of the transmissive display device 2 and the image processing device 9. The communication device 8 transmits and receives information between the crane control device 30 and the image processing device 9. The communication device 8 also transmits and receives information between the display device control device 7 and the image processing device 9.

[0060] The image processing device 9 is a control device that generates information If about the virtual image Im to be displayed on the transmissive display device 2. The image processing device 9 essentially includes a CPU, a ROM, a RAM, a HDD, etc., connected via a bus. Alternatively, the image processing device 9 may be configured with a one-chip LSI, etc. In this embodiment, the image processing device 9 is provided in the transmissive display device 2. The image processing device 9 stores various programs and data for controlling the operation of the transmissive display device 2, etc., and for processing image data. The image processing device 9 may be configured integrally with the display device control device 7 or the crane control device 30.

[0061] The image processing device 9 is connected to the display device control device 7 of the transmissive display device 2 via the communication device 8. The image processing device 9 acquires information Ifa about the user's position detected by the display device GNSS receiver 5 from the display device control device 7 via the communication device 8. The image processing device 9 also continuously acquires current images P captured by the display device camera 6 from the display device control device 7 via the communication device 8 at unit time intervals. The images P include information Ifd about the selected position Sp.

[0062] Furthermore, the image processing device 9 acquires information Ifb about the position of the crane 11 from the crane control device 30 via the communication device 8. The information Ifb about the position of the crane 11 includes the latitude, longitude, altitude, and vehicle direction of the location of the rotation center of the crane 11. Furthermore, the image processing device 9 transmits information Ife about the target position Lt of the main hook 20a, which is the reference member, to the crane control device 30 via the communication device 8.

[0063] Furthermore, the image processing device 9 acquires information Ifc about the attitude of the crane 11 from the crane control device 30 via the communication device 8. The information Ifc about the attitude of the crane 11 includes the swing angle of the crane 11, the boom hoisting angle, the boom length, the payout length of the main wire rope 24, the payout length of the sub-wire rope 26, and the operation direction and operation amount of each operating tool of the crane 11. In other words, the information Ifc about the attitude of the crane 11 includes, for example, information about the current position Lp of the main hook 20a, which is the reference member of the crane 11.

[0064] The image processing device 9 generates information If (If1, If2, If3, If4 described later) about the virtual image Im to be displayed on the transmissive display 4 of the transmissive display device 2 as a virtual image Im viewed from the user's viewpoint, based on the information Ifa about the user's position, the acquired information Ifb about the position of the crane 11, and the information Ifc about the attitude of the crane 11. The image processing device 9 transmits the generated information If about the virtual image Im to the display device control device 7.

[0065] The display device control device 7 of the transmissive display device 2 acquires information If about the virtual image Im via the communication device 8. The display device control device 7 displays the virtual image Im on the transmissive display 4 overlapping with the background Ls viewed by the user through the transmissive display 4.

[0066] 4, when the image processing device 9 detects a movement of a part of the user's body or a movement of a specific marker or the like from the acquired image P, the image processing device 9 recognizes the movement as information Ifd about the selected position Sp at which the user selected a part of the virtual image Im. The image processing device 9 calculates the selected position Sp from the information Ifd about the selected position Sp.

[0067] Furthermore, for example, when the user's hand overlaps with a part of the first virtual image Im1 included in the virtual image Im for moving the main hook 20a as viewed from the user's viewpoint, the image processing device 9 generates information Iff regarding the movement direction and movement speed of the main hook 20a, which is the reference member, based on the positional relationship between the first virtual image Im1 and the selected position Sp (see FIG. 3). The information Iff regarding the movement direction and movement speed of the main hook 20a includes movement speed, movement direction, and absolute position coordinates for moving the main hook 20a in any direction at any speed. The image processing device 9 transmits the generated information Iff regarding the movement direction and movement speed of the main hook 20a to the crane control device 30 of the crane 11, which is the work vehicle, via the communication device 8.

[0068] The operation assistance system 1 configured as described above acquires information Ifb about the position of the crane 11 and information Ifc about the attitude of the crane 11 from the crane control device 30 via the communication device 8, and acquires information Ifa about the position of the user from the display device control device 7. The operation assistance system 1 generates information If about the virtual image Im based on the information acquired by the image processing device 9. The operation assistance system 1 displays the virtual image Im on the transmission-type display device 2 based on the information If about the virtual image Im. Furthermore, when the image processing device 9 determines that the acquired image P includes information Ifd about the selected position Sp, it calculates the selected position Sp. The image processing device 9 generates information Iff about the movement direction and movement speed of the main hook 20a based on the positional relationship between the first virtual image Im1 and the selected position Sp.

[0069] A crane 11, which is a work vehicle according to a first embodiment of the present invention, will be described below with reference to Figures 5 and 6. Figure 5 shows the overall configuration of the crane 11 to which the operation assistance system 1 is connected. Figure 6 shows a control block diagram of the crane 11 to which the operation assistance system 1 is connected. Note that although a rough terrain crane will be described in this embodiment, any work vehicle equipped with a boom 19 that can be raised and lowered via a swivel base, such as an all-terrain crane, a truck crane, or an aerial work platform, may be used.

[0070] 5 and 6, the crane 11 is a mobile crane that can be moved to any location. The crane 11 includes a vehicle 12, a crane device 16, and a crane control device 30 (see FIG. 7).

[0071] The vehicle 12 is a traveling body that transports the crane device 16. The vehicle 12 has a plurality of wheels 13. The vehicle 12 travels using an engine 14 as a power source. The vehicle 12 has outriggers 15. The outriggers 15 have hydraulically extendable extension beams on both sides of the width of the vehicle 12, and hydraulic jack cylinders that can be extended in a direction perpendicular to the ground.

[0072] The crane apparatus 16 is a work device that lifts a load W using a wire rope. The crane apparatus 16 includes a swivel base 17, a boom 19, a main hook block 20, a sub-hook block 21, a hoisting hydraulic cylinder 22, a main winch 23, a main wire rope 24, a sub-winch 25, a sub-wire rope 26, a cabin 27, etc.

[0073] The swivel base 17 is a rotating device that allows the crane device 16 to rotate. The swivel base 17 is mounted on the frame of the vehicle 12 via an annular bearing. The swivel base 17 has a hydraulic swivel motor 17a, which serves as an actuator. The swivel base 17 is configured to be swivelable in one direction and the other direction by the swivel motor 17a.

[0074] The crane GNSS receiver 18 (see Figure 6) is a receiver that constitutes the Global Navigation Satellite System. The crane GNSS receiver 18 receives ranging radio waves from satellites and detects the absolute coordinates of the crane GNSS receiver 18, such as latitude, longitude, altitude, and direction. The crane GNSS receiver 18 is provided on the swivel base 17. The crane GNSS receiver 18 detects information Ifb about the position of the crane 11.

[0075] The boom 19 is a movable support that supports the main wire rope 24 and the sub wire rope 26. The boom 19 is made up of multiple boom members. The base end of the base boom member of the boom 19 is swingably mounted approximately in the center of the swivel base 17. The boom 19 has telescopic hydraulic cylinders 19a (see Figure 7) that are actuators that extend and retract each boom member, and a hoisting hydraulic cylinder 22 that raises and lowers the boom members. The boom 19 is extended and lowered in the axial direction by the telescopic hydraulic cylinders 19a. The boom 19 also has a jib 19c that is an extension member.

[0076] The main hook block 20 and the sub-hook block 21 are components that suspend the cargo W. The main hook block 20 has a plurality of hook sheaves around which the main wire rope 24 is wound, and a main hook 20a that suspends the cargo W. The sub-hook block 21 has a sub-hook 21a that suspends the cargo W.

[0077] The hoisting hydraulic cylinder 22 is an actuator that raises and lowers the boom 19 and maintains the posture of the boom 19. The end of the cylinder portion of the hoisting hydraulic cylinder 22 is connected to the swivel base 17 so that it can be raised and lowered freely. The end of the rod portion of the hoisting hydraulic cylinder 22 is connected to the base boom member of the boom 19 so that it can swing freely.

[0078] The main winch 23 is a device that reels in (winds up) and reels out (winds down) the main wire rope 24. The sub winch 25 is a device that reels in (winds up) and reels out (winds down) the sub wire rope 26.

[0079] The cabin 27 is a housing that covers the operator's seat. The cabin 27 is mounted on the swivel base 17. A not-shown operator's seat is provided in the cabin 27. The operator's seat is provided with a travel operating tool 28 for operating the vehicle 12 to travel, a crane operating tool 29 for the crane device 16, and the like (see FIG. 6).

[0080] As shown in FIG. 6, the crane controller 30 controls each actuator of the crane 11. The crane controller 30 is provided in the cabin 27. The crane controller 30 essentially comprises a CPU, ROM, RAM, HDD, etc. connected via a bus. Alternatively, the crane controller 30 may be configured with a one-chip LSI, etc. The crane controller 30 stores various programs and data for controlling the operation of each actuator, switching valve, etc., and for processing image data. The crane controller 30 has a crane communication device 30a. The crane communication device 30a transmits and receives information to and from the image processing device 9 of the operation assistance system 1. The crane controller 30 acquires information Iff regarding the movement direction and movement speed of the main hook 20a.

[0081] The crane control device 30 is connected to the crane GNSS receiver 18. The crane control device 30 acquires information Ifb about the position of the crane 11. The crane control device 30 is also connected to the traveling operation device 28 and the crane operation device 29. The crane control device 30 generates a control signal for the crane apparatus 16 based on an operation signal generated by operating the crane operation device 29. The crane control device 30 also generates control signals for each actuator by interlocking the swing hydraulic motor 17a, the extension hydraulic cylinder 19a, the hoisting hydraulic cylinder 22, the main winch 23, etc., so that the main hook 20a moves in the direction and at the speed included in the acquired information Iff about the movement direction and speed of the main hook 20a. The crane control device 30 transmits the generated control signals to each actuator.

[0082] The crane control device 30 is connected to the image processing device 9 of the operation assistance system 1 via the crane communication device 30a. The crane control device 30 transmits information Ifb about the position of the crane 11 and information Ifc about the attitude of the crane 11 to the image processing device 9. The crane control device 30 obtains information Ife about the target position Lt of the main hook 20a from the image processing device 9 via the crane communication device 30a.

[0083] The crane 11 configured in this manner can move the vehicle 12 to any position by operating the traveling operating device 28. Also, the crane 11 can transport the load W to any position by rotating, raising, lowering, and extending the boom 19 by operating the crane apparatus operating device 29. Also, the crane 11 can lift or lower the load W with the main winch 23 or the like by operating the crane apparatus operating device 29.

[0084] Next, using FIGS. 7 to 11, we will explain the operation assistance provided by the operation assistance system 1. FIG. 7 shows a block diagram of the operation assistance system 1 of the crane 11 generating information necessary for displaying the first virtual image Im1, the second virtual image Im2, and the third virtual image Im3. FIG. 8 shows a schematic diagram of the operation assistance system 1 of the crane 11 displaying the first virtual image Im1, the second virtual image Im2, and the third virtual image Im3 on the transmission-type display device 2. FIG. 9 shows a schematic diagram of the operation assistance system 1 of the crane 11 in a state in which the main hook 20a has moved when a user selects a portion of the first virtual image Im1 on the transmission-type display device 2. FIG. 10 shows a block diagram of the operation assistance system 1 of the crane 11 generating information necessary for displaying the fourth virtual image Im4. FIG. 11 shows a schematic diagram of the operation assistance system 1 of the crane 11 displaying the fourth virtual image Im4.

[0085] In this embodiment, the reference member is the main hook 20a of the crane 11. The reference member may also be the cargo W transported by the crane 11. The operator of the crane 11 is the user of the transmissive display device 2. The user operates the crane 11 with the transmissive display device 2 attached to their head. The user is also assumed to be located near the planned installation position of the cargo W suspended from the main hook 20a. In other words, the user is located outside the cabin 27 of the crane 11.

[0086] In this embodiment, the operation assistance system 1 selects an arbitrary position on the transmissive display 4 as seen from the user's viewpoint by hand tracking. The shape of the user's hand is registered in the image processing device 9. When the image processing device 9 determines that the image P captured by the display device camera 6 includes the user's hand, it executes hand tracking mode (see FIG. 4). In the hand tracking mode, the position on the transmissive display 4 of a specific part in the image of the hand, such as the tip of the index finger as seen from the user's viewpoint, is calculated as a selected position Sp. The image processing device 9 generates information Iff about the movement direction and movement speed of the main hook 20a from the positional relationship between the selected position Sp and the first virtual image Im1.

[0087] <Operation arrow display at current position Lp of main hook 20a> As shown in FIGS. 7 and 8, the piloting assistance system 1 displays an operation arrow of the main hook 20a, which is a reference member, as a first virtual image Im1 on the transmission-type display device 2. The first virtual image Im1 is a virtual image in which multiple arrows indicating the movement direction point in multiple directions based on the current position Lp of the main hook 20a. In this embodiment, the first virtual image Im1 is a multiple arrow positioned according to the orientation. The first virtual image Im1 includes, for example, an east arrow Ae pointing east, a west arrow Aw pointing west, a south arrow As pointing south, and a north arrow An pointing north. Note that arrows pointing between the respective directions, an upward arrow pointing vertically upward, a downward arrow pointing vertically downward, etc. may also be displayed.

[0088] 7, the image processing device 9 of the operation assistance system 1 acquires information Ifa about the position of the user and a current image P from the display device control device 7 via the communication device 8 (see FIG. 4). In addition, the image processing device 9 acquires information Ifb about the position of the crane 11 and information Ifc about the attitude of the crane 11 from the crane control device 30.

[0089] The image processing device 9 calculates first reference coordinates C1, which are a set of absolute coordinates for displaying an operation arrow of the main hook 20a based on the current position Lp of the main hook 20a, based on the acquired information Ifb about the position of the crane 11 and information Ifc about the attitude of the crane 11. The image processing device 9 generates information If1 about a first virtual image Im1, which is used to display a first virtual image Im1 viewed from the user's viewpoint based on the current position Lp of the main hook 20a, on the transmissive display 4, based on the acquired information Ifa about the user's position and information Ifb about the crane 11's position, and the calculated first reference coordinates C1. The image processing device 9 transmits the generated information If1 about the first virtual image Im1 to the display device control device 7. The display device control device 7 acquires the information If1 about the first virtual image Im1 via the communication device 8.

[0090] 8, the display device control device 7 displays the first virtual image Im1 on the transmissive display 4 based on information If1 related to the first virtual image Im1. The display device control device 7 displays an east arrow Ae, a west arrow Aw, a south arrow As, and a north arrow An as seen from the user's viewpoint at positions on the transmissive display 4 centered on the main hook 20a viewed by the user through the transmissive display 4.

[0091] The user perceives that the first virtual image Im1, which includes the east arrow Ae, west arrow Aw, south arrow As, and north arrow An, is displayed by the transmissive display device 2 so as to surround the main hook 20a, which is included in the actual background Ls, at the center. When the user points with his / her hand at the east arrow Ae, which indicates the direction in which he / she wants to move the main hook 20a, among the east arrow Ae, west arrow Aw, south arrow As, and north arrow An that he / she is viewing, the user's hand is located within the field of view of the display device camera 6. Therefore, the image P captured by the display device camera 6 includes the user's hand. The display device control device 7 continuously transmits the image P, which includes the user's hand, to the image processing device 9 via the communication device 8 at unit time intervals.

[0092] As shown in FIG. 7 , when the image processing device 9 determines that the acquired image P includes the user's hand, it acquires information Ifd about the selected position Sp arbitrarily selected by the user. The image processing device 9 starts a hand tracking mode and calculates the position of a part of the first virtual image Im1 that overlaps with the fingertips of the user's hand as the selected position Sp selected by the user. The image processing device 9 generates information Iff about the movement direction and speed of the main hook 20a based on the positional relationship between the selected position Sp and the eastward arrow Ae pointed by the user as viewed from the user's viewpoint. The image processing device 9 transmits the information Iff about the movement direction and speed of the main hook 20a, which is control information for moving the main hook 20a horizontally eastward at a predetermined speed, to the crane control device 30 of the crane 11 via the communication device 8. While the image processing device 9 continuously acquires information Ifd about the selected position Sp every unit time, it also generates and transmits the information Iff about the movement direction and speed of the main hook 20a to the crane control device 30 every unit time. Moreover, the image processing device 9 transmits information relating to the selected position Sp to the display device control device 7.

[0093] As shown in FIG. 8, the display device control device 7 highlights a portion of the first virtual image Im1 selected by the user based on information about the selection position Sp. While the crane control device 30 of the crane 11 receives information Iff about the movement direction and movement speed of the main hook 20a via the communication device 8, it activates each actuator with an actuation amount, actuation speed, actuation timing, and actuation time that causes the main hook 20a to move horizontally toward the east. As a result, the main hook 20a moves horizontally toward the east at a predetermined speed. The display device control device 7 changes the display position of the first virtual image Im1 in accordance with the movement of the main hook 20a.

[0094] 9, when the user moves his / her hand to a position that does not overlap with the first virtual image Im1 as viewed from the user's viewpoint, the image processing device 9 does not generate information Iff about the movement direction and movement speed of the main hook 20a based on the positional relationship between the selected position Sp and the east arrow Ae as viewed from the user's viewpoint. The crane 11 stops each actuator while not acquiring information Iff about the movement direction and movement speed of the main hook 20a.

[0095] Furthermore, if the user moves their hand outside the field of view of the display device camera 6 as viewed from the user's viewpoint, the image P captured by the display device camera 6 will not include the user's hand. If the image processing device 9 determines that the acquired image P does not include the user's hand, it will not calculate the selected position Sp selected by the user. Therefore, the image processing device 9 will not generate information Iff regarding the movement direction and movement speed of the main hook 20a from the positional relationship between the selected position Sp and the selected east arrow Ae as viewed from the user's viewpoint. The crane control device 30 stops each actuator while it is not acquiring information Iff regarding the movement direction and movement speed of the main hook 20a.

[0096] The above-described operation assistance system 1 displays a guide arrow for moving the main hook 20a as a first virtual image Im1 seen from the user's viewpoint on a background Ls viewed by the user. That is, the operation assistance system 1 displays graphic information, text information, etc., superimposed on the background Ls in the user's field of view, allowing a user at any position to move the main hook 20a in any direction at any speed. The operation assistance system 1 also generates information Iff regarding the movement direction and movement speed of the main hook 20a based on the positional relationship between an arbitrary selected position Sp in the field of view selected by the user of the transmissive display device 2 and the first virtual image Im1. The user can generate a control signal for the crane 11 by selecting the desired direction, position, etc., of the main hook 20a based on the first virtual image Im1 using a method such as hand tracking.

[0097] Furthermore, when the user selects a portion of the first virtual image Im1, the operation assistance system 1 generates information Iff about the movement direction and movement speed of the main hook 20a based on the selected portion of the first virtual image Im1. The user of the transmissive display device 2 can generate a control signal for the crane 11 by selecting a portion of the first virtual image Im1 that indicates the desired direction, position, etc. to which the main hook 20a should be moved by a method such as eye tracking or hand tracking. This allows the operation assistance system 1 to operate the crane 11 intuitively from any position.

[0098] <Display of target position Lt of main hook 20a> As shown in Figures 7 to 9, the operation assistance system 1 may display the target position Lt of the main hook 20a relative to the tip of the boom 19 as a second virtual image Im2 included in the virtual image Im on the transmission display device 2.

[0099] As shown in FIG. 7, the image processing device 9 acquires an image P (see FIG. 4) including a marker M from the user's viewpoint using the display device camera 6 (see FIG. 3), which is a position information acquisition device. The image processing device 9 determines whether the acquired image P contains information Ife about the target position Lt of the main hook 20a based on the user's hand movement or the type of marker M in the acquired image P. If the image processing device 9 determines that the acquired image P contains information Ife about the target position Lt of the main hook 20a, it calculates the target position Lt of the main hook 20a based on the shape, position, and moving direction of the operator's hand in the image P or the type, position, and direction of the marker M. Alternatively, the image processing device 9 acquires the information Ife about the target position Lt from an external source via the communication device 8. The means for acquiring the information Ife about the target position Lt is not limited to the marker M. The information Ife about the target position Lt can also be acquired by recognizing the surrounding environment using a known technology such as MR (Mixed Reality).

[0100] The image processing device 9 calculates a plurality of second reference coordinates C2 for displaying the target position Lt of the main hook 20a at the current height of the main hook 20a, based on the information Ifc about the attitude of the crane 11 and the target position Lt. The image processing device 9 generates information for displaying the main hook 20a at the target position Lt as seen from the user's viewpoint as a second virtual image Im2 on the transmissive display 4 at the current height of the main hook 20a, based on the acquired information Ifa about the user's position and the plurality of second reference coordinates C2. That is, the image processing device 9 generates information If2 about the second virtual image Im2 of the main hook 20a at the target position Lt as seen from the user's viewpoint. The image processing device 9 transmits the generated information If2 about the second virtual image Im2 to the display device control device 7. The display device control device 7 acquires the information If2 about the second virtual image Im2 via the communication device 8.

[0101] 8, the display device control device 7 displays the second virtual image Im2 of the main hook 20a at the target position Lt as viewed from the user's viewpoint based on information If2 about the second virtual image Im2 on the transmissive display 4. In this embodiment, the second virtual image Im2 is a graphic representing the main hook 20a.

[0102] The user perceives the main hook 20a displayed at the target position Lt as being superimposed on the actual background Ls through the transmissive display device 2. When the first virtual image Im1 is displayed at the current position Lp and the second virtual image Im2 is displayed at the target position Lt, the user can visually recognize on the ground surface from any location the positional relationship between the current position Lp of the main hook 20a and the target position Lt, and the positional relationship between the main hook 20a and the feature F at the target position Lt. Therefore, the user can obtain information necessary to move the main hook 20a to the target position Lt by using the main hook 20a displayed at the target position Lt within the user's field of view. This allows the user to operate the crane 11 from any position through intuitive operations using the operation assistance system 1.

[0103] <Display of perpendicular line at current position Lp of main hook 20a> As shown in FIGS. 7 to 9, the steering assistance system 1 may display, on the transmission-type display device 2, a perpendicular line passing through the current position Lp of the main hook 20a as a third virtual image Im3 included in the virtual image Im.

[0104] As shown in FIG. 7 , the image processing device 9 calculates multiple third reference coordinates C3 for displaying a perpendicular line passing through the current position Lp of the main hook 20a. The image processing device 9 generates information for displaying a third virtual image Im3 viewed from the user's viewpoint on the transmissive display 4 based on the acquired information Ifa about the user's position and the multiple third reference coordinates C3. That is, the image processing device 9 generates information about the perpendicular line passing through the current position Lp of the main hook 20a viewed from the user's viewpoint as information If3 about the third virtual image Im3 of the crane 11. The image processing device 9 transmits the generated information If3 about the third virtual image Im3 of the crane 11 to the display device control device 7. Note that the information If3 about the third virtual image Im3 of the crane 11 may include information about the perpendicular line passing through the target position Lt of the main hook 20a. The display device control device 7 acquires the information If3 about the third virtual image Im3 of the crane 11 via the communication device 8.

[0105] As shown in Figure 8, the display device control device 7 displays on the transmissive display 4 a perpendicular line passing through the current position Lp of the main hook 20a as seen from the user's perspective, which is the third virtual image Im3, based on information If3 regarding the third virtual image Im3 of the crane 11.

[0106] The user perceives the transmissive display device 2 as displaying a perpendicular line indicating the current position Lp of the main hook 20a superimposed on the actual background Ls. When the perpendicular line from the current position Lp of the main hook 20a, which is the third virtual image Im3, and the main hook 20a at the target position Lt, which is the second virtual image Im2, are both displayed, the user can move the main hook 20a toward the target position Lt while visually checking the positional relationship of the main hook 20a with the feature F at the current position Lp of the main hook 20a or the target position Lt. This allows the user to operate the crane 11 intuitively from any position.

[0107] <Display of the moving direction of the main hook 20a> 10 and 11, the operation assistance system 1 may display a fourth virtual image Im4, which is a vector diagram representing the moving direction of the main hook 20a, on the transmission-type display device 2. The fourth virtual image Im4 included in the virtual image Im is a sphere centered on the current position Lp of the main hook 20a. Contains .

[0108] As shown in FIG. 10 , the image processing device 9 acquires information Ifa about the user's position, the current image P, information Ifb about the position of the crane 11, and information Ifc about the attitude of the crane 11 via the communication device 8. Based on the acquired information Ifb about the position of the crane 11 and the information Ifc about the attitude of the crane 11, the image processing device 9 calculates fourth reference coordinates C4, which are a set of absolute coordinates for displaying a sphere centered on the current position Lp of the main hook 20a. Based on the acquired information Ifa about the user's position and information Ifb about the position of the crane 11 and the calculated fourth reference coordinates C4, the image processing device 9 generates information If4 about a fourth virtual image Im4 for displaying the fourth virtual image Im4 viewed from the user's viewpoint with the current position Lp of the main hook 20a as a reference on the transmissive display 4. The image processing device 9 transmits the generated information If4 about the fourth virtual image Im4 to the display device control device 7. The display device control device 7 acquires the information If4 about the fourth virtual image Im4 via the communication device 8.

[0109] 11 , the display device control device 7 displays a fourth virtual image Im4 on the transmissive display 4 based on information If4 related to the fourth virtual image Im4. The display device control device 7 displays a sphere centered on the main hook 20a, which is visually recognized by the user through the transmissive display 4, as the fourth virtual image Im4.

[0110] The user perceives the fourth virtual image Im4 as a sphere displayed by the transmissive display device 2, surrounding the main hook 20a included in the actual background Ls. The user points with his or her hand on the sphere he or she is viewing in the direction in which he or she wants to move the main hook 20a. The display device control device 7 continuously transmits an image P including the user's hand pointing at the sphere to the image processing device 9 via the communication device 8 at unit time intervals.

[0111] As shown in FIG. 10 , the image processing device 9 calculates, from information Ifd about the selected position Sp included in the image P, a portion of the fourth virtual image Im4 that overlaps with the fingertips of the user's hand as the selected position Sp selected by the user. The image processing device 9 generates information Iff about the movement direction and speed of the main hook 20a based on the positional relationship between the current position Lp of the main hook 20a, the selected position Sp, and the position on the sphere pointed to by the user as viewed from the user's viewpoint. The image processing device 9 transmits the information Iff about the movement direction and speed of the main hook 20a, which is control information for moving the main hook 20a toward the selected position Sp on the sphere at a predetermined speed, to the crane control device 30 of the crane 11 via the communication device 8. While the image processing device 9 continuously acquires information Ifd about the selected position Sp every unit time, the image processing device 9 generates and transmits the information Iff about the movement direction and speed of the main hook 20a to the crane control device 30 every unit time. The image processing device 9 also transmits the information about the selected position Sp to the display device control device 7.

[0112] As shown in Fig. 11, the display device control device 7 displays the movement direction of the main hook 20a selected by the user using a vector diagram V based on information relating to the current position Lp and selected position Sp of the main hook 20a. The crane control device 30 of the crane 11 operates each actuator with an actuation amount, actuation speed, actuation timing, and actuation time so that the main hook 20a moves from the current position Lp in the direction of the selected position Sp while acquiring information Iff relating to the movement direction and movement speed of the main hook 20a via the communication device 8. The display device control device 7 controls the movement direction of the main hook 20a based on information Iff relating to the movement direction and movement speed of the main hook 20a. sphere and change the display position of vector diagram V.

[0113] The user perceives a sphere centered on the current position Lp of the main hook 20a in the actual background Ls and a vector diagram V pointing from the sphere toward the selected position Sp as being superimposed on each other on the transmissive display device 2. The user can set the movement direction of the main hook 20a by selecting any position on the sphere.

[0114] As described above, by using the transmissive display device 2, the crane 11 having the operation assistance system 1 can be operated using a virtual image that supports the operation of the crane 11 and is displayed superimposed on a reference member such as the main hook 20a that suspends the load W. This allows the user of the transmissive display device 2 to move the reference member in any direction by intuitive operation using hand tracking, eye tracking, or the like, from any position, not limited to the operator's seat, but also near the load W, or from a high place where the crane 11 can be viewed from above, without using the crane apparatus operating tool 29.

[0115] <Other embodiments> In each of the above-described embodiments, the piloting assistance system 1 displays a guide arrow for moving the main hook 20a in the background Ls viewed by the user as the first virtual image Im1 seen from the user's viewpoint. However, the first virtual image may be a graphic, character, or the like that guides the direction and speed of movement of the standard member. Furthermore, input to the piloting assistance system 1 is not limited to eye tracking or hand tracking, and may be voice, gesture, mechanical input, or a combination thereof, corresponding to the first virtual image. For example, the user may select a portion of the first virtual image by eye tracking and then start moving the main hook 20a by voice.

[0116] In each of the above-described embodiments, when the user points with his or her hand at the east arrow Ae among the operational arrows that the user is viewing, the operation assistance system 1 transmits information Iff about the movement direction and speed of the main hook 20a to the crane control device 30 of the crane 11 via the communication device 8. While the crane control device 30 acquires the information Iff about the movement direction and speed of the main hook 20a, the main hook 20a moves horizontally toward the east at a predetermined speed (see FIG. 7 ). However, the operation assistance system may also calculate the movement speed and movement direction of the reference member based on the position of the selected position in the first virtual image. The image processing device of the operation assistance system generates information about the movement direction and movement speed of the reference member, in which the movement speed of the reference member is set to be larger the farther the position of the first virtual image Im1 selected by the user is from the reference member. The image processing device also generates information about the movement direction and movement speed based on the direction of the selected position relative to the reference member.

[0117] In each of the above-described embodiments, the image processing device 9 calculates the portion of the image P where the user's hand overlaps with a part of the virtual image Im as the selected position Sp. In this case, the image processing device may execute an eye tracking mode in which the selected position is selected based on the position of the user's line of sight. This allows the piloting assistance system 1 to select the movement direction of the reference member even when the user cannot use their hands.

[0118] In each of the above-described embodiments, the image processing device 9 calculates, as the selected position Sp, a portion of the first virtual image Im1 where the user's hand overlaps with the first virtual image Im1 as viewed from the user's viewpoint in the image P captured by the display device camera 6. However, the image processing device may determine that the first virtual image has been selected even if there is no overlapping portion of the first virtual image Im1, as long as the user's hand is within a predetermined range from the user's viewpoint. This allows the operation assistance system to allow the user to easily select the first virtual image Im1 on the transmissive display device.

[0119] In each of the above-described embodiments, the information Iff regarding the movement direction and movement speed of the main hook 20a generated by the image processing device 9 includes the actuation amount, actuation speed, actuation timing, actuation time, etc. of each actuator in the crane 11 for moving the main hook 20a in any direction at any speed. However, the movement range of the boom, etc., of a crane may be restricted to prevent contact with surrounding structures, etc. Therefore, when the movement range of each actuator is limited in the crane control device, the image processing device may generate information regarding a virtual image showing the movement range of the reference member as seen from the user's perspective on a transmissive display device. The image processing device acquires information regarding the limit on the movement range of the crane from the crane control device via a communication device. When the reference member is located within the movement range, the image processing device generates information regarding the movement direction and movement speed of the reference member.

[0120] In each of the above-described embodiments, the operation assistance system 1 for the crane 11 displays any one of the first virtual image Im1, the second virtual image Im2, the third virtual image Im3, and the fourth virtual image Im4 on the transmission-type display device 2. However, the operation assistance system 1 for the work vehicle may display at least one of the first to fourth virtual image images selected arbitrarily by the user of the transmission-type display device. Furthermore, the operation assistance system for the work vehicle is not limited to the virtual images displayed in the above-described embodiments, and may display any virtual image that assists in the operation of the work vehicle.

[0121] Furthermore, in the above-described embodiment, the transmissive display device 2 of the operation assistance system 1 may be worn by an operator operating the crane 11 inside the cabin 27 of the crane 11. The operator of the crane 11 inside the cabin 27 may move the main hook 20a using the operation assistance system 1 without using the crane apparatus operating tool 29.

[0122] Furthermore, in each of the above-described embodiments, the transparent display device 2 of the operation assistance system 1 for the crane 11 is worn by a worker other than the operator of the crane 11. However, the transparent display device may be worn by at least one of the operator of the crane 11 and the worker other than the operator of the crane 11. For example, if the operator and slinger of the crane 11 each wear a transparent display device 2, the operation assistance system 1 for the crane 11 causes each of the transparent display devices to display the same or different types of virtual images based on the same information about the attitude of the crane 11. As a result, the operator and slinger of the crane 11 view virtual images based on the same information about the attitude of the crane 11 within the field of view of their respective positions, allowing them to more accurately share the position of the main hook 20a relative to the load W, etc.

[0123] In the above-described embodiment, the operation assistance system 1 is configured as an operation assistance system 1 for a crane 11. However, the operation assistance system 1 may also be configured as an operation assistance system 1 for an aerial work vehicle, which is a work vehicle. The operation assistance system 1 for the aerial work vehicle acquires, via a communication device, information on the user's position, information on the position of the aerial work vehicle, information on the attitude of the aerial work vehicle, and information on a reference plane including the position and attitude of the reference plane that serves as a reference for displaying a virtual image. The image processing device of the operation assistance system for the aerial work vehicle generates information on a virtual image for displaying, on the transmission-type display device, a virtual image that is an operation arrow seen from the user's viewpoint at the current position of a bucket, which is a reference member moved by the aerial work vehicle. The transmission-type display device acquires information on the virtual image for the aerial work vehicle and displays the virtual image on a display screen.

[0124] In each of the above-described embodiments, the operation assistance system 1 for the crane 11 displays at least one of the first virtual image Im1 to the fourth virtual image Im4 using the transmission-type display device 2. However, the operation assistance system for the crane 11 may display at least one of the first virtual image to the fourth virtual image on an image captured by a camera capturing an image of the reference member. At least one of the first virtual image to the fourth virtual image can be operated on a display screen such as a touch panel.

[0125] In each of the above-described embodiments, the transmissive display device 2 acquires, via the display device GNSS receiver 5, information Ifa about the user's position, which is necessary for displaying the virtual image Im on the transmissive display 4. However, the transmissive display device may calculate the position of the transmissive display device by recognizing a marker at a specific location or the crane itself. For example, the transmissive display device calculates the position of the transmissive display device by recognizing a marker located inside the cabin 27 of the crane 11 or by recognizing the crane itself using a marker or MR. Furthermore, by utilizing self-position estimation technology, the amount of movement and orientation of the transmissive display device can be constantly calculated and updated, thereby constantly grasping the positional relationship between the crane and the transmissive display device. Furthermore, the transmissive display device calculates its position using an inertial measurement unit (IMU) based on the calculated position.

[0126] In each of the above-described embodiments, the transmissive display device 2 is a wearable terminal that is worn by a user. However, the transmissive display device may be any display device that superimposes a virtual image seen from the user's viewpoint on a display screen that allows the background to be seen from the user's viewpoint. The transmissive display device may be, for example, a head-up display that uses the windshield of the cabin as a display screen to display a virtual image seen from the pilot's viewpoint in the cockpit.

[0127] The above-described embodiment merely shows a typical form, and various modifications can be made without departing from the gist of one embodiment. Of course, the present invention can be embodied in various other forms, and the scope of the present invention is defined by the claims, and further includes the meaning of equivalents set forth in the claims, as well as all modifications within the scope of the claims. [Explanation of symbols]

[0128] 1. Pilot assistance system 2 Transparent display device 3 frames 4. Transmissive displays 5 GNSS receiver for display device 6. Camera for display devices 7 Display device control device 8. Communications Equipment 9 Image Processing Device 11 Crane 21a Main hook (reference member) 30 Crane control device Lp Current position Lt target position Sp selection position Ifa User location information Ifb Crane Location Information Ifc Crane attitude information Ifd Information about the selected position Ife Information on the target position of the main hook Iff Information on the direction and speed of movement of the main hook Im1 First virtual image Im2 Second virtual image Im3 Third virtual image Im4 4th virtual image C1 First reference coordinate C2 Second reference image C3 Third reference coordinate C4 Fourth reference image If1 Information about the first virtual image of the crane If2 Information about the second virtual image of the crane If3 Information about the third virtual image of the crane If4 Information about the fourth virtual image of the crane

Claims

1. a transmission type display device that displays a virtual image seen from a user's viewpoint on a display screen that allows the background to be seen; a position information acquisition device that generates information about a selected position arbitrarily selected by the user as viewed from the user's viewpoint on the display screen; an image processing device that generates information about the virtual image, acquires information about the selected position, and calculates the selected position; A work vehicle steering assistance system comprising: a control device for a work vehicle; the image processing device; and a communication device that transmits and receives information between the control device for the work vehicle, the image processing device, and the transmission type display device, The image processing device includes: calculating a current position of a reference member moved by the work vehicle based on information relating to the position of the work vehicle and information relating to the attitude of the work vehicle acquired via the communication device; generating information about a first virtual image for displaying on the display screen a first virtual image for moving the reference member as viewed from the user's viewpoint, based on information about the user's position acquired via the communication device and a current position of the reference member; The transmissive display device comprises: displaying the first virtual image on the display screen based on information about the first virtual image acquired via the communication device; The location information acquisition device generating information about the selected position on the display screen on which the first virtual image is displayed; The image processing device includes: obtaining information about the selected position and calculating the selected position; generating information regarding a moving direction of the reference member based on a positional relationship between the first virtual image displayed on the display screen and the selected position, and transmitting the information regarding the moving direction of the reference member to the control device via the communication device; A steering assistance system for work vehicles.

2. 2. The work vehicle steering assistance system according to claim 1, The image processing device includes: generating information about a moving speed of the reference member based on a positional relationship between the first virtual image displayed on the display screen and the selected position, and transmitting the information about the moving speed of the reference member to the control device via the communication device; A steering assistance system for work vehicles.

3. 3. The work vehicle steering assistance system according to claim 2, The image processing device includes: When the selection position overlaps a part of the first virtual image as viewed from the user's viewpoint, information about a moving direction and a moving speed of the reference member is generated based on the position of the first virtual image where the selection position overlaps. A steering assistance system for work vehicles.

4. 4. The work vehicle steering assistance system according to claim 2, The image processing device includes: generating information about a moving direction and a moving speed of the reference member based on a position of the selected position in the first virtual image as seen from the user's viewpoint; A steering assistance system for work vehicles.

5. The working vehicle steering assistance system according to any one of claims 1 to 4, The image processing device includes: acquiring information about the target position of the reference member by the position information acquisition device, and generating information about a second virtual image for displaying the target position of the reference member as a second virtual image on the display screen as seen from the user's viewpoint; The transmissive display device comprises: acquiring information about the second virtual image via the communication device, and displaying the second virtual image seen from the user's viewpoint on the display screen; A steering assistance system for work vehicles.

6. The working vehicle steering assistance system according to any one of claims 1 to 5, The image processing device includes: generating information about a third virtual image for displaying a perpendicular line passing through the reference member as seen from the user's viewpoint on the display screen as a third virtual image; The transmissive display device comprises: acquiring information about the third virtual image via the communication device, and displaying the third virtual image seen from the user's viewpoint on the display screen; A steering assistance system for work vehicles.

7. The working vehicle steering assistance system according to any one of claims 1 to 6, The image processing device includes: generating information about a fourth virtual image for displaying, on the display screen, a vector diagram representing a direction from the reference member to the selected position as seen from the user's viewpoint, as a fourth virtual image; The transmissive display device comprises: acquiring information about the fourth virtual image via the communication device, and displaying the fourth virtual image seen from the user's viewpoint on the display screen; A steering assistance system for work vehicles.

8. The working vehicle steering assistance system according to any one of claims 1 to 7, The location information acquisition device generating information about the selected position from the position of the user's hand or the position of the user's line of sight on the display screen as seen from the user's viewpoint; The image processing device includes: It is possible to perform hand tracking, which calculates the selection position selected by the user based on the position of the user's hand included in the acquired information about the selection position, or eye tracking, which calculates the selection position selected by the user based on the position of the user's gaze included in the acquired information about the selection position. A steering assistance system for work vehicles.

9. A work vehicle comprising the work vehicle steering assistance system according to any one of claims 1 to 6.

Images