Pilot assistance system and work vehicle

Abstract

A maneuvering assistance system according to the present invention assists maneuvering of a work vehicle having a boom, said maneuvering assistance system comprising: a display that is transparent and that is capable of displaying an image; a generation unit that generates a virtual image indicating a boom and / or a target member which includes a member moved together with the boom; and a control unit that displays the virtual image on the display such that the virtual image is superimposed on a landscape visible through the display in a mode in which a user of the display can recognize the position of the target member.

Description

[Technical field]

[0001] The present invention relates to a steering assist system for a work vehicle and a work vehicle having a steering assist system. [Background technology]

[0002] Conventionally, when transporting a load using a mobile crane, which is a work vehicle, the operator of the mobile crane operates multiple operating tools, such as a slewing operating tool, a hoisting operating tool, and a telescopic operating tool for operating the boom of the mobile crane, and a winch operating tool for lifting or lowering the load. In addition, while operating the multiple operating tools simultaneously, the operator must operate the mobile crane so that the load does not come into contact with structures and the like at the work site. Therefore, the operator obtains information on the position of the load and information on features from the rigger and the like to supplement information on the range and time that the operator cannot see. In other words, the operator operates the crane based on information from the operator's eyes and information from an external instructor.

[0003] In addition, the mobile crane has a boom camera or the like to obtain information on areas that are difficult for the operator to recognize from the operator's seat. The mobile crane can obtain images of the cargo being transported and surrounding structures from above using the boom camera. For example, see Patent Document 1.

[0004] The crane described in Patent Document 1 uses a boom camera attached to the tip of the boom to capture images of the work site and images of the cargo suspended from the hook, and also displays on the images at least one of a reference line centered on the cargo, a scale, a directed line indicating the direction of cargo movement, and a trajectory in which the cargo can turn, etc. The crane operator operates the crane by comprehensively assessing his / her own visual information, information from an instructor, and image information from the boom camera. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2019-156538 A Summary of the Invention [Problem to be solved by the invention]

[0006] When the crane operator cannot see the tip of the boom and the hook, etc., he or she operates the crane according to the instructions and timing of the instructor. At this time, the crane operator infers the positional relationship between the surrounding structures and the boom and hook, etc. from the instructions. Therefore, if the instructions from the instructor are not appropriate, the crane operator cannot operate the crane efficiently.

[0007] An object of the present invention is to provide an operation assistance system and a work vehicle that can efficiently operate a crane by suppressing the effect on operation of information obtained from an instructor. [Means for solving the problem]

[0008] The present inventors have studied configurations of a driving assistance system and a work vehicle that can suppress the influence of information obtained from an instructor on the driving and efficiently drive the work vehicle. As a result of extensive studies, the present inventors have come up with the following configuration.

[0009] One aspect of the piloting assistance system according to the present invention is to A steering assistance system for assisting in steering a work vehicle having a boom, comprising: A display that is transparent and capable of displaying an image; B Move with the team Hook A generating unit that generates a virtual image representing the The user of the display Hook In a manner that allows the position of The user Through the display Visually confirm and a control unit that displays a virtual image on the display so as to overlap with the scenery. When implementing a steering assistance system as described above, the control unit preferably calculates the current position of the hook based on an image taken by a camera that photographs the hook from the tip of the boom, and displays a virtual image showing the shape of the hook as seen from the user's perspective at a position on the display corresponding to the current position of the hook.

[0010] One aspect of the work vehicle according to the present invention is equipped with the above-mentioned steering assistance system.

[0011] The terminology used in this specification is used for the purpose of defining particular embodiments only, and is not intended to limit the invention.

[0012] As used herein, "and / or" includes all combinations of one or more of the associated listed members.

[0013] As used herein, the use of "including," "comprising," or "having" and variations thereof specify the presence of stated features, steps, elements, components, and / or equivalents thereof, but may include one or more of the steps, operations, elements, components, and / or groups thereof.

[0014] As used herein, the terms "attached," "connected," "coupled," and / or the like are used broadly to encompass both "direct and indirect" attachments, connections, and couplings. Furthermore, "connected" and "coupled" are not restricted to physical or mechanical connections or couplings, but can include direct or indirect connections or couplings.

[0015] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0016] Terms defined in commonly used dictionaries should be interpreted to have a meaning consistent with the meaning in the context of the relevant art and this disclosure, and should not be interpreted in an idealized or overly formal sense unless expressly defined herein.

[0017] It will be understood that in describing the present invention, a number of techniques and processes are disclosed, each of which has distinct advantages and can be used in conjunction with one or more, or in some cases all, of the other disclosed techniques.

[0018] Thus, for the sake of clarity, the description of the present invention refrains from unnecessarily repeating every possible combination of the individual steps, but the specification and claims should be read with the understanding that all such combinations are within the scope of the present invention.

[0019] In this specification, embodiments of a steering assistance system for a work vehicle and a work vehicle equipped with a steering assistance system according to the present invention will be described.

[0020] In the following description, numerous specific examples are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced without these specific examples.

[0021] The following disclosure is therefore to be considered as illustrative of the present invention and is not intended to limit the invention to the specific embodiments illustrated in the following drawings or description.

[0022] [Virtual Image] In this specification, a virtual image means an image or video other than the background (scenery) of the transmissive display device viewed by a user in a transmissive display device, and refers to a line, figure, image, video, three-dimensional image, and / or three-dimensional video displayed by the transmissive display device. In other words, a virtual image is an image that can be displayed naturally on the background of the transmissive display device viewed by a user by converting an image of an object that does not actually exist into an image of a shape seen from the user's viewpoint. The virtual image is displayed on the transmissive display device by superimposing it on the background of the transmissive display device viewed by the user. The virtual image includes a first virtual image, a second virtual image, a third virtual image, a fourth virtual image, and a fifth virtual image.

[0023] [perspective] In this specification, the term "viewpoint" refers to a standpoint from which an object is viewed. The term "user's viewpoint" refers to the position of the user relative to an object when the user is viewing the object.

[0024] [Information about virtual images] In this specification, information about a virtual image means information necessary for displaying the virtual image on the display screen of the transmissive display device as an image seen from the viewpoint of a user of the transmissive display device. The information about the virtual image includes the coordinates and / or the display state of the virtual image.

[0025] [Trajectory] In this specification, a track refers to a figure that represents, with a continuous line, a set of multiple coordinates through which a reference member of a work vehicle passes by operating each of the operating tools of the work vehicle. The track also includes a figure that represents, with a continuous line, a set of coordinates through which a reference member of a work vehicle passes by operating each of the operating tools, which is converted into coordinates on an arbitrary reference plane.

[0026] [Communication Device] In this specification, the communication device means a device that transmits and receives information between the image processing device and the display device control device, between the image processing device and the input device, and between the image processing device and the control device of the work vehicle. The communication device may be configured integrally with the display device control device and the image processing device.

[0027] [Reference part] In this specification, the reference member is a member that moves with the movement of the boom of the work vehicle and is photographed by a camera attached to the boom. In other words, the reference member is a member whose current position is recognized by the camera. The reference member is, for example, a main hook, main hook block, sub hook, sub hook block, main wire rope, and a load suspended by a hook (main hook or sub hook) suspended from the boom. Effect of the Invention

[0028] According to the present invention, it is possible to provide an operation assistance system and a work vehicle that can efficiently operate a crane by suppressing the influence on operation of information obtained from an instructor. [Brief description of the drawings]

[0029] [Figure 1] FIG. 1 is a perspective view showing a transmission type display device in a working vehicle steering assistance system according to a first embodiment of the present invention worn by a user. [Diagram 2] FIG. 2 shows a perspective view of the transmissive display device in the work vehicle driving assistance system according to the first embodiment of the present invention. [Diagram 3] FIG. 3 shows a block diagram of the working vehicle steering assistance system according to the first embodiment of the present invention. [Figure 4] FIG. 4 shows a side view of a crane to which the work vehicle steering assistance system according to the first embodiment of the present invention is connected. [Diagram 5] FIG. 5 shows a control block diagram of a crane to which the working vehicle steering assistance system according to the first embodiment of the present invention is connected. [Figure 6] FIG. 6 shows a block diagram when the driving assistance system for the work vehicle according to the first embodiment of the present invention generates information required for displaying a virtual image. [Figure 7] FIG. 7 is a schematic diagram showing a state in which the working vehicle driving assistance system according to the first embodiment of the present invention displays a virtual image. [Figure 8] FIG. 8 is a schematic diagram showing a state in which the work vehicle steering assistance system according to the first embodiment of the present invention displays a virtual image in a state in which a part of the boom cannot be visually recognized. [Figure 9] FIG. 9 shows a block diagram when the driving assistance system for a work vehicle according to the second embodiment of the present invention generates information required for displaying a virtual image. [Figure 10] FIG. 10 is a schematic diagram showing a state in which the working vehicle driving assistance system according to the second embodiment of the present invention displays a virtual image in which the ground surface is used as a reference plane. [Figure 11]FIG. 11 is a schematic diagram showing a state in which the working vehicle steering assistance system according to the second embodiment of the present invention displays a virtual image when the horizontal plane including the boom tip is set as the reference plane. [Figure 12] FIG. 12 shows a block diagram when the driving assistance system for a work vehicle according to the third embodiment of the present invention generates information required for displaying a virtual image. [Figure 13] FIG. 13 shows a block diagram when the driving assistance system for a work vehicle according to the fourth embodiment of the present invention generates information required for displaying a virtual image. [Figure 14] FIG. 14 shows a block diagram when the driving assistance system for a work vehicle according to the fifth embodiment of the present invention generates information required for displaying a virtual image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0030] [Embodiment 1] Hereinafter, a maneuvering support system 1 for a crane 11, which is a work vehicle having a boom 19 according to a first embodiment of the present invention, will be described with reference to Figs. 1 to 3. Fig. 1 is a perspective view of a transmissive display device 2 in the maneuvering support system 1 for a crane 11 according to a first embodiment of the present invention worn by a user. Fig. 2 is a perspective view of the transmissive display device 2 in the maneuvering support system 1 for a crane 11. Fig. 3 is a block diagram of the maneuvering support system 1 for a crane 11. In this embodiment, a rough terrain crane (hereinafter simply referred to as "crane") will be described as the work vehicle. However, the work vehicle may be any work vehicle provided with a boom that can be raised and lowered via a swivel such as an all-terrain crane, a truck crane, or an aerial work vehicle.

[0031] 1 and 2, a driving assistance system 1 for a crane 11 (see FIG. 4), which is a work vehicle, is a system that uses mixed reality to assist in the driving of the crane 11. The driving assistance system 1 has a transmissive display device 2, a communication device 8, and an image processing device 9.

[0032] The operation assistance system 1 is a system that assists in the operation of a crane 11 having a boom 19. The operation assistance system 1 of this embodiment includes a display (specifically, a transmissive display 4) that is transparent and capable of displaying an image, a generating unit (image processing device 9) that generates a virtual image (specifically, a first virtual image Im1) showing a target member including the boom 19 and / or a member (specifically, a wire rope and / or a hook) that moves with the boom 19, and a control unit (display device control device 7) that displays the virtual image on the image processing device 9 so as to overlap with a view seen through the image processing device 9 in a manner that allows a user of the image processing device 9 to recognize the position of the target member. A specific configuration of the operation assistance system 1 will be described below.

[0033] The transmissive display device 2 is a display device that allows a user to visually recognize a virtual image seen from the viewpoint of the user of the transmissive display device 2 in a state where the virtual image is superimposed on a background (scenery) Ls of the transmissive display device 2 seen from the viewpoint of the user. The transmissive display device 2 is a wearable terminal that is worn by the user on the body. In this embodiment, the transmissive display device 2 is described as a glasses-type wearable terminal. The transmissive display device 2 is worn by the operator of the crane 11, who is the user. The transmissive display device 2 has a frame 3, a transmissive display 4, an inertial measurement unit 5, a display device camera 6, and a display device control device 7.

[0034] The frame 3 supports the see-through display 4, the inertial measurement unit 5, the display device camera 6, and the display device control device 7. The frame 3 is configured as a glasses-type frame. The frame 3 is worn on the head of a user of the see-through display device 2. The frame 3 is supported by the nose and both ears of the user.

[0035] The transmissive display 4 is a display with a transmittance of approximately 50% or more. The transmissive display 4 does not have a backlight, a polarizing filter, or a color filter. The transmissive display 4 may be a liquid crystal display, an organic EL display, an inorganic EL display, or the like. In other words, the transmissive display 4 does not have any members such as a backlight or a filter that covers the back surface of the display screen that faces the user of the transmissive display device 2. Therefore, the transmissive display 4 allows the background that overlaps with the transmissive display 4 to be seen through from the viewpoint of the user of the transmissive display device 2.

[0036] The transmissive display 4 displays on the display screen a first virtual image Im1, a second virtual image Im2, a third virtual image Im3, a fourth virtual image Im4, and a fifth virtual image Im5 generated by the display device control device 7. The first virtual image Im1 includes at least one of a first boom virtual image Im1a which is a virtual image of the boom 19, and a first reference member virtual image Im1b which is a virtual image of the reference member Rm. The first virtual image Im1 (the first boom virtual image Im1a and the first reference member virtual image Im1b) corresponds to an example of a virtual image indicating a target member.

[0037] The transmissive display 4 transmits light incident from the back surface in a portion where the virtual image is not displayed. The transmissive display 4 also blocks at least a portion of the light incident from the back surface in a portion where the virtual image is displayed. Furthermore, the transmissive display 4 displays the virtual image in the light-blocked portion. At this time, the transmissive display 4 displays the virtual image as seen from the viewpoint of the user of the transmissive display device 2. As a result, the transmissive display 4 allows the user of the transmissive display device 2 to visually recognize a state in which the virtual image is superimposed on the background Ls as seen from the viewpoint of the user of the transmissive display device 2. The transmissive display 4 is provided on the frame 3.

[0038] The transmissive display 4 is positioned so as to cover the visual field of the right eye and the visual field of the left eye of a user wearing the frame 3. The transmissive display 4 is configured to be able to display a virtual image for the right eye and a virtual image for the left eye as parallax images on the display screen. This allows the transmissive display 4 to display a stereoscopic virtual image in almost the entire visual field of the user. Note that a technology for displaying a virtual image seen from the user's viewpoint within the visual field of the user in the transmissive display device 2 is well known.

[0039] An inertial measurement unit 5 (IMU) calculates the position and orientation of the transmissive display device 2. The inertial measurement unit 5 is located on the frame 3. The inertial measurement unit 5 calculates angles and angular accelerations around the X-axis, Y-axis, and Z-axis. In other words, the inertial measurement unit 5 calculates the movement direction and movement amount of the transmissive display device 2 based on an arbitrary position.

[0040] The display device camera 6 captures an image in the direction of the line of sight of a user wearing the transmissive display device 2. The display device camera 6 is provided on the frame 3. The display device camera 6 captures an image of a range approximately equal to the visual field range of the user from a position approximately equal to the viewpoint of the user of the transmissive display device 2. Note that, although the display device camera 6 is provided on the frame 3 in this embodiment, it may be provided in a position where it can capture an image of the visual field range of the user of the transmissive display device 2 from the viewpoint of the user of the transmissive display device 2.

[0041] As shown in FIG. 3 , the display device control device 7 is a control device that causes the transmissive display 4 to display a first virtual image Im1, a second virtual image Im2, a third virtual image Im3, a fourth virtual image Im4, and / or a fifth virtual image Im5. The display device control device 7 is actually configured with a CPU, a ROM, a RAM, a HDD, etc., connected via a bus. Alternatively, the display device control device 7 may be configured with a one-chip LSI, etc. The display device is connected to the transmissive display 4 and the display device camera 6. The display device control device 7 is connected to an image processing device 9 via a communication device 8.

[0042] The display device control device 7 can acquire the display device image P1 captured by the display device camera 6. The display device control device 7 can recognize a specific marker included in the display device image P1. When the display device control device 7 has acquired the coordinates of the specific marker in advance, it can calculate the position and / or attitude of the display device camera 6 by recognizing the specific marker.

[0043] The specific marker is located, for example, at a predetermined location in the cabin 27 of the crane 11, which will be described later. Furthermore, the display device control device 7 calculates the position and / or attitude, etc. of the display device camera 6 for each unit time using the inertial measurement device 5 based on the position and / or attitude, etc. of the display device camera 6 calculated based on the specific marker. In this way, when the past display device image P1 includes the specific marker, the display device control device 7 can calculate the current position and / or attitude, etc. of the display device camera 6 even if the current display device image P1 does not include the specific marker.

[0044] The position of the user of the transparent display device 2 is determined in advance with reference to the display device camera 6. Therefore, the display device control device 7 can generate information Ifa about the position of the user of the transparent display device 2 based on the calculated position and / or attitude of the display device camera 6. The information Ifa about the user's position includes coordinate information such as the latitude, longitude, and altitude of the location of the user of the transparent display device 2, as well as the viewing direction of the user of the transparent display device 2. The transparent display device 2 calculates the information Ifa about the position of the user of the transparent display device 2 by recognizing a marker located in the cabin 27 of the crane 11, which will be described later, for example.

[0045] The display device control device 7 acquires information If1 on the first virtual image Im1, information If2 on the second virtual image Im2, information If3 on the third virtual image Im3, information If4 on the fourth virtual image Im4, and / or information If5 on the fifth virtual image Im5 to be displayed on the transmissive display 4 from the image processing device 9. The display device control device 7 can display the first virtual image Im1, the second virtual image Im2, the third virtual image Im3, the fourth virtual image Im4, and the fifth virtual image Im5 viewed from a viewpoint of a user of the transmissive display device 2 on the transmissive display 4 based on the acquired information If1 on the first virtual image Im1, information If2 on the second virtual image Im2, information If3 on the third virtual image Im3, information If4 on the fourth virtual image Im4, and / or information If5 on the fifth virtual image Im5.

[0046] When worn on the head of a user, the transmissive display device 2 can display a first virtual image Im1, a second virtual image Im2, a third virtual image Im3, a fourth virtual image Im4, and / or a fifth virtual image Im5 on the display screen of the transmissive display 4. Furthermore, the transmissive display device 2 can calculate information Ifa about the position of the user that is required to display the first virtual image Im1, the second virtual image Im2, the third virtual image Im3, the fourth virtual image Im4, and / or the fifth virtual image Im5 on the transmissive display 4 by recognizing a specific marker.

[0047] The transmissive display device 2 can display at least one of the first virtual image Im1, the second virtual image Im2, the third virtual image Im3, the fourth virtual image Im4, and the fifth virtual image Im5 viewed from the viewpoint of the user of the transmissive display device 2 on the display screen of the transmissive display 4 through which the background Ls is transmitted. This allows the transmissive display device 2 to allow the user of the transmissive display device 2 to visually recognize at least one of the first virtual image Im1, the second virtual image Im2, the third virtual image Im3, the fourth virtual image Im4, and the fifth virtual image Im5 in a state where they are superimposed on the background Ls of the transmissive display device 2.

[0048] As shown in Fig. 3, the communication device 8 is a device that transmits and receives information between the transmissive display device 2, the image processing device 9, and the crane control device 30. The communication device 8 is provided in each of the display device control device 7 of the transmissive display device 2 and the image processing device 9. The communication device 8 transmits and receives information between the crane control device 30 and the image processing device 9. In addition, the communication device 8 transmits and receives information between the display device control device 7 and the image processing device 9.

[0049] The image processing device 9 corresponds to an example of a generating unit, and is a control device that generates information If1 related to the first virtual image Im1, information If2 related to the second virtual image Im2, information If3 related to the third virtual image Im3, information If4 related to the fourth virtual image Im4, and information If5 related to the fifth virtual image Im5. The image processing device 9 is substantially a CPU, a ROM, a RAM, a HDD, etc., connected via a bus. Alternatively, the image processing device 9 may be configured to include a one-chip LSI, etc. In this embodiment, the image processing device 9 is provided in the transmissive display device 2. In other words, the image processing device 9 is supported by the frame 3.

[0050] The image processing device 9 stores various programs and data for controlling the operation of the transmissive display device 2 and the like and for processing image data. The image processing device 9 may be configured integrally with the display device control device 7 or a crane control device 30 of a crane 11, which is a work vehicle described later.

[0051] The image processing device 9 is connected to the display device control device 7 of the transmissive display device 2 via the communication device 8. The image processing device 9 can acquire information Ifa relating to the position of the user from the display device control device 7 via the communication device 8. The image processing device 9 can also acquire a display device image P1 captured by the display device camera 6 from the display device control device 7 via the communication device 8. The image processing device 9 can also acquire a reference member image P2, which is an image of a predetermined range including the reference member Rm captured by the boom camera 19b (see FIG. 4) of the crane 11, which is an imaging device, via the communication device 8.

[0052] The image processing device 9 can acquire information Ifb about the position of the crane 11 from the crane control device 30 via the communication device 8. The information Ifb about the position of the crane 11 includes coordinate information such as the latitude, longitude, and altitude of the position of the rotation center of the crane 11, as well as the direction of the vehicle 12, etc.

[0053] The image processing device 9 can acquire information Ifc about the attitude of the crane 11 from the crane control device 30 via the communication device 8. The information Ifc about the attitude of the crane 11 includes specifications such as the slewing angle of the crane 11, the boom hoisting angle, the boom length, the payout length of the main wire rope 24, the payout length of the sub-wire rope 26, the angle of view of the boom camera 19b, and the like, as well as the operation direction and operation amount of each operating tool of the crane 11 (see Figs. 4 and 5).

[0054] The image processing device 9 can acquire information Ifi about the shape of the crane 11 from the crane control device 30 or an external server (not shown) via the communication device 8. The information Ifi about the shape of the crane 11 is information about three-dimensional coordinates (in other words, a three-dimensional image) about the shapes of members related to the boom 19, such as the boom 19, jib 19c, main hook block 20, main hook 20a, sub-hook block 21, sub-hook 21a, main wire rope 24, and sub-wire rope 26, which will be described later. The information Ifi about the shape of the crane 11 also includes coordinate information about the latitude, longitude, and altitude of a boom camera 19b provided on the boom 19, and the shooting direction of the boom camera 19b.

[0055] The image processing device 9 can acquire information Ifh about the shape of the reference member Rm from the crane control device 30 or the BIM held by an external server via the communication device 8. The reference member Rm is the main hook block 20, the main hook 20a, the sub-hook block 21, the sub-hook 21a, the main wire rope 24, the sub-wire rope 26, the cargo W, and the like, which move with the movement of the boom 19. The reference member Rm can be arbitrarily selected by the user of the transmissive display device 2. The information Ifh about the shape of the reference member Rm is information about three-dimensional coordinates (in other words, a three-dimensional image) of the reference member Rm. In this embodiment, the reference member Rm is the main hook 20a.

[0056] The image processing device 9 can acquire information Iff about the reference member image P2 from the crane control device 30 or the boom camera 19b which is an image capturing device of the crane 11 via the communication device 8. The information Iff about the reference member image P2 is the reference member image P2 captured by the boom camera 19b and the angle of view of the boom camera 19b when the reference member image P2 was captured, etc. The information Iff about the reference member image P2 includes information Ifg about the attitude of the reference member Rm.

[0057] The crane 11 operation assistance system 1 configured in this manner obtains, by the image processing device 9, information Ifa regarding the position of the user of the transparent display device 2, information Ifb regarding the position of the crane 11, information Ifc regarding the posture of the crane 11, information Iff regarding the reference member image P2, and information Ifi regarding the shape of the crane 11.

[0058] A crane 11, which is a work vehicle according to one embodiment of the present invention, will be described below with reference to Figures 4 and 5. Figure 4 is a side view of the crane 11 to which the operation assistance system 1 is connected. Figure 5 is a control block diagram of the crane 11 to which the operation assistance system 1 is connected.

[0059] 4 and 5, the crane 11 is a mobile crane that can be moved to any location. The crane 11 includes a vehicle 12, a crane device 16, a boom camera 19b that is an imaging device, and a crane control device 30.

[0060] The vehicle 12 is a traveling body that transports the crane device 16. The vehicle 12 has a plurality of wheels 13. The vehicle 12 travels using an engine 14 as a power source. The vehicle 12 has outriggers 15. The outriggers 15 have hydraulically extendable extension beams on both sides in the width direction of the vehicle 12, and hydraulic jack cylinders that can be extended in a direction perpendicular to the ground.

[0061] The crane apparatus 16 is a work device that hoists a load W with a wire rope. The crane apparatus 16 includes a swivel base 17, a crane GNSS receiver 18, a boom 19, a main hook block 20, a sub-hook block 21, a hoisting hydraulic cylinder 22, a main winch 23, a main wire rope 24, a sub-winch 25, a sub-wire rope 26, and a cabin 27.

[0062] The swivel table 17 is a rotating device that configures the crane device 16 to be rotatable. The swivel table 17 is mounted on the frame of the vehicle 12 via annular bearings. The swivel table 17 is configured to be freely rotatable around the center of the annular bearing. The swivel table 17 has a hydraulic swivel motor 17a that serves as an actuator. The swivel table 17 is configured to be rotatable in one direction and the other direction by the swivel hydraulic motor 17a.

[0063] The crane GNSS receiver 18 is a receiver that constitutes a Global Navigation Satellite System. The crane GNSS receiver 18 receives ranging radio waves from satellites and detects the absolute coordinates of the crane GNSS receiver 18, such as latitude, longitude, altitude, and direction. The crane GNSS receiver 18 is provided on the rotating base 17. The crane GNSS receiver 18 detects information Ifb relating to the position of the crane 11.

[0064] The boom 19 is a movable support that supports the main wire rope 24 and the sub wire rope 26. The boom 19 is composed of multiple boom members. The base end of the base boom member of the boom 19 is swingably mounted approximately at the center of the swivel base 17. The boom 19 has an extension hydraulic cylinder 19a, which is an actuator that extends and retracts each boom member, and a hoist hydraulic cylinder 22 that raises and lowers the boom member.

[0065] The boom 19 is extended and retracted in the axial direction by an extension hydraulic cylinder 19a. The boom 19 has a boom camera 19b which is an image capturing device. The boom 19 further has a jib 19c which is an extension member. The reference member image P2 which is an image captured by the boom camera 19b is transmitted to the image processing device 9 via the communication device 30a of the crane control device 30.

[0066] The main hook block 20 and the sub-hook block 21 are components that suspend a load W. The main hook block 20 has a plurality of hook sheaves around which the main wire rope 24 is wound, and a main hook 20a that suspends the load W. The sub-hook block 21 has a sub-hook 21a that suspends the load W.

[0067] The main winch 23 is a device that pays in (winds up) and out (winds down) the main wire rope 24. The sub winch 25 is a device that pays in (winds up) and out (winds down) the sub wire rope 26. In the main winch 23, a main drum around which the main wire rope 24 is wound is rotated by a main hydraulic motor (not shown) that serves as an actuator. In the sub winch 25, a sub drum around which the sub wire rope 26 is wound is rotated by a sub hydraulic motor (not shown) that serves as an actuator.

[0068] The cabin 27 is a housing that covers the operator's seat. The cabin 27 is mounted on the rotating base 17. The cabin 27 is provided with an operator's seat (not shown). The operator's seat is provided with crane apparatus operating tools 29, including a traveling operation tool 28 for operating the vehicle 12 to travel, a turning operation tool 29a for operating the turning hydraulic motor 17a of the crane apparatus 16, an extension operation tool 29b for operating the extension cylinder, a hoisting operation tool 29c for operating the hoisting hydraulic cylinder 22, a main operation tool 29d for operating the main winch 23, and a sub operation tool 29e for operating the sub winch 25 (see FIG. 5).

[0069] As shown in Fig. 5, the crane controller 30 controls each actuator of the crane 11. The crane controller 30 is provided inside the cabin 27. In reality, the crane controller 30 includes a CPU, a ROM, a RAM, a HDD, etc., which are connected via a bus. Alternatively, the crane controller 30 is composed of a one-chip LSI or the like.

[0070] The crane control device 30 stores various programs and data for controlling the operation of each actuator, switching valve, etc., and for processing image data. The crane control device 30 has a communication device 30a. The communication device 30a transmits and receives information to and from the image processing device 9 of the operation assistance system 1.

[0071] The crane control device 30 is connected to the crane GNSS receiver 18. The crane control device 30 can acquire information Ifb relating to the position of the crane 11. The crane control device 30 is connected to the traveling operation tool 28 and the crane equipment operation tool 29. The crane control device 30 can acquire operation signals of the traveling operation tool 28 and the crane equipment operation tool 29.

[0072] The crane controller 30 generates a control signal for the vehicle 12 based on an operation signal generated by operating the traveling operation tool 28. Similarly, the crane controller 30 generates a control signal for the crane apparatus 16 based on an operation signal generated by operating the crane apparatus operation tool 29. The crane controller 30 can transmit the generated control signal to each actuator of the crane apparatus 16.

[0073] The crane controller 30 can generate information Ifc about the posture of the crane 11 from the acquired operation signals of each of the crane operating tools 29. The crane controller 30 is assumed to hold in advance information Ifi about the shape of the crane 11 and information Ifh about the shape of the reference member Rm.

[0074] The crane control device 30 is connected to the image processing device 9 of the operation assistance system 1 via the communication device 30a. The crane control device 30 can transmit to the image processing device 9 information Ifb on the position of the crane 11, information Ifc on the attitude of the crane 11, information Ifi on the shape of the crane 11, information Ifh on the shape of the reference member Rm, and information Iff on the reference member image P2.

[0075] The crane 11 thus configured can move the vehicle 12 to any position by operating the traveling operation tool 28. Also, the crane 11 can transport the cargo W to any position by rotating, raising, lowering, and extending the boom 19 by operating the crane device operation tool 29. Also, the crane 11 can transmit various types of information to the operation assistance system 1.

[0076] <Current position display of the boom 19 and main hook 20a> 6 to 8, the display of the current positions of the boom 19, main hook block 20, and main hook 20a (hereinafter simply referred to as "main hook 20a") to the operator of the crane 11 will be specifically described as a first embodiment of the operation assistance system 1. FIG. 6 is a control block diagram when the operation assistance system 1 of the crane 11 displays the first virtual image Im1. FIG. 7 is a schematic diagram of a state in which the operation assistance system 1 of the crane 11 displays the first virtual image Im1, the third virtual image Im3, the fourth virtual image Im4, and the fifth virtual image Im5. FIG. 8 is a schematic diagram of a state in which the operation assistance system 1 of the crane 11 displays the first boom virtual image Im1a and the first reference member virtual image Im1b while parts of the boom 19 and the main hook 20a are not visible.

[0077] The operator of the crane 11 is also the user of the transmissive display device 2. The operator of the crane 11 operates the crane 11 while wearing the transmissive display device 2 on his head. The operator of the crane 11 also operates the crane 11 while seated in a cabin 27 (see FIG. 4) of the crane 11. In this embodiment, the reference member Rm is the main hook 20a of the crane 11.

[0078] As shown in FIG. 6, the image processing device 9 of the operation support system 1 connected to the crane control device 30 of the crane 11 calculates the first reference coordinates C1, which are a set of three-dimensional coordinates of the boom 19 in the current attitude of the boom 19, based on the information Ifb about the position of the crane 11, the information Ifc about the attitude of the crane 11, and the information Ifi about the shape of the crane 11. Furthermore, the image processing device 9 generates information If1 about the first virtual image Im1 for displaying the first boom virtual image Im1a showing the shape of the boom 19 seen from the viewpoint of the user of the transmissive display device 2 on the display screen of the transmissive display device 4, based on the information Ifa about the position of the user and the first reference coordinates C1. In other words, the image processing device 9 determines the position on the display where the virtual image (first boom virtual image Im1a) is to be displayed, based on the information about the position of the user of the display and the information about the position of the target member calculated based on the attitude of the work vehicle and the position of the work vehicle. Then, the image processing device 9 sends information about the position on the display where the determined virtual image is to be displayed to the display device control device 7. Note that when determining the position in this manner, the information Ifi about the shape of the crane 11 may be omitted.

[0079] Furthermore, the image processing device 9 extracts the reference member Rm from the reference member image P2 based on the information Ifh on the shape of the reference member Rm and the information Iff on the reference member image P2. Furthermore, the image processing device 9 calculates the current position RLp of the reference member Rm, including coordinate information such as the current latitude, longitude, and altitude (height from the tip of the boom 19) of the reference member Rm, and the direction of the reference member Rm, based on the reference member Rm extracted from the reference member image P2, the information Ifb on the position of the crane 11, the information Ifc on the attitude of the crane 11, and the information Ifi on the shape of the crane 11. The current position RLp of the reference member Rm is calculated from the position of the reference member Rm included in the reference member image P2 captured by the boom camera 19b (see FIG. 5) at every unit time. In other words, the current position RLp also includes position fluctuations due to the swaying of the reference member Rm, etc.

[0080] The image processing device 9 calculates second reference coordinates C2, which are a set of three-dimensional coordinates of the reference member Rm at the current position RLp of the reference member Rm, based on the current position RLp of the reference member Rm, information Ifc about the attitude of the crane 11, and information Ifh about the shape of the reference member Rm. Furthermore, the image processing device 9 generates information If1 about a first virtual image Im1 for displaying a first reference member virtual image Im1b showing the shape of the reference member Rm seen from the viewpoint of the user of the transmissive display device 2 on the display screen of the transmissive display 4, based on the information Ifa about the user's position and the second reference coordinates C2. In other words, the image processing device 9 determines a position on the display where a virtual image (first reference member virtual image Im1b) is to be displayed, based on information about the position of the user of the display and information about the position of the target member calculated based on the attitude of the work vehicle and the position of the work vehicle. Then, the image processing device 9 sends information about the position on the display where the determined virtual image is to be displayed to the display device control device 7. The method of calculating the current position RLp of the reference member Rm is not limited to the above-mentioned method. The current position RLp of the reference member Rm may be calculated, for example, based on information regarding the position of the crane and information regarding the attitude of the crane.

[0081] The image processing device 9 transmits information If1 regarding the first virtual image Im1 to the display device control device 7 to display at least one of the generated first boom virtual image Im1a and the first reference member virtual image Im1b on the display screen of the transmissive display 4.

[0082] The display device control device 7 acquires information If1 about the first virtual image Im1 from the image processing device 9 via the communication device 8.

[0083] 7, based on information If1 related to the first virtual image Im1, the display device control device 7 displays at least one of a first boom virtual image Im1a showing the shape of the boom 19 seen from the viewpoint of the user of the transmissive display device 2 and a first reference member virtual image Im1b showing the shape of the reference member Rm seen from the viewpoint of the user on a display screen of the transmissive display device 4 overlapping with the boom 19 visually recognized by the user of the transmissive display device 2 through the transmissive display 4. The first boom virtual image Im1a overlaps with the boom 19 visually recognized by the user of the transmissive display device 2 through the transmissive display 4. The first reference member virtual image Im1b overlaps with the main hook 20a which is the reference member Rm visually recognized by the user of the transmissive display device 2 through the transmissive display 4.

[0084] 8, even when at least a part of the boom 19 cannot be seen from the viewpoint of the user of the transmissive display device 2 because of being blocked by a structure C or the like, the transmissive display device 2 displays on the transmissive display 4 a first boom virtual image Im1a indicating the shape of the boom 19 seen from the viewpoint of the user of the transmissive display device 2. Similarly, even when at least a part of the reference member Rm cannot be seen from the viewpoint of the user of the transmissive display device 2 because of being blocked by a structure C or the like, the transmissive display device 2 displays on the transmissive display 4 a first reference member virtual image Im1b indicating the shape of the reference member Rm seen from the viewpoint of the user. In other words, when the target member is hidden by an obstacle and cannot be seen, the display device control device 7 displays virtual images (specifically, the first boom virtual image Im1a and the first reference member virtual image Im1b) on the transmissive display 4 so as to overlap the obstacle.

[0085] In this way, the image processing device 9 generates information If1 about the first virtual image Im1 for displaying the first boom virtual image Im1a, based on the information Ifa about the user's position, the information Ifb about the position of the crane 11, the information Ifc about the attitude of the crane 11, and the information Ifi about the shape of the crane 11. In addition, the image processing device 9 generates information If1 about the first virtual image Im1 for displaying the first reference member virtual image Im1b, based on the information Ifa about the user's position, the information Ifb about the position of the crane 11, the information Ifc about the attitude of the crane 11, the information Ifh about the shape of the reference member Rm, and the reference member image P2.

[0086] That is, when generating the information If1 related to the first virtual image Im1, the image processing device 9 does not use image information including the boom 19 and the reference member Rm as viewed from the viewpoint of the user of the transmissive display device 2. Therefore, the image processing device 9 can generate the information If1 related to the first virtual image Im1 for displaying the first boom virtual image Im1a and the first reference member virtual image Im1b even if at least a part of the boom 19 and the reference member Rm are blocked as viewed from the viewpoint of the user of the transmissive display device 2.

[0087] An operator of the crane 11 using the transmissive display device 2 can operate the crane 11 while objectively grasping the positional relationship between the surrounding structures C and the boom 19 and main hook 20a through at least one of the first boom virtual image Im1a and the first reference member virtual image Im1b, even if the boom 19 or the main hook 20a suspending the load W cannot be seen from inside the cabin 27 during transport of the load W. This reduces the effect that information obtained from the instructor has on the operation, and allows the crane 11 to be operated efficiently.

[0088] <Embodiment 2: Boom camera image display> Hereinafter, with reference to Figs. 9 to 11, image display of the boom camera 19b on an arbitrary reference plane S1 will be specifically described as embodiment 2 of the operation support system 1 for the operator of the crane 11. Fig. 9 shows a block diagram when the operation support system 1 of the crane 11 generates information If2 on the second virtual image required for displaying the second virtual image Im2. Fig. 10 shows a schematic diagram of a state in which the operation support system 1 of the crane 11 displays the second virtual image Im2 when the ground surface is set as the reference plane S1. Fig. 11 shows a schematic diagram of a state in which the operation support system 1 of the crane 11 displays the second virtual image Im2 when the horizontal plane including the current position BLp of the boom tip is set as the reference plane S1.

[0089] As shown in Figure 9, the operation support system 1 of the crane 11 displays a reference member image P2, which is an image of a predetermined range including the reference member Rm captured by the boom camera 19b of the boom 19, as a second virtual image Im2 on an arbitrary reference plane S1 on a transparent display device 2 (see Figure 3).

[0090] The display device control device 7 of the transmissive display device 2 has information Ifa about the position of the user and information Ifd about the reference plane S1. The information Ifd about the reference plane S1 includes the position and inclination of a virtual plane that is the reference for the position at which the second virtual image Im2 is displayed as seen from the user's viewpoint. In this embodiment, the reference plane S1 is at least one of the ground surface and a horizontal plane including the current position BLp of the tip of the boom 19, and is set in advance within the background (scenery) Ls. The reference plane S1 may be arbitrarily determined by the user.

[0091] The image processing device 9, which is connected to the display device control device 7 via the communication device 8, acquires information Ifa relating to the position of the user from the display device control device 7. In addition, the image processing device 9, which is connected to the crane control device 30 via the communication device 8, acquires, from the crane control device 30, the reference member image P2 captured by the boom camera 19b, information Ifb relating to the position of the crane 11, and information Ifc relating to the attitude of the crane 11. The image processing device 9 determines the reference plane S1 from the acquired information Ifa relating to the position of the user, information Ifb relating to the position of the crane 11, and information Ifc relating to the attitude of the crane 11.

[0092] 10, when the image processing device 9 determines that the user's viewing direction is below the horizontal direction based on the information Ifa about the user's position, the image processing device 9 sets the reference plane S1 to the ground plane. In other words, the image processing device 9 sets the reference plane based on the viewing direction of the display user.

[0093] As shown in Figure 11, when the image processing device 9 determines based on the information Ifa about the user's position that the user's viewing direction is above the horizontal direction, it sets the reference plane S1 to a horizontal plane including the current position BLp of the tip of the boom 19.

[0094] 9, when the reference plane S1 is a horizontal plane including the current position BLp, i.e., when the reference plane S1 is higher than the position of the user's horizontal line of sight, the image processing device 9 generates information If2 about the second virtual image Im2 for displaying an image in which the reference member image P2 is inverted as the second virtual image Im2 on the display screen of the transmissive display 4. When the reference plane S1 is the ground surface, i.e., when the reference plane S1 is lower than the position of the user's horizontal line of sight, the image processing device 9 generates information If2 about the second virtual image Im2 for displaying an image in which the reference member image P2 is not inverted as the second virtual image Im2 on the display screen of the transmissive display 4.

[0095] 10 and 11, the display device control device 7 of the transmissive display device 2 acquires information If2 related to the reference member image P2 and the second virtual image Im2 from the image processing device 9 via the communication device 8. Furthermore, the display device control device 7 displays the second virtual image Im2 viewed from the user's viewpoint on the reference plane S1 on the display screen of the transmissive display 4.

[0096] In the above-mentioned configuration, the operation support system 1 of the crane 11 generates information If2 on the second virtual image Im2 for displaying the reference member image P2 captured by the boom camera 19b provided on the boom 19 as the second virtual image Im2, which is an image seen from the viewpoint of the user of the transmissive display device 2, on the background Ls of the transmissive display device 2 viewed by the operator of the crane 11, who is the user of the transmissive display device 2. The operation support system 1 displays the second virtual image Im2 on the display screen of the transmissive display 4 so that the second virtual image Im2 is superimposed on an arbitrary reference plane S1 designated by the user. When the second virtual image Im2 includes an image of the main hook 20a that moves with the movement of the boom 19, the image of the main hook 20a is displayed so that it is superimposed on the main hook 20a when viewed in the vertical direction. The operator of the crane 11 can use the second virtual image Im2 on the reference plane S1 as a projection of the main hook 20a.

[0097] Therefore, even if the operator of the crane 11 receives insufficient instructions from the instructor and is unable to see the ground surface condition, the operator can receive supplemental information regarding the positional relationship between the boom 19 and the main hook 20a and the surrounding structure C, which is necessary for operating the boom 19, by displaying the second virtual image Im2 within the operator's field of vision. This allows the operator of the crane 11 to efficiently operate the crane 11 while suppressing the effect that the information obtained from the instructor has on the operation.

[0098] <Embodiment 3: Display of current position BLp and trajectory of the tip of the boom 19> 7, 8, and 12, a specific description will be given below of trajectory display at the current position BLp of the tip of the boom 19 as a third embodiment of the operation assistance system 1 for the operator of the crane 11. Fig. 12 is a control block diagram when the operation assistance system 1 of the crane 11 displays the third virtual image Im3.

[0099] As shown in Fig. 7, the operation support system 1 for the crane 11 displays, on the transmissive display device 2 (see Fig. 3), the tip of the boom 19 moved by the crane apparatus 16 when the crane apparatus operating tool 29 is operated and the trajectory of the tip of the boom 19 at the current position BLp as a third virtual image Im3 on the reference plane S1. That is, the third virtual image Im3 includes an image showing the trajectory of the tip of the boom corresponding to the attitude of the boom. The third virtual image Im3 is an image included in the virtual image showing the target member.

[0100] The display device control device 7 has information Ifd about the reference plane S1. In this embodiment, the reference plane S1 is the earth's surface.

[0101] 12, the image processing device 9 can acquire information Ifd about the reference plane S1 from the display device control device 7 via the communication device 8. The information Ifd about the reference plane S1 includes the position and inclination of a virtual plane that serves as a reference for the position at which the third virtual image Im3 is displayed as viewed from the user's viewpoint.

[0102] The image processing device 9 acquires information Ifb about the position of the crane 11 and information Ifc about the attitude of the crane 11 from the crane control device 30. Based on the acquired information Ifb about the position of the crane 11, information Ifc about the attitude of the crane 11, and information Ifd about the reference plane S1, the image processing device 9 calculates third reference coordinates C3 which are a set of coordinates for displaying, on the reference plane S1, the current position BLp of the tip of the boom 19 when the crane operating tool 29 is operated and the trajectory coordinates of the boom 19 from the current position BLp.

[0103] Furthermore, based on the information Ifa regarding the user's position and the third reference coordinate C3, the image processing device 9 generates information If3 regarding the third virtual image Im3 for displaying the third virtual image Im3 showing the current position BLp of the boom 19 as seen from the viewpoint of the user of the transparent display device 2 and the trajectory at the current position BLp on the display screen of the transparent display 4.

[0104] The image processing device 9 transmits information If3 relating to the generated third virtual image Im3 to the display device control device .

[0105] The transmissive display device 2 acquires information If3 related to the third virtual image Im3 from the image processing device 9 via the communication device 8.

[0106] 7, the transmissive display device 2 displays a third virtual image Im3 in addition to the first virtual image Im1. The transmissive display device 2 displays the third virtual image Im3, which indicates the current position BLp of the boom 19 seen from the viewpoint of the user of the transmissive display device 2, and the trajectory at the current position BLp, on the display screen of the transmissive display 4, based on information If3 related to the third virtual image Im3.

[0107] The third virtual image Im3 is displayed superimposed on the reference plane S1 visually recognized by the user of the transmissive display device 2 through the transmissive display 4. When the reference plane S1 is the ground surface, the display device control device 7 displays the current position BLp of the boom 19 on the ground surface as seen from the user's viewpoint and its trajectory from the current position BLp as the third virtual image Im3 on the display screen of the transmissive display 4 overlapping with the ground surface visually recognized by the user of the transmissive display device 2 through the transmissive display 4.

[0108] 8, a user of the transmissive display device 2 visually recognizes the current position BLp of the tip of the boom 19 and the trajectory from the current position BLp, which are displayed on the ground surface of the work site, which is the actual background Ls, by the transmissive display device 2. Even in a state in which the tip of the boom 19 or the reference member Rm cannot be seen from the cockpit due to a structure C or the like, the operator can confirm the current position BLp of the tip of the boom 19 and the trajectory from the current position BLp by the third virtual image Im3 displayed on the ground surface.

[0109] 7 and 8, the operation assistance system 1 displays the first virtual image Im1 and the third virtual image Im3 on a background Ls within the field of view of the user of the transmissive display device 2, thereby providing the user of the transmissive display device 2 with information for grasping the current position BLp of the tip of the boom 19, the current position RLp of the main hook 20a, and the positional relationships between the boom 19 and the main hook 20a and the structure C. In addition, the operation assistance system 1 displays the current position BLp of the boom 19 from any direction and position desired by the user of the transmissive display device 2 and the trajectory from the current position BLp within the field of view of the user of the transmissive display device 2.

[0110] Therefore, even if the instructions from the instructor in charge of slinging are insufficient, the user of the transparent display device 2 can supplement the information necessary to operate the crane 11 with the current position BLp of the boom 19 and the trajectory from the current position BLp displayed within the field of view of the user of the transparent display device 2. This allows the user of the transparent display device 2 to efficiently operate the crane 11 while suppressing the effect that the information obtained from the instructor has on the operation.

[0111] <Fourth embodiment: Display of target position RLt of reference member Rm> 7 and 13, a specific description will be given of the display of the target position RLt of the reference member Rm as a fourth embodiment of the operation assistance system 1 for the operator of the crane 11. Fig. 13 is a block diagram when the operation assistance system 1 of the crane 11 generates information required to display a fourth virtual image Im4.

[0112] As shown in FIG. 7, the operation support system 1 of the crane 11 displays the shape of the reference member Rm at the target position RLt of the reference member Rm as a fourth virtual image Im4.

[0113] 13, the image processing device 9 acquires information Ifb about the position of the crane 11, information Ife about the target position RLt of the reference member Rm, and information Ifh about the shape of the reference member Rm from an external server or the crane control device 30 via the communication device 8. The information Ife about the target position RLt of the reference member Rm is coordinate information such as the latitude, longitude, and altitude (corresponding to the payout amount from the tip of the boom 19) of the target position RLt of the reference member Rm.

[0114] The image processing device 9 calculates fourth reference coordinates C4, which is a set of three-dimensional coordinates of the reference member Rm at the target position RLt, based on the information Ifb on the position of the crane 11, the information Ife on the target position RLt of the reference member Rm, and the information Ifh on the shape of the reference member Rm. Furthermore, the image processing device 9 generates information If4 on a fourth virtual image Im4, which is used to display, on the display screen of the transmissive display 4, a fourth virtual image Im4 indicating the shape of the reference member Rm at the target position RLt as seen from the viewpoint of the user of the transmissive display device 2, based on the information Ifa on the user's position and the fourth reference coordinates C4.

[0115] The image processing device 9 transmits information If4 relating to the generated fourth virtual image Im4 to the display device control device .

[0116] The display device control device 7 acquires information If4 on the fourth virtual image Im4 from the image processing device 9 via the communication device 8.

[0117] 7, the display device control device 7 displays a fourth virtual image Im4 indicating the shape of the reference member Rm seen from the viewpoint of the user of the transmissive display device 2 on the display screen of the transmissive display 4 based on information If4 related to the fourth virtual image Im4. The fourth virtual image Im4 is displayed at the target position RLt of the reference member Rm visually recognized by the user of the transmissive display device 2 through the transmissive display 4.

[0118] 7, based on the information If1 about the first virtual image Im1, the transmissive display device 2 displays a first virtual reference member image Im1b indicating the shape of the reference member Rm at the current position RLp seen from the viewpoint of the user of the transmissive display device 2 on the display screen of the transmissive display 4. The first virtual reference member image Im1b overlaps with the main hook 20a, which is the reference member Rm visually recognized by the user of the transmissive display device 2 through the transmissive display 4.

[0119] The transmissive display device 2 displays on the transmissive display 4 a fourth virtual image Im4 indicating the shape of the target position RLt on the reference member Rm as seen from the viewpoint of the user of the transmissive display device 2, even when the target position RLt of the reference member Rm is blocked by a structure C or the like and cannot be seen from the viewpoint of the user of the transmissive display device 2.

[0120] Even if the operator of the crane 11 using the transmissive display device 2 cannot see the target position RLt of the reference member Rm from the viewpoint of the user of the transmissive display device 2 due to obstruction by a structure C or the like, the operator can objectively grasp the positional relationship between the current position RLp of the reference member Rm and the target position RLt, and the positional relationship between the surrounding structures C and the reference member Rm at the current position RLp and the target position RLt. Therefore, even if the instruction from the instructor is insufficient, the operator can supplement the information required to move the reference member Rm located at the current position RLp to the target position RLt of the reference member Rm displayed within the operator's field of vision. This suppresses the influence of the information obtained from the instructor on the operation, and allows the operator to efficiently operate the crane 11. The fourth virtual image Im4 may include a virtual image showing the boom 19 and / or the wire rope (the main wire rope 24 and / or the sub wire rope 26). In other words, the display device control device 7 may display on the transmissive display 4 a virtual image showing the boom 19 and / or wire rope (main wire rope 24 and / or sub wire rope 26) at the position of the boom 19 and / or wire rope (main wire rope 24 and / or sub wire rope 26) corresponding to the target position RLt of the reference member Rm.

[0121] <Fifth embodiment: Display of perpendicular line at current position BLp of boom 19> 7, 8 and 14, a specific description will be given of the display of a perpendicular line passing through the current position BLp of the boom 19 as a fifth embodiment of the operation assistance system 1 for the operator of the crane 11. Fig. 14 is a block diagram showing how the operation assistance system 1 of the crane 11 generates information required to display a fifth virtual image Im5.

[0122] As shown in FIG. 7, the operation support system 1 of the crane 11 displays a perpendicular line passing through the current position BLp of the boom 19 as a fifth virtual image Im5.

[0123] 14, the image processing device 9 calculates a fifth reference coordinate C5, which is a set of coordinates for displaying a perpendicular line passing through the current position BLp of the boom 19, based on the acquired information Ifb about the position of the crane 11, information Ifc about the attitude of the crane 11, and information Ifd about the reference plane S1. The image processing device 9 generates the fifth reference coordinate C5 as information If5 about a fifth virtual image Im5 seen from the viewpoint of the user of the transmissive display device 2, based on the acquired information Ifa about the position of the user and the fifth reference coordinate C5. The image processing device 9 transmits the generated information If5 about the fifth virtual image Im5 to the display device control device 7.

[0124] The display device control device 7 acquires information If5 about the fifth virtual image Im5 via the communication device 8. The display device control device 7 displays, on the display screen of the transmissive display 4, the fifth virtual image Im5, which indicates a perpendicular line passing through the current position BLp of the boom 19 as seen from the viewpoint of the user of the transmissive display device 2, based on the information If5 about the fifth virtual image Im5.

[0125] 7 and 8, the user of the transmissive display device 2 recognizes the current position BLp of the boom 19 from the fifth virtual image Im5 indicating the current position BLp of the boom 19 displayed within the actual background Ls on the transmissive display device 2. When the fourth virtual image Im4 indicating the shape of the target position RLt on the reference member Rm and the fifth virtual image Im5 are both displayed, the user of the transmissive display device 2 can recognize the positional relationship between the current position BLp of the boom 19 and the target position RLt of the reference member Rm even in a state in which the boom 19 is not visible from the cockpit. Therefore, even if the instructions from the instructor are insufficient, the user of the transmissive display device 2 can efficiently operate the crane 11 by suppressing the influence of information obtained from the instructor on the operation.

[0126] <Other embodiments> In each of the above-described embodiments, the operation assistance system 1 of the crane 11 displays any one of the first virtual image Im1, the second virtual image Im2, the third virtual image Im3, the fourth virtual image Im4, and the fifth virtual image Im5 on the transmissive display device 2. However, the operation assistance system 1 of the crane 11 may display at least one of the first virtual image Im1, the second virtual image Im2, the third virtual image Im3, the fourth virtual image Im4, and the fifth virtual image Im5, selected arbitrarily by the user of the transmissive display device 2. Furthermore, the operation assistance system 1 of the crane 11 may display a virtual image that assists in the operation of the work vehicle, not limited to the virtual images displayed in the above-described embodiments.

[0127] In the above embodiment, the transparent display device 2 of the operation assistance system 1 is worn by an operator who operates the crane 11 inside the cabin 27 of the crane 11. However, the transparent display device 2 may be worn by an operator who remotely operates the crane 11 from outside the crane 11. For example, when a slinger wears the transparent display device 2, the slinger can convey the position of the boom 19 or the reference member Rm to the user of the transparent display device 2 of the crane 11 even if the boom 19 or the reference member Rm is blocked by a structure C or the like by displaying the first virtual image Im1 and the second virtual image Im2 on the transparent display device 2.

[0128] 10, the transmissive display device 2 of the operation assistance system 1 displays a third virtual image Im3 indicating the current position BLp of the boom 19 and the trajectory at the current position BLp on the reference plane S1 on which the second virtual image Im2 is displayed. However, the third virtual image Im3 may be displayed on any reference plane on which the second virtual image Im2 is not displayed. Also, for example, in the case of a baggage W being transported, the third virtual image may be displayed as a trajectory passing through the baggage W when the reference member Rm is being transported.

[0129] In addition, as the above-mentioned embodiment, the operation assistance system 1 is configured as an operation assistance system 1 for a crane 11. However, the operation assistance system may be configured as an operation assistance system for a vehicle for working at height, which is a work vehicle. The operation assistance system for a vehicle for working at height acquires information on the position of the user, information on the position of the vehicle for working at height, information on the attitude of the vehicle for working at height, and information on a reference plane including the position and attitude of a reference plane that is a reference for displaying a virtual image via a communication device. The image processing device of the operation assistance system generates information on a virtual image of the bucket for displaying the current position of the bucket, which is a reference member, and a trajectory from the current position on the display screen of the transmissive display device as a virtual image seen from the viewpoint of the operator of the vehicle for working at height. The transmissive display device acquires information on the virtual image of the bucket for the vehicle for working at height via the communication device, and displays the virtual image on the display screen.

[0130] In each of the above-described embodiments, the operation support system 1 of the crane 11 displays at least one of the first virtual image Im1, the second virtual image Im2, the third virtual image Im3, the fourth virtual image Im4, and the fifth virtual image Im5 by the transmissive display device 2. However, the operation support system 1 of the crane 11 may display at least one of the first virtual image Im1, the second virtual image Im2, the third virtual image Im3, the fourth virtual image Im4, and the fifth virtual image Im5 on an image captured by the camera. At least one of the first virtual image Im1, the second virtual image Im2, the third virtual image Im3, the fourth virtual image Im4, and the fifth virtual image Im5 may be operable on a display screen such as a touch panel.

[0131] Furthermore, in the above-described embodiment, the image processing device 9 acquires the information Ife about the target position RLt of the reference member Rm from an external server or the crane control device 30 via the communication device 8. However, the image processing device 9 may acquire the information Ife about the target position RLt by a user of the transmissive display device 2 using an input device or the like. The image processing device 9 may input the target position RLt of the reference member Rm, for example, using the transmissive display device 2 as an input device.

[0132] Furthermore, in the above-described embodiment, the image processing device 9 acquires the information Ifg about the posture of the reference member Rm from the reference member image P2 captured by the boom camera 19b. However, the image processing device 9 may acquire the information Ifg about the posture of the reference member Rm from an external source. The image processing device 9 may acquire the information Ifg about the posture of the reference member Rm from an image captured by a camera provided outside the crane 11 via the communication device 8, for example.

[0133] Furthermore, in the above-described embodiment, the image processing device 9 acquires, via the communication device 8, information Ifi on the shape of the crane 11, information Ifa on the position of the user, information Ifb on the position of the crane 11, information Ifc on the attitude of the crane 11, information Ifh on the shape of the reference member Rm that moves with the movement of the boom 19, and information Ifg on the attitude of the reference member Rm. However, the image processing device may be configured to acquire at least one of these pieces of information.

[0134] Furthermore, in each of the above-described embodiments, the transmissive display device 2 recognizes a marker located at a specific position, and thereby calculates the information Ifa about the user's position that is required to display a virtual image on the transmissive display 4. However, the transmissive display device may calculate the information Ifa about the user's position from the position of the transmissive display device 2 detected by the GNSS receiver.

[0135] In each of the above-described embodiments, the transmissive display device 2 is a wearable terminal that is worn by the user. However, the transmissive display device may be any display device that allows the user to superimpose a virtual image seen from the user's viewpoint on a display screen that allows the background to be seen from the user's viewpoint. The transmissive display device may be, for example, a head-up display that uses the windshield of the cabin as a display screen to display a virtual image seen from the viewpoint of the user of the transmissive display device 2 seated in the cockpit.

[0136] In each of the above-described embodiments, the transmissive display device 2 calculates the moving direction and the moving amount by the inertial measurement device 5. However, the transmissive display device may calculate the moving direction and the moving amount by Visual Simultaneous Localization And Mapping (Visual SLAM), which generates a map of the surrounding environment and estimates the self-position based on image data. In Visual SLAM, feature points in an image acquired from a stereo camera or the like are grasped. Next, Visual SLAM estimates the self-position and creates a map from the movement of the feature points accompanying the movement of the stereo camera or the like. The transmissive display device can calculate the moving direction and the moving amount by using an image captured by a display device camera that captures the direction of the user's line of sight.

[0137] The above-mentioned embodiment is merely a representative embodiment, and various modifications can be made without departing from the gist of the embodiment. It goes without saying that the present invention can be embodied in various other forms, and the scope of the present invention is defined by the claims, and includes the equivalent meanings of the claims, as well as all modifications within the scope of the claims.

[0138] The disclosures of the specification, drawings, and abstract contained in Japanese application No. 2021-194526, filed on November 30, 2021, are incorporated herein by reference in their entirety. [Industrial Applicability]

[0139] The present invention is applicable to various working machines having a boom. [Explanation of symbols]

[0140] 1 Pilot Assistance System 2 Transparent display device 3 Frame 4. See-through display 5 Inertial Measurement Unit 6. Cameras for display devices 7 Display device control device 8. Communications Equipment 9. Image Processing Device 11. Crane 12 Vehicles 13 wheels 14 Engine 15 Outrigger 16 Crane equipment 17 Swivel table 17a Swing hydraulic motor 18 GNSS receiver for cranes 19. Boom 19a Hydraulic cylinder for extension 19b Boom Camera 19c Jib 20 Main Hook Block 20a Main hook (reference member) 21 Sub-hook block 21a Sub hook 22 Hydraulic cylinder for lifting 23 Main winch 24 Main Wire Rope 25 Sub winch 26 Sub-wire rope 27 Cabin 28 Driving controls 29 Crane equipment operating tools 29a Turning control device 29b Expansion / contraction operating tool 29c Raising operation tool 29d Main operating device 29e Sub-operating tool 30 Crane control device 30a Communication equipment Current position of BLp boom RLt Current position of the reference part RLt Target position of reference part S1 reference plane Rm Reference member Ifa Information about the user's location Ifb Crane location information Ifc Crane position information Ifd Reference Plane Information Ife Information about the target position of the reference part Iff Information on the reference part image Ifg Information about the orientation of the reference part Ifh Information on the shape of the reference part Ifi Crane Geometry Information Im1 First virtual image Im1a 1st boom virtual image Im1b First reference member virtual Im2 2nd virtual image Im3 3rd virtual Im4 4th virtual image Im5 5th virtual image C1 First reference coordinate If1 Information about the first virtual image If2 Information about the second virtual image If3 Information about the third virtual image If4 Information on the 4th virtual image If5 Information on the 5th virtual image

Claims

1. A steering assistance system for assisting in steering a work vehicle having a boom, comprising: A display that is transparent and capable of displaying an image; a generator for generating a virtual image showing a hook moving with the boom; a control unit that displays the virtual image on the display so as to overlap with a scene viewed by the user through the display in a manner that allows the user of the display to recognize the position of the hook; The control unit is Calculating a current position of the hook based on an image captured by a camera that captures an image of the hook from the tip of the boom; displaying the virtual image showing the shape of the hook as seen from the user's viewpoint at a position on the display corresponding to the current position of the hook; Pilot assistance system.

2. A steering assistance system for assisting in steering a work vehicle having a boom, comprising: A display that is transparent and capable of displaying an image; a generator for generating a virtual image showing a hook moving with the boom; a control unit that displays the virtual image on the display so as to overlap with a scene viewed by the user through the display in a manner that allows the user of the display to recognize the position of the hook; the virtual image is an image captured by a camera capturing an image of the hook from the tip of the boom, The control unit displays the virtual image on the display so as to overlap with a reference plane set on the scenery. Pilot assistance system.

3. When the hook is hidden by an obstacle and cannot be seen, the control unit displays the virtual image on the display so as to overlap the obstacle. The piloting assistance system according to claim 1 .

4. the control unit displays the virtual image at a position on the display determined based on information regarding a position of a user of the display and information regarding a position of the hook calculated based on the attitude of the work vehicle and the position of the work vehicle. The piloting assistance system according to claim 1 .

5. The generation unit sets, as the reference plane, a ground surface in the landscape or a horizontal plane including a position of a tip of the boom. The piloting assistance system according to claim 2.

6. The generation unit, When the direction in which the user is viewing is lower than the horizontal direction, the ground surface is set as the reference plane; When the direction in which the user is viewing is above the horizontal direction, the horizontal plane is set as the reference plane; The control unit is When the horizontal plane is set as the reference plane, an image obtained by flipping the image captured by the camera is displayed on the display as the virtual image; When the ground surface is set as the reference surface, the image captured by the camera is displayed on the display as the virtual image without being inverted. A piloting assistance system according to claim 5.

7. the virtual image includes a virtual image showing a trajectory of a tip of the boom corresponding to a posture of the boom; The control unit displays a virtual image showing the trajectory so as to overlap with a reference plane set in the scenery. A piloting assistance system according to claim 1 or 2.

8. The control unit displays the virtual image on the display at a portion that corresponds to the target position of the hook. The piloting assistance system according to claim 1 .

9. the virtual image includes a virtual image showing a perpendicular line passing through a tip of the boom; The control unit displays a virtual image indicating the perpendicular line on the display at a portion corresponding to the tip of the boom. A piloting assistance system according to claim 1 or 2.

10. A work vehicle comprising the steering assistance system according to claim 1 or 2.

Images