Crane operation assistance system and crane with operation assistance system

Abstract

To provide an operation support system of a crane capable of intuitively grasping a positional relation between a track of a hook of the crane and a specific object, and a crane having the operation support system.SOLUTION: An operation support system 1 includes a transmission type display device 2 and an image processing device 9 generating information If about a virtual image Im. The image processing device 9 generates information If1 about a first virtual image Im1 for displaying the first virtual image Im1, which shows an image about a present position Lp of a main hook 20a and a turning direction track Tt and a radial direction track Tr of the main hook 20a. The image processing device 9 determines an image about a specific object in an image P photographed by a boom camera 19b and generates information If2 about a second virtual image Im2 for displaying the second virtual image Im2, which shows an image about the specific object. The transmission type display device 2 displays the first virtual image Im1 and the second virtual image Im2 as seen from a viewpoint of a user.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] The present invention relates to a maneuvering assistance system for a crane and to a crane having a maneuvering assistance system. [Background technology]

[0002] Conventionally, when transporting a load using a mobile crane, which is a work vehicle, the operator of the mobile crane operates multiple operating tools, such as a swing operating tool, a hoisting operating tool, an extension operating tool, and a winch operating tool for lifting or lowering the load, all of which are used to operate the mobile crane's boom. The operator must also operate the multiple operating tools simultaneously while operating the mobile crane so as to prevent the load from coming into contact with structures at the work site. Therefore, mobile cranes are equipped with a boom camera or other device to obtain information about areas that are difficult for the operator to see from the operator's seat. The boom camera can be used to obtain overhead images of the load being transported and surrounding structures, etc., as shown in Patent Document 1, for example.

[0003] The crane described in Patent Document 1 uses a boom camera attached to the tip of the boom to capture images of the work site and images of the load suspended from the hook, and superimposes on the images at least one of a reference line centered on the load, a scale, a directed line indicating the direction of load movement, and the swing trajectory of the hook. The crane operator operates the crane by comprehensively assessing the image information from the boom camera in addition to the operator's own visual information and information from an instructor. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-156538 Summary of the Invention [Problem to be solved by the invention]

[0005] Boom cameras have limited imaging directions and ranges. Furthermore, the imaging conditions for boom camera images differ from those for images viewed by the operator with the naked eye. Therefore, crane operators must be highly skilled to use the boom camera to grasp the positional relationship between the hook's rotation trajectory and the person, and then operate the crane while checking the surrounding conditions with the naked eye. Furthermore, it may be difficult to see people on the ground using images from a boom camera located at the tip of the boom. Therefore, crane operators cannot obtain sufficient information needed to move a load to a desired location using images from the boom camera.

[0006] An object of the present invention is to provide a crane operation assistance system that enables intuitive understanding of the positional relationship between the trajectory of the crane hook and a specific object, and a crane equipped with the operation assistance system. [Means for solving the problem]

[0007] The present inventors have studied the configuration of a crane operation assistance system that allows a reference member to be moved in a desired direction at a desired speed by intuitive operation from any position, and a crane equipped with the operation assistance system. As a result of their intensive study, the present inventors have come up with the following configuration.

[0008] The crane operation assistance system of the present invention comprises a transmissive display device that displays a virtual image seen from a user's perspective on a display screen that allows the background to be seen through the display screen; a position information acquisition device that acquires position information of a specific object within a range that includes at least a portion of the movable range of a hook of a crane that has a telescopic boom that can be raised and lowered on a swivel base; an image processing device that generates information related to the virtual image to be displayed on the transmissive display device; and a communication device that transmits and receives information between the position information acquisition device and the image processing device, between the crane's control device and the image processing device, and between the transmissive display device and the image processing device.

[0009] The image processing device calculates the current position of the hook and the swing direction trajectory and radial trajectory of the hook relative to the boom swing center based on information about the crane position and information about the crane attitude acquired via the communication device. The image processing device generates information about a first virtual image for displaying on the display screen a first virtual image showing an image of the current position of the hook and the swing direction trajectory and radial trajectory of the hook as viewed from the user's viewpoint based on information about the user's position acquired via the communication device. The image processing device detects position information about a specific object in position information acquired by the position information acquisition device via the communication device, calculates the position of the specific object based on information about the conditions when the position information acquisition device acquired the position information and information about the user's position, and generates information about a second virtual image for displaying on the display screen a second virtual image showing an image of the specific object as viewed from the user's viewpoint.

[0010] The transmissive display device acquires information regarding the first virtual image and information regarding the second virtual image from the image processing device via the communication device, and displays the first virtual image and the second virtual image as viewed from the user's viewpoint on the display screen.

[0011] The above-described operation assistance system displays a first virtual image, which shows the current position of the hook, its rotational trajectory, and its radial trajectory as seen from the user's viewpoint, and a second virtual image, which shows the position of a specific object, on the background of the display screen viewed by the user of the transmissive display device. In other words, the operation assistance system displays graphical information, textual information, etc., superimposed on the specific object in the user's field of view, allowing the operator, who is the user of the transmissive display device, to grasp the hook trajectory and the position of the specific object, such as a person or vehicle, relative to the hook trajectory. A crane operator using a transmissive display device recognizes the current position of the hook, the hook's rotational trajectory or radial trajectory, and the presence of people, vehicles, etc., located near the rotational trajectory and radial trajectory, in the actual scene seen with the naked eye. The crane operator also recognizes the presence of people, vehicles, etc., even in blind spots caused by cabin pillars, etc. Furthermore, a slinger, etc., using a transmissive display device, recognizes the position of the hook's rotational trajectory and radial trajectory in the actual scene seen with the naked eye. This allows users to intuitively grasp the positional relationship between the trajectory of the crane hook and a specific object.

[0012] The crane operation assistance system of the present invention preferably includes the following configuration: the image processing device generates information related to a third virtual image for displaying on the display screen a third virtual image showing a highlighted image of the specific object located within a predetermined range based on the swing direction trajectory and the radial direction trajectory of the hook.

[0013] In the above configuration, the operation assistance system displays a third virtual image seen from the user's perspective on the display screen of the transmissive display device so that the third virtual image is further superimposed on the background of the display screen. In other words, the operation assistance system highlights people, vehicles, etc. located within a predetermined range based on the hook's rotational trajectory and radial trajectory. The crane operator, who is the user of the transmissive display device, can recognize the presence of people, vehicles, etc. that may come into contact with the load, hook, etc. during the crane's load transport in the actual scene seen with the naked eye, without having to individually check the current position of the hook, the positional relationship between the hook's rotational trajectory and radial trajectory, and the people, vehicles, etc. This allows the operator to intuitively grasp the positional relationship between the crane's hook trajectory and a specific object.

[0014] Furthermore, it is preferable that the crane operation assistance system includes the following configuration: when the user is located within a predetermined range based on the swing direction trajectory and radial direction trajectory of the hook, the image processing device generates information related to a fourth virtual image that shows the user that the user is located near the swing direction trajectory and radial direction trajectory. When the transmission-type display device acquires information related to the fourth virtual image from the image processing device via the communication device, it displays the fourth virtual image seen from the user's viewpoint on the display screen.

[0015] In the above configuration, when a user of the transmission-type display device is located within a predetermined range based on the swing direction trajectory and radial direction trajectory of the hook, the operation assistance system displays a fourth virtual image seen from the user's viewpoint on the display screen so that the fourth virtual image is superimposed on the background viewed by the user of the transmission-type display device. By viewing the fourth virtual image, the user of the transmission-type display device recognizes that they are located near the swing direction trajectory or radial direction trajectory of the hook. This allows the user to intuitively grasp the positional relationship between the trajectory of the crane hook and a specific object.

[0016] Furthermore, it is preferable that the crane operation assistance system according to one embodiment of the present invention includes the following configuration. When the specific object is located outside the range of the background overlapping with the display screen and within a predetermined range based on the swing direction trajectory and radial direction trajectory of the hook, as viewed from the user's viewpoint, the image processing device generates a fifth virtual image showing an image displaying the direction of the specific object located within the predetermined range based on the swing direction trajectory and radial direction trajectory of the hook. The transmission-type display device receives information about the fifth virtual image from the image processing device via the communication device and displays the fifth virtual image seen from the user's viewpoint on the display screen.

[0017] In the above configuration, when the specific object is located outside the range of the background overlapping the display screen from the user's viewpoint of the transmissive display device but within a predetermined range based on the swing trajectory and radial trajectory of the hook, the operation assistance system displays a fifth virtual image as seen from the user's viewpoint on the display screen so that it is superimposed on the background seen by the user's naked eye. By viewing the fifth virtual image, the user (i.e., the crane operator) recognizes the presence of a person, vehicle, etc. located near the swing trajectory or radial trajectory of the hook outside the range of the actual scene seen by the naked eye. Furthermore, by directing his or her gaze in the direction indicated by the fifth virtual image, the crane operator can visually confirm with the naked eye the person, vehicle, etc. located near the swing trajectory or radial trajectory of the hook. This allows the operator to intuitively grasp the positional relationship between the crane hook's trajectory and the specific object.

[0018] Furthermore, it is preferable that the crane operation assistance system according to one embodiment of the present invention includes the following configuration: The position information acquisition device is a LIDAR that acquires the distance and direction to the specific object by using a laser.

[0019] In the above configuration, the piloting assistance system acquires the distance and direction of a specific object, such as a person or a vehicle, using LIDAR (light detection and ranging). Therefore, even if the distance from the LIDAR to the specific object varies, the piloting assistance system detects the difference in distance. For example, the piloting assistance system can easily acquire the distance and direction between a person on the ground and a person on a structure using LIDAR. This allows the piloting assistance system to intuitively grasp the positional relationship between the trajectory of the crane hook and the specific object.

[0020] Furthermore, a crane operation assistance system according to one embodiment of the present invention preferably includes the following configuration: The position information acquisition device is an imaging device that captures an image of a range including at least a portion of the movable range of a hook of a crane having a telescopic boom that can be raised and lowered on a swivel base. The image processing device acquires information related to the angle of view and imaging position of the imaging device and the image captured by the imaging device via the communication device. The image processing device detects an image of a specific object in the image captured by the imaging device, calculates the position of the specific object based on the information related to the angle of view and imaging position of the imaging device and information related to the position of the user, and generates information related to a second virtual image for displaying on the display screen a second virtual image that shows the image of the specific object as seen from the user's viewpoint.

[0021] In the above configuration, the operation assistance system detects the current positions of the hook, people, vehicles, etc. based on an image of a range including at least a portion of the movable range of the crane hook captured by an imaging device. The current positions of the hook, people, vehicles, etc. can be detected in real time even if they are moving. This allows users to intuitively grasp the positional relationship between the trajectory of the crane hook and a specific object.

[0022] Furthermore, it is preferable that the crane operation assistance system according to one embodiment of the present invention includes the following configuration: The imaging device is a stereo camera.

[0023] In the above configuration, the operation assistance system recognizes the position of a specific object, such as a person or vehicle, based on the parallax of multiple cameras. Therefore, even if the distance from the stereo camera to the specific object varies, the operation assistance system detects the difference in distance. For example, the operation assistance system recognizes that a person on the ground and a person on a structure are located at different distances, and reflects this in the display mode of the virtual image. This allows users to intuitively grasp the positional relationship between the trajectory of the crane hook and the specific object.

[0024] According to another aspect, the crane operation assistance system of the present invention preferably includes the following configuration: the image processing device has training data for images of the specific object, and the image processing device detects images of the specific object from images captured by the imaging device by machine learning based on the training data.

[0025] In the above configuration, the operation assistance system identifies a specific object from an image captured by an imaging device using the results of machine learning based on training data on a specific object such as a person or vehicle. Therefore, the accuracy of identifying the specific object improves each time the operation assistance system identifies the specific object from the image. Therefore, the operation assistance system identifies the specific object without being affected by the image capturing conditions of the imaging device. This allows the positional relationship between the trajectory of the crane hook and the specific object to be accurately determined.

[0026] A crane according to one embodiment of the present invention includes any one of the above-described operator assistance systems.

[0027] In a crane equipped with one of the above-described operation assistance systems, when a user wearing the transmissive display device is the crane operator, the swing trajectory or radial trajectory of the hook and specific objects such as people and vehicles are displayed as virtual images superimposed on the actual scenery seen with the naked eye from the crane's operator's seat. The crane operator can visually recognize the swing trajectory or radial trajectory and the positions of the specific objects within the scenery seen with the naked eye from the crane's operator's seat through the virtual images. This allows the operator to intuitively grasp the positional relationship between the crane hook's trajectory and the specific objects.

[0028] The terminology used in this specification is used for the purpose of defining particular embodiments only, and is not intended to limit the invention.

[0029] As used herein, the use of "including," "comprising," or "having" and variations thereof identify the presence of stated features, steps, operations, elements, components, and / or equivalents thereof, but may include one or more of the steps, operations, elements, components, and / or groups thereof.

[0030] As used herein, the terms "attached," "connected," "coupled," and / or their equivalents are used broadly to encompass both "direct and indirect" attachments, connections, and couplings. Furthermore, "connected" and "coupled" are not limited to physical or mechanical connections or couplings, but can also include direct or indirect electrical connections or couplings.

[0031] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0032] Terms defined in commonly used dictionaries should be construed to have a meaning consistent with the relevant art and meaning in the context of this disclosure, and should not be construed in an idealized or overly formal sense unless expressly defined herein.

[0033] Accordingly, the following disclosure is to be considered as illustrative of the present invention and is not intended to limit the invention to the specific embodiments illustrated by the following drawings or description.

[0034] [Virtual image] In this specification, a virtual image is an image or video other than the background viewed by a user in a transmissive display device, and refers to a line, figure, image, video, 3D image, or 3D video displayed by the transmissive display device. In other words, a virtual image is an image that can be displayed without discomfort even when superimposed on the background viewed by the user by converting an image of an object that does not actually exist into an image of a shape seen from the user's viewpoint. A virtual image is displayed superimposed on the background viewed by the user in a transmissive display device.

[0035] [perspective] In this specification, the term "viewpoint" refers to a standpoint from which an object is viewed. The term "user's viewpoint" refers to the user's position relative to an object when the user is viewing the object.

[0036] [Information about virtual images] In this specification, the information about the virtual image means information necessary for displaying the virtual image on the display screen of the transmissive display device as an image on an arbitrary reference plane seen from the viewpoint of the user of the transmissive display device, including the coordinates of the virtual image, the display state, etc.

[0037] [Communication Device] In this specification, the communication device refers to a device that transmits and receives information between the image processing device and the display device control device, between the image processing device and the input device, and between the image processing device and the crane control device. The communication device may be configured integrally with the display device control device and the image processing device.

[0038] [Angle of view] In this specification, the field angle refers to the angle indicating the range actually captured by the imaging element. Specifically, the field angle is the diagonal field angle indicating the angle formed by the optical center of the optical lens and two points forming diagonals of the light-receiving surface of the imaging element. In other words, the diagonal field angle is determined by the focal length of the optical lens and the length of the diagonal of the light-receiving surface. For a given size of the light-receiving surface, the diagonal field angle becomes smaller as the focal length of the optical lens becomes longer, and becomes smaller as the diagonal of the light-receiving surface becomes shorter. [Effects of the Invention]

[0039] According to one embodiment of the present invention, the operation assistance system and the crane equipped with the operation assistance system can be operated by intuitive operation from any position. [Brief explanation of the drawings]

[0040] [Figure 1] FIG. 1 is a perspective view showing a transmission-type display device in a crane operation assistance system according to one embodiment of the present invention worn by a user. [Figure 2] FIG. 2 is a perspective view of a transmission type display device in a crane operation assistance system according to one embodiment of the present invention. [Figure 3] FIG. 3 shows a control block diagram of a crane operation assistance system according to one embodiment of the present invention. [Figure 4] FIG. 4 is a schematic diagram showing a process of discriminating between people and vehicles from an image by machine learning in a crane operation assistance system according to one embodiment of the present invention. [Figure 5] FIG. 5 shows a side view of a crane to which a crane operation assistance system according to one embodiment of the present invention is connected. [Figure 6] FIG. 6 shows a control block diagram of a crane to which a crane operation assistance system according to one embodiment of the present invention is connected. [Figure 7]Figure 7 shows a block diagram of a crane operation assistance system according to one embodiment of the present invention when generating information necessary for displaying the first virtual image, the second virtual image, the third virtual image, the fourth virtual image, and the fifth virtual image. [Figure 8] FIG. 8 shows an image captured by a boom camera in a crane operation assistance system according to one embodiment of the present invention. [Figure 9] FIG. 9 is a schematic diagram showing a state in which a user of a crane operation assistance system according to one embodiment of the present invention is viewing a first virtual image, a second virtual image, and a third virtual image on a transmission-type display device. [Figure 10] FIG. 10 is a schematic diagram showing a state in which a user of the crane operation assistance system according to one embodiment of the present invention is viewing a fourth virtual image on the transmission type display device. [Figure 11] FIG. 11 is a schematic diagram showing a state in which a user of the crane operation assistance system according to one embodiment of the present invention is viewing a fifth virtual image on the transmission display device. DETAILED DESCRIPTION OF THE INVENTION

[0041] A crane operation assistance system according to one embodiment of the present invention will be described below with reference to FIGS. 1 to 4. FIG. 1 is a perspective view of a user wearing a transmissive display device 2 in a crane operation assistance system 1 for a crane 11 according to one embodiment of the present invention. FIG. 2 is a perspective view of the transmissive display device 2 in the crane operation assistance system 1 for the crane 11. FIG. 3 is a control block diagram of the crane operation assistance system 1 for the crane 11. FIG. 4 is a schematic diagram of a process for identifying a person H and a vehicle V from an image P using machine learning in the crane operation assistance system 1 for the crane 11. Note that in this embodiment, a rough terrain crane (hereinafter simply referred to as "crane") will be described as the crane. However, the crane may be any work vehicle equipped with a boom that can be raised or lowered via a swivel, such as an all-terrain crane, a truck crane, or an aerial work platform. In the following description, the "user" refers to a person wearing the transmissive display device 2.

[0042] As shown in FIGS. 1 and 2, a maneuvering assistance system 1 for a crane 11 (see FIG. 5), which is a work vehicle, is a system that uses mixed reality to assist in the operation of the crane 11. The maneuvering assistance system 1 includes a transmissive display device 2, a boom camera 19b, which is a position information acquisition device, a communication device 8, and an image processing device 9. The maneuvering assistance system 1 displays, on the transmissive display device 2, a first virtual image Im1 that indicates the current position Lp of a main hook 20a (see FIG. 5), which is a reference member of the crane 11, and the turning direction trajectory Tt and radial direction trajectory Tr of the main hook 20a, and a second virtual image Im2 that indicates the positions of a person H and a vehicle V, which are specific objects. Note that the specific objects are not limited to the person H and the vehicle V.

[0043] The transmissive display device 2 is a display device that allows a user to view a background Ls (see FIG. 9 ) of the transmissive display 4 of the transmissive display device 2 through the transmissive display 4. The transmissive display device 2 is a wearable terminal that is worn on the body of the user. In this embodiment, the transmissive display device 2 is described as a glasses-type wearable terminal. The transmissive display device 2 is worn by the operator of the crane 11, who is the user. The transmissive display device 2 has a frame 3, a transmissive display 4, a display device GNSS receiver 5, a display device camera 6, and a display device control device 7.

[0044] The frame 3 supports a transmissive display 4, a display device GNSS receiver 5, a display device camera 6, and a display device control device 7. The frame 3 is configured as a glasses-type frame. The frame 3 is worn on the user's head. The frame 3 is supported by the user's nose and both ears.

[0045] The transmissive display 4 is a display with a transmittance of approximately 50% or more. The transmissive display 4 is composed of a liquid crystal display, an organic EL display, an inorganic EL display, or the like that does not have a backlight, a polarizing filter, or a color filter. In other words, the transmissive display 4 does not have components such as a backlight or a filter that cover the back surface of the display screen that faces the user. Therefore, the transmissive display 4 transmits a background Ls that overlaps with the display screen as seen from the user's viewpoint. Furthermore, the transmissive display 4 can display virtual images including a first virtual image Im1 and a second virtual image Im2, which are images generated by the function of the transmissive display 4, on the display screen.

[0046] The transmissive display 4 transmits light incident from the back surface in a portion where the first virtual image Im1 and the second virtual image Im2 are not displayed. The transmissive display 4 also blocks at least a portion of the light incident from the back surface in a portion where the first virtual image Im1 and the second virtual image Im2 are displayed. The transmissive display 4 then displays the first virtual image Im1 and the second virtual image Im2 in the light-blocked portion. At this time, the transmissive display 4 displays the first virtual image Im1 and the second virtual image Im2 as seen from the user's viewpoint. This allows the user to visually recognize the first virtual image Im1 and the second virtual image Im2 as seen from the user's viewpoint superimposed on the background Ls.

[0047] The transmissive display 4 is provided on the frame 3. The transmissive display 4 is positioned so as to cover the visual field of the right eye and the visual field of the left eye of a user wearing the frame 3. The transmissive display 4 is configured to be able to display a first virtual image Im1 for the right eye and a second virtual image Im2 for the left eye as parallax images on the transmissive display 4. This allows the transmissive display 4 to display a virtual image including the three-dimensional first virtual image Im1 and the second virtual image Im2 in almost the entire visual field of the user. Note that a technique for displaying the first virtual image Im1 and the second virtual image Im2 as seen from the user's viewpoint within the visual field of the user in the transmissive display device 2 is well known.

[0048] The GNSS receiver 5 for the display device is a receiver that constitutes a Global Navigation Satellite System. The GNSS receiver 5 for the display device receives ranging radio waves from satellites and detects the absolute coordinates of the GNSS receiver 5 for the display device, such as latitude, longitude, altitude, and direction. The GNSS receiver 5 for the display device is provided on the frame 3 worn by the user. The GNSS receiver 5 for the display device detects information Ifa about the position of the user (pilot) (see FIG. 3 ). The information Ifa about the user's position includes the latitude, longitude, altitude, and direction of the transmissive display device 2. In other words, the information Ifa about the user's position includes information about the position of the transmissive display device 2 and the direction of the background Ls that the transmissive display 4 transmits. In this embodiment, the GNSS receiver 5 for the display device is provided on the frame 3, but it may be located in any position where it can detect the information Ifa about the user's position. For example, the GNSS receiver 5 for the display device may be worn on the body of the user wearing the transmissive display device 2.

[0049] The display device camera 6 captures an image in the direction of the line of sight of a user wearing the transmissive display device 2. The display device camera 6 is provided on the frame 3. The display device camera 6 is configured to capture an image in a range approximately equal to the user's visual field from a position approximately equal to the user's viewpoint. In this embodiment, the display device camera 6 is provided on the frame 3, but it may be located in any position that allows it to capture an image of the user's visual field from the user's viewpoint.

[0050] As shown in FIG. 3 , the display device control device 7 is a control device that displays the first virtual image Im1 and the second virtual image Im2 on the transmissive display 4. The display device control device 7 actually includes a CPU, a ROM, a RAM, a HDD, and the like connected via a bus. Alternatively, the display device control device 7 may be configured with a one-chip LSI or the like. The display device control device 7 is connected to the transmissive display 4, the display device GNSS receiver 5, and the display device camera 6. The display device control device 7 is connected to an image processing device 9 via a communication device 8. The display device control device 7 has information Ifd about a reference plane S1 on which the first virtual image Im1 and the second virtual image Im2 are displayed. In this embodiment, the reference plane S1 is set on the ground surface. The information Ifd about the reference plane S1 includes the position and inclination of the reference plane S1, which serves as a reference for the position at which the first virtual image Im1 and the second virtual image Im2 are displayed as viewed from the user's viewpoint. The reference plane S1 can be set arbitrarily by the user.

[0051] The display device control device 7 can acquire information Ifa about the user's position from the display device GNSS receiver 5. The display device control device 7 can acquire images from the display device camera 6. The display device control device 7 can also acquire information If1 about the first virtual image Im1 and information If2 about the second virtual image Im2 from the image processing device 9. The display device control device 7 can generate the first virtual image Im1 and the second virtual image Im2 based on the acquired information If1 about the first virtual image Im1 and information If2 about the second virtual image Im2. The display device control device 7 can also display the first virtual image Im1 and the second virtual image Im2 on the transmissive display 4.

[0052] The transmissive display device 2 configured as described above can display the first virtual image Im1 and the second virtual image Im2 on the display screen of the transmissive display 4 (hereinafter simply referred to as the "transmissive display 4") while being worn on the user's head. The transmissive display device 2 can also acquire, by the display device GNSS receiver 5, information Ifa about the user's position, which is necessary for displaying the first virtual image Im1 and the second virtual image Im2 on the transmissive display 4. The information Ifa about the user's position includes the latitude, longitude, altitude, and vehicle orientation of the user of the transmissive display device 2. The transmissive display device 2 can display the first virtual image Im1 and the second virtual image Im2 as seen from the user's viewpoint on the transmissive display 4 through which the background Ls is transmitted, based on the information If1 about the first virtual image Im1 and the information If2 about the second virtual image Im2 and the information Ifa about the user's position. This allows the user to Through the transmissive display device 2, an image in which the first virtual image Im1 and the second virtual image Im2 are superimposed on the background Ls of the transmissive display 4 can be visually recognized.

[0053] The communication device 8 is a device that transmits and receives information between the transmissive display device 2, the image processing device 9, and the crane control device 30 of the crane 11. The communication device 8 is provided in each of the display device control device 7 of the transmissive display device 2 and the image processing device 9. The communication device 8 transmits and receives information between the crane control device 30 and the image processing device 9. The communication device 8 also transmits and receives information between the display device control device 7 and the image processing device 9.

[0054] The image processing device 9 is a control device that generates information If1 related to the first virtual image Im1 and information If2 related to the second virtual image Im2 to be displayed on the transmissive display device 2. The image processing device 9 is essentially a CPU, ROM, RAM, HDD, etc., connected via a bus. Alternatively, the image processing device 9 may be configured with a one-chip LSI, etc. In this embodiment, the image processing device 9 is provided in the transmissive display device 2. The image processing device 9 stores various programs and data for controlling the operation of the transmissive display device 2, etc., and for processing image data. The image processing device 9 may be integrated with the display device control device 7 or the crane control device 30. The image processing device 9 is a position information acquisition device and includes a discrimination unit 9a that discriminates people H and vehicles V (see FIG. 9 ) included in an image P captured by the boom camera 19b of the crane 11, which is also an imaging device. The image P includes people H and vehicles V located on the ground surface, etc. The people H and vehicles V include, for example, people such as slingers and civil engineering workers located on the ground surface within the imaging range of the boom camera 19b, and vehicles such as trucks and construction machinery.

[0055] The image processing device 9 is connected to the display device control device 7 of the transmissive display device 2 via the communication device 8. The image processing device 9 acquires information Ifa about the user's position detected by the display device GNSS receiver 5 from the display device control device 7 via the communication device 8. The image processing device 9 also acquires information Ifd about the reference plane S1 from the display device control device 7 via the communication device 8. The image processing device 9 has information about the boom camera 19b, such as the angle of view and focal length of the boom camera 19b.

[0056] The image processing device 9 acquires information Ifb about the position of the crane 11 from the crane control device 30 via the communication device 8. The information Ifb about the position of the crane 11 includes the latitude, longitude, altitude, and vehicle direction of the location of the rotation center of the crane 11.

[0057] The image processing device 9 acquires information Ifc about the attitude of the crane 11 from the crane control device 30 via the communication device 8. The information Ifc about the attitude of the crane 11 includes the swing angle of the crane 11, the boom hoisting angle, the boom length, the payout length of the main wire rope 24, the payout length of the sub-wire rope 26, and the operation direction and operation amount of each operating tool of the crane 11. The information Ifc about the attitude of the crane 11 also includes information about the current position Lp of the main hook 20a, which is the reference member of the crane 11, and information about the angle of view and position of the boom camera 19b, which are conditions when the boom camera 19b, which is a position information acquisition device, acquires an image P including position information.

[0058] The image processing device 9 continuously acquires current images P captured by the boom camera 19b at each unit time from the crane control device 30 via the communication device 8. The images P include images of a person H and a vehicle V, which are specific objects.

[0059] The image processing device 9 generates information If1 about the first virtual image Im1 for displaying the first virtual image Im1 on the reference plane S1 on the transmissive display 4 as the first virtual image Im1 seen from the viewpoint of the user of the transmissive display device 2, based on the information Ifa about the user's position, the acquired information Ifb about the position of the crane 11, the information Ifc about the attitude of the crane 11, and the information Ifd about the reference plane S1. The image processing device 9 transmits the generated information If1 about the first virtual image Im1 to the display device control device 7.

[0060] As shown in FIG. 4, the discrimination unit 9a of the image processing device 9 discriminates people H and vehicles V from the acquired image P using deep learning, which is a type of machine learning. Furthermore, the image processing device 9 generates information Ife about the positions of the people H and vehicles V discriminated by the discrimination unit 9a. The information Ife about the positions of the people H and vehicles V includes the latitude, longitude, altitude, and vehicle orientation of the people H and vehicles V, respectively. The discrimination unit 9a has a human and vehicle image discrimination model M in which a known parameter-trained model has been trained using a training dataset, which is training data including a large number of images of people H and vehicles V captured by the boom camera 19b of the crane 11. The discrimination unit 9a discriminates people H and vehicles V from the image P, which is input data, using deep learning with the human and vehicle image discrimination model M, and generates information Ife about the positions of the people H and vehicles V as output data.

[0061] The image processing device 9 generates information If2 about the second virtual image Im2 for displaying the second virtual image Im2 on the reference plane S1 on the transmissive display 4 as a second virtual image Im2 seen from the viewpoint of the user of the transmissive display device 2, based on the information about the boom camera 19b, the information Ifa about the position of the user, the information Ifb about the position of the crane 11, the information Ifc about the attitude of the crane 11, the information Ifd about the reference plane S1, and the information Ife about the positions of the person H and the vehicle V. The image processing device 9 transmits the generated information If2 about the second virtual image Im2 to the display device control device 7.

[0062] The display device control device 7 of the transmissive display device 2 acquires information If1 about the first virtual image Im1 and information If2 about the second virtual image Im2 via the communication device 8. The display device control device 7 displays the first virtual image Im1, the second virtual image Im2, and the background Ls superimposed on a reference plane S1 on the transmissive display 4 that is viewed by the user.

[0063] The image processing device 9 of the operation assistance system 1 configured as described above acquires information Ifa about the position of the user, information Ifb about the position of the crane 11, information Ifc about the attitude of the crane 11, and information Ifd about the reference plane S1. The image processing device 9 also generates information Ife about the positions of the person H and the vehicle V from the image P by deep learning. The image processing device 9 generates information If1 about a first virtual image Im1 based on the acquired information. The image processing device 9 also generates information If2 about a second virtual image Im2 based on the acquired information and the generated information. The display device control device 7 of the operation assistance system 1 displays the first virtual image Im1 from the information If1 about the first virtual image Im1, and displays the second virtual image Im2 from the information If2 about the second virtual image Im2.

[0064] A crane 11 according to a first embodiment of the present invention will be described below with reference to Figures 5 and 6. Figure 5 shows the overall configuration of the crane 11 to which the operation assistance system 1 is connected. Figure 6 shows a control block diagram of the crane 11 to which the operation assistance system 1 is connected.

[0065] 5 and 6, the crane 11 is a mobile crane that can be moved to any location. The crane 11 includes a crane vehicle 12, a crane device 16, and a crane control device 30 (see FIG. 7).

[0066] The crane vehicle 12 is a vehicle that transports a crane device 16. The crane vehicle 12 has a plurality of wheels 13. The crane vehicle 12 travels using an engine 14 as a power source. The crane vehicle 12 has outriggers 15. The outriggers 15 have hydraulically extendable extension beams on both sides of the width of the crane vehicle 12, and hydraulic jack cylinders that can be extended in a direction perpendicular to the ground.

[0067] The crane apparatus 16 is a work device that lifts a load W using a wire rope. The crane apparatus 16 includes a swivel base 17, a boom 19, a main hook block 20, a sub-hook block 21, a hoisting hydraulic cylinder 22, a main winch 23, a main wire rope 24, a sub-winch 25, a sub-wire rope 26, a cabin 27, etc.

[0068] The swivel base 17 is a rotating device that allows the crane apparatus 16 to rotate. The swivel base 17 is mounted on the frame of the crane vehicle 12 via an annular bearing. The swivel base 17 has a hydraulic swivel motor 17a, which serves as an actuator. The swivel base 17 is configured to be swivelable in one direction and the other direction by the swivel motor 17a.

[0069] The crane GNSS receiver 18 (see Figure 6) is a receiver that constitutes the Global Navigation Satellite System. The crane GNSS receiver 18 receives ranging radio waves from satellites and detects the absolute coordinates of the crane GNSS receiver 18, such as latitude, longitude, altitude, and direction. The crane GNSS receiver 18 is provided on the swivel base 17. The crane GNSS receiver 18 detects information Ifb about the position of the crane 11.

[0070] The boom 19 is a telescopic boom that supports a main wire rope 24 and a sub-wire rope 26. The boom 19 is composed of multiple boom members. The base end of the base boom member of the boom 19 is swingably mounted approximately at the center of the swivel base 17. The boom 19 has telescopic hydraulic cylinders 19a (see Figure 6) that act as actuators for extending and retracting each boom member, and hoisting hydraulic cylinders 22 that raise and lower the boom members. The boom 19 is extended and lowered in the axial direction by the telescopic hydraulic cylinders 19a. The boom 19 also has a boom camera 19b that is an imaging device and a jib 19c that is an extension member. The boom camera 19b is located at the tip of the boom 19. The boom camera 19b captures images of the area vertically downward from the tip of the boom 19 regardless of the boom hoisting angle. The vertically downward image P taken from the tip of the boom 19 captures an area that includes at least a portion of the movable range of the main hook 20a from above, where there are fewer obstacles than on the ground, and therefore allows the operator to obtain more information about the suspended load and its surroundings, which would be difficult for the operator to grasp due to obstacles, etc.

[0071] The main hook block 20 and the sub-hook block 21 are components that suspend the cargo W. The main hook block 20 has a plurality of hook sheaves around which the main wire rope 24 is wound, and a main hook 20a that suspends the cargo W. The sub-hook block 21 has a sub-hook 21a that suspends the cargo W.

[0072] The hoisting hydraulic cylinder 22 is an actuator that raises and lowers the boom 19 and maintains the posture of the boom 19. The end of the cylinder portion of the hoisting hydraulic cylinder 22 is connected to the swivel base 17 so that it can be raised and lowered freely. The end of the rod portion of the hoisting hydraulic cylinder 22 is connected to the base boom member of the boom 19 so that it can swing freely.

[0073] The main winch 23 is a device that reels in (winds up) and reels out (winds down) the main wire rope 24. The sub winch 25 is a device that reels in (winds up) and reels out (winds down) the sub wire rope 26.

[0074] The cabin 27 is a housing that covers the operator's seat. The cabin 27 is mounted on the swivel base 17. A not-shown operator's seat is provided in the cabin 27. The operator's seat is provided with a travel operating tool 28 for operating the crane vehicle 12 to travel, a crane device operating tool 29 for the crane device 16, and the like (see FIG. 6).

[0075] As shown in FIG. 6, the crane control device 30 controls each actuator of the crane 11. The crane control device 30 is provided inside the cabin 27. The crane control device 30 essentially comprises a CPU, ROM, RAM, HDD, etc. connected via a bus. Alternatively, the crane control device 30 may be composed of a one-chip LSI or the like. The crane control device 30 stores various programs and data for controlling the operation of each actuator, switching valve, etc., and for processing image data. The crane control device 30 has a crane communication device 30a. The crane communication device 30a transmits and receives information to and from the image processing device 9 of the operation assistance system 1.

[0076] The crane control device 30 is connected to the crane GNSS receiver 18. The crane control device 30 acquires information Ifb about the position of the crane 11. The crane control device 30 is connected to the boom camera 19b. The crane control device 30 continuously acquires current images P captured by the boom camera 19b per unit time via the communication device 8. The crane control device 30 is connected to the traveling operation device 28 and the crane operation device 29. The crane control device 30 generates a control signal for the crane apparatus 16 based on an operation signal generated by operating the crane operation device 29. The crane control device 30 transmits the generated control signal to each actuator.

[0077] The crane control device 30 is connected to the image processing device 9 of the operation assistance system 1 via the crane communication device 30a. The crane control device 30 transmits information Ifb about the position of the crane 11 and information Ifc about the attitude of the crane 11 to the image processing device 9.

[0078] The crane 11 configured in this manner can move the crane vehicle 12 to any position by operating the traveling operation device 28. Also, the crane 11 can transport the load W to any position by rotating, raising, lowering, and extending the boom 19 by operating the crane device operation device 29. Also, the crane 11 can lift or lower the load W with the main winch 23 or the like by operating the crane device operation device 29.

[0079] Next, using FIGS. 7 to 11, we will explain the operation assistance provided by the operation assistance system 1. FIG. 7 shows a block diagram of the operation assistance system 1 for the crane 11 when generating information necessary for displaying the first virtual image Im1, the second virtual image Im2, the third virtual image Im3, the fourth virtual image Im4, and the fifth virtual image Im5. FIG. 8 shows an image captured by the boom camera 19b in the operation assistance system 1 for the crane 11. FIG. 9 shows a schematic diagram of the operation assistance system 1 for the crane 11, in which a user is viewing the first virtual image Im1, the second virtual image Im2, and the third virtual image Im3 on the transmission-type display device 2. FIG. 10 shows a schematic diagram of the operation assistance system 1 for the crane 11, in which a user is viewing the fourth virtual image Im4 on the transmission-type display device 2. FIG. 11 shows a schematic diagram of the operation assistance system 1 for the crane 11, in which a user is viewing the fifth virtual image Im5 on the transmission-type display device 2.

[0080] In this embodiment, the reference member that serves as the reference for the trajectory is the main hook 20a of the crane 11. The reference member may also be the cargo W transported by the crane 11. The operator of the crane 11 is the user of the transparent display device 2. The user operates the crane 11 with the transparent display device 2 attached to his head. Furthermore, a slinger positioned near the planned installation position of the cargo W suspended from the main hook 20a attaches the transparent display device 2 to his head. In other words, the slinger is the user of the transparent display device 2.

[0081] In this embodiment, the driving assistance system 1 uses deep learning to distinguish people H and vehicles V from an image P using an image processing device 9. When the image processing device 9 acquires an image P captured by a boom camera 19b, it uses deep learning to distinguish people H and vehicles V from the acquired image P using a person and vehicle image discrimination model M. The image processing device 9 generates information Ife about the positions of the distinguished people H and vehicles V.

[0082] <Display of current position Lp of main hook 20a, turning direction trajectory Tt, and radial direction trajectory Tr> 7 to 9, the operation assistance system 1 displays the current position Lp of the main hook 20a, which is a reference member, and the rotation direction trajectory Tt or radial direction trajectory Tr of the main hook 20a as a first virtual image Im1 on the transmission-type display device 2. The first virtual image Im1 includes at least one of the rotation direction trajectory Tt, which is the trajectory that the main hook 20a passes through when the boom 19 rotates, or the radial direction trajectory Tr, which is the trajectory that the main hook 20a passes through when the boom 19 is raised or lowered or extended, based on the current position Lp of the main hook 20a. Note that the first virtual image Im1 may display a trajectory arbitrarily set by the operator, such as a straight line trajectory from the current position Lp of the main hook 20a to a target position.

[0083] 7, the image processing device 9 of the operation assistance system 1 acquires information Ifa about the position of the user, information Ifd about the reference plane S1, and a current image P from the display device control device 7 via the communication device 8. The image processing device 9 also acquires information Ifb about the position of the crane 11 and information Ifc about the attitude of the crane 11 from the crane control device 30. The image processing device 9 also continuously acquires the current image P captured by the boom camera 19b from the crane control device 30 via the communication device 8 at unit time intervals.

[0084] The image processing device 9 calculates first reference coordinates C1, which are a set of absolute coordinates for displaying the current position Lp of the main hook 20a and the rotation direction trajectory Tt and radial direction trajectory Tr of the main hook 20a based on the current position Lp of the main hook 20a, based on the acquired information Ifb about the position of the crane 11, information Ifc about the attitude of the crane 11, and information Ifd about the reference plane S1. The image processing device 9 generates information If1 about a first virtual image Im1 for displaying the first virtual image Im1 viewed from the user's viewpoint based on the current position Lp of the main hook 20a on the transmissive display 4, based on the acquired information Ifa about the user's position and the calculated first reference coordinates C1. The image processing device 9 transmits the generated information If1 about the first virtual image Im1 to the display device control device 7. The display device control device 7 acquires the information If1 about the first virtual image Im1 via the communication device 8.

[0085] 8, a discrimination unit 9a of an image processing device 9 uses deep learning with a human and vehicle image discrimination model M to determine whether an image P contains a person H and a vehicle V. When the discrimination unit 9a determines that the image P contains at least one of a person H or a vehicle V, the image processing device 9 generates information Ife relating to the positions of the person H and the vehicle V as output data.

[0086] As shown in FIG. 7 , the image processing device 9 calculates second reference coordinates C2, which are a set of absolute coordinates for displaying the person H and the vehicle V included in the image P surrounded by a frame, based on information about the boom camera 19b, information Ifa about the user's position, information Ifb about the position of the crane 11, information Ifc about the attitude of the crane 11, information Ifd about the reference plane S1, and information Ife about the positions of the person H and the vehicle V. The image processing device 9 generates information If2 about the second virtual image Im2 for displaying the second virtual image Im2 viewed from the user's viewpoint on the reference plane S1 on the transmissive display 4, based on the acquired information Ifa about the user's position and the calculated second reference coordinates C2. The image processing device 9 transmits the information If2 about the second virtual image Im2 to the display device control device 7. The display device control device 7 acquires the information If2 about the second virtual image Im2 via the communication device 8.

[0087] 9, the display device control device 7 displays the first virtual image Im1 on the transmissive display 4 based on information If1 related to the first virtual image Im1. The display device control device 7 displays the current position Lp of the main hook 20a as seen from the user's viewpoint, the turning direction trajectory Tt and the radial direction trajectory Tr of the main hook 20a in a portion of the transmissive display 4 that overlaps with the reference plane S1, with the main hook 20a being viewed by the user as the center through the transmissive display 4. The display device control device 7 displays a new first virtual image Im1 every time it acquires information If1 related to the first virtual image Im1.

[0088] The display device control device 7 displays the second virtual image Im2 on the transmissive display 4 based on information If2 related to the second virtual image Im2. The display device control device 7 displays the person H and the vehicle V viewed from the user's viewpoint using a frame in a portion of the transmissive display 4 that overlaps with the person H and the vehicle V viewed by the user through the transmissive display 4. The display device control device 7 displays a new second virtual image Im2 every time it acquires information If2 related to the second virtual image Im2. The display device control device 7 changes the display position of the first virtual image Im1 in accordance with the movement of the main hook 20a.

[0089] The user perceives that the current position Lp, turning direction trajectory Tt, and radial direction trajectory Tr of the main hook 20a, which are the first virtual image Im1, are displayed on the ground surface, which is the reference plane S1, based on the main hook 20a included in the background Ls of the transmissive display 4, by the transmissive display device 2. The user perceives that the display position of the first virtual image Im1 is changing in accordance with the movement of the main hook 20a.

[0090] The user perceives that the second virtual image Im2 is displayed as a surrounding frame superimposed on the person H and the vehicle V on the ground surface, which is the reference plane S1 included in the background Ls of the transmissive display 4, by the transmissive display device 2. The user also perceives that the display position of the second virtual image Im2 is changed in accordance with the movement of the person H and the vehicle V.

[0091] The above-described operation assistance system 1 displays on the transmissive display 4 a first virtual image Im1, which represents the trajectory of the main hook 20a as seen from the user's viewpoint, superimposed on the background Ls of the transmissive display 4 viewed by the user. The operation assistance system 1 also displays on the transmissive display 4 a second virtual image Im2, which represents a frame surrounding the person H and vehicle V as seen from the user's viewpoint, superimposed on the background Ls of the transmissive display 4 viewed by the user. The operation assistance system 1 provides visual information about the positional relationship of the person H and vehicle V with respect to the trajectory of the main hook 20a to a user at any position. In other words, the operation assistance system 1 allows the operator of the crane 11 using the transmissive display device 2 to confirm in advance the presence of the surrounding person H and vehicle V in the actual scene viewed with the naked eye as the main hook 20a moves. This allows the operator to intuitively grasp the positional relationship between the trajectory of the crane hook and a specific object.

[0092] <Highlighting of people H and vehicles V located near the reference members> 7 and 9, the steering assistance system 1 may display a third virtual image Im3 that highlights the person H and the vehicle V located within a predetermined range A (hereinafter simply referred to as "within the predetermined range A") based on the turning direction trajectory Tt and the radial direction trajectory Tr of the main hook 20a on the transmission-type display device 2. The predetermined range A is a vertical projection space in an area having a predetermined width in a direction horizontally perpendicular to the turning direction trajectory Tt and the radial direction trajectory Tr of the main hook 20a.

[0093] As shown in FIG. 7, the image processing device 9 calculates a predetermined range coordinate C1A, which is a set of absolute coordinates indicating a predetermined range A based on the turning direction trajectory Tt and radial direction trajectory Tr of the main hook 20a, based on the calculated first reference coordinate C1. The image processing device 9 determines whether the person H and vehicle V identified from the image P by the discrimination unit 9a are located within the predetermined range coordinate C1A. If the person H and vehicle V are located within the predetermined range coordinate C1A, the image processing device 9 generates information Iff regarding the positions of the person H and vehicle V located within the predetermined range coordinate C1A. The information Iff regarding the positions of the person H and vehicle V located within the predetermined range coordinate C1A includes the latitude, longitude, altitude, and vehicle orientation of the person H and vehicle V, respectively, located within the predetermined range coordinate C1A.

[0094] The image processing device 9 calculates third reference coordinates C3, which are a set of absolute coordinates for displaying the person H and vehicle V located within the predetermined range A in a surrounding frame, based on information about the boom camera 19b, information Ifa about the user's position, information Ifb about the position of the crane 11, information Ifc about the attitude of the crane 11, information Ifd about the reference plane S1, and information Iff about the positions of the person H and vehicle V located within the predetermined range coordinates C1A. Based on the acquired information Ifa about the user's position and the calculated third reference coordinates C3, the image processing device 9 generates information If3 about a third virtual image Im3 for displaying a third virtual image Im3 viewed from the user's viewpoint on the reference plane S1 on the transmissive display 4. The image processing device 9 transmits the information If3 about the third virtual image Im3 to the display device control device 7. The display device control device 7 acquires the information If3 about the third virtual image Im3 via the communication device 8.

[0095] As shown in FIG. 9 , the display device control device 7 displays the third virtual image Im3 on the transmissive display 4 based on information If3 related to the third virtual image Im3. The third virtual image Im3 is a graphic such as a frame that is further superimposed on the first virtual image Im1 indicating the positions of the person H and the vehicle V for highlighting. The display device control device 7 further surrounds the person H and the vehicle V viewed from the user's viewpoint with a frame in a portion of the transmissive display 4 that overlaps with the person H and the vehicle V viewed by the user through the transmissive display 4. The display device control device 7 displays a new third virtual image Im3 every time it acquires information If3 related to the third virtual image Im3.

[0096] The user perceives that the third virtual image Im3 is displayed as a surrounding frame superimposed on the person H and the vehicle V on the reference plane S1 included in the background Ls of the transmissive display 4 by the transmissive display device 2. Therefore, the user focuses on some of the people H and vehicles V that are highlighted by the third virtual image Im3 among the multiple people H and vehicles V. The user also perceives that the display position of the third virtual image Im3 is changed in accordance with the movement of the people H and the vehicles V.

[0097] The above-described operation assistance system 1 displays a third virtual image on the transmissive display 4 so that the third virtual image is superimposed on people H and vehicles V located within a predetermined range A from the user's viewpoint on the background Ls of the transmissive display 4 that the user is viewing. The operation assistance system 1 provides the user with visual information about the positions of people H and vehicles V that may come into contact with the main hook 20a or the load W being carried by the main hook 20a, among multiple people H and vehicles V that the operator of the crane 11 using the transmissive display device 2 is viewing with the naked eye. In other words, the operation assistance system 1 allows the operator of the crane 11 using the transmissive display device 2 to confirm in advance, in the actual scene viewed with the naked eye, the presence of people H and vehicles V that require special attention when moving the main hook 20a. This allows the operator to intuitively grasp the positional relationship between the trajectory of the crane hook and a specific object.

[0098] <Warning display for users located near the track of the reference member> As shown in Figures 7 and 10, the piloting assistance system 1 may display a fourth virtual image Im4 on the transmissive display device 2, which informs a user of the transmissive display device 2 located within a predetermined range A that the user is located within the predetermined range A.

[0099] 7, the image processing device 9 determines whether or not the user of the transmissive display device 2 is located within a predetermined range A, based on the acquired information Ifa about the user's position and the calculated predetermined range coordinates C1A. When the user of the transmissive display device 2 is located within the predetermined range A, the image processing device 9 generates information Ifg about the position of the user located within the predetermined range A. The information Ifg about the user located within the predetermined range A includes the latitude, longitude, altitude, and vehicle direction of each user located within the predetermined range A.

[0100] The image processing device 9 generates information If4 about the fourth virtual image Im4, which is used to display the fourth virtual image Im4 indicating a warning that the user is located within the predetermined range A, on the transmissive display 4, based on the information Ifa about the user's position and the information Ifg about the user's position within the predetermined range A. The image processing device 9 transmits the generated information If4 about the fourth virtual image Im4 to the display device control device 7 of the user located within the predetermined range A. The display device control device 7 acquires the information If4 about the fourth virtual image Im4 via the communication device 8.

[0101] As shown in FIG. 10 , the display device control device 7 displays the fourth virtual image Im4 on the transmissive display 4 based on information If4 related to the fourth virtual image Im4. The fourth virtual image Im4 is graphic, textual information, or the like indicating that the main hook 20a is located within a predetermined range A based on the turning direction trajectory Tt and radial direction trajectory Tr (not shown) of the main hook 20a. The display device control device 7 displays the fourth virtual image Im4 on the transmissive display 4 at a position and in a manner that allows the user to easily view it. If the display device control device 7 does not acquire information If4 related to the fourth virtual image Im4 for a certain period of time or longer, it stops displaying the fourth virtual image Im4.

[0102] When a user of the transmissive display device 2 is located within a predetermined range A, the above-described operation assistance system 1 displays a fourth virtual image Im4, which is a warning display, on the transmissive display device 2 used by the user. The operation assistance system 1 displays a first virtual image Im1 indicating the current position Lp of the main hook 20a, the rotation direction trajectory Tt of the main hook 20a, and the radial direction trajectory Tr of the main hook 20a, and a fourth virtual image Im4 indicating that the main hook 20a is located within a range where it may come into contact with the main hook 20a or the load W being carried by the main hook 20a, to the slinger or the like using the transmissive display device 2. In other words, the operation assistance system 1 allows the user of the transmissive display device 2, who needs to pay more attention to the main hook 20a, to check in advance the positional relationship with the main hook 20a in the actual scene viewed with the naked eye. This allows the user to intuitively grasp the positional relationship between the trajectory of the crane hook and a specific object.

[0103] <Direction indication of people H and vehicles V located near the reference member> As shown in Figures 7 and 11, the driving assistance system 1 may display a fifth virtual image Im5 on the transmissive display device 2, which displays the direction of a person H and a vehicle V located within a predetermined range A as seen from the user.

[0104] As shown in FIG. 7 , when a person H and a vehicle V are located within a predetermined range A, the image processing device 9 generates information Iff about the positions of the person H and the vehicle V located within the predetermined range A. The image processing device 9 determines whether the person H and the vehicle V located within the predetermined range A are included within the range of a background Ls that overlaps with the transmissive display 4, which is the display screen as seen from the viewpoint of the user of the transmissive display device 2, based on the information Ifa about the user's position and the information Iff about the positions of the person H and the vehicle V located within the predetermined range A. When the person H and the vehicle V located within the predetermined range A are not included within the range of the background Ls that overlaps with the transmissive display 4 as seen from the viewpoint of the user, the image processing device 9 generates information If5 about a fifth virtual image Im5 for displaying the fifth virtual image Im5 on the transmissive display 4, the fifth virtual image Im5 indicating the directions of the person H and the vehicle V located within the predetermined range A. The image processing device 9 transmits the information If5 about the fifth virtual image Im5 to the display device control device 7. The display device control device 7 acquires the information If5 about the fifth virtual image Im5 via the communication device 8.

[0105] As shown in FIG. 11 , the display device control device 7 displays the fifth virtual image Im5 on the transmissive display 4 based on information If5 about the fifth virtual image Im5. The fifth virtual image Im5 is a graphic such as an arrow that indicates the direction of a person H and a vehicle V included in a predetermined range A. The display device control device 7 displays the fifth virtual image Im5 on the transmissive display 4 at a position and in a manner that allows the user to reliably view it. If the display device control device 7 does not acquire information If5 about the fifth virtual image Im5 for a certain period of time or longer, it stops displaying the fifth virtual image Im5.

[0106] When a user of the transmissive display device 2 cannot see with the naked eye the person H and vehicle V located within the predetermined range A, the above-described operation assistance system 1 displays a fifth virtual image Im5, which is a graphic indicating the direction of the person H and vehicle V located within the predetermined range based on the rotation direction trajectory Tt and radial direction trajectory Tr of the main hook 20a, with the user's position as the reference. In other words, when the person H and vehicle V that require special attention when moving the main hook 20a are not located in the actual scene seen by the user's naked eye, the operation assistance system 1 can prompt the user to visually recognize the person H and vehicle V. This allows the user to intuitively grasp the positional relationship between the trajectory of the crane hook and a specific object.

[0107] <Other embodiments> In the above-described embodiment, the operation assistance system 1 acquires the image P using the monocular boom camera 19b as an imaging device. However, the imaging device of the boom camera 19b may be a stereo camera to detect the distance between the boom camera 19b and the person H and vehicle V from the image P. The imaging device configured as a stereo camera can detect the positional relationship between a specific object located on the ground surface and a specific object located on the roof of a building from an image containing the specific object. That is, the operation assistance system uses an image processing device to display a second virtual image Im2, a third virtual image Im3, and a fourth virtual image Im4 on a transmission display device, taking into account the distance from the imaging device to the specific object. This allows the positional relationship between the trajectory of the crane hook and the specific object to be more accurately detected and intuitively grasped.

[0108] In the above-described configuration, the maneuvering assistance system 1 calculates the distance and direction of a specific object, such as a person H or a vehicle V, based on the image P captured by the boom camera 19b. However, the maneuvering assistance system may also be configured to acquire the distance and direction of a specific object, such as a person H or a vehicle, using a laser as a position information acquisition device, using LIDAR (light detection and ranging). LIDAR is a well-known technology that measures the distance and direction from a laser source to the object and identifies the properties of the object by scanning the object with a laser and observing the scattered and reflected light of the laser. When LIDAR is used, the information Ifc about the attitude of the crane 11 includes the LIDAR position, laser irradiation conditions, irradiation range, etc., which are information about the conditions under which the LIDAR acquires the distance and direction of a specific object as position information.

[0109] In the above-described embodiment, the discrimination unit 9a of the image processing device 9 discriminates between the person H and the vehicle V by deep learning, which is included in machine learning. However, the maneuvering assistance system may also be configured to detect the person H and the vehicle V from an image P capturing the person H and the vehicle V having markers that identify them. The maneuvering assistance system does not need to perform machine learning using a training dataset that is training data. In addition, the person H and the vehicle V can be discriminated based on information that cannot be discerned from an image, such as the gender, age, and affiliation of the person H.

[0110] In the above-described embodiment, the maneuvering assistance system 1 displays a surrounding frame on the transmissive display device 2 as the first virtual image Im1 indicating the person H and the vehicle V. However, the maneuvering assistance system 1 may also use letters, arrows, coloring, etc., as long as they indicate the presence of the person and the vehicle.

[0111] In the above-described embodiment, the operation assistance system 1 acquires the image P using the boom camera 19b of the crane 11 as the imaging device. However, the operation assistance system may use a camera other than the boom camera 19b as the imaging device. For example, the operation assistance system may acquire images captured by the display device camera 6, the body camera of the crane 11, a camera installed at the construction site, a camera mounted on a drone, or the like as the imaging device.

[0112] In the above-described embodiment, when the movement range of the crane apparatus 16 is restricted to prevent contact with surrounding structures, the image processing device 9 may generate information about a virtual image indicating the movement range of the main hook 20a as viewed from the user's viewpoint on the transmissive display 4. The image processing device 9 acquires information about the restriction of the movement range of the crane 11 from the crane control device 30 via the communication device 8. When the person H and vehicle V identified by the discrimination unit 9a in the image P are located outside the movement range of the crane apparatus 16, the image processing device 9 does not generate information If3 about the third virtual image Im3, information If4 about the fourth virtual image Im4, or information If5 about the fifth virtual image Im5. That is, the operation assistance system 1 does not display the third virtual image Im3, the fourth virtual image Im4, or the fifth virtual image Im5 on the user's transmissive display device 2. This prevents the user, who is the operator of the crane 11, from paying attention to the person H and vehicle V located outside the movement range.

[0113] In each of the above-described embodiments, the operation assistance system 1 for the crane 11 displays any one of the first virtual image Im1, the second virtual image Im2, the third virtual image Im3, the fourth virtual image Im4, and the fifth virtual image Im5 on the transmission-type display device 2. However, the operation assistance system 1 may display at least one of the first virtual image Im1 to the fifth virtual image Im5 as desired by a user of the transmission-type display device 2. Furthermore, the operation assistance system 1 is not limited to the virtual images displayed in the above-described embodiments, and may display any virtual image that assists in the operation of the crane 11.

[0114] Furthermore, in each of the above-described embodiments, the transparent display device 2 of the operation assistance system 1 for the crane 11 is worn by the operator of the crane 11. However, the transparent display device 2 may also be worn by at least one of the operator of the crane 11 and a worker other than the operator of the crane 11. For example, if the operator and slinger of the crane 11 each wear a transparent display device 2, the operation assistance system 1 causes each of the transparent display devices 2 to display the same or different types of virtual images based on the same information Ifc about the attitude of the crane 11. As a result, the operator and slinger of the crane 11 view virtual images based on the same information Ifc about the attitude of the crane 11 within the field of view of their respective positions, and therefore can more accurately share information such as the position of the main hook 20a relative to the load W.

[0115] In the above-described embodiment, the operation assistance system 1 is configured as an operation assistance system for a crane 11. However, the operation assistance system 1 may also be configured as an operation assistance system for a vehicle for working on an aerial platform, which is a work vehicle. The operation assistance system for the vehicle for working on an aerial platform acquires, via a communication device, information on the user's position, information on the position of the vehicle for working on an aerial platform, information on the attitude of the vehicle for working on an aerial platform, and information on a reference plane, including the position and attitude of the reference plane that serves as a reference for displaying a virtual image. The image processing device of the operation assistance system for the vehicle for working on an aerial platform generates information on a virtual image for displaying, on the transmission-type display device, a virtual image showing the rotational trajectory and radial trajectory of the bucket, which is a reference member moved by the vehicle for working on an aerial platform, as seen from the user's viewpoint at the current position of the bucket. The transmission-type display device acquires information on the virtual image for the vehicle for working on an aerial platform and displays the virtual image on a display screen.

[0116] In each of the above-described embodiments, the operation assistance system 1 for the crane 11 displays at least one of the first virtual image Im1 to the fifth virtual image Im5 using the transmission-type display device 2. However, the operation assistance system for the crane 11 may display at least one of the first virtual image to the fifth virtual image on an image captured by the boom camera 19b capturing an image of the reference member. At least one of the first virtual image to the fifth virtual image can be operated on a display screen such as a touch panel.

[0117] In each of the above-described embodiments, the transmissive display device 2 acquires, by the display device GNSS receiver 5, information Ifa about the user's position, which is necessary for displaying the first virtual image Im1 and the second virtual image Im2 on the transmissive display 4. However, the transmissive display device may calculate the position of the transmissive display device by recognizing a marker at a specific position. For example, the transmissive display device calculates the position of the transmissive display device by recognizing a marker located inside the cabin 27 of the crane 11. Furthermore, the transmissive display device calculates the position using an inertial measurement unit (IMU) based on the calculated position.

[0118] In each of the above-described embodiments, the transmissive display device 2 is a wearable terminal that is worn by a user. However, the transmissive display device may be any display device that superimposes a virtual image seen from the user's viewpoint on a display screen that allows the background Ls to be seen from the user's viewpoint. The transmissive display device may be, for example, a head-up display that uses the windshield of the cabin as a display screen to display a virtual image seen from the pilot's viewpoint in the cockpit.

[0119] Furthermore, in each of the above-described embodiments, the image processing device 9 calculates the first reference coordinates C1, which are a set of absolute coordinates for displaying the current position Lp of the main hook 20a, based on the acquired information Ifb about the position of the crane 11, information Ifc about the attitude of the crane 11, and information Ifd about the reference plane S1. However, the image processing device may calculate the first reference coordinates, which are a set of absolute coordinates for displaying the current position of the main hook, based on the image P captured by the boom camera, in addition to the information about the position of the crane, the information about the attitude of the crane, and the information about the reference plane. This allows the image processing device to detect the position of the main hook even if the main hook is swinging.

[0120] In each of the above-described embodiments, when a user of the transmissive display device 2 is located within a predetermined range A, the piloting assistance system 1 displays a fourth virtual image Im4, which is a warning display, on the transmissive display device 2 used by the user. However, the piloting assistance system may also transmit a signal indicating that the mobile device is located within the predetermined range to a mobile device, such as a smartphone or a wearable device, registered in the piloting assistance system. When the piloting assistance system determines that the mobile device is located within the predetermined range based on the acquired position information of the mobile device, it transmits a signal, such as a vibration or a warning sound, to the mobile device.

[0121] The above-described embodiment merely shows a typical form, and various modifications can be made without departing from the gist of one embodiment. Of course, the present invention can be embodied in various other forms, and the scope of the present invention is defined by the claims, and further includes the meaning of equivalents set forth in the claims, as well as all modifications within the scope of the claims. [Explanation of symbols]

[0122] 1. Pilot assistance system 2 Transparent display device 3 frames 4. Transmissive displays 5 GNSS receiver for display device 6. Camera for display devices 7 Display device control device 8. Communications Equipment 9 Image Processing Device 11 Crane 19b Boom Camera 21a Main hook (reference member) 30 Crane control device Lp Current position Ifa User location information Ifb Crane Location Information Ifc Crane attitude information Ifd Reference Plane Information Ife Information about the location of people and vehicles Iff Information about the location of people and vehicles within a given area Im1 First virtual image Im2 Second virtual image Im3 Third virtual image Im4 4th virtual image Im5 Fifth virtual image C1 First reference coordinate C2 Second reference image C3 Third reference coordinate If1 Information about the first virtual image If2 Information about the second virtual image If3 Information about the third virtual image If4 Information about the fourth virtual image If5 Information about the fifth virtual image

Claims

1. a transmission type display device that displays a virtual image seen from a user's viewpoint on a display screen that allows the background to be seen; a position information acquisition device for acquiring position information of a specific object within a range including at least a portion of a movable range of a hook of a crane having a telescopic boom that can be raised and lowered on a swivel base; an image processing device that generates information about a virtual image to be displayed on the transmission type display device; A crane operation assistance system comprising a communication device that transmits and receives information between the position information acquisition device and the image processing device, between the crane control device and the image processing device, and between the transmissive display device and the image processing device, The image processing device includes: calculating a rotation direction trajectory and a radial direction trajectory of the hook based on the current position of the hook and the rotation center of the boom, based on information about the position of the crane and information about the attitude of the crane acquired via the communication device; generating information about a first virtual image for displaying on the display screen a first virtual image showing an image of the current position of the hook and a turning direction trajectory and a radial direction trajectory of the hook as viewed from the user's viewpoint, based on information about the user's position acquired via the communication device; Detecting position information about a specific object from the position information acquired by the position information acquisition device, calculating the position of the specific object based on information about the conditions when the position information acquisition device acquires the position information and information about the user's position acquired via the communication device, and generating information about a second virtual image for displaying on the display screen a second virtual image showing an image of the specific object as seen from the user's viewpoint; The transmissive display device comprises: acquiring information about the first virtual image and information about the second virtual image from the image processing device via the communication device, and displaying the first virtual image and the second virtual image viewed from the user's viewpoint on the display screen; Crane operation assistance system.

2. The crane operation assistance system according to claim 1, The image processing device includes: generating information about a third virtual image for displaying, on the display screen, a third virtual image showing a highlighted image of the specific object located within a predetermined range based on the rotational trajectory and the radial trajectory of the hook; Crane operation assistance system.

3. The crane operation assistance system according to claim 1 or 2, The image processing device includes: When the user is located within a predetermined range based on the rotational trajectory and the radial trajectory of the hook, information regarding a fourth virtual image is generated to display on the display screen a fourth virtual image showing that the user is located in the vicinity of the rotational trajectory and the radial trajectory, and the information regarding the fourth virtual image is transmitted to the image processing device of the user via the communication device; The transmissive display device comprises: When information about the fourth virtual image is acquired from the image processing device via the communication device, the fourth virtual image seen from the viewpoint of the user is displayed on the display screen. Crane operation assistance system.

4. The crane operation assistance system according to any one of claims 1 to 3, The image processing device includes: When the specific object is located outside the range of the background overlapping with the display screen and within a predetermined range based on the turning direction trajectory and the radial direction trajectory of the hook as viewed from the user's viewpoint, information related to a fifth virtual image is generated for displaying on the display screen a fifth virtual image showing an image displaying the direction of the specific object located within the predetermined range based on the turning direction trajectory and the radial direction trajectory of the hook; The transmissive display device comprises: When information about the fifth virtual image is acquired from the image processing device via the communication device, the fifth virtual image seen from the viewpoint of the user is displayed on the display screen. Crane operation assistance system.

5. The crane operation assistance system according to any one of claims 1 to 4, The location information acquisition device A LIDAR that acquires the distance and direction to the specific object using a laser. Crane operation assistance system.

6. The crane operation assistance system according to any one of claims 1 to 4, The location information acquisition device An imaging device for capturing an image of a range including at least a part of the movable range of a hook of a crane having a telescopic boom that can be raised and lowered on a swivel base, The image processing device includes: information about the angle of view and the imaging position of the imaging device and an image captured by the imaging device are acquired via the communication device; an image of a specific object is detected in an image captured by the imaging device, and the position of the specific object is calculated based on information about the angle of view and imaging position of the imaging device and information about the position of the user; and information about a second virtual image is generated to display on the display screen a second virtual image showing the image of the specific object as seen from the viewpoint of the user. Crane operation assistance system.

7. 7. The crane operation assistance system according to claim 6, The imaging device is It is a stereo camera, Crane operation assistance system.

8. The crane operation assistance system according to claim 6 or 7, The image processing device includes: having training data for images of the specific object, and determining images of the specific object from images captured by the imaging device by machine learning based on the training data; Crane operation assistance system.

9. A crane comprising a maneuvering assistance system according to any one of claims 1 to 8.

Images