Work vehicles

Abstract

To provide a work vehicle capable of moving a tip of a boom along a predetermined moving route irrespective of an operator's skill.SOLUTION: A control device 24 of a crane 1 acquires a position of a tip of a boom or a position of a rotation center of the boom with respect to a boom camera 9b, posture information Ip of the boom, view angle information Ii of an image, and an image P obtained by imaging the periphery of the crane 1 from above by the boom camera 9b, displays the image P in a display device 21, calculates tip coordinates B of the boom in a coordinate system of a display screen and rotation center coordinates C of the boom in the coordinate system of the display screen, calculates target coordinates T in the coordinate system of the display screen at the target position of the tip of the boom set on the image P, and displays present operation position coordinates Pa, target operation position coordinates Pb, and an operation order of an operation tool for the crane device of the actuator such that the tip of the boom reaches the target coordinates T along a predetermined moving route.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a work vehicle. [Background technology]

[0002] Conventionally, when transporting a load using a mobile crane, which is a work vehicle, the operator of the mobile crane needs to operate multiple operating tools, such as a slewing operating tool for operating the boom of the crane, a hoisting operating tool, an extension operating tool, and a winch operating tool for lifting and lowering the load. In addition, the operator of the mobile crane must operate the multiple operating tools simultaneously to operate the mobile crane so that the load does not come into contact with structures at the work site. With a mobile crane configured in this way, it may be difficult to transport the load to a target position depending on the operator's skill and the situation at the work site. Therefore, a crane that displays information about the operation of the crane based on the movement path of the load displayed on a 3D map of the work site is known. For example, see Patent Document 1.

[0003] The crane described in Patent Document 1 extracts three-dimensional information of a suspended load (luggage) based on three-dimensional information of a work site acquired by a stereo camera or the like at the end of the boom. Furthermore, the crane creates a plan view based on the three-dimensional information of the work site. Next, the crane calculates a movement path on the plan view. The crane displays information regarding operations required to move the suspended load along the calculated movement path.

[0004] The crane described in Patent Document 1 displays information about the operation required to move the load as numerical values, such as the amount of change in the rotation angle of the swivel base, the amount of change in the telescopic length of the boom, and the amount of change in the hoisting angle. The operator of the crane operates each of the control tools to which actuators with different movement speeds are assigned so that the swivel base and the boom are in positions based on the displayed numerical values. However, operators with low skill levels find it difficult to intuitively determine the amount of operation of each control tool according to the movement speed of the assigned actuator. Therefore, even if the crane described in Patent Document 1 displays information about the operation required to move the load, it may be difficult to operate depending on the operator's level of skill. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2018-95369 A Summary of the Invention [Problem to be solved by the invention]

[0006] An object of the present invention is to provide a work vehicle that is capable of moving the tip of a boom along a predetermined movement path, regardless of the skill level of the operator. [Means for solving the problem]

[0007] The problem to be solved by the present invention has been described above, and the means for solving this problem will now be described.

[0008] A work vehicle according to one embodiment of the present invention comprises a work vehicle having a boom that can be raised and lowered via a swivel base attached to a running body, a camera device that photographs the area around the work vehicle from above, a display device that displays images captured by the camera device and is capable of outputting the coordinates of any position on a display screen specified externally, and a control device that controls the actuators of the work vehicle.

[0009] The control device acquires the position of the tip of the boom or the position of the boom's rotation center relative to the imaging device, boom attitude information, angle of view information of the image, and an image of the surroundings of the work vehicle captured from above by the imaging device, displays the image on the display device, calculates the coordinate of the tip of the boom in the coordinate system of the display screen and the coordinate of the boom's rotation center in the coordinate system of the display screen, calculates the coordinate in the coordinate system of the display screen of a target position of the tip of the boom set on the image, and displays the current operating position and target operating position of each operating tool of the actuator so that the tip of the boom reaches the target position via a straight line path.

[0010] In the above-mentioned configuration, the control device of the work vehicle calculates the coordinates of the center of rotation of the work vehicle and the coordinates of the tip of the boom in an image including the surroundings of the work vehicle captured by the imaging device. By calculating the coordinates of the center of rotation of the work vehicle and the coordinates of the tip of the boom, the control device can convert the coordinate system of the display screen, which is an orthogonal coordinate system with its origin at a predetermined position on the display screen of the display device, into a crane coordinate system with the center of rotation of the mobile crane as its reference.

[0011] Therefore, the operator of the work vehicle can set the target position of the boom tip on the image while checking the situation around the work vehicle on the image in real time. The control device also displays the current operation position and the target operation position of each operating tool, which is information related to the operation of each operating tool that operates the actuator so that the boom tip reaches the target position along a predetermined movement path. The operator can move the boom tip to the target position by simple operation based on the information related to the operation of each operating tool while grasping the situation of the work site in real time using the work vehicle. Therefore, even an operator with low proficiency can visually grasp the operation amount of each operating tool to which the actuator with different movement speeds is assigned. This allows the work vehicle to move the boom tip along the predetermined movement path regardless of the operator's level of proficiency.

[0012] According to another aspect, the work vehicle of the present invention preferably includes the following configuration: The control device displays an operation sequence of each operating tool of the actuator.

[0013] In the above-mentioned configuration, the control device further displays an operation sequence of the operating devices that operate the actuators so that the tip of the boom reaches the target position along a predetermined movement path. The operator can move the tip of the boom to the target position with a simple operation that follows the operation sequence in addition to the operation amount of each operating device while grasping the situation of the work site in real time using the work vehicle. This allows the work vehicle to move the tip of the boom along the predetermined movement path regardless of the operator's level of skill.

[0014] According to another aspect, it is preferable that the work vehicle of the present invention includes the following configuration: The imaging device is provided at the tip of the boom. The control device sets the center position of the image to the tip position of the boom, and calculates the coordinates of the boom tip in the coordinate system of the display screen and the coordinates of the boom rotation center in the coordinate system of the display screen from the boom attitude information including the boom hoisting angle, the boom extension length, the boom rotation angle, and the shape and dimensions of the boom.

[0015] In the above-mentioned configuration, since the image is captured from the tip position of the boom, the control device can calculate the coordinates of the tip of the boom in the coordinate system of the display screen and the coordinates of the boom's rotation center in the coordinate system of the display screen from the angle of view information including information such as the ratio between the unit length in the image and the actual length of the corresponding part for each shooting distance and the boom attitude information. Therefore, the operator can move the tip of the boom to the target position with a simple operation based on the information on the operation of each of the operating tools while grasping the situation of the work site in real time using the work vehicle. This allows the work vehicle to move the tip of the boom along the predetermined movement path regardless of the operator's level of skill.

[0016] From another viewpoint, it is preferable that the work vehicle of the present invention includes the following configuration: The control device detects an image of the tip of the boom from the image captured by the imaging device, and calculates the coordinates of the boom's rotation center in the coordinate system of the display screen and the coordinates of the boom tip in the coordinate system of the display screen from the orientation and size of the image of the tip of the boom and attitude information of the boom.

[0017] In the above-mentioned configuration, the control device automatically detects the coordinates of the boom's rotation center and the coordinates of the boom's tip in the coordinate system of the display screen from an image including the work vehicle captured by the image capture device, by comparing the coordinates with an image of the boom's tip previously acquired. The control device can easily calculate the coordinates of the target position of the boom tip specified on the image based on the coordinates of the boom's rotation center. In addition, the operator can move the boom tip to the target position with a simple operation based on information on the operation of each of the operating tools while grasping the situation of the work site in real time using the work vehicle. This allows the work vehicle to move the boom tip along the predetermined movement path regardless of the operator's level of skill.

[0018] From another viewpoint, it is preferable that the work vehicle of the present invention includes the following configuration. The work vehicle is equipped with a first GNSS receiver that detects absolute coordinates of the boom's rotation center or absolute coordinates of the boom's tip, an orientation sensor that detects the orientation of the boom's extension direction, and a second GNSS receiver that detects absolute coordinates of the imaging device. The control device acquires detection values ​​of the first GNSS receiver, detection values ​​of the second GNSS receiver, detection values ​​of the orientation sensor, and attitude information of the boom, and calculates absolute coordinates of the boom's rotation center and absolute coordinates of the boom's tip.

[0019] In the above-mentioned configuration, the control device calculates the absolute coordinates of the boom tip from the absolute coordinates of the boom's rotation center or the absolute coordinates of the boom tip, the azimuth of the boom extension direction, the absolute coordinates of the camera device, the angle of view information of the camera device, and the boom attitude information. The control device also calculates the coordinates of the rotation center and the coordinates of the boom tip in the coordinate system of the display screen based on the absolute coordinates of the camera device and the angle of view information of the camera device. The operator can move the boom tip to the target position with a simple operation based on information on the operation of each of the operating tools while grasping the situation of the work site in real time using the work vehicle. This allows the work vehicle to move the boom tip along the predetermined movement path regardless of the operator's level of skill.

[0020] According to another aspect, the work vehicle of the present invention preferably includes the following configuration: The target operation positions of the operating tools of the actuators are displayed by graphics.

[0021] In the above-mentioned configuration, the control device transmits information regarding the operation of each of the operating tools to the operator by means of a graphic. Specifically, the control device displays the current operating positions, operating sequence, and target operating positions of each of the operating tools of the actuator by the position of the graphic. The operator can move the tip of the boom to the target position by operating each of the operating tools according to the target operating positions and the operating sequence for the current operating positions of each of the operating tools displayed, while grasping the situation of the work site in real time using the work vehicle. This allows the work vehicle to move the tip of the boom along the predetermined movement path regardless of the operator's level of skill.

[0022] According to another aspect, the work vehicle of the present invention preferably includes the following configuration: The display device is configured to be able to operate an operating tool for the actuator.

[0023] In the above-mentioned configuration, the display device allows the operator to operate each of the operating tools while visually checking information related to the operation of each of the operating tools. The operator can easily and accurately change the current operating positions of each of the operating tools to the target operating positions of each of the operating tools displayed on the display device while grasping the situation of the work site in real time using the work vehicle. This allows the work vehicle to move the tip of the boom along the predetermined movement path regardless of the operator's level of skill.

[0024] According to another aspect, the work vehicle of the present invention preferably includes the following configuration: When the control device acquires the coordinates of another work vehicle, the control device sets the position of the other work vehicle as the target position.

[0025] In the above-mentioned configuration, the control device displays on the display device information related to the operation of each of the operating devices required to move the tip of the boom along the predetermined movement path with the position of the other work vehicle as a target position. The operator can move the tip of the boom to the other work vehicle by operating each of the operating devices in accordance with the target operation position and the operation order relative to the current operation position of each of the operating devices displayed while grasping the situation of the work site in real time using the work vehicle. This allows the work vehicle to move the tip of the boom along the predetermined movement path regardless of the operator's level of skill.

[0026] The technical terms used in this specification are used for the purpose of defining only specific embodiments, and are not intended to limit the invention by said technical terms. Unless otherwise defined, all terms used in this specification (including technical and scientific terms) have the same meaning as commonly understood by those skilled in the art to which this invention belongs. Terms defined in commonly used dictionaries should be interpreted to have a meaning consistent with the meaning in the context of the relevant technology and this disclosure, and should not be interpreted in an ideal or overly formal sense unless expressly defined in this specification.

[0027] In the description of the present invention, it is understood that a number of techniques and steps are disclosed. Each of these has its own advantages and can be used with one or more, or in some cases all, of the other disclosed techniques. Therefore, for the sake of clarity, the description of the present invention refrains from repeating all possible combinations of the individual steps unnecessarily. However, the specification and claims should be read with the understanding that all such combinations are within the scope of the present invention.

[0028] In the following description, numerous specific examples are described to provide a thorough understanding of the present invention. However, it is clear to those skilled in the art that the present invention can be practiced without these specific examples. Therefore, the following disclosure should be considered as an example of the present invention, and is not intended to limit the present invention to the specific embodiments shown in the following drawings or description. Effect of the Invention

[0029] According to one embodiment of the present invention, the work vehicle can move the tip of the boom along a predetermined movement path, regardless of the operator's level of skill. [Brief description of the drawings]

[0030] [Figure 1] 1 is an overall view showing the overall configuration of a crane according to the present invention; [Diagram 2] FIG. 2 is a block diagram showing a control configuration of the crane according to the first embodiment of the present invention. [Diagram 3] FIG. 1 is a schematic diagram showing a control configuration of an image processing device according to a first embodiment of the present invention. [Figure 4] FIG. 3 is a block diagram showing a screen displaying the moving direction of the boom tip of the crane according to the first embodiment of the present invention and a target operation position of the crane equipment operating tool. [Diagram 5] FIG. 2 is a block diagram showing a state in which the boom tip of the crane according to the first embodiment of the present invention is moved by operating a crane equipment operation tool, and a screen showing the operation state of the crane equipment operation tool. [Figure 6] FIG. 11 is a diagram showing a crane and a camera for the crane according to a second embodiment of the present invention. [Figure 7] FIG. 11 is a block diagram showing a control configuration of a crane according to a second embodiment of the present invention. [Figure 8] FIG. 11 is a block diagram showing a control configuration of an image processing apparatus according to a second embodiment of the present invention. [Figure 9] FIG. 11 is a block diagram showing a control configuration of a crane according to a third embodiment of the present invention. [Figure 10] FIG. 11 is a block diagram showing a control configuration of an image processing apparatus according to a third embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0031] A crane 1, which is a work vehicle according to a first embodiment of the present invention, will be described below with reference to Figures 1 to 4. Note that, although a rough terrain crane will be described in this embodiment, any work vehicle equipped with a boom that can be raised and lowered via a swivel base, such as an all-terrain crane, a truck crane, or an aerial work vehicle, may be used.

[0032] 1 and 2, the crane 1 is a mobile crane that can be moved to any location. The crane 1 includes a vehicle 2, a crane device 6, a boom camera 9b that is an image capturing device, a display device 21 (see FIG. 2), an input device 22, an image processing device 23, and a control device 24 (see FIG. 2).

[0033] The vehicle 2 is a traveling body that transports a crane device 6. The vehicle 2 has a plurality of wheels 3, and travels using an engine 4 as a power source. The vehicle 2 is provided with outriggers 5. The outriggers 5 are composed of an extension beam that can be hydraulically extended on both sides in the width direction of the vehicle 2, and a hydraulic jack cylinder that can be extended in a direction perpendicular to the ground.

[0034] The crane apparatus 6 is a work device that hoists a load W by a wire rope. The crane apparatus 6 includes a swivel base 7, a boom 9, a main hook block 10, a sub-hook block 11, a hoisting hydraulic cylinder 12, a main winch 13, a main wire rope 14, a sub-winch 15, a sub-wire rope 16, a cabin 20, and a crane apparatus sensor 17.

[0035] The swivel table 7 is a rotating device that configures the crane device 6 to be rotatable. The swivel table 7 is mounted on the frame of the vehicle 2 via annular bearings. The swivel table 7 is configured to be freely rotatable around the center of the annular bearing. The swivel table 7 is provided with a hydraulic swivel motor 8 that serves as an actuator. The swivel table 7 is configured to be rotatable in one direction and the other direction by the swivel hydraulic motor 8.

[0036] The boom 9 is a movable support that supports the main wire rope 14 and the sub wire rope 16 in a state in which the load W can be hoisted. The boom 9 is composed of multiple boom members. The base end of the base boom member of the boom 9 is swingably mounted approximately at the center of the swivel base 7. The boom 9 is configured to be freely extendable in the axial direction by moving each boom member with an extension hydraulic cylinder 9a (see FIG. 2) which serves as an actuator. The boom 9 is also provided with a boom camera 9b, which serves as an imaging device, and a jib 9c. Images captured by the boom camera 9b are displayed on the display device 21.

[0037] The main hook block 10 and the sub-hook block 11 are used to suspend a load W. The main hook block 10 is provided with a plurality of hook sheaves around which the main wire rope 14 is wound, and a main hook 10a for suspending the load W. The sub-hook block 11 is provided with a sub-hook 11a for suspending the load W.

[0038] The hoisting hydraulic cylinder 12 is an actuator that raises and lowers the boom 9 and maintains the posture of the boom 9. The end of the cylinder portion of the hoisting hydraulic cylinder 12 is connected to the swivel base 7 so that it can be raised and lowered, and the end of the rod portion is connected to the base boom member of the boom 9 so that it can swing freely.

[0039] The main winch 13 and the sub winch 15 are used to pay in (wind up) and out (wind down) the main wire rope 14 and the sub wire rope 16. The main winch 13 is configured so that a main drum around which the main wire rope 14 is wound is rotated by a main hydraulic motor (not shown) which serves as an actuator. The sub winch 15 is configured so that a sub drum around which the sub wire rope 16 is wound is rotated by a sub hydraulic motor (not shown) which serves as an actuator.

[0040] The crane apparatus 6 has a plurality of crane apparatus sensors 17 required for controlling the swing hydraulic motor 8, the telescopic hydraulic cylinder 9a, the hoisting hydraulic cylinder 12, the main winch 13, and the sub winch 15. The plurality of crane apparatus sensors 17 are provided on the corresponding actuators or the control valves of the corresponding actuators. The plurality of crane apparatus sensors 17 detect the amount of operation of each actuator, etc. Furthermore, the detection values ​​of the crane apparatus sensors 17 are included in the attitude information Ip of the boom 9.

[0041] The cabin 20 is a housing that covers the operator's seat. The cabin 20 is mounted on the rotating base 7. A operator's seat (not shown) is provided. The operator's seat is provided with a travel operation tool 18 for operating the vehicle 2 to travel, a rotation operation tool for operating the rotation hydraulic motor 8 of the crane apparatus 6, a telescopic operation tool for operating the telescopic hydraulic cylinder, a hoisting operation tool for operating the hoisting hydraulic cylinder 12, crane apparatus operation tools 19 including a main operation tool for operating the main winch 13 and a sub operation tool for operating the sub winch 15, a display device 21, an input device 22, etc. (see FIG. 2).

[0042] The display device 21 displays the image P captured by the boom camera 9b. The display device 21 is composed of a liquid crystal monitor or the like having a touch panel function that allows input from the screen. The display device 21 is disposed at a position where the operator of the crane 1 can view it. The display device 21 is configured to be able to acquire the image P from the boom camera 9b. The display device 21 displays the image P captured by the boom camera 9b, information about the crane 1 calculated based on the image P captured by the boom camera 9b, and information about the operation of the crane operating tool 19 (see FIG. 4). The information about the operation of the crane operating tool 19 is information about a current operating position that is the operating position of the crane operating tool 19 in the current attitude of the boom 9, and a target operating position that is the operating position of the crane operating tool 19 for moving the tip of the boom 9 to a target position. The information about the operation of the crane operating tool 19 may include information about the operating order of the crane operating tool 19.

[0043] The input device 22 inputs the target position of the tip of the boom 9 and the operation signal of the crane device operating tool 19. The input device 22 also inputs the operation signal of the boom camera 9b. In this embodiment, the display device 21, which is configured as a touch panel that can be input from the screen, constitutes the input device 22. The input device 22 can input any position within the range in which the image P is displayed on the screen of the display device 21 as the target position of the tip of the boom 9. In other words, the display device 21, which is the input device 22, is configured to be able to output the coordinates of any position within the range in which the image P is displayed on the screen to the image processing device 23 or the like. The input device 22 can also input any position within the range in which information on the operation of the crane device operating tool 19 on the screen of the display device 21 is displayed as the target operation position of the crane device operating tool 19. In other words, the display device 21, which is the input device 22, is configured to be able to operate the crane device operating tool 19.

[0044] The image processing device 23 processes the image P (see FIG. 4) captured by the boom camera 9b. The image processing device 23 may be substantially configured with a CPU, a ROM, a RAM, a HDD, etc. connected via a bus, or may be configured with a one-chip LSI, etc. In this embodiment, the image processing device 23 is provided in the display device 21. The image processing device 23 stores various programs and data for controlling the operations of the display device 21, the input device 22, etc., and for processing image data. The image processing device 23 is connected to the control device 24. In other words, the image processing device 23 is considered to be electrically included in the control device 24. The image processing device 23 may be configured integrally with the control device 24.

[0045] As shown in FIG. 3, the image processing device 23 is connected to the display device 21. The image processing device 23 can acquire the position of the target position input from the display screen of the display device 21, which is the input device 22, as a target coordinate T in the coordinate system of the display screen (hereinafter, simply referred to as the "coordinate system of the display screen"), which is an orthogonal coordinate system with a predetermined position on the display screen as the origin. Furthermore, the image processing device 23 can acquire the current operation position of the crane apparatus operating tool 19 (see FIG. 2) displayed on the display screen as a current operation position coordinate Pa in the coordinate system of the display screen. The image processing device 23 can acquire a current operation signal Sa corresponding to the current operation position of the crane apparatus operating tool 19 from the control device 24. Furthermore, the image processing device 23 can acquire a target operation signal Sb corresponding to a target operation position of the crane apparatus operating tool 19 required to move the tip of the boom 9 (see FIG. 1) to the target position along a predetermined movement path R (see FIG. 4). The image processing device 23 can acquire posture information Ip of the boom 9, including the boom 9 hoisting angle, the boom 9 telescopic length, the boom 9 rotation angle, and the shape and dimensions of the boom 9, from the control device 24. The image processing device 23 can acquire an operation signal of the boom camera 9b from the display device 21.

[0046] The image processing device 23 can generate a current operation signal Sa of the crane operating tool 19 (see FIG. 2) based on the current operation position coordinate Pa of the crane operating tool 19 in the coordinate system of the display screen. The image processing device 23 can calculate the current operation position coordinate Pa of the crane operating tool 19 in the coordinate system of the display screen based on the acquired current operation signal Sa of the crane operating tool 19. Similarly, the image processing device 23 can calculate the target operation position coordinate Pb of the crane operating tool 19 in the coordinate system of the display screen based on the acquired target operation signal Sb of the crane operating tool 19. The image processing device 23 can generate a control signal for the boom camera 9b based on the acquired operation signal of the boom camera 9b.

[0047] The image processing device 23 can transmit the generated current operation signal Sa of the crane operating tool 19 (see FIG. 2) to the control device 24. The image processing device 23 can display a figure showing the crane operating tool 19 or a figure showing the operating position of the crane operating tool 19 at the current operating position coordinate Pa and the target operating position coordinate Pb of the crane operating tool 19 in the calculated coordinate system of the display screen. That is, the image processing device 23 can display the current operating position of the crane operating tool 19 and the target operating position of the crane operating tool 19, which are information related to the operation of the crane operating tool 19, on the display device 21. The image processing device 23 can transmit the generated operation signal of the boom camera 9b to the boom camera 9b. The image processing device 23 may be configured to display the operation sequence of the crane operating tool 19 on the display device 21.

[0048] 3, the image processing device 23 has angle-of-view information Ii including information such as the ratio between a unit length in an image P captured by the boom camera 9b and the actual length of a corresponding portion for each shooting distance. The image processing device 23 can calculate the transformed tip coordinate Bc of the boom 9, the transformed rotation center coordinate Cc of the boom 9, and the transformed target coordinate Tc of the tip of the boom 9 in a crane coordinate system based on the rotation center of the crane 1, from the angle-of-view information Ii of the boom camera 9b and the attitude information Ip of the boom 9.

[0049] Since the boom camera 9b is provided at the tip of the boom 9 (see FIG. 1), the image processing device 23 calculates the tip coordinate B of the boom 9 in the coordinate system of the display screen with the center position of the image P as the tip position of the boom 9. Next, the image processing device 23 calculates the height of the boom camera 9b based on the installation surface of the crane 1 based on the attitude information Ip of the boom 9. The image processing device 23 calculates the rotation center coordinate C of the boom 9 in the coordinate system of the display screen from the angle of view information Ii of the boom camera 9b, the height of the boom camera 9b, and the attitude information Ip of the boom 9. The image processing device 23 superimposes and displays the calculated positions of the tip coordinate B and the rotation center coordinate C on the image P displayed on the display device 21 (see FIG. 2). Note that if the rotation center coordinate C is not included in the range of the image P, it is not displayed on the image P.

[0050] The image processing device 23 converts the tip coordinate B of the boom 9 and the rotation center coordinate C of the boom 9 in the coordinate system of the display screen from the height of the boom camera 9b and the angle of view information Ii of the boom camera 9b into a converted tip coordinate Bc and a converted rotation center coordinate Cc in a crane coordinate system with the rotation center of the boom 9 as the origin.

[0051] Furthermore, the image processing device 23 acquires the target coordinate T in the coordinate system of the display screen of the target position input from the display device 21. Furthermore, the image processing device 23 displays, superimposed on the image P, a predetermined movement path R (see FIG. 4) from the tip coordinate B to the target coordinate T. The image processing device 23 converts the target coordinate T into a converted target coordinate Tc, which is a coordinate in the crane coordinate system, based on the height of the boom camera 9b and the viewing angle information Ii of the boom camera 9b. The image processing device 23 transmits the converted tip coordinate Bc, the converted rotation center coordinate Cc, and the converted target coordinate Tc to the control device 24.

[0052] The image processing device 23 calculates the current operation position coordinate Pa and the target operation position coordinate Pb of the crane operating device 19 in the coordinate system of the display screen based on the current operation signal Sa and the target operation signal Sb of the crane operating device 19 acquired from the control device 24. The image processing device 23 displays a figure representing the crane operating device 19 at the current operation position coordinate Pa of the crane operating device 19 in the calculated coordinate system of the display screen. Similarly, the image processing device 23 displays a figure representing the crane operating device 19 at the target operation position coordinate Pb of the crane operating device 19 in the calculated coordinate system of the display screen. In other words, the image processing device 23 displays the operation amount, operation direction and movement order of the crane operating device 19 for moving the tip of the boom 9 from the current position to the target position along a predetermined movement route R.

[0053] As shown in Figures 2 and 3, the control device 24 controls each actuator of the crane 1. The control device 24 is provided in the cabin 20. The control device 24 may be configured in such a way that a CPU, a ROM, a RAM, a HDD, etc. are connected via a bus, or may be configured as a one-chip LSI, etc. The control device 24 stores various programs and data for controlling the operations of the actuators, the switching valve, the crane device sensor 17, etc., and for processing image data. The control device 24 includes an image processing device 23.

[0054] The control device 24 is connected to the traveling operation tool 18 and the crane operation tool 19. The control device 24 can acquire the operation signals of the traveling operation tool 18 and the crane operation tool 19. The control device 24 generates a control signal for the vehicle 2 based on the operation signal generated by the operation of the traveling operation tool 18. Similarly, the control device 24 generates a control signal for the crane 6 based on the operation signal generated by the operation of the crane operation tool 19. The control device 24 can transmit the generated control signal to each actuator of the crane 6. The control device 24 is connected to the crane sensor 17. The control device 24 can control each actuator based on the detection value of the crane sensor 17. The control device 24 can also acquire the attitude information Ip of the boom 9 by the crane sensor 17.

[0055] The control device 24 is connected to the image processing device 23 by wire or wirelessly. The control device 24 can transmit posture information Ip of the boom 9 to the image processing device 23. The control device 24 can acquire, from the image processing device 23, a transformed tip coordinate Bc which is the coordinate of the tip of the boom 9 in the crane coordinate system, a transformed rotation center coordinate Cc which is the coordinate of the rotation center of the boom 9, and a transformed target coordinate Tc of the tip of the boom 9.

[0056] The control device 24 can generate a current operation signal Sa of the crane operating tool 19 in the current attitude of the boom 9 and a target operation signal Sb of the crane operating tool 19 for the tip of the boom 9 to reach the conversion target coordinate Tc along a predetermined moving path R at a predetermined speed from the acquired conversion tip coordinate Bc and conversion target coordinate Tc. The control device 24 can transmit the generated current operation signal Sa and target operation signal Sb of the crane operating tool 19 to the image processing device 23. That is, the control device 24 can calculate the operation amount and operation direction of the crane operating tool 19 for moving the load W along the predetermined moving path R at a predetermined speed. In this embodiment, the predetermined moving path R is a path connecting the conversion tip coordinate Bc to the conversion target coordinate Tc with a straight line (see FIG. 4). The control device 24 may also be configured to calculate the operation order of each operating tool of the crane operating tool 19.

[0057] The crane 1 thus configured can move the vehicle 2 to any position by operating the traveling operation tool 18. The crane 1 can also transport the load W to any position by rotating, raising, lowering, and extending the boom 9 by operating the crane apparatus operation tool 19. The crane 1 can also hoist or lower the load W with the main winch 13 or the like by operating the crane apparatus operation tool 19. Meanwhile, the crane 1 can generate an operation signal for the crane apparatus operation tool 19 for moving the tip of the boom 9 along a predetermined movement path R to a target position set based on the image P.

[0058] The input device 22, the display device 21 and the image processing device 23 may be configured as a remote control terminal that is detachable from the cabin 20. The remote control terminal further includes a communication device and the like, and is configured to be capable of communicating between the image processing device 23 and the control device 24. This allows the crane 1 to operate the boom 9 from a position away from the cabin 20.

[0059] 3 to 5, the image processing by the image processing device 23 and the control device 24 will be described below. Note that the image P captured by the boom camera 9b includes the work site around the crane 1. The crane 1 is in a state where the load W is lifted up to a predetermined height by the sub winch 15.

[0060] As shown in Figures 4 and 5, the display device 21 (see Figure 3) has an image display area in which the image P captured by the boom camera 9b is displayed, and an operating tool display area that displays the operating positions of the rotation operating tool for operating the rotation hydraulic motor 8, the extension operating tool for operating the extension hydraulic cylinder, the hoisting operating tool for operating the hoisting hydraulic cylinder 12, the main operating tool for operating the main winch 13, and the sub operating tool for operating the sub winch 15.

[0061] The operation tool display area displays the movable range of each crane operation tool 19, and graphics (see black circles) indicating the current operation position P1a of the rotation operation tool, the current operation position P2a of the extension operation tool, the current operation position P3a of the hoisting operation tool, the current operation position P4a of the main operation tool, and the current operation position P5a of the sub operation tool that constitute the crane operation tool 19. The operation tool display area can also display graphics (see two-dot chain circles) indicating the target operation position P1b of the rotation operation tool, the target operation position P2b of the extension operation tool, the target operation position P3b of the hoisting operation tool, the target operation position P4b of the main operation tool, and the target operation position P5b of the sub operation tool for moving the tip of the boom 9 to the target position.

[0062] As shown in Fig. 3, the image processing device 23 acquires an image P of the tip of the boom 9 of the crane 1 captured by the boom camera 9b every unit time. In addition, the image processing device 23 acquires posture information Ip of the boom 9 from the control device 24 every unit time. The image processing device 23 displays the acquired image P (see Figs. 4 and 5) in the image display area of ​​the display device 21. The operator inputs a target position of the tip of the boom 9 from the image P. The image processing device 23 calculates a target coordinate T in the coordinate system of the display screen of the input target position.

[0063] The image processing device 23 calculates the tip coordinate B of the boom 9 in the coordinate system of the display screen, with the center position of the image P being the tip position of the boom 9. Next, the image processing device 23 calculates the rotation center coordinate C of the boom 9 in the coordinate system of the display screen. As shown in Figs. 4 and 5, the image processing device 23 superimposes and displays the calculated tip coordinate B, rotation center coordinate C, and target coordinate T on the image P. Furthermore, the image processing device 23 superimposes and displays on the image P a predetermined movement path R from the tip coordinate B to the target coordinate T superimposed and displayed on the image P. It should be noted that in this embodiment, the image P does not include the rotation center coordinate C. Therefore, the rotation center coordinate C is not displayed on the image P.

[0064] 3, the image processing device 23 calculates the converted tip coordinate Bc, the converted rotation center coordinate Cc, and the converted target coordinate Tc in the crane coordinate system from the height of the boom camera 9b and the angle of view information Ii of the boom camera 9b. The image processing device 23 transmits the converted rotation center coordinate Cc, the converted target coordinate Tc, and the converted target coordinate Tc to the control device 24. Note that although the crane coordinate system is an orthogonal coordinate system in this embodiment, it may be a polar coordinate system.

[0065] The control device 24 generates a target operation signal Sb of the crane operating tool 19 for moving the tip of the boom 9 to the transformed target coordinate Tc along a predetermined movement path R, based on the transformed swing center coordinate Cc and the transformed tip coordinate Bc. The control device 24 transmits the generated target operation signal Sb of the crane operating tool 19 to the image processing device 23. The image processing device 23 calculates a target operation position coordinate Pb of the crane operating tool 19 in the coordinate system of the display screen, based on the target operation signal Sb of the crane operating tool 19 acquired from the control device 24.

[0066] The image processing device 23 displays figures indicating the target operating position P1b of the rotation operating tool, the target operating position P2b of the extension operating tool, the target operating position P3b of the elevation operating tool, the target operating position P4b of the main operating tool, and the target operating position P5b of the sub operating tool in the operating tool display area.

[0067] The image processing device 23 may display the operation order of the turning operating tool, the telescopic operating tool, the elevation operating tool, the main operating tool, and the sub operating tool. For example, the image processing device 23 displays the operation order of each operating tool in numbers such as (1), (2), (3), etc. in the operating tool display area of ​​each operating tool. When multiple operating tools are operated simultaneously, the image processing device 23 displays the same operation order for the operating tools operated simultaneously. In this embodiment, the image processing device 23 displays the same operation order "(2)" for the telescopic operating tool and the main operating tool operated simultaneously.

[0068] With this configuration, the crane 1 can set the target position of the boom 9 while checking the surrounding conditions with the image P. The crane 1 can also visually recognize the amount and direction of operation of the crane apparatus operating tool 19 required to move the tip of the boom 9 to the target position. The crane 1 also displays the operation sequence of the crane apparatus operating tool 19, allowing the user to visually recognize the operation sequence in addition to the amount and direction of operation of each operating tool.

[0069] As shown in Fig. 5, when the crane operating tool 19 included in the crane operating tool 19 is operated, the control device 24 acquires a current operation signal Sa for each operating tool. The control device 24 transmits the acquired current operation signal Sa to the image processing device 23. The image processing device 23 calculates a current operating position coordinate Pa of the crane operating tool 19 in the coordinate system of the display screen based on the current operation signal Sa of the crane operating tool 19 acquired from the control device 24. The image processing device 23 sets the calculated current operating position coordinate Pa of the crane operating tool 19 as a new current operating position coordinate Pa of the crane operating tool 19 after operation, and displays a figure showing the crane operating tool 19 at the current operating position coordinate Pa.

[0070] With this configuration, the figure representing the crane equipment operating tool 19 displayed on the display device 21 moves as the crane 1 operates the crane equipment operating tool 19. This allows the operator of the crane 1 to operate the crane equipment operating tool 19 while checking the situation around the crane 1 and the difference between the target operating position of the crane equipment operating tool 19 and the current operating position.

[0071] When at least one of the figures showing the current operating position P1a of the turning operating tool, the current operating position P2a of the telescopic operating tool, the current operating position P3a of the hoisting operating tool, the current operating position P4a of the main operating tool, and the current operating position P5a of the sub operating tool displayed in the operating tool display area is moved, the image processing device 23 generates a current operation signal Sa with the coordinates of the moved figure in the coordinate system of the display screen as the current operating position coordinates Pa of the corresponding operating tool. The image processing device 23 transmits the generated current operation signal Sa to the control device 24. The control device 24 transmits a control signal to the corresponding actuator based on the acquired current operation signal Sa.

[0072] With this configuration, the tip of the boom 9 of the crane 1 moves in accordance with the movement of the figures on the screen of the display device 21. The crane 1 can move the tip of the boom 9 to the target position along a predetermined movement path R while checking the situation around the crane 1 by moving, on the screen of the display device 21, the figures indicating the current operating position P1a of the telescopic operating tool, the current operating position P2a of the telescopic operating tool, the current operating position P3a of the hoisting operating tool, the current operating position P4a of the main operating tool, and the current operating position P5a of the sub operating tool to the target operating position coordinates Pb.

[0073] Therefore, the operator of the crane 1 can set the target position of the tip of the boom 9 on the image P displayed on the display device 21 while checking the situation around the crane 1 in real time. Moreover, the image processing device 23 calculates the converted tip coordinate Bc, the converted rotation center coordinate Cc, and the converted target coordinate Tc in the crane coordinate system, with the center position of the image P photographed by the boom camera 9b provided at the tip of the boom 9 being the tip position of the boom 9. Furthermore, the image processing device 23 displays the current operating position and the target operating position of the crane apparatus operating tool 19 that operates each actuator so that the tip of the boom 9 reaches the target position on a linear moving path R. The operator can move the tip of the boom 9 to the target position with a simple operation based on the information on the operation of the crane apparatus operating tool 19 while grasping the situation of the work site in real time using the crane 1. Therefore, the crane 1 allows even an operator with low skill to visually grasp the operation amount of each operating tool to which the actuators with different moving speeds are assigned. The image processing device 23 may be configured to display the operation order of the crane apparatus operating tool 19. This allows the crane 1 to move the tip of the boom 9 along the linear movement path R regardless of the operator's level of skill.

[0074] Next, a crane 1A according to a second embodiment of the present invention will be described with reference to Figures 6 to 8. Note that the crane 1A according to the following embodiments will refer to the same crane 1 shown in Figures 1 to 5, and the names, drawing numbers, and symbols used in the description will be used to refer to the same thing. Therefore, in the following embodiments, specific descriptions of the same points as in the embodiments already described will be omitted, and the description will focus on the differences.

[0075] As shown in Fig. 6, the crane 1A has an image capturing device 25, a remote control terminal 28 (see Fig. 7), and an image processing device 30 (see Fig. 7). The image capturing device 25 captures images of the crane 1A and the surroundings of the crane 1A at the work site. The image capturing device 25 includes an unmanned aerial vehicle 26 and a camera 27.

[0076] The unmanned aerial vehicle 26 is an autonomous unmanned aerial vehicle (drone) that can be remotely controlled and fly autonomously by a control signal. The unmanned aerial vehicle 26 is, for example, a multicopter having multiple propellers. The unmanned aerial vehicle 26 is configured to be able to transmit and receive control signals from the remote control terminal 28. This allows the unmanned aerial vehicle 26 to be moved to any position by the remote control terminal 28. The unmanned aerial vehicle 26 may be configured to follow a beacon attached to the boom 9 of the crane 1A, an image of the boom 9 detected by the image processing device 30, an AR tag attached to the boom 9, or the like.

[0077] The camera 27 is provided on the unmanned aerial vehicle 26. The camera 27 is configured to capture an image vertically below the unmanned aerial vehicle 26. The camera 27 is configured to be able to acquire control signals and the like from the remote control terminal 28. The camera 27 is also configured to be able to transmit captured image data to the image processing device 30. This allows the camera 27 to perform operations such as starting and stopping capture, determining the capture direction, and zooming, via the remote control terminal 28.

[0078] The remote control terminal 28 is an operation terminal that operates the unmanned air vehicle 26 and the camera 27. The remote control terminal 28 is arranged in the cabin 20. The remote control terminal 28 is provided with an operation tool for the unmanned air vehicle 26 and an operation tool for the camera 27 in a portable housing. In other words, the remote control terminal 28 is configured as a portable operation terminal that can be carried outside the cabin 20. The remote control terminal 28 can place the unmanned air vehicle 26 above the crane 1A by operating the operation tool, and can hover at any position. The remote control terminal 28 is configured to be able to transmit and receive control signals to the unmanned air vehicle 26 and the camera 27. In addition, the remote control terminal 28 is configured to be able to transmit and receive input signals, operation signals, etc. to the control device 24 of the crane 1A. The remote control terminal 28 is provided with a display device 21. The display device 21 may be arranged in the cabin 20 independently of the remote control terminal 28.

[0079] As shown in Fig. 7, the communication device 29 transmits and receives image data and control signals. The communication device 29 is provided in the remote control terminal 28. The communication device 29 can transmit and receive control signals for the unmanned air vehicle 26 and the camera 27 of the photographing device 25 to and from the image processing device 30. The communication device 29 can also transmit the image P captured by the camera 27 to the image processing device 30. The communication device 29 can transmit and receive control signals between the unmanned air vehicle 26 and the camera 27. The communication device 29 can receive the image P captured by the camera 27.

[0080] The image processing device 30 is provided in the remote control terminal 28. The image processing device 30 is connected to the display device 21. The image processing device 30 can acquire operation signals of the unmanned aerial vehicle 26 and the camera 27 from the display device 21. The image processing device 30 generates a control signal for the unmanned aerial vehicle 26 based on the operation signal for the unmanned aerial vehicle 26 input from the display device 21. Similarly, the image processing device 30 generates a control signal for the camera 27 based on the operation signal for the camera 27 input from the display device 21. The image processing device 30 is connected to a communication device 29. The image processing device 30 transmits the generated control signal to the unmanned aerial vehicle 26 or the camera 27 via the communication device 29.

[0081] The image processing device 30 detects the tip of the boom 9 from the image P captured by the camera 27 based on a learning model stored in advance. In this embodiment, the image processing device 30 uses vectorization by a color histogram of the image P, vectorization by an autoencoder, vectorization by a local feature, and the like in combination to calculate feature amounts by each vectorization. Vectorization by a color histogram is a method of vectorizing the color intensity and occurrence frequency as feature amounts. Vectorization by an autoencoder is a method of using the intermediate layer of the autoencoder as a feature amount representing the image P. The autoencoder is a three-layer neural network in which supervised learning is performed using the same data for the input layer and the output layer. Vectorization by a local feature is a method of vectorizing the image P using the KAZE local feature amount.

[0082] The image processing device 30 calculates three feature amounts by vectorization using a color histogram of the image P, vectorization using an autoencoder, and vectorization using a local feature amount. Note that the feature amount may be a luminance value, an edge histogram, a correlation function between color and edges, or the like, contained in the image P. The image processing device 30 may also calculate local feature amounts of the image P, such as HOG (Histogram of Oriented Gradients), SIFT (Scale Invariant Feature Transform), or the like.

[0083] Next, detection of the tip of the boom 9 and the extension direction of the boom 9 in the image processing device 30 of the crane 1A will be described with reference to Fig. 8. The image processing device 30 has a reference image holding unit 30a, a judgment image detection unit 30b, and a posture judgment unit 30c. The image processing device 30 has a learning model that has been subjected to supervised learning of a predetermined range of the tip of the boom 9. The image processing device 30 can detect the tip of the boom 9 and the extension direction of the boom 9 from three feature amounts of the image P captured by the camera 27.

[0084] As shown in FIG. 8, the reference image storage unit 30a is a part of the image processing device 30, and stores a reference image Is of the tip of the boom 9 in advance. The reference image Is is an image that serves as a reference for determining the tip of the boom 9. The reference image Is is composed of at least a part of an image of the tip of the boom 9 in an arbitrarily determined reference posture. In other words, the reference image Is is an image that shows the reference posture of the tip of the boom 9. In this embodiment, the reference posture is the boom 9 in a predetermined posture. In other words, the reference image Is is composed of at least a part of an image taken by the imaging device 25 in a predetermined posture from above, at a predetermined distance, of the tip of the boom 9 at a predetermined elevation angle, extension length, and rotation angle.

[0085] The judgment image detection unit 30b is part of the image processing device 30, and detects an image of a portion of the current image P that corresponds to the reference image Is as a judgment image Ij. The judgment image Ij is an image of the tip portion of the boom 9 in the current image P relative to the reference attitude of the tip portion of the boom 9. In other words, the judgment image Ij is an image for calculating the relative attitude of the tip portion of the boom 9 in the current image P with respect to the reference image Is. The judgment image Ij is composed of an image of at least a portion of the tip portion of the boom 9 in the current image P.

[0086] The judgment image detection unit 30b can calculate feature points of the selected reference image Is. Furthermore, the judgment image detection unit 30b can calculate feature points from the current image P. The judgment image detection unit 30b can compare the feature points of the reference image Is with the feature points of the current image P, and detect a portion of the current image P that corresponds to the reference image Is as a judgment image Ij.

[0087] The attitude determination unit 30c is a part of the image processing device 30, and determines the current orientation D of the extension direction of the boom 9. The attitude determination unit 30c is configured to be able to calculate the shape, direction, and inclination of the determination image Ij relative to the reference image Is by comparing the reference image Is with the determination image Ij. The attitude determination unit 30c can calculate the tip coordinates B of the boom 9 in the current coordinate system of the display screen and the orientation D of the extension direction of the boom 9 relative to the reference attitude of the tip portion of the boom 9 in the coordinate system of the display screen from the coordinates of each part of the reference image Is in the coordinate system of the display screen and the coordinates of each part of the determination image Ij in the coordinate system of the display screen.

[0088] The image processing device 30 can calculate the turning center coordinate C, which is the coordinate of the turning center of the boom 9 in the coordinate system of the display screen, from the attitude information Ip of the boom 9, such as the size in the detected image P of the tip portion of the boom 9, the extension direction of the boom 9, the hoisting angle of the boom 9, the telescopic length of the boom 9, the turning angle of the boom 9, and the shape and dimensions of the boom 9. The image processing device 30 superimposes and displays the calculated tip coordinate B, turning center coordinate C, and direction D of the boom 9 on the image P displayed on the display device 21.

[0089] The image processing device 30 can convert the coordinate system of the display screen into a crane coordinate system with the boom 9's rotation center as the origin, based on the tip coordinate B of the boom 9, the boom 9's rotation center coordinate C, and the boom 9's attitude information Ip in the coordinate system of the display screen. The image processing device 30 converts the tip coordinate B and the rotation center coordinate C in the coordinate system of the display screen into converted tip coordinate Bc and converted rotation center coordinate Cc in the crane coordinate system. The image processing device 30 transmits the converted tip coordinate Bc, converted rotation center coordinate Cc, and converted target coordinate Tc to the control device 24.

[0090] When performing image processing using the image processing device 30, the crane 1A hovers the unmanned aerial vehicle 26 of the imaging device 25 above the tip of the boom 9 before the start of transportation (see FIG. 6). Even if the boom 9 moves, the unmanned aerial vehicle 26 maintains the hovering position it was in before the boom 9 moved. At this time, the image processing device 30 performs sway correction so that the position on the screen of a target object such as a structure that has been automatically detected or designated by the operator does not fluctuate.

[0091] The image processing device 30 acquires, via the communication device 29, an image P taken from above the crane 1A by the camera 27 and information relating to the position, attitude, etc. of the image taking device 25 every unit time (see FIG. 4). In addition, the image processing device 30 acquires attitude information Ip of the boom 9 from the control device 24 every unit time. The image processing device 30 causes the display device 21 to display the acquired image P.

[0092] The image processing device 30 detects a determination image Ij, which is an image of the tip of the boom 9, from the latest image P captured by the camera 27. The image processing device 30 stores in advance a reference image Is, which is an image of the tip of the boom 9 detected using a Convolutional Neural Network (CNN) from images of the tip of the boom 9 of various cranes 1A (see FIG. 4).

[0093] The image processing device 30 detects a determination image Ij, which is an image of the tip of the boom 9, from the latest image P by image classification using CNN based on the reference image Is. From the detected determination image Ij and attitude information Ip of the boom 9, the image processing device 30 calculates the direction D of the axis of the boom 9 in the determination image Ij, a tip coordinate B indicating the position of the tip of the boom 9 on the axis in the coordinate system of the display screen, and a rotation center coordinate C indicating the rotation center of the boom 9 in the coordinate system of the display screen.

[0094] Specifically, the image processing device 30 calculates the length from the tip to the base of the boom 9 in the coordinate system of the display screen in which the boom 9 is photographed from above, from the width of the tip of the boom 9, the boom hoisting angle, and the extension length of the boom 9, which are the posture information Ip of the boom 9. The image processing device 30 calculates the rotation center coordinate C from the calculated tip coordinate B, the axis direction D, and the length from the tip to the base of the boom 9. As a result, the image processing device 30 can convert the coordinate system of the display screen into a crane coordinate system with the rotation center of the boom 9 as the origin, from the tip coordinate B, the rotation center coordinate C, the axis direction D, and the posture information Ip of the boom 9. The image processing device 30 converts the tip coordinate B and the rotation center coordinate C in the coordinate system of the display screen into the converted tip coordinate Bc and the converted rotation center coordinate Cc in the crane coordinate system. In this embodiment, the crane coordinate system is an orthogonal coordinate system, but it may be a polar coordinate system.

[0095] The operator inputs the target position of the tip of the boom 9 on the image P. The operator sets the target position while checking the status of the boom 9 of the crane 1A displayed in the image P, the lifting position of the load W, and the installation position of the load W from above the crane 1A. This enables the operator to intuitively set a transfer route that avoids obstacles, taking into account the layout of structures in the work site.

[0096] With this configuration, the image processing device 30 automatically calculates the rotation center coordinate C of the crane 1A and the tip coordinate B of the boom 9 by, for example, comparing the image P including the crane 1A and the surroundings of the crane 1A captured by the camera 27 with a previously acquired image P of the tip of the boom 9. By calculating the rotation center coordinate C and tip coordinate B of the boom 9 using the image processing device 30, the control device 24 can convert the coordinate system of the display screen into a crane coordinate system based on the rotation center of the crane 1A.

[0097] In the above embodiment, the image P may include an AR marker. The image processing device 30 calculates the tip coordinate B and the turning center coordinate C of the boom 9 from the shape of the boom 9 included in the image P. However, if the image P includes an image of an AR marker provided on a part of the boom 9, the image processing device 30 can calculate the tip coordinate B and the turning center coordinate C of the boom 9 from the position of the AR marker in the image P. The image processing device 30 can also calculate the turning center coordinate C from the yaw angle of the AR marker. The image processing device 30 can also calculate the elevation angle from the roll angle and pitch angle of the AR marker.

[0098] Next, a crane 1B according to a third embodiment of the present invention will be described with reference to Figures 9 and 10. Note that the crane 1B according to the following embodiments is assumed to be the same as the cranes 1 and A1 shown in Figures 1 to 8, and the names, drawing numbers, and symbols used in the description will be used to indicate the same thing.

[0099] 9, the crane 1B is a mobile crane that can be moved to any location. The crane 1B includes a vehicle 2, a crane device 6, an imaging device 25, a display device 21, a first GNSS receiver 31, a direction sensor 32, a second GNSS receiver 33, a communication device 34, an image processing device 30, and a control device 24.

[0100] The first GNSS receiver 31 is a receiver constituting a global navigation satellite system. The first GNSS receiver 31 receives ranging radio waves from a satellite and calculates the absolute coordinates of the receiver, that is, latitude, longitude, and altitude. The first GNSS receiver 31 is provided at the rotation center of the boom 9. The crane 1B can obtain the absolute coordinates of the rotation center of the boom 9 by the first GNSS receiver 31. In this embodiment, the first GNSS receiver 31 detects the absolute coordinates of the rotation center of the boom 9, but it is sufficient that the position is such that the attitude of the boom 9 can be specified. For example, the first GNSS receiver 31 may be configured to detect the absolute coordinates of the tip of the boom 9.

[0101] The orientation sensor 32 detects the orientation. The orientation sensor 32 is composed of a geomagnetic sensor such as an MR sensor or an MI sensor, a GNSS receiver having two antennas, and the like. The orientation sensor 32 detects the orientation in the extension direction of the boom 9. The orientation sensor 32 is provided at the rotation center of the boom 9 or at the tip of the boom 9. The first GNSS receiver 31 may also serve as the orientation sensor 32.

[0102] The second GNSS receiver 33 is a receiver that constitutes a global positioning satellite system. The second GNSS receiver 33 receives ranging radio waves from satellites and calculates the absolute coordinates of the receiver, which are latitude, longitude, and altitude. The second GNSS receiver 33 is provided in the unmanned air vehicle 26 of the imaging device 25. The crane 1B can obtain the absolute coordinates of the camera 27 provided in the unmanned air vehicle 26 by using the second GNSS receiver 33.

[0103] The communication device 34 can receive the absolute coordinates calculated by the first GNSS receiver 31. The communication device 34 can receive the absolute coordinates calculated by the second GNSS receiver 33.

[0104] As shown in FIG. 10, the image processing device 30 processes an image P captured by the camera 27. The image processing device 30 can convert the coordinate system of the display screen into an absolute coordinate system from the absolute rotation center coordinate Ca, which is the absolute coordinate of the rotation center received by the first GNSS receiver 31, the direction D of the extension direction of the boom 9 calculated by the direction sensor 32, the absolute camera coordinate Ia, which is the absolute coordinate of the camera 27 received by the second GNSS receiver 33, and the attitude information Ip of the boom 9. The image processing device 30 can calculate the absolute tip coordinate Ba, which is the absolute coordinate of the tip of the boom 9. In addition, the image processing device 30 can calculate the absolute target coordinate Ta from the target coordinate T of the tip of the boom 9 input to the display device 21 using a conversion formula. The image processing device 30 transmits the calculated absolute target coordinate Ta to the control device 24.

[0105] When performing image processing using the image processing device 30, the crane 1B hovers the unmanned aerial vehicle 26 of the imaging device 25 above the tip of the boom 9 before the start of transportation (see FIG. 6). Even if the boom 9 moves, the unmanned aerial vehicle 26 maintains the hovering position it was in before the boom 9 moved. At this time, the image processing device 30 performs sway correction so that the position on the screen of a target object such as a structure that is automatically detected or designated by the operator does not fluctuate.

[0106] The image processing device 30 of the crane 1B acquires an image P of the crane 1B taken by the camera 27 from above every unit time via the communication device 34. In addition, the image processing device 30 acquires posture information Ip of the boom 9 from the control device 24 every unit time. The image processing device 30 causes the display device 21 to display the acquired image P.

[0107] The image processing device 30 acquires the absolute turning center coordinate Ca from the first GNSS receiver 31, acquires the direction D of the extension direction of the boom 9 from the direction sensor 32, and acquires the absolute camera coordinate Ia from the second GNSS receiver 33. The image processing device 30 calculates the absolute tip coordinate Ba from the absolute turning center coordinate Ca, the direction D of the extension direction of the boom 9, and the attitude information Ip of the boom 9 based on the positional relationship between the absolute camera coordinate Ia and the absolute turning center coordinate Ca.

[0108] The image processing device 30 calculates the target coordinate T input by the operator from the screen of the display device 21. The image processing device 30 calculates the absolute target coordinate Ta from the calculated target coordinate T based on the absolute camera coordinate Ia and the angle of view information Ii of the camera 27. The image processing device 30 transmits the calculated absolute target coordinate Ta to the control device 24.

[0109] With this configuration, the control device 24 calculates the absolute tip coordinate Ba from the absolute rotation center coordinate Ca of the boom 9, the orientation D of the extension direction of the boom 9, the absolute camera coordinate Ia, the angle of view information Ii of the camera 27, and the attitude information Ip of the boom 9. The control device 24 also calculates a conversion formula for converting coordinates in the coordinate system of the display screen into absolute coordinates based on the absolute camera coordinate Ia and the angle of view information Ii of the camera 27. This makes it possible to convert the coordinate system of the display screen into the absolute coordinate system.

[0110] In each of the above-described embodiments, the cranes 1, 1A, 1B are configured to set a target position on the image P displayed on the display device 21, but the cranes may also be configured to set the coordinates of another work vehicle as the target coordinates T when position information including the coordinates of the other work vehicle is acquired.

[0111] As another embodiment of the present invention, when the other work vehicle (for example, a crane, aerial work platform vehicle, truck, etc.) is equipped with a GNSS receiver, the control device 24 of the cranes 1, 1A, 1B may be configured to acquire position information from the GNSS receiver of the other work vehicle via a communication device or the like. In addition, when the other work vehicle has an AR marker, the control device 24 of the cranes 1, 1A, 1B may be configured to acquire position information of the other work vehicle by photographing the AR marker with the photographing device 25. The control device 24 calculates the conversion target coordinates Tc, which are the coordinates of the other work vehicle in the crane coordinate system of the cranes 1, 1A, 1B, from the acquired position information of the other work vehicle. In addition, the control device 24 generates an operation signal for the crane apparatus operating tool 19 for the tip of the boom 9 to reach the conversion target coordinates Tc along a predetermined moving route R at a predetermined speed.

[0112] The control device 24 displays on the display device 21 information related to the operation of the crane apparatus operating tool 19 required to move the tip of the boom 9 along a predetermined movement path R with the other work vehicle as a target position. The operator can operate the cranes 1, 1A, 1B while grasping the situation at the work site in real time using the imaging device 25. The operator can also move the tip of the boom 9 to the other work vehicle by operating the crane apparatus operating tool 19 according to a target operation position and operation sequence relative to the current operation position of the crane apparatus operating tool 19. This allows the work vehicle to move the tip of the boom 9 along the predetermined movement path R regardless of the operator's level of skill.

[0113] In each of the above-described embodiments, the control device 24 of the cranes 1, 1A, 1B transmits control signals to each actuator of the crane apparatus 6 while the crane apparatus operating tool 19 is being operated or the graphic representing the position of the crane apparatus operating tool 19 on the display device 21 is being moved. However, the control device 24 may be configured to stop the movement of the boom 9 when the tip of the boom 9 moves away from the predetermined movement path R by a predetermined distance or more, or to notify that effect.

[0114] In each of the above-mentioned embodiments, the image processing device of the cranes 1, 1A, 1B displays the current operating position and the target operating position of each operating tool as a graphic on the display device 21, but is not limited to this. The image processing device may be configured to display the current operating position and the target operating position of each operating tool in a form that allows the current operating position and the target operating position to be distinguished and visually recognized, such as by displaying them with a level indicator, by displaying them with an arrow, or by changing the shape and color of the graphic.

[0115] In addition, in each of the above-mentioned embodiments, the control device 24 of the cranes 1, 1A, 1B may be configured to inform the operator that the current operating position of each operating tool matches the target operating position by displaying a figure, color, letter, icon, etc., alerting with a sound, vibrating the operating tool, fixing the operating tool, increasing the operating resistance of the operating tool, etc.

[0116] In each of the above-described embodiments, the control device 24 displays on the display device 21 information relating to the operation of the crane apparatus operating tool 19 required for moving the tip of the boom 9 along a predetermined movement path R with the other work vehicle as the target position. However, the control device 24 may set a person or an object as the target position as long as it can recognize the positional relationship with the cranes 1, 1A, 1B using an AR marker or the like.

[0117] The above-mentioned embodiment is merely a representative embodiment, and various modifications can be made without departing from the gist of the embodiment. Of course, the present invention can be embodied in various other forms, and the scope of the present invention is defined by the claims, and includes the equivalent meanings of the claims and all modifications within the scope of the claims. [Explanation of symbols]

[0118] 1. Crane 6 Crane equipment 9. Boom 9b Boom Camera 17 Sensors for crane equipment 18 Driving controls 19 Crane equipment operating tools 21 Display device 22 Input Devices 23 Image Processing Device 24 Control device P Image C Turning center coordinates B Tip coordinate T target coordinates Sa Current operation signal Sb Current operation signal Pa Current operation position coordinates Pb Target operation position coordinates

Claims

1. A work vehicle having a boom that can be raised and lowered via a swivel on a traveling body; An imaging device that images the surroundings of the work vehicle from above; a display device that displays the image captured by the image capture device and is capable of outputting coordinates of any position on the display screen that is specified from outside; A control device for controlling an actuator of the work vehicle, The control device includes: a work vehicle that acquires the position of the tip of the boom or the position of the boom's rotation center relative to the imaging device, attitude information of the boom, angle of view information of the image, and an image of the surroundings of the work vehicle captured from above by the imaging device, displays the image on the display device, calculates the coordinate of the tip of the boom in the coordinate system of the display screen and the coordinate of the boom's rotation center in the coordinate system of the display screen, calculates the coordinate in the coordinate system of the display screen of a target position of the tip of the boom set on the image, and displays current operating positions and target operating positions of each operating tool of the actuator so that the tip of the boom reaches the target position via a predetermined movement path.

2. The work vehicle according to claim 1 , wherein the control device displays an operation sequence of each operating tool of the actuator.

3. The imaging device is Provided at the tip of the boom, The control device includes:

3. A work vehicle as described in claim 1 or claim 2, wherein the center position of the image is the tip position of the boom, and the coordinates of the tip of the boom in the coordinate system of the display screen and the coordinates of the boom's rotation center in the coordinate system of the display screen are calculated from the boom posture information.

4. The control device includes:

3. The work vehicle according to claim 1, wherein an image of the tip of the boom is detected from an image captured by the photographing device, and the coordinates of the boom's center of rotation in the coordinate system of the display screen and the coordinates of the boom tip in the coordinate system of the display screen are calculated from the orientation and size of the image of the tip of the boom and attitude information of the boom.

5. A first GNSS receiver that detects absolute coordinates of a rotation center of the boom or absolute coordinates of a tip of the boom; An orientation sensor for detecting an orientation of the extension direction of the boom; A second GNSS receiver that detects absolute coordinates of the imaging device, The control device includes:

3. The work vehicle according to claim 1 or 2, further comprising: a detection value of the first GNSS receiver, a detection value of the second GNSS receiver, a detection value of the orientation sensor, and attitude information of the boom; and an absolute coordinate of a rotation center of the boom and an absolute coordinate of a tip of the boom are calculated by acquiring the detection value of the first GNSS receiver, the detection value of the second GNSS receiver, the detection value of the orientation sensor, and attitude information of the boom.

6. The work vehicle according to claim 1 , wherein the operating positions of the operating devices of the actuators are displayed by graphics.

7. The display device includes: The work vehicle according to claim 1 , wherein an operating tool for the actuator is operable.

8. The control device includes: The work vehicle according to claim 1 , wherein when coordinates of another work vehicle are acquired, the position of the other work vehicle is set as the target position.

Images