Guidance information display device and crane provided with the device, guidance information display method

Abstract

Disclosed is a guide information display device, a crane provided with the guide information display device, and a guide information display method. Information related to the shape and position of a hoist object and a ground object existing around the hoist object is accurately presented regardless of the posture and movement of the crane. In the guide information display device (50), a camera (61) photographs a partial area of a work area (SA) of a crane (1), a laser scanner (62) acquires point group data (P) of the work area (SA) being photographed by the camera (61), and a data processing section (70) removes point group data (P) existing between a hoist object (W) suspended by the crane (1) and a top end portion of a telescopic boom 22 of the crane (1) from the point group data (P) acquired by the laser scanner (62), infers an upper surface of the hoist object (W), a ground surface (F) of the work area (SA), and an upper surface of a ground object (C) existing in the work area (SA) from the remaining point group data (P), and generates guide frames (GD1) respectively surrounding the upper surface of the hoist object (W) and the upper surface of the ground object (C), superimposes the guide frames (GD1) generated by the data processing section (70) on an image (M) photographed by the camera (61), and displays the image (M) on a data display section (80).

Description

[0001] This application is a divisional application of the application with the application number 201880047676.2, the application date 19 July 2018, and the invention title "Guidance information display device and crane provided with the device, guidance information display method". TECHNICAL FIELD

[0002] The present application relates to a technology of a guidance information display device that assists a crane operation performed by an operator, a crane provided with the guidance information display device, and a guidance information display method. BACKGROUND

[0003] In the related art, there is known a technology of achieving efficiency of a crane operation by confirming a hoist and a state of the surroundings of the hoist based on guidance information displayed on a monitor. Such a technology is disclosed in, for example, Patent Literature 1.

[0004] Patent Literature 1 discloses a technology related to a height information notification system that notifies of a height of an object in the surroundings of a hoist. In the height information notification system of the object in the surroundings of the hoist described in Patent Literature 1, a distance measuring unit such as a laser distance sensor, a microwave distance sensor, or a stereo camera is used to measure a distance from a top end of a hoist arm to the surroundings of the hoist. Further, the system is configured to detect a position of the object in the surroundings of the hoist (an object on the ground) and calculate a height using a result of the measurement of the distance, and to generate a processed image (guidance information) in which the position and the height of the object in the surroundings of the hoist are associated with a captured image captured by a camera, and display the guidance information on a monitor.

[0005] In the guidance information display device (height information notification system) described in Patent Literature 1, the position information and the height information of the hoist and the object in the surroundings of the hoist are transmitted to the operator by using the guidance information, so that contact of the hoist with the object in the surroundings is suppressed.

[0006] Related Art Documents

[0007] Patent Literature

[0008] Patent Literature 1: Japanese Patent Application Publication No. 2013-120176 SUMMARY

[0009] Problems to be Solved by the Invention

[0010] However, in the case where the steel wire rope exists in the measurement range of the distance measuring unit, the guidance information display device described in Patent Literature 1 measures not only the distance from the top end of the boom to the periphery of the hoisted object but also the distance from the top end of the boom to the steel wire rope. In addition, the guidance information display device displays guidance information including position information of the ground object around the hoisted object and the like on the monitor based on the measurement result of the distance measuring unit. Therefore, when the steel wire rope exists in the measurement range of the distance measuring unit, there is a case where the guidance information related to the ground object is affected by the measurement result of the steel wire rope, and thus the guidance information display device displays the position information of the steel wire rope mixed with the position information of the hoisted object and the object around the hoisted object. That is, in the existing guidance information display device, there is a case where the operator cannot accurately grasp the shape of the hoisted object and the ground object around the hoisted object based on the guidance information due to the posture and the motion of the crane.

[0011] The present application has been made in view of such a current problem, and has an object to provide a guidance information display device capable of accurately presenting information related to the shape and position of a hoisted object and a ground object around the hoisted object regardless of the posture and the motion of a crane.

[0012] Means for solving the problem

[0013] The problem to be solved by the present application is as described above, and the means for solving the problem will be described next.

[0014] That is, the guidance information display device according to the present application is a guidance information display device of a crane provided with a camera, a laser scanner, a data processing section, and a data display section, the camera photographs a part of a work area of the crane, the laser scanner acquires point group data of the work area being photographed by the camera, the data processing section removes point group data existing between a hoisted object hung by the crane and a top end portion of a boom of the crane from the point group data acquired by the laser scanner, and infers an upper surface of the hoisted object, a ground surface of the work area, and upper surfaces of ground objects existing in the work area based on the remaining point group data, and generates guidance frame images respectively surrounding the upper surface of the hoisted object and the upper surfaces of the ground objects, and further superimposes the guidance frame images generated by the data processing section on an image photographed by the camera and displays on the data display section.

[0015] Further, in the guide information display device of the present application, the data processing section estimates point group data in which the difference between the hoist and the top end of the boom of the crane and any elevation value is within a predetermined elevation difference as being on the same plane, thereby forming a plane cluster, calculates a straight line connecting the center of gravity of the hoist and the hoisting position of the hoist at the top end of the boom, calculates the hoist width from the center of gravity position of the hoist to the outermost circumference of the hoist from the center of gravity position when viewed from the direction of the straight line, calculates the horizontal distance between the center of gravity of the plane cluster and the straight line, and sets the point group data forming the plane cluster as point group data to be removed from the point group data acquired by the laser scanner when the horizontal distance is smaller than the hoist width or when the plane cluster overlaps the hoist when viewed from the vertical direction.

[0016] Further, in the guide information display device of the present application, the data processing section sets a removal region between the hoist and the top end of the boom, and sets the point group data acquired in the removal region as point group data to be removed from the point group data acquired by the laser scanner.

[0017] Further, the crane of the present application is provided with a guide information display device.

[0018] Further, in the guide information display method of the guide information display device according to the present application, the guide information display device includes a camera, a laser scanner, a data acquisition unit, a data processing unit, and a data display unit, the camera photographs a partial region of a work area of a crane, the laser scanner acquires point group data from above a hoisted object in the work area, the data acquisition unit acquires point group data of the work area photographed by the camera by the laser scanner, the data processing unit estimates an upper surface of the hoisted object, a ground surface of the work area, and an upper surface of a ground object present in the work area based on the point group data, and generates guide frame images that respectively surround the upper surface of the hoisted object and the upper surface of the ground object, and the data display unit superimposes and displays the guide frame images generated by the data processing unit on an image photographed by the camera, the guide information display method of the guide information display device includes a plane cluster acquisition step of acquiring a plane cluster formed by estimating point group data having a difference from an arbitrary elevation value within a predetermined elevation difference as being on the same plane, a contour line acquisition step of acquiring a contour line having the highest elevation value among contour lines of the plane cluster, a hoisting position calculation step of calculating a center of gravity of the contour line by regarding the center of gravity of the contour line as a hoisting position of the hoisted object at a top end portion of the boom, a straight line calculation step of calculating a straight line connecting the center of gravity of the hoisted object to the hoisting position, a horizontal distance calculation step of calculating a horizontal distance between a center of gravity of the plane cluster and the straight line, and a plane cluster removal step of removing point data forming the plane cluster from point group data acquired by the laser scanner in a case where the horizontal distance is smaller than a hoisted object width that is a distance from a center of gravity position of the hoisted object to a farthest outer periphery of the hoisted object when viewed from above in a straight line direction, or in a case where the hoisted object overlaps the plane cluster when viewed from above in a vertical direction.

[0019] The present application has the following advantageous effects.

[0020] As an effect of the present application, the following effects are provided.

[0021] According to the present application, information related to the shape and position of a hoisted object and a ground object present in the vicinity of the hoisted object can be accurately prompted without regard to the posture and movement of a crane. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1is a schematic view showing the overall structure of a crane, which is an application object of the guide information display device according to an embodiment of the present application.

[0023] Figure 2 is a schematic view showing the overall structure of a crane, which is an application object of the guide information display device according to an embodiment of the present application.

[0024] Figure 3 is a schematic view showing the overall structure of a crane, which is an application object of the guide information display device according to an embodiment of the present application.

[0025] Figure 4 is a schematic view showing the overall structure of a crane, which is an application object of the guide information display device according to an embodiment of the present application.

[0026] Figure 5 is a schematic view showing the overall structure of a crane, which is an application object of the guide information display device according to an embodiment of the present application.

[0027] Figure 6 is a schematic view showing the overall structure of a crane, which is an application object of the guide information display device according to an embodiment of the present application.

[0028] Figure 7 is a schematic view showing the overall structure of a crane, which is an application object of the guide information display device according to an embodiment of the present application. Figure 7 (A) in A-A sectional view.

[0029] Figure 8 is a schematic view showing the overall structure of a crane, which is an application object of the guide information display device according to an embodiment of the present application.

[0030] Figure 9 is a schematic view showing the overall structure of a crane, which is an application object of the guide information display device according to an embodiment of the present application.

[0031] Figure 10 is a schematic view showing the overall structure of a crane, which is an application object of the guide information display device according to an embodiment of the present application.

[0032] Figure 11 is a schematic view showing the overall structure of a crane, which is an application object of the guide information display device according to an embodiment of the present application.

[0033] Figure 12 is a schematic view showing the overall structure of a crane, which is an application object of the guide information display device according to an embodiment of the present application.

[0034] Figure 13is a graph showing point group data acquired by the data acquisition section, in which (A) is a graph in which the point group data is plotted on an XYZ coordinate system, and (B) is a graph in which the point group data plotted on the XYZ coordinate system is divided into a plurality of groups.

[0035] Figure 14 is a schematic view showing a case where a ground surface is designated by an operator.

[0036] Figure 15 is an explanatory view of a calculation method of a reference height at the time of designating a ground surface, in which (A) is a graph showing the calculation method of the reference height, and (B) is a schematic view showing a division case of a hoisting object region.

[0037] Figure 16 is an explanatory view of a generation method of a plane cluster.

[0038] Figure 17 is a flowchart showing an inference step of a plane.

[0039] Figure 18 is an explanatory view of an inference step (STEP-201 to 202) of a plane.

[0040] Figure 19 is an explanatory view of an inference step (STEP-203 to 205) of a plane.

[0041] Figure 20 is an explanatory view of an inference step (STEP-206 to 208) of a plane.

[0042] Figure 21 is an explanatory view of an inference step (STEP-206 to 208 (second time)) of a plane.

[0043] Figure 22 is an explanatory view of an inference step (inference case of an upper surface) of a plane.

[0044] Figure 23 is a flowchart of a combination method of planes existing in different groups.

[0045] Figure 24 is an explanatory view of a combination method of planes existing in different groups.

[0046] Figure 25 is an explanatory view of a clustering process of the same region.

[0047] Figure 26 is an explanatory view of hierarchical clustering, in which (A) is a case of a terrestrial object of a first example, and (B) is a case of a terrestrial object of a second example.

[0048] Figure 27 is an explanatory view of a generation step of a guide frame.

[0049] Figure 28 FIG. 7 is a diagram showing setting examples of the same region cluster in hierarchical clustering, in which (A) is a case where the same region cluster is set to encompass the entirety, (B) is a case where the plane cluster with the highest elevation value is set as the other same region cluster, and (C) is a case where the other same region cluster is set to include all the plane clusters with a difference in elevation value above a threshold value.

[0050] Figure 29 FIG. 8 is a schematic diagram showing alarm display of the data display section.

[0051] Figure 30 FIG. 9 is a flowchart showing the flow of deletion of the plane cluster by the data processing section.

[0052] Figure 31 FIG. 10 is an explanatory diagram of deletion of the plane cluster by the data processing section, in which (A) is an explanatory diagram of deletion of the plane cluster between the hoisting object and the top end portion of the boom, and (B) is an explanatory diagram of observation in the Y-axis direction.

[0053] Figure 32 FIG. 11 is an explanatory diagram of setting of the region removal. DETAILED DESCRIPTION

[0054] Next, the embodiment of the application will be described.

[0055] As shown in FIG. 1, a crane 1 is an example of a crane that is an application object of the guidance information display device according to an embodiment of the application, and is a mobile crane that can move to a desired location. Figure 1

[0056] The crane 1 includes a traveling vehicle 10 and a crane device 20.

[0057] The traveling vehicle 10 is a vehicle that transports the crane device 20, has a plurality of (four in this embodiment) wheels 11, and travels with an engine (not shown) as a power source.

[0058] An outrigger 12 is provided at each of the four corners of the traveling vehicle 10. The outrigger 12 is composed of an outrigger fork 12a that can extend to both sides in the width direction of the traveling vehicle 10 by hydraulic pressure and a hydraulic hoist cylinder 12b that can extend in a direction perpendicular to the ground. Furthermore, the traveling vehicle 10 can become a state in which the crane 1 can be operated by grounding the hoist cylinder 12b, and can increase the operable range (operational radius) of the crane 1 by increasing the extension length of the outrigger fork 12a.

[0059] ​The crane device 20 is a device that lifts the hoisted object W using a wire rope, and is provided with a slewing base 21, an extendable boom 22, a main hook pulley 23, a sub hook pulley 24, a luffing cylinder 25, a main winch 26, a main wire rope 27, a sub winch 28, a sub wire rope 29, and a cab 30.

[0060] The slewing base 21 is configured to be able to slew, and is provided on the frame of the traveling vehicle 10 via a circular ring-shaped bearing. The circular ring-shaped bearing is disposed in a manner that its center of rotation is perpendicular to the surface on which the traveling vehicle 10 is disposed. The slewing base 21 is configured to be able to freely rotate in one direction and another direction with the center of the circular ring-shaped bearing as the center of rotation. In addition, the slewing base 21 is rotated by a hydraulic slewing motor (not shown).

[0061] The extendable boom 22, which is a boom, supports the wire rope in a state in which the hoisted object W is able to be lifted. The extendable boom 22 is configured by a plurality of base arm members 22a, a second section arm member 22b, a third section arm member 22c, a fourth section arm member 22d, a fifth section arm member 22e, and a top arm member 22f, which are boom members. Each of the boom members is inserted in a nested manner in order of the size of the cross-sectional area. The extendable boom 22 is configured so that each of the boom members is moved by an extendable cylinder (not shown), and is thus freely extendable in the axial direction. The base end of the base arm member 22a of the extendable boom 22 is swingably provided on the slewing base 21. Thus, the extendable boom 22 is configured to be able to horizontally rotate and freely swing on the frame of the traveling vehicle 10.

[0062] The main hook pulley 23 is a device for hooking the hoisted object W, and is provided with a plurality of hook pulleys that wind the main wire rope 27 and a main hook 32 that suspends the hoisted object W.

[0063] The crane device 20 is provided with the main hook pulley 23, and is also provided with the sub hook pulley 24 for hooking the hoisted object W, on which a sub hook 33 that suspends the hoisted object W is provided.

[0064] The luffing cylinder 25 makes the extendable boom 22 stand up and lie down, and holds the posture of the extendable boom 22. The luffing cylinder 25 is configured by a hydraulic cylinder that is configured by a cylinder portion and a rod portion.

[0065] The main winch 26 performs winding (winding up) and unwinding (unwinding) of the main wire rope 27, and is configured by a hydraulic winch in the present embodiment.

[0066] The main winch 26 is configured to rotate the main drum on which the main wire rope 27 is wound by a main hydraulic motor. The main winch 26 is configured to unwind the main wire rope 27 wound on the main drum by supplying working oil in one direction by rotating the main hydraulic motor, and to wind the main wire rope 27 back onto the main drum by supplying working oil in the other direction by rotating the main hydraulic motor.

[0067] In addition, the auxiliary winch 28 performs the winding and unwinding of the auxiliary wire rope 29, and in this embodiment, it is composed of a hydraulic winch.

[0068] The cab 30 covers the driver's seat 31 where the operator sits and is located on the side of the telescopic boom 22 on the rotary table 21.

[0069] The crane 1 constructed in this way can move the crane device 20 to any position by moving the traveling vehicle 10. In addition, by using the pitch cylinder 25, the telescopic boom 22 can be erected at any pitch angle, thereby extending the telescopic boom 22 to any telescopic boom length.

[0070] In addition, the crane 1 is equipped with a controller 34 that controls the movement of the gyroscope 21, the telescopic boom 22, the pitch cylinder 25, etc. (i.e., the movement of the crane 1). The controller 34 can output to the outside information related to the movement status of the gyroscope 21, the telescopic boom 22, the pitch cylinder 25, etc., information related to the inherent performance of the crane 1, and the weight of the hoisted object W, etc.

[0071] Furthermore, in this description, the axial direction of the pitch fulcrum of the telescopic boom 22 is used as a reference for... Figure 1 The XYZ coordinate system shown is predetermined (as will be explained below).

[0072] The X-axis direction (also known as the lateral direction) is a horizontal direction parallel to the axis of the pitch support point of the telescopic boom 22. Additionally, the Y-axis direction (also known as the elevation direction) is a vertical direction. Furthermore, the Z-axis direction (also known as the depth direction) is a horizontal direction perpendicular to the axis of the pitch support point of the telescopic boom 22. That is, as... Figure 2 As shown, the XYZ coordinate system is predefined as a local coordinate system based on the telescopic boom 22.

[0073] Next, a guide information display device according to one embodiment of the present invention will be described.

[0074] Crane 1 has Figure 3 The guidance information display device 50 shown is shown.

[0075] The guidance information display device 50 is an example of the guidance information display device involved in this invention, designed for efficient and safe implementation.Figure 1 An apparatus for displaying information (hereinafter referred to as guide information) of a region including a hoisted object W (hereinafter referred to as a hoisted object region WA) by an image and giving a prompt to an operator in relation to the operation of a crane 1.

[0076] As shown in Figure 2 and Figure 4 The hoisted object region WA is set as a region including the hoisted object W when viewed in the Y-axis direction within the operation region SA of the crane 1, and is a region that becomes the target of generation of the guide information.

[0077] The hoisted object region WA is set to include the region directly below the top arm member 22f of the telescopic boom 22 of the crane 1, and the hoisted object W, the ground surface F, and the ground object C present in the hoisted object region WA become the measurement targets of the guide information display apparatus 50. The hoisted object region WA is displaced in accordance with the swing operation, the tilt operation, and the telescopic operation of the telescopic boom 22.

[0078] In addition, the guide information is information that assists the judgment of the operator in relation to the length, the swing position, the tilt angle, the amount of pay-out of the wire rope, and the like of the telescopic boom 22 when the operator transports the hoisted object W using the crane 1, and includes image information of the hoisted object region WA, information related to the shapes of the hoisted object W and the ground object C, height information of the hoisted object W, height information of the ground object C, information related to the movement route of the hoisted object W, and the like.

[0079] As shown in Figure 3 and Figure 4 The guide information display apparatus 50 is composed of a data acquisition section 60, a data processing section 70, a data display section 80, and a data input section 90.

[0080] The data acquisition section 60 is a section that acquires data required for generation of the guide information in the hoisted object region WA, and as shown in Figure 3 is provided with a camera 61, a laser scanner 62, and an inertial measurement device (IMU) 63.

[0081] As shown in Figure 4 The data acquisition section 60 is attached to the top arm member 22f located at the top end of the telescopic boom 22 of the crane 1, and is disposed in a state in which it can grasp the situation directly below from the boom top end located directly above the hoisted object W. In addition, the "directly above" of the hoisted object W is a concept including the position directly above the hoisted object W and the position of a fixed range (for example, the range of the upper surface of the hoisted object W) based on the position.

[0082] Data acquisition unit 60 via universal joint 67 (see Figure 1 The top arm component 22f, attached to the top of the telescopic boom 22, maintains a relatively fixed posture (or orientation towards the Y-axis) for the data acquisition unit 60 during the boom's pitching, rotating, and telescopic movements. This ensures that the camera 61 and laser scanner 62 are always facing the suspended object W. Therefore, the data acquisition unit 60 can continuously acquire data from the suspended object W and the ground surface F below it (i.e., the suspended object area WA) via the camera 61 and laser scanner 62. Furthermore, if an object C is present within the suspended object area WA, data on the object C can be acquired via the camera 61 and laser scanner 62.

[0083] like Figure 5 As shown in (A) and (B), camera 61 is a digital camera used to capture images of the hoisted area WA, and has the function of outputting the captured images to the outside in real time. Camera 61 has Figure 5 The field of view (horizontal field of view θh and vertical field of view θv) is as shown in (A) and (B). Furthermore, the camera 61 has a number of pixels, frame rate, and image transmission rate that take into account the amount of data required to generate appropriate guidance information.

[0084] like Figure 3 As shown, the laser scanner 62 is a device that obtains point data of a measurement object by irradiating the object with a laser and receiving the reflected light from the object. The objects measured by the laser scanner 62 are suspended objects W, ground objects C, and the ground surface F. Additionally, a first GNSS receiver 65 is connected to the laser scanner 62 to obtain the measurement time.

[0085] In the guidance information display device 50, three-dimensional point data of the plane is acquired in real time by a laser scanner 62.

[0086] like Figure 6 As shown, the laser scanner 62 has a total of 16 laser transceiver sensors, capable of simultaneously irradiating the object being measured with 16 laser beams to acquire point group data of the object. The 16 laser transceiver sensors of the laser scanner 62 are arranged with irradiation angles differing by 2° in the Z-axis direction, and are configured to irradiate the object with lasers over a total range of 30°. Furthermore, each laser transceiver sensor of the laser scanner 62 is configured to rotate 360° (omnidirectional) around the Z-axis. In the following description, the trajectory traced by the laser irradiating towards the suspended object area WA is referred to as a laser sideline. The laser sidelines are parallel to the X-axis direction, and the laser scanner 62 simultaneously traces 16 laser sidelines.

[0087] Furthermore, the laser scanner 62 is configured such that the laser side line is parallel to the X-axis direction. Additionally, the reference axis of the laser scanner 62 for changing the laser irradiation angle is parallel to the Z-axis direction.

[0088] like Figure 3 As shown, the Inertial Measurement Unit (IMU) 63 is a device for acquiring attitude data from the camera 61 and the laser scanner 62 at the time of data acquisition. The IMU 63 can measure attitude angles in real time and has measurement accuracy that can be used in correcting point group data acquired by the laser scanner 62. Additionally, a second GNSS receiver 66 is connected to the IMU 63 for acquiring the measurement time.

[0089] like Figure 7 As shown in (A) and (B), the data acquisition unit 60 is a sensor unit that integrates a camera 61, a laser scanner 62, and an inertial measurement device (IMU) 63 with respect to a frame body 64.

[0090] The frame 64 is an approximately cuboid-shaped object constructed from five sheet metal panels. The four side surfaces of the cube are formed by four sheet metal panels, and the top surface is formed by the remaining sheet metal, thus creating a shape with an opening at the bottom. In the data acquisition unit 60, a camera 61 and a laser scanner 62 are attached to the inside of the side surfaces of the frame 64, and an IMU 63 is attached to the top surface of the frame 64. Figure 7 As shown in (A), when viewed along the Y-axis, the center position of the imaging element of the camera 61 and the center position of the laser of the laser scanner 62 are separated by a distance Δzh along the Z-axis. Furthermore, the center position of the laser refers to the rotation center of the laser of the laser scanner 62, which is located on the Z-axis.

[0091] In addition, such as Figure 7 As shown in (B), when viewed in the X-axis direction, the center position of the imaging element of the camera 61 and the center position of the laser of the laser scanner 62 are separated by a distance Δyv in the Y-axis direction.

[0092] The data acquisition unit 60 is configured such that one of a pair of opposing side surfaces of the frame body 64 is perpendicular to the Z-axis, and the other of a pair of opposing side surfaces is perpendicular to the X-axis. Additionally, the data acquisition unit 60 is configured such that the upper surface of the frame body 64 is perpendicular to the Y-axis.

[0093] Next, we will explain the method for converting the X coordinate of a point (x, y) in the XYZ coordinate system to the Xc coordinate in the camera space coordinate system.

[0094] In the guidance information display device 50, in order to overlay the guidance information GD (described later) onto the image M captured by the camera 61 and display it on the data display unit 80, a coordinate value conversion process is performed between the XYZ coordinate system and the camera space coordinate system. In the guidance information display device 50, a three-dimensional camera space coordinate system Xc, Yc, Zc is predetermined within the image space of the camera 61.

[0095] like Figure 5 As shown in (A), the distance from the point (x, y) to the point on the X-axis, extending from the center of the lens of the camera 61, is dh, and the maximum screen width of the camera 61 in the horizontal direction is wh. Furthermore, the point (x, y) is defined as the position of the point on the X-axis from the center of the screen. The Xc coordinates of the point (x, y) in the camera space are then represented by equations (1) and (2).

[0096] Furthermore, in the following mathematical formula, the horizontal difference between the position of the imaging element of camera 61 and the laser center is denoted as Δzh (see...). Figure 7 In the middle (A)), the horizontal width of the camera image is set to wh, the horizontal field of view of camera 61 is set to θh, and the temporary variable is set to tmp1.

[0097] tmp1=(y-Δzh)×tan(π×θh / 360)···(1)

[0098] Xc=wh / 2-wh×x / (2×tmp1)···(2)

[0099] Next, we will explain the method for converting the Z coordinate of a point (y, z) in the XYZ coordinate system to the Zc coordinate in the camera space coordinate system.

[0100] like Figure 5 As shown in (B), the distance along the Z-axis from point (y, z) to the laser center is dv, and the maximum horizontal screen width of camera 61 is wv. Additionally, the position of point (y, z) along the Z-axis from the screen center is taken as z. At this time, the Zc coordinates of point (y, z) in camera space are represented by the following mathematical formulas (3) and (4).

[0101] Furthermore, in the following mathematical formula, the difference in the vertical direction between the position of the imaging element of camera 61 and the laser center of laser scanner 62 is denoted as Δyv (see [reference]). Figure 7 In step (B), the vertical width of the camera image is set to wv, the vertical field of view of camera 61 is set to θv, and the temporary variable is set to tmp2.

[0102] tmp2=Y×tan(π×θv / 360)···(3)

[0103] Zc=wv / 2+wv×(Z-Δyv) / (2×tmp2)···(4)

[0104] In the guidance information display device 50, the coordinates of the point group data obtained by the laser scanner 62 and the like in the XYZ coordinate system are converted into the camera space coordinate system using the above mathematical formulas (1) to (4), thereby matching the guidance information GD position to the image M captured by the camera 61 and displaying it.

[0105] Furthermore, considering the maximum reach height of the telescopic boom 22, a device capable of measuring the three-dimensional shape of the object to be measured from its maximum reach height (e.g., approximately 100m) is selected as the laser scanner 62. Additionally, considering the amount of data and data accuracy required to generate appropriate guidance information, a device with predetermined performance for various specifications such as measurement speed, number of measurement points, and measurement accuracy is selected as the laser scanner 62.

[0106] Furthermore, although this embodiment illustrates the use of a laser scanner 62 equipped with a total of 16 laser transceiver sensors, the guidance information display device according to the present invention is not limited by the number of laser transceiver sensors constituting the laser scanner. In other words, in the guidance information display device according to the present invention, the optimal laser scanner is appropriately selected based on factors such as the maximum reach height of the crane boom.

[0107] The data acquired by the data acquisition unit 60 in the hoisting area WA includes image data obtained by the camera 61 capturing images of the hoisting object W, the ground surface F below the hoisting object W, and ground objects C surrounding the hoisting object W. Additionally, the data acquired by the data acquisition unit 60 in the hoisting area WA includes point data obtained by the laser scanner 62 scanning the hoisting object W, the ground surface F, and the ground objects C. Furthermore, the ground surface F, as used here, broadly includes the surface that serves as the starting and ending point for the hoisting object W's transport, and includes not only the ground surface but also the floor slabs and roof surfaces of buildings.

[0108] like Figure 3 As shown, the data processing unit 70 is a part used to process the data acquired by the data acquisition unit 60 to generate guidance information GD to prompt the operator. In this embodiment, it is composed of a general-purpose personal computer with a predetermined data processing program installed.

[0109] In addition, the data processing unit 70 is electrically connected to the controller 34 of the crane 1, and the "crane information" output from the controller 34 is input into the data processing unit 70.

[0110] The data display unit 80 is a part used to display guidance information GD to prompt operators, and is composed of a display device connected to the data processing unit 70.

[0111] like Figure 8 As shown in (A), the image M of the hoisted object area WA captured by the camera 61 is displayed in real time on the data display unit 80.

[0112] like Figure 8 As shown in (B), the guidance information GD includes a guide frame image (GD1) representing the shape of the hoisted object W and the ground object C when viewed in the Y-axis direction, the height information of the lower surface of the hoisted object W (GD2), the height information of the upper surface of the ground object C (GD3), the working radius information (GD4) representing the movement route of the hoisted object W, and the axis information (GD5) representing the axis direction of the telescopic boom 22.

[0113] Furthermore, on the data display unit 80, the guidance information GD generated by the data processing unit 70 is overlaid with the image M.

[0114] like Figure 3 As shown, the data input unit 90 is a part for inputting setting values, etc., to the data processing unit 70, and is composed of a touch panel, mouse, keyboard device, etc.

[0115] In addition, such as Figure 9 As shown in (A), the guidance information display device 50 preferably integrates the data processing unit 70, the data display unit 80, and the data input unit 90 into a single unit using a tablet-type general-purpose personal computer (hereinafter also referred to as a tablet computer). Furthermore, as... Figure 9 As shown in (B), the guidance information display device 50 can also be a touch panel display device that integrates the data display unit 80 and the data input unit 90, and is configured to connect the data processing unit 70, which is a general-purpose PC, to the touch panel display device.

[0116] like Figure 4 As shown, the data display unit 80 and the data input unit 90 are positioned in front of the driver's seat 31 within the cab 30, in a location easily visible to the operator. Preferably, the data processing unit 70 is positioned near the data acquisition unit 60. Furthermore, if the data processing unit 70, data display unit 80, and data input unit 90 are integrated using a tablet computer, the data processing unit 70 can also be positioned within the cab 30.

[0117] The data transmission between the data acquisition unit 60 and the data processing unit 70 preferably uses a wired LAN. Alternatively, the data transmission between the data acquisition unit 60 and the data processing unit 70 may also use a wireless LAN, or powerline communication.

[0118] In addition, such as Figure 9 As shown in (A), the guidance information display device 50 preferably integrates the data processing unit 70, the data display unit 80, and the data input unit 90 into a single unit using a tablet-type general-purpose personal computer (hereinafter also referred to as a tablet computer). Furthermore, as... Figure 9 As shown in (B), the guidance information display device 50 can also be integrated into the data display unit 80 and the data input unit 90 via a touch panel display device, and is configured to connect the data processing unit 70, which is a general-purpose PC, to the touch panel display device.

[0119] Here, the data acquisition process of the data acquisition unit 60 will be explained.

[0120] In the data acquisition unit 60, the camera 61 continuously captures images of the hoisted object area WA, thereby acquiring an image M of the hoisted object area WA.

[0121] like Figure 10 As shown, in the data acquisition unit 60, the laser scanner 62 continuously scans the hoisting area WA to acquire point group data of the measurement objects within the hoisting area WA. Hereinafter, the point group data acquired by the laser scanner 62 will be referred to as point group data P. Point group data P is a set of point data p, where point data p represents a point located on the upper surface of the ground surface F, the hoisting object W, and the ground object C existing in the hoisting area WA. Furthermore, as... Figure 11 As shown, the point data p contains information such as the distance a from the object being measured (e.g., object C on the ground) to the laser scanner 62 and the illumination angle b of the laser scanner 62 when the point data p is obtained.

[0122] like Figure 3 As shown, a first GNSS receiver 65 is connected to the laser scanner 62. While acquiring point group data P, the first GNSS receiver 65 receives time information from multiple positioning satellites. Furthermore, the data processing unit 70 assigns information related to the acquisition time of the point data p to the point data p. That is, the information related to the point data p includes not only the distance a and the illumination angle b, but also the acquisition time tp.

[0123] In addition, in the data acquisition section 60, while the point group data P is acquired by the laser scanner 62, the posture data Q of the laser scanner 62 is acquired by the IMU 63 at a predetermined cycle. The posture data Q contains information related to the angle and acceleration of each axis of the laser scanner 62 with respect to the X, Y, and Z axes. Furthermore, the acquisition cycle of the posture data Q of the IMU 63 is shorter than the acquisition cycle of the point group data P of the laser scanner 62. The posture data Q is a collection of individual posture data q measured at each measurement cycle.

[0124] The second GNSS receiver 66 is connected to the IMU 63, and receives time information from a plurality of positioning satellites by the second GNSS receiver 66 at the same time as the posture data Q is acquired. The data processing section 70 assigns the acquisition time tq to the individual posture data q as information related to the acquisition time of the individual posture data q. That is, the acquisition time tq is included in the information related to the individual posture data q.

[0125] Next, the processing of the data by the data processing section 70 will be described.

[0126] As shown in FIG. 6, in the data processing by the data processing section 70, first, "frame extraction processing" is performed (STEP-101). Figure 12

[0127] In the data processing of the point group data P by the data processing section 70, the point group data P of 1 frame amount is cut out from the stream data of the point group data P and output. The point group data P of 1 frame amount is a collection of point data p acquired during one revolution of the irradiation direction of the laser of the laser scanner 62 around the Z axis.

[0128] As shown in FIG. 6, in the data processing by the data processing section 70, next, "point group data and posture data synchronization processing" is performed (STEP-102). Figure 12

[0129] The data processing section 70 synchronizes the point data p included in the point group data P of 1 frame amount with the posture data Q acquired by the IMU 63.

[0130] Specifically, in each point data p, the acquisition time tq of the individual posture data q closest to the acquisition time tp of the point data p is searched for, and the individual posture data q of the acquisition time tq is associated with the point data p, whereby synchronization is performed.

[0131] Thus, the data processing section 70 outputs the point data p synchronized with the individual posture data q.

[0132] Furthermore, as shown in FIG. 6, in the data processing by the data processing section 70, next, "point group data and posture data synchronization processing" is performed (STEP-102). Figure 11 ​​As shown, the data processing section 70 calculates the distance h from the laser center position of the laser scanner 62 to the point data p based on the distance a and the irradiation angle b. Moreover, the "distance h" here is the distance from the laser center position of the laser scanner 62 to the horizontal plane where the point data p exists.

[0133] In addition, in the data processing section 70, when calculating the distance h of the point data p, correction is performed using the individual posture data q corresponding to the point data p. By this, it is possible to eliminate the error due to the posture of the laser scanner 62, and furthermore, to calculate the distance h of the point data p with higher accuracy.

[0134] That is, in the guide information display device 50, the data acquisition section 60 is provided with the IMU 63 that acquires the posture data Q of the laser scanner 62, and the data processing section 70 corrects the point group data P based on the posture data Q of the laser scanner 62 acquired by the IMU 63.

[0135] In the guide information display device 50, it is possible to present more reliable guide information GD to the operator by such a structure.

[0136] When the point group data P of 1 frame amount is plotted on the XYZ coordinate system, it is expressed as in (A) of Figure 13 (A). (A) of Figure 13 (A) is the point group data P (collection of point data p) viewed from the Z-axis direction.

[0137] As shown in (A) of Figure 12 , in the data processing performed by the data processing section 70, next, "ground surface inference processing" is performed (STEP-103). The data processing section 70 performs processing to infer the ground surface F.

[0138] First, the case where the ground surface F is inferred with reference to a specific position on the image will be described. Moreover, although the case where the operator specifies the specific position on the image by hand is exemplified here, it can also be configured so that the data processing section 70 automatically decides and specifies the specific position on the image.

[0139] In the guide information display device 50, by specifying the position of the ground surface in the data display section 80 and the data input section 90, it is possible to decide the ground surface F that becomes the reference.

[0140] In the case where the specification is performed manually, first, as shown in the upper drawing of Figure 14 , the operator specifies the position that is definitely the ground surface by the image displayed on the data display section 80. Then, as shown in the middle drawing of Figure 14 , the data processing section 70 generates a reference circle of a predetermined radius with the specified position (point) as the center. Moreover, as shown in the lower drawing of Figure 14As shown in the lower drawing of FIG. 10, the data processing portion 70 detects the overlap of the point data p located on the laser side line and selects a plurality of point data p included in the reference circle.

[0141] Further, as shown in (A) of FIG. 11, the data processing portion 70 first extracts the point data p having the distance h of the maximum distance hmax from the selected plurality of point data p. The point data p having the maximum distance hmax is presumed to be the point data p existing at the lowest position. Then, the data processing portion 70 extracts the point data p having the distance h from the maximum distance hmax and the deviation D of the distance h within a fixed range (7 cm in the present embodiment) and calculates the average value of the distance h of the extracted point data p. The data processing portion 70 infers the average value calculated in this manner as the distance h to the ground surface F and determines the height of the ground surface F (hereinafter referred to as the reference height Ho) based on the distance. Figure 15

[0142] Further, the data processing portion 70 calculates the elevation value H of the point data p from the distance h and the reference height Ho. As shown in (A) of FIG. 12, the elevation value H is the height of the point data p from the reference height Ho. Figure 10

[0143] In the guide information display device 50, it is configured to generate the guide information GD based on the reference height Ho of the ground surface F obtained with high precision through the above processing. Therefore, in the guide information display device 50, it is possible to calculate the shape of the hoist W and the ground object C existing in the periphery thereof with high precision based on the height of the ground surface F.

[0144] Next, the case where the ground surface F is automatically inferred will be described.

[0145] Although the structure in which the operator designates the ground surface F is shown in the above description, in the guide information display device 50, it can also be configured as a structure in which the data processing portion 70 automatically infers the ground surface F.

[0146] In the case where the ground surface F is automatically inferred by the data processing portion 70, as shown in (B) of FIG. 13, the data processing portion 70 divides the hoist area WA into a plurality of (160 in the present embodiment) small areas S having equal areas. Figure 15

[0147] Next, the data processing portion 70 extracts the point data p having the maximum distance h (the distance h is the maximum distance hmax) in each small area S, as shown in (A) of FIG. 14, and extracts the point data p having the distance h from the maximum distance hmax and the deviation D of the distance h within a fixed range (the deviation D is 7 cm in the present embodiment). Figure 15

[0148] ​​​​Next, within each of the small regions S, the data processing section 70 calculates the average value of the distances h of the extracted point data p. The data processing section 70 automatically infers the reference height H0 of the ground surface F in each of the small regions S from the average value of the distances h thus calculated.

[0149] Alternatively, the data processing section 70 further averages the average values of the distances h calculated in each of the small regions S over all of the small regions S, and automatically infers the reference height H0 of the ground surface F in the hoist region WA from the average value. In this case, the data processing section 70 uses only the small regions S whose deviation D from the maximum value among the average values of the distances h in the respective small regions S is within a predetermined threshold value, to calculate the reference height H0, with reference to the maximum value.

[0150] As shown in Figure 12 , next, in the data processing performed by the data processing section 70, "planar inference processing" is implemented (STEP-104). The data processing section 70 infers the upper surfaces of the hoist W and the ground object C, which are the measurement targets, present within the hoist region WA, by the upper surface inference method shown below.

[0151] When the point group data P of one frame amount is plotted on the hoist region WA represented in the XYZ coordinate system, it is represented in the manner shown in (A) of Figure 13 . Also, when such point group data P present within the hoist region WA is schematically represented, it is represented as in the upper drawing of Figure 16 .

[0152] The data processing section 70 divides the point group data P taken in the hoist region WA shown in the upper drawing of Figure 16 in the manner shown in the middle drawing of Figure 16 , in the Y-axis direction, into layers at a predetermined thickness d, and divides the point group data P into a plurality of groups (see (B) of Figure 13 ).

[0153] At this time, the data processing section 70 assigns a separate group ID (here, ID: 001 to 006) to each of the divided groups, thereby associating each of the point data p with the group ID.

[0154] Also, in each of the groups, the data processing section 70 infers a plane using the plurality of point data p included in the group. The "plane" referred to here is a plane present in an upward direction in the hoist W and the ground object C, that is, the "upper surface" of the hoist W and the ground object C.

[0155] Specifically, first, as shown in Figure 17 and Figure 18As shown in the upper drawing of FIG. 2, the data processing section 70 selects two point data p, p from among the plurality of point data p, p,... included in the same group (two-point selection step: STEP-201).

[0156] Moreover, as shown in the lower drawing of FIG. 2, the data processing section 70 calculates the distance Ll between the two selected point data p, p (inter-point distance calculation step: STEP-202). Figure 17 and Figure 18 As shown in the upper drawing of FIG. 3, the data processing section 70 calculates the distance Ll between the two selected point data p, p (inter-point distance calculation step: STEP-202).

[0157] Next, as shown in the upper drawing of FIG. 4, the data processing section 70 determines whether the distance Ll between the two points is below a predetermined threshold value rl (STEP-203). If the distance Ll is below the threshold value rl, the data processing section 70 regards the two points (the two point data p, p shown by the dotted line) as being points on the same plane (two-point same-plane step: STEP-204). Figure 17 and Figure 19 As shown in the lower drawing of FIG. 4, the data processing section 70 calculates the center of gravity Gl of the points regarded as being on the same plane (here, the two selected points) (center-of-gravity calculation step: STEP-205). If the determination in (STEP-203) is NO, the process returns to (STEP-201) and new two points are selected. Figure 17 Figure 19 Next, as shown in the upper drawing of FIG. 5, the data processing section 70 searches for point data p that is a nearby point with respect to the calculated center of gravity Gl (nearby-point search step: STEP-206). The "nearby point" here is a point whose inter-point distance with respect to the center of gravity Gl is below the threshold value rl.

[0158] Next, as shown in the upper drawing of FIG. 5, the data processing section 70 searches for point data p that is a nearby point with respect to the calculated center of gravity Gl (nearby-point search step: STEP-206). The "nearby point" here is a point whose inter-point distance with respect to the center of gravity Gl is below the threshold value rl. Figure 17 Figure 20 As shown in the lower drawing of FIG. 5, the data processing section 70 regards the point data p found as a nearby point (STEP-207) as being on the same plane as the two previously selected point data p, p (nearby-point same-plane step: STEP-208).

[0159] Moreover, as shown in the lower drawing of FIG. 5, the data processing section 70 regards the point data p found as a nearby point (STEP-207) as being on the same plane as the two previously selected point data p, p (nearby-point same-plane step: STEP-208). Figure 17 Figure 20 As shown in the upper drawing of FIG. 6, the data processing section 70 returns to (STEP-205) and calculates a new center of gravity G2 from the points regarded as being on the same plane (here, the three point data p, p, p shown by the dotted line).

[0160] As shown in the upper drawing of FIG. 6, the data processing section 70 returns to (STEP-205) and calculates a new center of gravity G2 from the points regarded as being on the same plane (here, the three point data p, p, p shown by the dotted line). Figure 17 Figure 21 As shown in the lower drawing of FIG. 6, the data processing section 70 returns to (STEP-205) and calculates a new center of gravity G2 from the points regarded as being on the same plane (here, the three point data p, p, p shown by the dotted line).

[0161] The data processing section 70 shifts to (STEP-206) and further searches for point data p that is a nearby point with respect to the center of gravity G2. Moreover, as shown in the lower drawing of FIG. 6, the data processing section 70 regards the point data p found as a nearby point (STEP-207) as being on the same plane as the three previously selected point data p, p, p (nearby-point same-plane step: STEP-208). Figure 17 and​​​​Figure 21 As shown in the figure below, if the data processing unit 70 further discovers point data p as a nearby point (STEP-207), it will also regard the point data p as a nearby point as point data p on the same plane as the previously selected points (STEP-208).

[0162] Then, the data processing unit 70 calculates the new centroid while searching for nearby points, and repeatedly performs the processing from (STEP-205) to (STEP-208) until no point data p can be detected as a nearby point.

[0163] Moreover, such as Figure 17 As shown, if the data processing unit 70 does not find any new nearby points, it determines "No" in (STEP-207), as follows. Figure 22 As shown, clustering is performed on a subset (cluster) of point data p considered to be on the same plane to infer the plane (STEP-209). The "clustering" referred to here is the process of dividing the point data group P, which is the set of point data p, into clusters, such that the point data p contained in each cluster have the common characteristic that they are on the same plane.

[0164] The data processing unit 70 divides the point group data P into point data p that are considered to be on the same plane, and sets the plane cluster CL1 (see...). Figure 16 (See the diagram below). It is possible to define a plane (i.e., the "upper surface" of the hoisting object W and the ground object C) based on the point data p belonging to the plane cluster CL1. Furthermore, it also handles the case where multiple plane clusters CL1 exist within a group assigned the same group ID.

[0165] Then, the data processing unit 70 infers the "width" of the plane based on the maximum and minimum values ​​of the X-coordinates of the point data p belonging to the plane cluster CL1, and infers the "depth" of the plane based on the maximum and minimum values ​​of the Z-coordinates. The data processing unit 70 defines the plane formed by the plane cluster CL1 in this way. Furthermore, the plane defined here can also be a polygon other than a rectangle.

[0166] In this method of inferring the upper surface, the upper surfaces of the hoisted object W and the ground object C can be inferred solely from the point set data P corresponding to the upper surface obtained by the laser scanner 62. Therefore, according to the method of inferring the upper surface shown in this embodiment, the upper surfaces of the hoisted object W and the ground object C can be inferred in a short time based on the point set data P obtained by the laser scanner 62, and real-time inference of the upper surfaces of the hoisted object W and the ground object C can be achieved.

[0167] In addition, according to such a top surface estimation method, it is possible to estimate the top surfaces of the hoisted object W and the ground object C without using a statistical method, and it is possible to reduce the amount of calculation required to estimate the top surfaces of the hoisted object W and the ground object C compared to a case where a statistical method is used. Thus, according to the top surface estimation method illustrated in the present embodiment, it is possible to estimate the top surfaces of the hoisted object W and the ground object C in a shorter time based on the point group data P acquired by the laser scanner 62.

[0168] Further, although in the top surface estimation method of the hoisted object W and the ground object C illustrated in the present embodiment, a case is exemplified in which the data acquisition section 60 is provided at the top arm member 22f of the telescopic boom 22 in the crane 1 and the point group data P related to the hoisted object W, the ground object C, and the ground surface F is acquired from directly above the hoisted object W by the laser scanner 62, the top surface estimation method of the measurement target object of the present application is not particularly limited to a method applied to a case where a hoisted object of a crane and an article existing around the hoisted object are measurement target objects.

[0169] That is, the top surface estimation method illustrated in the present embodiment can be widely applied to, for example, a case in which a laser scanner is provided at a top end of a boom of a work vehicle (e.g., an aerial work vehicle or the like) or a drone or the like, and point group data of a measurement target object existing directly below the laser scanner is acquired from above, and a top surface of the measurement target object is estimated from the acquired point group data.

[0170] Next, the data processing section 70 combines the estimated respective plane clusters CL1 (top surfaces) together.

[0171] As shown in the upper diagram of Figure 23 and Figure 24 , the data processing section 70 selects two plane clusters CL1, CL1 among the estimated plane clusters CL1 to which different group IDs are assigned, and calculates a difference dH in the elevation values H of the respective plane clusters CL1 (STEP-301: elevation value difference calculation process).

[0172] Here, the data processing section 70 searches for combinations in which the difference dH is within a threshold value r2 (STEP-302). The elevation value H of the plane cluster CL1 referred to here is an average value of the elevation values H of the respective point data p belonging to the plane cluster CL1.

[0173] Next, as shown in the lower diagram of Figure 23 and Figure 24As shown in the middle figure, when the data processing unit 70 detects a combination of planar clusters CL1 whose elevation value difference dH is within the threshold r2, it performs an X-axis overlap dW detection (STEP-303: overlap detection process) on these planar clusters CL1. Here, "overlap" refers to the degree of overlap and separation of the planes defined by the planar clusters CL1 in the X-axis direction, such as... Figure 23 and Figure 24 As shown, if the overlap of the width dW1 is detected (dW1 > 0) or the separation dW2 is below the predetermined threshold r3 (0 ≤ dW2 ≤ r3), it is considered that "overlap" has been detected.

[0174] Moreover, such as Figure 23 and Figure 24 As shown, when the data processing unit 70 detects "overlap" (STEP-304), it considers the point data p belonging to these planar clusters CL1, CL1 to exist on the same plane, and thus combines the two planar clusters CL1, CL1 and updates them as a new planar cluster CL1 (STEP-305: Planar Combination Process). Furthermore, at this time, a new elevation value H is calculated based on the point data p belonging to the new planar cluster CL1.

[0175] like Figure 23 As shown, the data processing unit 70 repeatedly performs the above processing until there is no combination of plane clusters CL1 and CL1 that meet the conditions (STEP-306), thereby inferring the plane that exists across multiple groups.

[0176] Then, the data processing unit 70 outputs the plane (i.e., the plane cluster CL1) that has been combined through the above combined processing.

[0177] The plane defined by the plane cluster CL1 is the plane that exists in an upward manner in the hoisting object W and the ground object C, that is, the upper surface of the hoisting object W and the ground object C.

[0178] In this method of plane inference, the plane can be inferred without using the normal vectors of the point set data P. Therefore, compared with the case of inferring the plane using the normal vectors of the point set data P, it has the characteristic of requiring less computation.

[0179] Furthermore, in such a planar inference method, the three-dimensional shape of the hoisting object W and the ground object C can be determined by inferring the upper surface of the hoisting object W and the ground object C, without needing to obtain point data p of the side surfaces of the hoisting object W and the ground object C.

[0180] like Figure 12As shown, in the data processing performed in the data processing unit 70, the next step is to perform "clustering processing of the same region" (STEP-105). The "clustering" referred to here is the process of dividing the point data P, which is a set of data, into clusters so that the point data p contained in each cluster have the common characteristic of being in the "same region".

[0181] The "clustering process for the same region" implemented here is to cluster the generated planar cluster CL1 (plane) based on the different perspective of whether it exists in the "same region," regardless of whether it constitutes the same plane.

[0182] Specifically, such as Figure 25 As shown in the figure above, the data processing unit 70 extracts planar clusters CL1, including point data p with elevation values ​​H at the maximum value Hh, and planar clusters CL1 not associated with these planar clusters CL1. Then, the data processing unit 70 calculates the difference ΔH between the elevation values ​​H of each extracted planar cluster CL1. If the difference ΔH is below a predetermined threshold, the process moves to the next judgment.

[0183] When moving on to the next judgment, such as Figure 25 As shown in the middle figure, the data processing unit 70 confirms the overlap when observing two planar clusters CL1 and CL2 with a difference ΔH below a predetermined threshold in the Y-axis direction.

[0184] Here, when the two planar clusters CL1 and CL2 overlap when viewed along the Y-axis, as follows: Figure 25 As shown in the figure below, the data processing unit 70 considers these planar clusters CL1 and CL1 to be in the "same region", and these planar clusters CL1 and CL1 form a same region cluster CL2.

[0185] Furthermore, the data processing unit 70 further searches for planar cluster CL1, which includes point data p with the maximum value Hh of elevation value H, and planar cluster CL1 that is not combined with the planar cluster CL1. If an uncombined planar cluster CL1 is extracted, a judgment based on the difference ΔH and confirmation of overlap when viewed in the Y-axis direction are performed. If a planar cluster CL1 that meets the above conditions is found, it is further added to the above-mentioned same area cluster CL2.

[0186] The data processing unit 70 repeatedly performs this process until no other planar cluster CL1 is found that is not combined with the planar cluster CL1 containing the point data p with the maximum value Hh of the elevation value H. The data processing unit 70 forms a cluster CL2 of the same region through the above-described process.

[0187] Furthermore, in the display of the guidance information GD described later, the point data p belonging to the same area cluster CL2 formed in this way is processed as point data that is a cluster in shape, and the guidance box GD1 is displayed in a manner that surrounds the same area cluster CL2.

[0188] Furthermore, such "clustering of the same region" is preferred. Figure 26 The diagrams (A) and (B) show hierarchical clustering using a tree structure based on elevation values. In the "clustering processing of the same region," the data processing unit 70 creates a tree structure for each ground object C using its elevation value H. An example is shown here for... Figure 26 In the first example shown in (A), the ground object C implements hierarchical clustering using a tree structure and is targeted at... Figure 26 The second example shown in (B) illustrates a case where the ground object C implements hierarchical clustering using a tree structure.

[0189] In hierarchical clustering using a tree structure based on elevation values, the data processing unit 70 designates the planar cluster CL1 with the smallest average elevation value H as the "root". Furthermore, whenever there is a planar cluster CL1 that overlaps with the planar cluster CL1 constituting the "root" when viewed in the Y-axis direction, the data processing unit 70 extends a "branch" from the "root" and adds the overlapping planar cluster CL1 at the front end of the "branch". Moreover, the data processing unit 70 designates the planar cluster CL1 with the largest average elevation value H as the "child".

[0190] Here, the method for generating the guide box GD1 is explained.

[0191] The data processing unit 70 obtains the tree structure of the ground objects C generated in the "clustering process of the same region". In addition, the data processing unit 70 obtains the point data p contained in each planar cluster CL1 that constitutes the tree structure.

[0192] Next, as Figure 27 As shown in the figure above, the data processing unit 70 obtains the point data p of each laser side line located on the innermost side in the Z-axis direction from the point data p of the planar cluster CL1 of the "sub". Then, the data processing unit 70 forms a rectangle that is offset by 1 / 2 of the distance to the adjacent laser side line in the Z-axis direction and has a width in the X-axis direction that can surround each point data p.

[0193] Next, assuming the point data p exists on the laser sideline adjacent to the constructed rectangle, as follows: Figure 27 As shown in the figure below, the data processing unit 70 deforms the rectangle in such a way that it includes all the point data p on the laser side line, thereby forming the outline.

[0194] Then, the data processing section 70 searches for the point data p on the adjacent laser side line and repeatedly performs the above-described processing until there is no point data p on the laser side line that becomes the object.

[0195] Finally, the data processing section 70 creates an outline line that encloses all the plane clusters CL1 included in the selected tree structure.

[0196] Further, the data processing section 70 outputs only the outline line that meets the condition from the created outline line as the guide frame GD1.

[0197] As the condition to be output as the guide frame GD1, for example, as shown in (A) of FIG. 10, a condition can be selected that displays only the outline line of the above-ground object C as a large frame. In a case where this condition is selected, on the data display section 80, one guide frame GD1 that encloses the entire above-ground object C is displayed with respect to this above-ground object C. Figure 28 Further, as the condition to be output as the guide frame GD1, for example, as shown in (B) of FIG. 10, a condition can be selected that displays, in addition to the outline line of the above-ground object C as a large frame, the outline line (small frame) of the plane cluster CL1 in which the elevation value H is highest in each branch with respect to the difference (difference AH) in the elevation value H from the "root" that is above a threshold value. In a case where this condition is selected, in the data display section 80, a first guide frame GD1 that encloses the entire above-ground object C and a second guide frame GD1 that is included inside the first guide frame GD1 are displayed, and more detailed guide information GD that takes into account the three-dimensional shape of the above-ground object C is also displayed.

[0198] Figure 28 Further, as the condition to be output as the guide frame GD1, for example, as shown in (C) of FIG. 10, a condition can be selected that displays, in addition to the outline line of the above-ground object C as a large frame, all of the outline lines (small frames) of the difference (difference AH) in the elevation value H from the "root" that is above a threshold value. In a case where this condition is selected, in the data display section 80, a first guide frame GD1 that encloses the entire above-ground object C and a second guide frame GD1 that is included inside the first guide frame GD1 are displayed, and more detailed guide information GD that takes into account the three-dimensional shape of the above-ground object C is also displayed.

[0199] Further, as the condition to be output as the guide frame GD1, for example, as shown in (C) of FIG. 10, a condition can be selected that displays, in addition to the outline line of the above-ground object C as a large frame, all of the outline lines (small frames) of the difference (difference AH) in the elevation value H from the "root" that is above a threshold value. In a case where this condition is selected, in the data display section 80, a first guide frame GD1 that encloses the entire above-ground object C and a second guide frame GD1 that is included inside the first guide frame GD1 are displayed, and more detailed guide information GD that takes into account the three-dimensional shape of the above-ground object C is also displayed. Figure 28

[0200] Such a display condition can also be implemented by adjusting the threshold value of the difference AH. The operator can appropriately select the display condition of the guide frame GD1 so that the display of the guide information GD is easier to view.

[0201] ​​That is, in the guide information display device 50, the guide frame GD1 is generated based on the same region cluster CL2, so that the three-dimensional shape of the ground object C can be considered, and the guide frame GD1 that more specifically represents the ground object C is further generated. In the guide information display device 50, the guide frame GD1 that collectively surrounds the planar clusters CL1 existing in the same region can be generated. That is, according to the guide information display device 50, the guide information GD that is more specific and easy to view can be presented.

[0202] As shown in FIG. 6, in the data processing performed by the data processing section 70, next, "synchronization processing of point group data and camera image" is performed (STEP-106). Figure 12

[0203] Here, as shown in (A) and (B) of FIG. 7, the point group data P obtained in the XYZ coordinate system is converted into coordinate values of the camera space coordinate system, and is synchronized to the image M captured by the camera 61 (position matching), and then is output to the data display section 80. Figure 5

[0204] As shown in FIG. 8, in the data processing performed by the data processing section 70, next, "guide display processing" is performed (STEP-107). Figure 12

[0205] The data processing section 70 generates the guide information GD based on the information of the generated same region cluster CL2, and outputs the guide information GD to the data display section 80.

[0206] Further, in the "guide display processing", "crane information" output from the controller 34 of the crane 1 is used. The "crane information" used here includes information related to the length of the telescopic boom 22, the pitch angle, the work radius of the crane 1, the weight of the hoisted object W, and the like.

[0207] Although the series of processes of the data processing performed by the data processing section 70 is described, in such a configuration, without obtaining the point data p on the side surface of the measurement target object, the three-dimensional shape of the hoisted object W and the ground object C can be reliably grasped with a small amount of calculation, and the guide information GD can be generated. In such a configuration, since the data calculation amount is small, it is suitable for use in which the shape of the hoisted object W and the ground object C is grasped in real time, and the data processing section 70 can be configured with a simple hardware structure.

[0208] Next, the content of the guide information GD will be described.

[0209] In the guide information display device 50, the guide information GD is displayed by the data display section 80. In the guide information GD displayed by the data display section 80, the guide information GD includes​​​Figure 8 information on the specified position of the ground surface F designated by the operator.

[0210] Further, in the guide information display device 50, the hoisted object W can be designated. As with the case where the operator designates the ground surface F, it can be set that the upper surface of the hoisted object W is indicated by the plane (upper surface) existing at the designated position by indicating the hoisted object W on the screen. It is preferable that, after the hoisted object W is designated, the guide frame GD1 related to the hoisted object W and the guide frame GD1 related to the ground object C are displayed in distinction by changing the line color, line thickness, or the like.

[0211] The information on the designated positions of the ground surface F and the hoisted object W is displayed by a mark indicated in a figure such as a circle.

[0212] Further, in the guide information GD displayed by the data display section 80, the guide frame GD1 generated by the data processing section 70 is included.

[0213] The data processing section 70 outputs the guide frame GD1 based on the set same-area cluster CL2. Further, in the data processing section 70, a margin for reliably avoiding collision can be set for the guide frame GD1 of the hoisted object W, and a frame line offset only by a predetermined distance outward from the outline of the hoisted object W is output as the guide frame GD1. Such a guide frame GD1 becomes a frame display that surrounds the upper surface (plane cluster CL1) estimated in the hoisted object W and the ground object C in a line segment.

[0214] Further, in the guide information GD displayed by the data display section 80, the height information GD2 from the reference height Ho to the lower surface of the hoisted object W and the height information GD3 from the reference height Ho to the upper surface of the ground object C are included.

[0215] It is preferable that a separate area is provided at an easily viewable position on the screen of the data display section 80, and the height information GD2 of the hoisted object W is displayed in the area.

[0216] In the guide information display device 50, by such a structure, the height information GD2 of the hoisted object W and the height information GD3 of the ground object C are prevented from being mistaken.

[0217] The data processing section 70 calculates the height information GD2 by subtracting the height of the hoisted object W from the upper surface height of the plane cluster CL1 estimated as the upper surface of the hoisted object W.

[0218] In the guide information display device 50, an operator inputs information related to the hoisted object W (hereinafter referred to as "hoisted object information") to the data processing section 70 in advance. The input of the "hoisted object information" by the operator is performed from the data input section 90. Then, the data processing section 70 acquires the height of the hoisted object W using the "hoisted object information".

[0219] In the guide information display device 50, the height information GD3 of the ground object C is displayed inside the guide frame GD1 that surrounds the ground object C. Alternatively, in the guide information display device 50, the display is performed in a manner that the guide frame GD1 is partially overlapped with the guide frame GD1 in the case where the guide frame GD1 is small.

[0220] In the guide information display device 50, by such a configuration, the correspondence between the ground object C and the height information GD3 is made clear.

[0221] Further, in the guide information display device 50, the data processing section 70 is configured to change the line color of the guide frame GD1 according to the elevation value H of the planar cluster CL1 corresponding to the guide frame GD1.

[0222] In the guide information display device 50, with such a configuration, the operator can perceive the approximate elevation value (height) of the hoisted object W and the ground object C in a sense by observing the guide frame GD1. Therefore, in the guide information display device 50, the height of the hoisted object W and the ground object C can be more reliably prompted.

[0223] Further, in the guide information display device 50, the data processing section 70 is configured to change the font color of the height information GD2 according to the elevation value H of the planar cluster CL1 corresponding to the guide frame GD1.

[0224] In the guide information display device 50, with such a configuration, the operator can perceive the approximate elevation value (height) of the hoisted object W and the ground object C in a sense by observing the height information GD2. Therefore, in the guide information display device 50, the height of the hoisted object W and the ground object C can be more reliably prompted.

[0225] Further, in the display of the guide information GD performed by the guide information display device 50, the movement route information of the hoisted object W is included. In the movement route information of the hoisted object W, the work radius information GD4 of the hoisted object W and the axis information GD5 of the telescopic boom 22 of the crane 1 are included.

[0226] The work radius information GD4 is a reference of the movement route of the hoisted object W when the telescopic boom 22 performs a swing operation from the current state, and the hoisted object W moves along the circular arc shown as the work radius information GD4.

[0227] Further, the axis information GD5 becomes a reference for the moving route of the load W when the telescopic boom 22 is subjected to the luffing operation and the telescopic operation from the present state, and the load W moves along the straight line shown as the operation radius information GD4.

[0228] In the guidance information display device 50, the operation radius information GD4 of the load W and the axis information GD5 of the telescopic boom 22 are generated based on the "crane information".

[0229] The data processing section 70 calculates the operation radius of the crane 1 based on the "crane information", generates a circular arc showing the operation radius, and outputs it as the operation radius information GD4.

[0230] Further, the data processing section 70 calculates the axis direction of the telescopic boom 22 based on the "crane information", generates a straight line showing the axis direction, and outputs it as the axis information GD5.

[0231] Further, in the guidance information display device 50, the line showing the operation radius information GD4 and the axis information GD5 is displayed by a broken line, and the length and interval of the broken line are displayed by a reference length (hereinafter referred to as a reference length). For example, in the case where the reference length is set to 1 m, the length and interval of the broken line are changed on the display according to the size of the load area WA displayed on the data display section 80 in terms of the operation radius information GD4 and the axis information GD5, and the length and interval corresponding to 1 m are displayed on the ground surface F at the scale at that time.

[0232] In the guidance information display device 50, the length and interval of the broken line are displayed by the reference length (for example, 1 m), so that the operator can perceive the scale of the load W and the ground object C from the guidance information GD.

[0233] Further, the data processing section 70 calculates the height of the data acquisition section 60 based on the "crane information", and calculates the size of the load area WA and the size of the display range of the data display section 80, and changes the scale (the size of the broken line and the interval thereof) of the broken line displayed as the operation radius information GD4 and the axis information GD5 according to the calculation result.

[0234] Further, in the display of the guidance information GD implemented by the guidance information display device 50, an alarm display for preventing the contact of the load W with the ground object C is included.

[0235] In the case where the horizontal distance when the load W and the ground object C are projected on the horizontal plane is equal to or less than a predetermined threshold value (for example, 1 m) and the vertical distance is equal to or less than a predetermined threshold value (for example, 1 m), the data processing section 70 determines that the contact is possible.

[0236] As Figure 29 shown, at this time, the data processing section 70 outputs the guide frame GD1 and the height information GD2 of the ground object C in a manner of emphasizing the guide frame GD1 and the height information GD2 of the ground object C which is likely to be in contact with the hoisting object W. Alternatively, the data processing section 70 outputs the guide frame GD1 and the height information GD2 of the ground object C in a manner of flickering the guide frame GD1 and the height information GD2 of the ground object C. In the guide information display device 50, by outputting the guide frame GD1 and the height information GD2 of the ground object C as an alarm display by the data processing section 70 and displaying on the data display section 80, it is possible to urge the operator to pay attention.

[0237] In addition, in the guide information display device 50, it is configured that the point group data P existing between the hoisting object W and the top arm member 22f is removed when the guide information GD is generated by the data processing section 70. The Figure 30 and Figure 31 The processing of removing the point group data P existing between the hoisting object W and the top arm member 22f will be described.

[0238] The data processing section 70 determines whether or not there is an outline made for the hoisting object W (STEP-401). In the case where there is an outline made for the hoisting object W, the data processing section 70 acquires all the plane clusters CL1 (plane cluster acquisition process: STEP-402), and in the case where there is no outline made for the hoisting object W, the data processing section 70 ends the processing of removing the point group data P existing between the hoisting object W and the top arm member 22f.

[0239] Further, the data processing section 70 acquires the outline of the plane cluster CL1 having the highest elevation value H (outline acquisition process: STEP-403).

[0240] The data processing section 70 calculates the center of gravity of the acquired outline and sets the center of gravity as the hanging position WP of the hoisting object W (see Figure 4 and Figure 31 in (A) (STEP-404).

[0241] The data processing section 70 sequentially repeats the following (STEP-405) to (STEP-411) described below only in accordance with the number of the acquired plane clusters CL1. First, the data processing section 70 determines whether or not the elevation value H of the selected plane cluster CL1 is higher than the elevation value H of the height information GD2 of the hoisting object W (see Figure 31(A) (STEP-405). When the data processing unit 70 determines that the elevation value H of the selected planar cluster CL1 is higher than the elevation value H of the height information GD2 of the hoisting object W, it calculates the straight line connecting the hoisting position WP of the hoisting object W and the center of gravity G3 of the hoisting object W (see...). Figure 31 In the process of (A) (straight line calculation procedure: STEP-406), if it is determined that the elevation value H of the planar cluster CL1 is not higher than the elevation value H of the height information GD2 of the hoisting object W, the selected planar cluster CL1 is not removed.

[0242] Furthermore, the data processing unit 70 calculates the horizontal distance HD between the centroid G3 of the selected planar cluster CL1 and the calculated straight line (see...). Figure 31 (A) (Horizontal distance calculation procedure: STEP-407).

[0243] The data processing unit 70 calculates the width WD of the hoisting object W from the center of gravity (center of gravity G3) to the outermost perimeter of the hoisting object W (the outline of the hoisting object W) that is furthest from the center of gravity when viewed from above along the calculated straight line (see [reference]). Figure 31 The data processing unit 70 determines whether the calculated horizontal distance HD is less than the width WD of the hoisting object (STEP-408). If the data processing unit determines that the calculated horizontal distance is less than the width WD of the hoisting object, it removes the point group data P used to form the selected planar cluster CL1 (planar cluster removal process: STEP-411). Furthermore, when calculating the width WD of the hoisting object, as mentioned above, it can be calculated based on the obtained point group data P, or it can be calculated using the "hoisting object information" (width and depth of the hoisting object W) input by the operator.

[0244] If the data processing unit 70 determines that the horizontal distance is not less than the width WD of the suspended object, it sets the selected planar cluster CL1 in the camera space coordinate system. Figure 31 (not shown in the figure) Determine whether it overlaps with the suspended object W when viewed in the Yc axis direction (STEP-409). If the data processing unit 70 determines that the selected planar cluster CL1 overlaps with the suspended object W when viewed in the Yc axis direction in the camera space coordinate system, it removes the point group data P used to form the selected planar cluster CL1 (STEP-411).

[0245] If the data processing unit 70 determines that the selected planar cluster CL1 does not overlap with the suspended object W when viewed in the Yc axis direction of the camera space coordinate system, then it determines the position of the selected planar cluster CL1 in the XYZ coordinate system (see [reference]). Figure 31In step (A), when viewed along the Y-axis, does the selected planar cluster CL1 overlap with the hoisting object W (STEP-410)? If the data processing unit 70 determines that the selected planar cluster CL1 overlaps with the hoisting object W when viewed along the Y-axis in the XYZ coordinate system, it removes the point group data P used to form the selected planar cluster CL1 (STEP-411). If the data processing unit 70 determines that the selected planar cluster CL1 does not overlap with the hoisting object W when viewed along the Y-axis in the XYZ coordinate system, it does not remove the selected planar cluster CL1.

[0246] The point group data P removed in this way includes the point group data P corresponding to the main wire rope 27. Therefore, the guidance information display device 50 can accurately provide information related to the shape and position of the hoisted object W and the ground objects C surrounding the hoisted object W, regardless of the crane 1's posture or movement. Furthermore, by removing the point group data P corresponding to the main wire rope 27 and generating the guide frame GD1, the guidance information display device 50 can display more accurate and easily viewable guidance information GD.

[0247] In addition, such as Figure 32 As shown, in another embodiment, a removal area JA for the removal point group data P can be set between the hoisting object W and the top arm component 22f. The removal area JA is set to be within an arbitrary range relative to the straight line connecting the hoisting position WP of the hoisting object W and the center of gravity of the hoisting object W. In this embodiment, the set range is set to the range with the upper surface of the hoisting object W as the lower end. Furthermore, considering that it will not affect the generation of the guide frame GD1 of the hoisting object W, it is preferable to set the lower end height of the removal area JA at a position separated from the upper surface of the hoisting object W by a predetermined distance.

[0248] The point data P obtained within the removal area JA includes point data P corresponding to the main wire rope 27. Therefore, the guidance information display device 50 can accurately provide information related to the shape and position of the hoisted object W and the ground objects C surrounding the hoisted object W, regardless of the crane 1's posture or movement. Furthermore, by removing the point data P corresponding to the main wire rope 27 and generating a guide frame GD1, the guidance information display device 50 can provide more accurate and easily viewable guidance information GD.

[0249] In the guide information display device 50 configured in this way, the guide frame GD1 indicating the shape of the hoisted object W and the guide information GD including the height information GD2, GD3 indicating the height can be reliably presented to the operator of the crane 1 with respect to the hoisted object W and the ground object C existing in the periphery of the hoisted object W. Moreover, as long as the guide information display device 50 configured in this way is used, the operator can efficiently and safely perform the work of the crane 1 based on the guide information GD presented by the guide information display device 50 even in a case where the operator cannot directly visually confirm the hoisted object W, for example.

[0250] Industrial applicability

[0251] The present application can be applied to a guide information display device, a crane provided with the guide information display device, and a guide information display method.

[0252] Explanation of symbols

[0253] 1 Crane

[0254] 50 Guide information display device

[0255] 60 Data acquisition section

[0256] 61 Camera

[0257] 62 Laser scanner

[0258] 70 Data processing section

[0259] 80 Data display section

[0260] P Point group data

[0261] W Hoisted object

[0262] F Ground surface

[0263] C Ground object

[0264] M Image (captured by the camera)

[0265] GD Guide information

[0266] GD1 Guide frame

[0267] GD2 Height information (of the lower surface of the hoisted object)

[0268] GD3 Height information (of the upper surface of the ground object)

[0269] WA Hoisted object region

[0270] CL1 Plane cluster

[0271] CL2 Same region cluster

[0272] HD horizontal distance

[0273] WD load width

[0274] WP hanging position

[0275] JA removal area.

Claims

1. A guide information display device having: a data display section; a data acquisition section that acquires image data of a hoist area captured by a camera and point group data obtained by scanning, from above, objects including a hoist and a ground object within the hoist area and a ground surface within the hoist area by a laser scanner; and a data processing section that, based on remaining point group data obtained by removing useless point group data from the point group data, estimates an upper surface of the objects and the ground surface, generates a guide frame that encloses the upper surface, and displays the generated guide frame on the data display section so as to overlap the objects within the image data. The data processing section estimates the upper surface of the objects by layering the point group data in a distance direction from the laser scanner and clustering point group data belonging to the same layer and having an inter-point distance of less than a predetermined value to form a plane cluster, generates the guide frame for each plane cluster, and removes point group data that forms a plane cluster having an elevation value higher than an elevation value of the hoist as the useless point group data.

2. The guide information display device according to claim 1, The removed useless point group data is a steel wire rope that suspends the hoist.

3. The guide information display device according to claim 1, The data processing section, calculates a straight line that connects a hoisting position of the hoist and a center of gravity of the hoist, judges whether the point group data is useless based on whether a horizontal distance from a center of gravity of the plane cluster to the straight line is less than a hoist width that is a distance from the center of gravity of the hoist to an outermost circumference of the hoist in the distance direction from the laser scanner or whether the hoist and the plane cluster overlap when viewed from above in the distance direction from the laser scanner.

4. The guide information display device according to claim 1, The data processing section sets a removal area between the hoist and the hoisting position of the hoist, sets point group data acquired in the removal area as the useless point group data to be removed from the point group data acquired by the laser scanner.

5. The guide information display device according to claim 3, The data processing section regards a center of gravity of an outline of a plane cluster having the highest elevation value in the plane clusters as the hoisting position of the hoist.

6. A work machine that is provided with the guide information display device according to any one of claims 1 to 5. ​ ​ ​

Images