A crane safety identification method, device, system and storage medium

Abstract

The present invention relates to the technical field of construction machinery, and particularly relates to a crane safety identification method, device, system and storage medium. The method includes: acquiring point cloud data and the position of the hook of the crane; constructing a load detection area based on the position of the hook, where the load detection area is located in a preset orientation of the hook; performing clustering segmentation on the point cloud data located within the load detection area to obtain a plurality of independent circumscribed envelopes; determining the size of the load according to the positions of the vertices of each independent circumscribed envelope; and performing safety identification on the crane based on the size of the load. By identifying the size of the load, the present application enables the crane to more accurately and effectively perform safety identification in combination with the surrounding environmental conditions according to the size of the load during operation, improving the accuracy, sensitivity and stronger adaptability of safety identification.

Description

Technical Field

[0001] The present invention relates to the technical field of construction machinery, and particularly relates to a crane safety identification method, device, system and storage medium. Background Art

[0002] Cranes are widely used in the field of construction, mainly for cargo transfer. At present, during the process of transporting the suspended load, the judgment of the presence of obstacles around mainly relies on manual visual observation.

[0003] Since the size of the suspended load during the operation of the crane cannot be limited, and the working environment is basically high-altitude operation, which is often restricted by the field of vision. Therefore, it is often impossible to determine the degree of potential safety hazards in a timely and effective manner based on the surrounding environment and the size of the suspended load. As a result, collisions between the crane boom, wire rope, hook and suspended load and obstacles often occur, and safety accidents occur frequently. Summary of the Invention

[0004] The present application provides a crane safety identification method, device, system and storage medium, aiming to provide a more efficient and accurate safety identification method for cranes, effectively improving the safety perception performance.

[0005] In a first aspect, an embodiment of the present application provides a crane safety identification method, which includes:

[0006] Obtain point cloud data and the position of the hook of the crane; construct a suspended load detection area based on the position of the hook, and the suspended load detection area is located at a preset azimuth of the hook; perform clustering segmentation on the point cloud data located in the suspended load detection area to obtain a plurality of independent circumscribed envelopes; determine the size of the suspended load according to the positions of the vertices of each independent circumscribed envelope; perform safety identification on the crane based on the size of the suspended load.

[0007] Based on the above solution, a crane safety identification method provided by the present application constructs a suspended load detection area centered on the hook, more accurately detects and identifies the suspended load based on this suspended load detection area, obtains the point cloud data of the suspended load, and performs clustering segmentation on the point cloud data of the suspended load to obtain a plurality of independent circumscribed envelopes, so that the size of the suspended load can be determined based on the positions of the vertices of each independent circumscribed envelope, effectively providing a detection and identification solution for the size of the suspended load, so that during the operation of the crane, according to the size of the suspended load, it can more accurately and effectively combine the surrounding environment to perform safety identification, improve the accuracy of safety identification, improve the sensitivity of crane safety perception, and have stronger adaptability.

[0008] In some alternative embodiments, determining the size of the suspended load according to the positions of the vertices of each independent circumscribed envelope includes:

[0009] Project the positions of the vertices of each independent circumscribed envelope onto a target plane to obtain the projection points of the vertices in the target plane. The target plane is perpendicular to the optical axis of the sensor looking downward from the perspective of the auxiliary running device of the crane; determine the size of the suspended load according to the distances of the respective projection points from the center point of the target plane.

[0010] As an example, the auxiliary running device described in the embodiments of the present application can be set according to the actual situation. For example, the auxiliary running device can be the trolley device in the tower crane structure. For another example, the auxiliary running device can also be a device formed by adding a pan-tilt to a truck crane and installing a sensor on the pan-tilt platform. This is not limited here. For the sake of concise description, the trolley will be used as an example for introduction later.

[0011] Based on the above solution, the present application screens out the smallest circumscribed envelope on the outermost side of the suspended load, and thus determines the size of the suspended load based on the distance between the vertex of the smallest circumscribed envelope and the center of the target plane, which is more targeted, effectively saves the calculation amount, abandons the complex and useless calculation process, improves the calculation efficiency, is more convenient and fast, and has stronger adaptability.

[0012] In some alternative embodiments, determining the size of the suspended load according to the distances of the respective projection points from the center point of the target plane includes:

[0013] Determine the maximum distance from each projection point to the center point as the maximum size of the suspended load.

[0014] It can be understood that the embodiments of the present application can further determine the relevant dimensions of the suspended load according to actual needs. For example, in order to better reduce the collision probability between the suspended load and surrounding obstacles and better ensure the safety of the suspended load, the embodiments of the present application can further obtain the maximum size of the suspended load, so that a safer anti-collision area can be delineated based on the maximum size of the suspended load. For another example, the embodiments of the present application can also, in order to better obtain the relevant information of the suspended load and better understand the shape of the suspended load, and thus better determine the hoisting stability of the suspended load, etc., the embodiments of the present application can obtain the overall circumscribed size of the suspended load based on the distances of the respective projection points from the center point of the target plane.

[0015] In some alternative embodiments, obtaining the point cloud data and the position of the hook of the crane includes:

[0016] Obtain the working condition information of the crane, and obtain the estimated position of the hook according to the working condition information. The working condition information includes the luffing working condition and the hoisting working condition of the crane; construct a hook recognition area based on the estimated position of the hook, and detect the hook in the hook recognition area by matching the knowledge base with the point cloud features corresponding to the hook to obtain the actual position of the hook; the point cloud feature matching knowledge base includes the shape feature, statistical feature and reflection intensity feature of the hook.

[0017] As an example, the embodiments of the present application can not only identify and determine the position of the hook based on point cloud data, but also determine the position of the hook through the data collected by an image sensor. For example, the embodiments of the present application can determine the position of the hook through the data collected by a binocular camera or an RGBD camera, etc., which is not limited herein.

[0018] Based on the above solution, when determining the position of the hook in the present application, the actual hook position is measured by combining the estimated hook position of the working condition information, which can provide a targeted hook recognition area, effectively narrow the hook recognition range, and improve the hook detection and recognition efficiency. Moreover, after determining the hook recognition area, combining the point cloud feature matching knowledge base corresponding to the hook can more accurately and effectively improve the accuracy of hook position recognition, which is more convenient and fast.

[0019] In some optional embodiments, the method further includes:

[0020] Track the actual position of the hook, and predict the spatial position of the hook at the current moment; update the hook recognition area according to the predicted spatial position of the hook.

[0021] Based on the above solution, the present application can effectively improve the accuracy of hook recognition and positioning by tracking and detecting the position of the hook.

[0022] In some optional embodiments, the constructing a load detection area based on the position of the hook includes:

[0023] Determine the height between the hook and the load according to the position of the hook; construct the load detection area with the position of the hook as the center, according to the height and a preset radius.

[0024] Based on the above solution, the present application provides a way to construct a load detection area. For example, the present application can obtain a cylindrical area according to the height between the hook and the load, the preset radius and the position of the hook, and thus determine the cylindrical area as the load detection area. By defining the load detection area, the range of load recognition and detection can be effectively determined, avoiding the interference of invalid point cloud data, and making it more convenient and fast to recognize the load.

[0025] In some optional embodiments, determining the height between the hook and the load according to the position of the hook includes:

[0026] Obtaining first load cell data corresponding to when the load stays on the ground, and a first height between the hook and the boom of the crane; obtaining second load cell data corresponding to when the load is lifted by the hook and leaves the ground; when it is determined that the change in the second load cell data obtained in a continuous threshold number of frames is less than a threshold difference, obtaining a second height between the current hook and the boom of the crane; determining the height between the hook and the load through the first height, the second height, and the hook height.

[0027] In some optional embodiments, performing safety identification on the crane based on the size of the load includes:

[0028] Constructing a three-dimensional simulation area between the hook and the auxiliary operating device of the crane according to the position of the hook to obtain a first area; constructing a three-dimensional simulation area centered on the load according to the height between the hook and the load and the size of the load to obtain a second area; performing safety identification on the crane based on the first area and the second area.

[0029] Based on the above solution, the present application plans an area that needs to be focused on for safety hazard investigation by constructing the first area and the second area, so that safety identification can be carried out targeted according to the collected surrounding environment point cloud data, combined with the first area and the second area, which is faster and more efficient, and improves the accuracy of safety identification.

[0030] In some optional embodiments, performing safety identification on the crane based on the first area and the second area includes:

[0031] Removing the point cloud data corresponding to the first area and the second area from the collected point cloud data to obtain the point cloud data of the obstacle; dividing the point cloud data of the obstacle by an octree to obtain a plurality of grids; calculating the collision distance between each grid in the plurality of grids and the first area and the second area respectively to obtain the safety identification result of the crane.

[0032] In a second aspect, an embodiment of the present application provides a crane safety identification device, and the device includes:

[0033] A determination module, configured to obtain point cloud data and the position of the hook of the crane;

[0034] A building module for building a load detection area based on the position of the hook, where the load detection area is located in a preset orientation of the hook;

[0035] A processing module for performing clustering segmentation on the point cloud data within the load detection area to obtain a plurality of independent circumscribed envelopes;

[0036] The determining module is further configured to determine the size of the load according to the positions of the vertices of each independent circumscribed envelope;

[0037] The processing module is further configured to perform safety identification on the crane based on the size of the load.

[0038] In some optional embodiments, the determining module is specifically configured to:

[0039] Project the positions of the vertices of each independent circumscribed envelope onto a target plane to obtain the projection points of each vertex within the target plane, where the target plane is perpendicular to the sensor optical axis with a downward view of the auxiliary running device of the crane; determine the size of the load according to the distances of each projection point from the center point of the target plane.

[0040] In some optional embodiments, the determining module is specifically configured to:

[0041] Determine the maximum distance from each projection point to the center point as the maximum size of the load.

[0042] In some optional embodiments, the determining module is specifically configured to:

[0043] Obtain the working condition information of the crane, obtain the estimated position of the hook according to the working condition information, where the working condition information includes the luffing working condition and the hoisting working condition of the crane; build a hook recognition area based on the estimated position of the hook, and perform hook detection on the hook recognition area through the point cloud feature matching knowledge base corresponding to the hook to obtain the actual position of the hook; the point cloud feature matching knowledge base includes the shape feature, statistical feature and reflection intensity feature of the hook.

[0044] In some optional embodiments, the processing module is specifically configured to:

[0045] Track the actual position of the hook, predict the spatial position of the hook at the current moment; update the hook recognition area according to the predicted spatial position of the hook.

[0046] In some optional embodiments, the building module is specifically configured to:

[0047] Determine the height between the hook and the load according to the position of the hook; construct a load detection area centered on the position of the hook according to the height and a preset radius.

[0048] In some optional embodiments, the determining module is specifically configured to:

[0049] Obtain the first load cell data corresponding to when the load stays on the ground, and the first height between the hook and the boom of the crane; obtain the second load cell data corresponding to when the load is lifted by the hook and leaves the ground; when it is determined that the data change of the second load cell data obtained in consecutive threshold frames is less than the threshold difference, obtain the second height between the hook and the boom of the crane at present; determine the height between the hook and the load through the first height, the second height, and the hook height.

[0050] In some optional embodiments, the processing module is specifically configured to:

[0051] Construct a three-dimensional simulation area between the hook and the auxiliary operation device of the crane according to the position of the hook to obtain a first area; construct a three-dimensional simulation area centered on the load according to the height between the hook and the load and the size of the load to obtain a second area; perform safety identification on the crane based on the first area and the second area.

[0052] In some optional embodiments, the processing module is specifically configured to:

[0053] Remove the point cloud data corresponding to the first area and the second area from the collected point cloud data to obtain the point cloud data of the obstacle; divide the point cloud data of the obstacle by an octree to obtain a plurality of grids; calculate the collision distance between each grid in the plurality of grids and the first area and the second area respectively to obtain the safety identification result of the crane.

[0054] In a third aspect, an embodiment of the present application provides an electronic device, which includes at least one processor and at least one memory. Among them, the memory stores a computer program, and when the program is executed by the processor, the processor is enabled to execute the crane safety identification method according to any one of the first aspects above.

[0055] In a fourth aspect, an embodiment of the present application provides a chip system, which includes a processor and an interface. The processor is used to call and execute a computer program from the interface. When the processor executes the computer program, the method described in the first aspect or any possible design of the first aspect can be implemented.

[0056] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program for executing the method described in the first aspect or any possible design of the first aspect above.

[0057] In a sixth aspect, an embodiment of the present application further provides a computer program product, including a computer program, which can implement the method described in the first aspect or any possible design of the first aspect above when the computer program is executed.

[0058] The beneficial effects produced by any one of the second to sixth aspects above are the same as those of the first aspect. For specific details, reference can be made to the detailed description of the first aspect above, and no further elaboration will be provided here. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 FIG. is a schematic diagram of a crane safety identification method provided by an embodiment of the present application;

[0060] Figure 2 FIG. is a schematic diagram of a crane safety identification system provided by an embodiment of the present application;

[0061] Figure 3 FIG. is a schematic diagram of a scanned area display provided by an embodiment of the present application;

[0062] Figure 4 FIG. is a schematic diagram of a space construction provided by an embodiment of the present application;

[0063] Figure 5 FIG. is a schematic structural diagram of a crane safety identification device provided by an embodiment of the present application;

[0064] Figure 6 FIG. is a schematic block diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0065] In order to make the objectives, technical solutions, and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0066] Cranes are widely used in the field of construction, mainly for cargo transfer. Among them, since the size of the suspended load during the operation of the crane cannot be limited, and the working environment is basically high-altitude operation, restricted by the field of vision, it often leads to the inability to determine the degree of potential safety hazards in a timely and effective manner according to the surrounding environment and the size of the suspended load. As a result, collisions often occur between the crane boom, wire rope, hook and the suspended load and obstacles, and safety accidents occur frequently.

[0067] For example, one current method for safety identification based on cranes mainly involves establishing a boom model of a luffing tower crane, calculating the shortest spatial distance between the booms of two adjacent luffing tower cranes, so as to judge dangerous operations of the tower crane's behavior. However, this method can only be applied to tower cranes with known models. For unknown tower cranes or other obstacles, the tower crane cannot identify them. In addition, even based on a known tower crane model, there are often problems with potential safety hazards caused by the oversized size of the suspended load.

[0068] Another example is that another current method for safety identification based on cranes mainly involves real-time monitoring of the rotation state of the boom of adjacent fixed cranes. When the boom of one crane enters the warning area of another crane, a warning signal is sent to remind the staff. And this method is also to obtain the status information of adjacent cranes for anti-collision detection between multiple cranes, while the single crane itself does not have functions such as anti-collision detection and safety identification.

[0069] In summary, the current safety identification scheme based on cranes can only perform safety identification for simple scenarios. For unknown tower cranes or other obstacles, it cannot flexibly detect according to the actual size of its own suspended load in a timely and effective manner, and there are still relatively high potential safety hazards.

[0070] Based on this, the present application provides a crane safety identification method, device and storage medium. By identifying the size of the suspended load, the crane can, during operation, more accurately and effectively combine the surrounding environmental conditions for safety identification according to the size of the suspended load, improving the accuracy, sensitivity and adaptability of safety identification.

[0071] Furthermore, the crane safety identification method provided by the embodiments of the present application can also actively detect dynamic and static obstacles existing around the crane boom - wire rope - hook - suspended load through sensors such as lidar and cameras installed on the crane, and perform anti-collision detection in real time and efficiently, improving the hoisting operation efficiency and making the operation of the crane safer.

[0072] The following explains the terms involved in the embodiments of the present application. The specific explanations are not limited to the following descriptions:

[0073] (1) Polar coordinates refer to taking a fixed point O in a plane, called the pole, drawing a ray Ox, called the polar axis, and then selecting a unit of length and the positive direction of the angle (usually the counterclockwise direction). For any point M in the plane, let ρ represent the length of the line segment OM, and θ represent the angle from Ox to OM. ρ is called the polar radius of point M, and θ is called the polar angle of point M. The ordered pair (ρ, θ) is called the polar coordinates of point M. The coordinate system established in this way is called the polar coordinate system.

[0074] (2) An envelope is a figure formed by the interweaving of many elliptical curves.

[0075] Among them, the minimum envelope is a bounded convex set in three-dimensional space. For a given set of points, there exists a minimum envelope that completely encloses these points. The minimum envelope can be defined by calculating a set of points (vertices) in the point set, and these point sets define the geometric shape of the envelope.

[0076] Next, in combination with the accompanying drawings and specific embodiments, the technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail. These several specific embodiments below can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0077] The embodiment of the present application provides a crane safety recognition method, as Figure 1 shown, the method may include:

[0078] Step S101: Obtain the point cloud data and the position of the hook of the crane.

[0079] Among them, the point cloud data obtained by the embodiment of the present application in step S101 can be the point cloud data collected based on a set target area. For example, the target area can be an observation area set for the crane; for another example, the target area can be the area that can be collected by the collection device on the crane, and can be specifically obtained according to the actual situation, and is not limited herein.

[0080] Further, the embodiment of the present application provides various ways to determine the position of the hook of the crane, which are not limited to the following several:

[0081] Determination method 1: Identify and determine the position of the hook based on the point cloud data.

[0082] Determination method 2: Determine the position of the hook through the data collected by the image sensor.

[0083] For example, the embodiment of the present application can determine the position of the hook through the data collected by a binocular camera, or an RGBD camera, etc., and is not limited herein.

[0084] For a better description of the embodiments of the present application, the embodiments of the present application are based on determination method 1, and the content of determining the position of the hook is introduced in detail:

[0085] Specifically, obtain the working condition information of the crane, and obtain the estimated position of the hook according to the working condition information. The working condition information includes the luffing working condition and the hoisting working condition of the crane; construct a hook recognition area based on the estimated position of the hook, and perform hook detection on the hook recognition area by matching the point cloud features corresponding to the hook with the knowledge base, so as to obtain the actual position of the hook; the point cloud feature matching knowledge base includes the shape features, statistical features and reflection intensity features of the hook.

[0086] Among them, in order to better and more accurately locate and identify the hook, the embodiments of the present application can track the actual position of the hook, predict the spatial position of the hook at the current moment, and thus update the hook recognition area according to the predicted spatial position of the hook. For example, the embodiments of the present application can locate and identify the hook by combining a detector and a tracker.

[0087] Exemplarily, the embodiments of the present application can first perform initial hook positioning and identification based on a detector. For example, the point cloud feature matching knowledge base designed for the hook in the embodiments of the present application can include: a weighted combination of shape features (such as aspect ratio, height difference, bounding box size, etc.), statistical features (such as three-dimensional invariant moments, three-dimensional covariance matrices), and reflection intensity features. By collecting a large number of hook training samples (hook point cloud data at different distances), according to the variation law of each feature, calculate the feature weights, and finally ensure that the combined feature output result of the hook remains within a certain interval change range to achieve automatic detection of the hook.

[0088] Then, when the embodiments of the present application perform positioning and identification on the hook subsequently, the actual detection position of the hook can be obtained through the detector, and the spatial position of the hook at the current moment can be predicted by tracking the detection result. Among them, when the detector does not detect the hook or misdetects other objects, the prediction result of the tracker can be used to replace the detector, and at the same time, other misdetections are filtered, further narrowing the point cloud space R2 of the detector in the next frame, replacing R2 with R1, and then using the detector to detect the hook in the R2 space, finally improving the accuracy of hook recognition and positioning.

[0089] Step S102: Construct a load detection area based on the position of the hook, and the load detection area is located at a preset orientation of the hook.

[0090] Specifically, in the embodiments of the present application, the height between the hook and the load can be determined according to the position of the hook, and then, with the position of the hook as the center, the load detection area is constructed according to the height and a preset radius.

[0091] For example, the load detection area constructed in the embodiments of the present application may be a cylindrical area formed based on the position of the hook, the height between the hook and the load, and a preset radius.

[0092] As an example, the embodiments of the present application may determine the height between the hook and the load in the following manner.

[0093] Specifically, obtain the first load cell data corresponding to when the load stays on the ground, and the first height between the hook and the boom of the crane. Then, obtain the second load cell data corresponding to when the load is lifted by the hook and leaves the ground, and when it is determined that the change in the second load cell data obtained for a continuous threshold number of frames is less than the threshold difference, obtain the second height between the current hook and the boom of the crane. Finally, determine the height between the hook and the load through the first height, the second height, and the hook height.

[0094] Exemplarily, when hanging a load by a hook and the load stays on the ground, the sling between the hook and the load is in an unloaded state. Record the load cell data G1 at this time and the height H1 between the current hook and the boom of the crane. Then, when the hook is lifted and the load leaves the ground, the sling is in a loaded state during this process, and record the load cell data G2 at this time.

[0095] Among them, when the load cell data G2 obtained for several consecutive frames remains relatively stable, it can be considered that the load has left the ground, and record the height H2 between the hook and the boom of the crane at this time.

[0096] Referring to the following formula 1, the distance H between the hook and the load can be roughly calculated through H1 and H2:

[0097] H = H2 - H1 + D Formula 1

[0098] Among them, D in the above formula 1 is used to indicate the average hook height when a person stands on the ground. In some examples, according to actual experience, the value of D can be set to 1.7, that is, D = 1.7.

[0099] Step S103: Perform clustering segmentation on the point cloud data located in the load detection area to obtain a plurality of independent circumscribed envelopes.

[0100] Exemplarily, with the hook as the center, construct a cylindrical lidar point cloud space C with a radius of 2 meters and a height of H. The radius length can be set according to the actual situation. In order to better avoid false detection, the radius is generally set to be less than or equal to 2 meters. The height H is the height value between the hook and the load.

[0101] Then, in space C, using the point cloud clustering and segmentation method, the detected objects are used to generate individual circumscribed envelopes obj_c. Taking obj_c as the object, in space O constructed with the acquisition device, such as a lidar device with a downward view on the trolley, as the original center point, using the clustering method, calculate the minimum circumscribed envelope obj_o of obj_c in the original point cloud space O, where the size of obj_o may be larger than that of obj_c.

[0102] Step S104: Determine the size of the suspended load according to the vertex positions of each individual circumscribed envelope.

[0103] Specifically, in the embodiments of the present application, the positions of the vertices of each individual circumscribed envelope can be projected onto the target plane to obtain the projection points of each vertex in the target plane. Then, according to the distances of each projection point from the center point of the target plane, the size of the suspended load is determined.

[0104] As an example, in the embodiments of the present application, the target plane can be perpendicular to the optical axis of the sensor with a downward view of the auxiliary running device of the crane, and the center point can be the center of the plane of the target plane, or the projection point of the sensor on the target plane, which can be specifically determined according to the actual situation.

[0105] For example, taking the sensor with a downward view of the auxiliary running device of the crane as the perspective, looking down at the suspended load, then in space, the three-dimensional suspended load can present a two-dimensional planar graph in this downward view. Among them, the plane where the two-dimensional coordinate system showing the two-dimensional planar graph is located can be understood as the target plane of the embodiments of the present application, and the projection point of the sensor on the target plane can be understood as the center point of the embodiments of the present application.

[0106] Further, when the embodiments of the present application need to obtain the maximum size of the suspended load, the maximum distance from each projection point to the center point can be determined as the maximum size of the suspended load.

[0107] Exemplarily, based on the minimum circumscribed envelope obj_o obtained in the above step S103, the coordinates of each vertex p(x, y, z) of each obj_o envelope are projected onto the plane perpendicular to the lidar optical axis, and then the distances from each vertex p(y, z) in this plane to the center point O are calculated. The distance R_max of the vertex p(y, z)_max with the maximum distance is selected as the maximum size of the suspended load.

[0108] Step S105: Perform safety identification on the crane based on the size of the suspended load.

[0109] As an example, embodiments of the present application can construct a three-dimensional simulation area between the hook and the auxiliary running device according to the position of the hook to obtain a first area, and construct a three-dimensional simulation area centered on the load according to the height between the hook and the load and the size of the load to obtain a second area, so as to perform safety identification on the crane based on the first area and the second area.

[0110] Specifically, embodiments of the present application can perform safety identification on the crane based on the first area and the second area in the following manner:

[0111] For example, embodiments of the present application can remove the point cloud data corresponding to the first area and the second area from the point cloud data collected in step S101 above to obtain the point cloud data of the obstacle. Then, the point cloud data of the obstacle is divided by an octree to obtain a plurality of grids. Finally, the collision distance between each grid in the plurality of grids and the first area and the second area is calculated to obtain the global safety identification result of the crane.

[0112] Exemplarily, embodiments of the present application can construct a cylindrical area C_hook with a radius of 1 meter between the hook and the trolley based on the positioning result of the hook as a three-dimensional detection space wire rope-hook simulator, that is, the first area. And based on the distance between the load and the hook and the maximum size R_max of the load, construct a spherical area S_obj with the load as the center and a radius of R_max as a simulator of the load in the three-dimensional detection space, that is, the second area.

[0113] Then, within the space range sensed by the lidar, the point clouds in the first area and the second area can be regarded as the equipment body, and other point clouds are regarded as environmental obstacles. The obstacle point cloud space is divided into grids by an octree, and the collision distance between each grid and the first area and the second area is calculated to achieve the global safety active identification of the wire rope-hook-load and surrounding obstacles.

[0114] Furthermore, after determining the point cloud of the obstacle, embodiments of the present application can also determine the global safety active warning prompt in the following manner:

[0115] For example, embodiments of the present application can calculate the angle α required for the tower crane to rotate and brake at the current speed according to the rotational angular velocity of the tower crane and the length of the crane boom; then calculate the included angle β between the point on the obstacle envelope box closest to the crane boom and the crane boom, and compare it with the braking angle α. If β ≤ α, the obstacle is displayed in red on the interface (alarm state), and the voice and light alarms are linked for automatic speed limit processing; otherwise, it is only displayed in green on the interface (warning state).

[0116] Furthermore, in order to better apply the crane safety recognition method provided in this application, the embodiments of this application can adopt a crane device as shown in Figure 2 . For example, as shown in (a) of Figure 2 , the embodiments of this application can install acquisition devices on the tower crane boom, counterweight boom, and trolley respectively. The acquisition device can include a lidar and a camera, etc. For example, as shown in (b) of Figure 2 , the acquisition device can be a combination of a lidar and a camera. By setting acquisition devices at multiple positions, acquisition perception in multiple angles and regions can be realized. As shown in (c) of Figure 2 , acquisition recognition of the area formed by the horizontal rotation of the tower base boom, acquisition recognition of the area formed by the horizontal rotation of the tower base counterweight boom, and acquisition perception of the three-dimensional space area formed by the trolley in the vertical downward direction are realized, which can more effectively and comprehensively perform safety perception of the objects around the crane, effectively improve the accuracy of acquisition recognition, improve the sensitivity of crane safety perception, and have stronger adaptability.

[0117] In addition, in order to facilitate subsequent calculations more conveniently, the embodiments of this application can also define the coordinate systems of modules such as lidar, camera, crane boom, trolley, and tower crane, and finally unify them into the tower crane coordinate system.

[0118] Among them, there are various situations of the crane safety perception described in the embodiments of this application, and specifically, it is not limited to the following two:

[0119] Perception situation 1: Active perception in the horizontal rotation direction of the crane boom.

[0120] Exemplarily, as shown in Figure 3 , the embodiments of this application can number the sensors in each detection area as 1, 2, 3, and 4 in the order of the left and right sides of the boom, and the left and right sides of the counterweight boom.

[0121] Among them, for the slewing motion of the tower crane, the tower crane controller can determine the current motion direction of the tower crane. According to the motion direction, the data collected by the sensors within the range in the same direction as the motion direction of the tower crane are automatically selected. If the tower crane is currently slewing counterclockwise, the radar data in the 1st and 4th detection areas are obtained; if the tower crane is slewing clockwise, the radar data in the 2nd and 3rd areas are obtained.

[0122] As an example, when the embodiments of this application adopt the above system for safety recognition, in order to better improve the recognition accuracy rate, the obstacles in the construction scene can be divided into two categories, one category is large-volume obstacles, and the other category is small obstacles. Based on different obstacle types, different recognition and detection methods can be adopted.

[0123] Obstacle type 1: Large-sized obstacles.

[0124] Based on obstacle type 1, in the embodiments of the present application, through radar and vision fusion detection technology, the large-sized obstacles existing around the crane can be actively identified, the spatial position information of each coordinate point of the 3D optimal bounding box of the obstacle outline can be calculated, and the rectangular envelope box of the cube in the top-down view of the tower crane is taken as the detection result of the obstacle. The detection result is unified to the reference coordinate system of the tower crane through coordinate transformation, and the vertices of the circumscribed rectangle of the obstacle are represented by polar coordinates.

[0125] Obstacle type 2: Small-sized obstacles.

[0126] For the identification of small-sized obstacles (such as steel wires and electric wires), during the detection process, when the distance is far, there are problems such as difficult imaging, sparse point clouds, and being easily filtered as noise points. Therefore, in the embodiments of the present application, through the fusion technology of two-way interaction between vision and radar information, the identification ability of small-sized obstacles can be further improved.

[0127] Exemplarily, in the embodiments of the present application, data pre-fusion is performed between the camera (with a long focal length and a far recognition distance) and the radar. The steel wire rope at a long distance is recognized by the camera, and the coordinate position in the corresponding three-dimensional point cloud space is obtained. Then, with the estimation error as the radius, local space search is performed in the original data sphere of the lidar to determine whether there are sparse irregular points formed by steel wire ropes or electric wires.

[0128] Perception situation 2: Active perception in the direction of the trolley vertically downward

[0129] Exemplarily, the embodiments of the present application can Figure 4 As shown, collect the lidar and video image data under the trolley, identify the hook and the suspended load, and perform active collision detection on the steel wire rope - hook - suspended load object based on the surrounding environment.

[0130] Then, according to the trolley luffing and hook hoisting working conditions of the tower crane, the rough positioning P1(X1, Y1, Z1) of the hook from the jib is obtained, and a cylinder with P1 as the center, R as the radius, and H for both the upper and lower heights is set as the region of interest R1 for the fine positioning of the hook. The lidar point cloud data within the space of R1 is used for hook detection. By setting R1, background interference is effectively reduced.

[0131] The above method, through sensors such as lidar and cameras installed on the crane, combines the operating conditions of the crane to actively detect dynamic and static obstacles around the crane boom - wire rope - hook - load, better perform anti-collision detection, improve the efficiency of lifting operations, and make the operation of the crane safer. In addition, through the positioning and identification of the hook and the load in the embodiments of the present application, it is possible to be not restricted by the shape, size, type, etc. of the load, so as to identify and locate more accurately and have stronger adaptability.

[0132] As Figure 5 shown, based on the same inventive concept, the embodiments of the present application provide a crane safety identification device 500, including:

[0133] A determination module 501, configured to obtain point cloud data and the position of the hook of the crane;

[0134] A construction module 502, configured to construct a load detection area based on the position of the hook, and the load detection area is located in a preset orientation of the hook;

[0135] A processing module 503, configured to perform clustering segmentation on the point cloud data located in the load detection area to obtain a plurality of independent circumscribed envelopes;

[0136] The determination module 501 is further configured to determine the size of the load according to the positions of the vertices of each independent circumscribed envelope;

[0137] The processing module 503 is further configured to perform safety identification on the crane based on the size of the load.

[0138] In some optional embodiments, the determination module 501 is specifically configured to:

[0139] Project the positions of the vertices of each independent circumscribed envelope onto a target plane to obtain the projection points of each vertex in the target plane, where the target plane is perpendicular to the optical axis of the sensor with a downward view in the auxiliary operating device of the crane; determine the size of the load according to the distances of the projection points from the center point of the target plane.

[0140] In some optional embodiments, the determination module 501 is specifically configured to:

[0141] Determine the maximum distance from each projection point to the center point as the maximum size of the load.

[0142] In some optional embodiments, the determination module 501 is specifically configured to:

[0143] Obtain the working condition information of the crane, and obtain the estimated position of the hook according to the working condition information, where the working condition information includes the luffing working condition and the hoisting working condition of the crane; construct a hook recognition area based on the estimated position of the hook, and detect the hook in the hook recognition area by matching the point cloud features corresponding to the hook with the knowledge base to obtain the actual position of the hook; the point cloud feature matching knowledge base includes the shape feature, statistical feature and reflection intensity feature of the hook.

[0144] In some optional embodiments, the processing module 503 is specifically configured to:

[0145] Track the actual position of the hook, and predict the spatial position of the hook at the current moment; update the hook recognition area according to the predicted spatial position of the hook.

[0146] In some optional embodiments, the constructing module 502 is specifically configured to:

[0147] Determine the height between the hook and the load according to the position of the hook; construct the load detection area with the position of the hook as the center according to the height and the preset radius.

[0148] In some optional embodiments, the determining module 501 is specifically configured to:

[0149] Obtain the first load cell data corresponding to the load staying on the ground, and the first height between the hook and the boom of the crane; obtain the second load cell data corresponding to the load being lifted by the hook and leaving the ground; when it is determined that the data change of the second load cell obtained in the continuous threshold number of frames is less than the threshold difference, obtain the second height between the hook and the boom of the crane at present; determine the height between the hook and the load through the first height, the second height, and the hook height.

[0150] In some optional embodiments, the processing module 503 is specifically configured to:

[0151] Construct a three-dimensional simulation area between the hook and the auxiliary operation device of the crane according to the position of the hook to obtain a first area; construct a three-dimensional simulation area centered on the load according to the height between the hook and the load and the size of the load to obtain a second area; perform safety identification on the crane based on the first area and the second area.

[0152] In some optional embodiments, the processing module 503 is specifically configured to:

[0153] Remove the point cloud data corresponding to the first region and the second region from the collected point cloud data to obtain the point cloud data of the obstacle; divide the point cloud data of the obstacle by an octree to obtain a plurality of grids; calculate the collision distance between each of the plurality of grids and the first region and the second region respectively to obtain the safety recognition result of the crane.

[0154] In some optional embodiments, the processing module 503 is further configured to:

[0155] Obtain and identify the coordinate positions of the small obstacles in the target area through a camera; search the space within a preset radius range of the coordinate positions through a lidar to determine whether there are sparse irregular points formed by the small obstacles.

[0156] Since this device is the device in the method of the embodiments of the present application, and the principle of this device to solve problems is similar to that of the method, the implementation of this device can refer to the implementation of the method, and the repeated parts will not be elaborated.

[0157] As Figure 6 shown, based on the same inventive concept, an embodiment of the present application provides an electronic device 600, including: a processor 601 and a memory 602;

[0158] The memory 602 may be a volatile memory, such as a random-access memory (RAM); the memory 602 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 602 is any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 602 may be a combination of the above memories.

[0159] The processor 601 may include one or more central processing units (CPUs), a graphics processing unit (GPU), or a digital processing unit, etc.

[0160] In the embodiments of the present application, the specific connection medium between the above-mentioned memory 602 and the processor 601 is not limited. In the embodiments of the present application Figure 6 it is connected between the memory 602 and the processor 601 through a bus 603, and the bus 603 is inFigure 6 Shown in thick lines, the bus 603 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 it is shown as only one thick line herein, but it does not mean that there is only one bus or one type of bus.

[0161] Among them, the memory 602 stores program codes. When the program codes are executed by the processor 601, the processor 601 is caused to execute the following processes:

[0162] Obtain point cloud data and the position of the hook of the crane; construct a load detection area based on the position of the hook, and the load detection area is located in a preset orientation of the hook; perform clustering segmentation on the point cloud data within the load detection area to obtain a plurality of independent circumscribed envelopes; determine the size of the load based on the positions of the vertices of each independent circumscribed envelope; perform safety identification on the crane based on the size of the load.

[0163] In some alternative embodiments, the processor 601 is specifically configured to:

[0164] Project the positions of the vertices of each independent circumscribed envelope onto a target plane to obtain the projection points of each vertex within the target plane, and the target plane is perpendicular to the sensor optical axis with a downward view of the auxiliary running device of the crane; determine the size of the load based on the distances of each projection point from the center point of the target plane.

[0165] In some alternative embodiments, the processor 601 is specifically configured to:

[0166] Determine the maximum distance from each projection point to the center point as the maximum size of the load.

[0167] In some alternative embodiments, the processor 601 is specifically configured to:

[0168] Obtain the working condition information of the crane, obtain the estimated position of the hook according to the working condition information, and the working condition information includes the luffing working condition and the hoisting working condition of the crane; construct a hook recognition area based on the estimated position of the hook, and perform hook detection on the hook recognition area through the point cloud feature matching knowledge base corresponding to the hook to obtain the actual position of the hook; the point cloud feature matching knowledge base includes the shape feature, statistical feature and reflection intensity feature of the hook.

[0169] In some alternative embodiments, the processor 601 is specifically configured to:

[0170] Track the actual position of the hook and predict the spatial position of the hook at the current moment; update the hook recognition area according to the predicted spatial position of the hook.

[0171] In some alternative embodiments, the processor 601 is specifically configured to:

[0172] Determine the height between the hook and the load according to the position of the hook; construct a load detection area centered on the position of the hook according to the height and a preset radius.

[0173] In some alternative embodiments, the processor 601 is specifically configured to:

[0174] Obtain the first load cell data corresponding to the load staying on the ground and the first height between the hook and the boom of the crane; obtain the second load cell data corresponding to the load being lifted by the hook and leaving the ground; when it is determined that the data change of the second load cell obtained in consecutive threshold frames is less than the threshold difference, obtain the second height between the current hook and the boom of the crane; determine the height between the hook and the load through the first height, the second height, and the hook height.

[0175] In some alternative embodiments, the processor 601 is specifically configured to:

[0176] Construct a three-dimensional simulation area between the hook and the auxiliary operating device of the crane according to the position of the hook to obtain a first area; construct a three-dimensional simulation area centered on the load according to the height between the hook and the load and the size of the load to obtain a second area; perform safety identification on the crane based on the first area and the second area.

[0177] In some alternative embodiments, the processor 601 is specifically configured to:

[0178] Remove the point cloud data corresponding to the first area and the second area from the collected point cloud data to obtain the point cloud data of the obstacle; divide the point cloud data of the obstacle by an octree to obtain a plurality of grids; calculate the collision distance between each grid in the plurality of grids and the first area and the second area respectively to obtain the safety identification result of the crane.

[0179] In some alternative embodiments, the processor 601 is further configured to:

[0180] Obtain and identify the coordinate positions of the small obstacles within the target area through a camera; search the space within a preset radius of the coordinate positions through a lidar to determine whether there are sparse irregular points formed by the small obstacles.

[0181] In some optional embodiments, the processor 601 is further configured to:

[0182] Obtain the first spatial point cloud data and the second spatial point cloud data corresponding to the construction scene, where the construction scene time corresponding to the first spatial point cloud data is earlier than the construction scene time corresponding to the second spatial point cloud data; process the first spatial point cloud data to obtain the first object bounding box set corresponding to the first spatial point cloud data, and process the second spatial point cloud data to obtain the second object bounding box set corresponding to the second spatial point cloud data; update the first object bounding box set based on the second object bounding box set; construct a three-dimensional model corresponding to the construction scene according to the updated first object bounding box set.

[0183] In some optional embodiments, the processor 601 is specifically configured to:

[0184] Traverse the first object bounding box set and the second object bounding box set to determine whether the object bounding boxes located in different object bounding box sets satisfy the first condition;

[0185] If satisfied, merge the object bounding boxes that satisfy the first condition, and update the first object bounding box set based on the merged object bounding boxes; or,

[0186] If not satisfied, supplement the object bounding boxes in the second object bounding box set that do not satisfy the first condition into the first object bounding box set.

[0187] In some optional embodiments, the processor 601 is specifically configured to:

[0188] Traverse the first object bounding box set and the second object bounding box set to determine whether the object bounding boxes located in different object bounding box sets satisfy the first condition;

[0189] If satisfied, replace the object bounding boxes in the first object bounding box set that satisfy the first condition with the object bounding boxes in the second object bounding box set; or,

[0190] If not satisfied, supplement the object bounding boxes in the second object bounding box set that do not satisfy the first condition into the first object bounding box set.

[0191] In some optional embodiments, the first condition includes:

[0192] The coordinate displacement difference between the object bounding boxes in different sets of object bounding boxes is not greater than a first threshold; and / or, the volume difference between the object bounding boxes in different sets of object bounding boxes is not greater than a second threshold.

[0193] In some alternative embodiments, the processor 601 is specifically configured to:

[0194] Determine the updated object bounding boxes in the first set of object bounding boxes;

[0195] Update the three-dimensional model corresponding to the constructed construction scene according to the updated object bounding boxes.

[0196] In some alternative embodiments, the processor 601 is specifically configured to:

[0197] Obtain and remove the point cloud data representing the ground from the first spatial point cloud data, and update the first spatial point cloud data; perform clustering segmentation on the updated first spatial point cloud data to obtain the first set of object bounding boxes;

[0198] Obtain and remove the point cloud data representing the ground from the second spatial point cloud data, and update the second spatial point cloud data; perform clustering segmentation on the updated second spatial point cloud data to obtain the second set of object bounding boxes.

[0199] In some alternative embodiments, the processor 601 is further configured to:

[0200] Perform clustering segmentation on the first spatial point cloud data and the second spatial point cloud data based on a target coordinate system to respectively obtain the first set of object bounding boxes and the second set of object bounding boxes based on the same coordinate system; or,

[0201] Perform clustering segmentation on the first spatial point cloud data and the second spatial point cloud data to respectively obtain the first set of object bounding boxes and the second set of object bounding boxes; unify the coordinate systems of all the objects in the first set of object bounding boxes and the second set of object bounding boxes to the target coordinate system.

[0202] Since this electronic device is the electronic device that executes the method in the embodiments of the present application, and the principle by which this electronic device solves problems is similar to that of the method, the implementation of this electronic device can refer to the implementation of the method, and the repeated parts will not be elaborated.

[0203] The embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of the crane boom safety recognition method as described above are implemented. Among them, the readable storage medium can be a non-volatile readable storage medium.

[0204] The foregoing is described with reference to block diagrams and / or flowchart illustrations of methods, apparatuses (systems) and / or computer program products according to embodiments of the present application. It should be understood that one block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, a special purpose computer, and / or other programmable devices to produce a machine, such that the instructions executed via the computer processor and / or other programmable devices create a method for implementing the functions / actions specified in the block diagrams and / or flowchart blocks.

[0205] Accordingly, the present application can also be implemented by hardware and / or software (including firmware, resident software, microcode, etc.). Further still, the present application can take the form of a computer program product on a computer-usable or computer-readable storage medium having computer-usable or computer-readable program code embodied in the medium for use by or in connection with an instruction execution system. In the context of the present application, a computer-usable or computer-readable medium can be any medium that can contain, store, communicate, transmit, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

[0206] Although the preferred embodiments of the present application have been described, additional changes and modifications can be made to these embodiments by those skilled in the art once they learn of the basic creative concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications falling within the scope of the present application.

[0207] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application is also intended to include these changes and modifications.

Claims

1. A crane safety identification method, characterized in that, The method includes: Obtaining point cloud data and the position of the hook of the crane; Constructing a load detection area based on the position of the hook, where the load detection area is located in a preset orientation of the hook; Performing clustering segmentation on the point cloud data within the load detection area to obtain multiple independent circumscribed envelopes; Determining the size of the load according to the positions of the vertices of each independent circumscribed envelope; Performing safety identification on the crane based on the size of the load; The performing safety identification on the crane based on the size of the load includes: Constructing a three-dimensional simulation area between the hook and the auxiliary operating device of the crane according to the position of the hook to obtain a first area; Constructing a three-dimensional simulation area centered on the load according to the height between the hook and the load and the size of the load to obtain a second area; Removing the point cloud data corresponding to the first area and the second area from the collected point cloud data to obtain the point cloud data of the obstacle; Dividing the point cloud data of the obstacle by an octree to obtain multiple grids; Calculating the collision distance between each of the multiple grids and the first area and the second area respectively to obtain the safety identification result of the crane.

2. The method according to claim 1, characterized in that The determining the size of the load according to the positions of the vertices of each independent circumscribed envelope includes: Projecting the positions of the vertices of each independent circumscribed envelope onto a target plane to obtain the projection points of the vertices within the target plane, where the target plane is perpendicular to the sensor optical axis facing downwards from the perspective of the auxiliary operating device of the crane; Determining the size of the load according to the distances of the projection points from the center point of the target plane.

3. The method according to claim 1, characterized in that, The obtaining point cloud data and the position of the hook of the crane includes: Obtaining the working condition information of the crane, and obtaining the estimated position of the hook according to the working condition information, where the working condition information includes the luffing working condition and the hoisting working condition of the crane; Constructing a hook identification area based on the estimated position of the hook, and performing hook detection on the hook identification area through a point cloud feature matching knowledge base of the hook to obtain the actual position of the hook; The point cloud feature matching knowledge base includes at least one of the shape feature, statistical feature, and reflection intensity feature of the hook.

4. The method according to claim 3, wherein The method further includes: Tracking the actual position of the hook and predicting the spatial position of the hook at the current moment; Updating the hook identification area according to the predicted spatial position of the hook.

5. The method according to any one of claims 1 to 4, characterized in that The constructing a load detection area based on the position of the hook includes: Determining the height between the hook and the load according to the position of the hook; Constructing the load detection area with the position of the hook as the center according to the height and a preset radius.

6. The method according to claim 5, wherein The determining the height between the hook and the load according to the position of the hook includes: Obtaining the first weighing sensor data corresponding to when the load stays on the ground, and the first height between the hook and the boom of the crane; Obtain the second weighing sensor data corresponding to when the suspended load is lifted by the hook and leaves the ground; When it is determined that the change in the second weighing sensor data obtained in consecutive threshold frames is less than the threshold difference, obtain the second height between the current hook and the boom of the crane; Determine the height between the hook and the suspended load based on the first height, the second height, and the hook height.

7. A crane safety identification device, characterized in that, The device applies the method described in any one of claims 1 to 6, and the device includes: A determination module, configured to obtain point cloud data and the position of the hook of the crane; A construction module, configured to construct a suspended load detection area based on the position of the hook, and the suspended load detection area is located in a preset orientation of the hook; A processing module, configured to perform clustering segmentation on the point cloud data located in the suspended load detection area to obtain a plurality of independent circumscribed envelopes; The determination module is further configured to determine the size of the suspended load according to the positions of the vertices of each independent circumscribed envelope; The processing module is further configured to perform safety identification on the crane based on the size of the suspended load.

8. An electronic device, characterized in that, The electronic device includes at least one processor and at least one memory, wherein the memory stores a computer program, and when the program is executed by the processor, the processor is caused to execute the method described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, It stores a computer program executable by an electronic device, and when the program runs on the electronic device, the electronic device is caused to execute the method described in any one of claims 1 to 6.

Images