Crane safety identification method and apparatus, system, and storage medium

Abstract

The present application relates to the technical field of engineering machinery, and in particular to a crane safety identification method and apparatus, a system, and a storage medium. The method comprises: acquiring point cloud data and a position of a lifting hook of a crane; constructing a hoisted object detection area on the basis of the position of the lifting hook, wherein the hoisted object detection area is located at a preset orientation relative to the lifting hook; performing clustering segmentation on point cloud data located in the hoisted object detection area to obtain a plurality of independent circumscribed envelopes; determining the size of a hoisted object on the basis of the positions of vertexes of each independent circumscribed envelope; and performing safety identification on the crane on the basis of the size of the hoisted object.

Description

Crane safety identification method, device, system and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Chinese patent application 202410237784.3 filed on March 1, 2024, the contents of which are incorporated herein by reference. Technical Field

[0003] The present application relates to the technical field of engineering machinery, and in particular to a crane safety identification method, device, system and storage medium. Background Art

[0004] Cranes are widely used in the field of construction, mainly for cargo delivery. Among them, during the process of transporting the hoisted objects, the judgment of the existence of obstacles in the surrounding area is currently mainly based on manual visual observation.

[0005] Since the size of the load that the crane can lift during operation cannot be limited, and the working environment is basically high-altitude operation, which is often limited by vision, it is often impossible to timely and effectively determine the degree of safety hazards based on the surrounding environment and the size of the load. As a result, the crane boom, wire rope, hook and load often collide with obstacles, resulting in frequent safety accidents. Summary of the Invention

[0006] The present application provides a crane safety identification method, device, system and storage medium, which are used to provide a more efficient and accurate safety identification method for cranes, effectively improving safety perception performance.

[0007] In a first aspect, an embodiment of the present application provides a crane safety identification method, the method comprising:

[0008] Acquire point cloud data and the position of a crane hook; construct a load detection area based on the position of the hook, wherein the load detection area is located at a preset position of the hook; cluster and segment the point cloud data located within the load detection area to obtain multiple independent circumscribed volumes; determine the size of the load based on the positions of the vertices of each independent circumscribed volume; and safely identify the crane based on the size of the load.

[0009] Based on the above scheme, the present application provides a crane safety identification method, which constructs a load detection area with the hook as the center, detects and identifies the load more accurately based on the load detection area, obtains the point cloud data of the load, and obtains multiple independent external envelopes by clustering and segmenting the point cloud data of the load, so that the size of the load can be determined based on the vertex positions of each independent external envelope, and effectively provides a detection and identification scheme for the size of the load, so that the crane can perform safety identification more accurately and effectively in combination with the surrounding environment during operation according to the size of the load, thereby improving the accuracy of safety identification, improving the sensitivity of the crane's safety perception, and being more adaptable.

[0010] In some optional implementations, determining the size of the suspended object based on the positions of the vertices of each independent circumscribed envelope includes:

[0011] The positions of the vertices of each independent circumscribed envelope are projected onto a target plane to obtain projection points of each vertex within the target plane. The target plane is perpendicular to the optical axis of a sensor located at a downward viewing angle of the auxiliary operating device of the crane. The size of the load is determined based on the distance of each projection point relative to the center point of the target plane.

[0012] As an example, the auxiliary operation device described in the embodiment of the present application can be set according to actual conditions. For example, the auxiliary operation device can be a trolley device in a tower crane structure. For another example, the auxiliary operation device can also be a device formed by adding a pan-tilt platform to a car crane and placing sensors on the cloud platform. This is not limited here. For the sake of simplicity, the trolley will be used as an example for introduction later.

[0013] Based on the above scheme, this application screens out the smallest circumscribed envelope of the outermost part of the load, and thus determines the size of the load based on the distance between the vertex of the smallest circumscribed envelope and the center of the target plane. This is more targeted, effectively saves calculation amount, abandons complex and useless calculation processes, improves calculation efficiency, is more convenient and quick, and has stronger adaptability.

[0014] In some optional implementations, determining the size of the suspended object based on the distances of the projection points relative to the center point of the target plane includes:

[0015] The maximum distance from each projection point to the center point is determined as the maximum size of the suspended object.

[0016] It can be understood that the embodiments of the present application can further determine the relevant dimensions of the suspended object according to actual needs. For example, in order to better reduce the probability of collision between the suspended object and surrounding obstacles and better ensure the safety of the suspended object, the embodiments of the present application can further obtain the maximum dimension of the suspended object, so as to circle a safer anti-collision area based on the maximum dimension of the suspended object; for another example, in order to better obtain relevant information of the suspended object, better understand the shape of the suspended object, and thus better determine the hanging stability of the suspended object, the embodiments of the present application can obtain the overall circled dimensions of the suspended object based on the distance of each projection point relative to the center point of the target plane, etc.

[0017] In some optional implementations, obtaining the point cloud data and the position of the crane hook includes:

[0018] The working condition information of the crane is obtained, and the estimated position of the hook is obtained based on the working condition information, wherein the working condition information includes the luffing working condition and the lifting working condition of the crane; a hook recognition area is constructed based on the estimated position of the hook, and hook detection is performed on the hook recognition area through a point cloud feature matching knowledge base corresponding to the hook to obtain the actual position of the hook; the point cloud feature matching knowledge base includes the shape features, statistical features, and reflection intensity features of the hook.

[0019] As an example, the embodiment of the present application can not only determine the position of the hook based on point cloud data recognition, but also determine the position of the hook through data collected by the image sensor. For example, the embodiment of the present application can determine the position of the hook through data collected by a binocular camera or an RGBD camera, etc., which is not limited here.

[0020] Based on the above scheme, when determining the position of the hook, the present application determines the actual hook position by combining the estimated hook position with the working condition information, which can provide a targeted hook identification area, effectively narrow the hook identification range, and improve the hook detection and identification efficiency. Moreover, after determining the hook identification area, combining the point cloud feature matching knowledge base corresponding to the hook can more accurately and effectively improve the accuracy of hook position identification, which is more convenient and quick.

[0021] In some optional implementations, the method further comprises:

[0022] The actual position of the hook is tracked to predict the spatial position of the hook at the current moment; and the hook identification area is updated according to the predicted spatial position of the hook.

[0023] Based on the above scheme, this application can effectively improve the accuracy of hook identification and positioning by tracking and detecting the position of the hook.

[0024] In some optional implementations, establishing a load detection area based on the position of the hook includes:

[0025] According to the position of the hook, the height between the hook and the load is determined; with the position of the hook as the center, the load detection area is constructed according to the height and a preset radius.

[0026] Based on the above scheme, the present application provides a method for constructing a suspended object detection area. For example, the present application can obtain a cylindrical area according to the height between the hook and the suspended object, the preset radius and the position of the hook, and thus determine the cylindrical area as the suspended object detection area. By limiting the suspended object detection area, the range of suspended object identification and detection can be effectively determined, avoiding the interference of invalid point cloud data, and performing suspended object identification more conveniently and quickly.

[0027] In some optional implementations, determining the height between the hook and the load based on the position of the hook includes:

[0028] Obtain first weighing sensor data corresponding to when the load is resting on the ground, and a first height between the hook and the boom of the crane; obtain second weighing sensor data corresponding to when the load is lifted by the hook and leaves the ground; when it is determined that the change in the second weighing sensor data obtained for a consecutive threshold number of frames is less than a threshold difference, obtain the second height between the current hook and the boom of the crane; determine the height between the hook and the load through the first height, the second height, and the hook height.

[0029] In some optional implementations, the safety identification of the crane based on the size of the load includes:

[0030] According to the position of the hook, a three-dimensional simulation area is constructed between the hook and the auxiliary operating device of the crane to obtain a first area; according to the height between the hook and the load and the size of the load, a three-dimensional simulation area centered on the load is constructed to obtain a second area; based on the first area and the second area, the crane is safely identified.

[0031] Based on the above scheme, this application plans the areas where safety hazards need to be checked by constructing the first area and the second area, so that targeted safety identification can be carried out based on the collected surrounding environment point cloud data, combined with the first area and the second area, which is faster and more efficient, and improves the accuracy of safety identification.

[0032] In some optional implementations, the performing safety identification on the crane based on the first area and the second area includes:

[0033] The collected point cloud data is removed from the point cloud data corresponding to the first area and the second area to obtain point cloud data of the obstacle; the point cloud data of the obstacle is divided using an octree to obtain multiple grids; and a collision distance is calculated between each grid in the multiple grids and the first area and the second area to obtain a safety identification result of the crane.

[0034] In a second aspect, an embodiment of the present application provides a crane safety identification device, the device comprising:

[0035] a determination module for acquiring point cloud data and the position of the crane's hook;

[0036] A construction module, configured to construct a load detection area based on the position of the hook, wherein the load detection area is located at a preset position of the hook;

[0037] a processing module, configured to cluster and segment the point cloud data within the suspended object detection area to obtain a plurality of independent external envelopes;

[0038] The determining module is further configured to determine the size of the suspended object based on the positions of the vertices of each independent circumscribed envelope;

[0039] The processing module is further configured to perform safety identification on the crane based on the size of the load.

[0040] In some optional implementations, the determining module is specifically configured to:

[0041] The positions of the vertices of each independent circumscribed envelope are projected onto a target plane to obtain projection points of each vertex within the target plane. The target plane is perpendicular to the optical axis of a sensor located at a downward viewing angle of the auxiliary operating device of the crane. The size of the load is determined based on the distance of each projection point relative to the center point of the target plane.

[0042] In some optional implementations, the determining module is specifically configured to:

[0043] The maximum distance from each projection point to the center point is determined as the maximum size of the suspended object.

[0044] In some optional implementations, the determining module is specifically configured to:

[0045] The working condition information of the crane is obtained, and the estimated position of the hook is obtained based on the working condition information, wherein the working condition information includes the luffing working condition and the lifting working condition of the crane; a hook recognition area is constructed based on the estimated position of the hook, and hook detection is performed on the hook recognition area through a point cloud feature matching knowledge base corresponding to the hook to obtain the actual position of the hook; the point cloud feature matching knowledge base includes the shape features, statistical features, and reflection intensity features of the hook.

[0046] In some optional implementations, the processing module is specifically configured to:

[0047] The actual position of the hook is tracked to predict the spatial position of the hook at the current moment; and the hook identification area is updated according to the predicted spatial position of the hook.

[0048] In some optional implementations, the building blocks are specifically configured to:

[0049] According to the position of the hook, the height between the hook and the load is determined; with the position of the hook as the center, the load detection area is constructed according to the height and a preset radius.

[0050] In some optional implementations, the determining module is specifically configured to:

[0051] Obtain first weighing sensor data corresponding to when the load rests on the ground, and a first height between the hook and the boom of the crane; obtain second weighing sensor data corresponding to when the load is lifted by the hook and leaves the ground; when it is determined that the change in the data of the second weighing sensor obtained for a consecutive threshold number of frames is less than a threshold difference, obtain the second height between the current hook and the boom of the crane; determine the height between the hook and the load through the first height, the second height, and the hook height.

[0052] In some optional implementations, the processing module is specifically configured to:

[0053] According to the position of the hook, a three-dimensional simulation area is constructed between the hook and the auxiliary operating device of the crane to obtain a first area; according to the height between the hook and the load and the size of the load, a three-dimensional simulation area centered on the load is constructed to obtain a second area; based on the first area and the second area, the crane is safely identified.

[0054] In some optional implementations, the processing module is specifically configured to:

[0055] The collected point cloud data is removed from the point cloud data corresponding to the first area and the second area to obtain point cloud data of the obstacle; the point cloud data of the obstacle is divided using an octree to obtain multiple grids; and a collision distance is calculated between each grid in the multiple grids and the first area and the second area to obtain a safety identification result of the crane.

[0056] In a third aspect, an embodiment of the present application provides an electronic device, comprising at least one processor and at least one memory, wherein the memory stores a computer program, and when the program is executed by the processor, the processor executes any crane safety identification method described in the first aspect.

[0057] In a fourth aspect, an embodiment of the present application provides a chip system, comprising: a processor and an interface, wherein the processor is used to call and execute a computer program from the interface. When the processor executes the computer program, the method described in the above-mentioned first aspect or any possible design of the first aspect can be implemented.

[0058] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program for executing the method described in the first aspect or any possible design of the first aspect.

[0059] In a sixth aspect, an embodiment of the present application further provides a computer program product, comprising a computer program, which, when executed, can implement the method described in the first aspect or any possible design of the first aspect.

[0060] The beneficial effects produced by any of the second to sixth aspects are the same as those of the first aspect. For details, please refer to the detailed introduction of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] FIG1 is a schematic diagram of a crane safety identification method provided by an embodiment of the present application;

[0062] FIG2 is a schematic diagram of a crane safety identification system provided in an embodiment of the present application;

[0063] FIG3 is a schematic diagram showing a scanning area according to an embodiment of the present application;

[0064] FIG4 is a schematic diagram of a space construction provided in an embodiment of the present application;

[0065] FIG5 is a schematic structural diagram of a crane safety identification device provided in an embodiment of the present application;

[0066] FIG6 is a schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] To make the objectives, technical solutions, and advantages of this application more clear, this application will be further described in detail below with reference to the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0068] Cranes are widely used in construction, primarily for cargo delivery. However, since the size of the loads a crane can lift cannot be specified during operation, and the operating environment is typically high-altitude, with limited visibility, it's often impossible to effectively and timely assess the extent of safety hazards based on the surrounding environment and the size of the load. Consequently, collisions between the crane boom, wire rope, hook, and load often occur, leading to frequent accidents.

[0069] For example, one current method for crane-based safety identification involves establishing a boom tower crane model and calculating the shortest distance between the booms of two adjacent boom tower cranes, thereby determining whether the crane is operating dangerously. However, this method only works for tower cranes with known models. It cannot identify unknown tower cranes or other obstacles. Furthermore, even with known tower crane models, safety hazards often arise due to the oversized load.

[0070] For example, another current method of crane-based safety identification is mainly through real-time monitoring of the rotation status of adjacent fixed crane booms. When the boom of one crane enters the warning area of ​​another crane, an early warning signal is issued to alert the staff. This method also obtains the status information of adjacent cranes and performs anti-collision detection between multiple machines. However, a single machine itself does not have anti-collision detection, safety identification and other functions.

[0071] In summary, the current crane-based safety identification solution can only perform safety identification in simple scenarios. For unknown tower cranes or other obstacles, it cannot perform flexible detection based on the actual size of the load in a timely and effective manner, and there is still a high safety hazard.

[0072] Based on this, the present application provides a crane safety identification method, device and storage medium. By identifying the size of the hoisted object, the crane can perform safety identification more accurately and effectively in combination with the surrounding environment during operation based on the size of the hoisted object, thereby improving the accuracy, sensitivity and adaptability of safety identification.

[0073] Furthermore, the crane safety identification method provided in the embodiment of the present application can also actively detect dynamic and static obstacles around the crane boom, wire rope, hook and load through sensors such as laser radar and cameras installed on the crane, combined with the crane operating conditions, and perform anti-collision detection in real time and efficiently, thereby improving the efficiency of lifting operations and making crane operation safer.

[0074] The following is an explanation of the terms involved in the embodiments of the present application, which are not limited to the following descriptions:

[0075] (1) Polar coordinates refer to taking a fixed point O in a plane, called the pole, drawing a ray Ox, called the polar axis, and then selecting a unit of length and a positive angle direction (usually counterclockwise). For any point M in the plane, ρ represents the length of the line segment OM, and θ represents the angle from Ox to OM. ρ is called the polar diameter of point M, and θ is called the polar angle of point M. The ordered number pair (ρ, θ) is called the polar coordinates of point M. The coordinate system established in this way is called a polar coordinate system.

[0076] (2) Envelope refers to a figure formed by the interweaving of many elliptical curves.

[0077] The minimum enveloping volume is a bounded convex set in three-dimensional space. For a given set of points, there exists a minimum enveloping volume that completely contains these points. The minimum enveloping volume can be defined by computing a set of points (vertices) in the set that define the geometry of the enveloping volume.

[0078] The following will be combined with the accompanying drawings and specific embodiments to explain in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0079] An embodiment of the present application provides a crane safety identification method, as shown in FIG1 , which may include:

[0080] Step S101: Acquire point cloud data and the position of the crane hook.

[0081] Among them, the point cloud data obtained in step S101 of the embodiment of the present application can be point cloud data collected based on a set target area. For example, the target area can be an observation area set for a crane; for another example, the target area can be an area that can be collected by a collection device on a crane. The specific area can be obtained according to actual conditions and is not limited here.

[0082] Furthermore, the embodiments of the present application provide multiple methods for determining the position of the crane hook, which are not limited to the following:

[0083] Determination method 1: Determine the position of the hook based on point cloud data recognition.

[0084] Determination method 2: Determine the position of the hook through data collected by the image sensor.

[0085] For example, the embodiment of the present application can determine the position of the hook through data collected by a binocular camera, an RGBD camera, etc., which is not limited here.

[0086] In order to better describe the embodiment of the present application, the embodiment of the present application is based on determination method 1 and describes in detail the content of determining the position of the hook:

[0087] Specifically, the working condition information of the crane is obtained, and the estimated position of the hook is obtained based on the working condition information, wherein the working condition information includes the boom length variation condition and the lifting condition of the crane; a hook recognition area is constructed based on the estimated position of the hook, and the hook detection is performed on the hook recognition area through the point cloud feature matching knowledge base corresponding to the hook to obtain the actual position of the hook; the point cloud feature matching knowledge base includes the shape features, statistical features and reflection intensity features of the hook.

[0088] In order to better and more accurately locate and identify the hook, the embodiment of the present application can track the actual position of the hook and predict the spatial position of the hook at the current moment, thereby updating the hook identification area based on the predicted spatial position of the hook. For example, the embodiment of the present application can locate and identify the hook by combining a detector and a tracker.

[0089] Exemplarily, the embodiment of the present application can first perform initial hook positioning and identification based on the detector. For example, the point cloud feature matching knowledge base of the hook designed in the embodiment of the present application may include: shape features (such as aspect ratio, height difference, bounding box size, etc.), statistical features (such as three-dimensional invariant moment, three-dimensional covariance matrix), and a weighted combination of reflection intensity features. By collecting a large number of hook training samples (hook point cloud data at different distances), the feature weight is calculated according to the change law of each feature, and finally it is ensured that the combined feature output result of the hook remains within a certain interval change range, thereby realizing automatic detection of the hook.

[0090] Then, when the embodiment of the present application subsequently locates and identifies the hook, the actual detection position of the hook can be obtained through the detector, and the spatial position of the hook at the current moment can be predicted by tracking the detection results. If the detector fails to detect the hook or mistakenly detects another object, the tracker's prediction result can be used to replace the detector, while filtering out other false detections, further reducing the point cloud space R2 of the next frame detector, replacing R1 with R2, and then using the detector to detect the hook in the R2 space, ultimately improving the accuracy of hook identification and positioning.

[0091] Step S102: constructing a load detection area based on the position of the hook, wherein the load detection area is located at a preset position of the hook.

[0092] Specifically, in the implementation of this application, the height between the hook and the load can be determined based on the position of the hook, and then the load detection area can be constructed based on the height and a preset radius with the position of the hook as the center.

[0093] For example, the load detection area constructed in the embodiment of the present application can be a cylindrical area formed based on the position of the hook, the height between the hook and the load, and a preset radius.

[0094] As an example, the embodiment of the present application can determine the height between the hook and the load in the following manner.

[0095] Specifically, first weighing sensor data corresponding to when the load is resting on the ground and a first height between the hook and the boom of the crane are obtained, and then second weighing sensor data corresponding to when the load is lifted by the hook and leaves the ground are obtained, and when it is determined that the change in the second weighing sensor data obtained for a consecutive threshold number of frames is less than a threshold difference, the second height between the current hook and the boom of the crane is obtained, and finally, the height between the hook and the load is determined by the first height, the second height, and the hook height.

[0096] For example, when a load is hung by a hook and the load stays on the ground, the rope between the hook and the load is in an unstressed state, and the weighing sensor data G1 and the current height H1 between the hook and the crane boom are recorded at this time; then, when the hook is raised, the load leaves the ground, and the rope is in a stressed state during this process, and the weighing sensor data G2 is recorded at this time.

[0097] When the weighing sensor data G2 acquired for several consecutive frames remains relatively stable, it can be considered that the load has left the ground, and the height H2 between the hook and the crane boom at this time is recorded.

[0098] Refer to the following formula 1, and the distance H between the hook and the load can be roughly calculated using H1 and H2: H=H2-H1+D Formula 1

[0099] Wherein, D in the above formula 1 is used to indicate the average hook height of a person standing on the ground. In some examples, the value of D can be set to 1.7 based on practical experience, that is, D=1.7.

[0100] Step S103: performing clustering and segmentation on the point cloud data within the suspended object detection area to obtain a plurality of independent external envelopes.

[0101] For example, a cylindrical LiDAR point cloud space C is constructed with a hook as the center, a radius of 2 meters, and a height of H. The radius can be set based on actual conditions; to better avoid false detections, the radius is generally set to be less than or equal to 2 meters. The height H is the height between the hook and the load.

[0102] Then, within space C, point cloud clustering and segmentation are used to generate independent bounding boxes obj_c for each detected object. Taking obj_c as the object, a clustering method is used to calculate the minimum bounding box obj_o of obj_c in space O, which is constructed with the acquisition device (e.g., a downward-facing lidar device located on a vehicle) as the original center point. The size of obj_o may be larger than that of obj_c.

[0103] Step S104: determining the size of the suspended object according to the positions of the vertices of each independent circumscribed envelope.

[0104] Specifically, the embodiment of the present application can project the positions of each vertex of each independent external envelope onto the target plane to obtain the projection points of each vertex within the target plane, and then determine the size of the suspended object based on the distance of each projection point relative to the center point of the target plane.

[0105] As an example, the target plane described in the embodiment of the present application can be perpendicular to the optical axis of the sensor located at the auxiliary operation device of the crane facing downward, and the center point can be the center of the plane of the target plane or the projection point of the sensor on the target plane. The specific method can be determined based on actual conditions.

[0106] For example, if the auxiliary operating device of the crane takes the sensor facing downward as the perspective and looks down at the load, the three-dimensional load in space can be presented as a two-dimensional plane figure under this top-down perspective. The plane where the two-dimensional coordinate system displaying the two-dimensional plane figure is located can be understood as the target plane of the embodiment of the present application, and the projection point of the sensor on the target plane can be understood as the center point of the embodiment of the present application.

[0107] Furthermore, when the embodiment of the present application needs to obtain the maximum size of the suspended object, the maximum distance from each projection point to the center point can be determined as the maximum size of the suspended object.

[0108] Exemplarily, based on the minimum circumscribed envelope obj_o obtained in the above step S103, the coordinates of each vertex p(x, y, z) of each obj_o envelope are projected onto a plane perpendicular to the laser radar optical axis, and then the distance from each vertex p(y, z) in the plane to the center point O is calculated, and the distance R_max of the vertex p(y, z)_max with the largest distance is selected as the maximum size of the suspended object.

[0109] Step S105: Based on the size of the hoisted object, perform safety identification on the crane.

[0110] As an example, an embodiment of the present application can construct a three-dimensional simulation area between the hook and the auxiliary operating device according to the position of the hook to obtain a first area, and construct a three-dimensional simulation area centered on the load according to the height between the hook and the load and the size of the load to obtain a second area, thereby safely identifying the crane based on the first area and the second area.

[0111] Specifically, in the embodiment of the present application, the crane can be safely identified based on the first area and the second area in the following manner:

[0112] For example, an embodiment of the present application can remove the point cloud data corresponding to the first area and the second area from the point cloud data collected in the above step S101 to obtain the point cloud data of the obstacle, and then divide the point cloud data of the obstacle using an octree to obtain multiple grids. Finally, each grid in the multiple grids is respectively calculated with the first area and the second area to obtain the global safety identification result of the crane.

[0113] For example, in this embodiment, based on the hook positioning results, a cylindrical region C_hook with a radius of 1 meter between the hook and the trolley can be constructed as the wire rope-hook simulation in the three-dimensional detection space, i.e., the first region. Furthermore, based on the distance between the load and the hook and the maximum size R_max of the load, a spherical region S_obj with a radius R_max centered on the load can be constructed as the load simulation in the three-dimensional detection space, i.e., the second region.

[0114] Then, based on the spatial range perceived by the lidar, the point clouds in the first and second areas can be used as the device body, and the other point clouds can be used as environmental obstacles. The point cloud space of these obstacles can be divided into grids using an octree, and the collision distance between each grid and the first and second areas is calculated to achieve full-area safe active identification of the wire rope-hook-load and surrounding obstacles.

[0115] Furthermore, after determining the point cloud of the obstacle, the embodiment of the present application can also determine the global safety active warning prompt in the following manner:

[0116] For example, the embodiment of the present application can calculate the angle α required for the tower crane's slewing braking at the current speed based on the tower crane's slewing angular velocity and the crane boom length; then calculate the angle β between the nearest point on the obstacle envelope box to the crane boom and the crane boom, and compare it with the braking angle α. If β≤α, the obstacle is displayed in red on the interface (alarm state), and a voice and light alarm is triggered, and the speed limit is automatically processed; otherwise, it is only displayed in green on the interface (warning state).

[0117] Furthermore, in order to better apply the crane safety identification method provided by the present application, the embodiment of the present application can adopt the crane device shown in Figure 2. For example, as shown in Figure 2 (a), the embodiment of the present application can install collection devices on the tower crane boom, balance arm, and mobile trolley respectively. The collection device may include a laser radar and a camera, etc. For example, as shown in Figure 2 (b), the collection device may be a combination of a laser radar and a camera. By setting the collection device at multiple positions, multi-angle and multi-region collection and perception can be achieved. As shown in Figure 2 (c), the collection and identification of the area formed by the horizontal rotation of the tower base boom and the collection and identification of the area formed by the horizontal rotation of the tower base balance arm are achieved, as well as the collection and perception of the three-dimensional space area formed by the vertical downward direction of the trolley. It can more effectively and comprehensively perceive the safety of objects around the crane in multiple areas, effectively improve the accuracy of collection and identification, improve the sensitivity of crane safety perception, and be more adaptable.

[0118] In addition, in order to facilitate subsequent calculations, the embodiment of the present application can also define the coordinate systems of modules such as the lidar, camera, crane arm, trolley, tower crane, etc., and finally unify them into the tower crane coordinate system.

[0119] Among them, the crane safety perception conditions described in the embodiments of the present application include multiple, but not limited to the following two:

[0120] Perception situation 1: Active perception of the horizontal rotation direction of the crane's boom.

[0121] For example, as shown in FIG3 , in an embodiment of the present application, the sensors in each detection area may be numbered as 1, 2, 3, and 4 in the order of the left side and right side of the boom, and the left side and right side of the balance arm.

[0122] Among them, the tower crane's rotation movement can be determined by the tower crane controller. According to the movement direction, the sensor collection data in the same direction as the tower crane's movement direction is automatically screened. If the tower crane is currently rotating counterclockwise, the radar data of detection areas 1 and 4 are obtained; if the tower crane is rotating clockwise, the radar data of areas 2 and 3 are obtained.

[0123] As an example, when the embodiment of the present application adopts the above-mentioned system for safety identification, in order to better improve the recognition accuracy, the obstacles in the construction scene can be divided into two categories, one is larger obstacles and the other is small obstacles. Based on different obstacle types, different identification and detection methods can be adopted.

[0124] Obstacle type 1: Large obstacles.

[0125] Based on obstacle type 1, the embodiment of the present application can actively identify large obstacles around the crane through radar and vision fusion detection technology, calculate the spatial position information of each coordinate point of the 3D optimal bounding box of the obstacle's outer contour, and take the rectangular envelope box of the cube from the tower crane's top-down perspective as the obstacle detection result. The detection result is unified to the tower crane's reference coordinate system through coordinate transformation, and each vertex of the obstacle's circumscribed rectangle is represented by polar coordinates.

[0126] Obstacle type 2: small obstacles.

[0127] Regarding the identification of small obstacles (such as wire ropes and electric wires), during the detection process, when the distance is far, there are problems such as difficult imaging, sparse point clouds, and easy filtering out as noise points. Therefore, the embodiment of the present application can further improve the recognition ability of small obstacles through the fusion technology of two-way interaction of visual and radar information.

[0128] Illustratively, the embodiment of the present application performs data pre-fusion through a camera (with a longer focal length and a longer recognition distance) and a radar, identifies a steel wire rope at a distance through the camera, obtains the coordinate position of the corresponding three-dimensional point cloud space, and then uses the estimated error as the radius to search the local space of the laser radar raw data sphere to determine whether there are sparse irregular points formed by steel wire ropes or electric wires.

[0129] Perception situation 2: The car actively perceives the vertical downward direction

[0130] Exemplarily, an embodiment of the present application can collect lidar and video image data from under the trolley, as shown in FIG4 , to identify the hook and load, and implement active collision detection of the wire rope-hook-load based on the surrounding environment.

[0131] Then, according to the boom length variation and hook lifting conditions of the tower crane trolley, the coarse positioning P1 (X1, Y1, Z1) of the hook from the boom is obtained, and a cylinder with P1 as the center, R as the radius, and H as the upper and lower heights is set as the precise positioning area of ​​interest R1 for the hook. The hook is detected on the lidar point cloud data in the R1 space. By setting R1, background interference is effectively reduced.

[0132] The above method, using sensors such as a LiDAR and camera installed on the crane and combined with the crane's operating conditions, proactively detects dynamic and static obstacles around the crane's boom, wire rope, hook, and load, enabling better collision avoidance detection, improving lifting efficiency, and making crane operation safer. Furthermore, by locating and identifying the hook and load, the embodiments of the present application are independent of the shape, size, and type of the load, resulting in more accurate identification and greater adaptability.

[0133] As shown in FIG5 , based on the same inventive concept, an embodiment of the present application provides a crane safety identification device 500 , comprising:

[0134] Determining module 501, for obtaining point cloud data and the position of the crane hook;

[0135] A construction module 502 is configured to construct a load detection area based on the position of the hook, wherein the load detection area is located at a preset position of the hook;

[0136] The processing module 503 is used to cluster and segment the point cloud data located in the suspended object detection area to obtain multiple independent external envelopes;

[0137] The determining module 501 is further configured to determine the size of the suspended object according to the positions of the vertices of each independent circumscribed envelope;

[0138] The processing module 503 is further configured to perform safety identification on the crane based on the size of the load.

[0139] In some optional implementation manners, the determining module 501 is specifically configured to:

[0140] The positions of the vertices of each independent circumscribed envelope are projected onto a target plane to obtain projection points of each vertex within the target plane. The target plane is perpendicular to the optical axis of a sensor located at a downward viewing angle of the auxiliary operating device of the crane. The size of the load is determined based on the distance of each projection point relative to the center point of the target plane.

[0141] In some optional implementation manners, the determining module 501 is specifically configured to:

[0142] The maximum distance from each projection point to the center point is determined as the maximum size of the suspended object.

[0143] In some optional implementation manners, the determining module 501 is specifically configured to:

[0144] The working condition information of the crane is obtained, and the estimated position of the hook is obtained based on the working condition information, wherein the working condition information includes the luffing working condition and the lifting working condition of the crane; a hook recognition area is constructed based on the estimated position of the hook, and hook detection is performed on the hook recognition area through a point cloud feature matching knowledge base corresponding to the hook to obtain the actual position of the hook; the point cloud feature matching knowledge base includes the shape features, statistical features, and reflection intensity features of the hook.

[0145] In some optional implementations, the processing module 503 is specifically configured to:

[0146] The actual position of the hook is tracked to predict the spatial position of the hook at the current moment; and the hook identification area is updated according to the predicted spatial position of the hook.

[0147] In some optional implementations, the building module 502 is specifically configured to:

[0148] According to the position of the hook, the height between the hook and the load is determined; with the position of the hook as the center, the load detection area is constructed according to the height and a preset radius.

[0149] In some optional implementation manners, the determining module 501 is specifically configured to:

[0150] Obtain first weighing sensor data corresponding to when the load rests on the ground, and a first height between the hook and the boom of the crane; obtain second weighing sensor data corresponding to when the load is lifted by the hook and leaves the ground; when it is determined that the change in the data of the second weighing sensor obtained for a consecutive threshold number of frames is less than a threshold difference, obtain the second height between the current hook and the boom of the crane; determine the height between the hook and the load through the first height, the second height, and the hook height.

[0151] In some optional implementations, the processing module 503 is specifically configured to:

[0152] According to the position of the hook, a three-dimensional simulation area is constructed between the hook and the auxiliary operating device of the crane to obtain a first area; according to the height between the hook and the load and the size of the load, a three-dimensional simulation area centered on the load is constructed to obtain a second area; based on the first area and the second area, the crane is safely identified.

[0153] In some optional implementations, the processing module 503 is specifically configured to:

[0154] The collected point cloud data is removed from the point cloud data corresponding to the first area and the second area to obtain point cloud data of the obstacle; the point cloud data of the obstacle is divided using an octree to obtain multiple grids; and a collision distance is calculated between each grid in the multiple grids and the first area and the second area to obtain a safety identification result of the crane.

[0155] In some optional implementations, the processing module 503 is further configured to:

[0156] The coordinate positions of small obstacles in the target area are acquired and identified by a camera; the space within a preset radius of the coordinate positions is searched by a laser radar to determine whether there are sparse irregular points formed by the small obstacles.

[0157] Since the device is the device in the method in the embodiment of the present application, and the principle of solving the problem by the device is similar to that of the method, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0158] As shown in FIG6 , based on the same inventive concept, an embodiment of the present application provides an electronic device 600 , including: a processor 601 and a memory 602 ;

[0159] Memory 602 may be a volatile memory, such as random-access memory (RAM); a non-volatile memory, such as read-only memory, flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 602 may be a combination of the aforementioned memories.

[0160] The processor 601 may include one or more central processing units (CPUs), graphics processing units (GPUs) or digital processing units, etc.

[0161] The specific connection medium between the memory 602 and the processor 601 is not limited in the embodiments of the present application. In FIG6 , the memory 602 and the processor 601 are connected via a bus 603. Bus 603 is represented by a bold line in FIG6 . Bus 603 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, FIG6 shows only one bold line, but this does not mean that there is only one bus or only one type of bus.

[0162] The memory 602 stores program codes. When the program codes are executed by the processor 601, the processor 601 performs the following process:

[0163] Acquire point cloud data and the position of a crane hook; construct a load detection area based on the position of the hook, wherein the load detection area is located at a preset position of the hook; cluster and segment the point cloud data located within the load detection area to obtain multiple independent circumscribed volumes; determine the size of the load based on the positions of the vertices of each independent circumscribed volume; and safely identify the crane based on the size of the load.

[0164] In some optional implementations, the processor 601 is specifically configured to:

[0165] The positions of the vertices of each independent circumscribed envelope are projected onto a target plane to obtain projection points of each vertex within the target plane. The target plane is perpendicular to the optical axis of a sensor located at a downward viewing angle of the auxiliary operating device of the crane. The size of the load is determined based on the distance of each projection point relative to the center point of the target plane.

[0166] In some optional implementations, the processor 601 is specifically configured to:

[0167] The maximum distance from each projection point to the center point is determined as the maximum size of the suspended object.

[0168] In some optional implementations, the processor 601 is specifically configured to:

[0169] The working condition information of the crane is obtained, and the estimated position of the hook is obtained based on the working condition information, wherein the working condition information includes the luffing working condition and the lifting working condition of the crane; a hook recognition area is constructed based on the estimated position of the hook, and hook detection is performed on the hook recognition area through a point cloud feature matching knowledge base corresponding to the hook to obtain the actual position of the hook; the point cloud feature matching knowledge base includes the shape features, statistical features, and reflection intensity features of the hook.

[0170] In some optional implementations, the processor 601 is specifically configured to:

[0171] The actual position of the hook is tracked to predict the spatial position of the hook at the current moment; and the hook identification area is updated according to the predicted spatial position of the hook.

[0172] In some optional implementations, the processor 601 is specifically configured to:

[0173] According to the position of the hook, the height between the hook and the load is determined; with the position of the hook as the center, the load detection area is constructed according to the height and a preset radius.

[0174] In some optional implementations, the processor 601 is specifically configured to:

[0175] Obtain first weighing sensor data corresponding to when the load rests on the ground, and a first height between the hook and the boom of the crane; obtain second weighing sensor data corresponding to when the load is lifted by the hook and leaves the ground; when it is determined that the change in the data of the second weighing sensor obtained for a consecutive threshold number of frames is less than a threshold difference, obtain the second height between the current hook and the boom of the crane; determine the height between the hook and the load through the first height, the second height, and the hook height.

[0176] In some optional implementations, the processor 601 is specifically configured to:

[0177] According to the position of the hook, a three-dimensional simulation area is constructed between the hook and the auxiliary operating device of the crane to obtain a first area; according to the height between the hook and the load and the size of the load, a three-dimensional simulation area centered on the load is constructed to obtain a second area; based on the first area and the second area, the crane is safely identified.

[0178] In some optional implementations, the processor 601 is specifically configured to:

[0179] The collected point cloud data is removed from the point cloud data corresponding to the first area and the second area to obtain point cloud data of the obstacle; the point cloud data of the obstacle is divided using an octree to obtain multiple grids; and a collision distance is calculated between each grid in the multiple grids and the first area and the second area to obtain a safety identification result of the crane.

[0180] In some optional implementations, the processor 601 is further configured to:

[0181] The coordinate positions of small obstacles in the target area are acquired and identified by a camera; the space within a preset radius of the coordinate positions is searched by a laser radar to determine whether there are sparse irregular points formed by the small obstacles.

[0182] In some optional implementations, the processor 601 is further configured to:

[0183] Acquire first spatial point cloud data and second spatial point cloud data corresponding to a construction scene, wherein a construction scene moment corresponding to the first spatial point cloud data is earlier than a construction scene moment corresponding to the second spatial point cloud data; process the first spatial point cloud data to obtain a first object envelope box set corresponding to the first spatial point cloud data, and process the second spatial point cloud data to obtain a second object envelope box set corresponding to the second spatial point cloud data; update the first object envelope box set based on the second object envelope box set; and construct a three-dimensional model corresponding to the construction scene based on the updated first object envelope box set.

[0184] In some optional implementations, the processor 601 is specifically configured to:

[0185] Traversing the first object envelope box set and the second object envelope box set to determine whether a first condition is satisfied between object envelope boxes in different object envelope box sets;

[0186] If so, merging the object envelope boxes that meet the first condition, and updating the first object envelope box set based on the merged object envelope boxes; or

[0187] If not, the object envelope boxes in the second object envelope box set that do not meet the first condition are added to the first object envelope box set.

[0188] In some optional implementations, the processor 601 is specifically configured to:

[0189] Traversing the first object envelope box set and the second object envelope box set to determine whether a first condition is satisfied between object envelope boxes in different object envelope box sets;

[0190] If the condition is satisfied, the object envelope box in the second object envelope box set that meets the first condition is used to replace the corresponding object envelope box in the first object envelope box set; or

[0191] If not, the object envelope boxes in the second object envelope box set that do not meet the first condition are added to the first object envelope box set.

[0192] In some optional implementations, the first condition includes:

[0193] The coordinate displacement difference between the object envelope boxes located in different object envelope box sets is not greater than a first threshold; and / or the volume difference between the object envelope boxes located in different object envelope box sets is not greater than a second threshold.

[0194] In some optional implementations, the processor 601 is specifically configured to:

[0195] determining an updated object envelope box in the first object envelope box set;

[0196] The constructed three-dimensional model corresponding to the construction scene is updated according to the updated object envelope box.

[0197] In some optional implementations, the processor 601 is specifically configured to:

[0198] removing point cloud data representing the ground from the first spatial point cloud data and updating the first spatial point cloud data; performing clustering and segmentation on the updated first spatial point cloud data to obtain a first object envelope box set;

[0199] The point cloud data used to represent the ground in the second spatial point cloud data is removed, and the second spatial point cloud data is updated; clustering and segmentation are performed on the updated second spatial point cloud data to obtain the second object envelope box set.

[0200] In some optional implementations, the processor 601 is further configured to:

[0201] Performing clustering and segmentation on the first spatial point cloud data and the second spatial point cloud data based on a target coordinate system to obtain the first object envelope box set and the second object envelope box set based on the same coordinate system respectively; or,

[0202] Clustering and segmenting the first spatial point cloud data and the second spatial point cloud data to obtain the first object envelope box set and the second object envelope box set, respectively; and aligning the coordinate systems of all objects in the first object envelope box set and the second object envelope box set to the target coordinate system.

[0203] Since the electronic device is the electronic device that executes the method in the embodiment of the present application, and the principle of solving the problem by the electronic device is similar to that of the method, the implementation of the electronic device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0204] The embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the crane boom safety identification method described above.

[0205] The present application is described above with reference to block diagrams and / or flow charts illustrating methods, devices (systems) and / or computer program products according to embodiments of the present application. It should be understood that a block of a block diagram and / or flow chart, as well as a combination of blocks of a block diagram and / or flow chart, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer and / or other programmable device to produce a machine, so that the instructions executed by the computer processor and / or other programmable device create a method for implementing the functions / actions specified in the block diagram and / or flow chart block.

[0206] Accordingly, the present application may also be implemented using hardware and / or software (including firmware, resident software, microcode, etc.). Furthermore, the present application may take the form of a computer program product on a computer-usable or computer-readable storage medium having computer-usable or computer-readable program code embodied in the medium for use by or in conjunction with an instruction execution system. In the context of the present application, a computer-usable or computer-readable medium may be any medium that can contain, store, communicate, transmit, or convey a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0207] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0208] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A crane safety identification method, the method comprising: Obtain point cloud data and the position of the crane's hook; Establishing a load detection area based on the position of the hook, wherein the load detection area is located at a preset position of the hook; Clustering and segmenting the point cloud data within the load detection area to obtain multiple independent external envelopes; Determining the size of the suspended object according to the positions of the vertices of each independent circumscribed envelope; The crane is safely identified based on the size of the load.

2. The method according to claim 1, wherein Determining the size of the suspended object according to the positions of the vertices of the independent circumscribed envelopes includes: Projecting the positions of the vertices of each independent circumscribed envelope onto a target plane to obtain projection points of each vertex within the target plane, wherein the target plane is perpendicular to the optical axis of a sensor located at a downward viewing angle of the auxiliary operating device of the crane; The size of the suspended object is determined according to the distance between each projection point and the center point of the target plane.

3. The method according to claim 1, wherein The step of obtaining the point cloud data and the position of the crane hook includes: Acquiring operating condition information of the crane, and obtaining an estimated position of the hook based on the operating condition information, wherein the operating condition information includes a luffing operating condition and a lifting operating condition of the crane; A hook recognition area is constructed based on the estimated position of the hook, and hook detection is performed on the hook recognition area through a point cloud feature matching knowledge base corresponding to the hook to obtain the actual position of the hook; The point cloud feature matching knowledge base includes at least one of the shape features, statistical features and reflection intensity features of the hook.

4. The method of claim 3, further comprising: Tracking the actual position of the hook and predicting the spatial position of the hook at the current moment; The hook identification area is updated according to the predicted spatial position of the hook.

5. The method according to any one of claims 1 to 4, wherein The step of establishing a load detection area based on the position of the hook includes: Determining the height between the hook and the load according to the position of the hook; The load detection area is constructed based on the height and the preset radius with the position of the hook as the center.

6. The method according to claim 5, wherein: Determining the height between the hook and the load according to the position of the hook includes: Acquire first load cell data corresponding to the load resting on the ground, and a first height between the hook and the boom of the crane; Acquire the second weighing sensor data corresponding to when the load is lifted by the hook and leaves the ground; When it is determined that the change in the second load cell data obtained for the consecutive threshold number of frames is less than the threshold difference, obtaining a second height between the current hook and the boom of the crane; The height between the hook and the load is determined by the first height, the second height, and the hook height.

7. The method according to any one of claims 1 to 4, wherein The safety identification of the crane based on the size of the load includes: constructing a three-dimensional simulation area between the hook and the auxiliary operation device of the crane according to the position of the hook to obtain a first area; constructing a three-dimensional simulation area centered on the load according to the height between the hook and the load and the size of the load to obtain a second area; Removing the point cloud data corresponding to the first area and the second area from the collected point cloud data to obtain point cloud data of the obstacle; Divide the point cloud data of the obstacle by using an octree to obtain multiple grids; A collision distance is calculated between each grid in the plurality of grids and the first area and the second area, respectively, to obtain a safety identification result of the crane.

8. A crane safety identification device, comprising: a determination module for acquiring point cloud data and the position of the crane's hook; A construction module, configured to construct a load detection area based on the position of the hook, wherein the load detection area is located at a preset position of the hook; a processing module, configured to cluster and segment the point cloud data within the suspended object detection area to obtain a plurality of independent external envelopes; The determining module is further configured to determine the size of the suspended object based on the positions of the vertices of each independent circumscribed envelope; The processing module is further configured to perform safety identification on the crane based on the size of the load.

9. An electronic device comprising at least one processor and at least one memory, wherein: The memory stores a computer program, and when the program is executed by the processor, the processor is caused to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program executable by an electronic device, wherein when the program is run on the electronic device, the electronic device executes the method according to any one of claims 1 to 7.

Images