A method, apparatus, storage medium, and electronic equipment for segmenting suspended object point clouds based on multi-frame point clouds.

Abstract

This application discloses a method, apparatus, storage medium, electronic device, and computer program product for point cloud segmentation of suspended objects based on multi-frame point clouds, belonging to the field of construction technology. The method includes: during the lifting of suspended objects by engineering machinery with lifting function, collecting a point cloud sequence of the area where the suspended object is located, the point cloud sequence including at least two frames of point clouds; determining multiple changing point cloud clusters in each frame of point cloud based on adjacent frames of point clouds in the point cloud sequence; determining multiple candidate point cloud clusters of suspended objects in the area from below the hook to the ground from each frame of point cloud; and segmenting the point cloud of the suspended object in each frame of point cloud based on the changing point cloud clusters and the candidate point cloud clusters of suspended objects. Therefore, it is not only applicable to point cloud segmentation of various types of suspended objects, but also has a wide range of applications, high accuracy in suspended object recognition, and good recognition effect. Moreover, it does not require the use of deep learning models or manual point cloud calibration.

Description

Technical Field

[0001] This application belongs to the field of building construction technology, and in particular relates to a method, device, storage medium, electronic device and computer program product for segmenting suspended object point clouds based on multi-frame point clouds. Background Technology

[0002] Tower cranes, also known as tower hoists, are important engineering facilities. Through lifting, luffing, and slewing movements, tower cranes can move and transport large materials vertically and horizontally, making them widely used in construction sites and similar environments. However, traditional tower cranes require an operator in the control room. Avoiding obstacles (such as stacked materials and buildings) and planning the crane's trajectory during operation relies heavily on the operator's skills and experience, which can easily lead to safety accidents. Therefore, intelligent tower cranes have emerged as a solution.

[0003] Intelligent tower cranes are intelligent tower crane equipment that integrates advanced electronic technology, automation control technology, sensor technology, and other engineering technologies, enabling unmanned operation and autonomous operation. For intelligent tower cranes, trajectory planning, collision detection, and obstacle avoidance are three crucial research areas. During the lifting process, the dimensions of the load are essential for safe and reliable trajectory planning, which is also a prerequisite for automatic collision detection and obstacle avoidance during lifting. However, due to the diverse range of loads lifted by tower cranes, current technologies struggle to accurately identify the load's dimensions, leading to difficulties or inaccuracies in load size calculations. Summary of the Invention

[0004] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a method, apparatus, storage medium, electronic device, and computer program product for segmenting suspended object point clouds based on multi-frame point clouds, which can accurately segment point clouds of various suspended objects with high object recognition accuracy.

[0005] Firstly, this application provides a method for segmenting point clouds of suspended objects based on multi-frame point clouds, including:

[0006] During the lifting process of engineering machinery with hoisting function, a point cloud sequence of the collection area where the hoisted object is located is acquired, and the point cloud sequence includes at least two frames of point cloud;

[0007] Based on the point cloud of adjacent frames in the point cloud sequence, determine multiple changed point cloud clusters in each frame point cloud;

[0008] From each frame of point cloud, identify multiple candidate point cloud clusters of suspended objects in the area from below the hook to the ground;

[0009] Based on the changed point cloud cluster and the candidate point cloud cluster of the suspended object, the point cloud of the suspended object in each frame of the point cloud is segmented.

[0010] In some embodiments, determining multiple changed point cloud clusters in each frame of the point cloud based on adjacent frame point clouds in the point cloud sequence includes:

[0011] All frame point clouds in the point cloud sequence are voxelized according to a unified standard.

[0012] Coarsely align the horizontal coordinates of any two adjacent voxelized point clouds to determine the point clouds that have changed in the corresponding two frame point clouds, and obtain the set of changed point clouds for each frame point cloud.

[0013] The point clouds in the changed point cloud set are filtered, and the point clouds in the filtered changed point cloud set are clustered to obtain multiple changed point cloud clusters that have changed in the corresponding frame point cloud.

[0014] In some embodiments, determining a plurality of candidate point cloud clusters of suspended objects in the region from below the hook to the ground from each frame of point cloud includes:

[0015] Determine the hook point cloud and the ground point cloud in each frame of the point cloud;

[0016] The target point cloud is obtained by determining the point cloud located below the hook point cloud in each frame of the point cloud based on the three-dimensional position coordinates of each point cloud.

[0017] The target point cloud in each frame of the point cloud is clustered to obtain multiple clusters;

[0018] The clusters that are not connected to the ground point cloud are selected as candidate point cloud clusters for the suspended object.

[0019] In some embodiments, segmenting the point cloud of the suspended object in each frame of the point cloud based on the changed point cloud cluster and the candidate point cloud cluster of the suspended object includes:

[0020] Determine the minimum distance between any of the changed point cloud clusters and any of the candidate point cloud clusters for the suspended object corresponding to each frame of point cloud;

[0021] Based on the minimum distance, at least one associated point cloud cluster combination is determined from the candidate point cloud cluster of the suspended object and the variable point cloud cluster, and each associated point cloud cluster combination includes one candidate point cloud cluster of the suspended object and at least one variable point cloud cluster.

[0022] The point cloud of the suspended object in the corresponding frame point cloud is segmented based on the associated point cloud cluster combination.

[0023] In some embodiments, determining at least one associated point cloud cluster combination from the candidate point cloud clusters of the suspended object and the variable point cloud clusters based on the minimum distance includes:

[0024] Determine whether the minimum distance is less than or equal to a first preset threshold;

[0025] If so, the corresponding candidate point cloud clusters of suspended objects and the change point cloud clusters are grouped into a single associated point cloud cluster combination, and different associated point cloud cluster combinations having the same candidate point cloud cluster of suspended objects are merged.

[0026] In some embodiments, segmenting the point cloud of the suspended object in the corresponding frame point cloud based on the associated point cloud cluster combination includes:

[0027] Count the total number of the first point cloud of the candidate point cloud cluster for the suspended object in each of the associated point cloud cluster combinations, and the total number of the second point cloud of the changed point cloud cluster;

[0028] The confidence level of the corresponding candidate point cloud clusters for suspended objects is calculated based on the total number of the first point cloud and the total number of the second point cloud.

[0029] All point clouds in the candidate point cloud clusters of the suspended object corresponding to the confidence scores greater than or equal to the second preset threshold are used as the point clouds of the suspended object in the corresponding frame point cloud.

[0030] Secondly, this application provides a device for segmenting suspended object point clouds based on multi-frame point clouds, comprising:

[0031] The acquisition unit is used to acquire a point cloud sequence of the acquisition area where the hoisted object is located during the hoisting process of engineering machinery with hoisting function; the point cloud sequence includes at least two frames of point cloud.

[0032] The first determining unit is used to determine multiple changing point cloud clusters in each frame of the point cloud based on the adjacent frame point clouds in the point cloud sequence.

[0033] The second determining unit is used to determine multiple candidate point cloud clusters of suspended objects in the area from below the hook to the ground from each frame of point cloud;

[0034] The identification unit is used to segment the point cloud of the suspended object in each frame of the point cloud based on the changed point cloud cluster and the candidate point cloud cluster of the suspended object.

[0035] In some embodiments, the first determining unit is specifically used for:

[0036] All frame point clouds in the point cloud sequence are voxelized according to a unified standard.

[0037] Coarsely align the horizontal coordinates of any two adjacent voxelized point clouds to determine the point clouds that have changed in the corresponding two frame point clouds, and obtain the set of changed point clouds for each frame point cloud.

[0038] The point clouds in the changed point cloud set are filtered, and the point clouds in the filtered changed point cloud set are clustered to obtain multiple changed point cloud clusters that have changed in the corresponding frame point cloud.

[0039] In some embodiments, the second determining unit is specifically used for:

[0040] Determine the hook point cloud and the ground point cloud in each frame of the point cloud;

[0041] The target point cloud is obtained by determining the point cloud located below the hook point cloud in each frame of the point cloud based on the three-dimensional position coordinates of each point cloud.

[0042] The target point cloud in each frame of the point cloud is clustered to obtain multiple clusters;

[0043] The clusters that are not connected to the ground point cloud are selected as candidate point cloud clusters for the suspended object.

[0044] In some embodiments, the identification unit is specifically used for:

[0045] Determine the minimum distance between any of the changed point cloud clusters and any of the candidate point cloud clusters for the suspended object corresponding to each frame of point cloud;

[0046] Based on the minimum distance, at least one associated point cloud cluster combination is determined from the candidate point cloud cluster of the suspended object and the variable point cloud cluster, and each associated point cloud cluster combination includes one candidate point cloud cluster of the suspended object and at least one variable point cloud cluster.

[0047] The point cloud of the suspended object in the corresponding frame point cloud is segmented based on the associated point cloud cluster combination.

[0048] In some embodiments, the identification unit is specifically used for:

[0049] Determine whether the minimum distance is less than or equal to a first preset threshold;

[0050] If so, the corresponding candidate point cloud clusters of suspended objects and the change point cloud clusters are grouped into a single associated point cloud cluster combination, and different associated point cloud cluster combinations having the same candidate point cloud cluster of suspended objects are merged.

[0051] In some embodiments, the identification unit is specifically used for:

[0052] Count the total number of the first point cloud of the candidate point cloud cluster for the suspended object in each of the associated point cloud cluster combinations, and the total number of the second point cloud of the changed point cloud cluster;

[0053] The confidence level of the corresponding candidate point cloud clusters for suspended objects is calculated based on the total number of the first point cloud and the total number of the second point cloud.

[0054] All point clouds in the candidate point cloud clusters of the suspended object corresponding to the confidence scores greater than or equal to the second preset threshold are used as the point clouds of the suspended object in the corresponding frame point cloud.

[0055] Thirdly, this application provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the above-described method for segmenting suspended object point clouds based on multi-frame point clouds.

[0056] Fourthly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described method for segmenting suspended object point clouds based on multi-frame point clouds.

[0057] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method for segmenting suspended object point clouds based on multi-frame point clouds.

[0058] The method, apparatus, storage medium, electronic device, and computer program product for segmenting suspended objects based on multi-frame point clouds provided in this application embodiment acquire a point cloud sequence corresponding to the collection area where the suspended object is located during the lifting of suspended objects by engineering machinery with lifting functions. The point cloud sequence includes at least two frames of point clouds. Based on the adjacent frames of point clouds in the point cloud sequence, multiple changing point cloud clusters are determined in each frame of point cloud. Multiple candidate point cloud clusters of suspended objects are determined from each frame of point cloud in the area from below the hook to the ground. Based on the changing point cloud clusters and the candidate point cloud clusters of suspended objects, the point cloud of the suspended object in each frame of point cloud is segmented. That is, the suspended object point cloud is segmented based on the characteristic that the position of the suspended object changes during lifting and the characteristic that it is located below the hook and does not contact the ground. This effectively improves the segmentation accuracy of the suspended object point cloud, and the suspended object recognition effect is good. It is not only applicable to the segmentation of various types of suspended object point clouds, but also has a wide range of applications. Moreover, it does not require the use of deep learning models or manual point cloud calibration. Attached Figure Description

[0059] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0060] Figure 1 This is a flowchart illustrating the point cloud segmentation method for suspended objects based on multi-frame point clouds provided in an embodiment of this application.

[0061] Figure 2This is a schematic diagram of the tower crane provided in the embodiments of this application;

[0062] Figure 3 This is another flowchart illustrating the method for segmenting suspended object point clouds based on multi-frame point clouds provided in this application embodiment;

[0063] Figure 4 This is a schematic diagram of the processing flow of the change point cloud cluster and the candidate point cloud cluster of the suspended object provided in the embodiments of this application;

[0064] Figure 5 This is a schematic diagram of the structure of the suspended object point cloud segmentation device based on multi-frame point clouds provided in the embodiments of this application;

[0065] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0066] Figure 7 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0067] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0068] During the lifting process of intelligent tower cranes, it is necessary to know the size information of the load to provide safe and reliable trajectory planning. This is also a prerequisite for automatic collision detection and obstacle avoidance during the lifting process. In existing technologies, the size of the load can be obtained through manual measurement. This method not only suffers from measurement errors but also increases labor costs. In other implementations, the size of the load can be measured based on a deep learning model. Specifically, a deep learning model for identifying load point clouds is trained using a large number of load point cloud samples and labeled data. The deep learning model then segments the load point cloud collected in the tower crane's working area. The size of the load is then calculated based on the segmented load point cloud. While this method can achieve automatic measurement of load size, the variety of loads makes it difficult to obtain comprehensive training point cloud samples when training the deep learning model. This results in insufficient feature learning for the deep learning model, making it unable to accurately segment various load point clouds. The model's applicability is limited, and the accuracy of load identification is low, leading to difficulties or inaccuracies in load size calculation.

[0069] To address at least one of the aforementioned technical problems, embodiments of this application provide a method, apparatus, storage medium, electronic device, and computer program product for segmenting suspended object point clouds based on multi-frame point clouds.

[0070] Please see Figure 1 , Figure 1 This is a flowchart illustrating the multi-frame point cloud-based object segmentation method provided in this application. This multi-frame point cloud-based object segmentation method is applied to an electronic device, which can be implemented as a user terminal or a server. The user terminal includes laptops, tablets, desktop computers, mobile devices (e.g., mobile phones, personal digital assistants, dedicated messaging devices), etc. Specifically, the multi-frame point cloud-based object segmentation method may include the following steps 101-104, wherein:

[0071] 101. During the lifting process of engineering machinery with hoisting function, acquire the point cloud sequence of the collection area where the hoisted object is located, and the point cloud sequence includes at least two frames of point cloud.

[0072] The engineering machinery with lifting capabilities includes, but is not limited to, cranes, tower cranes, and gantry cranes. The engineering machinery includes a hook for lifting the load and a steel cable suspending the hook, as well as a boom and a moving trolley mounted on the boom, with the steel cable fixed to the moving trolley. This application will use a tower crane as an example to introduce a method for segmenting the point cloud of a suspended object based on multi-frame point clouds. Please refer to... Figure 2 , Figure 2 This is a schematic diagram of the tower crane structure provided in this application embodiment. The tower crane 20 includes a base 21, a tower body 22 vertically fixed to the base 21, a boom 23 and a counterweight boom 24 disposed on the tower body 22, a hook 25, and a moving trolley 26. The moving trolley 26 is disposed on the boom 23 and is equipped with a motor that drives the moving trolley 26 to move left and right along the boom 23. The top of the steel cable 27 is suspended from the moving trolley 26, and the suspension point of the steel cable can be regarded as the center point of the top of the steel cable 27. The hook 25 is suspended from the bottom of the steel cable 27. When the moving trolley 26 moves left and right, the hook 25 moves left and right accordingly, and the position of the hook 25 can be adjusted at any time to facilitate the lifting of heavy objects and the transport of heavy objects to the required location. It should be noted that... Figure 2 Only one structure of the tower crane 20 is shown. The tower crane 20 can also have other structures, which are not limited here.

[0073] During the lifting process by a tower crane, point cloud acquisition devices such as LiDAR or binocular cameras can be used to continuously collect point cloud data of the area where the object is located. The collected point cloud data includes the geometric position and attribute data of the point cloud, such as color information and reflectivity information. The point cloud acquisition device can be mounted on a mobile trolley and move with it, or it can be fixedly installed around the tower crane; there are no restrictions on this.

[0074] In some embodiments, after each frame point cloud is acquired, the frame point cloud can be preprocessed. This preprocessing includes filtering processes such as noise reduction and outlier removal, as well as downsampling processes, etc., to improve the quality and accuracy of the point cloud data and facilitate subsequent point cloud analysis and processing.

[0075] 102. Based on the point clouds of adjacent frames in the point cloud sequence, determine the multiple changed point cloud clusters in each frame of the point cloud that have changed.

[0076] Among them, the point cloud in the changing point cloud cluster mainly refers to the point cloud of objects that have moved relative to the point cloud acquisition device, such as moving horizontally or vertically. When the point cloud acquisition device is installed on a mobile trolley, these objects may include the ground, surrounding buildings, swaying suspended objects, steel cables, etc. When the point cloud acquisition device is fixedly installed around the tower crane, these objects may include suspended objects, components on the tower crane (such as hooks, steel cables, etc.).

[0077] In some embodiments, see Figure 3 , Figure 3 This is another flowchart illustrating the method for segmenting suspended object point clouds based on multi-frame point clouds provided in this application embodiment. Step 102 above may specifically include steps 1021-1023, wherein:

[0078] 1021. Perform standardized voxelization on all frame point clouds in the point cloud sequence;

[0079] 1022. Perform coarse horizontal coordinate alignment on any two adjacent voxelized point clouds to determine the point clouds that have changed in the corresponding two frame point clouds, and obtain the set of changed point clouds for each frame point cloud.

[0080] 1023. Filter the point clouds in the set of changed point clouds, and cluster the point clouds in the filtered set of changed point clouds to obtain multiple clusters of changed point clouds that have changed in the corresponding frame point clouds.

[0081] In this approach, considering the differences in 3D position coordinates before and after a change in a point cloud, the point cloud frames in the sequence are first voxelized using the same reference coordinates. Then, adjacent frames are coarsely aligned in terms of horizontal coordinates to identify points whose 3D position coordinates have changed. All changed points in each frame are grouped into a set of changed point clouds. Next, outlier removal and other filtering processes are applied to the points in each set. Finally, point cloud clustering is performed to obtain multiple clusters of changed point clouds corresponding to each frame. Outliers generally refer to points that deviate from the majority of the data. Point cloud clustering divides the point cloud into different groups based on its spatial distribution and characteristics. Outlier removal methods include statistical methods and domain-based methods, while point cloud clustering methods include K-Means clustering, Euclidean clustering, and DBSCAN clustering, among others. No specific restrictions are imposed here.

[0082] 103. Identify multiple candidate point cloud clusters of suspended objects in the area from below the hook to the ground from each frame of point cloud.

[0083] In some embodiments, please continue to see Figure 3 Step 103 above may specifically include the following steps 1031-1033, wherein:

[0084] 1031. Determine the hook point cloud and the ground point cloud in each frame of the point cloud;

[0085] 1032. Based on the three-dimensional position coordinates of each point cloud, determine the point cloud located below the hook point cloud in each frame of the point cloud to obtain the target point cloud;

[0086] 1033. Cluster the target point cloud in each frame of point cloud to obtain multiple clusters, and select the clusters that are not connected to the ground point cloud as candidate point cloud clusters for the suspended object.

[0087] Since the shape features of the hook and the ground are relatively fixed, a semantic segmentation model for the point cloud can be trained. This trained model can then be used to identify and segment the point clouds of the hook and the ground. In other embodiments, the actual location and direction of the cable suspension point (cable suspension condition) can be determined first. The cable point cloud can then be segmented based on the cable suspension condition. Finally, the hook point cloud can be further segmented based on the actual cable point cloud and the cable suspension condition, thereby effectively improving the segmentation accuracy of the hook point cloud.

[0088] After obtaining the hook point cloud, the point cloud with the lowest height relative to the ground (e.g., the hook point cloud with the smallest z-axis coordinate value) can be found based on its three-dimensional position coordinates (e.g., XYZ axis coordinates). This yields the point cloud at the lowest point of the hook. Then, using the horizontal plane containing this point cloud as the boundary, the point cloud below the hook (i.e., the target point cloud) is cropped to remove the point clouds of the hook and objects above it, thus narrowing the point cloud recognition range for the suspended object. Next, for the target point cloud in each frame, point cloud clustering can be performed to group point clouds with the same spatial distribution and characteristics into a single cluster. These clusters represent the point clouds of objects below the hook, including not only the suspended object but also buildings and other objects. Considering that the suspended object is generally suspended in the air and does not contact the ground, clusters connected to the ground point cloud need to be removed to further narrow the point cloud recognition range for the suspended object.

[0089] 104. Based on the changed point cloud cluster and the candidate point cloud cluster of the suspended object, segment the point cloud of the suspended object in each frame of the point cloud.

[0090] In some embodiments, please continue to see Figure 3 Step 104 above may specifically include steps 1041-1043, wherein:

[0091] 1041. Determine the minimum distance between any changed point cloud cluster and any candidate point cloud cluster for a suspended object corresponding to each frame of point cloud.

[0092] For example, if the changed point cloud cluster corresponding to the point cloud in frame q is denoted as The candidate point cloud cluster for the suspended object corresponding to the point cloud in frame q is denoted as . If i represents the number of variable point cloud clusters, and j represents the number of candidate point cloud clusters for the suspended object, then please refer to [the relevant documentation / reference]. Figure 4 , Figure 4 This is a schematic diagram of the processing flow for variable point cloud clusters and candidate suspended object point cloud clusters provided in the embodiments of this application. It is necessary to calculate the minimum distance d between each variable point cloud cluster in i variable point cloud clusters and each candidate suspended object point cloud cluster in j candidate suspended object point cloud clusters. mn m∈[1,i], n∈[1,j].

[0093] 1042. Determine at least one associated point cloud cluster combination from the candidate point cloud cluster and the variable point cloud cluster based on the minimum distance, each associated point cloud cluster combination including a candidate point cloud cluster and at least one variable point cloud cluster.

[0094] In some embodiments, step 1042 may specifically include:

[0095] Determine whether the minimum distance is less than or equal to the first preset threshold;

[0096] If so, the corresponding candidate point cloud clusters and change point cloud clusters of suspended objects will be grouped into a single associated point cloud cluster combination, and different associated point cloud cluster combinations with the same candidate point cloud cluster of suspended objects will be merged.

[0097] Among them, since the change point cloud clusters correspond to moving objects (including suspended objects), and the suspended object candidate point cloud clusters correspond to moving objects (including suspended objects) located below the hook and not in contact with the ground, and non-moving objects, if the change point cloud clusters and suspended object candidate point cloud clusters are the same moving object, the distance between the point cloud clusters should be relatively small (ideally 0). Therefore, the minimum distance between the point cloud clusters should not be greater than the first preset threshold. The change point cloud clusters and suspended object candidate point cloud clusters with small distances can be found, thereby effectively filtering out the point cloud clusters of non-moving objects and narrowing the point cloud segmentation range of suspended objects.

[0098] Typically, if a single candidate point cloud cluster for a suspended object finds one or more variable point cloud clusters that are relatively close to it, for example, assuming a candidate point cloud cluster for a suspended object... and two variable point cloud clusters If the minimum distance between all points is less than or equal to the first preset threshold, then the candidate point cluster for the suspended object is considered to be in this cluster. If the corresponding object is located below the hook but not in contact with the ground, and has moved, this object is highly likely to be the suspended object. Therefore, this candidate suspended object cluster should be included in the cloud. and corresponding change point cloud clusters These are grouped into a single cluster (associated point cloud clusters) for further object point cloud segmentation. If a single object candidate point cloud cluster cannot find a variation point cloud cluster with a small distance to it, for example, assuming the object candidate point cloud cluster... And arbitrary change point cloud clusters If the minimum distance between all points is less than or equal to the first preset threshold, then the candidate point cluster for the suspended object is considered to be in this cluster. Although the corresponding object is located below the hook and not in contact with the ground, it has not moved. Therefore, this object should not be the suspended object, and the candidate object cluster is indicated. This is not a dotted cloud formation indicating a hanging object. Please continue reading. Figure 4 When based on the minimum distance d between point cloud clusters mn After combining all point cloud clusters that meet the distance condition, there may be only e candidate point cloud clusters of suspended objects with a change point cloud cluster with a small distance to them, e≤j, that is, the number of associated point cloud cluster combinations is e.

[0099] 1043. Based on the associated point cloud cluster combination, segment out the point cloud of the suspended object in the corresponding frame point cloud.

[0100] In some embodiments, step 1043 above may specifically include:

[0101] Count the total number of the first point cloud in the candidate point cloud cluster of the suspended object in each associated point cloud cluster combination, and the total number of the second point cloud in the changed point cloud cluster;

[0102] The confidence level of the corresponding candidate point cloud clusters for suspended objects is calculated based on the total number of the first point cloud and the total number of the second point cloud.

[0103] All point clouds in the candidate point cloud clusters of the suspended object corresponding to the confidence level that is greater than or equal to the second preset threshold are used as the point clouds of the suspended object in the corresponding frame point cloud.

[0104] In this context, the candidate point cloud clusters for suspended objects in the associated point cloud cluster combination only indicate point clouds of moving objects located below the hook and not in contact with the ground, not necessarily point clouds of the suspended objects themselves. Considering that the volume of the suspended object is generally relatively large in this area below the hook and not in contact with the ground, while the volume of other interfering objects is relatively small, the suspended object point cloud can be accurately and quickly extracted based on the proportion of point clouds in each candidate point cloud cluster. Specifically, the confidence level of a candidate point cloud cluster belonging to the suspended object is first determined based on the proportion of point clouds in each candidate point cloud cluster, and then the point clouds belonging to the suspended object are extracted based on this confidence level. The specific formula for calculating the confidence level *con* of the candidate point cloud clusters for suspended objects in each associated point cloud cluster combination is as follows:

[0105]

[0106] This represents the total number of point clouds in the k-th associated point cloud cluster combination corresponding to the point cloud of the q-th frame (i.e., the total number of the first point clouds mentioned above). It is the sum of the number of points in all the changed point cloud clusters in the kth associated point cloud cluster combination corresponding to the point cloud of the qth frame (i.e., the total number of the second point cloud mentioned above).

[0107] Then, ensure that all confidence levels con are not less than the second preset threshold T. con The point cloud in the candidate point cloud cluster of the suspended object is identified as the point cloud of the suspended object, and then the point cloud of the suspended object is segmented.

[0108] In other words, the segmentation of the suspended object point cloud in this application is not achieved by training a semantic segmentation model of the point cloud. It does not require a large number of suspended object point cloud samples or point cloud calibration. Instead, it relies on the characteristics of the changing position of the suspended object during hoisting and the characteristics of its specific suspension area to segment the suspended object point cloud. Specifically, it identifies and clusters the point clouds whose positions change in each frame based on the point clouds of adjacent frames. At the same time, it identifies and clusters the point clouds in the specific suspension area located below the hook and not in contact with the ground in each frame. Then, it combines these two types of point cloud clusters to automatically segment the suspended object point cloud. This not only achieves automated segmentation of various types of suspended object point clouds, but also saves the workload and operational errors of manual point cloud calibration, improving the speed and accuracy of point cloud segmentation.

[0109] As described above, the multi-frame point cloud segmentation method for suspended objects provided in this application obtains a point cloud sequence corresponding to the collection area where the suspended object is located during the lifting process of engineering machinery with lifting function. The point cloud sequence includes at least two frames of point clouds. Based on the adjacent frames of point clouds in the point cloud sequence, multiple changing point cloud clusters in each frame of point cloud are determined. Multiple candidate point cloud clusters of suspended objects are determined from each frame of point cloud in the area from below the hook to the ground. Based on the changing point cloud clusters and the candidate point cloud clusters of suspended objects, the point cloud of the suspended object in each frame of point cloud is segmented. That is, the suspended object point cloud is segmented by relying on the characteristic that the position of the suspended object changes during lifting and the characteristic that it is located below the hook and does not contact the ground in a specific suspension area. This effectively improves the segmentation accuracy of the suspended object point cloud, and the suspended object recognition effect is good. It is not only applicable to the segmentation of various types of suspended object point clouds, but also has a wide range of applications. Moreover, it does not require the use of deep learning models or manual point cloud calibration.

[0110] Based on the method described in the above embodiments, this application also provides a hanging object point cloud segmentation device based on multi-frame point clouds, used to perform the steps in the above-described hanging object point cloud segmentation method based on multi-frame point clouds. Please refer to... Figure 5 , Figure 5 This is a schematic diagram of the structure of the multi-frame point cloud-based object segmentation device 300 provided in this application embodiment. The multi-frame point cloud-based object segmentation device 300 is applied in an electronic device, which can be implemented as a user terminal or a server. The user terminal includes laptops, tablets, desktop computers, mobile devices (e.g., mobile phones, personal digital assistants, dedicated messaging devices), etc. Specifically, the multi-frame point cloud-based object segmentation device 300 includes an acquisition unit 301, a first determination unit 302, a second determination unit 303, and an identification unit 304, wherein:

[0111] The acquisition unit 301 is used to acquire a point cloud sequence of the acquisition area where the hoisted object is located during the hoisting process of the engineering machinery with hoisting function. The point cloud sequence includes at least two frames of point cloud.

[0112] The first determining unit 302 is used to determine multiple changed point cloud clusters in each frame of point cloud based on the adjacent frame point clouds in the point cloud sequence;

[0113] The second determining unit 303 is used to determine multiple candidate point cloud clusters of suspended objects in the area from below the hook to the ground from each frame of point cloud;

[0114] The identification unit 304 is used to segment the point cloud of the suspended object in each frame of the point cloud based on the changing point cloud cluster and the candidate point cloud cluster of the suspended object.

[0115] In some embodiments, the first determining unit 302 is specifically used for:

[0116] All frame point clouds in the point cloud sequence are voxelized according to a unified standard.

[0117] Coarsely align the horizontal coordinates of any two adjacent voxelized point clouds to determine the point clouds that have changed in the corresponding two frame point clouds, and obtain the set of changed point clouds for each frame point cloud.

[0118] The point clouds in the changed point cloud set are filtered, and the point clouds in the filtered changed point cloud set are clustered to obtain multiple changed point cloud clusters that have changed in the corresponding frame point cloud.

[0119] In some embodiments, the second determining unit 303 is specifically used for:

[0120] Determine the hook point cloud corresponding to the hook and the ground point cloud corresponding to the ground in each frame of the point cloud;

[0121] The target point cloud is obtained by determining the point cloud located below the hook point cloud in each frame of the point cloud based on the three-dimensional position coordinates of each point cloud.

[0122] The target point cloud in each frame of the point cloud is clustered to obtain multiple clusters;

[0123] The clusters that are not connected to the ground point cloud are selected as candidate point cloud clusters for the suspended object.

[0124] In some embodiments, the identification unit 304 is specifically used for:

[0125] Determine the minimum distance between any change point cloud cluster and any candidate point cloud cluster for the suspended object corresponding to each frame of point cloud;

[0126] Based on the minimum distance, at least one associated point cloud cluster combination is determined from the candidate point cloud cluster of the suspended object and the variable point cloud cluster, each associated point cloud cluster combination including one candidate point cloud cluster of the suspended object and at least one variable point cloud cluster;

[0127] The point cloud corresponding to the suspended object in the corresponding frame point cloud is determined based on the combination of the associated point cloud clusters.

[0128] In some embodiments, the identification unit 304 is specifically used for:

[0129] Determine whether the minimum distance is less than or equal to the first preset threshold;

[0130] If so, the corresponding candidate point cloud clusters and change point cloud clusters of suspended objects will be grouped into a single associated point cloud cluster combination, and different associated point cloud cluster combinations with the same candidate point cloud cluster of suspended objects will be merged.

[0131] In some embodiments, the identification unit 304 is specifically used for:

[0132] Count the total number of the first point cloud in the candidate point cloud cluster of the suspended object in each associated point cloud cluster combination, and the total number of the second point cloud in the changed point cloud cluster;

[0133] The confidence level of the corresponding candidate point cloud clusters for suspended objects is calculated based on the total number of the first point cloud and the total number of the second point cloud.

[0134] All point clouds in the candidate point cloud clusters of the suspended object whose confidence level is greater than or equal to the second preset threshold are used as the point cloud of the suspended object in the corresponding frame point cloud.

[0135] It should be noted that the specific details of each module unit in the above-mentioned multi-frame point cloud-based suspended object point cloud segmentation device 300 have been described in detail in the embodiments of the above-mentioned multi-frame point cloud-based suspended object point cloud segmentation method, and will not be repeated here.

[0136] In some embodiments, the multi-frame point cloud-based object segmentation device in this application can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal device. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application does not specifically limit the device.

[0137] In some embodiments, such as Figure 6 As shown, this application embodiment also provides an electronic device 400, including a processor 401, a memory 402, and a computer program stored in the memory 402 and executable on the processor 401. When the program is executed by the processor 401, it implements the various processes of the above-described embodiment of the suspended object point cloud segmentation method based on multi-frame point clouds and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0138] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0139] Figure 7 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0140] The electronic device 500 includes, but is not limited to, components such as: radio frequency unit 501, network module 502, audio output unit 503, input unit 504, sensor 505, display unit 506, user input unit 507, interface unit 508, memory 509, and processor 510.

[0141] Those skilled in the art will understand that the electronic device 500 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 510 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 7 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0142] It should be understood that, in this embodiment, the input unit 504 may include a graphics processing unit (GPU) 5041 and a microphone 5042. The GPU 5041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 506 may include a display panel 5061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 507 includes at least one of a touch panel 5071 and other input devices 5072. The touch panel 5071 is also called a touch screen. The touch panel 5071 may include a touch detection device and a touch controller. Other input devices 5072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0143] The memory 509 can be used to store software programs and various data. The memory 509 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 509 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 509 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0144] Processor 510 may include one or more processing units; processor 510 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 510.

[0145] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described embodiment of the suspended object point cloud segmentation method based on multi-frame point clouds and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0146] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0147] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method for segmenting suspended object point clouds based on multi-frame point clouds.

[0148] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0149] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0150] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0151] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0152] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0153] In the description of this application, "multiple" means two or more.

[0154] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0155] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for segmenting suspended object point clouds based on multi-frame point clouds, characterized in that, include: During the lifting process of engineering machinery with hoisting function, a point cloud sequence of the collection area where the hoisted object is located is acquired, and the point cloud sequence includes at least two frames of point cloud; Based on the point cloud of adjacent frames in the point cloud sequence, determine multiple changed point cloud clusters in each frame point cloud; From each frame of point cloud, identify multiple candidate point cloud clusters of suspended objects in the area from below the hook to the ground; Based on the changed point cloud cluster and the candidate point cloud cluster of the suspended object, the point cloud of the suspended object in each frame of the point cloud is segmented; The step of segmenting the point cloud of the suspended object in each frame of the point cloud based on the changed point cloud cluster and the candidate point cloud cluster of the suspended object includes: Determine the minimum distance between any of the changed point cloud clusters and any of the candidate point cloud clusters for the suspended object corresponding to each frame of point cloud; Based on the minimum distance, at least one associated point cloud cluster combination is determined from the candidate point cloud cluster of the suspended object and the variable point cloud cluster, and each associated point cloud cluster combination includes one candidate point cloud cluster of the suspended object and at least one variable point cloud cluster. The point cloud of the suspended object in the corresponding frame point cloud is segmented according to the associated point cloud cluster combination; The step of determining at least one associated point cloud cluster combination from the candidate point cloud clusters of the suspended object and the changed point cloud clusters based on the minimum distance includes: Determine whether the minimum distance is less than or equal to a first preset threshold; If so, the corresponding candidate point cloud clusters of suspended objects and the change point cloud clusters are grouped into a group of associated point cloud clusters, and different groups of associated point cloud clusters that have the same candidate point cloud cluster of suspended objects are merged. The step of segmenting the point cloud of the suspended object in the corresponding frame point cloud based on the associated point cloud clusters includes: Count the total number of the first point cloud of the candidate point cloud cluster for the suspended object in each of the associated point cloud cluster combinations, and the total number of the second point cloud of the changed point cloud cluster; The confidence level of the corresponding candidate point cloud clusters for suspended objects is calculated based on the total number of the first point cloud and the total number of the second point cloud. All point clouds in the candidate point cloud clusters of the suspended object corresponding to the confidence scores greater than or equal to the second preset threshold are used as the point clouds of the suspended object in the corresponding frame point cloud.

2. The method for segmenting suspended object point clouds based on multi-frame point clouds according to claim 1, characterized in that, The step of determining multiple changed point cloud clusters in each frame of the point cloud based on adjacent frame point clouds in the point cloud sequence includes: All frame point clouds in the point cloud sequence are voxelized according to a unified standard. Coarsely align the horizontal coordinates of any two adjacent voxelized point clouds to determine the point clouds that have changed in the corresponding two frame point clouds, and obtain the set of changed point clouds for each frame point cloud. The point clouds in the changed point cloud set are filtered, and the point clouds in the filtered changed point cloud set are clustered to obtain multiple changed point cloud clusters that have changed in the corresponding frame point cloud.

3. The method for segmenting suspended object point clouds based on multi-frame point clouds according to claim 1, characterized in that, The determination of multiple candidate point cloud clusters of suspended objects from each frame of point cloud in the area below the hook to the ground includes: Determine the hook point cloud and the ground point cloud in each frame of the point cloud; The target point cloud is obtained by determining the point cloud located below the hook point cloud in each frame of the point cloud based on the three-dimensional position coordinates of each point cloud. The target point cloud in each frame of the point cloud is clustered to obtain multiple clusters; The clusters that are not connected to the ground point cloud are selected as candidate point cloud clusters for the suspended object.

4. A device for segmenting suspended object point clouds based on multi-frame point clouds, characterized in that, The suspended object point cloud segmentation device is used to execute the suspended object point cloud segmentation method based on multi-frame point clouds as described in any one of claims 1-3, and the suspended object point cloud segmentation device comprises: The acquisition unit is used to acquire a point cloud sequence of the acquisition area where the hoisted object is located during the hoisting process of engineering machinery with hoisting function; the point cloud sequence includes at least two frames of point cloud. The first determining unit is used to determine multiple changing point cloud clusters in each frame of the point cloud based on the adjacent frame point clouds in the point cloud sequence. The second determining unit is used to determine multiple candidate point cloud clusters of suspended objects in the area from below the hook to the ground from each frame of point cloud; The identification unit is used to segment the point cloud of the suspended object in each frame of the point cloud based on the changed point cloud cluster and the candidate point cloud cluster of the suspended object.

5. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the method for segmenting suspended object point clouds based on multi-frame point clouds as described in any one of claims 1-3.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method for segmenting suspended object point clouds based on multi-frame point clouds as described in any one of claims 1-3.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the method for segmenting suspended object point clouds based on multi-frame point clouds as described in any one of claims 1-3.

Images