Hook point cloud segmentation method and device, storage medium and electronic equipment

Abstract

The application discloses a hook point cloud segmentation method and device, a storage medium, an electronic device and a computer program product, and belongs to the technical field of building construction. The method comprises the following steps: according to a steel cable suspension direction and a steel cable point cloud set, a hook candidate point cloud set is screened out from frame point clouds from a steel cable suspension point position to the ground; according to the steel cable suspension point position, the steel cable suspension direction and the hook candidate point cloud set, the height coordinate interval of the hook is determined; and according to the height coordinate interval, all point clouds of the hook are segmented from the hook candidate point cloud set, that is, the hook point cloud is segmented according to the actual steel cable point cloud and the steel cable suspension condition, so that the hook point cloud can be accurately segmented even in the case that the steel cable shakes, the point cloud is unevenly dense, the hook deformation form changes and the like, and the hook point cloud segmentation effect is good.

Description

Technical Field

[0001] This application belongs to the field of building construction technology, and in particular relates to a hook point cloud segmentation method, device, storage medium, electronic device and computer program product. Background Technology

[0002] Tower cranes, also known as tower hoists, are important engineering facilities. Through lifting, luffing, and slewing movements, tower cranes can move and transport large materials vertically and horizontally, making them widely used in construction sites and similar environments. However, traditional tower cranes require an operator in the control room. Avoiding obstacles (such as stacked materials and buildings) and planning the crane's trajectory during operation relies heavily on the operator's skills and experience, which can easily lead to safety accidents. Therefore, intelligent tower cranes have emerged as a solution.

[0003] Intelligent tower cranes are intelligent tower crane equipment that integrates advanced electronic technology, automation control technology, sensor technology, and other engineering technologies, enabling unmanned operation and autonomous operation. For intelligent tower cranes, trajectory planning, collision detection, and obstacle avoidance are three crucial research areas. During the lifting process, the position and dimensions of the hook are essential for safe and reliable trajectory planning, which is also a prerequisite for automatic collision detection and obstacle avoidance during lifting. Current technologies struggle to accurately segment the hook point cloud, resulting in poor point cloud segmentation performance. Summary of the Invention

[0004] This application aims to at least solve one of the technical problems existing in the prior art. To this end, this application proposes a hook point cloud segmentation method, apparatus, storage medium, electronic device, and computer program product, which can achieve accurate segmentation of hook point clouds with good segmentation effect.

[0005] Firstly, this application provides a method for segmenting a hook point cloud, including:

[0006] During the lifting and hoisting process of engineering machinery with hoisting function, a candidate point cloud set for the hook is selected from the frame point cloud from the location of the steel cable suspension point to the ground, based on the location of the steel cable suspension point, the direction of the steel cable suspension, and the steel cable point cloud set.

[0007] The height coordinate range of the hook is determined based on the suspension direction of the steel cable and the candidate point cloud set of the hook.

[0008] Based on the height coordinate range, all point clouds of the hook are segmented from the candidate point cloud set of the hook.

[0009] In some embodiments, the step of determining the cable point cloud set includes:

[0010] Establish a tree-shaped data structure corresponding to the frame point cloud from the location of the steel cable suspension point to the ground;

[0011] Select the cable seed point from the tree data structure;

[0012] Based on the seed points of the steel cables, a set of steel cable point clouds is determined from the tree data structure using the region growing method.

[0013] In some embodiments, determining the height coordinate range of the hook based on the cable suspension direction and the candidate point cloud set of the hook includes:

[0014] Based on the suspension direction of the steel cable and the candidate point cloud set of the hook, multiple coordinate intervals are divided on the two-dimensional plane, and the height coordinate interval of the hook is determined according to the maximum distance of each coordinate interval, the actual length value of the top surface of the hook, and the actual length value of the bottom surface of the hook.

[0015] In some embodiments, the step of dividing a two-dimensional plane into multiple coordinate intervals based on the cable suspension direction and the candidate point cloud set of the hook, and determining the height coordinate interval of the hook based on the maximum distance of each coordinate interval, the actual length value of the top surface of the hook, and the actual length value of the bottom surface of the hook, includes:

[0016] Based on the suspension direction of the steel cable, each point cloud in the candidate point cloud set of the hook is converted into a planar coordinate point in a two-dimensional plane to obtain a set of planar coordinate points. The coordinate system of the two-dimensional plane includes a first coordinate axis and a second coordinate axis.

[0017] The two-dimensional position coordinates corresponding to the set of planar coordinate points are divided into multiple coordinate intervals along the direction of the first coordinate axis, and the maximum distance of the interval is determined based on the planar coordinate points in the coordinate interval.

[0018] Based on the maximum distance of each coordinate interval, the actual length of the top surface of the hook, and the actual length of the bottom surface of the hook, determine the top boundary position coordinates and bottom boundary position coordinates of the hook on the two-dimensional plane;

[0019] The height coordinate range of the hook is determined based on the coordinates of the top boundary position and the coordinates of the bottom boundary position.

[0020] In some embodiments, determining the top boundary position coordinates and bottom boundary position coordinates of the hook on the two-dimensional plane based on the maximum distance of each coordinate interval, the actual length value of the top surface of the hook, and the actual length value of the bottom surface of the hook includes:

[0021] The first target coordinate interval is determined from the coordinate interval based on the actual length value of the top surface of the hook, and the second target coordinate interval is determined from the coordinate interval based on the actual length value of the bottom surface of the hook;

[0022] The coordinates of the end point in the first target coordinate interval are used as the top boundary position coordinates of the hook on the two-dimensional plane, and the coordinates of the end point in the second target interval are used as the bottom boundary position coordinates of the hook on the two-dimensional plane.

[0023] In some embodiments, determining the maximum distance of the interval based on the planar coordinate points in the coordinate interval includes:

[0024] Clustering the planar coordinate points in each coordinate interval yields multiple clusters;

[0025] Calculate the maximum distance of each cluster on the second coordinate axis, and select the maximum distance with the largest value as the maximum distance of the corresponding coordinate interval.

[0026] In some embodiments, the step of determining the cable suspension direction includes:

[0027] Determine the minimum bounding box of the cable point cloud set;

[0028] The first straight line is obtained by determining the point cloud that is furthest from the suspension point of the steel cable in the point cloud set and the suspension point of the steel cable;

[0029] Determine the straight lines in the three directions of the length, width, and height of the minimum bounding box to obtain three second straight lines;

[0030] The direction of the second line, which has the smallest angle with the first line, is selected as the direction of the cable suspension.

[0031] In some embodiments, the step of filtering a candidate point cloud set for the hook from the frame point cloud from the cable suspension point location to the ground based on the cable suspension point location, cable suspension direction, and cable point cloud set includes:

[0032] Fit the plane equation corresponding to the point cloud set of the steel cable to obtain the steel cable plane;

[0033] Determine a third straight line passing through the location of the cable suspension point and the direction of the cable suspension;

[0034] Based on the maximum length and maximum width of the hook on the horizontal plane, a set of candidate hook point clouds is selected from the frame point cloud from the location of the cable suspension point to the ground.

[0035] Secondly, this application provides a hook point cloud segmentation device, comprising:

[0036] The filtering unit is used to filter out a candidate hook point cloud set from the frame point cloud from the location of the cable suspension point to the ground during the lifting of the load by the engineering machinery with lifting function, based on the location of the cable suspension point, the direction of the cable suspension, and the cable point cloud set.

[0037] The determining unit is used to determine the height coordinate range of the hook based on the suspension direction of the steel cable and the set of candidate points of the hook;

[0038] A segmentation unit is used to segment all point clouds of the hook from the candidate point cloud set of the hook according to the height coordinate interval.

[0039] In some embodiments, the filtering unit is further configured to:

[0040] Establish a tree-shaped data structure corresponding to the frame point cloud from the location of the steel cable suspension point to the ground;

[0041] Select the cable seed point from the tree data structure;

[0042] Based on the seed points of the steel cables, a set of steel cable point clouds is determined from the tree data structure using the region growing method.

[0043] In some embodiments, the determining unit is specifically used for:

[0044] Based on the suspension direction of the steel cable and the candidate point cloud set of the hook, multiple coordinate intervals are divided on the two-dimensional plane, and the height coordinate interval of the hook is determined according to the maximum distance of each coordinate interval, the actual length value of the top surface of the hook, and the actual length value of the bottom surface of the hook.

[0045] In some embodiments, the determining unit is specifically used for:

[0046] Based on the suspension direction of the steel cable, each point cloud in the candidate point cloud set of the hook is converted into a planar coordinate point in a two-dimensional plane to obtain a set of planar coordinate points. The coordinate system of the two-dimensional plane includes a first coordinate axis and a second coordinate axis.

[0047] The two-dimensional position coordinates corresponding to the set of planar coordinate points are divided into multiple coordinate intervals along the direction of the first coordinate axis, and the maximum distance of the interval is determined based on the planar coordinate points in the coordinate interval.

[0048] Based on the maximum distance of each coordinate interval, the actual length of the top surface of the hook, and the actual length of the bottom surface of the hook, determine the top boundary position coordinates and bottom boundary position coordinates of the hook on the two-dimensional plane;

[0049] The height coordinate range of the hook is determined based on the coordinates of the top boundary position and the coordinates of the bottom boundary position.

[0050] In some embodiments, the determining unit is specifically used for:

[0051] The first target coordinate interval is determined from the coordinate interval based on the actual length value of the top surface of the hook, and the second target coordinate interval is determined from the coordinate interval based on the actual length value of the bottom surface of the hook;

[0052] The coordinates of the end point in the first target coordinate interval are used as the top boundary position coordinates of the hook on the two-dimensional plane, and the coordinates of the end point in the second target interval are used as the bottom boundary position coordinates of the hook on the two-dimensional plane.

[0053] In some embodiments, the determining unit is specifically used for:

[0054] Clustering the planar coordinate points in each coordinate interval yields multiple clusters;

[0055] Calculate the maximum distance of each cluster on the second coordinate axis, and select the maximum distance with the largest value as the maximum distance of the corresponding coordinate interval.

[0056] In some embodiments, the filtering unit is further configured to:

[0057] Determine the minimum bounding box of the cable point cloud set;

[0058] The first straight line is obtained by determining the point cloud that is furthest from the suspension point of the steel cable in the point cloud set and the suspension point of the steel cable;

[0059] Determine the straight lines in the three directions of the length, width, and height of the minimum bounding box to obtain three second straight lines;

[0060] The direction of the second line, which has the smallest angle with the first line, is selected as the direction of the cable suspension.

[0061] In some embodiments, the filtering unit is further configured to:

[0062] Fit the plane equation corresponding to the point cloud set of the steel cable to obtain the steel cable plane;

[0063] Determine a third straight line passing through the location of the cable suspension point and the direction of the cable suspension;

[0064] Based on the maximum length and maximum width of the hook on the horizontal plane, a set of candidate hook point clouds is selected from the frame point cloud from the location of the cable suspension point to the ground.

[0065] Thirdly, this application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the hook point cloud segmentation method described in any of the above claims.

[0066] Fourthly, this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the hook point cloud segmentation method described in any of the preceding claims.

[0067] Fifthly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the hook point cloud segmentation method described in any of the preceding claims.

[0068] The hook point cloud segmentation method, device, storage medium, electronic device, and computer program product provided in this application embodiment, during the lifting of loads by engineering machinery with lifting functions, selects a candidate hook point cloud set from the frame point cloud from the cable suspension point position to the ground based on the cable suspension point position, cable suspension direction, and cable point cloud set; determines the height coordinate range of the hook based on the cable suspension direction and the candidate hook point cloud set; and segments all hook point clouds from the candidate hook point cloud set based on the height coordinate range. That is, the hook point cloud is segmented based on the actual cable point cloud and cable suspension condition, so that even when the cable sways, the point cloud density is uneven, or the hook's appearance is deformed, the hook point cloud can still be accurately segmented, resulting in good hook point cloud segmentation effect. Attached Figure Description

[0069] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0070] Figure 1 This is a flowchart illustrating the hook point cloud segmentation method provided in an embodiment of this application;

[0071] Figure 2 This is a schematic diagram of the tower crane provided in the embodiments of this application;

[0072] Figure 3 This is another flowchart illustrating the hook point cloud segmentation method provided in this application embodiment;

[0073] Figure 4 This is a schematic diagram of the structure of the hook point cloud segmentation device provided in the embodiments of this application;

[0074] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0075] Figure 6This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0076] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0077] During the lifting process of intelligent tower cranes, the position and size information of the hook are required to provide safe and reliable trajectory planning. This is also a necessary condition for automatic collision detection and obstacle avoidance during the lifting process. Research on hook point cloud segmentation faces challenges such as uneven point cloud density, ghosting caused by cable swaying, and deformation of the hook's appearance, all of which significantly increase the difficulty of hook point cloud segmentation.

[0078] Existing technologies employ various methods, such as using mechanical structures and PLCs to determine the expected position of the hook. However, factors like inertia, wind speed, uneven weight distribution, and design flaws in the suspension structure can cause the hook to sway unpredictably, resulting in a discrepancy between the actual hook position and the position calculated by the mechanical structure. This discrepancy is particularly pronounced when point clouds are acquired using uncalibrated equipment. Other methods train a deep learning model to identify hook point clouds using a large number of hook point cloud samples and labeled data. This model then segments the hook point cloud from the tower crane's working area. However, due to inherent variations in point cloud density and hook distortion, obtaining sufficiently diverse point cloud samples for model training is challenging, leading to poor hook point cloud segmentation results.

[0079] To address at least one of the aforementioned technical problems, embodiments of this application provide a hook point cloud segmentation method, apparatus, storage medium, electronic device, and computer program product.

[0080] Please see Figure 1 , Figure 1 This is a flowchart illustrating the hook point cloud segmentation method provided in this application embodiment. The hook point cloud segmentation method is applied to an electronic device, which can be implemented as a user terminal or a server. The user terminal includes laptops, tablets, desktop computers, mobile devices (e.g., mobile phones, personal digital assistants, dedicated messaging devices), etc. Specifically, the hook point cloud segmentation method may include the following steps 101-103, wherein:

[0081] 101. During the lifting and transporting of objects by engineering machinery with hoisting function, a candidate point cloud set for the hook is selected from the frame point cloud from the location of the cable suspension point to the ground based on the cable suspension point location, cable suspension direction, and cable point cloud set.

[0082] The engineering machinery with lifting capabilities includes, but is not limited to, hoisting machinery, tower cranes, and gantry cranes. The engineering machinery includes a hook for lifting the load and a steel cable suspending the hook, as well as a boom and a moving trolley mounted on the boom, with the steel cable fixed to the moving trolley. This application will use a tower crane as an example to introduce a hook point cloud segmentation method. Please see [link to relevant documentation]. Figure 2 , Figure 2 This is a schematic diagram of the tower crane structure provided in this application embodiment. The tower crane 20 includes a base 21, a tower body 22 vertically fixed to the base 21, a boom 23 and a counterweight boom 24 disposed on the tower body 22, a hook 25, and a moving trolley 26. The moving trolley 26 is disposed on the boom 23 and is equipped with a motor that drives the moving trolley 26 to move left and right along the boom 23. The top of the steel cable 27 is suspended from the moving trolley 26, and the suspension point of the steel cable can be regarded as the center point of the top of the steel cable 27. The hook 25 is suspended from the bottom of the steel cable 27. When the moving trolley 26 moves left and right, the hook 25 moves left and right accordingly, and the position of the hook 25 can be adjusted at any time to facilitate the lifting of heavy objects and the transport of heavy objects to the required location. It should be noted that... Figure 2 Only one structure of the tower crane 20 is shown. The tower crane 20 can also have other structures, which are not limited here.

[0083] During the lifting process by a tower crane, point cloud acquisition devices such as LiDAR or binocular cameras can be used to continuously collect point cloud data of the area where the object is located. The collected point cloud data includes the geometric position and attribute data of the point cloud, such as color information and reflectivity information. The point cloud acquisition device can be mounted on a mobile trolley and move with it, or it can be fixedly installed around the tower crane; there are no restrictions on this.

[0084] In some embodiments, after each frame point cloud is acquired, the frame point cloud can be preprocessed. This preprocessing includes filtering processes such as noise reduction and outlier removal, as well as downsampling processes, etc., to improve the quality and accuracy of the point cloud data and facilitate subsequent point cloud analysis and processing.

[0085] In some embodiments, the frame point cloud from the cable suspension point to the ground refers to all point clouds in the frame point cloud whose height relative to the ground does not exceed the cable suspension point position (e.g., denoted as point Q). That is, point clouds in the frame point cloud whose z-axis coordinate value in the three-dimensional position coordinates is less than or equal to the z-axis coordinate value of point Q. This filters out the point clouds of the boom and objects above it, leaving only those with a height not greater than Z. q The point cloud is obtained, thereby removing the point cloud of the remaining structure above the steel cable, and reducing the segmentation range of the point cloud of the suspended object and the point cloud of the steel cable.

[0086] In some embodiments, the step of determining the location of the cable suspension point specifically includes:

[0087] Obtain the amplitude value of the moving trolley, the slewing angle of the boom, and the minimum height of the boom relative to the ground;

[0088] The center point of the moving trolley is determined based on the amplitude value, the rotation angle, and the minimum height.

[0089] The location of the steel cable suspension point is determined based on the center point location and the preset calibration value.

[0090] The amplitude variation of the moving trolley is the slewing radius or working radius of the tower crane. At different data collection times, either the slewing angle of the boom or the minimum height of the boom relative to the ground may change, thus affecting the position (x) of the cable suspension point (Q). q y q , z q The value of ) changes and needs to be calculated in real time. The specific calculation formula is as follows:

[0091]

[0092] Among them, (x m y m , z m (x0, y0, z0) represents the center point of the moving trolley, R represents the amplitude of the moving trolley, α represents the slewing angle of the boom, H represents the bottom height of the boom (i.e., the lowest height relative to the ground), and (x0, y0, z0) represents the calibration value from the suspension point Q to the center point of the moving trolley (i.e., the preset calibration value mentioned above).

[0093] In this embodiment of the application, when determining the position of the cable suspension point, the real-time movement of the moving trolley and the boom is determined, and the position of the cable suspension point is calculated based on their real-time movement. This enables accurate estimation of the position of the cable suspension point, which is beneficial to improving the accuracy of the segmentation of the cable point cloud and the hook point cloud when performing segmentation of the cable point cloud and the hook point cloud based on the position of the cable suspension point.

[0094] In some embodiments, the step of determining the cable point cloud set specifically includes:

[0095] Establish a tree-structured data structure corresponding to the frame point cloud from the location of the steel cable suspension point to the ground;

[0096] Select the cable seed point from this tree-shaped data structure;

[0097] Based on the seed point of the steel cable, the set of steel cable point clouds is determined from the tree data structure by the region growing method.

[0098] The basic idea of ​​the region growing method is to merge points with similar properties together. For each region, a seed point is first designated as the starting point for growth. Then, the points in the surrounding neighborhood of the seed point are compared with the seed point, and points with similar properties are merged to continue growing outwards until no more points meeting the conditions are included. This completes the growth of a region. In this application, it is as follows: First, let R be the frame point cloud from the location of the cable suspension point to the ground. e Establish R e The kd-tree (a tree-like data structure used to store and retrieve points in K-dimensional space) is used to search for a point (e.g., point A) that is closest to the cable suspension point (Q point position) and does not exceed a first preset distance d1 (default empirical value). This point is then selected as a seed point on the cable. Based on the cable seed point A, it is determined whether the point cloud in the nearby search range meets the point cloud growth criteria. Specifically, with the cable seed point A as the center, the point cloud in the search range (the coverage area of ​​the preset search radius r) near point A in the kd-tree is first found as the target point cloud set {S}. Then, it is determined whether {S} meets the point cloud growth criteria. If it does, point cloud growth is performed, and {S} is added to the cable point cloud set {T}. The search continues in the next range until all point clouds in the kd-tree have been searched. The point cloud growth process based on the cable seed point ends, and the next cable seed point is selected. If the condition is not met, then search in the kd-tree for the next point closest to point Q and not exceeding d1 (e.g., point B) as the next cable seed point. Continue repeating the point cloud growth operation above until no cable seed point within d1 is found, at which point the entire point cloud growth ends. Using the region growing method, all point clouds of cables in the kd-tree can be found.

[0099] The formula for point cloud growth standards can be summarized as follows:

[0100]

[0101] The normal vector of the plane equation fitted to the target point cloud set {S} is denoted as... Let C be the center point cloud of {S}, and let the direction formed by point A and the center point cloud C be denoted as . α is and The angle between them, k is the total number of point clouds in {S}. thre_α and thre_k are thresholds; only when α and k simultaneously meet the above conditions is the point cloud growth standard deemed met, and growth proceeds; otherwise, growth stops.

[0102] In this embodiment, when segmenting the point cloud of the steel cable, the segmentation range is initially narrowed by first filtering out the frame point cloud from the steel cable suspension point to the ground. Then, the steel cable seed points are selected based on the steel cable suspension point position, and the steel cable point cloud is segmented based on the region growing method. Thus, even if the point cloud itself has uneven density or the steel cable sways, the segmentation of the steel cable point cloud can be achieved quickly and effectively with high segmentation accuracy. This is beneficial for improving the segmentation accuracy of the hook point cloud when segmenting the hook point cloud based on the steel cable point cloud.

[0103] In some embodiments, the step of determining the suspension direction of the steel cable specifically includes:

[0104] Determine the minimum bounding box of the cable point cloud set;

[0105] The first straight line is obtained by determining the point cloud that is farthest from the suspension point of the steel cable in the point cloud set and the straight line that passes through the suspension point of the steel cable.

[0106] Determine the straight lines in the three directions of the length, width, and height of the minimum bounding box to obtain three second straight lines;

[0107] The direction of the second line, which has the smallest angle with the first line, is selected as the direction of the cable suspension.

[0108] That is, if the point cloud set of the steel cable is denoted as {T}, and the position of the suspension point of the steel cable is denoted as Q, then we can first determine the minimum bounding box of {T}, and the straight lines {L} in the length, width, and height directions of the bounding box are defined by the following: j Let j be the index of the three lines. Find the point in {T} that is farthest from point Q (e.g., point cloud P1), and denote the line passing through points Q and P1 as L. Then, from {L... j Select the straight line with the smallest angle to L from the given line, and use it as the straight line parallel to the direction of the steel cable, thus defining the direction of the steel cable suspension. generally, The angle between the direction and the vector pointing from point Q to point P1 is an acute angle.

[0109] In some embodiments, see Figure 3 , Figure 3 This is another flowchart illustrating the hook point cloud segmentation method provided in this application embodiment, wherein step 101 may specifically include:

[0110] 1011. Fit the plane equation corresponding to the point cloud set of the steel cable to obtain the steel cable plane, and determine the third straight line passing through the position of the steel cable suspension point and the direction of steel cable suspension;

[0111] 1012. Based on the maximum length and maximum width of the hook on the horizontal plane, filter out the candidate point cloud set of the hook from the frame point cloud from the cable suspension point to the ground.

[0112] That is, let R be the frame point cloud from the point where the steel cable is suspended to the ground. e Fit the plane equation of the point cloud set {T} of the steel cable, and determine the points passing through the suspension point Q of the steel cable and the direction of the suspension of the steel cable. The straight line (third straight line F). Then, a second preset distance d2 is set based on the maximum width of the hook on the horizontal plane (e.g., the XY plane), and a third preset distance d3 is set based on the maximum length of the hook on the horizontal plane. d2 is generally greater than the maximum width, and d3 is greater than the maximum length; for example, d3 should be 50% larger than the maximum length. And from the point cloud set R to be filtered... e In the process, the point cloud that is less than the second preset distance d2 from the cable plane and less than the third preset distance d3 from the third straight line F is segmented, which is the hook candidate point cloud set {G}. roi These point clouds are the point clouds within the predicted area where the hook will appear.

[0113] By limiting the second preset distance d2 and the third preset distance d3 mentioned above, the finally estimated hook appearance area can be considered as a coordinate region where the X-axis coordinate value is slightly greater than the maximum length value of the hook, the Y-axis coordinate value is slightly greater than the maximum width value of the hook, and the Z-axis coordinate value is not higher than the coordinate of the cable suspension point Q, thus effectively reducing the point cloud segmentation range of the hook.

[0114] 102. Based on the suspension direction of the steel cable and the candidate point cloud set of the hook, determine the height coordinate range of the hook. Further details can be found in the following sections. Figure 3 The above step 102 specifically includes: dividing multiple coordinate intervals on a two-dimensional plane according to the suspension direction of the steel cable and the candidate point cloud set of the hook, and determining the height coordinate interval of the hook according to the maximum distance of each coordinate interval, the actual length value of the top surface of the hook and the actual length value of the bottom surface of the hook.

[0115] In some embodiments, step 102 specifically includes:

[0116] Based on the suspension direction of the steel cable, each point cloud in the candidate point cloud set of the hook is converted into a planar coordinate point in a two-dimensional plane to obtain a set of planar coordinate points. The coordinate system of the two-dimensional plane includes a first coordinate axis and a second coordinate axis.

[0117] Divide the two-dimensional position coordinates corresponding to the set of plane coordinate points into multiple coordinate intervals along the direction of the first coordinate axis, and determine the maximum distance of the interval based on the plane coordinate points in the coordinate interval;

[0118] Based on the maximum distance of each coordinate interval, the actual length of the top surface of the hook, and the actual length of the bottom surface of the hook, determine the top boundary position coordinates and bottom boundary position coordinates of the hook on the two-dimensional plane;

[0119] The height coordinate range of the hook is determined based on the coordinates of the top boundary and the bottom boundary.

[0120] The actual lengths of the hook top and hook ground were measured in advance. First, the direction of the steel cable suspension was determined. and the candidate point cloud set {G roi Projecting them onto the same two-dimensional plane yields... Direction on a two-dimensional plane and {G roi The set of projection points on a two-dimensional plane (denoted as the set of plane coordinate points {D}) 2}), and, can be directional Let X1 be the first coordinate axis of the two-dimensional plane, perpendicular to the direction. direction Let Y1 be the second coordinate axis of the two-dimensional plane, and let each projection point {D} be... 2 The corresponding 3D position coordinates of the point cloud are transformed from the point cloud spatial coordinate system (e.g., the XYZ coordinate system) to the 2D position coordinates in the 2D planar coordinate system X1Y1, thereby realizing {G roi The point cloud in} to {D 2 Transformation of plane coordinate points in}

[0121] After that, we can first work on {D} 2} is filtered to remove noise and outliers, and then along the first coordinate axis X1, {D} is filtered at intervals of the fourth preset distance d4 (default empirical value). 2 The corresponding coordinate region (i.e., the coordinate region with the minimum X1 coordinate value and the maximum X1 coordinate value as endpoints) is divided into m equally spaced coordinate intervals. Then, the maximum distance of each coordinate interval is calculated. Based on the maximum distance of each coordinate interval, the actual length value of the top surface of the hook and the actual length value of the bottom surface of the hook, the top boundary position coordinates S1 and the bottom boundary position coordinates S2 of the hook on the two-dimensional plane (i.e., the two-dimensional position coordinates under the two-dimensional plane coordinate system X1Y1) are determined. The two-dimensional S1 and S2 are projected onto the vertical coordinate axis (i.e., the Z axis) in the three-dimensional point cloud space coordinate system XYZ to obtain the minimum height coordinate h1 corresponding to S1 and the maximum height coordinate h2 corresponding to S2, and then the height coordinate interval [h1, h2] is obtained.

[0122] In some embodiments, the above step of "determining the top boundary position coordinates and bottom boundary position coordinates of the hook on the two-dimensional plane based on the maximum distance of each coordinate interval, the actual length value of the top surface of the hook, and the actual length value of the bottom surface of the hook" specifically includes:

[0123] The first target coordinate interval is determined from the coordinate interval based on the actual length value of the top surface of the hook, and the second target coordinate interval is determined from the coordinate interval based on the actual length value of the bottom surface of the hook.

[0124] The coordinates of the end point in the first target coordinate interval are taken as the top boundary position coordinates of the hook on the two-dimensional plane, and the coordinates of the end point in the second target interval are taken as the bottom boundary position coordinates of the hook on the two-dimensional plane.

[0125] That is, first set the fifth preset distance d5 based on the actual length of the top surface of the hook, and set the sixth preset distance d6 based on the actual length of the bottom surface of the hook. Then, along the first coordinate axis... The maximum distance between each coordinate interval is compared sequentially from front to back. The first coordinate interval whose maximum distance is greater than the fifth preset distance d5 (the first target coordinate interval), such as the second coordinate interval, is taken as the top boundary b1 of the hook. At this time, the coordinates of the end point in the first target coordinate interval can be taken as the coordinates of the top boundary b1 of the hook (i.e., the top boundary position coordinates S1). Then, the maximum distance between subsequent coordinate intervals is compared. The first coordinate interval whose distance is less than the sixth preset distance d6 (the second target coordinate interval), such as the fifth coordinate interval, is taken as the bottom boundary b2 of the hook. At this time, the coordinates of the end point in the second target coordinate interval can be taken as the coordinates of the bottom boundary b2 of the hook (i.e., the top boundary position coordinates S2).

[0126] For the first target coordinate interval and the second target coordinate interval, their respective endpoint coordinates can be determined by the following formula:

[0127]

[0128] Where d4 is the fourth preset distance mentioned above, that is, the interval distance between the coordinate intervals. min For the set of planar coordinate points {D 2 The first coordinate axis in} The smallest coordinate component in the direction.

[0129] In some embodiments, the step of "determining the maximum distance of the interval based on the plane coordinate point in the coordinate interval" specifically includes:

[0130] Clustering the planar coordinate points in each coordinate interval yields multiple clusters;

[0131] Calculate the maximum distance for each cluster on the second coordinate axis, and select the maximum distance with the largest value as the maximum distance of the corresponding coordinate interval.

[0132] That is, clustering of planar coordinate points is performed within each coordinate interval, and the value of each cluster is calculated in the direction of the second coordinate axis Y1 (i.e., The maximum distance in the direction is used as the maximum distance of the corresponding coordinate interval. 103. Based on the height coordinate interval, segment all point clouds of the hook from the candidate point cloud set.

[0133] Among them, the candidate point cloud set {G} of the hook can be... roi The point cloud of the Z-axis coordinate component in the height coordinate interval [h1, h2] is used as the point cloud of the hook to realize the point cloud segmentation of the hook.

[0134] As described above, the hook point cloud segmentation method provided in this application segmentes hook candidate point cloud sets by filtering from the frame point cloud from the cable suspension point location to the ground based on the cable suspension point location, cable suspension direction, and cable point cloud set; determining the height coordinate range of the hook based on the cable suspension direction and the hook candidate point cloud set; and segmenting all hook point clouds from the hook candidate point cloud set based on the height coordinate range. That is, the hook point cloud is segmented based on the actual cable point cloud and cable suspension condition, so that even when the cable sways, the point cloud density is uneven, or the hook appearance is deformed, the hook point cloud can still be accurately segmented, resulting in good hook point cloud segmentation effect.

[0135] Based on the method described in the above embodiments, this application also provides a hook point cloud segmentation device for performing the steps in the above hook point cloud segmentation method. Please refer to... Figure 4 , Figure 4 This is a schematic diagram of the hook point cloud segmentation device provided in this application embodiment. The hook point cloud segmentation device 300 is applied in an electronic device, which can be implemented as a user terminal or a server. The user terminal includes laptops, tablets, desktop computers, mobile devices (e.g., mobile phones, personal digital assistants, dedicated messaging devices), etc. Specifically, the hook point cloud segmentation device 300 includes a filtering unit 301, a determining unit 302, and a segmentation unit 303, wherein:

[0136] The filtering unit 301 is used to filter out a candidate point cloud set of hooks from the frame point cloud from the location of the cable suspension point to the ground during the process of lifting objects by engineering machinery with lifting function, based on the location of the cable suspension point, the direction of the cable suspension, and the point cloud set of the cable.

[0137] The determining unit 302 is used to determine the height coordinate range of the hook based on the suspension direction of the steel cable and the candidate point cloud set of the hook;

[0138] The segmentation unit 303 is used to segment all point clouds of the hook from the candidate point cloud set of the hook according to the height coordinate interval.

[0139] In some embodiments, the filtering unit 301 is further configured to:

[0140] Establish a tree-structured data structure corresponding to the frame point cloud from the location of the steel cable suspension point to the ground;

[0141] Select the cable seed point from this tree-shaped data structure;

[0142] Based on the seed point of the steel cable, the set of steel cable point clouds is determined from the tree data structure by the region growing method.

[0143] In some embodiments, the determining unit 302 is specifically used for:

[0144] Based on the suspension direction of the steel cable and the candidate point cloud set of the hook, multiple coordinate intervals are divided on the two-dimensional plane. The height coordinate interval of the hook is determined according to the maximum distance of each coordinate interval, the actual length of the top surface of the hook, and the actual length of the bottom surface of the hook.

[0145] In some embodiments, the determining unit 302 is specifically used for:

[0146] Based on the suspension direction of the steel cable, each point cloud in the candidate point cloud set of the hook is converted into a planar coordinate point in a two-dimensional plane to obtain a set of planar coordinate points. The coordinate system of the two-dimensional plane includes a first coordinate axis and a second coordinate axis.

[0147] Divide the two-dimensional position coordinates corresponding to the set of plane coordinate points into multiple coordinate intervals along the direction of the first coordinate axis, and determine the maximum distance of the interval based on the plane coordinate points in the coordinate interval;

[0148] Based on the maximum distance of each coordinate interval, the actual length of the top surface of the hook, and the actual length of the bottom surface of the hook, determine the top boundary position coordinates and bottom boundary position coordinates of the hook on the two-dimensional plane;

[0149] The height coordinate range of the hook is determined based on the coordinates of the top boundary and the bottom boundary.

[0150] In some embodiments, the determining unit 302 is specifically used for:

[0151] The first target coordinate interval is determined from the coordinate interval based on the actual length value of the top surface of the hook, and the second target coordinate interval is determined from the coordinate interval based on the actual length value of the bottom surface of the hook.

[0152] The coordinates of the end point in the first target coordinate interval are taken as the top boundary position coordinates of the hook on the two-dimensional plane, and the coordinates of the end point in the second target interval are taken as the bottom boundary position coordinates of the hook on the two-dimensional plane.

[0153] In some embodiments, the determining unit 302 is specifically used for:

[0154] Clustering the planar coordinate points in each coordinate interval yields multiple clusters;

[0155] Calculate the maximum distance for each cluster on the second coordinate axis, and select the maximum distance with the largest value as the maximum distance of the corresponding coordinate interval.

[0156] In some embodiments, the filtering unit 301 is further configured to:

[0157] Determine the minimum bounding box of the cable point cloud set;

[0158] The first straight line is obtained by determining the point cloud that is farthest from the suspension point of the steel cable in the point cloud set and the straight line that passes through the suspension point of the steel cable.

[0159] Determine the straight lines in the three directions of the length, width, and height of the minimum bounding box to obtain three second straight lines;

[0160] The direction of the second line, which has the smallest angle with the first line, is selected as the direction of the cable suspension.

[0161] In some embodiments, the filtering unit 301 is further configured to:

[0162] Fit the plane equation corresponding to the point cloud set of the steel cable to obtain the steel cable plane;

[0163] Determine the third straight line passing through the location of the cable suspension point and the direction of the cable suspension;

[0164] Based on the maximum length and maximum width of the hook on the horizontal plane, a set of candidate hook point clouds is selected from the frame point cloud from the location of the cable suspension point to the ground.

[0165] It should be noted that the specific details of each module unit in the hook point cloud segmentation device 300 have been described in detail in the embodiments of the hook point cloud segmentation method, and will not be repeated here.

[0166] In some embodiments, the hook point cloud segmentation device in this application can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal device. Exemplary examples include mobile phones, tablets, laptops, handheld computers, in-vehicle electronic devices, mobile internet devices (MIDs), augmented reality (AR) / virtual reality (VR) devices, robots, wearable devices, ultra-mobile personal computers (UMPCs), netbooks, or personal digital assistants (PDAs), etc. It can also be a server, network attached storage (NAS), a personal computer (PC), a television set (TV), an ATM, or a self-service machine, etc. This application does not impose specific limitations.

[0167] In some embodiments, such as Figure 5 As shown, this application embodiment also provides an electronic device 400, including a processor 401, a memory 402, and a computer program stored in the memory 402 and executable on the processor 401. When the program is executed by the processor 401, it implements the various processes of the above-described hook point cloud segmentation method embodiment and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0168] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0169] Figure 6 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0170] The electronic device 500 includes, but is not limited to, components such as: radio frequency unit 501, network module 502, audio output unit 503, input unit 504, sensor 505, display unit 506, user input unit 507, interface unit 508, memory 509, and processor 510.

[0171] Those skilled in the art will understand that the electronic device 500 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 510 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 6The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0172] It should be understood that, in this embodiment, the input unit 504 may include a graphics processing unit (GPU) 5041 and a microphone 5042. The GPU 5041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 506 may include a display panel 5061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 507 includes at least one of a touch panel 5071 and other input devices 5072. The touch panel 5071 is also called a touch screen. The touch panel 5071 may include a touch detection device and a touch controller. Other input devices 5072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0173] The memory 509 can be used to store software programs and various data. The memory 509 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 509 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 509 in this embodiment includes, but is not limited to, these and any other suitable types of memory.

[0174] Processor 510 may include one or more processing units; processor 510 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 510.

[0175] This application also provides a non-transitory computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the various processes of the above-described hook point cloud segmentation method embodiment and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0176] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0177] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described hook point cloud segmentation method.

[0178] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0179] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0180] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0181] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

[0182] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0183] In the description of this application, "multiple" means two or more.

[0184] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0185] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for segmenting point clouds using a hook, characterized in that, include: During the lifting process of engineering machinery with hoisting function, the plane equation corresponding to the point cloud set of steel cable is fitted to obtain the steel cable plane, and the third straight line passing through the position of the steel cable suspension point and the direction of steel cable suspension is determined; Based on the maximum length and maximum width of the hook on the horizontal plane, a set of candidate hook point clouds is selected from the frame point cloud from the location of the cable suspension point to the ground. Based on the suspension direction of the steel cable, each point cloud in the candidate point cloud set of the hook is converted into a planar coordinate point in a two-dimensional plane to obtain a set of planar coordinate points. The coordinate system of the two-dimensional plane includes a first coordinate axis and a second coordinate axis. The two-dimensional position coordinates corresponding to the set of planar coordinate points are divided into multiple coordinate intervals along the direction of the first coordinate axis. The planar coordinate points in each coordinate interval are clustered to obtain multiple clusters. Calculate the maximum distance of each cluster on the second coordinate axis, and select the maximum distance with the largest value as the maximum distance of the corresponding coordinate interval; Based on the maximum distance of each coordinate interval, the actual length of the top surface of the hook, and the actual length of the bottom surface of the hook, determine the top boundary position coordinates and bottom boundary position coordinates of the hook on the two-dimensional plane; The height coordinate range of the hook is determined based on the coordinates of the top boundary position and the coordinates of the bottom boundary position; Based on the height coordinate range, all point clouds of the hook are segmented from the candidate point cloud set of the hook.

2. The hook point cloud segmentation method according to claim 1, characterized in that, The steps for determining the steel cable point cloud set include: Establish a tree-shaped data structure corresponding to the frame point cloud from the location of the steel cable suspension point to the ground; Select the cable seed point from the tree data structure; Based on the seed points of the steel cables, a set of steel cable point clouds is determined from the tree data structure using the region growing method.

3. The hook point cloud segmentation method according to claim 1, characterized in that, The step of determining the top and bottom boundary coordinates of the hook on the two-dimensional plane based on the maximum distance of each coordinate interval, the actual length of the hook top surface, and the actual length of the hook bottom surface includes: The first target coordinate interval is determined from the coordinate interval based on the actual length value of the top surface of the hook, and the second target coordinate interval is determined from the coordinate interval based on the actual length value of the bottom surface of the hook; The coordinates of the end point in the first target coordinate interval are used as the top boundary position coordinates of the hook on the two-dimensional plane, and the coordinates of the end point in the second target coordinate interval are used as the bottom boundary position coordinates of the hook on the two-dimensional plane.

4. The hook point cloud segmentation method according to claim 1, characterized in that, The steps for determining the suspension direction of the steel cable include: Determine the minimum bounding box of the cable point cloud set; The first straight line is obtained by determining the point cloud that is furthest from the suspension point of the steel cable in the point cloud set and the suspension point of the steel cable; Determine the straight lines in the three directions of the length, width, and height of the minimum bounding box to obtain three second straight lines; The direction of the second line, which has the smallest angle with the first line, is selected as the direction of the cable suspension.

5. A hook-based point cloud segmentation device, characterized in that, include: The filtering unit is used to fit the plane equation corresponding to the point cloud set of steel cables during the lifting of objects by engineering machinery with lifting function, to obtain the steel cable plane, and to determine the third straight line passing through the position of the steel cable suspension point and the direction of steel cable suspension; Based on the maximum length and maximum width of the hook on the horizontal plane, a set of candidate hook point clouds is selected from the frame point cloud from the location of the cable suspension point to the ground. The determining unit is used to convert each point cloud in the candidate point cloud set of the hook into planar coordinate points in a two-dimensional plane according to the suspension direction of the steel cable, so as to obtain a set of planar coordinate points. The coordinate system of the two-dimensional plane includes a first coordinate axis and a second coordinate axis. The two-dimensional position coordinates corresponding to the set of planar coordinate points are divided into multiple coordinate intervals along the direction of the first coordinate axis. The planar coordinate points in each coordinate interval are clustered to obtain multiple clusters. Calculate the maximum distance of each cluster on the second coordinate axis, and select the maximum distance with the largest value as the maximum distance of the corresponding coordinate interval; Based on the maximum distance of each coordinate interval, the actual length of the top surface of the hook, and the actual length of the bottom surface of the hook, determine the top boundary position coordinates and bottom boundary position coordinates of the hook on the two-dimensional plane; The height coordinate range of the hook is determined based on the coordinates of the top boundary position and the coordinates of the bottom boundary position; A segmentation unit is used to segment all point clouds of the hook from the candidate point cloud set of the hook according to the height coordinate interval.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the hook point cloud segmentation method as described in any one of claims 1-4.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the hook point cloud segmentation method as described in any one of claims 1-4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the hook point cloud segmentation method as described in any one of claims 1-4.

Images