Grab safety protection method and device based on laser radar point cloud

Abstract

This application discloses a method and device for grab bucket safety protection based on lidar point clouds. The excavator method includes: collecting current frame point cloud data of the ship's cabin at a preset frequency using a lidar pre-installed on the excavator; constructing and updating a latest three-dimensional point cloud map of the cabin based on the current frame point cloud data; identifying the bulkhead and its point cloud, the hull and its point cloud, and the grab bucket and its point cloud based on the latest three-dimensional point cloud map; calculating the horizontal safety distance between the grab bucket and the bulkhead, and the vertical safety distance between the grab bucket and the hull in real time based on the bulkhead point cloud, the hull point cloud, and the grab bucket point cloud; and determining whether to generate a warning message and an emergency stop command based on a comparison of whether the horizontal and vertical safety distances exceed preset thresholds. Therefore, using the embodiments of this application can effectively prevent collision accidents, significantly reduce the reliance on operator experience, and improve the safety and efficiency of complex cabin operations.

Description

Technical Field

[0001] This application relates to the field of port machinery safety technology, and in particular to a grab bucket safety protection method and device based on lidar point cloud. Background Technology

[0002] In port bulk cargo handling operations, especially inside the holds of various cargo ships, tracked or wheeled excavators equipped with grab buckets are typically used for unloading or cleaning bulk cargoes such as coal, ore, and grain. Due to the special structure of ship holds, their interior spaces are usually narrow, deep, and poorly lit, resulting in numerous blind spots. The excavator operator, located in the cab, has their field of vision severely limited by hatches, bulkheads, and materials, making it difficult to accurately and in real-time perceive the three-dimensional spatial distances between the grab bucket body and teeth and the hard structures such as bulkheads and bottom.

[0003] In related technologies, safety protection for the aforementioned work scenarios mainly relies on the operator's personal experience and visual observation. However, relying entirely on the operator's senses and experience, especially in dimly lit and limited-view cabins, can easily lead to misjudgments of distance, causing the grab bucket to scrape or collide with the cabin walls, or the grab bucket to be lowered excessively and hit the bottom of the cabin, resulting in equipment damage and reducing the safety of excavator operations. Summary of the Invention

[0004] This application provides a grab safety protection method and apparatus based on lidar point clouds. To provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. This summary is not intended as a general commentary, nor is it intended to identify key / important components or describe the scope of protection of these embodiments. Its sole purpose is to present some concepts in a simple form as a prelude to the detailed description that follows.

[0005] In a first aspect, embodiments of this application provide a grab bucket safety protection method based on lidar point clouds, applied to excavators, the method comprising: The current frame point cloud data inside the ship's cabin is collected by a lidar pre-installed on the excavator at a preset frequency. Based on the current frame point cloud data, construct and update the latest 3D point cloud map of the cabin; Based on the latest 3D point cloud map, the bulkhead and its bulkhead point cloud, the hull and its hull point cloud, and the grab bucket and its grab bucket point cloud were identified. Based on the bulkhead point cloud, hull point cloud, and grab point cloud, calculate in real time the horizontal safety distance between the grab and the bulkhead, and the vertical safety distance between the grab and the hull. Based on the comparison results of whether the horizontal and vertical safety distances exceed preset thresholds, it is determined whether to generate warning information and emergency stop instructions.

[0006] Optionally, based on the current frame's point cloud data, construct and update the latest 3D point cloud map of the ship's cabin, including: The current frame point cloud data is downsampled and filtered to remove noise, resulting in denoised point cloud data; The point cloud data of each point in the denoised point cloud data is uniformly transformed from the sensor coordinate system of the lidar itself to the excavator body coordinate system with the excavator's slewing center as the origin, to obtain the preprocessed point cloud data of the current frame; If the previous frame of historical preprocessed point cloud data exists, obtain the historical 3D point cloud map of the cabin; The current frame preprocessed point cloud data is registered with the previous frame historical preprocessed point cloud data to obtain the registered current frame point cloud. The registered point cloud of the current frame is fused into the historical 3D point cloud map in the form of a voxel mesh to obtain the latest 3D point cloud map corresponding to the registered point cloud of the current frame, which serves as the latest 3D point cloud map of the ship's cabin.

[0007] Optionally, the preprocessed point cloud data of the current frame is registered with the historical preprocessed point cloud data of the previous frame to obtain the registered point cloud of the current frame, including: Based on the historical preprocessed point cloud data of the previous frame, a probability distribution model is constructed; By maximizing the probability density of the current frame preprocessed point cloud data under the probability distribution model, the first transformation matrix of the current frame preprocessed point cloud data relative to the previous frame historical preprocessed point cloud data is solved to complete the coarse registration. Using the first transformation matrix as the initial transformation, the iterative nearest point algorithm is used to perform fine registration between the preprocessed point cloud data of the current frame and the historical preprocessed point cloud data of the previous frame. The optimal rigid body transformation is calculated by minimizing the distance between corresponding point pairs, and the second transformation matrix is ​​obtained. Using the second transformation matrix, spatial transformation is performed on the preprocessed point cloud data of the current frame to obtain the registration point cloud; Calculate the registration error. If the registration error is less than the preset error threshold, use the registered point cloud as the current frame point cloud after registration.

[0008] Optionally, based on the latest 3D point cloud map, identify the bulkhead and its point cloud, the hull and its point cloud, and the grab bucket and its point cloud, including: From the latest 3D point cloud map, fit and separate the static structural point cloud set of the plane; From the latest 3D point cloud map, remove the static structure point cloud set to obtain the remaining point cloud set; From the remaining point cloud set, segment out the non-planar dynamic structure point cloud set; Extract geometric features from static and dynamic structured point cloud sets; The geometric features are input into a pre-trained machine learning classifier, which outputs the semantic category of each point cloud set. Based on semantic categories, the bulkhead and its point cloud, the hull and its point cloud, and the grab bucket and its point cloud were identified.

[0009] Optionally, fit and separate a set of static structural point clouds representing the bulkhead and floor planes, including: From the latest 3D point cloud map, three points are randomly selected to fit a candidate planar model; Calculate the vertical distance from all points in the latest 3D point cloud map to the candidate plane model, and determine the points whose distance is less than the preset distance threshold as interior points; Record the number of interior points corresponding to the current candidate plane model; Continue executing the step of randomly selecting three points from the latest 3D point cloud map to fit the candidate plane model until the number of selections reaches the preset threshold. Then, the candidate plane with the most interior points is taken as the valid plane, and the interior point set of the valid plane is removed from the latest 3D point cloud map and added to the static structure point cloud candidate set. Continue executing the step of randomly selecting three points from the latest 3D point cloud map to fit a candidate planar model until it is impossible to segment a planar model that meets the minimum number of interior points from the remaining point cloud, thereby generating a static structural point cloud set representing the bulkhead and bottom plane.

[0010] Optionally, from the remaining point cloud set, a non-planar dynamic structure point cloud set is segmented, including: Construct a spatial data structure for the remaining point cloud set to accelerate nearest neighbor search; Select an unvisited point from the remaining point cloud set as the seed point; In a spatial data structure, search for all points in the neighborhood of a seed point whose spatial Euclidean distance to the seed point is less than a preset clustering distance threshold. Add the searched points to the current cluster and mark them as visited; Select a new point from the current cluster that has not been used as a seed point, and continue to select an unvisited point from the remaining point cloud set as a seed point until there are no points in the current cluster that have not been used as seed points, and obtain the segmentation result of the point cloud clustering. Continue executing the steps of building a spatial data structure for the remaining point cloud set until all points in the remaining point cloud set have been visited, and the segmentation results of multiple point cloud clusters are obtained; The segmentation results of multiple point cloud clusters are discarded if the number of points is less than the preset minimum number of cluster points, and the segmentation results with the number of points greater than the preset maximum number of cluster points are split or discarded to obtain a non-planar dynamic structure point cloud set.

[0011] Optionally, based on the bulkhead point cloud, hull point cloud, and grab point cloud, the horizontal safety distance between the grab and the bulkhead, and the vertical safety distance between the grab and the hull are calculated in real time, including: For the point cloud of the grab bucket, extract geometric features, shape features, statistical features and projection features as feature vectors; The feature vector is input into a pre-trained support vector machine classifier to obtain the recognition result of whether the grab point cloud is a grab. Acquire images of the ship's cabin captured by a camera pre-installed on the excavator; Based on the cabin images, visual verification is performed using a pre-set YOLO object detection algorithm to obtain the verification results. When both the identification and verification results indicate that the grab point cloud is a grab, the Kalman filter algorithm is used to continuously track and smooth the state of the grab to obtain the optimal bounding box parameters of the grab. Determine the current grab bounding box for distance calculation based on the optimal bounding box parameters of the grab; Based on the bulkhead point cloud, hull point cloud, and current grab bucket surround box, calculate the horizontal safe distance between the grab bucket and the bulkhead, and the vertical safe distance between the grab bucket and the hull.

[0012] Optionally, based on the bulkhead point cloud, hull point cloud, and current grab bin surround box, calculate the horizontal safety distance between the grab and the bulkhead, and the vertical safety distance between the grab and the hull, including: For the bulkhead point cloud and the hull point cloud respectively, construct the first KD tree and the second KD tree; Based on the four sides of the current grab bucket enclosure, calculate the distance from each side to the nearest point in the bulkhead point cloud indexed by the first KD tree, and take the minimum distance as the candidate horizontal distance; or, based on the projection of the current grab bucket enclosure onto the horizontal plane, calculate the nearest distance on the horizontal plane to the bulkhead point cloud, and take it as the candidate horizontal distance. Calculate the distance from the bottom plane of the current grab enclosure to the nearest point in the bilge point cloud indexed by the second KD tree, and use it as a candidate vertical distance; Multiple candidate horizontal and vertical distances obtained in consecutive calculation cycles are subjected to moving average filtering to obtain smooth and stable horizontal and vertical safe distances.

[0013] Optionally, based on the comparison results of whether the horizontal and vertical safety distances exceed preset thresholds, it is determined whether to generate warning information and emergency stop commands, including: Obtain a first warning distance threshold, a second warning distance threshold, and an emergency stop distance threshold; the first warning distance threshold is greater than the second warning distance threshold, and the second warning distance threshold is greater than or equal to the emergency stop distance threshold. Determine whether the horizontal and / or vertical safety distances are less than or equal to the first warning distance threshold; If the horizontal safety distance and / or vertical safety distance are less than or equal to the first warning distance threshold and greater than the second warning distance threshold, a first-level warning is triggered. The first-level warning includes generating a first type of prompt information on the human-computer interaction interface. If the horizontal safety distance and / or vertical safety distance are less than or equal to the second warning distance threshold and greater than the emergency stop distance threshold, a second-level warning is triggered. The second-level warning includes generating a second type of prompt information, and the warning intensity of the second type of prompt information is higher than that of the first type of prompt information. If the horizontal and / or vertical safety distances are less than or equal to the emergency stop distance threshold, a Level 3 warning is triggered. The Level 3 warning includes generating a control signal, which is used to control the excavator to perform an emergency stop operation.

[0014] Secondly, embodiments of this application provide a grab bucket safety protection device based on lidar point clouds, the device comprising: The point cloud data acquisition module is used to acquire the current frame point cloud data inside the ship's cabin at a preset frequency using a lidar pre-installed on the excavator. The 3D point cloud map building module is used to build and update the latest 3D point cloud map of the cabin based on the current frame point cloud data. The environment recognition module is used to identify the bulkhead and its bulkhead point cloud, the hull and its hull point cloud, and the grab bucket and its grab bucket point cloud based on the latest 3D point cloud map. The distance calculation module is used to calculate the horizontal safe distance between the grab and the bulkhead and the vertical safe distance between the grab and the bulkhead in real time based on the bulkhead point cloud, the hull point cloud, and the grab bucket point cloud. The early warning module is used to determine whether to generate early warning information and emergency stop instructions based on the comparison results of whether the horizontal and vertical safety distances exceed preset thresholds.

[0015] The technical solutions provided in this application embodiment may include the following beneficial effects: In this embodiment, high-precision point cloud data is automatically collected by a laser radar with a preset frequency, and a three-dimensional point cloud map is constructed and updated based on this data. The precise positions of the bulkhead, hull, and grab bucket are automatically identified from the map in a systematic way. This allows for the real-time and automatic calculation of the horizontal and vertical safe distances between the grab bucket and the hull structure. Based on this objective measurement result rather than subjective judgment, a decision is made on whether to generate a warning or emergency stop command. This fundamentally replaces unreliable human visual judgment and realizes a shift from passive reaction to proactive prevention, greatly improving the safety and reliability of excavators operating in complex cabins.

[0016] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] Figure 1 This is a schematic diagram of the method flow of a grab bucket safety protection method based on lidar point cloud provided in an embodiment of this application; Figure 2 This is a schematic diagram of a lidar installation location provided in an embodiment of this application; Figure 3 This is a schematic diagram of the layout of a human-computer interaction interface provided in an embodiment of this application; Figure 4 This is a system architecture diagram provided in an embodiment of this application; Figure 5 This is a schematic diagram of a grab bucket safety protection process based on lidar point clouds provided in an embodiment of this application; Figure 6 This is a schematic diagram illustrating the process of generating warning information and emergency stop commands according to an embodiment of this application; Figure 7 This is a schematic diagram of the structure of a grab bucket safety protection device based on lidar point clouds provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] The following description and accompanying drawings fully illustrate specific embodiments of this application to enable those skilled in the art to practice them.

[0020] It should be understood that the described embodiments are merely some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0021] In the following description, when referring to the accompanying drawings, the same numbers in different drawings denote the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0022] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances. Furthermore, in the description of this application, unless otherwise stated, "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0023] This application provides a method and device for grab bucket safety protection based on lidar point clouds to solve the problems existing in the aforementioned related technologies. In the embodiments of this application, high-precision point cloud data is automatically collected by lidar at a preset frequency, and a three-dimensional point cloud map is constructed and updated based on this data. The precise positions of the bulkhead, hull, and grab bucket are systematically and automatically identified from the map, thereby enabling real-time and automatic calculation of the horizontal and vertical safe distances between the grab bucket and the hull structure. Based on this objective measurement result rather than subjective judgment, a decision is made on whether to generate a warning or emergency stop command, fundamentally replacing unreliable human visual judgment, realizing a shift from passive reaction to active prevention, and greatly improving the safety and reliability of excavators operating in complex cabins. The following is a detailed description using exemplary embodiments.

[0024] The following will be combined with the appendix Figure 1 -Appendix Figure 6 This application provides a detailed description of the grab safety protection method based on lidar point clouds provided in its embodiments. This method can be implemented using a computer program and can run on a grab safety protection device based on lidar point clouds and a von Neumann architecture. This computer program can be integrated into the application or run as a standalone utility application.

[0025] Please see Figure 1 This document presents a flowchart illustrating a grab bucket safety protection method based on lidar point clouds, applicable to excavators, as an embodiment of this application. Figure 1 As shown, the method in this application embodiment includes the following steps: S101 uses a lidar pre-installed on an excavator to collect current frame point cloud data inside the ship's cabin at a preset frequency. LiDAR, an active optical remote sensing sensor, acquires the distance and angle information of a target in three-dimensional space by emitting a laser beam and measuring the time it takes for the beam to reflect back from the target object. This generates a discrete set of points representing the spatial distribution of the object's surface. For example, a 96-line mechanical rotating LiDAR can be installed on top of an excavator cab via a metal frame fixed to the eye bolt holes. It operates at 9V-32V, consumes 8W, and has an IP67 / IP6K9K protection rating, making it suitable for harsh port environments.

[0026] For example Figure 2 As shown, Figure 2 This application provides a schematic diagram of a lidar installation location. The lidar and computing system are installed on the top of the excavator cab using a metal frame fixed to the eye bolt holes to ensure stability and reliability.

[0027] The preset frequency is a time interval pre-set by the system for the lidar to perform one complete data acquisition cycle. The current frame point cloud data is the three-dimensional spatial dataset acquired by the lidar during one scan cycle at the preset frequency. This dataset consists of a series of discrete points with three-dimensional coordinates (X, Y, Z), which collectively describe the surface contours and spatial positions of all detected objects within the lidar's field of view at the moment of scanning. The ship's interior refers to the internal working space formed by the enclosed or semi-enclosed hull structure used for loading bulk cargo on a cargo ship, mainly including the space defined by the bulkheads, bilge, hatches, and the materials within the hold.

[0028] In some embodiments of this application, the operator starts the excavator and activates the grab bucket safety protection detection system. After the system is powered on, preset system parameters (such as safety distance thresholds, alarm sound settings, and preset cabin dimensions) are loaded. The lidar sensing module, data processing computing system, and human-machine interface display terminal, all mounted on the top of the excavator cab, are activated sequentially. The lidar performs a self-test, completes internal calibration, and begins emitting laser light at a preset frequency of 10Hz, preparing to receive echoes. The computing system starts its main control program, establishing communication connections with the lidar, camera, cab display terminal, and port control network.

[0029] At this time, the lidar rotates and scans at a constant rate, measuring the entire interior space of the excavator's cabin (360° horizontal field of view, 0-90° vertical field of view). Each completed scan generates a current frame of point cloud data, containing the three-dimensional coordinates and reflectivity information of hundreds of thousands of points. This point cloud data is transmitted in real-time via Ethernet UDP protocol to a computing system also installed on top of the cab for processing.

[0030] S102, Based on the current frame point cloud data, construct and update the latest three-dimensional point cloud map of the cabin; In some embodiments of this application, the specific process of constructing and updating the latest three-dimensional point cloud map of the cabin based on the current frame point cloud data includes: downsampling and filtering the current frame point cloud data to obtain denoised point cloud data; uniformly transforming the point cloud data of each point in the denoised point cloud data from the sensor coordinate system of the lidar itself to the excavator body coordinate system with the excavator's slewing center as the origin to obtain the current frame preprocessed point cloud data; if there is historical preprocessed point cloud data from the previous frame, obtaining the historical three-dimensional point cloud map of the cabin; performing point cloud registration between the current frame preprocessed point cloud data and the historical preprocessed point cloud data from the previous frame to obtain the registered current frame point cloud; and fusing the registered current frame point cloud into the historical three-dimensional point cloud map in the form of a voxel mesh to obtain the latest three-dimensional point cloud map corresponding to the registered current frame point cloud, which serves as the latest three-dimensional point cloud map of the cabin.

[0031] In one possible implementation, after receiving the raw point cloud, the computing system first filters it to remove noise points caused by dust, water mist, etc., and then downsamples it to improve the efficiency of subsequent processing. The coordinates of each point are transformed from the lidar's own coordinate system to the excavator's body coordinate system with the excavator's swing center as the origin through a pre-calibrated transformation matrix, thus obtaining the preprocessed point cloud data of the current frame.

[0032] It should be noted that point cloud registration can be achieved by combining the NDT (Normal Distributions Transform) algorithm with the ICP (Iterative Closest Point) algorithm.

[0033] The NDT algorithm works by dividing the point cloud space into multiple voxel grids, calculating the mean and covariance matrix of points within each grid, and constructing a probability distribution model. For the point cloud to be registered, its probability density under the target distribution is calculated, and the optimal transformation matrix is ​​solved by maximizing the probability density. The NDT algorithm has low requirements for initial pose and fast convergence speed, making it suitable for coarse registration.

[0034] The ICP algorithm works by performing fine registration based on coarse registration using NDT. The ICP algorithm iteratively finds the closest point pairs between the source and target point clouds, calculates the rigid body transformation (rotation matrix R and translation vector t) that minimizes the distance between corresponding points, and iterates until convergence. While the ICP algorithm offers high accuracy, it is sensitive to initial pose and is therefore often used in conjunction with the NDT algorithm.

[0035] Specifically, the process of registering the preprocessed point cloud data of the current frame with the historical preprocessed point cloud data of the previous frame to obtain the registered point cloud of the current frame includes: constructing a probability distribution model based on the historical preprocessed point cloud data of the previous frame; solving the first transformation matrix of the preprocessed point cloud data of the current frame relative to the historical preprocessed point cloud data of the previous frame by maximizing the probability density of the preprocessed point cloud data of the current frame under the probability distribution model to complete the coarse registration; using the first transformation matrix as the initial transformation, using the iterative nearest point algorithm to perform fine registration of the preprocessed point cloud data of the current frame with the historical preprocessed point cloud data of the previous frame, and calculating the optimal rigid body transformation by minimizing the distance between corresponding point pairs to obtain the second transformation matrix; using the second transformation matrix to perform spatial transformation on the preprocessed point cloud data of the current frame to obtain the registered point cloud; calculating the registration error, and if the registration error is less than a preset error threshold, the registered point cloud is used as the registered point cloud of the current frame.

[0036] Specifically, a voxel-based fusion strategy is adopted to integrate the registered point cloud data into the global map. For points within the same voxel, a weighted average method is used to update the voxel center position and normal vector. By maintaining the voxel occupancy probability, dynamic point cloud updates and static map construction are achieved. The final constructed ship cabin point cloud map includes geometric information of structures such as bulkheads, hulls, and hatches.

[0037] In one possible implementation, the preprocessed current frame point cloud is registered with the historical point cloud generated from previous frames. First, the NDT algorithm is used for fast and robust coarse registration to obtain a preliminary position transformation relationship. Then, using this result as initial values, the ICP algorithm is used for fine registration to obtain an accurate pose transformation matrix. Using this transformation matrix, the current frame point cloud is accurately stitched into the constructed historical 3D point cloud map. Through a voxel-based fusion strategy, the average position and normal vector of each point within each voxel in the map are updated, thereby dynamically maintaining and updating a globally consistent, up-to-date 3D point cloud map reflecting the complete structure of the current ship compartment (bulwarks, hull, hatches).

[0038] S103, based on the latest 3D point cloud map, identifies the bulkhead and its bulkhead point cloud, the hull and its hull point cloud, and the grab bucket and its grab bucket point cloud; In some embodiments of this application, the specific process of identifying the bulkhead and its bulkhead point cloud, the hull and its hull point cloud, and the grab bucket and its grab bucket point cloud based on the latest 3D point cloud map includes: fitting and separating a set of planar static structural point clouds from the latest 3D point cloud map; removing the set of static structural point clouds from the latest 3D point cloud map to obtain a set of remaining point clouds; segmenting a set of non-planar dynamic structural point clouds from the set of remaining point clouds; extracting geometric features from the set of static structural point clouds and the set of dynamic structural point clouds; inputting the geometric features into a pre-trained machine learning classifier to output the semantic category of each point cloud set; and identifying the bulkhead and its bulkhead point cloud, the hull and its hull point cloud, and the grab bucket and its grab bucket point cloud through the semantic category.

[0039] It should be noted that the RANSAC (Random Sample Consensus) algorithm was used for plane segmentation to extract the bulkhead and hull planes. The RANSAC algorithm fits a planar model by randomly sampling a subset of point clouds, calculates the distance from all points to the plane, counts the number of interior points, and iteratively selects the planar model with the most interior points. The algorithm's distance threshold was set to 5 cm, the number of iterations was set to 1000, and the minimum proportion of interior points was set to 10%. By analyzing the plane's normal vector direction, the bulkhead plane (vertical plane) and the hull plane (horizontal plane) were distinguished.

[0040] Specifically, the process of fitting and separating the static structural point cloud set representing the bulkhead and hull plane includes: randomly selecting three points from the latest 3D point cloud map to fit a candidate plane model; calculating the vertical distance from all points in the latest 3D point cloud map to the candidate plane model, and determining points with distances less than a preset distance threshold as inliers; recording the number of inliers corresponding to the current candidate plane model; continuing to execute the step of randomly selecting three points from the latest 3D point cloud map to fit a candidate plane model until the number of selections reaches a preset threshold, then taking the candidate plane with the most inliers as the valid plane, removing the set of inliers of the valid plane from the latest 3D point cloud map, and adding it to the static structural point cloud candidate set; continuing to execute the step of randomly selecting three points from the latest 3D point cloud map to fit a candidate plane model until it is impossible to segment a plane model that meets the minimum number of inliers requirement from the remaining point cloud, thus generating the static structural point cloud set representing the bulkhead and hull plane.

[0041] It should be noted that for point clouds with non-planar structures (such as grab buckets), Euclidean clustering is used for segmentation. This algorithm clusters points based on the Euclidean distance between them, grouping points with a distance less than a threshold into the same cluster. The parameters of this algorithm are set as follows: clustering distance threshold is 10cm, minimum cluster size is 100 points, and maximum cluster size is 10000 points. The clustering results can separate the grab bucket point cloud from other object point clouds.

[0042] Specifically, the process of segmenting a non-planar dynamic structure point cloud set from the remaining point cloud set includes: constructing a spatial data structure for the remaining point cloud set to accelerate nearest neighbor search; selecting an unvisited point from the remaining point cloud set as a seed point; searching the spatial data structure for all points within the seed point's neighborhood whose spatial Euclidean distance to the seed point is less than a preset clustering distance threshold; adding the searched points to the current cluster and marking them as visited; selecting a new point from the current cluster that has not been used as a seed point, and continuing the process from the remaining point cloud set... The process involves selecting an unvisited point from the set as a seed point until no unvisited point remains in the current cluster, resulting in a point cloud clustering segmentation result. The process continues by constructing a spatial data structure for the remaining point cloud set until all points in the remaining point cloud set have been visited, resulting in multiple point cloud clustering segmentation results. Segmentation results with fewer points than a preset minimum clustering number are removed, while segmentation results with more points than a preset maximum clustering number are split or removed, resulting in a non-planar, dynamically structured point cloud set.

[0043] Specifically, based on geometric segmentation, a feature-based semantic segmentation method is used to classify point clouds. Geometric features of each point cloud cluster (such as the number of points, bounding box size, normal vector distribution, curvature, etc.) are extracted, and semantic annotation of the point cloud clusters is performed in combination with a machine learning classifier (such as SVM or random forest) to identify targets such as bulkheads, hatches, and grabs.

[0044] In one possible implementation, on a constructed global point cloud map, the RANSAC plane segmentation algorithm is used to iteratively fit a planar model to separate point cloud sets representing the bulkhead (vertical plane) and the hull (horizontal plane), which are then used as a static background. These static point clouds are removed from the global map, leaving point clouds that may contain dynamic targets such as grabs and scattered materials. Euclidean clustering is applied to this portion of the point cloud to group points that are spatially close together, thus separating independent object clusters. Geometric features (such as the number of points, bounding box size, centroid position, and shape index) are extracted from each cluster, and the feature vectors are then input into a pre-trained SVM classifier for semantic judgment to identify which object cluster contains the grab. The system also incorporates images from a third-party camera (visual perception module) and uses visual detection algorithms such as YOLO for auxiliary verification to improve recognition accuracy.

[0045] S104, based on the bulkhead point cloud, hull point cloud, and grab point cloud, calculate in real time the horizontal safety distance between the grab and the bulkhead and the vertical safety distance between the grab and the hull. In some embodiments of this application, the specific process of calculating the horizontal safety distance between the grab bucket and the bulkhead and the vertical safety distance between the grab bucket and the bulkhead in real time based on the bulkhead point cloud, the hull point cloud, and the grab bucket point cloud includes: extracting geometric features, shape features, statistical features, and projection features from the grab bucket point cloud as feature vectors; inputting the feature vectors into a pre-trained support vector machine classifier to obtain the recognition result of whether the grab bucket point cloud is a grab bucket; acquiring the hull image captured by a camera pre-installed on the excavator; performing visual verification based on the hull image and a preset YOLO target detection algorithm to obtain the verification result; when both the recognition result and the verification result indicate that the grab bucket point cloud is a grab bucket, using a Kalman filter algorithm to continuously track and smooth the grab bucket's state to obtain the optimal bounding box parameters of the grab bucket; determining the current grab bucket bounding box for distance calculation based on the optimal bounding box parameters of the grab bucket; and calculating the horizontal safety distance between the grab bucket and the bulkhead and the vertical safety distance between the grab bucket and the hull bottom based on the bulkhead point cloud, the hull point cloud, and the current grab bucket bounding box.

[0046] In one possible implementation, feature extraction is performed on the segmented grab bucket point cloud for identification and tracking. The extracted features include geometric features, shape features, statistical features, and projection features. Geometric features include: point cloud quantity, bounding box dimensions (length, width, height), volume, and centroid location; shape features include: eigenvalues ​​of the covariance matrix of the point cloud distribution, and shape indices (sphericity, cylindricity, flatness); statistical features include: point cloud density, normal vector distribution, and curvature distribution statistics (mean, variance); projection features include: projected contour features in the XY, XZ, and YZ planes. Based on the extracted feature vectors, a Support Vector Machine (SVM) classifier is used for grab bucket identification. Grab bucket sample data is pre-collected to train the SVM classification model. In real-time detection, features are extracted from clustered point cloud clusters and input into the SVM classifier to determine whether they represent grab buckets. Simultaneously, YOLO object detection algorithm is used for visual verification in conjunction with third-party camera images to improve recognition accuracy.

[0047] It should be noted that a Kalman filter algorithm is used to achieve continuous and stable tracking of the grab's position. The Kalman filter estimates the optimal state of the grab by fusing historical and current observation data through a prediction-update approach.

[0048] It should be noted that a KD-tree-based nearest neighbor search algorithm is used to calculate the distances between the grab and the bulkhead and bilge. A KD-tree is a spatially partitioned data structure that enables efficient nearest neighbor search. The Best Bin First (BBF) strategy is employed for approximate nearest neighbor search, improving search speed while maintaining accuracy. The search time complexity is reduced from O(n) to O(log n).

[0049] Specifically, the process of calculating the horizontal safety distance between the grab and the bulkhead and the vertical safety distance between the grab and the bulkhead, based on the bulkhead point cloud, the hull point cloud, and the current grab enclosure, includes: constructing a first KD tree and a second KD tree for the bulkhead point cloud and the hull point cloud, respectively; calculating the distance from each of the four sides of the current grab enclosure to the nearest point in the bulkhead point cloud indexed by the first KD tree, and taking the minimum distance as the candidate horizontal distance; or, calculating the nearest distance on the horizontal plane to the bulkhead point cloud based on the projection of the current grab enclosure onto the horizontal plane, and taking it as the candidate horizontal distance; calculating the distance from the bottom plane of the current grab enclosure to the nearest point in the hull point cloud indexed by the second KD tree, and taking it as the candidate vertical distance; and performing moving average filtering on the multiple candidate horizontal and vertical distances obtained in multiple consecutive calculation cycles to obtain smooth and stable horizontal and vertical safety distances.

[0050] In one possible implementation, to simplify distance calculation and collision detection, a bounding box method is used to approximate the grab. AABB (Axis-Aligned Bounding Box): Calculates the minimum and maximum values ​​of the grab's point cloud in the X, Y, and Z directions, constructing a cuboid bounding box aligned with the coordinate axes. AABB is computationally simple, but the bounding box is not tight enough. OBB (Oriented Bounding Box): Calculates the principal axis directions of the point cloud using Principal Component Analysis (PCA), constructing a bounding box aligned with the principal axes. OBB provides a tighter bounding box and is suitable for irregularly shaped grabs. Distance Calculation: Fast collision detection and distance estimation are performed based on the bounding boxes. When the bounding boxes do not intersect, the minimum distance between them is calculated as a safe distance estimate; when the bounding boxes intersect, a collision warning is triggered.

[0051] For example, horizontal distance: calculate the projected distance between the grab enclosure and the bulkhead plane on the XY plane. The minimum distance from the four sides of the grab enclosure to the bulkhead plane is taken as the horizontal safety distance. Vertical distance: calculate the vertical distance from the bottom of the grab to the bottom plane. The distance from the bottom of the grab enclosure to the bottom plane is taken as the vertical safety distance. Distance filtering: apply a moving average filter to the calculated distance values ​​to eliminate measurement noise and obtain a smooth and stable distance estimate. The filtering window size is set to 5 frames, using a weighted average method, with newer frames having a higher weight.

[0052] S105, based on the comparison results of whether the horizontal safety distance and the vertical safety distance exceed the preset threshold, determine whether to generate a warning message and an emergency stop command.

[0053] In some embodiments of this application, the specific process of determining whether to generate a warning message and an emergency stop command based on the comparison results of whether the horizontal and vertical safety distances exceed preset thresholds includes: obtaining a first warning distance threshold, a second warning distance threshold, and an emergency stop distance threshold; the first warning distance threshold is greater than the second warning distance threshold, and the second warning distance threshold is greater than or equal to the emergency stop distance threshold; determining whether the horizontal and / or vertical safety distances are less than or equal to the first warning distance threshold; if the horizontal and / or vertical safety distances are less than or equal to the first warning distance threshold and greater than the second warning distance threshold, a first-level warning is triggered, which includes generating a first type of prompt information on the human-machine interface; if the horizontal and / or vertical safety distances are less than or equal to the second warning distance threshold and greater than the emergency stop distance threshold, a second-level warning is triggered, which includes generating a second type of prompt information, the warning intensity of which is higher than that of the first type of prompt information; if the horizontal and / or vertical safety distances are less than or equal to the emergency stop distance threshold, a third-level warning is triggered, which includes generating a control signal, the control signal being used to control the excavator to perform an emergency stop operation.

[0054] In one possible implementation, the calculated distance is compared with a safety threshold to determine if the system is in a safe state. When the distance is less than the safety threshold, an audible and visual alarm is triggered, the edges of the human-machine interface flash brightly, and an emergency stop signal is simultaneously output to the port control network. The system employs a tiered safety protection strategy: Level 1 Warning (Alert): When the grab bucket enters the hatch projection range, the interface edges flash twice to alert the operator; Level 2 Warning (Warning): When the distance between the grab bucket and the bulkhead approaches the safety threshold, the interface edges flash continuously and an audible warning is played; Level 3 Warning (Emergency): When the distance between the grab bucket and the bulkhead or hull bottom is less than the safety threshold, the interface edges flash rapidly, an alarm sound is played, and an emergency stop signal is output to forcibly stop the excavator's operation. For example... Figure 3 As shown, Figure 3 This is a layout diagram of a human-computer interaction interface provided in this application. The outer contour is the contour of the ship's bulkhead, and the inner side displays the hatch contour, the excavator position point (fixed), the grab bucket position point (changing in real time), and the sea side direction mark.

[0055] Furthermore, a grab bucket safety protection detection component window is set up in the excavator's remote control interface, which has the following functions: Display content: Displays a top-down view of the ship's cabin. The outer contour of the component is the outline of the cabin wall, and the inner side of the outline displays the hatch outline, the excavator's position point, and the position point of the grab bucket when it is above the hatch. Directional markings: Dot-shaped markings are set on one side of the hatch outline to indicate the sea side, making it easier to determine the direction. The system can automatically determine the sea side direction based on the direction the grab enters. Entry Reminder: Whenever the grab enters the hatch projection range and is detected, the window edge will flash twice to indicate that the grab has entered the upper part of the cabin and is about to fall, reminding the operator to observe the movement of the grab in the component; Collision avoidance warning: When the projected position of the grab bucket is too close to the position of the excavator, the edge of the component will flash brightly at high speed, play an alarm sound, and at the same time the system will output a collision avoidance emergency stop signal; Adjustable parameters: alarm distance threshold is adjustable, alarm sound can be turned on or off, window position and size, content color and opacity can be customized.

[0056] For example Figure 6 As shown, Figure 6 This application provides a schematic diagram of the process for generating early warning information and emergency stop commands, involving real-time acquisition of point cloud data (continuous scanning of the ship's cabin by lidar). First, the grab's position is calculated, then it is determined whether the grab has entered the hatch area. If not: return directly to real-time point cloud data acquisition and continue looping monitoring. If yes: trigger a Level 1 early warning (interface flashes twice), then calculate the distance from the grab to the bulkhead (d_wall) and hull bottom (d_bottom). Determine if the distance d is less than a preset threshold: if not: return to real-time point cloud data acquisition and continue looping monitoring. If yes: trigger a Level 3 early warning (high-speed flashing + alarm sound, outputting an emergency stop signal).

[0057] For example Figure 4 As shown, Figure 4 This application provides a system architecture diagram of the overall architecture of the grab bucket safety protection and detection system, including the connection relationship between the lidar, computing system, third-party camera, display terminal and port control network.

[0058] For example Figure 5 As shown, Figure 5This application provides a schematic diagram of a grab bucket safety protection process based on LiDAR point clouds. A LiDAR installed on an excavator scans the interior space of the ship's cabin at a preset frequency (e.g., 10Hz), collecting current frame point cloud data (each frame contains a large number of points with three-dimensional coordinates, reflecting the spatial distribution of objects on the surface of the ship's cabin at the moment of scanning). The original point cloud is filtered, denoised, and transformed to obtain a denoised and coordinate-uniform preprocessed point cloud. The current frame preprocessed point cloud is registered with historical frame preprocessed point clouds (commonly using NDT / ICP algorithms). Using the registered current frame point cloud, combined with the historical three-dimensional point cloud map, a global three-dimensional point cloud map of the ship's cabin is updated and constructed through voxel mesh fusion and other methods. In the global three-dimensional point cloud map, the point cloud is divided into three subsets using the RANSAC plane segmentation algorithm: bulkhead point cloud; hull point cloud; and grab bucket point cloud. For the segmented bulkhead, hatch, and grab bucket point clouds, feature extraction and Kalman filter tracking are performed respectively. This system enables continuous target identification and dynamic tracking, providing accurate target shape and location information for distance calculation. Based on this information, it outputs the horizontal safety distance between the grab bucket and the bulkhead, and the vertical safety distance between the grab bucket and the bottom / hatch, quantifying the physical indicators of safety risk. The calculated horizontal / vertical safety distances are compared with preset thresholds (warning threshold and emergency stop threshold): if the distance is less than or equal to the warning threshold, a warning message is issued (e.g., audible and visual alerts, interface alarm); if the distance is less than or equal to the emergency stop threshold, an emergency stop command is triggered (controlling the excavator to stop operation to avoid collision). Finally, a safety protection decision is output, completing a closed loop from sensing point clouds to ensuring safety.

[0059] In this embodiment, high-precision point cloud data is automatically collected by a laser radar with a preset frequency, and a three-dimensional point cloud map is constructed and updated based on this data. The precise positions of the bulkhead, hull, and grab bucket are automatically identified from the map in a systematic way. This allows for the real-time and automatic calculation of the horizontal and vertical safe distances between the grab bucket and the hull structure. Based on this objective measurement result rather than subjective judgment, a decision is made on whether to generate a warning or emergency stop command. This fundamentally replaces unreliable human visual judgment and realizes a shift from passive reaction to proactive prevention, greatly improving the safety and reliability of excavators operating in complex cabins.

[0060] The following are embodiments of the apparatus of this application, which can be used to execute the embodiments of the method of this application. For details not disclosed in the embodiments of the apparatus of this application, please refer to the embodiments of the method of this application.

[0061] Please see Figure 7This illustration shows a schematic diagram of a grab safety protection device based on lidar point clouds provided in an exemplary embodiment of this application. This lidar point cloud-based grab safety protection device can be implemented as all or part of an electronic device through software, hardware, or a combination of both. The device 1 includes a point cloud data acquisition module 10, a 3D point cloud map construction module 20, an environment recognition module 30, a distance calculation module 40, and an early warning module 50.

[0062] The point cloud data acquisition module 10 is used to acquire the current frame point cloud data inside the ship's cabin at a preset frequency using a lidar pre-installed on the excavator. The 3D point cloud map building module 20 is used to build and update the latest 3D point cloud map of the cabin based on the current frame point cloud data. The environment recognition module 30 is used to identify the bulkhead and its bulkhead point cloud, the hull and its hull point cloud, and the grab bucket and its grab bucket point cloud based on the latest three-dimensional point cloud map. The distance calculation module 40 is used to calculate the horizontal safe distance between the grab and the bulkhead and the vertical safe distance between the grab and the bulkhead in real time based on the bulkhead point cloud, the hull point cloud and the grab bucket point cloud. The early warning module 50 is used to determine whether to generate early warning information and emergency stop command based on the comparison results of whether the horizontal safety distance and the vertical safety distance exceed the preset threshold.

[0063] It should be noted that the above embodiments of the grab safety protection device based on lidar point clouds, when executing the grab safety protection method based on lidar point clouds, are only illustrative examples of the above-described functional module divisions. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the grab safety protection device based on lidar point clouds and the grab safety protection method embodiments based on lidar point clouds belong to the same concept, and their implementation process is detailed in the method embodiments, which will not be repeated here.

[0064] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0065] In this embodiment, high-precision point cloud data is automatically collected by a laser radar with a preset frequency, and a three-dimensional point cloud map is constructed and updated based on this data. The precise positions of the bulkhead, hull, and grab bucket are automatically identified from the map in a systematic way. This allows for the real-time and automatic calculation of the horizontal and vertical safe distances between the grab bucket and the hull structure. Based on this objective measurement result rather than subjective judgment, a decision is made on whether to generate a warning or emergency stop command. This fundamentally replaces unreliable human visual judgment and realizes a shift from passive reaction to proactive prevention, greatly improving the safety and reliability of excavators operating in complex cabins.

[0066] This application also provides a computer-readable medium having program instructions stored thereon, which, when executed by a processor, implement the grab bucket safety protection method based on lidar point cloud provided in the above-described method embodiments.

[0067] This application also provides a computer program product containing instructions that, when run on a computer, causes the computer to execute the grab bucket safety protection method based on lidar point cloud of the above-described method embodiments.

[0068] Please see Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 8 As shown, the electronic device 1000 may include: at least one processor 1001, at least one network interface 1004, a user interface 1003, a memory 1005, and at least one communication bus 1002.

[0069] The communication bus 1002 is used to realize the connection and communication between these components.

[0070] The user interface 1003 may include a display screen and a camera. Optionally, the user interface 1003 may also include a standard wired interface and a wireless interface.

[0071] The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0072] The processor 1001 may include one or more processing cores. The processor 1001 connects to various parts within the electronic device 1000 using various interfaces and lines. It executes various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1005, and by calling data stored in the memory 1005. Optionally, the processor 1001 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 1001 may integrate one or more of the following: a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and a modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed on the screen; and the modem handles wireless communication. It is understood that the modem may also be implemented as a separate chip, without being integrated into the processor 1001.

[0073] The memory 1005 may include random access memory (RAM) or read-only memory. Optionally, the memory 1005 may include a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 1005 may also be at least one storage system located remotely from the aforementioned processor 1001. Figure 8 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a grab bucket safety protection application based on lidar point clouds.

[0074] exist Figure 8In the illustrated electronic device 1000, the user interface 1003 is mainly used to provide an input interface for the user and to acquire user input data; while the processor 1001 can be used to call the grab bucket safety protection application based on lidar point cloud stored in the memory 1005, and specifically perform the following operations: The current frame point cloud data inside the ship's cabin is collected by a lidar pre-installed on the excavator at a preset frequency. Based on the current frame point cloud data, construct and update the latest 3D point cloud map of the cabin; Based on the latest 3D point cloud map, the bulkhead and its bulkhead point cloud, the hull and its hull point cloud, and the grab bucket and its grab bucket point cloud were identified. Based on the bulkhead point cloud, hull point cloud, and grab point cloud, calculate in real time the horizontal safety distance between the grab and the bulkhead, and the vertical safety distance between the grab and the hull. Based on the comparison results of whether the horizontal and vertical safety distances exceed preset thresholds, it is determined whether to generate warning information and emergency stop instructions.

[0075] In one embodiment, when the processor 1001 constructs and updates the latest 3D point cloud map of the cabin based on the current frame point cloud data, it specifically performs the following operations: The current frame point cloud data is downsampled and filtered to remove noise, resulting in denoised point cloud data; The point cloud data of each point in the denoised point cloud data is uniformly transformed from the sensor coordinate system of the lidar itself to the excavator body coordinate system with the excavator's slewing center as the origin, to obtain the preprocessed point cloud data of the current frame; If the previous frame of historical preprocessed point cloud data exists, obtain the historical 3D point cloud map of the cabin; The current frame preprocessed point cloud data is registered with the previous frame historical preprocessed point cloud data to obtain the registered current frame point cloud. The registered point cloud of the current frame is fused into the historical 3D point cloud map in the form of a voxel mesh to obtain the latest 3D point cloud map corresponding to the registered point cloud of the current frame, which serves as the latest 3D point cloud map of the ship's cabin.

[0076] In one embodiment, when processor 1001 performs point cloud registration by matching the current frame preprocessed point cloud data with the previous frame historical preprocessed point cloud data to obtain the registered current frame point cloud, it specifically performs the following operations: Based on the historical preprocessed point cloud data of the previous frame, a probability distribution model is constructed; By maximizing the probability density of the current frame preprocessed point cloud data under the probability distribution model, the first transformation matrix of the current frame preprocessed point cloud data relative to the previous frame historical preprocessed point cloud data is solved to complete the coarse registration. Using the first transformation matrix as the initial transformation, the iterative nearest point algorithm is used to perform fine registration between the preprocessed point cloud data of the current frame and the historical preprocessed point cloud data of the previous frame. The optimal rigid body transformation is calculated by minimizing the distance between corresponding point pairs, and the second transformation matrix is ​​obtained. Using the second transformation matrix, spatial transformation is performed on the preprocessed point cloud data of the current frame to obtain the registration point cloud; Calculate the registration error. If the registration error is less than the preset error threshold, use the registered point cloud as the current frame point cloud after registration.

[0077] In one embodiment, when the processor 1001 identifies the bulkhead and its bulkhead point cloud, the hull and its hull point cloud, and the grab bucket and its grab bucket point cloud based on the latest 3D point cloud map, it specifically performs the following operations: From the latest 3D point cloud map, fit and separate the static structural point cloud set of the plane; From the latest 3D point cloud map, remove the static structure point cloud set to obtain the remaining point cloud set; From the remaining point cloud set, segment out the non-planar dynamic structure point cloud set; Extract geometric features from static and dynamic structured point cloud sets; The geometric features are input into a pre-trained machine learning classifier, which outputs the semantic category of each point cloud set. Based on semantic categories, the bulkhead and its point cloud, the hull and its point cloud, and the grab bucket and its point cloud were identified.

[0078] In one embodiment, when the processor 1001 performs fitting and separation of a set of static structural point clouds representing the bulkhead and floor planes, it specifically performs the following operations: From the latest 3D point cloud map, three points are randomly selected to fit a candidate planar model; Calculate the vertical distance from all points in the latest 3D point cloud map to the candidate plane model, and determine the points whose distance is less than the preset distance threshold as interior points; Record the number of interior points corresponding to the current candidate plane model; Continue executing the step of randomly selecting three points from the latest 3D point cloud map to fit the candidate plane model until the number of selections reaches the preset threshold. Then, the candidate plane with the most interior points is taken as the valid plane, and the interior point set of the valid plane is removed from the latest 3D point cloud map and added to the static structure point cloud candidate set. Continue executing the step of randomly selecting three points from the latest 3D point cloud map to fit a candidate planar model until it is impossible to segment a planar model that meets the minimum number of interior points from the remaining point cloud, thereby generating a static structural point cloud set representing the bulkhead and bottom plane.

[0079] In one embodiment, when the processor 1001 performs the operation of segmenting a non-planar dynamic structure point cloud set from the remaining point cloud set, it specifically performs the following operations: Construct a spatial data structure for the remaining point cloud set to accelerate nearest neighbor search; Select an unvisited point from the remaining point cloud set as the seed point; In a spatial data structure, search for all points in the neighborhood of a seed point whose spatial Euclidean distance to the seed point is less than a preset clustering distance threshold. Add the searched points to the current cluster and mark them as visited; Select a new point from the current cluster that has not been used as a seed point, and continue to select an unvisited point from the remaining point cloud set as a seed point until there are no points in the current cluster that have not been used as seed points, and obtain the segmentation result of the point cloud clustering. Continue executing the steps of building a spatial data structure for the remaining point cloud set until all points in the remaining point cloud set have been visited, and the segmentation results of multiple point cloud clusters are obtained; The segmentation results of multiple point cloud clusters are discarded if the number of points is less than the preset minimum number of cluster points, and the segmentation results with the number of points greater than the preset maximum number of cluster points are split or discarded to obtain a non-planar dynamic structure point cloud set.

[0080] In one embodiment, when the processor 1001 performs real-time calculation of the horizontal safety distance between the grab bucket and the bulkhead, and the vertical safety distance between the grab bucket and the bulkhead based on the bulkhead point cloud, the hull point cloud, and the grab bucket point cloud, it specifically performs the following operations: For the point cloud of the grab bucket, extract geometric features, shape features, statistical features and projection features as feature vectors; The feature vector is input into a pre-trained support vector machine classifier to obtain the recognition result of whether the grab point cloud is a grab. Acquire images of the ship's cabin captured by a camera pre-installed on the excavator; Based on the cabin images, visual verification is performed using a pre-set YOLO object detection algorithm to obtain the verification results. When both the identification and verification results indicate that the grab point cloud is a grab, the Kalman filter algorithm is used to continuously track and smooth the state of the grab to obtain the optimal bounding box parameters of the grab. Determine the current grab bounding box for distance calculation based on the optimal bounding box parameters of the grab; Based on the bulkhead point cloud, hull point cloud, and current grab bucket surround box, calculate the horizontal safe distance between the grab bucket and the bulkhead, and the vertical safe distance between the grab bucket and the hull.

[0081] In one embodiment, when the processor 1001 calculates the horizontal safety distance between the grab and the bulkhead and the vertical safety distance between the grab and the hull based on the bulkhead point cloud, the hull point cloud, and the current grab bounding box, it specifically performs the following operations: For the bulkhead point cloud and the hull point cloud respectively, construct the first KD tree and the second KD tree; Based on the four sides of the current grab bucket enclosure, calculate the distance from each side to the nearest point in the bulkhead point cloud indexed by the first KD tree, and take the minimum distance as the candidate horizontal distance; or, based on the projection of the current grab bucket enclosure onto the horizontal plane, calculate the nearest distance on the horizontal plane to the bulkhead point cloud, and take it as the candidate horizontal distance. Calculate the distance from the bottom plane of the current grab enclosure to the nearest point in the bilge point cloud indexed by the second KD tree, and use it as a candidate vertical distance; Multiple candidate horizontal and vertical distances obtained in consecutive calculation cycles are subjected to moving average filtering to obtain smooth and stable horizontal and vertical safe distances.

[0082] In one embodiment, when the processor 1001 determines whether to generate a warning message and an emergency stop command based on the comparison results of whether the horizontal safety distance and the vertical safety distance exceed preset thresholds, it specifically performs the following operations: Obtain a first warning distance threshold, a second warning distance threshold, and an emergency stop distance threshold; the first warning distance threshold is greater than the second warning distance threshold, and the second warning distance threshold is greater than or equal to the emergency stop distance threshold. Determine whether the horizontal and / or vertical safety distances are less than or equal to the first warning distance threshold; If the horizontal safety distance and / or vertical safety distance are less than or equal to the first warning distance threshold and greater than the second warning distance threshold, a first-level warning is triggered. The first-level warning includes generating a first type of prompt information on the human-computer interaction interface. If the horizontal safety distance and / or vertical safety distance are less than or equal to the second warning distance threshold and greater than the emergency stop distance threshold, a second-level warning is triggered. The second-level warning includes generating a second type of prompt information, and the warning intensity of the second type of prompt information is higher than that of the first type of prompt information. If the horizontal and / or vertical safety distances are less than or equal to the emergency stop distance threshold, a Level 3 warning is triggered. The Level 3 warning includes generating a control signal, which is used to control the excavator to perform an emergency stop operation.

[0083] In this embodiment, high-precision point cloud data is automatically collected by a laser radar with a preset frequency, and a three-dimensional point cloud map is constructed and updated based on this data. The precise positions of the bulkhead, hull, and grab bucket are automatically identified from the map in a systematic way. This allows for the real-time and automatic calculation of the horizontal and vertical safe distances between the grab bucket and the hull structure. Based on this objective measurement result rather than subjective judgment, a decision is made on whether to generate a warning or emergency stop command. This fundamentally replaces unreliable human visual judgment and realizes a shift from passive reaction to proactive prevention, greatly improving the safety and reliability of excavators operating in complex cabins.

[0084] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program for grab safety protection based on lidar point clouds can be stored in a computer-readable storage medium. When executed, the program can include the processes of the embodiments of the above methods. The storage medium for the grab safety protection program based on lidar point clouds can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.

[0085] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A grab bucket safety protection method based on lidar point clouds, characterized in that, Applied to excavators, the method includes: The current frame point cloud data inside the ship's cabin is collected by a lidar pre-installed on the excavator at a preset frequency. Based on the current frame point cloud data, construct and update the latest three-dimensional point cloud map of the cabin; Based on the latest 3D point cloud map, the bulkhead and its bulkhead point cloud, the hull and its hull point cloud, and the grab bucket and its grab bucket point cloud were identified. Based on the bulkhead point cloud, the hull point cloud, and the grab bucket point cloud, the horizontal safe distance between the grab bucket and the bulkhead, and the vertical safe distance between the grab bucket and the hull are calculated in real time. Based on the comparison results of whether the horizontal safety distance and the vertical safety distance exceed the preset threshold, it is determined whether to generate a warning message and an emergency stop command.

2. The method according to claim 1, characterized in that, The step of constructing and updating the latest 3D point cloud map of the ship's cabin based on the current frame point cloud data includes: The current frame point cloud data is downsampled and filtered for noise reduction to obtain denoised point cloud data; The point cloud data of each point in the denoised point cloud data is uniformly transformed from the sensor coordinate system of the lidar itself to the excavator body coordinate system with the excavator's slewing center as the origin, to obtain the preprocessed point cloud data of the current frame. If the previous frame of historical preprocessed point cloud data exists, obtain the historical three-dimensional point cloud map of the cabin; The current frame preprocessed point cloud data is registered with the previous frame historical preprocessed point cloud data to obtain the registered current frame point cloud. The registered current frame point cloud is fused into the historical 3D point cloud map in the form of a voxel mesh to obtain the latest 3D point cloud map corresponding to the registered current frame point cloud, which serves as the latest 3D point cloud map of the cabin.

3. The method according to claim 2, characterized in that, The step of performing point cloud registration between the current frame preprocessed point cloud data and the previous frame historical preprocessed point cloud data to obtain the registered current frame point cloud includes: Based on the historical preprocessed point cloud data of the previous frame, a probability distribution model is constructed; By maximizing the probability density of the current frame preprocessed point cloud data under the probability distribution model, the first transformation matrix of the current frame preprocessed point cloud data relative to the previous frame historical preprocessed point cloud data is solved to complete coarse registration. Using the first transformation matrix as the initial transformation, the iterative nearest point algorithm is used to perform fine registration between the current frame preprocessed point cloud data and the previous frame historical preprocessed point cloud data, and the optimal rigid body transformation is calculated by minimizing the distance between corresponding point pairs to obtain the second transformation matrix. Using the second transformation matrix, spatial transformation is performed on the preprocessed point cloud data of the current frame to obtain a registration point cloud; Calculate the registration error. If the registration error is less than a preset error threshold, use the registered point cloud as the current frame point cloud after registration.

4. The method according to claim 1, characterized in that, The process of identifying the bulkhead and its point cloud, the hull and its point cloud, and the grab bucket and its point cloud based on the latest 3D point cloud map includes: From the latest 3D point cloud map, fit and separate the static structural point cloud set of the plane; From the latest 3D point cloud map, the static structure point cloud set is removed to obtain the remaining point cloud set; From the remaining point cloud set, a non-planar dynamic structure point cloud set is segmented; Extract the geometric features of the static structural point cloud set and the dynamic structural point cloud set; The geometric features are input into a pre-trained machine learning classifier, which outputs the semantic category of each point cloud set. Based on the semantic categories, the bulkhead and its bulkhead point cloud, the hull and its hull point cloud, and the grab bucket and its grab bucket point cloud are identified.

5. The method according to claim 4, characterized in that, The fitted and separated set of static structural point clouds representing the bulkhead and floor planes includes: From the latest 3D point cloud map, three points are randomly selected to fit a candidate planar model; Calculate the vertical distance from all points in the latest 3D point cloud map to the candidate plane model, and determine points whose distance is less than a preset distance threshold as interior points; Record the number of interior points corresponding to the current candidate plane model; Continue executing the step of randomly selecting three points from the latest 3D point cloud map to fit a candidate plane model until the number of selections reaches a preset threshold. Then, take the candidate plane with the most interior points as the valid plane and remove the set of interior points of the valid plane from the latest 3D point cloud map and add it to the static structure point cloud candidate set. Continue executing the step of randomly selecting three points from the latest 3D point cloud map to fit a candidate planar model until a planar model that meets the minimum number of interior points cannot be segmented from the remaining point cloud, generating a static structural point cloud set representing the bulkhead and bottom plane.

6. The method according to claim 4, characterized in that, The step of segmenting a non-planar dynamic structure point cloud set from the remaining point cloud set includes: A spatial data structure is constructed for the remaining point cloud set to accelerate nearest neighbor search; Select an unvisited point from the remaining point cloud set as a seed point; In the spatial data structure, search for all points in the neighborhood of the seed point whose spatial Euclidean distance to the seed point is less than a preset clustering distance threshold; Add the searched points to the current cluster and mark them as visited; Select a new point from the current cluster that has not been used as a seed point, and continue to execute the step of selecting an unvisited point from the remaining point cloud set as a seed point, until there are no points in the current cluster that have not been used as seed points, and obtain the segmentation result of the point cloud clustering; Continue executing the step of constructing a spatial data structure for the remaining point cloud set until all points in the remaining point cloud set have been visited, and obtain the segmentation results of multiple point cloud clusters; The segmentation results of the multiple point cloud clusters are discarded if the number of points is less than the preset minimum number of cluster points, and the segmentation results with the number of points greater than the preset maximum number of cluster points are split or discarded to obtain the non-planar dynamic structure point cloud set.

7. The method according to claim 1, characterized in that, The step of calculating in real time the horizontal safety distance between the grab and the bulkhead and the vertical safety distance between the grab and the bulkhead based on the bulkhead point cloud, the hull point cloud, and the grab point cloud includes: Geometric features, shape features, statistical features, and projection features are extracted from the point cloud of the grab bucket and used as feature vectors. The feature vector is input into a pre-trained support vector machine classifier to obtain the recognition result of whether the grab bucket point cloud is a grab bucket; Acquire images of the ship's cabin captured by a camera pre-installed on the excavator; Based on the cabin image, visual verification is performed using a preset YOLO target detection algorithm to obtain the verification result. When both the identification results and the verification results indicate that the grab point cloud is a grab, the Kalman filter algorithm is used to continuously track and smooth the state of the grab to obtain the optimal bounding box parameters of the grab. Based on the optimal bounding box parameters of the grab, determine the current grab bounding box used for distance calculation; Based on the bulkhead point cloud, the hull point cloud, and the current grab bucket surround box, calculate the horizontal safety distance between the grab bucket and the bulkhead, and the vertical safety distance between the grab bucket and the hull.

8. The method according to claim 7, characterized in that, The step of calculating the horizontal safety distance between the grab and the bulkhead and the vertical safety distance between the grab and the hull based on the bulkhead point cloud, the hull point cloud, and the current grab surround box includes: For the bulkhead point cloud and the hull point cloud, respectively, a first KD tree and a second KD tree are constructed; Based on the four sides of the current grab enclosure, calculate the distance from each side to the nearest point in the bulkhead point cloud indexed by the first KD tree, and take the minimum distance as the candidate horizontal distance; or, based on the projection of the current grab enclosure onto the horizontal plane, calculate the nearest distance on the horizontal plane to the bulkhead point cloud, and take it as the candidate horizontal distance. Calculate the distance from the bottom plane of the current grab enclosure to the nearest point in the bilge point cloud indexed by the second KD tree, and use it as a candidate vertical distance; The candidate horizontal distances and candidate vertical distances obtained in multiple consecutive calculation cycles are subjected to moving average filtering to obtain smooth and stable horizontal and vertical safe distances.

9. The method according to claim 1, characterized in that, The process of determining whether to generate a warning message and an emergency stop command based on a comparison of whether the horizontal safety distance and the vertical safety distance exceed a preset threshold includes: Obtain a first warning distance threshold, a second warning distance threshold, and an emergency stop distance threshold; the first warning distance threshold is greater than the second warning distance threshold, and the second warning distance threshold is greater than or equal to the emergency stop distance threshold. Determine whether the horizontal safety distance and / or the vertical safety distance are less than or equal to the first warning distance threshold; If the horizontal safety distance and / or the vertical safety distance is less than or equal to the first warning distance threshold and greater than the second warning distance threshold, a first-level warning is triggered, wherein the first-level warning includes generating a first type of prompt information on the human-computer interaction interface; If the horizontal safety distance and / or the vertical safety distance is less than or equal to the second warning distance threshold and greater than the emergency stop distance threshold, a secondary warning is triggered. The secondary warning includes generating a second type of prompt information, the warning intensity of which is higher than that of the first type of prompt information. If the horizontal safety distance and / or the vertical safety distance is less than or equal to the emergency stop distance threshold, a level 3 warning is triggered. The level 3 warning includes generating a control signal, which is used to control the excavator to perform an emergency stop operation.

10. A grab bucket safety protection device based on lidar point clouds, characterized in that, The device includes: The point cloud data acquisition module is used to acquire the current frame point cloud data inside the ship's cabin at a preset frequency using a lidar pre-installed on the excavator. The 3D point cloud map construction module is used to construct and update the latest 3D point cloud map of the cabin based on the current frame point cloud data. The environment recognition module is used to identify the bulkhead and its bulkhead point cloud, the hull and its hull point cloud, and the grab bucket and its grab bucket point cloud based on the latest three-dimensional point cloud map. The distance calculation module is used to calculate the horizontal safe distance between the grab and the bulkhead and the vertical safe distance between the grab and the bulkhead in real time based on the bulkhead point cloud, the hull point cloud, and the grab point cloud. The early warning module is used to determine whether to generate early warning information and emergency stop command based on the comparison results of whether the horizontal safety distance and the vertical safety distance exceed a preset threshold.

Images