A method and device for controlling collaborative operation of ship unloaders

Abstract

This application discloses a collaborative operation control method and device for ship unloaders, relating to the field of port ship unloading technology. The method includes acquiring ship unloader position data collected by a positioning component, ship tilt data collected by an attitude sensing component, and complete point cloud data preprocessed by an edge computing module. The original point cloud data corresponding to the complete point cloud data is obtained by scanning the material pile inside the hold using a lidar component. After identifying the hatch spatial coordinates using a target detection algorithm on the complete point cloud data, the hold point cloud and the material pile point cloud are segmented based on the hatch spatial coordinates. The material pile point cloud is then reconstructed in three dimensions to generate a triangular mesh model. Based on the material pile distribution density obtained from the triangular mesh model, different operating areas are divided. Combining the ship unloader position data and the ship tilt data optimized by Kalman filtering, a multi-dimensional anti-collision strategy is established, and the operating areas of each ship unloader are dynamically adjusted to effectively prevent collisions and significantly improve collaborative operation efficiency.

Description

Technical Field

[0001] This application relates to the field of port unloading operation technology, and in particular to a method and device for controlling the coordinated operation of unloading machines. Background Technology

[0002] In port unloading operations, most terminals still rely on manual operation, where the unloader operator, located in the high-altitude cab, directly observes the distribution of the material pile inside the hold and manually operates the control levers to control the movement of the unloader trolley, the pitching of the horizontal boom, the swinging of the vertical boom, and the opening and closing of the grab bucket.

[0003] The entire operation is highly dependent on the driver's experience. After long periods of continuous operation, increased driver fatigue leads to decreased operational precision and significant fluctuations in the amount of material handled per operation. Furthermore, multiple ship unloaders avoid conflicts by pre-setting fixed operating areas. Even if a material pile in one area has been cleared, it is necessary to wait for other areas to complete their operations before crossing over to another area, resulting in low efficiency. Summary of the Invention

[0004] In view of the above-mentioned defects or deficiencies in the related technologies, it is desirable to provide a method and device for controlling the collaborative operation of ship unloaders, which can dynamically and accurately schedule each ship unloader, prevent collisions, and improve the efficiency of collaborative operation.

[0005] In a first aspect, this application provides a collaborative operation control method for ship unloaders, wherein each ship unloader is equipped with a lidar component, a positioning component, an attitude sensing component, and an edge computing module, and the collaborative operation control method for ship unloaders includes: The system acquires the unloader position data collected by the positioning component, the ship tilt data collected by the attitude sensing component, and the complete point cloud data preprocessed by the edge computing module. The original point cloud data corresponding to the complete point cloud data is obtained by the lidar component scanning the material pile inside the cabin. After identifying the hatch spatial coordinates using a target detection algorithm on the complete point cloud data, the ship's cabin point cloud and the material pile point cloud are segmented based on the hatch spatial coordinates, and the material pile point cloud is reconstructed in three dimensions to generate a material pile triangular mesh model. Based on the material pile distribution density obtained from the triangular mesh model, different operating areas are divided. Combined with the unloader position data and the ship tilt data optimized by Kalman filtering, a multi-dimensional anti-collision strategy is established, and the operating areas of each unloader are dynamically adjusted.

[0006] Optionally, in some embodiments of this application, the segmentation of the ship's hold point cloud and the stockpile point cloud based on the hatch spatial coordinates includes: Within the area enclosed by the hatch spatial coordinates, a temporary plane is obtained by randomly sampling three deck surface points each time according to the number of iterations and fitting the data. The equation of the temporary plane is: , Indicates the direction along the coast. Indicates the direction perpendicular to the shoreline. Indicate the vertical upward direction and calculate the normal vector of the temporary plane; Calculate the normal vector of the temporary plane respectively. - flat, - The projection slope on the plane is determined as follows: when both projection slopes meet the preset slope threshold, the temporary plane is determined as a valid plane. Then, the distance from each point in the complete point cloud data to the valid plane is calculated sequentially, and the total number of inliers whose distance is not greater than the preset distance threshold is counted. Compare the total number of interior points recorded in all iterations, and select the effective plane with the largest total number of interior points as the boundary plane to segment the cabin point cloud and the material pile point cloud.

[0007] Optionally, the number of iterations described in some embodiments of this application , Indicates the confidence level. This indicates the preset threshold for the percentage of interior points. This represents the number of random sampling points.

[0008] Optionally, in some embodiments of this application, the three-dimensional reconstruction of the point cloud of the material pile to generate a triangular mesh model of the material pile includes: Along the vertical upward direction, the point cloud of the material pile is divided into multiple horizontal slice layers according to a preset thickness, and the two-dimensional contour is extracted from the point cloud in each horizontal slice layer to obtain the two-dimensional boundary point set of each horizontal slice layer. After stacking and combining the two-dimensional boundary point sets of each horizontal slice layer, the Delaunay triangulation algorithm is used to generate the material pile triangular mesh model.

[0009] Optionally, in some embodiments of this application, the extraction of two-dimensional contours from the point cloud within each horizontal slice layer to obtain a two-dimensional boundary point set for each horizontal slice layer includes: Traverse the two-dimensional points within each horizontal slice layer, search for the neighborhood point set of the two-dimensional point within a preset range, and select any intermediate point in the neighborhood point set. Then, calculate the coordinates of two center points based on the coordinates of the two-dimensional point, the coordinates of the intermediate point, and the preset radius corresponding to the preset range. The preset radius is half of the preset range. If all remaining points in the neighborhood point set, excluding the intermediate point, are located at distances greater than the two center coordinates of the circle, then the two-dimensional point is determined to be a constituent element of the two-dimensional boundary point set of each horizontal slice layer.

[0010] Optionally, in some embodiments of this application, the coordinates of the two center points are... and ; ; ; in, , , Indicates the preset radius. Represents the coordinates of a two-dimensional point. This represents the coordinates of the midpoint.

[0011] Optionally, the preprocessing described in some embodiments of this application includes filtering and completion processing; During the filtering process, the filtering weights of each point in the original point cloud data are calculated sequentially, and the coordinates of each point in the original point cloud data are updated based on the filtering weights. In addition, the point cloud space corresponding to the original point cloud data is divided into a three-dimensional grid according to the preset voxel size, and the average coordinates of all points in each three-dimensional grid are calculated. The average coordinates are retained as the voxel representative point, and other points are deleted to obtain sparse point cloud data. During the completion process, the sparse point cloud data is input into a pre-trained point cloud repair network to fill holes and generate the complete point cloud data. The encoder of the point cloud repair network introduces a local feature integration module, which includes a local spatial coding unit and an attention pooling unit. The decoder of the point cloud repair network generates a tiled point cloud through three fully connected layers.

[0012] Optionally, in some embodiments of this application, the filtering weight of each point in the original point cloud data is the normalized product of the spatial kernel function and the grayscale kernel function; The space kernel function , Represents the coordinates of the current point. Represents the coordinates of neighboring points, 1≤ ≤30, Indicates spatial standard deviation; The grayscale kernel function , This represents the grayscale value of the current point. Represents the gray value of neighboring points. This represents the standard deviation of gray levels.

[0013] Optionally, the dynamic adjustment of the operating area of ​​each ship unloader described in some embodiments of this application includes: The different operating areas are sorted from high to low according to the material pile distribution density. The operating area with the highest ranking is extracted and assigned to the ship unloader with the lowest current load. The operating areas of adjacent ship unloaders are checked. If the overlap width exceeds the preset overlap threshold, the area boundary is adjusted.

[0014] Secondly, this application provides a collaborative operation control device for ship unloaders, wherein each ship unloader is equipped with a lidar component, a positioning component, an attitude sensing component, and an edge computing module. The collaborative operation control device for ship unloaders includes an acquisition module, a three-dimensional reconstruction module, and a dynamic adjustment module that are interconnected. The acquisition module is used to acquire the unloader position data collected by the positioning component, the ship tilt data collected by the attitude sensing component, and the complete point cloud data preprocessed by the edge computing module. The original point cloud data corresponding to the complete point cloud data is obtained by the lidar component scanning the material pile inside the cabin. The three-dimensional reconstruction module is used to identify the hatch spatial coordinates of the complete point cloud data using a target detection algorithm, then segment the ship cabin point cloud and the material pile point cloud based on the hatch spatial coordinates, and perform three-dimensional reconstruction of the material pile point cloud to generate a material pile triangular mesh model. The dynamic adjustment module is used to divide different working areas based on the material pile distribution density obtained from the material pile triangular mesh model, and after combining the unloader position data and the ship tilt data optimized by Kalman filtering, establish a multi-dimensional anti-collision strategy and dynamically adjust the working area of ​​each unloader.

[0015] As can be seen from the above technical solutions, the present invention has the following advantages: This invention provides a collaborative operation control method and device for ship unloaders. The method utilizes lidar components deployed on each ship unloader to automatically scan the material pile inside the hold, positioning components to automatically collect ship unloader position data, attitude sensing components to automatically collect ship tilt data, and an edge computing module to preprocess the raw point cloud data into complete point cloud data. This process requires no manual intervention, ensuring accuracy and efficiency. Based on this, the material pile point cloud is segmented from the complete point cloud data and reconstructed in three dimensions to generate a triangular mesh model of the material pile. Then, based on the material pile distribution density obtained from the triangular mesh model, different operating areas are divided. Finally, by combining the ship unloader position data and the ship tilt data optimized by Kalman filtering, a multi-dimensional anti-collision strategy is established, and the operating areas of each ship unloader are dynamically adjusted to effectively prevent collisions and significantly improve collaborative operation efficiency. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a collaborative operation control method for a ship unloader provided in this application embodiment; Figure 2 A schematic diagram of a control process provided in an embodiment of this application; Figure 3 A structural block diagram of a ship unloader collaborative operation control device provided in this application embodiment; Figure 4 This is a structural block diagram of a terminal device provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. The following examples illustrate this. Figures 1 to 4 The present application describes in detail the ship unloader collaborative operation control method and device provided in the embodiments.

[0021] Please refer to Figure 1 This is a flowchart illustrating a collaborative operation control method for a ship unloader provided in this application embodiment. The method specifically includes the following steps: S101, acquire the unloader position data collected by the positioning component, the ship tilt data collected by the attitude perception component, and the complete point cloud data preprocessed by the edge computing module. The original point cloud data corresponding to the complete point cloud data is obtained by the lidar component scanning the material pile inside the cabin.

[0022] It should be noted that each ship unloader is equipped with lidar components, positioning components, attitude sensing components, and edge computing modules. Taking the coordinated coal unloading operation of the bridge grab ship unloader as an example, the grab bucket volume is 24.4m³. 3The horizontal cross-section is 4.2m × 3.5m. A hybrid solid-state lidar assembly is installed 1.5m below the front end of the horizontal arm of each ship unloader. The laser wavelength is 1550nm, with a horizontal field of view of 120° and a vertical field of view of 25°. The ranging range is 0.3m to 500m, the ranging accuracy is ±2cm, and the sampling frequency is 50Hz. An absolute encoder is installed at the center of the driven wheel of the ship unloader's trolley, and a Beidou RTK (real-time navigation system) is installed on the maintenance platform at the front of the horizontal arm. The kinematic (real-time dynamic) mobile station, with its base station located on the roof of the dock's waiting building, boasts a positioning accuracy of ±3cm. The positioning components include an absolute encoder and a BeiDou RTK mobile station, enabling real-time acquisition of the unloader's trolley position, horizontal boom pitch angle, vertical boom swing angle, and cantilever rotation angle. It is further aided by a lidar component to ensure the scanning range consistently covers the hatch area. A high-precision inclinometer, with a sampling frequency of 50Hz, is installed on the front side of the maintenance platform atop the vertical boom, serving as an attitude sensing component. This inclinometer collects real-time tilt data of the hull relative to the horizontal plane. Ultrasonic rangefinders are also installed on both sides of the grab bucket to monitor the distance between the grab bucket and the bulkhead. Finally, the edge computing module features an 8-core CPU and a 1024-core GPU, running the JetPack system and pre-installed with Ubuntu 18.04 and a robot operating system.

[0023] In practical use, combined with Figure 2 As shown, each ship unloader is equipped with nine high-definition cameras, including three 4-megapixel PTZ cameras and six 3-megapixel infrared bullet cameras, covering the areas of trolley travel, boom tilt, and grab bucket operation. Video data is stored on a hard disk recorder and then transmitted to the remote control room. The ship unloader and the control room communicate via dual-core single-mode fiber optic cable, with control and video signals transmitted separately over a 1000Mbps fiber optic bandwidth. The control room is equipped with two industrial-grade host computers and one 3D visualization workstation. The host computers use i7 series CPUs and 3080Ti series GPUs, running the ship unloader collaborative operation control method. The 3D visualization workstation runs the VTK Visualization Toolkit, which can display the triangular mesh model of the material pile inside the hopper, the ship unloader's attitude, and collision avoidance warning information in real time. Furthermore, a firewall is deployed between the coal conveying distributed control system (DCS) and the ship unloader control system in the control room. This setup prevents unauthorized access and ensures network security.

[0024] Furthermore, after the lidar component is activated, it scans the material pile inside the cabin with a 120° horizontal field of view and a sampling frequency of 50Hz, outputting 600,000 raw point cloud data per frame. The data includes the three-dimensional coordinates and grayscale values ​​of the points, and is transmitted to the edge computing module in real time through the robot operating system. The absolute encoder of the positioning component outputs the trolley's travel distance every 10ms with a resolution of 0.1mm, the Beidou RTK mobile station outputs the three-dimensional coordinates of the horizontal arm's front end (WGS84 coordinate system) every 50ms, and the high-precision inclinometer of the attitude perception component outputs the vertical arm's swing angle and the hull's tilt angle every 20ms with a sampling frequency of 50Hz. When the hull tilt angle exceeds 0.5°, an attitude anomaly warning is triggered, and the positioning data sampling frequency is synchronously increased to 100Hz. All positioning data is transmitted to the edge computing module via the Modbus-TCP protocol and then uniformly converted into a global coordinate system with the dock shore as the origin. Indicates the direction along the coast. Indicates the direction perpendicular to the shoreline. It indicates the vertical upward direction.

[0025] The preprocessing of the edge computing module includes, but is not limited to, filtering and completion processing. During filtering, the filtering weights of each point in the original point cloud data are calculated sequentially. Based on these weights, the coordinates of each point in the original point cloud data are updated to remove noise. Additionally, according to a preset voxel size, the point cloud space corresponding to the original point cloud data is divided into a 3D grid. The average coordinates of all points within each 3D grid are calculated, and the average coordinates are retained as the representative voxel point. Other points are deleted to obtain sparse point cloud data, completing downsampling. The filtering weights of each point in the original point cloud data are the normalized product of the spatial kernel function and the grayscale kernel function. Gray-scale kernel function , Represents the coordinates of the current point. Represents the coordinates of neighboring points, 1≤ ≤30, Indicates spatial standard deviation. This represents the grayscale value of the current point. Represents the gray value of neighboring points. This represents the standard deviation of grayscale. For example, setting the spatial standard deviation. =0.5m, this value is determined based on the surface roughness characteristics of bulk cargo terminal stockpiles, which can effectively preserve the surface details of the stockpiles; grayscale standard deviation =20, this value is determined by combining the grayscale value range of the point cloud collected by the lidar component. It can balance the sensitivity and anti-interference ability of grayscale similarity judgment, and avoid misjudging normal grayscale differences in different areas of the material pile as noise. The grayscale value range is 0 to 255; for the current point Search its corresponding 30 neighborhood points And calculate the filter weights. Then the updated coordinates of the current point c This removes coal dust noise points (points where grayscale values ​​change abruptly). For example, setting the voxel size to 0.05m × 0.05m × 0.05m takes into account both the accuracy requirements and computational efficiency of subsequent 3D modeling. This significantly reduces the amount of point cloud data, decreasing it to 1 / 8 to 1 / 10 of the original amount, while ensuring a high level of detail retention in the material pile, thus greatly improving processing efficiency.

[0026] During the completion process, sparse point cloud data is input into a pre-trained point cloud repair network to fill holes and generate complete point cloud data. The encoder of the point cloud repair network incorporates a Local Feature Aggregation (LFA) module, which includes a Local Spatial Encoding Unit (LSI) and an Attention Pooling Unit (APU). The decoder of the point cloud repair network generates a tiled point cloud through three fully connected layers. The LSI uses a 3×3×3 three-dimensional convolutional kernel with a stride of 1 and a "same" padding method to encode neighborhood point features. Its receptive field covers 0.3m×0.3m×0.3m, effectively capturing the local geometry of the material pile. The APU employs a multi-head self-attention mechanism with four heads, each with a feature dimension of 64. By calculating the similarity weights between the neighborhood points and the center point's point cloud features, it strengthens important features and suppresses redundant features. , Point With point Cosine similarity of features Point The set of neighborhood points, Point With point Cosine similarity of features. The number of fully connected layers and neuron configuration in the decoder need to be determined based on the feature mapping and coordinate output requirements. For example, the first fully connected layer contains 1024 neurons, which can map the 256-dimensional global features output by the encoder to high-dimensional features. The activation function is ReLU. The second fully connected layer contains 512 neurons. To further refine the features, the activation function is LeakyReLU, and the negative slope is set to 0.01 to enhance the network's ability to fit sparse features. The third fully connected layer contains 3×2048 neurons, outputting the 3D coordinates of 2048 completion points. There is no activation function, ensuring the continuity and rationality of the coordinate values. The loss function adopts the improved Chamfer distance loss, which is achieved by setting weight parameters. To avoid the point cloud deviating from the original distribution, the calculation formula is as follows: In this formula This represents the original sparse point cloud. It is a subset of the original sparse point cloud. This indicates the point cloud after completion. To complete the subset of the point cloud, Set to 1.2, by increasing arrive The distance weighting helps prevent the completed point cloud from deviating excessively from the original point cloud distribution. When training the point cloud repair network, a dataset containing 5000 ship cabin and material pile models was used. This dataset was constructed by mixing data synthesized using 3D modeling software with actual on-site scanning data. It includes samples with different hatch shapes, material pile morphologies, and different degrees of occlusion (such as grab bucket occlusion and bulkhead occlusion) to enhance the model's generalization ability. The batch size was set to 32, the initial learning rate was 0.001, and the Adam optimizer was used. The network was trained for 200 epochs with Chamfer distance as the loss function until the loss value converged to below 0.02. Furthermore, when deploying the point cloud repair network, TensorRT was used to accelerate the model, and for cases such as 5cm × 5cm holes formed by material pile occlusion, each frame of point cloud could be completed within 200ms.

[0027] S102, after identifying the hatch spatial coordinates using a target detection algorithm on the complete point cloud data, the ship's hold point cloud and the material pile point cloud are segmented based on the hatch spatial coordinates, and the material pile point cloud is reconstructed in three dimensions to generate a material pile triangular mesh model.

[0028] It should be noted that during the initialization phase, the unloader is adjusted to the preset operating starting point by the positioning component to ensure that the scanning range of the lidar component covers the hatch. If it still does not cover the hatch after multiple attempts, the horizontal boom pitch angle or the trolley position is adjusted to rescan.

[0029] In some embodiments of this application, the target detection algorithm may employ the YOLOv8 algorithm, and a pre-trained target detection model may be used to output the hatch spatial coordinates. It is understood that those skilled in the art may also choose other types of deep learning target detection networks, such as Faster R-CNN or SSD (Single Shot Multibox Detector), depending on actual accuracy and speed requirements, as long as the hatch spatial coordinates can be identified. If the hatch region in the point cloud data is incomplete, the hatch region point cloud can be completed and coordinates calculated by combining the prior model of the ship's structure and the unloader position data provided by the positioning component. The YOLOv8 object detection model consists of a backbone, a neck, and a head. The backbone contains five CSP modules, each consisting of residual blocks and two convolutional layers. The number of residual blocks is 1, 2, 8, 8, and 4, respectively, and the convolutional kernel size is 3×3. The stride is set to 1 or 2 depending on the downsampling requirements. The CSP structure reduces computation while preserving gradient information. The neck uses top-down and bottom-up feature fusion methods, with fused feature map scales of 13×13, 26×26, and 52×52, respectively. The number of channels is adjusted to 256 using 1×1 convolutions, and then feature redundancy is eliminated after fusion using 3×3 convolutions. The head uses an anchor-free mode, predicting one bounding box and one class confidence for each feature point. The bounding box regression uses CIoU loss, and the class prediction uses cross-entropy loss. During model training, a hybrid dataset and a specific scheduling strategy are employed. The dataset is constructed by mixing original and synthetic images, which can simulate different lighting and occlusion scenarios. The dataset is expanded to 8000 images. Cosine annealing learning rate scheduling is used, with an initial learning rate of 0.01, which is reduced to 0.001 after 50 iterations and further reduced to 0.0001 after 100 iterations to ensure stable convergence of the model in later stages. Label smoothing technology is also used, with a smoothing coefficient set to 0.1, thereby reducing the model's sensitivity to label errors.

[0030] Furthermore, when segmenting the hatch point cloud and the stockpile point cloud based on the hatch spatial coordinates, firstly, within the area enclosed by the hatch spatial coordinates, three deck surface points are randomly sampled each time according to the number of iterations. A temporary plane is obtained by fitting coordinates close to the hatch height (i.e., within a preset coordinate range). The equation of this temporary plane is: , Indicates the direction along the coast. Indicates the direction perpendicular to the shoreline. Indicates the vertical upward direction. , , and These are the intermediate calculation parameters, and the normal vector of the temporary plane is calculated, for example, the number of iterations. , Indicates the confidence level. This indicates the preset threshold for the percentage of interior points. Indicates the number of random sampling points. It can be 0.99. It can be 0.7. The number of iterations calculated by this formula can be 3. This represents the percentage of interior points in the hypothetical data. In the case where all sampling points in at least one iteration come from interior points, the probability reaches The minimum number of attempts required ensures a high success rate while avoiding infinite iteration or premature termination; then, the normal vector of the temporary plane is calculated respectively. - flat, - The projection slope on the plane is , When both projection slopes satisfy a preset slope threshold, for example... , All values ​​are less than 0.05. This preset slope threshold is determined based on the characteristic that the deck surface is usually close to the horizontal plane, which can greatly improve the accuracy of deck surface recognition and avoid missegmentation. After determining the temporary plane as the effective plane, the distance from each point in the complete point cloud data to the effective plane is calculated in turn, and the total number of interior points whose distance is not greater than the preset distance threshold is counted. For example, the preset distance threshold is 0.1m, which is 5 times the distance measurement accuracy, which can accommodate point cloud measurement error and avoid misjudging non-deck surface points as interior points. Then, the total number of interior points recorded in all iterations is compared, and the effective plane with the largest total number of interior points is selected as the boundary plane to segment the ship cabin point cloud and the material pile point cloud.

[0031] Furthermore, when performing 3D reconstruction of the point cloud of the material pile to generate a dynamically updated triangular mesh model of the material pile, along the vertically upward direction (i.e., (Direction), the point cloud of the material pile is divided into multiple horizontal slices according to a preset thickness, for example, a preset thickness of 0.05m, and the point cloud within each horizontal slice is... - Two-dimensional contour extraction is performed on the plane to obtain the two-dimensional boundary point set of each horizontal slice layer. Finally, the two-dimensional boundary point sets of each horizontal slice layer are stacked and combined, and the Delaunay triangulation algorithm is used to generate a material pile triangular mesh model. This material pile triangular mesh model is updated every 100ms and synchronously transmitted to the 3D display workstation in the central control room. Specifically, the point cloud within each horizontal slice layer is... - When extracting the 2D contour on a plane and obtaining the 2D boundary point set of each horizontal slice layer, the 2D points within each horizontal slice layer are traversed. The search is performed on the neighborhood point set of each 2D point within a preset range. After selecting any intermediate point from the neighborhood point set, the coordinates of two circle centers are calculated based on the coordinates of the 2D point, the coordinates of the intermediate point, and the preset radius corresponding to the preset range. The preset radius is half of the preset range. For example, the coordinates of the two circle centers are... and ,Right now: ; ; In the above formula, , , and These are intermediate calculation parameters. Indicates the preset radius. It can be 0.3m. Represents the coordinates of a two-dimensional point. This represents the coordinates of the intermediate point. If the distances from all remaining points in the neighborhood point set (excluding the intermediate point) to the coordinates of the two center points are greater than the preset radius, then the two-dimensional point is determined as a component element of the two-dimensional boundary point set of each horizontal slice layer.

[0032] S103, based on the material pile distribution density obtained from the triangular mesh model of the material pile, divides different working areas, and combines the unloader position data and the ship tilt data optimized by Kalman filtering to establish a multi-dimensional anti-collision strategy, and then dynamically adjusts the working area of ​​each unloader.

[0033] In some embodiments of this application, Kalman filtering can be used to optimize the processing of hull tilt angle data, thereby improving measurement accuracy. For example, the state vector is first initialized. Error covariance Process noise covariance and measurement noise covariance Secondly, state prediction, i.e., calculation , , They represent different times. These are intermediate calculation parameters. Here is the state transition matrix. for The state vector at time t, For the control matrix, for Timing control input, and calculation To update the prediction error covariance, These are intermediate calculation parameters. for The error covariance at time, Indicates transpose; measure and update again, i.e., calculate Kalman gain. , Update the state vector for the observation matrix. , For the measurement vector, and update the error covariance , The system uses an identity matrix; finally, it outputs optimized hull tilt angle data. This process is executed every 5ms, improving accuracy to ±0.05°. Since hull tilt is a slowly changing process, redundant states such as angular velocity are ignored, and the state vector only contains the hull tilt angle, simplifying model calculations. The process noise covariance is set to 0.001, determined based on the rate of change of the hull tilt angle, avoiding over-correction. The measurement noise covariance is set to 0.005, determined based on the inclinometer measurement error, balancing the weights of measured and predicted values. The filtering process is executed every 5ms, a cycle determined based on the frequency of the hull tilt angle change, satisfying the Nyquist sampling theorem, ensuring no signal aliasing, and avoiding the waste of computational resources caused by high-frequency filtering.

[0034] In some embodiments of this application, multi-dimensional collision avoidance strategies include, but are not limited to, hatch safety zone division, grab-bulls collision avoidance, grab-hull collision avoidance, and collision avoidance between unloaders, facilitating hierarchical control. When dividing the hatch safety zone, firstly, based on real-time hull tilt angle data optimized by Kalman filtering, tilt compensation is performed on the spatial coordinates of the four corner points of the identified hatch to obtain the projected boundary coordinates of the hatch on the horizontal reference plane under the current tilt state. Using the compensated projected boundary as a reference, a specific distance is offset outward to divide the area into a U-shape, i.e., the safety zone is 0.8m from the projected boundary, and the edge deceleration zone is 1.2m from the projected boundary. During grab-bulls collision avoidance, an ultrasonic rangefinder measures the first distance between the grab and the bulkhead every 100ms. The positioning component calculates the second distance between the two unloaders using coordinates. The first sub-filter processes the first distance to obtain relevant parameters, the second sub-filter processes the second distance to obtain relevant parameters, and then the main filter uses a federated Kalman filter architecture for data fusion. Assuming the optimal distance estimate output by the first sub-filter is... The estimated error covariance is The optimal distance estimate output by the second sub-filter is The estimated error covariance is Then the main filter passes through Calculate the final distance estimate after fusion. When the final distance estimate after fusion When the distance is less than 1.2m, a deceleration signal is sent to the unloader, causing the grab bucket's descent speed to decrease from 180m / min to 60m / min, ensuring sufficient time to decelerate to a stop, and the final distance estimate after fusion. When the distance is less than 0.8m, an emergency stop signal is triggered. This value takes into account the maximum swing amplitude of the grab bucket to ensure that there is still a safe distance between the grab bucket and the bulkhead after an emergency stop.

[0035] For grab-bottom collision avoidance, the height of the material pile can be obtained based on the triangular mesh model of the material pile, and the minimum safe landing height can be calculated. This minimum safe landing height = material pile height + grab bucket closing height + 0.3m. This value is the reserved grab bucket closing error and material pile settlement to avoid bottoming out. If the actual height of the grab bucket measured by the Beidou RTK mobile station is lower than this minimum safe landing height, an audible and visual alarm will be triggered in the central control room. Furthermore, for collision avoidance between ship unloaders, the coordinates of the front ends of the horizontal arms of the two ship unloaders are obtained through the Beidou RTK mobile station, and the minimum distance is calculated. If the distance is less than 2.5m, the operation path is adjusted.

[0036] Furthermore, when dynamically adjusting the operating areas of each ship unloader, different operating areas can be sorted from high to low according to the material pile distribution density. The operating area at the top of the sorted list is then extracted and assigned to the ship unloader with the lowest current load. The operating areas of adjacent ship unloaders are checked; if the overlap width exceeds a preset overlap threshold, the area boundaries are adjusted. For example, the area inside the hold can be divided into 1m×1m grid cells, and the number of point clouds in each grid cell can be calculated. Based on the material pile distribution density from high to low, high-density, medium-density, and low-density areas are defined, with the number of point clouds in the high-density areas not less than 50 per m. 3 The number of point clouds in the medium-density region is 20 / m. 3 Up to 49 per m 3 Between these, the number of point clouds in low-density areas is less than 20 / m. 3 For example, initialize the operating load of each ship unloader to 0, and calculate the real-time operating load. , Indicates real-time traffic. Indicates the rated flow rate. Indicates the duration of the work. This indicates the total estimated duration. The flow rate as a percentage of 0.6 is higher than the duration as a percentage of 0.4. This setting is based on the characteristic that flow rate directly reflects the current operational efficiency, which can prevent the unloader from being continuously overloaded due to high duration and low flow rate, and improve the load balancing effect. The first grid cell in the sorted order is extracted and assigned to the unloader with the lowest current load. After assignment, the working areas of adjacent unloaders are checked. If the overlap width exceeds 0.5m, the boundary is adjusted. The real-time operational load is recalculated every 10 seconds.

[0037] During operation, flow rate needs to be monitored in real time. When the flow rate of a certain ship unloader does not meet the requirements, the operation area is redistributed. Flow rate monitoring is specifically achieved using a 2D LiDAR, which is mounted on a fixed bracket approximately 1.5m directly above the conveyor belt of the ship unloader. The scanning plane is perpendicular to the belt's running direction, with a scanning frequency of 10Hz, a scanning angle of 180°, and a resolution of 0.1°. By scanning the cross-section of the belt, a total of 1801 scanning points are obtained. The distance data of each scanning point is converted into the height on the belt cross-section, and the cross-sectional area of ​​the material is calculated using trapezoidal integration. Combined with the coal density and the belt speed measured by the conveyor encoder, the real-time flow rate is calculated. To ensure accurate flow calculation, when the flow rate of a ship unloader falls below 70% of its rated value for 30 consecutive seconds, its operating area is reassigned to ensure balanced load across all unloaders. This timeout setting eliminates the impact of short-term flow fluctuations. The 70% rated flow rate is based on the minimum effective operating efficiency of the unloader; a flow rate below this value indicates that the material pile in the operating area is depleted or there is an equipment malfunction, requiring timely reassignment. During reassignment, high-density areas should be prioritized to avoid cross-area scheduling, and high-density areas should be allocated to the unloader with the lowest load. This reduces the travel distance of the unloader's trolley, lowers the proportion of non-operating time, and ensures rapid recovery of operating efficiency after reassignment. Operational results show that the hatch identification accuracy reaches 98.5%, the material pile distribution density calculation error is less than 5%, the anti-collision warning response time is less than 100ms, and no collisions occurred between the grab bucket and the bulkhead, the grab bucket and the bilge, or between the unloaders. The load balancing rate of the two ship unloaders does not exceed 15%, the operating efficiency has been improved from 50 seconds / cycle for manual operation to 35 seconds / cycle, the daily coal unloading volume has increased from 18,000 tons to 25,000 tons, and the labor intensity of workers has been reduced by 80%, which fully meets the needs of efficient, safe and unmanned ship unloading operations at bulk cargo terminals.

[0038] The collaborative operation control method for ship unloaders provided in this application automatically scans the material pile inside the hold using lidar components deployed on each ship unloader, automatically collects the ship unloader's position data using positioning components, automatically collects the ship's tilt data using attitude sensing components, and preprocesses the raw point cloud data into complete point cloud data using an edge computing module. This process requires no manual intervention, is accurate and efficient. Based on this, the material pile point cloud is segmented from the complete point cloud data, and a three-dimensional reconstruction of the material pile point cloud is performed to generate a triangular mesh model of the material pile. Then, based on the material pile distribution density obtained from the triangular mesh model, different operating areas are divided. Finally, by combining the ship unloader's position data and the ship's tilt data optimized by Kalman filtering, a multi-dimensional anti-collision strategy is established, and the operating areas of each ship unloader are dynamically adjusted to effectively prevent collisions and significantly improve the efficiency of collaborative operations.

[0039] Based on the foregoing embodiments, this application provides a ship unloader collaborative operation control device 100, which can be applied to... Figures 1 to 2 In the corresponding embodiment of the ship unloader collaborative operation control method, each ship unloader is equipped with a lidar component, a positioning component, an attitude sensing component, and an edge computing module. Please refer to... Figure 3 The ship unloader collaborative operation control device 100 includes: The acquisition module 101 is used to acquire the unloader position data collected by the positioning component, the ship tilt data collected by the attitude perception component, and the complete point cloud data preprocessed by the edge computing module. The original point cloud data corresponding to the complete point cloud data is obtained by the lidar component scanning the material pile inside the cabin. The 3D reconstruction module 102 is used to identify the hatch spatial coordinates of the complete point cloud data using a target detection algorithm, then segment the ship's cabin point cloud and the material pile point cloud based on the hatch spatial coordinates, and perform 3D reconstruction on the material pile point cloud to generate a material pile triangular mesh model. The dynamic adjustment module 103 is used to divide different working areas based on the material pile distribution density obtained from the triangular mesh model of the material pile, and after combining the unloader position data and the ship tilt data optimized by Kalman filtering, establish a multi-dimensional anti-collision strategy and dynamically adjust the working area of ​​each unloader.

[0040] Optionally, in some embodiments of this application, the three-dimensional reconstruction module 102 is specifically used to obtain a temporary plane by randomly sampling three deck surface points each time within the area enclosed by the hatch spatial coordinates, according to the number of iterations. The equation of the temporary plane is: , Indicates the direction along the coast. Indicates the direction perpendicular to the shoreline. Indicate the vertical upward direction and calculate the normal vector of the temporary plane; Calculate the normal vector of the temporary plane respectively. - flat, - The projection slope on the plane is determined as follows: when both projection slopes meet the preset slope threshold, the temporary plane is determined as the effective plane. Then, the distance from each point in the complete point cloud data to the effective plane is calculated in turn, and the total number of inliers whose distance is not greater than the preset distance threshold is counted. Compare the total number of interior points recorded in all iterations, and select the effective plane with the largest total number of interior points as the boundary plane to segment the point cloud of the ship's cabin and the point cloud of the material pile.

[0041] Optionally, the number of iterations in some embodiments of this application , Indicates the confidence level. This indicates the preset threshold for the percentage of interior points. This represents the number of random sampling points.

[0042] Optionally, in some embodiments of this application, the three-dimensional reconstruction module 102 is specifically used to divide the point cloud of the material pile into multiple horizontal slice layers according to a preset thickness along the vertical upward direction, and to extract the two-dimensional contour of the point cloud in each horizontal slice layer to obtain the two-dimensional boundary point set of each horizontal slice layer. After stacking and combining the two-dimensional boundary point sets of each horizontal slice layer, the Delaunay triangulation algorithm is used to generate the triangular mesh model of the material pile.

[0043] Optionally, in some embodiments of this application, the three-dimensional reconstruction module 102 is also specifically used to traverse the two-dimensional points in each horizontal slice layer, search for the neighborhood point set of the two-dimensional point within a preset range, and select any intermediate point in the neighborhood point set. Then, it calculates the coordinates of the two-dimensional point, the coordinates of the intermediate point, and the preset radius corresponding to the preset range. The preset radius is half of the preset range. If all remaining points in the neighborhood point set, excluding the middle point, are located at distances greater than the preset radii from the coordinates of the two center points, then the two-dimensional points are determined as constituent elements of the two-dimensional boundary point set of each horizontal slice layer.

[0044] Optionally, in some embodiments of this application, the coordinates of the two center points are... and ; ; ; in, , , Indicates the preset radius. Represents the coordinates of a two-dimensional point. This represents the coordinates of the midpoint.

[0045] Optionally, in some embodiments of this application, the preprocessing includes filtering and completion processing; During the filtering process, the filtering weights of each point in the original point cloud data are calculated sequentially, and the coordinates of each point in the original point cloud data are updated based on the filtering weights. In addition, the point cloud space corresponding to the original point cloud data is divided into a three-dimensional grid according to the preset voxel size, and the average coordinates of all points in each three-dimensional grid are calculated. The average coordinates are retained as the voxel representative point, and other points are deleted to obtain sparse point cloud data. During the completion process, sparse point cloud data is input into a pre-trained point cloud repair network to fill holes and generate complete point cloud data. The encoder of the point cloud repair network introduces a local feature integration module, which includes a local spatial coding unit and an attention pooling unit. The decoder of the point cloud repair network generates a tiled point cloud through three fully connected layers.

[0046] Optionally, in some embodiments of this application, the filtering weight of each point in the original point cloud data is the normalized product of the spatial kernel function and the grayscale kernel function; Spatial kernel function , Represents the coordinates of the current point. Represents the coordinates of neighboring points, 1≤ ≤30, Indicates spatial standard deviation; Gray-scale kernel function , This represents the grayscale value of the current point. Represents the gray value of neighboring points. This represents the standard deviation of gray levels.

[0047] Optionally, in some embodiments of this application, the dynamic adjustment module 103 is specifically used to sort different working areas from high to low according to the material pile distribution density, sequentially extract the working area at the top of the sort and assign it to the unloader with the lowest current load, and check the working areas of adjacent unloaders. If the overlap width exceeds a preset overlap threshold, the area boundary is adjusted.

[0048] It should be noted that the descriptions of the same steps and contents as in other embodiments in this embodiment can be found in the descriptions in other embodiments, and will not be repeated here.

[0049] The ship unloader collaborative operation control device provided in this application embodiment automatically scans the material pile inside the hold using lidar components deployed on each ship unloader, automatically collects the ship unloader's position data using positioning components, automatically collects the ship's tilt data using attitude sensing components, and preprocesses the raw point cloud data into complete point cloud data using an edge computing module. This process requires no manual intervention, is accurate and efficient. Based on this, the material pile point cloud is segmented from the complete point cloud data, and the material pile point cloud is reconstructed in three dimensions to generate a material pile triangular mesh model. Then, based on the material pile distribution density obtained from the material pile triangular mesh model, different operating areas are divided. Finally, by combining the ship unloader's position data and the ship's tilt data optimized by Kalman filtering, a multi-dimensional anti-collision strategy is established, and the operating areas of each ship unloader are dynamically adjusted to effectively prevent collisions and significantly improve the efficiency of collaborative operation.

[0050] Based on the foregoing embodiments, this application provides a terminal device. Please refer to... Figure 4 The terminal device 200 may include a processor 201 and a memory 202. The memory 202 stores at least one instruction, at least one program, code set, or instruction set, which is loaded and executed by the processor 201 to implement [the desired functionality]. Figures 1 to 2 The steps of the ship unloader cooperative operation control method in the corresponding embodiment.

[0051] In another aspect, embodiments of this application provide a computer-readable storage medium for storing program code for executing the aforementioned... Figures 1 to 2 Any implementation of the ship unloader collaborative operation control method in the corresponding embodiment.

[0052] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0053] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed between each other can be through some interfaces, indirect coupling or communication connection between devices or modules, and can be electrical, mechanical, or other forms. Modules described as separate components may or may not be physically separate, and components shown as modules may or may not be physical units, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0054] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more units can be integrated into one module. The integrated unit can be implemented in hardware or as a software functional unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0055] Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the ship unloader collaborative operation control method of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0056] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0057] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for controlling the coordinated operation of a ship unloader, characterized in that, Each ship unloader is equipped with a lidar module, a positioning module, an attitude sensing module, and an edge computing module. The collaborative operation control method for the ship unloaders includes: The system acquires the unloader position data collected by the positioning component, the ship tilt data collected by the attitude sensing component, and the complete point cloud data preprocessed by the edge computing module. The original point cloud data corresponding to the complete point cloud data is obtained by the lidar component scanning the material pile inside the cabin. After identifying the hatch spatial coordinates using a target detection algorithm on the complete point cloud data, the ship's cabin point cloud and the material pile point cloud are segmented based on the hatch spatial coordinates, and the material pile point cloud is reconstructed in three dimensions to generate a material pile triangular mesh model. Based on the material pile distribution density obtained from the triangular mesh model, different operating areas are divided. Combined with the unloader position data and the ship tilt data optimized by Kalman filtering, a multi-dimensional anti-collision strategy is established, and the operating areas of each unloader are dynamically adjusted.

2. The method for controlling the coordinated operation of a ship unloader according to claim 1, characterized in that, The segmentation of the ship's hold point cloud and the stockpile point cloud based on the hatch spatial coordinates includes: Within the area enclosed by the hatch spatial coordinates, a temporary plane is obtained by randomly sampling three deck surface points each time according to the number of iterations and fitting the data. The equation of the temporary plane is: , Indicates the direction along the coast. Indicates the direction perpendicular to the shoreline. Indicate the vertical upward direction and calculate the normal vector of the temporary plane; Calculate the normal vector of the temporary plane respectively. - flat, - The projection slope on the plane is determined as follows: when both projection slopes meet the preset slope threshold, the temporary plane is determined as a valid plane. Then, the distance from each point in the complete point cloud data to the valid plane is calculated sequentially, and the total number of inliers whose distance is not greater than the preset distance threshold is counted. Compare the total number of interior points recorded in all iterations, and select the effective plane with the largest total number of interior points as the boundary plane to segment the cabin point cloud and the material pile point cloud.

3. The ship unloader cooperative operation control method according to claim 2, characterized in that, The number of iterations , Indicates the confidence level. This indicates the preset threshold for the percentage of interior points. This represents the number of random sampling points.

4. The method for controlling the coordinated operation of a ship unloader according to claim 1, characterized in that, The step of performing three-dimensional reconstruction of the point cloud of the material pile to generate a triangular mesh model of the material pile includes: Along the vertical upward direction, the point cloud of the material pile is divided into multiple horizontal slice layers according to a preset thickness, and the two-dimensional contour is extracted from the point cloud in each horizontal slice layer to obtain the two-dimensional boundary point set of each horizontal slice layer. After stacking and combining the two-dimensional boundary point sets of each horizontal slice layer, the Delaunay triangulation algorithm is used to generate the material pile triangular mesh model.

5. The ship unloader cooperative operation control method according to claim 4, characterized in that, The step of extracting two-dimensional contours from the point cloud within each horizontal slice layer to obtain a two-dimensional boundary point set for each horizontal slice layer includes: Traverse the two-dimensional points within each horizontal slice layer, search for the neighborhood point set of the two-dimensional point within a preset range, and select any intermediate point in the neighborhood point set. Then, calculate the coordinates of two center points based on the coordinates of the two-dimensional point, the coordinates of the intermediate point, and the preset radius corresponding to the preset range. The preset radius is half of the preset range. If all remaining points in the neighborhood point set, excluding the intermediate point, are located at distances greater than the two center coordinates of the circle, then the two-dimensional point is determined to be a constituent element of the two-dimensional boundary point set of each horizontal slice layer.

6. The ship unloader cooperative operation control method according to claim 5, characterized in that, The coordinates of the two center points are: and ; ； ； in, , , Indicates the preset radius. Represents the coordinates of a two-dimensional point. This represents the coordinates of the midpoint.

7. The method for controlling the coordinated operation of a ship unloader according to any one of claims 1 to 6, characterized in that, The preprocessing includes filtering and completion processing; During the filtering process, the filtering weights of each point in the original point cloud data are calculated sequentially, and the coordinates of each point in the original point cloud data are updated based on the filtering weights. In addition, the point cloud space corresponding to the original point cloud data is divided into a three-dimensional grid according to the preset voxel size, and the average coordinates of all points in each three-dimensional grid are calculated. The average coordinates are retained as the voxel representative point, and other points are deleted to obtain sparse point cloud data. During the completion process, the sparse point cloud data is input into a pre-trained point cloud repair network to fill holes and generate the complete point cloud data. The encoder of the point cloud repair network introduces a local feature integration module, which includes a local spatial coding unit and an attention pooling unit. The decoder of the point cloud repair network generates a tiled point cloud through three fully connected layers.

8. The method for controlling the coordinated operation of a ship unloader according to claim 7, characterized in that, The filtering weight of each point in the original point cloud data is the normalized product of the spatial kernel function and the grayscale kernel function; The space kernel function , Represents the coordinates of the current point. Represents the coordinates of neighboring points, 1≤ ≤30, Indicates spatial standard deviation; The grayscale kernel function , This represents the grayscale value of the current point. Represents the gray value of neighboring points. This represents the standard deviation of gray levels.

9. The method for controlling the coordinated operation of a ship unloader according to any one of claims 1 to 6, characterized in that, The dynamic adjustment of the operating area of ​​each ship unloader includes: The different operating areas are sorted from high to low according to the material pile distribution density. The operating area with the highest ranking is extracted and assigned to the ship unloader with the lowest current load. The operating areas of adjacent ship unloaders are checked. If the overlap width exceeds the preset overlap threshold, the area boundary is adjusted.

10. A control device for coordinated operation of a ship unloader, characterized in that, Each ship unloader is equipped with a lidar component, a positioning component, an attitude sensing component, and an edge computing module. The ship unloader collaborative operation control device includes an acquisition module, a three-dimensional reconstruction module, and a dynamic adjustment module that are interconnected. The acquisition module is used to acquire the unloader position data collected by the positioning component, the ship tilt data collected by the attitude sensing component, and the complete point cloud data preprocessed by the edge computing module. The original point cloud data corresponding to the complete point cloud data is obtained by the lidar component scanning the material pile inside the cabin. The three-dimensional reconstruction module is used to identify the hatch spatial coordinates of the complete point cloud data using a target detection algorithm, then segment the ship cabin point cloud and the material pile point cloud based on the hatch spatial coordinates, and perform three-dimensional reconstruction of the material pile point cloud to generate a material pile triangular mesh model. The dynamic adjustment module is used to divide different working areas based on the material pile distribution density obtained from the material pile triangular mesh model, and after combining the unloader position data and the ship tilt data optimized by Kalman filtering, establish a multi-dimensional anti-collision strategy and dynamically adjust the working area of ​​each unloader.

Images