A multi-modal fusion anti-collision system and method for a RTG trolley

Abstract

The application discloses an RTG trolley multimodal fusion anti-collision system and method, and relates to the technical field of safety of port container loading and unloading equipment. In the method, after receiving two-dimensional image data and three-dimensional point cloud data, the point cloud data is adaptively preprocessed, deep fusion is realized based on a cross-modal Transformer network, the fusion weight is dynamically adjusted according to the real-time pose of a spreader, online calibration verification and drift compensation are carried out by using IMU data, trajectory prediction is carried out in combination with RTG kinematic constraints, and a warning strategy is adaptively adjusted based on a reinforcement learning algorithm. The system comprises a visual sensor module, a three-dimensional sensor module and an edge computing unit. The application can maintain stable detection performance under adverse conditions such as metal reflection, spreader shielding and vibration, realizes real-time processing with an end-to-end delay of less than 80 ms, and can maintain a high detection success rate and stability under different illumination and weather conditions.

Description

Technical Field

[0001] This invention relates to the field of safety technology for port container handling equipment, and in particular to a multimodal fusion anti-collision system and method for RTG large vehicles. Background Technology

[0002] As a core loading and unloading equipment in port container yards, RTGs (Rubber-Tired Gantry Cranes) face complex collision risks when their trolleys operate on rails. To ensure operational safety, collision avoidance systems are crucial safety devices for RTGs. However, existing publicly available collision avoidance technologies have the following shortcomings:

[0003] (1) Limitations of a single sensor: Millimeter-wave radar is susceptible to multipath reflection interference in dense metal environments and cannot be classified; pure vision solutions suffer from performance degradation in bad weather and at night and cannot accurately measure distances.

[0004] (2) Shortcomings of existing fusion strategies: Most existing solutions are "result-level fusion" (sensors detect each other and then correlate them), which cannot make full use of the complementary information between different modalities (2D images and 3D point clouds) data, and have poor robustness in complex scenarios.

[0005] (3) Challenges of the special environment of the terminal: Container metal reflection, spreader obstruction, equipment vibration and other factors lead to false detection, data loss and calibration drift, and existing technologies lack targeted solutions.

[0006] (4) It is difficult to balance accuracy and real-time performance: Deep learning models have high latency and lightweight models have low accuracy. Early warning strategies are mostly based on fixed thresholds and cannot be adjusted adaptively, resulting in false alarms or false negatives. Summary of the Invention

[0007] To address the problems of low detection accuracy, poor robustness, insufficient real-time performance, and lack of targeted scenario optimization in the existing technology using a single sensor in complex dock environments, this invention provides an RTG large vehicle multimodal fusion collision avoidance system and method.

[0008] The technical solution adopted in this invention is:

[0009] A multimodal collision avoidance method for large RTG vehicles includes the following steps:

[0010] Step S1: In real time, acquire two-dimensional image data of the surrounding environment of the RTG vehicle collected by at least one vision sensor module, and three-dimensional point cloud data collected by at least one three-dimensional sensor module.

[0011] Step S2: Based on the reflection intensity and spatiotemporal consistency of the three-dimensional point cloud data, perform adaptive preprocessing on the three-dimensional point cloud data;

[0012] Step S3: Input the preprocessed 3D point cloud data and the 2D image data into a multimodal fusion network based on an attention mechanism. The multimodal fusion network performs feature extraction and deep fusion on the 3D point cloud data and the 2D image data to obtain fused features.

[0013] Step S4: Based on the fused features, detect and identify targets around the RTG vehicle, generate target detection results through a unified detection head, and output motion trajectories with unique identifiers through a multi-target tracking algorithm;

[0014] Step S5: Based on the target's historical motion trajectory, predict the target's future motion trajectory;

[0015] Step S6: Based on the future motion trajectory, calculate the collision risk between the RTG vehicle and the target, and according to the collision risk, generate and output corresponding collision avoidance control commands to the RTG control system through an adaptive decision-making strategy based on a reinforcement learning model.

[0016] Furthermore, the specific process of step S2 includes:

[0017] Step S21: Calculate the reflection intensity of each point in the three-dimensional point cloud data, and mark the points whose reflection intensity exceeds a preset statistical threshold as abnormally high reflection points;

[0018] Step S22: Perform a geometric consistency check on the abnormally high reflectivity points. Points whose local plane fitting residual with neighboring points is greater than a preset geometric consistency threshold are identified as false reflectivity points and removed.

[0019] Step S23: Utilize the temporal correlation of point clouds across multiple consecutive frames to perform spatiotemporal consistency verification on suspected false reflection points and eliminate points whose frequency of occurrence is lower than a preset frequency threshold.

[0020] Furthermore, in step S3, the multimodal fusion network is a Cross-Modal Transformer network, which calculates the attention weights between two-dimensional image features and three-dimensional point cloud features through a cross-modal bidirectional attention module, thereby achieving feature complementarity enhancement between the two-dimensional image data and the three-dimensional point cloud data.

[0021] Furthermore, if the three-dimensional sensor module has two or more components, step S3 may include the following steps before implementation:

[0022] Step S31: Acquire the pose information of the RTG lifting device in real time, and calculate the dynamic occlusion of the monitoring area by each of the three-dimensional sensor modules based on the pose information;

[0023] Step S32: Based on the dynamic occlusion situation, dynamically adjust the weights of the 3D point cloud data from different 3D sensor modules during feature fusion.

[0024] Furthermore, before step S4 is performed, the following is also included:

[0025] Step S4A1: Real-time acquisition of motion attitude data of the RTG vehicle collected by the attitude sensor module, and motion compensation of the three-dimensional point cloud data using the motion attitude data;

[0026] Step S4A2: Extract the corresponding feature points from the 3D point cloud data and the 2D image, and calculate the reprojection error;

[0027] Step S4A3: When the reprojection error exceeds a preset threshold, an online calibration compensation process is triggered to correct the extrinsic parameters between the 3D sensor module and the vision sensor module.

[0028] Further, step S5 includes:

[0029] Step S51: Input the historical motion trajectory of the target into the Transformer encoder to capture its temporal dependencies;

[0030] Step S52: Encode the kinematic constraints of the RTG vehicle into a vector and fuse it with the output of the Transformer encoder. The kinematic constraints include at least one of track constraints, speed limits, and acceleration limits.

[0031] Step S53: Based on the fused features, decode and generate multiple probabilistic future motion trajectories, and output the covariance matrix of each future motion trajectory.

[0032] Furthermore, in step S6, the adaptive decision-making strategy based on the reinforcement learning model is trained using the SoftActor-Critic algorithm;

[0033] The state space of the adaptive decision-making strategy includes the target state, RTG state and environment state, and the action space includes no operation, warning, speed limit and emergency braking.

[0034] And / or, in step S6, when calculating the collision risk, the time to collision and the collision probability are evaluated together, and the target is divided into different risk levels according to the evaluation results.

[0035] Furthermore, when multiple RTGs exist within the storage yard besides this RTG, cooperative obstacle avoidance operations are also included, specifically:

[0036] Step S71: Interact with other RTGs in the yard to exchange status information and trajectory prediction information;

[0037] Step S72: Based on the information exchanged, calculate the potential conflicts between multiple machines using a distributed collaborative algorithm;

[0038] Step S73: Determine and execute the obstacle avoidance strategy of this RTG according to the negotiated priority.

[0039] Based on the same inventive concept, the present invention also provides an RTG large vehicle multimodal fusion collision avoidance system, comprising:

[0040] The vision sensor module is used to acquire two-dimensional image data of the surrounding environment of the RTG truck;

[0041] The 3D sensor module is used to collect 3D point cloud data of the surrounding environment of the TG truck.

[0042] The edge computing unit is communicatively connected to the vision sensor module and the 3D sensor module, and is configured as follows:

[0043] Receive the two-dimensional image data and the three-dimensional point cloud data;

[0044] Based on the reflection intensity and spatiotemporal consistency of the three-dimensional point cloud data, adaptive preprocessing is performed on the three-dimensional point cloud data;

[0045] The preprocessed 3D point cloud data and the 2D image data are input into a multimodal fusion network based on an attention mechanism for feature extraction and deep fusion to obtain fused features;

[0046] Based on the fusion features, targets around the RTG vehicle are detected and identified. Target detection results are generated through a unified detection head, and motion trajectories with unique identifiers are output through a multi-target tracking algorithm.

[0047] Based on the target's historical trajectory, predict the target's future trajectory;

[0048] Based on the future trajectory, the collision risk between the RTG vehicle and the target is calculated, and according to the collision risk, an adaptive decision-making strategy based on a reinforcement learning model is used to generate and output corresponding collision avoidance control commands.

[0049] Furthermore, it also includes:

[0050] An attitude sensor module is used to monitor the motion attitude data of the RTG vehicle.

[0051] The edge computing unit synchronously receives the motion posture data collected by the posture sensor module and uses the posture data to perform motion compensation on the three-dimensional point cloud data.

[0052] Extract the corresponding feature points from the 3D point cloud data and the 2D image, and calculate the reprojection error;

[0053] When the reprojection error exceeds a preset threshold, an online calibration compensation process is triggered to correct the extrinsic parameters between the 3D sensor module and the vision sensor module.

[0054] The beneficial effects of this invention are:

[0055] 1. Deep fusion of 2D image data and 3D point cloud data is achieved through the Cross-Modal Transformer fusion network, which can maintain high performance under various weather and lighting conditions. The target detection recall rate reaches 98.5% during the day, 96.2% at night, and 94.8% in rainy and foggy weather, which are about 8%, 21%, and 30% higher than existing technologies, respectively.

[0056] 2. An adaptive point cloud filtering algorithm designed for the metal reflection environment of the dock reduces the false detection rate from 15% to below 2% through reflection intensity analysis, geometric consistency test and spatiotemporal consistency verification, which significantly improves the robustness of the system in the dense metal container environment.

[0057] 3. By adopting a sensor redundancy configuration and dynamic fusion weight adjustment strategy, the system effectively addresses scenarios where the lifting device is obstructed. Under such circumstances, the target tracking recall rate remains above 92%, which is about 22% higher than existing technologies.

[0058] 4. Through online calibration verification and IMU-assisted compensation mechanism, the reprojection error is maintained within 3 pixels under vibration environment, which is significantly reduced compared with the 8-12 pixels of the existing technology, ensuring the long-term stability of calibration accuracy.

[0059] 5. The spatiotemporal joint trajectory prediction network incorporating RTG kinematic constraints achieves a 3-second trajectory prediction error (ADE) of 0.35 meters, which is about 60% lower than the 0.8-1.2 meters of existing technologies, significantly improving the accuracy of collision risk assessment.

[0060] 6. An adaptive early warning strategy based on reinforcement learning dynamically adjusts the early warning threshold according to scene characteristics and historical data, reducing the false alarm rate from 12% of the traditional fixed threshold method to 3.5%, while keeping the collision false alarm rate below 0.5%.

[0061] 7. The end-to-end inference latency is 65-85ms, which is about 50-70% lower than the existing technology's 150-300ms, meeting the requirements for real-time early warning.

[0062] 8. Supports multi-RTG collaborative obstacle avoidance. Through distributed collaborative algorithms and priority negotiation mechanisms, it can improve the overall operation efficiency of the yard by 15-20% while ensuring the safety of multi-machine operation. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is a schematic diagram of the RTG large vehicle multimodal fusion collision avoidance control method in Example 1. Detailed Implementation

[0065] The embodiments of the invention will now be described in detail with reference to the accompanying drawings. It should be noted that the terms used in the following embodiments are defined as follows:

[0066] Recall rate: refers to the percentage of correctly detected targets out of the actual total number of targets;

[0067] Average positioning error: refers to the average distance between the center of the detection frame and the center of the actual target;

[0068] End-to-end delay: refers to the total processing time from sensor data acquisition to the output of control commands;

[0069] False alarm rate: refers to the percentage of falsely detected targets out of the total number of detections;

[0070] Trajectory prediction error: refers to the average distance between the predicted position and the actual position.

[0071] Example 1

[0072] This embodiment provides a multimodal fusion collision avoidance system for RTG large vehicles, including a vision sensor module, a 3D sensor module, and an edge computing unit. Furthermore, it also includes an attitude sensor module.

[0073] Specifically, the vision sensor module includes a front-view camera and two side-view cameras. The front-view camera is a Hikvision DS-2CD4A26FWD-IZ model, with a resolution of 1920×1080, a frame rate of 30fps, a lens focal length of 2.8-12mm with motorized zoom, supports a 120dB wide dynamic range and intelligent infrared illumination, and has a night vision distance of 50m. The front-view camera is mounted at the center front of the RTG truck, 4.5m above the ground, with a 15° downward tilt angle. The left-side camera is a Dahua DH-IPC-HFW5449E-ZH model, with a resolution of 1280×720, a frame rate of 25fps, a lens focal length of 2.7-12mm with motorized zoom, and a horizontal field of view of 110°. The left-side camera is mounted on the left side of the RTG truck, 3.2m above the ground, and horizontally tilted 20° to the left. The right-side camera has the same configuration as the left-side camera, symmetrically mounted on the right side. The field of view of the forward-looking camera is precisely calculated to ensure that the side-looking camera can still observe the target even when the main sensor is obstructed by the rigging. Pin 1 (positive power) of the forward-looking camera is connected to the DC 24V power module, pin 2 (ground) is connected to ground, pins 3 (Ethernet TX+) and 4 (Ethernet TX-) are connected to port 1 of the Gigabit Ethernet switch on the edge computing unit via shielded twisted-pair cables, and pins 5 (Ethernet RX+) and 6 (Ethernet RX-) are also connected to the switch via shielded twisted-pair cables. The pin connections of the side-looking camera are the same as those of the forward-looking camera, connected to ports 2 and 3 of the switch, respectively.

[0074] The 3D sensor module includes a main LiDAR and two auxiliary LiDARs. The main LiDAR is a Hesai Pandar64 model, a 64-line LiDAR with a vertical field of view of ±25°, a horizontal field of view of 360°, a ranging range of 200m, an accuracy of ±2cm, a point frequency of approximately 2.3 million points / second, and a rotation frequency of 10Hz. The main LiDAR is mounted at the center of the top of the RTG vehicle, approximately 12m above the ground. Both the left and right auxiliary LiDARs are RoboSense RS-LiDAR-32 models, 32-line LiDARs with a vertical field of view of ±15°, a horizontal field of view of 120°, a ranging range of 150m, and an accuracy of ±2cm. The left auxiliary LiDAR is mounted on the front left side of the RTG vehicle, 4m above the ground, and horizontally deflected 30° to the left; the right auxiliary LiDAR is symmetrically mounted on the front right side. The main LiDAR's pin 1 (positive power) is connected to the DC 24V power module, pin 2 (ground) is connected to ground, and pin 3 (Ethernet data output) is connected to port 4 of the edge computing unit's Gigabit Ethernet switch via a shielded network cable. The auxiliary LiDAR's pin connections are the same as the main LiDAR, connected to switch ports 5 and 6 respectively. The arrangement of the three LiDARs forms a 120° field of view coverage, allowing the auxiliary LiDARs to continue monitoring the target even when the main LiDAR is obstructed by the hoisting equipment.

[0075] The edge computing unit utilizes the NVIDIA Jetson AGX Xavier industrial-grade embedded platform, configured with a 512-core Volta GPU (32 TOPS computing power), an 8-core ARM CPU, 32GB of RAM, and 64GB of eMMC storage. The edge computing unit is installed inside the electrical control cabinet of the RTG (Remote Transportation Truck) and communicates with the RTG's control system (core of which is a PLC controller) via a CAN bus. The edge computing unit's CAN interface uses a PEAK PCAN-USB interface card with a communication rate of 500kbps. Pin 1 (CAN_H) of the edge computing unit connects to the CAN_H port of the RTG's PLC controller, and pin 2 (CAN_L) connects to the CAN_L port of the control system (core of which is the PLC controller). The edge computing unit receives all sensor data through a gigabit Ethernet switch; the switch is an industrial-grade gigabit switch supporting eight gigabit ports.

[0076] The attitude sensor module includes an Inertial Measurement Unit (IMU) and an inclinometer. The IMU is an Xsens MTi-G-710 model, a six-axis MEMS inertial measurement unit with a gyroscope accuracy of 0.05° / s, an accelerometer accuracy of 0.002m / s², and an output frequency of 100Hz. The IMU is mounted at the center of the RTG's main beam, adjacent to the main LiDAR, for real-time monitoring of the RTG's vibration acceleration and tilt angle changes. The inclinometer is a Beiwei Sensing BW-VG400C, with pitch ±90º, roll ±180º, and an accuracy of 0.01°, used to measure the longitudinal and lateral tilt angles of the RTG. Pin 1 (positive power) of the IMU is connected to the DC 12V power module, pin 2 (ground) is connected to ground, and pins 3 (UART TX) and 4 (UART RX) are connected to serial port 1 of the edge computing unit via an RS-232 interface. The inclinometer's pin connections are the same as the IMU, connected to serial port 2 of the edge computing unit.

[0077] Figure 1 This is a flowchart illustrating the RTG large vehicle multimodal fusion collision avoidance method based on the RTG large vehicle multimodal fusion collision avoidance system in Example 1. The following specific operations are performed in cooperation between the various components of the system.

[0078] (1) The edge computing unit performs adaptive preprocessing on the three-dimensional point cloud data collected by the three-dimensional sensor module.

[0079] Calculate the reflection intensity I of each point in the 3D point cloud data and establish a reflection intensity histogram. Calculate the mean I_mean and standard deviation σ of the local reflection intensity of the point cloud, and mark points that satisfy I>I_mean+2σ (i.e., the preset statistical threshold) as abnormally high reflection points. For the marked abnormally high reflection points, perform a geometric consistency check and calculate the local plane fitting residual r_i between the point and its neighboring points (points within a radius of 0.2m). Use the RANSAC algorithm to fit the local plane of the neighboring points, with the plane equation ax+by+cz+d=0; where (a,b,c) are the plane normal vectors. Calculate the distance r_i from the abnormally high reflection point to the fitted plane: r_i=|ax_i+by_i+cz_i+d| / √(a²+b²+c²); if r_i>τ_geo, where the preset geometric consistency threshold τ_geo is 0.08m, then the point is determined to be a false reflection point and is removed. By utilizing the spatiotemporal consistency of data points in three consecutive frames, suspected false reflection points are checked for their frequency of occurrence in the preceding and following three frames. If the point appears less than twice in the three frames (frequency lower than a preset frequency threshold of 67%), it is discarded.

[0080] Furthermore, in this embodiment, to address the challenges faced by RTGs in real, dynamic port operation environments and to provide a high-quality, reliable data foundation for subsequent obstacle detection and collision warning, an improved RANSAC algorithm combined with tilt information provided by the IMU can be used for ground segmentation. This ensures that the system uses limited computing resources effectively, maintaining high resolution at close range where precise sensing is crucial, while reducing resolution to improve speed at distant ranges where only warning is needed. This is one of the key technical means to achieve both high accuracy and low latency in the system. The longitudinal tilt angle α and lateral tilt angle β of the RTG are obtained from the inclinometer, and a rotation matrix R_IMU is constructed. The ground normal vector is initialized as n_init=R_IMU·[0,0,1]^T. The RANSAC algorithm iterates 50 times, randomly selecting 3 points to fit the plane each time, calculating the distance from the points to the plane, and designating points with a distance less than 0.15m as interior points. The plane with the most interior points is selected as the ground plane, and the segmented ground points are used for subsequent obstacle height calculation. The VoxelGrid voxel size is dynamically adjusted based on the point cloud density. For close-range areas less than 20m from the RTG, a voxel size of 0.05m is used; for medium-range areas between 20-50m, a voxel size of 0.1m is used; and for distant areas greater than 50m, a voxel size of 0.2m is used.

[0081] (2) Establish a sensor field of view model and calculate in real time the visibility of each sensor (i.e., main LiDAR, left auxiliary LiDAR, and right auxiliary LiDAR) in the three-dimensional sensor module to each region of space.

[0082] The 80m×30m×8m space in front of the RTG is divided into 0.5m×0.5m×0.5m grids, totaling 80 / 0.5×30 / 0.5×8 / 0.5=153600 grids. A visibility matrix V∈R^(S×G) is defined, where S=3 represents the number of 3D sensor modules (i.e., main LiDAR, left auxiliary LiDAR, and right auxiliary LiDAR), and G=153600 represents the number of spatial grids. For each spatial grid j, the ray from the main LiDAR, left auxiliary LiDAR, and right auxiliary LiDAR to that grid is calculated to determine if it is blocked by the lifting device. The pose information (i.e., position coordinates and attitude angles) of the lifting device is obtained in real time from the RTG's control system (core being a PLC controller). Based on the lifting device's geometric model (6m long × 2.4m wide × 0.8m high), the voxels occupied by the lifting device in the spatial grid are calculated. If the ray from sensor i to spatial grid j passes through the voxel occupied by the hanger, then V_ij = 0 (invisible); otherwise, V_ij = 1 (visible). The fusion weights of each sensor are dynamically adjusted according to the real-time occlusion situation. For example, for spatial position p, the weight of sensor i is calculated as w_i(p) = V_i(p)·Q_i / ∑_{j=1}^S V_j(p)·Q_j; where Q_i is the intrinsic quality factor of sensor i. In this embodiment, the intrinsic quality factor of the main LiDAR is preset to Q_1 = 1.0, and the intrinsic quality factors of the left auxiliary LiDAR and the right auxiliary LiDAR are Q_2 = Q_3 = 0.8 (due to the small number of lines and low point cloud density). V_i(p) is the visibility indication function. For example, if sensor i can see spatial position p, then V_i(p) = 1; otherwise, V_i(p) = 0.

[0083] For a target observed by multiple sensors, feature fusion is performed at the feature level, projecting features from different modalities into a unified embedding space and using an attention mechanism for adaptive weighting. The fused feature is f_fused=∑_{i=1}^S α_i·f_i, where the attention weight α_i=exp(a_i) / ∑_j exp(a_j), the attention weight a_i is dynamically calculated by a small neural network based on feature quality, and f_i is the feature extracted from the original 3D point cloud data collected by sensor i.

[0084] (3) The edge computing unit uses IMU data for online calibration and verification.

[0085] The IMU acquires the running attitude data (acceleration and angular velocity data) of the RTG vehicle at a frequency of 100Hz, and extracts the vibration components through high-pass filtering (cutoff frequency 5Hz). For the rotating main LiDAR, the acquisition time of each point is different, and motion compensation is performed on the point cloud using the running attitude data acquired by the IMU. Let time t_0 be the frame start time, and time t_i be the acquisition time of point i. The pose transformation matrix T(t_i) at time t_i is estimated by IMU integration and Kalman filtering. The corrected coordinates of point i are p_i^corrected = T(t_0)^(-1)·T(t_i)·p_i^raw, where the superscripts corrected and raw represent corrected and original, respectively.

[0086] Design an automated online calibration and verification algorithm to extract corner points and edge features from 3D point cloud data (using the LOAM algorithm) and extract Harris corner points from 2D images. Project the 3D feature points extracted from the 3D point cloud data onto the image plane using the projection formula u=π(K[R|t]P), where K is the camera intrinsic matrix, R and t are the extrinsic parameters (rotation matrix and translation vector) of the transformation from the LiDAR coordinate system to the camera coordinate system, P is the coordinates of the 3D feature point, and u is the pixel coordinates after projection. Calculate the distance d_proj between the projected point and the nearest 2D feature point. Maintain a sliding window (length 100 frames) and calculate the average reprojection error. and standard deviation σ_proj. If the current frame reprojection error satisfies d_proj > If +3σ_proj is applied, a calibration anomaly alarm will be triggered.

[0087] For small-amplitude drift (reprojection error < 5 pixels), an online optimization algorithm is used to fine-tune the extrinsic parameters of the transformation from the LiDAR coordinate system to the camera coordinate system. The optimization objective is defined as min_{ΔR,Δt}∑_{k=1}^N ||u_k-π(K(ΔR·R_0|Δt+t_0)p_k)||^2, where (R_0,t_0) are the initial calibration parameters and (ΔR,Δt) are the drift amount to be estimated. The Levenberg-Marquardt algorithm is used to solve the problem. Specifically, the steps of the online calibration compensation process include: (1) When online calibration compensation is triggered, the edge computing unit takes out the effective feature matching pair {P_j, u_j} of the nearest M frames (e.g., M=20) from the sliding window, where P_j is the 3D feature point extracted from the 3D point cloud and u_j is the corresponding 2D feature point extracted from the 2D image. The pose drift parameters to be optimized are initialized as follows: the rotation matrix increment ΔR is initialized to a 3x3 identity matrix I, and the translation vector increment Δt is initialized to a zero vector 0. (2) The Levenberg-Marquardt algorithm is used to perform nonlinear least squares optimization on the objective function. In each iteration: a) Calculate the reprojection error e_j = u_j - π( K(ΔR·R_0 | Δt + t_0) P_j ) of all feature points under the current parameters (ΔR, Δt). b) Calculate the Jacobian matrix J of the error (i.e. the derivative of the error e with respect to the parameters (ΔR, Δt), where ΔR is parameterized using the perturbation model on the Lie algebra so(3). c) Construct the increment equation: (J^TJ + λ I) δ = -J^T e, where λ is the damping factor, I is the identity matrix, and δ is the parameter increment to be calculated. d) Solve the incremental equation to obtain δ, and update the parameters: (ΔR, Δt) ← (ΔR, Δt) ⊕ δ, where ⊕ is the update operation on the Lie group. e) Adjust the damping factor λ dynamically according to whether the error decreases. (3) The optimization process terminates when any of the following conditions are met: a) Error condition: The average reprojection error of all feature points is less than the preset error threshold ε_th (e.g., ε_th = 0.5 pixels). b) Increment condition: The norm of the parameter increment δ is less than the preset increment threshold δ_th. c) Iteration number condition: The number of iterations reaches the preset maximum number of iterations N_max (e.g., N_max = 50). (4) Combine (ΔR, Δt) obtained after optimization convergence with the original extrinsic parameters (R_0, t_0) to obtain the updated calibration parameters (R_new, t_new) = (ΔR·R_0, Δt + t_0), and use them for data processing of subsequent frames.In this embodiment, the use of sliding window multi-frame data increases the constraints of the optimization problem and improves robustness in cases of vibration or few feature points. The Levenberg-Marquardt algorithm, by introducing a damping factor, combines the convergence speed of the Gauss-Newton method with the stability of the gradient descent method, effectively avoiding getting trapped in local optima, thereby ensuring the reliability and accuracy of online calibration compensation.

[0088] (4) The edge computing unit uses the Cross-Modal Transformer fusion network to deeply fuse two-dimensional image features and three-dimensional point cloud features; it generates target detection results (i.e., target category, center coordinates, width and height, confidence of 2D box, center coordinates x, y, z, size l, w, h, orientation angle θ, and confidence of 3D box) through a unified detection head; based on the fused target detection results, it tracks the detected target and outputs a motion trajectory with a unique identifier.

[0089] The 2D branch uses the lightweight convolutional neural network EfficientNet-B0 to extract image features. The input size is 640×640, and the output is a multi-scale feature map F_2D^l (l=1,2,3), with feature map sizes of 80×80, 40×40, and 20×20, and 64, 128, and 256 channels, respectively. The 3D branch uses the PointPillars network to process the point cloud, dividing it into 0.2m×0.2m pillars. Each pillar samples a maximum of 32 points. Point features include coordinates (x,y,z), reflection intensity I, and relative position to the pillar center (x_c,y_c), totaling 7 dimensions. PointNet is used to encode the points within each pillar, generating 64-dimensional pillar features. These pillar features are then scattered onto a pseudo-image to form a BEV feature map F_3D∈R^(400×400×64), corresponding to a physical area of ​​80m×80m. A 2D convolutional network is applied to the BEV feature map with four downsampling stages and the number of channels is 64, 128, 256 and 512 respectively, to output multi-scale BEV features; a cross-modal bidirectional attention module is used to achieve deep interaction between two-dimensional image features and three-dimensional point cloud features.

[0090] The pixel-level features of the 2D feature map are used as the query, and the spatial features of the 3D point cloud are used as the key and value. For the l-th layer features:

[0091] Query_2D=Linear(F_2D^l)∈R^(H_1W_1×d),

[0092] Key_3D=Linear(F_3D^l)∈R^(H_2W_2×d),

[0093] Value_3D=Linear(F_3D^l)∈R^(H_2W_2×d);

[0094] Where d=256 is the attention dimension, and H_1, W_1 and H_2, W_2 are the spatial dimensions of the 2D and BEV feature maps, respectively.

[0095] Calculate attention weights:

[0096] Attn_{2D→3D}=Softmax(Query_2D·Key_3D^T / √d)·Value_3D.

[0097] Symmetrically compute Attn_{3D→2D}, add the attention output to the original features (residual connection), F_2D^enhanced = LayerNorm(F_2D + Attn_{2D→3D}), and then further process it through MLP, F_2D^fused = MLP(F_2D^enhanced). The MLP contains two fully connected layers with the GELU activation function in between, and the hidden layer dimension is 1024. In the actual implementation, 8 attention heads are used, each with a dimension of 32 (total dimension 256). Simultaneously, this embodiment adds learnable positional encodings to the 2D and BEV features to help the model understand spatial relationships.

[0098] A unified detection head is applied to the fused features, simultaneously outputting 2D and 3D bounding boxes. The detection head contains three convolutional layers, with the last layer outputting N_a×(5+C+7) channels, where N_a=3 is the number of anchors, 5 is the 2D bounding box parameters (center coordinates, width, height, confidence), C=5 is the number of target categories (person, forklift, container truck, container, other obstacles), and 7 is the 3D bounding box parameters (center coordinates x, y, z, dimensions l, w, h, orientation angle θ).

[0099] Anchors of different sizes were designed for different target categories: people (0.6m×0.6m×1.8m), forklifts (2.0m×1.5m×2.0m), trucks (6.0m×2.5m×3.0m), and containers (6.0m×2.4m×2.6m). The loss function was L_total=λ_cls·L_cls+λ_2D·L_2D+λ_3D·L_3D+λ_iou·L_iou, where L_cls is the classification loss (FocalLoss), L_2D is the 2D bounding box regression loss (Smooth L1 Loss), L_3D is the 3D bounding box regression loss (Smooth L1 Loss), and L_iou is the 3D IoU loss. The weights were set to λ_cls=1.0, λ_2D=2.0, λ_3D=2.0, and λ_iou=1.5.

[0100] The network training employs a two-stage strategy: In the first stage (pre-training), the cross-modal bidirectional attention module is frozen, and the 2D and 3D branches are trained separately for 50 epochs each. In the second stage (joint training), all parameters are unfrozen, and the entire network is trained end-to-end for 100 epochs using a small learning rate of 0.0001 to avoid corrupting the pre-trained weights. In the second stage (joint training), edge computing units compute the prediction results output by the multimodal fusion network via forward propagation and optimize all parameters of the multimodal fusion network end-to-end using the backpropagation algorithm, aiming to minimize the loss function L_total. The composition of the loss function L_total, the calculation method of each component, and its balancing weights (λ_cls, λ_2D, λ_3D, λ_iou) remain consistent with the settings deployed in the unified detection head after the first stage of pre-training. Joint training simultaneously optimizes the parameters of the 2D image feature extraction branch, the 3D point cloud feature extraction branch, the cross-modal bidirectional attention module, and the unified detection head, enabling the multimodal fusion network to learn the optimal mapping from multimodal sensor data to the final target detection and localization results. Training was performed using a self-built dataset (containing 50,000 labeled images covering different weather and lighting conditions) with a batch size of 32, an initial learning rate of 0.01, and cosine annealing scheduling. Data augmentation included random cropping, flipping, color dithering, and mosaic enhancement. After training, TensorRT was used for INT8 quantization, with the quantization calibration dataset containing 1000 representative images. After quantization, the model size decreased from 14MB to 3.8MB, the inference speed increased from 25ms to 8ms (2D branch), and the 3D branch decreased from 45ms to 18ms, with a total inference time of 35ms for the fusion network.

[0101] This embodiment employs an improved SORT algorithm, combined with Kalman filtering and the Hungarian algorithm, to track detected targets based on the fused target detection results. The tracking state vector is x=[x,y,z,v_x,v_y,v_z,l,w,h,θ]^T, where (x,y,z) are the target center coordinates, (v_x,v_y,v_z) are the velocities, (l,w,h) are the bounding box dimensions, and θ is the orientation angle. A constant velocity model is used, and the state transition equation is x_{k+1}=F·x_k+w_k, where the state transition matrix F is a block diagonal matrix, with the position block [1,Δt;0,1], the velocity block [1,0;0,1], and the size and orientation blocks being identity matrices, and Δt=0.1s being the time interval. The process noise covariance matrix Q = diag([0.1,0.1,0.1,0.5,0.5,0.5,0.01,0.01,0.01,0.05]). The measurement vector is z = [x,y,z,l,w,h,θ]^T (directly from the detection results), and the observation matrix H is [I_3,0_{3×3},0_{3×4};0_{4×3},0_{4×3},I_4]. The measurement noise covariance matrix R is dynamically adjusted according to the detection confidence, R = (1 / conf)·diag([0.2,0.2,0.2,0.1,0.1,0.1,0.1]); where conf is the detection confidence (between 0 and 1). For the predicted motion trajectory i and the detection j, the association cost is defined as:

[0102] C_{ij}=α(1-IoU_3D(box_i^pred,box_j^det))+β||pos_i^pred-pos_j^det|| / d_max+γ(1-class_match_{ij});

[0103] Wherein, IoU_3D is the IoU of the 3D bounding box, d_max=10m is the normalized distance, and class_match is the target class matching indicator (1 for the same, 0 for different). Weights are set to α=1.0, β=0.5, and γ=2.0 (with a larger penalty for class mismatch). The Hungarian algorithm is used to solve for the optimal assignment, minimizing the total association cost. The association threshold is set to 0.7; pairings with a total association cost greater than the threshold are rejected. Unmatched detections are initialized as new running trajectories, and unmatched motion trajectories enter a lost state. Lost motion trajectories are deleted after 5 consecutive frames without association. Three motion trajectory states are defined: Tentative, Confirmed, and Lost. New motion trajectories are initially Tentative, become Confirmed after 3 consecutive frames of association, and become Lost and are deleted after 5 consecutive frames without association. A unique ID (incrementing integer) is assigned to each confirmed motion trajectory, and the ID remains unchanged throughout the trajectory's lifecycle. A historical state queue (maximum length 30 frames, corresponding to 3 seconds) is maintained for each motion trajectory for subsequent trajectory prediction.

[0104] (5) Use a spatiotemporal joint trajectory prediction network with RTG kinematic constraints to predict the future trajectory of the target.

[0105] The input to the Cross-Modal Transformer fusion network is the historical motion trajectory of the target, {s_t}_{t=-T_h}^0, where T_h = 20 frames (2 seconds), and s_t = [x_t,y_t,z_t,v_x,v_y,l,w,h,θ]. The state at each time step is mapped to 128 dimensions through a linear layer. A 4-layer Transformer encoder is used, with each layer containing 8 attention heads and a hidden dimension of 512. The Transformer encoder can capture the long-term temporal dependencies of the historical trajectory, and the encoder output H = TransformerEncoder({s_t}_{t=-T_h}^0).

[0106] Meanwhile, in this embodiment, the kinematic constraints of the RTG (i.e., at least one of track constraints, speed limits, and acceleration limits) are used as additional inputs. The RTG runs along a fixed track with limited lateral displacement, and the track centerline equation is defined as y=f(x). For the future trajectory, a soft constraint is applied: L_track=∑_{t=1}^{T_f}||y_t-f(x_t)||^2; where T_f=30 frames (3 seconds) is the prediction duration. The maximum speed of the RTG is v_max=2.5m / s, and the maximum acceleration is a_max=0.5m / s². When predicting speed and acceleration, clipping is performed: v_t=clip(v_{t-1}+a_t·Δt,0,v_max). The turning radius of the RTG is R_min≥10m, and the curvature κ of the future trajectory must satisfy |κ|≤1 / R_min. Based on the target category, predefined kinematic parameters (maximum speed, maximum acceleration, minimum turning radius) are queried, and these parameters are encoded into a 64-dimensional vector, which is then concatenated with the Transformer output.

[0107] The MLP decoder outputs K=6 candidate future motion trajectories (each containing 30 time points) and their probability distributions {(Traj_k,p_k)}_{k=1}^K, where ∑_{k=1}^K p_k=1. Each candidate future motion trajectory is represented as {ŝ_t^(k)}_{t=1}^30 with a probability of p^(k). Winner-Takes-All (WTA) loss is used for training, applying the loss only to the candidate future motion trajectory closest to the ground truth. The optimal trajectory index is defined as k^*=argmin_k (1 / T_f)∑_{t=1}^{T_f}||p_t^(k)-p_t^gt||^2; where p_t is the position coordinate. The trajectory loss is L_traj=(1 / T_f)∑_{t=1}^{T_f}||p_t^(k^*)-p_t^gt||^2. The classification loss is L_cls = -log p^(k^*). The kinematic constraint loss is L_vel = ∑_{t=1}^{T_f}max(0,||v̂_t||-v_max)^2. Similarly, the acceleration constraint loss L_acc and the curvature constraint loss L_curv are defined. The total loss is L_total = L_traj + 0.5L_cls + 0.1(L_vel + L_acc + L_curv).

[0108] Output the covariance matrix Σ_t∈R^(3×3) (only considering the uncertainty of position) for each future motion trajectory at each prediction time. The covariance matrix is parameterized by Cholesky decomposition, Σ_t = L_t·L_t^T; where L_t is a lower triangular matrix. The network outputs 6 parameters (l_{11}, l_{21}, l_{22}, l_{31}, l_{32}, l_{33}). The prediction network is relatively small (with about 5M parameters) and the inference time is about 12ms. Further, after optimization using TensorRT in this embodiment, it is reduced to 5ms.

[0109] (6) Calculate the predicted time to collision (TTC) and collision probability based on the future motion trajectory; adaptively adjust the collision warning strategy based on the reinforcement learning algorithm, and output a control command to the control system of the RTG (the core is a PLC controller) when the collision risk exceeds the dynamic threshold.

[0110] For each tracked target, calculate the TTC by a simplified method or an exact method.

[0111] Among them, the simplified method assumes that the tracked target and the RTG both move in a straight line at the current speed, and calculates the minimum distance d_min and the corresponding time t_min. If d_min < D_safe (the safety distance, set to 3m), then TTC = t_min; otherwise TTC = ∞.

[0112] The exact method calculates the TTC based on the future motion trajectory. For each future motion trajectory k, calculate the minimum distance d_t^(k) between the RTG bounding box and the target bounding box at each time. Considering the orientation and shape of the bounding box, use the Separating Axis Theorem to calculate the exact distance. TTC is defined as TTC^(k) = min{t|d_t^(k) ≤ D_safe}. The final TTC is the probability-weighted average of all future motion trajectories TTC = ∑_{k = 1}^K p^(k)·TTC^(k).

[0113] Considering the uncertainty of prediction, the collision probability is calculated in the following way in this embodiment. Assume that the target position follows a Gaussian distribution N(p̂_t,Σ_t), and the collision probability is P_collision = max_{t∈[0,T_f]} P(p_t∈CollisionZone); where CollisionZone is the area after expanding the RTG bounding box by D_safe. The Monte Carlo sampling (1000 samples) is used to approximately calculate this collision probability. The targets are divided into four collision risk levels according to TTC and collision probability: ① Safe: TTC > 5s or P_collision < 0.1; ② Low: 3 < TTC ≤ 5s and 0.1 ≤ P_collision < 0.3; ③ Medium: 1.5 < TTC ≤ 3s and 0.3 ≤ P_collision < 0.6; ④ High: TTC ≤ 1.5s or P_collision ≥ 0.6.

[0114] In this embodiment, the Soft Actor-Critic (SAC) algorithm is specifically used to adaptively adjust the collision warning strategy. The Actor network takes the state space S as input and outputs the action probability distribution π(a|S); the network structure of the Actor network is a 3-layer MLP, with the hidden layer dimension of 256-256, and the Softmax is used in the output layer. The state space S includes the target state (TTC, collision probability, relative position, relative velocity, category of the highest-risk target), RTG state (current speed, acceleration, braking state, gear), environmental state (visibility, road slipperiness, current time period), and historical actions (actions at the previous 3 time moments), with a total dimension of about 50. The action space A is a discrete action space, including 4 actions: a_0 no operation (maintain the current state), a_1 acoustic and optical warning (trigger the buzzer and warning light without changing the movement of the RTG), a_2 speed limit (send a deceleration command to the RTG controller, and the target speed is reduced to 50% of the current speed), a_3 emergency braking (send an emergency stop command and brake with the maximum deceleration).

[0115] The Critic network contains two Q networks (Q1 and Q2, used to reduce overestimation), takes the state-action pair (S,a) as input, and outputs the Q value; the network structure of the Critic network is a 3-layer MLP, with the hidden layer dimension of 256-256. The temperature parameter α controls the balance between exploration and exploitation and is automatically adjusted to match the target entropy (set to -|A| = -4). The experience replay maintains a replay buffer with a capacity of 100k to store the experience tuples (S,a,r,S'). 256 samples are randomly sampled each time for update. The network is updated once every 10 new samples are collected, and each update iterates 10 steps.

[0116] The reward function is defined as R(s,a)=R_safety+R_efficiency+R_comfort; the safety reward R_safety=-100 (if a collision occurs)+10 (if high risk is avoided)+0 (other cases); the efficiency reward R_efficiency=-5·1_{false_alarm}+0.1·v_current / v_max, with a false alarm penalty of -5 points and a small reward for maintaining high speed; the comfort reward R_comfort=-0.5·|a_current-a_previous|, with frequent switching of penalty actions to avoid sudden braking and stopping.

[0117] The training process consists of three phases: Phase 1 (Simulation Pre-training): 100,000 steps of training are performed in the CARLA simulator to learn the basic strategy. Phase 2 (Real Vehicle Fine-tuning): The strategy is deployed on a real RTG and run in exploration mode (high temperature parameters) to collect data and update the strategy in the cloud. Strategy parameters are updated and distributed weekly. Phase 3 (Safety Verification): Before deployment, the new strategy must pass a series of safety tests (such as extreme scenario simulations) to ensure that it will not make dangerous decisions.

[0118] The rule layer defines hard rules: Rule 1, if TTC < 0.5s, enforce emergency braking; Rule 2, if personnel are detected and the distance is < 5m, enforce speed limit to 0.5m / s; Rule 3, if the sensor malfunctions (e.g., LiDAR has no data), enforce speed reduction to safe mode (< 1m / s). Under the premise of satisfying the rules, the strategy layer determines the specific action by the RL strategy. The strategy considers long-term cumulative rewards and can make more intelligent decisions (e.g., early warning instead of temporary emergency braking). The execution layer converts the actions output by the strategy into specific control instructions and sends them to the RTG controller via the CAN bus. Example of instruction format: CAN ID 0x200, Data[0] command type (0x01=warning, 0x02=speed limit, 0x03=braking), Data[1] target speed (valid only when speed limit is limited, unit 0.1m / s), Data[2-7] reserved. The system monitors the RTG's response in real time (receiving status feedback via CAN ID 0x201). If the RTG fails to execute the command (e.g., a braking system malfunction), a level 2 alarm is triggered (notifying manual intervention).

[0119] (7) Multi-RTG collaborative obstacle avoidance processing.

[0120] Each RTG is equipped with a 5G communication module and communicates with the edge computing unit through a dedicated network (slice). The communication protocol uses MQTT, and the message format is JSON. The message types include heartbeat messages (1Hz): containing RTG ID, location, speed, orientation, status (running / stopped / fault); trajectory messages (0.5Hz): containing the future motion trajectory (next 5 seconds) and confidence; warning messages (event-triggered): sent immediately when a high-risk target is detected, containing target information and recommended obstacle avoidance actions. The end-to-end latency of the 5G network is about 20 - 30ms, meeting the real-time requirements.

[0121] Deploy high-performance servers (configured with 32-core CPUs and 128GB of memory) in the edge computer room of the yard to run the coordination algorithm. The coordinator maintains a high-precision map of the yard (with an accuracy of 10cm), including RTG tracks, container stacking areas, no-go areas, etc. The map is updated regularly (once a day) to reflect changes in the yard layout. The coordinator tracks the positions and statuses of all RTGs in real time, using Kalman filtering to fuse multiple data sources (RTG self-reported positions, GPS, UWB positioning).

[0122] Steps of the conflict detection algorithm: Step 1, predict the future motion trajectories of each RTG (using the same prediction network as the single machine); Step 2, for each pair of RTGs (i, j), calculate the minimum distance d_{ij}(t) = min_{t'∈[t,t+T]}||pos_i(t') - pos_j(t')|| in space and time of their trajectories; Step 3, if d_{ij}(t) < D_safe_multi (multi-machine safety distance, set to 8m), then it is determined that there is a conflict.

[0123] Priority assignment uses a weighted scoring function to calculate the priority of each RTG: Priority_i = 0.4·TaskUrgency_i + 0.3·(1 - load_i / load_max) + 0.3·v_i / v_max, where TaskUrgency_i is the task urgency (0 - 1, provided by the scheduling system), load_i is the current load, and v_i is the current speed. RTGs with higher priorities pass through the conflict area first. For low-priority RTG_j, obstacle avoidance instructions are generated: decelerate, calculate the required deceleration amount so that j reaches the conflict area after the high-priority RTG_i has passed; stop, if deceleration is not sufficient to avoid the conflict, instruct j to stop and wait; detour, if there is an alternative path (such as switching to an adjacent track), instruct j to detour. The obstacle avoidance instructions are sent to the corresponding RTG through MQTT, and the RTG executes and gives feedback confirmation after receiving them.

[0124] Distributed negotiation is triggered when an RTG loses connection with the coordinator for more than 5 seconds, or when the coordinator fails. Each RTG independently calculates its obstacle avoidance strategy based on received broadcast information from neighboring RTGs (via DSRC short-range communication, range 300m). Multi-RTG obstacle avoidance is modeled as a non-cooperative game, where each RTG chooses its optimal strategy (decelerate / stop / stay) to maximize its own utility function U_i(a_i,a_{-i})=-t_{delay,i}-λ·1_{collision}, where t_{delay,i} is the delay time of i, a_{-i} is the action of other RTGs, and λ=1000 is the collision penalty coefficient. The Iterated Best Response algorithm is used to solve for the Nash equilibrium. Through multiple rounds of broadcasting and negotiation (up to 5 rounds), the decisions of each RTG are ensured to be compatible. If consensus cannot be reached, all RTGs adopt a conservative strategy (stop).

[0125] When a person is detected entering a danger zone (within 10m of any RTG), an emergency avoidance mechanism is triggered. The RTG that detected the person immediately broadcasts an emergency message (via DSRC and 5G dual channels), containing the person's location and a suggested parking area. All RTGs receiving the message immediately decelerate to a safe speed (0.5m / s) or stop, waiting for the person to leave before resuming operation. The person's location is continuously tracked using a forward-looking camera, two side-looking cameras, a main LiDAR, and two auxiliary LiDARs until they leave the danger zone (beyond 10m of all RTGs), at which point the emergency is lifted.

[0126] To ensure deterministic real-time performance and absolute security of the distributed cooperative obstacle avoidance algorithm under limited communication and computing resources, this embodiment designs explicit algorithm termination conditions and exception handling strategies: ① The iterative optimal response algorithm operates within a preset maximum number of negotiation rounds, for example, 5 rounds. Each round of negotiation includes each RTG broadcasting its own policy, receiving neighboring policies, and calculating the optimal response for that round based on the utility function U_i. When the number of negotiation rounds reaches this maximum, the algorithm forcibly terminates the negotiation process for the current cycle, regardless of whether a strict Nash equilibrium has been reached. ② After each round of negotiation, each RTG checks its own policy and the policies received from other RTGs. If it is detected that the policies of all RTGs have not changed in two consecutive rounds, it is determined that consensus has been reached (i.e., convergence has been achieved), the negotiation is immediately terminated, and each RTG executes its currently determined optimal policy. ③ To cope with communication packet loss, delays, or single-machine computing failures, a timeout timer is set for each round of negotiation and the entire negotiation cycle. If an RTG does not receive policy broadcasts from all neighboring RTGs within the predetermined time, or times out while calculating the optimal response, the RTG will immediately switch to an abnormal handling mode. ④ Negotiation is considered a failure if any of the following occurs: a) After reaching the maximum number of negotiation rounds, the RTGs still fail to reach a consensus; b) An RTG triggers an abnormal handling mode before the negotiation period times out; c) Based on current policy predictions, the minimum distance between any RTGs will be less than the emergency safety distance within a very short time (e.g., 2 seconds). Once negotiation fails, all participating RTGs will automatically execute a predefined, conservative global safety policy. This policy prioritizes absolute safety; for example, all relevant RTGs will decelerate to a safe speed (e.g., 0.5 m / s) or perform an emergency stop, while simultaneously sending an alarm to the central monitoring system. The system will wait for the conflict risk to be resolved or for manual dispatcher intervention before resuming automatic operation. In this implementation, the collaborative obstacle avoidance method ensures that even in the worst-case scenario (algorithm non-convergence, communication interruption), the system can react within a defined time boundary and always prioritize safety, thereby meeting the stringent requirements of the port operation environment for system real-time performance, reliability, and safety.

[0127] (8) The calibration drift detection and compensation algorithm process includes online calibration verification, IMU-assisted calibration compensation and adaptive calibration update.

[0128] Online calibration verification utilizes static reference objects (ground markings, fixed pillars) in the scene to check calibration consistency. Reference features are automatically extracted every 100 frames, and the reprojection error E_reproj=(1 / M)∑_{j=1}^M||u_j-π(K[R|t]P_j)||^2 is calculated. When the error exceeds a preset threshold (3 pixels), the online calibration compensation process is triggered.

[0129] IMU-assisted calibration compensation monitors the vibration acceleration and tilt angle changes of the RTG in real time. Based on the IMU data, it calculates the small changes in the relative pose of the sensor and performs dynamic compensation on the external parameter [R|t]: [R'|t']=ΔT·[R|t], where ΔT is the incremental transformation matrix obtained by IMU integration.

[0130] Adaptive calibration updates initiate background calibration optimization when persistent calibration drift is detected. Using reference feature data from the most recent 100 frames, the extrinsic parameters are re-optimized using the Levenberg-Marquardt algorithm, employing a sliding window strategy to prevent single-frame anomalies from affecting the calibration results. Calibration quality metrics are established: reprojection error, feature matching rate, and calibration convergence speed. When these metrics fall below a threshold, a calibration warning is issued to the operator. Historical calibration data is recorded for predictive maintenance.

[0131] Performance testing was conducted on the RTG multimodal fusion collision avoidance system for large vehicles in this embodiment. The test site was a container yard at a port, covering an area of ​​approximately 100,000 square meters, equipped with 8 RTGs. The testing period was from June to August 2025, a total of 3 months, with a cumulative operating time of approximately 2000 hours. The test environment conditions covered various weather and lighting conditions, including sunny, cloudy, rainy, foggy, daytime, and nighttime. The test results are shown in the table below:

[0132]

[0133] Note: The test conditions were RTG operating speed of 0.5-2.5m / s, target distance of 5-80m, and ambient temperature of -10℃ to 40℃.

[0134] As shown in the table, under clear daytime conditions, the recall rate reaches 98.5%, the precision rate reaches 96.8%, the average positioning error is 0.16m, and the end-to-end delay is 75ms. Under nighttime conditions, the recall rate reaches 96.2%, the precision rate reaches 94.5%, the average positioning error is 0.21m, and the end-to-end delay is 78ms. Under rainy conditions, the recall rate reaches 94.8%, the precision rate reaches 93.1%, the average positioning error is 0.24m, and the end-to-end delay is 82ms. Under foggy conditions, the recall rate reaches 92.3%, the precision rate reaches 90.5%, the average positioning error is 0.28m, and the end-to-end delay is 85ms. Under conditions where the equipment is obstructed, the recall rate reaches 92.1%, the precision rate reaches 89.7%, the average positioning error is 0.19m, and the end-to-end delay is 77ms.

[0135] The time distribution for each stage from sensor data acquisition to output control commands is as follows: sensor data acquisition takes 10ms (LiDAR rotation period 100ms, taken as 1 / 10 frame), data preprocessing takes 12ms (point cloud filtering, downsampling), target detection takes 35ms (fusion network inference), target tracking takes 10ms (Kalman filtering, data association), trajectory prediction takes 5ms (predictive network inference), collision risk assessment takes 3ms (TTC calculation, probability estimation), early warning strategy decision takes 2ms (RL strategy network inference), and control command transmission takes 1ms (CAN bus communication), with a total latency of 78ms. The system meets the real-time early warning requirement (requirement <100ms).

[0136] The trajectory prediction performance test results show that the average displacement error (ADE) for 1-second prediction is 0.12m, and the final displacement error (FDE) is 0.15m; the ADE for 2-second prediction is 0.25m, and the FDE is 0.38m; and the ADE for 3-second prediction is 0.35m, and the FDE is 0.68m. The warning performance test results show that the collision false alarm rate is <0.5%, the false alarm rate is 3.5%, and the average warning lead time is 2.8 seconds. During the 3-month testing period, the system accumulated 2000 hours of operation, processed approximately 7.2 million frames of data, issued 1523 valid warnings, successfully avoided 15 collisions (after manual confirmation), and triggered emergency braking events without error. System availability was 99.2% (downtime was mainly due to planned maintenance), calibration drift occurred 3 times (all were successfully corrected by automatic compensation algorithm without manual recalibration), sensor failure occurred 2 times (1 time was due to camera lens damage, and 1 time was due to LiDAR data cable loosening, both of which were found and repaired during routine maintenance), and software anomalies occurred 0 times (no crashes or deadlocks).

[0137] Example 2

[0138] The difference from Embodiment 1 is that this embodiment provides a low-cost configuration of an RTG large vehicle multimodal fusion collision avoidance system. Hardware simplifications include: using only one forward-looking camera and one 16-line main LiDAR; eliminating side-looking cameras and auxiliary LiDARs; using a low-cost IMU (such as the MPU-6050, with reduced accuracy but meeting basic requirements); using an NVIDIA Jetson Nano (0.5 TOPS computing power, costing approximately 1000 RMB) instead of the Jetson Xavier; and eliminating the 5G module, using only WiFi or 4G for data upload (non-real-time).

[0139] The algorithm simplifications include: using a lighter YOLOv8-n model and further pruning it (reducing the number of channels by 50%), reducing the inference time to 15ms; simplifying point cloud processing to a traditional method based on Euclidean clustering, without using deep learning; removing the Cross-Modal Transformer and using simple late-stage fusion (based on IoU matching); using a linear extrapolation model (constant velocity assumption), without using a Transformer network; and using a fixed threshold for hierarchical warnings, without using reinforcement learning.

[0140] Performance test results show that the detection recall rate is 90% during the day, 80% at night, and 75% in rainy weather (a decrease of approximately 5-15% compared to the configuration in Example 1); the end-to-end latency is 150ms (an increase of approximately 2 times compared to the standard configuration); the false alarm rate is 8% (an increase of approximately 2 times compared to the configuration in Example 1); and the trajectory prediction accuracy with a 3-second ADE is approximately 0.8m (a decrease of approximately 2 times compared to the configuration in Example 1). The total cost of the low-cost solution is approximately 30,000-40,000 RMB (a reduction of approximately 70% compared to the 80,000-120,000 RMB configuration in Example 1), making it suitable for budget-constrained or low-risk scenarios (such as low-speed areas inside a storage yard).

[0141] Compare with Example 1

[0142] Compared to Example 1, which employs a traditional post-result-level fusion method where each sensor detects independently before associating results, this approach fails to fully utilize the complementary information between different modalities. Specifically, the 2D detection branch uses YOLOv8 for image target detection, outputting 2D bounding boxes and categories. The 3D detection branch uses PointPillars for point cloud target detection, outputting 3D bounding boxes and categories. The fusion strategy employs IoU matching, projecting the 3D bounding boxes onto the image plane and calculating the IoU with the 2D bounding boxes; pairs with an IoU greater than 0.5 are considered the same target. The Hungarian algorithm is used to find the optimal assignment. Tracking uses the standard SORT algorithm, employing Kalman filtering and IoU matching. Trajectory prediction uses a linear extrapolation model, assuming the target maintains its current velocity and orientation. The warning strategy uses a fixed threshold, issuing a warning when TTC < 3s.

[0143] Performance test results show that the detection recall rate is 90% during the day, 75% at night, and 65% in rainy weather (a decrease of 8%, 21%, and 30% respectively compared to Example 1); the end-to-end latency is 100ms (an increase of 25ms compared to Example 1); the false alarm rate is 12% (an increase of 8.5% compared to Example 1); and the trajectory prediction accuracy is approximately 0.8m with a 3-second ADE (a decrease of approximately 2 times compared to Example 1). The detection performance of Example 1 significantly decreases under adverse weather conditions, mainly because the later fusion cannot fully utilize the complementary information between modalities, and the association method based on hand-crafted rules has poor robustness in complex scenarios.

[0144] Compare with Example 2

[0145] In contrast to Example 2, a purely vision-based approach is used, employing only a camera for target detection without LiDAR or millimeter-wave radar. Specifically, YOLOv8 is used for 2D target detection, outputting 2D bounding boxes and categories. Target distance is estimated using a monocular depth estimation network (such as MonoDepth2). Tracking utilizes the DeepSORT algorithm, combining appearance and motion features. Trajectory prediction employs an LSTM network. The warning strategy uses a fixed threshold, issuing a warning when TTC < 3s.

[0146] Performance test results show that the detection recall rate is 92% during the day, 68% at night, and 58% in rainy weather (a decrease of 6.5%, 28%, and 37% respectively compared to Example 1); the end-to-end latency is 80ms (an increase of 5ms compared to Example 1); the false alarm rate is 15% (an increase of 11.5% compared to Example 1); and the average positioning error is 0.5m (an increase of 0.34m compared to Example 1). The detection performance of Control Example 2 deteriorates significantly at night and in inclement weather, mainly because the pure vision solution is greatly affected by lighting and weather conditions, and the monocular depth estimation accuracy is low, making it unable to accurately obtain the target's three-dimensional position information.

[0147] In summary, Example 1 employs a Cross-Modal Transformer to deeply fuse 2D vision and 3D point clouds, maintaining high performance under various weather and lighting conditions. Daytime recall reaches 98.5%, nighttime recall reaches 96.2%, and rain / fog recall reaches 94.8%, representing improvements of approximately 8%, 21%, and 30% respectively compared to Control Example 1, and improvements of approximately 6.5%, 28%, and 37% respectively compared to Control Example 2. An adaptive point cloud filtering algorithm designed for the metal reflection environment of the dock, through reflection intensity analysis, geometric consistency verification, and spatiotemporal consistency verification, reduces the false detection rate from 15% to below 2%. Employing sensor redundancy configuration and dynamic fusion weight adjustment strategies effectively addresses scenarios with spreader occlusion, maintaining a target tracking recall rate above 92% even under spreader occlusion conditions, an improvement of approximately 22% compared to Control Example 1. Through online calibration verification and IMU-assisted compensation mechanisms, the reprojection error is maintained within 3 pixels under vibration conditions, a significant reduction compared to the 8-12 pixels of traditional methods. A spatiotemporal joint trajectory prediction network incorporating RTG kinematic constraints achieves an 3-second trajectory prediction error (ADE) of 0.35 meters, a reduction of approximately 60% compared to 0.8 meters in Control Examples 1 and 2. An adaptive warning strategy based on reinforcement learning dynamically adjusts the warning threshold according to scene characteristics and historical data, reducing the false alarm rate from 12% in Control Example 1 and 15% in Control Example 2 to 3.5%, while maintaining a collision false alarm rate below 0.5%. The end-to-end inference latency is 75ms, a reduction of approximately 25% compared to 100ms in Control Example 1, meeting real-time warning requirements. Multi-RTG cooperative obstacle avoidance is supported; through distributed cooperative algorithms and priority negotiation mechanisms, overall yard operation efficiency is improved by 15-20%, while ensuring the safety of multi-machine operations.

[0148] It should be noted that the core of the "multimodal fusion network based on attention mechanism" described in this invention lies in dynamically calculating and fusing features from different modalities (such as 2D images and 3D point clouds) through an attention mechanism to achieve feature complementarity and enhancement. The attention mechanism allows the network to adaptively focus on the most critical information region in another modality corresponding to the current task while processing data from one modality. Those skilled in the art will understand that the network architecture for implementing this fusion is not limited to the Cross-Modal Transformer specifically shown in the embodiments. For example, encoder-decoder structures based on cross-attention, gated attention fusion networks, or other common attention fusion architectures such as introducing spatial and channel attention modules (such as CBAM) into a Feature Pyramid Network (FPN) for multi-scale feature fusion can also be used. As long as the network utilizes the attention mechanism to achieve dynamic interaction and weight allocation between 2D and 3D features, it falls within the core conceptual scope of this invention for fusion networks.

[0149] It should be noted that the core of the "adaptive decision-making strategy based on a reinforcement learning model" described in this invention lies in dynamically optimizing the collision avoidance warning strategy through interactive learning with the environment, in order to balance safety, efficiency, and comfort. The reinforcement learning model defines a state space, an action space, and a reward function, and the agent (in this system, an edge computing unit) learns a mapping strategy from the environmental state to the optimal action. Those skilled in the art will understand that the algorithm for implementing this adaptive decision-making is not limited to the Soft Actor-Critic (SAC) algorithm specifically shown in the embodiments. For example, other common deep reinforcement learning algorithms such as Deep Deterministic Policy Gradient (DDPG), Proximal Policy Optimization (PPO), Dual-Delay Deep Deterministic Policy Gradient (TD3), or Deep Q-Network (DQN) and its variants based on Q-learning can also be used. As long as the strategy is trained through a reinforcement learning framework and can adaptively generate warning, speed limit, or braking commands based on real-time assessed collision risks, it falls within the core conceptual scope of the adaptive decision-making strategy of this invention.

Claims

1. A multimodal fusion collision avoidance method for large RTG vehicles, characterized in that, Includes the following steps: Step S1: In real time, acquire two-dimensional image data of the surrounding environment of the RTG vehicle collected by at least one vision sensor module, and three-dimensional point cloud data collected by at least one three-dimensional sensor module. Step S2: Based on the reflection intensity and spatiotemporal consistency of the three-dimensional point cloud data, perform adaptive preprocessing on the three-dimensional point cloud data; Step S3: Input the preprocessed 3D point cloud data and the 2D image data into a multimodal fusion network based on an attention mechanism. The multimodal fusion network performs feature extraction and deep fusion on the 3D point cloud data and the 2D image data to obtain fused features. Step S4: Based on the fused features, detect and identify targets around the RTG vehicle, generate target detection results through a unified detection head, and output motion trajectories with unique identifiers through a multi-target tracking algorithm; Step S5: Based on the target's historical motion trajectory, predict the target's future motion trajectory; Step S6: Based on the future motion trajectory, calculate the collision risk between the RTG vehicle and the target, and according to the collision risk, generate and output corresponding collision avoidance control commands to the RTG control system through an adaptive decision-making strategy based on a reinforcement learning model.

2. The RTG large vehicle multimodal fusion collision avoidance method according to claim 1, characterized in that, The specific process of step S2 includes: Step S21: Calculate the reflection intensity of each point in the three-dimensional point cloud data, and mark the points whose reflection intensity exceeds a preset statistical threshold as abnormally high reflection points; Step S22: Perform a geometric consistency check on the abnormally high reflectivity points. Points whose local plane fitting residual with neighboring points is greater than a preset geometric consistency threshold are identified as false reflectivity points and removed. Step S23: Utilize the temporal correlation of point clouds across multiple consecutive frames to perform spatiotemporal consistency verification on suspected false reflection points and eliminate points whose frequency of occurrence is lower than a preset frequency threshold.

3. The RTG large vehicle multimodal fusion collision avoidance method according to claim 1, characterized in that, In step S3, the multimodal fusion network is a Cross-Modal Transformer network, which calculates the attention weights between two-dimensional image features and three-dimensional point cloud features through a cross-modal bidirectional attention module, thereby achieving feature complementarity enhancement between the two-dimensional image data and the three-dimensional point cloud data.

4. The RTG large vehicle multimodal fusion collision avoidance method according to claim 1, characterized in that, If the three-dimensional sensor module has two or more components, the step S3 may include the following before implementation: Step S31: Acquire the pose information of the RTG lifting device in real time, and calculate the dynamic occlusion of the monitoring area by each of the three-dimensional sensor modules based on the pose information; Step S32: Based on the dynamic occlusion situation, dynamically adjust the weights of the 3D point cloud data from different 3D sensor modules during feature fusion.

5. The RTG large vehicle multimodal fusion collision avoidance method according to claim 1, characterized in that, Before step S4 is performed, the following also includes: Step S4A1: Real-time acquisition of motion attitude data of the RTG vehicle collected by the attitude sensor module, and motion compensation of the three-dimensional point cloud data using the motion attitude data; Step S4A2: Extract the corresponding feature points from the 3D point cloud data and the 2D image, and calculate the reprojection error; Step S4A3: When the reprojection error exceeds a preset threshold, an online calibration compensation process is triggered to correct the extrinsic parameters between the 3D sensor module and the vision sensor module.

6. The RTG large vehicle multimodal fusion collision avoidance method according to claim 3, characterized in that, Step S5 includes: Step S51: Input the historical motion trajectory of the target into the Transformer encoder to capture its temporal dependencies; Step S52: Encode the kinematic constraints of the RTG vehicle into a vector and fuse it with the output of the Transformer encoder. The kinematic constraints include at least one of track constraints, speed limits, and acceleration limits. Step S53: Based on the fused features, decode and generate multiple probabilistic future motion trajectories, and output the covariance matrix of each future motion trajectory.

7. The RTG large vehicle multimodal fusion collision avoidance method according to claim 1, characterized in that, In step S6, the adaptive decision-making strategy based on the reinforcement learning model is trained using the Soft Actor-Critic algorithm. The state space of the adaptive decision-making strategy includes the target state, RTG state and environment state, and the action space includes no operation, warning, speed limit and emergency braking. And / or, in step S6, when calculating the collision risk, the time to collision and the collision probability are evaluated together, and the target is divided into different risk levels according to the evaluation results.

8. The RTG large vehicle multimodal fusion collision avoidance method according to any one of claims 1 to 7, characterized in that, When multiple RTGs exist within the storage yard besides this RTG, cooperative obstacle avoidance operations are also included, specifically: Step S71: Interact with other RTGs in the yard to exchange status information and trajectory prediction information; Step S72: Based on the information exchanged, calculate the potential conflicts between multiple machines using a distributed collaborative algorithm; Step S73: Determine and execute the obstacle avoidance strategy of this RTG according to the negotiated priority.

9. A multimodal fusion collision avoidance system for RTG large vehicles, characterized in that, include: The vision sensor module is used to acquire two-dimensional image data of the surrounding environment of the RTG truck; The 3D sensor module is used to collect 3D point cloud data of the surrounding environment of the RTG truck. The edge computing unit is communicatively connected to the vision sensor module and the 3D sensor module, and is configured as follows: Receive the two-dimensional image data and the three-dimensional point cloud data; Based on the reflection intensity and spatiotemporal consistency of the three-dimensional point cloud data, adaptive preprocessing is performed on the three-dimensional point cloud data; The preprocessed 3D point cloud data and the 2D image data are input into a multimodal fusion network based on an attention mechanism for feature extraction and deep fusion to obtain fused features; Based on the fusion features, targets around the RTG vehicle are detected and identified. Target detection results are generated through a unified detection head, and motion trajectories with unique identifiers are output through a multi-target tracking algorithm. Based on the target's historical trajectory, predict the target's future trajectory; Based on the future trajectory, the collision risk between the RTG vehicle and the target is calculated, and according to the collision risk, an adaptive decision-making strategy based on a reinforcement learning model is used to generate and output corresponding collision avoidance control commands.

10. The RTG large vehicle multimodal fusion collision avoidance system according to claim 9, characterized in that, Also includes: An attitude sensor module is used to monitor the motion attitude data of the RTG vehicle. The edge computing unit synchronously receives the motion posture data collected by the posture sensor module and uses the posture data to perform motion compensation on the three-dimensional point cloud data. Extract the corresponding feature points from the 3D point cloud data and the 2D image, and calculate the reprojection error; When the reprojection error exceeds a preset threshold, an online calibration compensation process is triggered to correct the extrinsic parameters between the 3D sensor module and the vision sensor module.

Images