Method and system for predicting motion track of crane based on RGBD image

Abstract

The invention discloses a crane motion track prediction method based on an RGBD image, and belongs to the technical field of crane intelligent control, and the crane motion track prediction method comprises the following steps: S1, synchronously collecting multi-modal data, deploying a binocular RGBD camera array, and obtaining an RGB image and depth point cloud data of a crane operation area in real time; s2, cross-modal feature alignment and fusion are carried out; s3, track prediction and physical constraint correction; and S4, calculating the minimum safety distance between the crane and the dynamic object in real time. According to the method, a cross-modal space-time alignment mechanism is constructed by integrating the binocular RGBD camera array, the millimeter wave radar, the laser range finder and IMU data, the deformable convolutional network is adopted to dynamically compensate sensor data offset, multi-source data conflicts are eliminated, and compared with a traditional single sensor scheme, the method has the advantages that the method is simple in structure and convenient to operate. The multi-mode cooperative mechanism effectively solves the problem of sensing failure in a sensor blind area and a severe environment.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent control of cranes, and particularly to a method and system for predicting the motion trajectory of a crane based on RGBD images. Background Art

[0002] In the fields of modern industrial production and logistics transportation, as a key material handling equipment, the safety and efficiency of the operation of cranes have a decisive impact on the overall production process. With the rapid development of automation and intelligent technologies, the control system of cranes is undergoing a profound transformation.

[0003] Traditional crane control systems mostly rely on a single data source (such as a two-dimensional camera or a millimeter-wave radar), lacking a multi-sensor cooperation mechanism. For example, the trajectory prediction scheme proposed in Chinese Patent CN114275681B is only based on historical slewing speed and static structure parameters, without fusing depth information or inertial data, resulting in limited dynamic compensation ability. In complex working conditions (such as counterweight occlusion, rain, fog, or strong light interference), a single sensor is prone to perception failure due to blind spots or environmental interference, with a relatively high miss detection rate.

[0004] Existing solutions have not effectively solved the problem of unifying the coordinate systems of multi-sensors, and conflicts are prone to occur due to spatio-temporal deviations during data fusion. For example, Chinese Patent CN116253248B uses a fixed camera or a millimeter-wave radar, but lacks a cross-modal spatio-temporal alignment mechanism, resulting in insufficient trajectory prediction accuracy for dynamic objects in occluded areas or during high-speed movement. In addition, sensor data is not combined with physical constraints (such as sling elastic deformation, joint torque limits), and there are deviations between the prediction results and the actual mechanical motion laws. Summary of the Invention

[0005] Aiming at the above-mentioned shortcomings of the existing technology, the present invention provides a method and system for predicting the motion trajectory of a crane based on RGBD images.

[0006] To achieve the above objectives, the present invention is realized through the following technical solutions: A method for predicting the motion trajectory of a crane based on RGBD images, comprising the following steps: S1: Synchronously collect multi-modal data, deploy a binocular RGBD camera array, and obtain RGB images and depth point cloud data of the crane operation area in real time; Transmit electromagnetic waves at a frequency of 75 - 80 GHz through a millimeter-wave radar and receive the reflected waves to measure the distance and speed of dynamic objects; Transmit laser pulses with an accuracy of ±2 mm through a laser rangefinder and receive the reflected light to measure the distance information of dynamic objects; Implement time synchronization of the IEEE 1588v2 protocol through FPGA hardware to ensure the time consistency of all sensor data; S2: Cross-modal feature alignment and fusion. Use a deformable convolutional network to generate the deformation field parameter Δp, and the calculation formula is:

[0007] where F rgb is the RGB feature map, F depth is the depth feature map, T imu is the inertial measurement unit data, and Conv 1×1 is a 1×1 convolution operation; When a dynamic object enters the sensor blind area, predict its acceleration through IMU data and dynamically adjust Δp to compensate for trajectory deviation; S3: Trajectory prediction and physical constraint correction. Build a spatio-temporal graph convolutional model (a hybrid architecture of TCN-GAT), input the fused multi-modal data, and output the crane movement trajectory within the next 2 - 4 seconds. Verify the feasibility of the trajectory through the rigid body dynamics equation, and the constraint conditions are:

[0008] where J T is the transpose of the Jacobian matrix, and F ext is the external force; S4: Real-time calculation of the minimum safety distance between the crane and the dynamic object:

[0009] where , M is the mass of the suspended item, and t react = 0.2s is the system response time; S5: Hierarchical braking control. When a dynamic object enters the 1.5D safe range, trigger an audible and visual warning. When the dynamic object breaks through Dsafe and the operator does not respond, implement an emergency stop through the redundant braking unit.

[0010] Furthermore, the cross-modal feature alignment further includes: Sensor outlier removal. Detect data consistency through Mahalanobis distance and remove abnormal data:

[0011] where X is the sensor data vector, μ is the mean vector, and Σ is the covariance matrix; Unify the radar and lidar ranging data to the crane base coordinate system, and the transformation matrix is:

[0012] where R is the rotation matrix and t is the translation vector.

[0013] Furthermore, the working mode of the spatio-temporal graph convolution model includes: a gated TCN module that uses three layers of dilated convolution, with each layer containing 32 3×3 convolutional kernels; Calculating node weights through the multi-head attention mechanism:

[0014] where W is a learnable weight matrix, and h i , h j are node feature vectors.

[0015] Furthermore, the physical constraint correction process includes: Elastic deformation compensation, dynamically adjusting the amount of deformation according to the sling tension FF:

[0016] Trajectory reparameterization, when the joint torque exceeds the threshold, using the sequential quadratic programming (SQP) algorithm to optimize the trajectory, with a convergence threshold of 1e-4 and the number of iterations ≤ 50 times.

[0017] Furthermore, the working logic of the redundant braking unit includes: Main brake failure determination, the main brake unit sends a heartbeat packet through the CAN bus every 100 ms. If it is lost continuously for 3 times, it is determined to have failed; Dual-coil solenoid valve drive, and the drive current formula is:

[0018] where V brake is the brake pedal voltage.

[0019] A crane motion trajectory prediction system for a crane motion trajectory prediction method based on RGBD images as described above, includes: A multi-modal sensing module, with RGBD cameras distributed in an isosceles triangle, and the baseline length is:

[0020] Millimeter-wave radars are installed at the four corners of the crane turntable, covering a 360° horizontal field of view; An edge computing unit, which reduces the calculation latency through model quantization and operator fusion technologies to achieve real-time trajectory prediction; A redundant braking module, including a main brake and a redundant brake, and realizing dual-system hot standby through a brake switching unit; A safety assessment module, which calculates the trajectory confidence in real time and triggers the backup prediction mode when the confidence is lower than the threshold.

[0021] Furthermore, the deployment method of the multi-modal sensing module is the RGBD camera baseline length,

[0022] Among them, z = 10 m is the typical working distance, f = 8 mm is the camera focal length, and d min = 0.1 m is the minimum recognizable obstacle size, and N pixel = 5 is the minimum number of pixels occupied by the target when the detection confidence level is ≥95%; Installation of millimeter-wave radar. Four groups of radars are installed at the four corners of the crane turntable with a horizontal inclination angle θ = 15° and a vertical depression angle ϕ = 10°, covering a 360° horizontal field of view angle, with a maximum detection distance of 80 m and an accuracy of ±0.02 m; Calibration of the laser rangefinder. The optical axis offset is corrected in real time through IMU data, with an error tolerance of ±0.05°. In rainy and foggy environments, a multi-scale MRF model is used to suppress noise.

[0023] Furthermore, the edge computing unit performs as follows: Model quantization. Dynamically quantize the FP32 floating-point weights into the INT8 format. The dynamic range of the calibration set is [-2.5, 3.8]. Optimize the truncation threshold through the KL divergence algorithm, and the quantization error rate ≤ 1.5%; Operator fusion. Combine the convolution, batch normalization, and ReLU activation layers into a single computing core. The formula is:

[0024] Among them, Conv is convolution and BN is batch normalization.

[0025] Furthermore, the safety assessment module includes: Trajectory deviation detection. Calculate the Hausdorff distance between the predicted trajectory P and the real-time observed trajectory Q:

[0026] When H(P, Q) > 0.5 m, trigger confidence degradation and start redundant braking; Environmental complexity index. The formula is:

[0027] The dynamic coefficient increases according to the movement speed. The safety margin = 1.2 - 0.05M. When ECI ≥ 2.0, the warning threshold is tightened to 70% of the normal value; when ECI < 1.0, the threshold is relaxed to 130%.

[0028] Furthermore, the test protocol of the redundant braking module includes: Perform full-load emergency stop and no-load fine movement tests every month, and record t 实测 and t 标称 , and the judgment condition is:

[0029] When Δt > 15%, a maintenance alarm is triggered; The data is uploaded to the cloud by encrypting and transmitting the pressure curve, current waveform, and temperature data through the 5G NR protocol. The JSON format includes the timestamp, device ID, and error code; Abnormal data is synchronized to the work order system in real time, and a maintenance task is automatically generated. Technicians need to replace the faulty components on site within 72 hours.

[0030] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. By integrating a binocular RGBD camera array, millimeter-wave radar, laser rangefinder, and IMU data, the present invention constructs a cross-modal spatio-temporal alignment mechanism, adopts a deformable convolutional network to dynamically compensate for sensor data offset, and eliminates multi-source data conflicts. Compared with traditional single-sensor solutions, the multi-modal collaboration mechanism of the present invention effectively solves the problems of sensor blind spots and perception failure in harsh environments, realizes 360° full-view coverage, reduces the dynamic target missed detection rate, and improves the positioning accuracy.

[0031] 2. The present invention has high-precision spatio-temporal modeling and trajectory prediction. Based on the spatio-temporal graph convolutional model, combined with gated dilated convolution and multi-head attention mechanism, it accurately captures the long-term temporal dependencies of crane movements and the complex spatial relationships between dynamic objects. By introducing rigid body dynamics equations to verify the feasibility of the trajectory and compensating for the elastic deformation of the sling in real time, it solves the problem of trajectory deviation caused by traditional models ignoring physical constraints.

[0032] 3. The load-adaptive dynamic safety control of the present invention proposes a dynamic calculation model for the safety distance based on the load mass, speed, and acceleration, replacing the traditional fixed threshold scheme. By real-time feedback of load parameters through a hook weighing sensor and combining edge computing optimization, it realizes the dynamic matching of the safety distance and mechanical constraints, reduces the collision false alarm rate under heavy load conditions, significantly improves compared with traditional schemes, and the warning distance is adaptively adjusted, enhancing the safety margin.

[0033] 4. Through the design of a dual-coil solenoid valve and the CAN bus heartbeat detection mechanism, the present invention realizes seamless switching between the main / backup braking units, improves the response efficiency compared with the prior art, and combines the regular self-checking protocol and 5G cloud data monitoring to solve the risk of braking failure caused by long-term idleness. The system failure rate approaches zero, and the maintenance response time limit is strictly controlled within 72 hours.

[0034] 5. Through model quantization and operator fusion technology, the present invention compresses the inference delay to, reduces the memory occupancy, and combines KL divergence optimization truncation threshold and multi-sensor outlier rejection mechanism to ensure the stability of data fusion. Detailed implementation manners

[0035] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be described clearly and completely below. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention. Embodiment

[0036] A method for predicting the motion trajectory of a crane based on RGBD images includes the following steps: S1: Synchronously collect multi-modal data, deploy a binocular RGBD camera array, and obtain RGB images and depth point cloud data of the crane operation area in real time; The binocular RGBD camera calculates depth information through the parallax of two cameras, and combines RGB images to provide rich visual features and spatial information. Its working principle is based on the parallax principle, that is, two cameras observe the same object from different positions, and depth information is obtained by calculating the parallax, that is:

[0037] By calculating the parallax, the depth information of the object can be obtained. The binocular camera combines RGB images to provide rich visual features and spatial information, and is suitable for 3D reconstruction and object recognition under complex working conditions.

[0038] The millimeter-wave radar emits electromagnetic waves at a frequency of 75 - 80 GHz and receives the reflected waves to measure the distance and speed of dynamic objects; The millimeter-wave radar emits electromagnetic waves at a frequency of 75 - 80 GHz and receives the reflected waves to measure the distance and speed of dynamic objects. Its working principle is based on frequency-modulated continuous wave (FMCW) technology. By analyzing the frequency change of the reflected wave, the distance and speed of the target are calculated. FMCW emits a signal with a linearly changing frequency over time and analyzes the frequency change of the reflected wave to calculate the distance and speed of the target. The specific formula is as follows:

[0039] where c is the speed of light, Δf is the frequency difference, K is the frequency modulation slope, f 0 is the carrier frequency, v is the target speed. The millimeter-wave radar has strong robustness to ambient light and weather changes and is suitable for real-time perception of dynamic objects.

[0040] The laser rangefinder emits laser pulses with an accuracy of ±2 mm and receives the reflected light to measure the distance information of dynamic objects; The laser rangefinder measures the distance by emitting laser pulses and receiving the reflected light, using the Time-of-Flight (ToF) technology. ToF calculates the distance by measuring the time difference between the emission and reflection of the laser pulse. The formula is as follows:

[0041] where c is the speed of light and t is the time difference. The laser rangefinder has a high precision of ±2mm and is suitable for high-precision distance measurement.

[0042] Time synchronization of the IEEE 1588v2 protocol is implemented through FPGA hardware to ensure the time consistency of all sensor data. The specific method is as follows: A1. Optimize the recording accuracy of timestamps. Implement a hardware timestamp module in the FPGA to directly detect and mark the timestamps of IEEE 1588 messages between the Ethernet physical layer and the MAC layer to ensure the accuracy of timestamps; A2. Use a high-precision clock source (such as a 200MHz clock) and frequency multiplication processing technology to improve the resolution of timestamps. The specific implementation method is as follows: Use a high-precision crystal oscillator as the clock source, such as a 200MHz crystal oscillator; Perform frequency multiplication processing through the Phase-Locked Loop (PLL) of the FPGA to increase the clock frequency to a higher frequency, such as 400MHz; Record the timestamp within each crystal oscillator cycle; A3. Design a dynamic frequency compensation module to adjust the frequency of the local system clock in real time to reduce the frequency deviation between the master and slave clocks. By measuring the frequency deviation between the master and slave clocks, calculate the frequency offset that needs to be adjusted, and calculate the frequency adjustment coefficient according to the following formula:

[0043] where t1, t2, t3, and t4 are the timestamps of the master and slave clocks, and T is the synchronization period; Dynamically adjust the frequency of the local clock according to the calculated frequency adjustment coefficient to ensure that the frequencies of the master and slave clocks are consistent; A4. Transmission delay compensation. By interacting with Delay_Req and Delay_Resp messages, measure the transmission delay between the master and slave clocks. According to the measured delay value, compensate the timestamp of the slave clock to ensure the accuracy of time synchronization. The specific formula is as follows:

[0044] According to the calculated delay value, compensate the timestamp of the slave clock to ensure the accuracy of time synchronization. Adopt a transparent clock architecture to reduce the clock deviation caused by multi-hop routing. The transparent clock is one of End to End or Peer to Peer. The specific method is as follows: E2E (End to End) transparent clock. The correction of the E2E transparent clock for the delay only includes the residence time of this node. By measuring the round-trip delay (RTT) between the master and slave clocks, calculate the path delay and use it for the correction of the time deviation. The formula is as follows: E2E (End to End) transparent clock

[0045] Among them, D is the path delay, and C1 is the queuing time in the direction from the master clock to the slave clock; P2P (Peer to Peer) transparent clock. The correction of the P2P transparent clock for the delay includes not only the residence time of this node but also adds the delay on the transmission path. Measure the path delay through PDelay_Req and PDelay_Resp messages and put it into the correctionField; The formula is as follows:

[0046] Among them, X is the path delay from the master clock to the transparent clock, Y is the path delay from the transparent clock to the slave clock, C1 is the queuing time in the direction from the master clock to the transparent clock. The transparent clock is usually implemented in the switch, records the timestamp and calculates the path delay through the FPGA or dedicated hardware module. On the switch supporting the PTP protocol, the transparent clock function is implemented through firmware update or configuration; By the above method, reduce the clock deviation caused by multi-hop routing, improve the time synchronization accuracy of the entire network, and thus ensure the time consistency of all sensor data.

[0047] S2: Cross-modal feature alignment and fusion. Use a deformable convolutional network to generate the deformation field parameter Δp. The calculation formula is:

[0048] Among them, F rgb is the RGB feature map, F depth is the depth feature map, T imu is the inertial measurement unit data, Conv 1×1 is a 1×1 convolution operation; When a dynamic object enters the sensor blind area, predict its acceleration through the IMU data and dynamically adjust Δp to compensate for the trajectory deviation; Extract visual features from RGB images through a Convolutional Neural Network (CNN) to capture the appearance and texture information of objects. Extract geometric features from depth point cloud data through a depth perception network (such as a depth convolutional network) to capture the spatial position and shape information of objects. Obtain acceleration and angular velocity data through an Inertial Measurement Unit (IMU) to provide motion information of dynamic objects. Concatenate the extracted RGB feature maps, depth feature maps, and IMU data to form a multi-modal feature map:

[0049] Fuse the concatenated feature maps through 1×1 convolutional operations to generate deformation field parameters Δp:

[0050] The role of 1×1 convolution is to reduce the number of channels of the feature map while keeping the spatial resolution unchanged, thus achieving efficient feature fusion; The deformable convolutional network dynamically adjusts the sampling positions of the convolutional kernels by learning the deformation field parameters Δp, so as to better capture the spatial variations in the feature map. Specifically, input the multi-modal feature map F concat into the deformable convolutional network, and generate deformation field parameters Δp through 1×1 convolutional operations for adjusting the sampling positions of the convolutional kernels:

[0051] Adjust the sampling positions of the convolutional kernels according to Δp to ensure that the sampling positions of the feature map can adapt to the spatial distribution differences of different modal data; When a dynamic object enters the sensor blind area, predict its acceleration through IMU data, dynamically adjust Δp to compensate for trajectory deviation, and predict the acceleration of the dynamic object through the Kalman filtering algorithm:

[0052] where is the predicted acceleration, A is the state transition matrix, B is the control input matrix, is the control input. According to the predicted acceleration, dynamically adjust the deformation field parameter to compensate for trajectory deviation: where α is the adjustment coefficient, is the predicted acceleration change; By fusing RGB, depth, and IMU data, the system can provide more comprehensive perception capabilities under complex working conditions (such as occlusion or bad weather). It predicts the acceleration of dynamic objects through IMU data, dynamically adjusts the deformation field parameter Δp, compensates for trajectory deviation, and improves the accuracy of trajectory prediction. In the case of sensor blind spots or data loss, IMU data can provide reliable dynamic information to ensure the robustness and stability of the system. The 1×1 convolution operation can effectively reduce the number of channels of the feature map, reduce the computational complexity, and keep the spatial resolution unchanged.

[0053] S3: Trajectory prediction and physical constraint correction. Build a spatio-temporal graph convolution model (a hybrid architecture of TCN-GAT), input the fused multi-modal data, and output the crane motion trajectory within the next 2 - 4 seconds. Verify the feasibility of the trajectory through the rigid body dynamics equation. The constraint conditions are:

[0054] Among them, is the transpose of the Jacobian matrix, is the external force, represents the maximum allowable joint moment (or torque), and the unit is Newton-meter (N·m); Extract features from RGB images, depth point cloud data, and IMU data respectively, splice the extracted features to form a multi-modal feature map, and fuse the spliced feature map through a 1×1 convolution operation to generate the deformation field parameter Δp. The spatio-temporal graph convolution model is: Spatio-temporal graph convolution network, capturing the time dependence in time series data; Graph attention network, modeling the relationship between features and capturing spatial dependence; Trajectory prediction, using an LSTM network to predict the crane motion trajectory within the next 2 - 4 seconds; The trajectory adjustment moment for physical constraint correction is calculated as follows: Calculate the joint moment to ensure that it does not exceed the maximum allowable value. If the predicted trajectory exceeds the physical constraints, adjust the trajectory parameters to ensure compliance with the constraint conditions; Use a long short-term memory network (LSTM) to predict the crane motion trajectory within the next 2 - 4 seconds, verify the feasibility of the trajectory through the rigid body dynamics equation, ensure that the predicted trajectory conforms to the physical constraints, calculate the transpose of the Jacobian matrix, and convert the external force into the force in the joint space:

[0055] Among them, x, y, z are the positions of the end effector, and q1, q2, …, qn are the joint variables; External force F extIncluding gravity, friction, and externally applied forces, ensure that the joint torque does not exceed the maximum allowable value of 150 N·m. If the predicted trajectory exceeds the physical constraints, adjust the trajectory parameters to make it comply with the constraints, and calculate the joint torque:

[0056] If τ > 150·m, adjust the trajectory parameters to ensure τ ≤ 150 N·m; Through the above technical means, trajectory prediction and physical constraint correction based on the TCN-GAT hybrid architecture can significantly improve the accuracy and physical feasibility of trajectory prediction, providing reliable technical support for high-precision requirements in complex dynamic environments; S4: Real-time calculate the minimum safety distance between the crane and the dynamic object:

[0057] Among them, , M is the mass of the suspended item, t react = 0.2 s is the system response time; Measure in real time through the hook weighing sensor, and the mass data is uploaded via the CAN bus at a frequency of 500 Hz. The coefficient 0.8 and the denominator 50 are calibrated based on experiments: when M = 50 kg, k1 = 0.8, ensuring that the safety margin under medium load is 80% of the empirical value; the denominator 50 is used to normalize the mass effect to avoid too large a parameter range; M / 100 reflects the gain effect of load inertia on the braking distance. When M > 100, k 2 Non-linearly grows to match the braking requirements under high-inertia conditions of the crane; V is the instantaneous velocity of the dynamic object relative to the crane (unit: m / s), obtained by parsing the FMCW signal of the millimeter-wave radar. The formula is:

[0058] Among them, f 0 Is the carrier frequency, and Δf is the Doppler frequency shift; a is the acceleration of the dynamic object (unit: m / s²), predicted in real time by the Kalman filter from the IMU data. The prediction equation is:

[0059] Among them, A is the state transition matrix and B is the control input matrix; t react = 0.2 s is the system response time, including sensor data acquisition (10 ms), edge computing inference (80 ms), braking instruction transmission (10 ms), and redundant brake action delay (100 ms). Ensure that the timing error of each link ≤ 1 ms through FPGA hardware-level timestamp synchronization; S5: Hierarchical braking control. When a dynamic object enters the 1.5D safe range, an audible and visual warning is triggered. When the dynamic object breaks through D safe and the operator does not respond, an emergency stop is achieved through the redundant braking unit; Furthermore, the cross-modal feature alignment further includes: Sensor outlier rejection. The consistency of the sensors is detected through Mahalanobis distance, and outliers with D M > 2.5 are rejected:

[0060] where X is the sensor data vector, μ is the mean vector, and Σ is the covariance matrix; Unify the radar and laser ranging data to the crane base coordinate system. The transformation matrix is:

[0061] where R is the rotation matrix and t is the translation vector.

[0062] The radar speed measurement and the IMU predicted acceleration are fused through weighted averaging, with weights w 雷达 = 0.7, w IMU = 0.3 to reduce the error influence of the radar in occluded scenarios. When the sling swing causes instantaneous noise in the load cell, a sliding window filter (window length 10 frames, with an exponentially decaying weight) is used to smooth the mass data, which is updated every 50 ms D safe , and the calculation result is verified by the safety assessment module; When a dynamic object enters the 1.5D safe range, an audible and visual warning is triggered; If the object breaks through D safe and the operator does not respond within 0.5 s, the redundant braking unit is activated, and the braking acceleration a brake = 5 m / s 2 , ensuring a complete stop within time; Verify the joint torque through the transpose of the Jacobian matrix J T ;

[0063] If τ > 150 N·m, trigger trajectory reparameterization (SQP algorithm iteration ≤ 50 times) to ensure compatibility between safety distance calculation and mechanical limit; In rainy and foggy environments, the laser rangefinder suppresses noise through a multi-scale Markov random field (MRF) model to ensure that the measurement errors of a and V are ≤ 5%; For example, light load, M = 30 kg;

[0064] The system gives an early warning when the object enters 0.774 m. And due to the large load inertia, the braking instruction is issued 20% in advance, and k is dynamically corrected by the mass M 1 、k 2 , replacing the traditional fixed threshold, making the safety distance match the actual working conditions, and performing weighted fusion of radar, IMU, and weighing data to improve the measurement robustness of V and a. FPGA hardware synchronization and edge computing optimization (INT8 quantization + operator fusion) ensure that the full process calculation is completed within a 50 ms cycle; When the laser rangefinder detects that the rain and fog interference level L rain > 3 (judgment of the reflection intensity threshold), the upper limit of the braking acceleration is adjusted:

[0065] The compensation term added to the solenoid valve drive current is:

[0066] Where: ΔI is the compensation term of the drive current; L rain is the rain and fog interference level; When the rain and fog interference level L rain exceeds 3, the system automatically adjusts the upper limit of the braking acceleration a max to 4.0 m / s 2 to meet the braking requirements in the rain and fog environment.

[0067] At the same time, a compensation term ΔI is added to the drive current of the solenoid valve, and its value is 0.2 times the rain and fog interference level L rain to offset the influence of the increased viscosity of the hydraulic oil.

[0068] The working mode of the spatio-temporal graph convolution model includes: a gated TCN module, which uses three layers of dilated convolution, with 32 3×3 convolutional kernels in each layer, and the number of input and output channels remains the same; Calculate the node weights through the multi-head attention mechanism:

[0069] e ij represents the attention score (or edge weight) between node i and node j; LeakyReLU represents the ReLU activation function with leakage; α represents the negative slope parameter in the ReLU activation function with leakage, which is given as 0.2 here; W represents the learnable weight matrix for linearly transforming the node features; h i and h j represent the feature vectors of node i and node j respectively.

[0070] α T Denotes the transpose of the weight vector α; For each layer, dilation factors d = 1, 2, 4 are adopted respectively, and the receptive field sizes of the corresponding layers are 3×3, 7×7, 15×15. By exponentially amplifying the perceptual range in the time dimension, the long-term temporal dependence features of the crane motion trajectory are captured, and the receptive field is exponentially enlarged layer by layer to effectively capture the long-term temporal dependence features of the crane motion. Each layer is configured with 32 3×3 3×3 convolutional kernels, and the output is calculated through the dilated convolution formula:

[0071] where w k is the convolutional kernel weight, x t−d·k is the delayed sampling point of the input sequence, and K = 3 is the convolutional kernel size; A gating mechanism is introduced in the output of each layer to dynamically adjust the information flow and suppress noise interference. The calculation formula is:

[0072] W g1 and W g2 are learnable parameter matrices, ⊙ represents element-wise multiplication, and σ is the Sigmoid function. The gating mechanism enhances the adaptability of the model to dynamic working conditions (such as sudden changes in wind speed or load swing) through non-linear interaction, and dynamically allocates node interaction weights through the multi-head attention mechanism. The specific steps are as follows: 1. The node feature vectors hi, hj (dimension d model = 64) are linearly transformed by the weight matrix W ∈ R dmodel×dmodel and then concatenated. The concatenation operation is:

[0073] The concatenated vector captures the joint features of nodes i and j, providing a spatial correlation representation for attention calculation; is a learnable parameter vector, and α = 0.2 alleviates the problem of gradient vanishing; 2. Split e ij into H = 4 independent heads, and each head calculates local attention:

[0074] h = 1, …, H scaling factor, used to stabilize the training process; Normalize the attention coefficients of each head:

[0075] Denote the attention coefficient from node $i$ to node $j$ in the $h$-th layer attention mechanism; Softmax is a function used to convert a set of real numbers into a probability distribution, that is, the output values of the function are all between 0 and 1, and the sum of all output values is 1; Denote the unnormalized attention score (or attention score) from node $i$ to node $j$ in the $h$-th layer attention mechanism; Exp represents the exponential function, that is, the power of the base $e$ of the natural logarithm; $N$ i Denote the set of neighboring nodes of node $i$, which are dynamically constructed based on the crane joint topology and obstacle positions; Denote the unnormalized attention score from node $i$ to node $k$ within its neighborhood in the $h$-th layer; Denote the summation over all neighboring nodes $k$ of node $i$; 3. The attention weights of each head are weighted and summed for the node features, and the results are concatenated and output:

[0076] Are the independent weight matrices for each head; Final output Fuse multi-perspective spatial dependency relationships; Denote the output feature vector of node $i$ after being processed by the multi-head attention mechanism; $H$ denotes the number of heads in the multi-head attention mechanism; Denote the summation over all neighboring nodes $k$ of node $i$; Denote the attention coefficient from node $i$ to node $j$ in the $h$-th head; : Denote the learnable weight matrix of the $h$-th head, which is used for linear transformation of node features.

[0077] $h$ j Denote the input feature vector of node $j$; Specifically, the time features (dimension 64) extracted by the TCN module are used as the initial features $h$ of the nodes i ; Construct the spatio-temporal graph adjacency matrix $N$ according to the crane manipulator joint topology and dynamic object positions i ; Through the multi-head attention mechanism, adaptively allocate the interaction weights between joint nodes and obstacle nodes; The aggregated features are added to the output of the TCN to retain the original temporal information:

[0078] h final represents the final node feature vector, which is obtained by adding the features aggregated by the attention mechanism to the output features of the TCN; represents the node feature vector after being processed by the attention mechanism. This vector contains the information obtained by weighted summing the neighborhood node features with the attention weights of different heads; h TCN represents the output feature vector of the Temporal Convolutional Network (TCN), which is used to capture the temporal dependencies in time series data. Here, it processes the time series information of node features; Dilated convolutions capture long-term temporal dependencies, and multi-head attention enhances local spatial sensitivity. The node interaction weights are adjusted in real-time through attention coefficients to match the requirements of complex working conditions.

[0079] Furthermore, the physical constraint correction process includes: Elastic deformation compensation, dynamically adjusting the deformation amount according to the sling tension F:

[0080] Trajectory reparameterization. When the joint torque exceeds the threshold, the Sequential Quadratic Programming (SQP) algorithm is used to optimize the trajectory, with a convergence threshold of 1e-4 and the number of iterations ≤ 50; F is the sling tension (unit: N), which is measured in real-time by a tension sensor; 0.02 is the calibration coefficient, which is obtained by fitting based on the elastic modulus E = 200 GPa of high-strength steel cables and experimental data; The exponent 1.5 reflects the non-linear deformation characteristics (Hooke's law is only applicable within the elastic limit, and actual deformation needs to consider material plastic deformation and the friction effect of multi-strand steel cables); The tension F is collected at a frequency of 1 kHz by a strain gauge sensor at the bottom of the hook, and the data is input into the compensation module after being filtered by a moving average filter (window length N = 50); Calculate ΔL in real-time according to the formula and convert it into the hook position offset:

[0081] where θ is the angle between the sling and the vertical direction, and Δx and Δy are superimposed on the end coordinates of the predicted trajectory to compensate for the positioning error caused by elastic deformation; When the joint torque

[0082] triggers trajectory reparameterization, and the specific steps are as follows: Problem modeling to minimize the trajectory adjustment amount

[0083] Iterative optimization, using the original trajectory P old as the initial guess, update J according to the current joint angle q T , solve the constrained optimization problem by the Lagrange multiplier method, the iteration step size α = 0.1, and the convergence threshold is 1×10 −4 , satisfying the torque constraint or reaching the upper limit of the number of iterations (50 times);

[0084] Precisely match the characteristics of the steel cable through a non-linear formula, avoid the cumulative error of the traditional linear model under heavy loads, and have a closed-loop feedback with the tension sensor data. The positioning accuracy of the compensated trajectory is improved. The SQP algorithm is combined with the dynamic update of the Jacobian matrix to ensure that the adjusted trajectory meets the torque constraint and the safety distance. Elastic compensation and reparameterization work together to solve the problem of trajectory deviation caused by mechanical deformation or sudden load.

[0085] Furthermore, the working logic of the redundant braking unit includes: Main brake failure determination. The main brake unit sends a heartbeat packet through the CAN bus every 100 ms. If it is lost continuously for 3 times, it is determined to fail; Among them, the heartbeat detection fault tolerance mechanism includes: When the inertial measurement unit (IMU) detects that the vibration acceleration a vib exceeds 2g, automatically extend the heartbeat determination window to 500 ms; the formula is:

[0086] The data frame header uses CRC-8 check (polynomial is 0x07), and the frame tail uses CRC-16 check (polynomial is 0x8005); Adopt differential Manchester encoding to shield short-time pulse interference (less than 10 ms): Shielding conditions:

[0087] Through the above mechanism, the heartbeat detection fault tolerance mechanism can adapt to the vibration environment and ensure the reliability and stability of communication; Dual-coil solenoid valve drive, the drive current formula is: ; Among them, V brake is the brake pedal voltage, and I drive is the drive current; Heartbeat packet sending mechanism: The main brake unit sends at T heartbeatBroadcast a heartbeat packet through the CAN bus (ID: 0x701) with a period of 100 ms. The data frame contains an 8-bit binary code representing braking pressure, temperature, and fault codes (such as over-temperature and over-current), as well as a CRC-16 check code with a polynomial of 0x8005;

[0088] Among them, β = 0.01 is the load mass compensation coefficient, and M is the load mass (unit: kg) real-time feedback by the hook weighing sensor; When M > 50 kg, activate the compensation term to match the braking force demand under the heavy-load working condition of the crane; If the slave node (redundant braking controller) does not receive the heartbeat packet within it is determined that the main brake fails, and a sliding window filter (window length N = 5) is introduced to filter out false triggers caused by occasional communication interference; The failure response process is that the redundant braking controller takes over the control right, switches the braking oil circuit through a relay, records the failure timestamp, the content of the last valid heartbeat packet, and environmental parameters (temperature, voltage), displays the fault code (such as E001) through the HMI panel, and sends an alarm signal to the cloud monitoring platform; The driving current formula for the double-coil solenoid valve is: (unit: A); Among them, the proportional coefficient 2.5 is determined through the solenoid valve force-current calibration experiment to ensure the linear relationship between current and braking force (R 2 ≥0.99), and the offset 0.3 compensates for the starting dead zone of the solenoid valve to eliminate the response delay caused by friction; The principle of the double-coil redundant structure is that the main coil works under normal conditions and is driven by the main braking controller. The redundant coil is connected in parallel with the main coil and is independently powered (24V DC isolated power supply) and is driven by the redundant controller. Fault isolation adopts an optocoupler isolation drive circuit to avoid simultaneous failure of the double coils caused by a single-point fault; PWM modulation calculates the PWM duty cycle according to I drive :

[0089] D represents the duty cycle of the PWM signal; I drive represents the driving current, which is the current value to be controlled through PWM modulation; I max represents the maximum current, 5 amperes (A), which is the maximum current value allowed by the system; Current closed-loop control real-time feedbacks the current value through a Hall sensor, and the PID regulator (K p = 0.8, K i = 0.2, Kd Ensure that the output error ≤ 2% (when the input signal-to-noise ratio ≥ 0.05); The CAN bus heartbeat mechanism combined with sliding window filtering reduces the false trigger rate.

[0090] A crane motion trajectory prediction system for the crane motion trajectory prediction method based on RGBD images as described above, comprising: A multi-modal sensing module, with RGBD cameras distributed in an isosceles triangle, and the baseline length is:

[0091] Millimeter-wave radars are installed at the four corners of the crane turntable, covering a 360° horizontal field of view; An edge computing unit, which reduces the computing latency through model quantization and operator fusion technologies to achieve real-time trajectory prediction; A redundant braking module, including a main brake and a redundant brake, and realizing dual-system hot standby through a braking switching unit; A safety assessment module, which calculates the trajectory confidence in real time and triggers a backup prediction mode when the confidence is lower than the threshold; When the safety assessment module detects that the hook swing angle θ > 5°, the redundant braking switches to a progressive braking mode: a brake = k·a max ·cosθ Where: a brake is the actual braking acceleration; a max is the maximum braking acceleration; θ is the hook swing angle; k is the braking force adjustment coefficient, and the value range is 0.5 ≤ k ≤ 1.0; When the hook swing angle θ exceeds 5°, the system enters the progressive braking mode; The braking acceleration a brake is dynamically adjusted by the adjustment coefficient k, the maximum braking acceleration a max and the cosine value of the swing angle cosθ.

[0092] The value range of the adjustment coefficient k is from 0.5 to 1.0, which is used to control the magnitude of the braking acceleration to adapt to different swing situations; By reducing the proportion of the lateral braking force, the risk of secondary collision caused by load swing is suppressed.

[0093] Furthermore, the deployment method of the multi-modal sensing module is the baseline length of the RGBD camera,

[0094] Among them, z = 10 m is the typical working distance, f = 8 mm is the camera focal length, and d min = 0.1 m is the minimum recognizable obstacle size, and N pixel = 5 is the minimum number of pixels occupied by the target when the detection confidence level is ≥ 95%; Installation of millimeter-wave radar. Four groups of radars are installed at the four corners of the crane turntable with a horizontal inclination angle θ = 15° and a vertical depression angle ϕ = 10°, covering a 360° horizontal field of view angle, with a maximum detection distance of 80 m and an accuracy of ±0.02 m; Calibration of the laser rangefinder. The optical axis offset is corrected in real time through IMU data, with an error tolerance of ±0.05°. In rainy and foggy environments, a multi-scale MRF model is used to suppress noise.

[0095] The horizontal inclination angle θ = 15° optimizes the detection ability of the radar beam for low obstacles (such as ground equipment); The vertical depression angle ϕ = 10° avoids the radar beam being blocked by the crane's own structure and improves the coverage in the elevation direction; The laser optical axis offset caused by the crane vibration is measured in real time through IMU, and the compensation formula is: Δθ = k·avibrate where k = 0.05 rad / (m / s 2 )k = 0.05 rad / (m / s2), and a vibrate is the vibration acceleration; Furthermore, the operations performed by the edge computing unit are as follows: Model quantization. The FP32 floating-point weights are dynamically quantized into the INT8 format. The dynamic range of the calibration set is [-2.5, 3.8]. The truncation threshold is optimized through the KL divergence algorithm, and the quantization error rate ≤ 1.5%; Operator fusion. The convolution, batch normalization, and ReLU activation layers are merged into a single computational kernel, and the formula is:

[0096] Among them, y represents the output after being processed by the batch normalization and ReLU activation functions; ReLU represents the ReLU activation function, which sets all negative values to 0 and keeps positive values unchanged, that is, ReLU(x) = max(0, x); γ is the learnable scaling parameter in batch normalization, which is used to adjust the range of the normalized values; β BN is the learnable offset parameter in batch normalization, which is used to translate the normalized values; x represents the data input to the batch normalization layer.

[0097] μ BN represents the mean value of the input x in the current mini-batch data.

[0098] σ BN Represents the standard deviation of the input x in the current mini - batch of data.

[0099] b conv Represents the bias term of the convolutional layer.

[0100] Furthermore, the security assessment module includes: Trajectory deviation detection, calculating the Hausdorff distance between the predicted trajectory P and the real - time observed trajectory Q:

[0101] H(P, Q) represents the Hausdorff distance between the point set P and the point set Q; sup represents the supremum, that is, the least upper bound of a set; inf represents the infimum, that is, the greatest lower bound of a set; p ∈ P means that the point p belongs to the point set P; q ∈ Q means that the point q belongs to the point set Q; d(p, q) represents the distance between the point p and the point q; When H(P, Q)>0.5 m, trigger confidence degradation and start redundant braking; Environmental complexity index, the formula is:

[0102] The dynamic coefficient increases according to the movement speed, safety margin = 1.2−0.05M. When ECI≥2.0, the warning threshold tightens to 70% of the normal value; when ECI < 1.0, the threshold relaxes to 130%; Increases linearly according to the speed v of the obstacle (coefficient = 1 + 0.1v), 1.2−0.05M1.2−0.05M. The larger the load mass M, the stricter the safety threshold.

[0103] The test protocol of the redundant braking module includes: performing a full - load emergency stop and an empty - load fine - motion test every month, and recording the response time t 实测 (representing the actually measured response time) and t 标称 (representing the nominal or expected response time), and the judgment condition is:

[0104] When > 15%, trigger a maintenance alarm; Data is uploaded to the cloud by encrypting and transmitting the pressure curve, current waveform and temperature data through the 5G NR protocol. The JSON format contains the timestamp, device ID, and error code; Abnormal data is synchronously transferred to the work order system in real time, and maintenance tasks are automatically generated. Technicians are required to replace faulty components on-site within 72 hours. The INT8 quantization process is as follows: Calibration set selection, with a dynamic range of [−2.5, 3.8] covering 99% of the activation value distribution. KL divergence optimization to minimize the difference between the FP32 and INT8 distributions, with a truncation threshold T = 3.2. Error control, reducing the impact of zero-point offset through symmetric quantization, with a quantization error rate ≤ 1.5%. Merge convolution (Conv), batch normalization (BN), and ReLU into a single computational core to reduce the number of memory accesses and lower latency.

[0105] Experimental Example 1 Performance test of multi-modal sensor fusion Purpose: To verify the improvement in the perception ability of the multi-modal data fusion mechanism under occlusion and harsh environments.

[0106] Test scenarios: Scenario A: Counterweight occlusion area (30% blind area), dynamic target (size 0.1m × 0.1m) moving horizontally at 2m / s. Scenario B: Rain and fog environment (visibility ≤ 10m), target surface reflectivity ≤ 10%. Test method: Data acquisition, synchronously collecting target data using a binocular RGBD camera (baseline B = 0.64m), millimeter-wave radar (detection range 80m), and laser rangefinder (accuracy ±2mm). Implement IEEE 1588v2 time synchronization through FPGA (timing error ≤ 1ms).

[0107] Control group: Single millimeter-wave radar solution. Experimental group: The multi-modal fusion solution of the present invention (weighted fusion weights: radar 0.7, IMU 0.3). Test results are shown in Table 1:

[0108] As can be seen from Table 1, the multi-modal fusion mechanism significantly improves the target detection rate and positioning accuracy under occlusion and rain and fog environments, verifying the effectiveness of cross-modal spatio-temporal alignment.

[0109] Experimental Example 2 Verification of the trajectory prediction accuracy of the spatio-temporal graph convolutional model Purpose: To evaluate the prediction performance of the TCN-GAT model in dynamic obstacle avoidance scenarios.

[0110] Test scenario: A crane hoists a load of M = 100 kg, and a sudden obstacle (speed 1.5 m / s) cuts into the working area. It is required to predict the obstacle avoidance trajectory within 2 seconds.

[0111] Test method: Model deployment: Baseline model, LSTM network (64 hidden layers); Comparison model: The TCN-GAT model of the present invention (expansion factor d = 1, 2, 4, number of multi-head attention heads H = 4).

[0112] Evaluation index: Root mean square error (RMSE) of trajectory prediction; Obstacle avoidance response time (from the appearance of the obstacle to the completion of trajectory adjustment).

[0113] The test results are shown in Table 2:

[0114] As can be seen from the above table, the prediction error of the TCN-GAT model is significantly reduced, and the response time is greatly shortened, proving its advantage in spatio-temporal dependence modeling.

[0115] Experimental Example 3 Verification of the load adaptive safety distance model; Purpose: To compare the safety differences between the dynamic safety distance model and the traditional fixed threshold scheme.

[0116] Test scenario: Condition 1, light load (M = 30 kg), a dynamic object approaches at a constant speed of 1 m / s; Condition 2, heavy load (M = 200 kg), a dynamic object approaches with an acceleration of 3 m / s (a = 1 m / s 2 )

[0117] Test method: Safety distance calculation: Traditional scheme, fixed threshold D_fixed = 1.0 m; The dynamic calculation of D_safe of the present invention (k 1 = 1.6, k 2 = 3.6).

[0118] Trigger result: Record the warning distance and the actual collision risk (false trigger rate).

[0119] The test results are shown in Table 3:

[0120] As can be seen from the above table, the false trigger rate of the dynamic model is significantly reduced under heavy load conditions, and the safety distance matches the actual load requirements.

[0121] Experimental Example 4 Redundant braking system reliability test; Purpose: To verify the main-standby braking switching efficiency and long-term operation stability.

[0122] Test scenarios: Scenario A: Simulate the interruption of the main braking CAN communication (300 ms) to test the redundant braking switching; Scenario B: Continuously operate for 6 months, and perform a full-load emergency stop test every month (M = 200 kg, braking pressure 20 MPa).

[0123] Test methods: Use a high-speed camera (1000 fps) to record the rising curve of the braking oil pressure; Calculate the total response time according to ISO 13849-1, that is, the total time from triggering to reaching the safe state.

[0124] Long-term stability: Count the number of fault alarms and the maintenance response timeliness.

[0125] Control group: Adopt Chinese Patent CN116253248B; Experimental group: The redundant braking switching system of the present invention.

[0126] The test results are shown in Table 4:

[0127] As can be seen from the above table, the switching time and pressure stability of the redundant braking system are better than the prior art, and the failure rate during long-term operation is low.

[0128] Experimental Example 5 Verification of edge computing real-time performance; Purpose: To evaluate the optimization effect of model quantization and operator fusion on the calculation delay.

[0129] Test methods: Deployment environment: Hardware platform, NVIDIA Jetson AGX Xavier; Traditional baseline scheme: FP32 model; Scheme of the present invention: INT8 quantization + operator fusion.

[0130] The test results are shown in Table 5:

[0131] As can be seen from the above table, the edge computing optimization of the present invention improves the inference speed and greatly reduces the memory occupancy.

[0132] Experimental Example 6 Swing condition test; Test conditions: Hoisting mass M = 150% of the rated load; Artificially induce the lateral swing angle of the hook θ = 8°; Test purpose: Verify the anti-shock ability of the braking system under high-load and large-swing conditions.

[0133] Judgment of oil pressure stability, dynamic adaptation of load pressure fluctuation tolerance:

[0134] Where: is the allowable pressure fluctuation range; P max is the maximum oil pressure; P min is the minimum oil pressure; M is the hoisting mass (unit: kg); During the test, the hoisting mass is set to 150% of the rated load, and the lateral swing angle of the hook is artificially induced to be 8°. In this way, the anti-shock ability of the braking system under high-load and large-swing conditions can be verified.

[0135] Judgment of oil pressure stability: Allowable pressure fluctuation range Determined by the difference between the maximum oil pressure Pmax and the minimum oil pressure Pmin; In the formula indicates that the pressure fluctuation tolerance is proportional to the hoisting mass M, and the unit is MPa; For example, when the hoisting mass M = 150% of the rated load, the pressure fluctuation tolerance is:

[0136] The test results are shown in Table 6:

[0137] It can be seen from the above table that the redundant braking unit of the present invention can still operate stably under the swing condition of the crane, realizes fast switching through vibration threshold filtering and dynamic current compensation, avoids communication interruption caused by mechanical vibration, and combines the braking degradation strategy of the hook swing angle θ to ensure that the braking force direction matches the load inertia during heavy load swing. The traditional scheme does not design a redundant mechanism for the crane swing scenario, and vibration interference may cause control signal loss or solenoid valve jamming, ultimately leading to system deadlock.

[0138] Suppress rain and fog noise through the multi-scale MRF model of the laser rangefinder, and dynamically correct it in combination with millimeter-wave radar data to compensate for the change in the viscosity of the hydraulic oil. According to the rain and fog interference level L rain Adjust the upper limit of the braking acceleration to avoid overshoot of the braking force caused by the decrease in visibility.

[0139] The traditional solution relies on a single sensor, and the noise interference is significant in rainy and foggy environments. The braking force calculation does not introduce an environmental compensation mechanism, resulting in an increased deviation.

[0140] In summary, the double-coil solenoid valve, vibration filtering, and dynamic current compensation significantly improve the reliability under extreme conditions, and the multi-sensor fusion and rain and fog interference compensation mechanism enhance the system robustness.

[0141] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments or perform equivalent replacements for some of the technical features. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A crane motion trajectory prediction method based on RGBD images, characterized in that: The following steps are involved: S1: Multi-modal data synchronous acquisition, deploying a binocular RGBD camera array to obtain RGB images and depth point cloud data of the crane operation area in real time; The millimeter-wave radar emits electromagnetic waves at a frequency of 75-80GHz and receives reflected waves to measure the distance and speed of dynamic objects; The laser rangefinder emits laser pulses and receives reflected light with an accuracy of ±2mm to measure the distance information of dynamic objects; Implement time synchronization of IEEE 1588v2 protocol through FPGA hardware to ensure the time consistency of all sensor data; S2: Cross-modal feature alignment and fusion, using a deformable convolutional network to generate the deformation field parameter Δp, the calculation formula is: Among them, F rgb is the RGB feature map, F depth is the deep feature map, T imu is the inertial measurement unit data, Conv 1×1 It is a 1×1 convolution operation; When a dynamic object enters the sensor blind spot, its acceleration is predicted through IMU data, and Δp is dynamically adjusted to compensate for trajectory deviation; S3: Trajectory prediction and physical constraint correction, build a spatiotemporal graph convolution model, input the fused multimodal data, output the crane motion trajectory within the next 2-4 seconds, and verify the feasibility of the trajectory through the rigid body dynamics equation. The constraints are: Among them, J T is the transpose of the Jacobian matrix, F ext is an external force; S4: Real-time calculation of the minimum safe distance between the crane and dynamic objects: in, , M is the mass of the suspended object, t react is the system response time; S5: hierarchical braking control. When the dynamic object enters the preset range value, an audible and visual warning is triggered. When the dynamic object exceeds the preset range value and the operator does not respond, an emergency stop is achieved through the redundant braking unit.

2. The crane motion trajectory prediction method based on RGBD image according to claim 1 is characterized in that: The cross-modal feature alignment further includes: Sensor outliers are eliminated by detecting data consistency through Mahalanobis distance and eliminating abnormal data: Where X is the sensor data vector, μ is the mean vector, and Σ is the covariance matrix; Unify the radar and laser ranging data into the crane base coordinate system, and the conversion matrix is: Among them, R is the rotation matrix and t is the translation vector.

3. The crane motion trajectory prediction method based on RGBD image according to claim 1 is characterized in that: The working mode of the spatiotemporal graph convolution model includes: a gated TCN module, which uses three layers of dilated convolution, each layer containing 32 3×3 convolution kernels; Calculate node weights through a multi-head attention mechanism: Among them, W is the learnable weight matrix, h i 、h j is the node feature vector.

4. The crane motion trajectory prediction method based on RGBD image according to claim 3 is characterized in that: The physical constraint correction process includes: Elastic deformation compensation, dynamically adjust the deformation according to the sling tension F: Trajectory reparameterization, when the joint torque exceeds the threshold, the sequential quadratic programming algorithm is used to optimize the trajectory, with a convergence threshold of 1e-4 and the number of iterations ≤ 50 times.

5. The crane motion trajectory prediction method based on RGBD image according to claim 1 is characterized in that: The working logic of the redundant braking unit includes: Main brake failure judgment: the main brake unit sends a heartbeat packet through the CAN bus every 100ms. If it is lost for three consecutive times, it is judged as failed. Double coil solenoid valve drive, the driving current formula is: Among them, V brake is the brake pedal voltage.

6. The crane motion trajectory prediction system according to any one of claims 1 to 5, characterized in that: include: Multimodal sensing module, RGBD cameras are distributed in an isosceles triangle, and the baseline length is: Millimeter-wave radars are installed at the four corners of the crane turntable, covering a 360° horizontal field of view; An edge computing unit, which reduces computing latency through model quantization and operator fusion technology to achieve real-time trajectory prediction; Redundant brake module, including main brake and redundant brake, realizes dual system hot standby through brake switching unit; The safety assessment module calculates the trajectory confidence in real time and triggers the backup prediction mode when the confidence is lower than the threshold.

7. The crane motion trajectory prediction system according to claim 6, characterized in that: The deployment method of the multimodal sensor module is as follows: In the calculation formula of the baseline length of the RGBD camera, z = 10 m is the typical working distance, f = 8 mm is the focal length of the camera, and d min = 0.1m is the minimum identifiable obstacle size, N pixel =5 is the minimum number of pixels occupied by the target when the detection confidence is ≥95%; for millimeter wave radar installation, at least 4 sets of radars are installed at the four corners of the crane turntable with a horizontal inclination angle of θ=15° and a vertical depression angle of ϕ=10°, covering a 360° horizontal field of view, with a maximum detection distance of 80m and an accuracy of ±0.02m; Laser rangefinder calibration, real-time correction of optical axis offset through IMU data, error tolerance ±0.05°, and multi-scale MRF model is used to suppress noise in rainy and foggy environments.

8. The crane motion trajectory prediction system according to claim 7, characterized in that: The edge computing unit performs: Model quantization: dynamically quantize FP32 floating-point weights to INT8 format. The dynamic range of the calibration set is [-2.5, 3.8]. The truncation threshold is optimized by the KL divergence algorithm, and the quantization error rate is ≤1.5%; Operator fusion combines convolution, batch normalization, and ReLU activation layers into a single computational kernel. The formula is: Among them, Conv stands for convolution and BN stands for batch normalization.

9. The crane motion trajectory prediction system according to claim 6, characterized in that: The safety assessment module includes: Trajectory deviation detection, calculate the Hausdorff distance between the predicted trajectory P and the real-time observed trajectory Q: When H(P,Q)>0.5 m, the confidence level is downgraded and redundant braking is initiated; Environmental complexity index, the formula is: The dynamic coefficient increases according to the movement speed, with a safety margin of 1.2−0.05M. When ECI ≥ 2.0, the warning threshold is tightened to 70% of the normal value; when ECI < 1.0, the threshold is relaxed to 130%.

10. The crane motion trajectory prediction system according to claim 6, characterized in that: The test protocol for the redundant brake module includes: performing full-load emergency stop and no-load inching tests every month, recording t 实测 and t 标称 , the judgment conditions are: When Δt>15%, a maintenance alarm is triggered; Data upload uses the 5G NR protocol to encrypt and transmit pressure curves, current waveforms, and temperature data to the cloud. The JSON format includes timestamps, device IDs, and error codes. Abnormal data is synchronized to the work order system in real time, and maintenance tasks are automatically generated. Technicians are required to replace faulty parts on site within 72 hours.

Images