Crane end-to-end motion trajectory prediction method based on rgbd images

Abstract

This invention discloses a method for predicting the end-to-end motion trajectory of a crane based on RGBD images. This method belongs to the field of intelligent control and includes two stages: training and prediction. During training, a sequence of RGB images and depth images from the operator's cab perspective, collected during crane operation, and motion control commands from the operator are input. The RGB images are segmented using a semantic segmentation network to obtain a semantic segmentation map, which is then fused with a category index map and depth map to obtain a semantic depth map. A dual-branch RLSTM-Transformer network extracts features from the semantic segmentation map and semantic depth map and predicts motion control commands. The deviation between the predicted values ​​and the true values ​​is minimized, and the neural network model is trained to obtain a prediction model. During prediction, the RGB images and depth images are input into a trained semantic RLSTM-Transformer crane end-to-end motion trajectory prediction model, which then outputs the crane's motion control commands.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of intelligent control of cranes, and particularly relates to a crane end-to-end motion trajectory prediction method based on RGBD images. BACKGROUND

[0002] Under the background of rapid development of intelligent manufacturing and industrial automation, improving the autonomy level and overall performance of the crane system has become the focus of current research.

[0003] The intelligent control problem of cranes has always been faced with many challenges, especially due to its under-actuated nature, the control system often needs to deal with redundant degrees of freedom and complex dynamic characteristics. Specifically, the hoist and load of the crane will produce nonlinear swing, and external disturbances (such as wind, unstable load, etc.) will also have a negative impact on control accuracy and operational safety. Therefore, the existing crane control method usually relies on local control, that is, segmented control for specific subsystems or operation processes. Although local control can improve operational performance to some extent, its limitations become increasingly apparent when faced with more complex working conditions and multi-variable control. This control method is difficult to achieve global optimization, resulting in low operational efficiency and the system's intelligence level is still at a low stage.

[0004] In contrast, end-to-end control technology has made breakthrough progress in the field of autonomous vehicles in recent years. End-to-end control combines perception, decision-making and control closely, directly from sensor data (such as cameras, lidar, etc.) to control output, establishing a unified model. This method no longer relies on the traditional hierarchical control structure, but through artificial intelligence technologies such as deep learning, it autonomously learns the optimal decision-making in complex environments from a large amount of data. This enables autonomous vehicles to more accurately and flexibly plan and control globally when faced with complex traffic environments, dynamic scenarios and diverse inputs, demonstrating excellent intelligence level.

[0005] However, compared with the field of autonomous driving, the research on end-to-end control of crane systems is still in a blank state. Current intelligent control of cranes mostly relies on local algorithms, lacking a global and unified intelligent control framework, which shows certain shortcomings in dealing with complex working conditions, dynamic environments, and diversified task requirements. However, there are differences between end-to-end control of vehicles and end-to-end control of cranes. Cranes are typical under-actuated systems, especially due to the presence of ropes, which causes the load to sway during operation, increasing the uncertainty and complexity of the system; while autonomous vehicles are fully actuated systems, and the control task is relatively direct. The operation scene of the crane is usually a small range of unstructured scene with certain dynamics, and the external interference (such as wind, ground vibration) and rope swing make the control more challenging; while the operation scene of the autonomous vehicle is a relatively structured road environment, and the control task pays more attention to global navigation and dynamic obstacle avoidance.

[0006] By introducing the end-to-end control method, the crane system can directly go from sensor data to output control signals, achieving global optimization and dynamic decision-making, and thus improving the automation level, operation efficiency and safety of the system. Especially in the face of complex environments or task requirements, end-to-end control is expected to overcome the bottlenecks of traditional local control and provide a more intelligent and more robust control scheme.

[0007] Therefore, this paper proposes an end-to-end motion trajectory prediction method based on RGBD images for crane systems. The method collects RGB images, depth images and operation end crane motion control instructions from the driving cab perspective of the crane operation scene, segments the RGB images based on a semantic segmentation model, obtains a pseudo-color image and a class index image, superimposes the pseudo-color image and the RGB image with a certain transparency to obtain a semantic RGB image, and fuses the class index image and the depth image to obtain a three-channel semantic depth image. Through the RLSTM-Transformer model, the features of the input image sequence are extracted, the motion control instructions of the crane are predicted, and the end-to-end intelligent control of the crane is realized. SUMMARY

[0008] The technical problem to be solved by the present application is to propose an end-to-end motion trajectory prediction method based on RGBD images for crane systems, to solve the problems of low information transmission efficiency, difficulty in coping with various complex environments, and difficulty in realizing overall control in the current control method of separating perception and control in crane research. By introducing the end-to-end control method, the crane system can directly go from sensor data to output control signals, achieving global optimization and dynamic decision-making, and thus improving the automation level, operation efficiency and safety of the system.

[0009] In order to achieve the above object, the technical scheme adopted by the present application is:

[0010] An end-to-end crane motion trajectory prediction method based on an RGBD image, the method comprising two stages of training and prediction, the steps of the training stage being steps one to seven, and the steps of the prediction stage being step eight:

[0011] Step one: obtain an RGB image and a depth image under the cab perspective of the crane during operation through an RGBD camera, and simultaneously obtain a motion control instruction of the operating end, the motion control instruction of the operating end comprising: C1-lifting, C2-descending, C3-amplification, C4-deamplification, C5-left rotation, C6-right rotation, C7-forward, C8-backward, C9-grasping, and C10-releasing;

[0012] Step two: segment the RGB image based on a semantic segmentation model to obtain a pseudo-color image and a class index image, superimpose the pseudo-color image and the RGB image with a certain transparency to obtain a semantic RGB image, and simultaneously fuse the class index image and the depth image, fuse a single-channel class index image, a single-channel depth image, and set a third channel as blank to obtain a three-channel semantic depth image. The output of the semantic segmentation model is represented as:

[0013] I pseudo , I index =Segmentation(I RGB )

[0014] wherein I RGB is the input RGB image, I pseudo is the pseudo-color image output by the semantic segmentation model, and I index is the single-channel class index image output by the semantic segmentation model.

[0015] The fusion process of the pseudo-color image and the RGB image is:

[0016] I semantic_RGB =α·I pseudo +(1-α)·I RGB

[0017] wherein α is a transparency coefficient, and I semantic_RGB is the fused semantic RGB image.

[0018] The generation mode of the three-channel semantic depth image I semantic_depth is that the class index image is taken as the first channel, the depth image is taken as the second channel, and a blank channel is taken as the third channel, and is represented as:

[0019] I semantic_depth =Concatenate(I index , I depth , Iempty )

[0020] I depth is the input single-channel depth map, I empty For blank channels, I empty Size and I depth and I index Concatenate(·) means concatenating images in the channel dimension.

[0021] Step 3: Establish a Multi-RLSTM local spatiotemporal feature extraction model by optimizing the model unit RLSTM;

[0022] Step 4: Input the semantic depth image into the local spatiotemporal feature extraction sub-network. The local spatiotemporal feature extraction sub-network uses CNN, Multi-RLSTM and multi-head attention modules to extract the local temporal features of the input sub-image sequence and predict the output of each sub-sequence. The local spatiotemporal feature extraction sub-network adopts a staged feature extraction strategy. First, the long image sequence X∈R N×C×T×H×W , where N is the batch size, C is the number of channels, T is the sequence length, H is the image height, and W is the image width; the input sequence is divided into image subsequences X with each sample length T (n) ∈R N×C×H×w , representing the image sequence of the nth sample; the image subsequence is input into the 3D convolution layer to extract spatiotemporal features and generate a temporal feature representation. After processing, the temporal data can still maintain its own temporal nature; by optimizing the RLSTM model unit, a Multi-RLSTM model is established to further extract the temporal features of these subsequences and gradually capture higher-level temporal dependencies; a multi-head attention mechanism is introduced after the Multi-RLSTM layer to process the entire sequence and capture global dependencies.

[0023] Step 5: Input the semantic RGB image into the global spatiotemporal feature extraction sub-network, which uses ResNet and Transformer to extract the deep spatial features of the image sequence and establish global dependencies, and predict the output of each sub-sequence.

[0024] Step 6: Finally, the outputs of the two branches are fused through the weighted fusion network to obtain the final output of the model. local ∈R N×d represents the crane motion state predicted by the local spatiotemporal prediction subnetwork, X global ∈R N×d Represents the crane motion state predicted by the global spatiotemporal prediction sub-network. The fusion of these two sub-networks adopts a weighted approach. fused =α·X local +β·Xglobal wherein a, b are weighting coefficients, representing the different emphasis on local and long-term prediction results, and the final fused prediction result y fused ∈R N×d , d is the total number of categories of crane motion control instructions, representing the crane motion control instruction prediction after combining local and long-term information decision-making.

[0025] Step seven: the collected motion control instructions of the operating end are taken as the labels of the images, the neural network model is trained by minimizing the deviation between the predicted value and the true value to optimize the loss function, and the prediction model is obtained.

[0026] Step eight: when executed online, the RGB image and the depth image collected by the RGBD camera under the cab perspective in the crane operation scene are obtained, the images are input into the trained semantic RLSTM-Transformer crane end-to-end motion trajectory prediction model, and the required crane motion control instructions can be obtained through model prediction.

[0027] The further improvement of the technical scheme of the present application is that the method for making the data set is:

[0028] The RGB image and the depth image under the cab perspective during crane operation are obtained by the RGBD camera, and the motion control instructions of the operating end are obtained at the same time.

[0029] The grab bucket, the material, the car bucket and the background in the RGB image are segmented based on the semantic segmentation model, the semantic segmentation pseudo-color image and the semantic segmentation category index image containing the grab bucket, the material, the car bucket and the background are obtained through model inference, the pseudo-color image containing the grab hook, the material, the car bucket and the background segmented and output is superimposed on the original image with a certain transparency to obtain the RGB image containing semantic information, and the depth image and the semantic segmentation category index image are fused to obtain the depth image with semantic information. The output of the semantic segmentation model is represented as:

[0030] I pseudo , I index =Segmentation(I RGB )

[0031] wherein I RGB is the input RGB image, I pseudo is the pseudo-color image output by the semantic segmentation model, and I index is the single-channel category index image output by the semantic segmentation model.

[0032] The fusion process of the pseudo-color image and the RGB image is:

[0033] I semantic_RGB =α·I pseudo +(1-α)·IRGB

[0034] where alpha is the transparency coefficient, I Semantic_RGB is the fused semantic RGB image.

[0035] The three-channel semantic depth image I semantic_depth is generated by taking the class index image as the first channel, the depth image as the second channel, and a blank channel as the third channel, and is represented as:

[0036] I semantic_depth = Concatenate(I index , I depth , I empty )

[0037] I depth is the input single-channel depth image, I empty is a blank channel, i empty has the same size as I depth and I index . Concatenate(·) represents concatenating images in the channel dimension.

[0038] Each semantic RGB image corresponds to a semantic depth image. The images are aligned with the crane motion control instructions collected by the operation end using a timestamp. The crane motion control instructions are used as the label of the picture, and the data set is preprocessed to obtain the preprocessed data set.

[0039] The further improvement of the technical scheme of the application is that the specific calculation formula of the optimization model unit RLSTM is:

[0040] f t = sigma(W f * [h t-1 , x t ] + b f )

[0041] i t = sigma(W i * [h t-1 , x t ] + b i )

[0042]

[0043] o t = sigma(W O [h t-1 , x t ] + b O )

[0044] h′ t = o ttanh(C t )

[0045] h t =tanh(h′ t +W h ·h′ t-1 +b h )

[0046] σ is a Sigmoid activation function, W f , W i , W o are weight parameters of the forget gate, the input gate and the output gate respectively, b f , b i , b o are bias parameters of the forget gate, the input gate and the output gate respectively, f t , i t , o t are the forget gate, the input gate and the output gate respectively, x t is the current sequence input, h t-1 is the state of the hidden layer at the last time, C t is the cell state under the current time input sequence, is the cell state updated through the output of the forget gate and the input gate, tanh is an activation function, h′ t is the output of the hidden layer of the LSTM at t before modification, h t is the output of the hidden layer of the RLSTM at t after modification. The calculation of the RLSTM model unit on the hidden state of each time step in the LSTM introduces a residual connection of the hidden state of the last time step, and introduces a learnable weight W h and a bias b h to adjust the contribution of the residual connection. The delayed residual connection can directly pass the hidden state of the last time step to the current time step, so that the model can make a more agile response to the rapid change of the crane state.

[0047] Further improvement of the technical scheme of the application is that the Multi-RLSTM local space-time feature extraction model is specifically:

[0048] R(h i , h j ) = linear(concat(h i , h j ))

[0049] h1 = tanh(LSTM1(x))

[0050] h2 = tanh(LSTM2(h1))

[0051] h3 = tanh (LSTM3 (h2))

[0052] h4 = tanh (LSTM4 (R (h3, h1))) = tanh (LSTM4 (linear (concat (h3, h1))))

[0053] h5 = tanh (LSTM5 (h4))

[0054] h6 = tanh (LSTM6 (R (h5, h2))) = tanh (LSTM6 (linear (concat (h5, h2))))

[0055]

[0056] where h i and h j are the output of the i-th and j-th layer hidden state, respectively, and their shape is (batch_size, seq_len, hidden_dim), R represents the residual connection function, concat (h i , h j ) represents concatenating h i and h j in the last dimension, linear is a linear transformation. In the Multi-RLSTM local spatio-temporal feature extraction model, a stacked LSTM structure is used, and each RLSTM layer can learn different levels of temporal features, so that the model can gradually extract rich high-dimensional features from shallow to deep. Dynamic residual connection is introduced between the RLSTM layers, which concatenates the output of the previous LSTM with the output of the subsequent LSTM layer in the feature dimension. This dynamic residual connection allows information to quickly propagate through the network, which helps to improve the response speed of the model to the changes in the joystick trajectory, especially when facing complex or rapid dynamic changes, it can capture the change trend more quickly.

[0057] Further improvement of the technical scheme of the application is that the local spatio-temporal feature extraction subnetwork, specifically:

[0058] The local spatio-temporal feature extraction subnetwork adopts a phased feature extraction strategy. First, the long image sequence X after semantic segmentation is divided into a plurality of image sub-sequences X N×C×T×H×W , where N is the batch size, C is the number of channels, T is the sequence length, H is the image height, and W is the image width. The input sequence is divided into image sub-sequences X (n) ∈R N×C×H×w, represents the image sequence of the nth sample. The image sub-sequence is input into the 3D convolution layer, and the convolution layer focuses on extracting the local spatio-temporal information of the input picture sequence, reflecting the short-term changes of the system, and is suitable for processing the short-term oscillation or sudden events of the crane system. The 3D convolution layer adopts multiple convolution blocks (3x3 convolution + ELU activation function + batch normalization) to extract the features of the input picture sequence, and introduces a residual convolution layer in the convolution layer to relieve the gradient disappearance problem in the deep network, while improving the training efficiency and model performance.

[0059] By optimizing the RLSTM model unit, a Multi-RLSTM model is established to further extract the time sequence features of these sub-sequences and gradually capture higher-level time dependence.

[0060] The multi-head attention mechanism is introduced after the Multi-RLSTM layer to process the entire sequence and capture global dependence, specifically:

[0061] F i n i = f(chunk co )

[0062] F cat n n = [F1; F2;...; Fn]

[0063] chunk i i represents the ith sub-sequence, f(chunk i ) represents the feature extraction process of the ith sub-sequence through 3D convolution and Multi-RLSTM layer, F i i represents the feature representation of the ith sub-sequence, with a dimension of (N, T, D), F concat is the feature representation after concatenation in the time dimension, with a dimension of (N, nT, D)

[0064] Q i = F concat W i Q

[0065] K i = F concat W i K

[0066] V i = F concat W i V

[0067] where F concat is the representation of the input sequence, W i Q , W iK , W i V are the weight matrices of Q, K and V of the i-th attention head, respectively, the similarity between Q and K is calculated by using a dot product operation for each attention head, and the output of the i-th attention head is normalized by Sof t max function:

[0068]

[0069] MultiHead(Q, K, V) = Concat(head1, head2, head3) W O

[0070] The outputs of all heads are spliced, and a linear transformation is performed to generate the output of the multi-head self-attention module, with a dimension of (N, nT, D).

[0071] The features of each sub-sequence are aggregated using max pooling, and a fully connected layer is introduced after the pooling layer to generate the final prediction result, specifically:

[0072]

[0073] where n represents the batch index, c represents the sub-sequence index, d represents the feature dimension index, and t represents the time step index, and the above formula represents average pooling for each time step of the sub-sequence. The pooled input is linearly transformed by the fully connected layer to gradually reduce the number of features from the input tensor to the shape of the output tensor, and the output is the crane movement control instruction.

[0074] A further improvement of the technical scheme of the application is that the global spatio-temporal feature extraction sub-network is specifically:

[0075] The global spatio-temporal feature extraction sub-network predicts the input long-time sequence image, and the input tensor is frames∈R N×C×T×H×W where N is the batch size, C is the number of channels, T is the number of time steps, H and W are the height and width of the image, respectively, i.e. frames = {f n，c，t，h,w}. The reshape operation is performed on the input tensor to combine the N and T dimensions into N×T, i.e. frames' = {f' n′，c，h，w}, n' = B×T, frames∈R NT×C×H×SThe global spatio-temporal feature extraction subnetwork combines ResNet18 and Transformer architecture. First, the reshaped entire image sequence is input into the modified ResNet18 model for preliminary feature extraction. ResNet18 serves as the image position encoder, extracting the spatial features of each image through convolution operations and converting them into fixed-size feature vectors. Meanwhile, a linear layer is used to map the extracted ResNet features to a low dimension as input for the subsequent temporal model. In the network architecture, ResNet18 is chosen as the encoder for image sequences because it has multiple convolutional layers and residual connections, enabling it to extract complex spatial features in crane operations, such as object displacement and rocker position changes. The network then uses a position encoding module to introduce sequence position information. This step introduces position information into the feature sequence, allowing the Transformer to perceive the order in the frame sequence and help capture global temporal dependencies. Position encoding is generated by a predefined function PE(t), which is: Next, the network inputs the position-encoded image sequence into the Transformer-based temporal encoder, which is: is the feature vector after the input image is extracted by Resnet18 and the features are mapped to low-dimensional features. The Transformer can consider all past states at each time step and identify key time points and trends through multi-head attention mechanisms. This allows the model to quickly capture sudden events (such as sudden shaking) in the short term while maintaining tracking of long-term trends, thereby improving prediction accuracy. The combination of ResNet18 and Transformer architecture enables the model to make decisions based on comprehensive information from the entire operation sequence, rather than being limited to single-frame images or local time periods, allowing it to better capture changes in motion patterns caused by complex dynamics.

[0076] Finally, the feature vector f = {f t，n，k}∈R T×N×d where T is the sequence length, N is the batch size, and d is the feature dimension. Max-pooling is performed in the time dimension T, which is: is the feature after max pooling. The pooled feature is dimensionally reduced, and then mapped to the final crane motion control instruction output through a fully connected layer. The global spatio-temporal feature prediction subnetwork extracts features from the entire batch of images and models the temporal relationship of sequences by means of position encoding and transformers, understands the long-term motion of the crane, and predicts the motion trend of the crane from a global perspective.

[0077] Compared with the prior art, the above technical scheme adopted by the present application has the advantages of:

[0078] The present application fuses semantic information and RGB information, and fuses semantic information and depth information. The semantic RGB image not only contains original visual information, but also clearly distinguishes key objects such as grabs, materials and carriages, so that the model can better understand the objects and relationships in the scene. By fusing depth images and semantic class index images, the model can simultaneously obtain the geometric depth information and object class information of the scene, help the model more accurately judge the position and distance of the object, and improve the adaptability and robustness of the model to complex scenes and the prediction accuracy.

[0079] The present application optimizes the model unit RLSTM, and proposes a Multi-RLSTM local spatio-temporal feature extraction model. RLSTM retains the hidden state of the previous time step through residual connection, which can help deal with nonlinear changes in the system and improve the tracking ability of complex motion trajectories. RLSTM can retain and pass important state information of the previous time step through residual connection, which helps the model better cope with local motion changes, making the model more sensitive to local feature changes and making accurate adjustments.

[0080] In the Multi-RLSTM local spatio-temporal feature extraction model, a stacked LSTM structure is used, and each RLSTM can learn different levels of time sequence features, so that the model can gradually extract rich high-dimensional features from shallow to deep. Dynamic residual connection is introduced between the RLSTM layers, and the output of the previous LSTM layer is spliced with the output of the subsequent LSTM layer in the feature dimension. This dynamic residual connection allows information to quickly propagate in the network, which helps to improve the response speed of the model to joystick trajectory changes, especially in the face of complex or rapid dynamic changes, the model can more quickly capture the change trend.

[0081] The RLSTM-Transformer end-to-end motion trajectory prediction model for crane systems proposed in this paper features a unique dual-branch network structure. The local spatiotemporal feature extraction subnetwork utilizes CNN, Multi-RLSTM, and a multi-head attention module to extract local temporal features from the input sub-image sequence. The global spatiotemporal feature extraction subnetwork utilizes ResNet and Transformer to extract deep spatial features from long image sequences and establish global dependencies. By combining local and global spatiotemporal feature extraction, the model can balance short-term motion details and long-term trajectory continuity in crane motion prediction, enhancing its understanding and modeling capabilities for complex spatiotemporal dynamics. This not only improves the model's prediction accuracy but also enhances its stability and robustness in actual operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 This is a schematic diagram of the RLSTM model unit structure;

[0083] Figure 2 This is a schematic diagram of the Multi-RLSTM model structure;

[0084] Figure 3 This is a schematic diagram of the semantic RLSTM-Transformer end-to-end crane motion trajectory prediction model structure;

[0085] Figure 4 It is a prediction flow chart. DETAILED DESCRIPTION

[0086] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products or apparatuses.

[0087] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments:

[0088] A crane end-to-end motion trajectory prediction method based on RGBD images includes two stages: training and prediction. The training stage includes steps 1 to 7, and the prediction stage includes step 8:

[0089] Step 1: On the gantry crane real machine experiment platform set up in the laboratory, the RGB image, depth image and operation end motion control instruction under the cab view during crane operation are obtained through the Azure Kinect camera, including: C1-up, C2-down, C3-amplification, C4-deamplification, C5-left rotation, C6-right rotation, C7-forward, C8-backward, C9-grab, C10-release.

[0090] Specifically, the RGB image and depth image under the cab view of the gantry crane collected by the Azure Kinect sensor and the operation end crane motion control instruction are obtained, the collection frequency is 10FPS, and the time stamp is retained.

[0091] On the real crane experiment platform, the Azure Kinect is installed on the boom of the crane to capture the images of the grab bucket operation at 10FPS. The data under the cab view during crane operation is collected by the Azure Kinect sensor through ROS and recorded as a rosbag file, and then the corresponding depth map and color map are extracted from the rosbag file through a script. The collected 50000 frames of RGB image, depth map and operation end motion control instruction contain 50 groups, each group of data contains a complete crane loading material process, the image resolution is 640*480, and the size is cropped to 160*210 in the later stage. The data is collected under different light conditions, the relative position of the sand pile and the car hop is adjusted, different levels of wind are simulated, and the weight of the lifted object is changed to improve the diversity of the data.

[0092] The input port of the crane controller is connected to the switch, which is used to detect the operator's command. According to different input switch signal, the output port connected relay is controlled through Siemens PLC program to realize the control of motor, so as to realize the precise control of different movements of the crane. The state of the input switch is captured and saved at 10FPS, and the time stamp is recorded at the same time.

[0093] During collection, each RGB image corresponds to a depth image. The collected images and the operation end motion control instruction are aligned by using the time stamp, and the operation end motion control instruction is used as the label of the picture.

[0094] Step 2, manually label the grab bucket, material, and car bucket in the RGB image, train the PaddleSeg semantic segmentation model, input all the RGB pictures into the trained semantic segmentation model, and get the semantic segmentation pseudo-color image and semantic segmentation class index image containing the grab bucket, material, car bucket, and background. Superimpose the pseudo-color image output by segmentation and the original image with a certain transparency to get the RGB image fused with semantic information. The superimposed image not only retains the rich color and texture information in the original image, but also highlights the target information that needs to be focused on in the crane decision-making task. Fuse the single-channel depth map and the single-channel semantic segmentation class index image, set the third channel to be blank, and fuse them into a three-channel semantic depth map. Each pixel value in the semantic segmentation class index image represents the class to which the pixel belongs, and the fused depth map with semantic information is obtained.

[0095] The output of the semantic segmentation model is represented as:

[0096] I pseudo , I index = Segmentation(I RGB )

[0097] where I RGB is the input RGB image, I pseudo is the pseudo-color image output by the semantic segmentation model, and I index is the single-channel class index image output by the semantic segmentation model.

[0098] The fusion process of the pseudo-color image and the RGB image is:

[0099] I semantic_RGB = a·I pseudo + (1-a)·I RGB

[0100] where a is the transparency coefficient, and I semantic_RGB is the fused semantic RGB image.

[0101] The generation method of the three-channel semantic depth map I semantic_depth is to take the class index image as the first channel, the depth map as the second channel, and the blank channel as the third channel, represented as:

[0102] I semantic_depth = Concatenate(I index , I depth , I empty )

[0103] I depth is the input single-channel depth map, I empty is the blank channel, and I empty has the same size as I depth and Iindex Concatenate(·) denotes concatenating images in the channel dimension.

[0104] Step 3, build the Multi-RLSTM local spatio-temporal feature extraction model by optimizing the model unit RLSTM;

[0105] Specifically, the calculation of the hidden state of each time step introduces the residual connection of the hidden state of the previous time step, and introduces the learnable weight W h and bias b h to adjust the contribution of the residual connection, such weighted residual connection can provide more flexibility, allowing the model to learn how to better integrate information from the previous time step. The RLSTM forward propagation process is:

[0106] f t = σ(W f · [h t-1 , x t ] + b f )

[0107] i t = σ(W i · [h t-1 , x t ] + b i )

[0108]

[0109] o t = σ(W O [h t-1 , x t ] + b O )

[0110] h′ t = o t *tanh(C t )

[0111] h t = tanh(h′ t + W h ·h′ t-1 + b h )

[0112] σ is the Sigmoid activation function, S f , W i , W o are the weight parameters of the forget gate, input gate and output gate respectively; b f , b i , b o are the bias parameters of the forget gate, input gate and output gate respectively, f t, i t , o t are forget gate, input gate and output gate respectively, x t is the current sequence input, h t-1 is the state of the hidden layer at the last time, C t is the cell state under the current time input sequence, is the cell state updated by the output of the forget gate and the input gate, tanh is the activation function, h′ t is the state of the hidden layer at time t before the model is improved, and the output value thereof is changed to h t by the formula.

[0113] In the forward Multi-RLSTM, the input is in time sequence from time 1 to time t, and the mathematical expression of the Multi-RLSTM local spatiotemporal feature extraction model is as follows:

[0114] R(h i , h j ) = linear(concat(h i , h j ))

[0115] h1 = tanh(LSTM1(x))

[0116] h2 = tanh(LSTM2(h1))

[0117] h3 = tanh(LSTM3(h2))

[0118] h4 = tanh(LSTM4(R(h3, h1))) = tanh(LSTM4(linear(concat(h3, h1))))

[0119] h5 = tanh(LSTM5(h4))

[0120] h6 = tanh(LSTM6(R(h5, h2))) = tanh(LSTM6(linear(concat(h5, h2))))

[0121]

[0122] where h i and h j are the outputs of the i-th layer and the j-th layer hidden state respectively, and their shapes are (batch_size, seq_len, hidden_dim), R represents the residual connection function, and concat(h i , h j ) represents concatenating h i and hj linear is a linear transformation.

[0123] The Multi-RLSTM network layer adopts the structure of stacked LSTM, stacks multiple LSTM layers, and introduces dynamic residual connection between the LSTM layers, splices the output of the LSTM of the previous layer and the tensor output by the LSTM layer of the subsequent layer in the feature dimension, and the spliced feature tensor is used as the input of the next layer LSTM.

[0124] Step 4, using a local spatio-temporal feature extraction sub-network to extract local spatio-temporal features of the semantic depth image sequence;

[0125] Specifically, the local spatio-temporal feature extraction sub-network adopts a phased feature extraction strategy. First, the long image sequence X after semantic segmentation is divided into a plurality of image sub-sequences X N×C×T×H×S , where N is the batch size, C is the number of channels, T is the sequence length, H is the picture height, and W is the picture width; the input sequence is divided into picture sub-sequences X (n) ∈R N×C×H×W , which represents the image sequence of the nth sample; the image sub-sequence is input into a 3D convolution layer to extract spatio-temporal features and generate time sequence feature representation; the time sequence data after processing still maintains its time sequence; by optimizing the RLSTM model unit, a Multi-RLSTM model is established to further extract the time sequence features of the sub-sequences and gradually capture higher-level time dependency; a multi-head attention mechanism is introduced after the Multi-RLSTM layer to process the entire sequence and capture global dependency.

[0126] First, a batch of long image sequences is input into a predefined [N, T, C, H, W] five-dimensional tensor. Wherein, the batch size N is set to 8, and the picture sequence length is set to 15. The long sequence is divided into sub-sequences each with a length of 15. The semantic depth image sub-sequences are input into a 3D convolution network to extract local spatio-temporal features of the image sequence; the input image sequence is processed by multiple 3D convolution layers to extract spatio-temporal features and generate time sequence feature representation; the time sequence data after processing still maintains its time sequence. The convolution layer focuses on extracting local spatio-temporal information of the input picture sequence, reflecting the short-term changes of the system, and is suitable for processing short-term oscillation or sudden events of the crane system; the 3D convolution layer uses multiple convolution blocks (3x3 convolution + ELU activation function + batch normalization) to extract features of the input picture sequence, and introduces a residual convolution layer in the convolution layer to relieve the gradient vanishing problem in the deep network, while improving the training efficiency and model performance.

[0127] The 3D convolution kernel slides in the depth, height and width directions of the input data to calculate the weighted sum of the local region. The output Y(d, h, w) at each position (d, h, w) can be represented as:

[0128]

[0129] where Y(d, h, w) is the value of the output data at position (d, h, w); K(c, i, j, k) is the weight of the convolution kernel at channel c and position (i, j, k); X(c, d·S d +i, h·S h +j, w·S w +k) is the value of the input data at channel c and position (d·S d +i, h·S h +j, w·S w +k); C is the number of channels of the input data; K d , K h , K w are the depth, height, and width of the convolution kernel, respectively; S d , S h , S w are the strides in the depth, height, and width directions, respectively.

[0130] The size of the feature map obtained in the previous calculation is reduced after each convolution calculation. The shape of the output feature map after applying a 3D convolution layer can be expressed as follows:

[0131] Output depth:

[0132] Output height:

[0133] Output width:

[0134] where D in , H in , W in are the input depth, width, and height; K d , K h , K W are the depth, height, and width of the convolution kernel; S d , S h , S W are the strides in the depth, height, and width directions; and P is the padding size, which is 0.

[0135] An ELU activation function is introduced after each convolution operation to introduce a nonlinear transformation to the network, enabling the network to fit and represent complex nonlinear relationships and improving the performance of the model.

[0136] A 3D batch normalization layer is introduced after the ELU activation function to accelerate the training process and reduce sensitivity to initialization, which can be expressed as follows:

[0137]

[0138] Residual connections are introduced between different 3D convolutional layers to alleviate the problems of gradient vanishing and gradient explosion, and to accelerate the training of the model. The residual can be expressed as follows:

[0139] Y = F(X, {S}) + X

[0140] Where F represents the mapping function from input X to output Y, and {W} is the weight of the layer. Through backpropagation, the gradient can be directly transmitted to the shallow layer:

[0141]

[0142] Secondly, by optimizing the model unit RLSTM, a Multi-RLSTM local spatio-temporal feature extraction model is established. The features extracted by 3D convolution on the sub-sequence are input into the Multi-RLSTM local spatio-temporal feature extraction model to further extract high-dimensional time sequence features of the sub-sequence, which is specifically:

[0143] R(h i , h j ) = linear(concat(h i , h j ))

[0144] h1 = tanh(LSTM1(x))

[0145] h2 = tanh(LSTM2(h1))

[0146] h3 = tanh(LSTM3(h2))

[0147] h4 = tanh(LSTM4(R(h3, h1))) = tanh(LSTM4(linear(concat(h3, h1))))

[0148] h5 = tanh(LSTM5(h4))

[0149] h6 = tanh(LSTM6(R(h5, h2))) = tanh(LSTM6(linear(concat(h5, h2))))

[0150]

[0151] Finally, a multi-head attention mechanism is introduced after the Multi-RLSTM local spatio-temporal feature extraction model. The local features of the sub-sequence are combined, and the entire sequence is input into the multi-head attention mechanism to capture the global dependency relationship of the entire sequence, which is specifically:

[0152] Output features of the concatenated sub-sequences:

[0153] F i = f(chunk i )

[0154] F concat = [F1; F2;...; F n ]

[0155] chunk i represents the i-th sub-sequence, f(chunk i ) represents the process of feature extraction on the i-th sub-sequence through 3D convolution and Multi-RLSTM layers, F i represents the feature representation of the i-th sub-sequence, with dimension (N, T, D), F concat represents the feature representation after concatenation in the time dimension, with dimension (N, nT, D).

[0156] Input the feature after concatenation of sub-sequences into the multi-head attention module:

[0157] Q i = F concat W i Q

[0158] K i = F concat W i K

[0159] V i = F concat W i V

[0160] where F concat is the representation of the input sequence, W i Q , W i K , W i V are the weight matrices of Q, K and V of the i-th attention head, respectively, for each attention head, the similarity between Q and K is calculated by using the dot product operation, and normalized by the Softmax function:

[0161]

[0162] MultiHead(Q, K, V) = Concat(head1, head2, head3) W O

[0163] The outputs of all heads are spliced, and a linear transformation generates the output of the multi-head self-attention module, with a dimension of (N, T, D). The features of the sub-sequences are aggregated and mapped to the final output using a fully connected layer. Specifically, the output of each sub-sequence not only contains the feature representation of the sub-sequence itself, but also contains the information of the entire sequence, which can generate more comprehensive and detailed feature representations. The features of each sub-sequence are aggregated using max-pooling, and a fully connected layer is introduced after the pooling layer to generate the final prediction result. Specifically:

[0164]

[0165] where n represents the batch index, c represents the sub-sequence index, d represents the feature dimension index, and t represents the time step index. The above formula represents average pooling for each sub-sequence time step.

[0166] The pooled input is linearly transformed by the fully connected layer to gradually reduce the number of features from the input tensor to the shape of the output tensor, and finally mapped to the movement control instruction of the crane.

[0167] The local spatio-temporal prediction sub-network focuses on modeling the spatio-temporal relationship of the input sub-image sequence, and understands the movement trend of the crane from a local and short-term perspective, and predicts the movement trend of the crane.

[0168] Step 5, use the global spatio-temporal feature extraction sub-network to extract the features of the semantic RGB image sequence and predict the movement control instruction;

[0169] Specifically, the global spatio-temporal feature extraction sub-network predicts the input long-time sequence image. The input tensor is frames∈R N×C×T×H×W where N=8 is the batch size, C=3 is the channel number, T=15 is the time step number, H=160 and W=210 are the height and width of the image respectively, i.e. frames={f n，c，t，h，w}. The reshape operation is performed on the input tensor to combine the N and T dimensions into N×T, i.e. frames'={f' n′，c，h，w}, n'=N×T, frames∈R 120×3×160×210The global spatio-temporal feature extraction subnetwork combines ResNetl8 and Transformer architecture. First, the reshaped entire image sequence is input into the modified ResNetl8 model for preliminary feature extraction. ResNetl8 serves as the image position encoder, extracting the spatial features of each image through convolution operations and converting them into fixed-size feature vectors. Meanwhile, a linear layer is used to map the extracted ResNet features to 512 dimensions as input for the subsequent temporal model. In the network architecture, ResNetl8 is chosen as the encoder for the image sequence because ResNetl8 has multiple convolutional layers and residual connections, which can extract complex spatial features in crane operations, such as object displacement and rocker position changes, thereby extracting rich spatial information at each time step to help the model better understand the crane's operating state. Then the network uses a position encoding module to introduce the sequence's position information. This step introduces position information into the feature sequence, enabling the Transformer to perceive the order in the frame sequence and help capture global temporal dependencies. Position encoding is generated by a predefined function PE(t), which is: Next, the network inputs the position-encoded image sequence into the Transformer-based temporal encoder, which is: is the feature vector after the input image is extracted by Resnetl8 and the features are mapped to 512-dimensional features. The Transformer can consider all past states at each time step and identify key time points and trends through the multi-head attention mechanism. This allows the model to quickly capture sudden events (such as sudden shaking) in the short term while maintaining tracking of long-term trends, thereby improving prediction accuracy. The combination of ResNetl8 and Transformer architecture enables the model to make decisions based on comprehensive information from the entire operation sequence, rather than being limited to single-frame images or local time periods, and better captures changes in motion patterns caused by complex dynamics.

[0170] Finally, the feature vector f = {f t，n，k}∈R T×N×d where T = 15 is the sequence length, N = 8 is the batch size, and d is the feature dimension. Max-pooling is performed in the time dimension T, which is: is the feature after max pooling. The dimensionality-reduced feature is mapped to the final crane motion control instruction through a fully connected layer. The global spatio-temporal feature prediction subnetwork understands the long-term motion of the crane by extracting features from the entire batch of images and modeling the temporal relationship of the sequence with the help of position encoding and the transformer, and predicts the motion trend of the crane from a global perspective.

[0171] Step 6, finally, the outputs of the two branches are fused by a weighted fusion network to obtain the final output of the model; specifically, X local ∈R N×d represents the crane motion state predicted by the local spatio-temporal prediction subnetwork, X global ∈R N×d represents the crane motion state predicted by the global spatio-temporal prediction subnetwork, the fusion of the two subnetworks adopts a weighted manner, y fused =α·X local +β·X global , where alpha = 0.4 and beta = 0.6 are the weighting coefficients, representing the different emphasis on local and long-term prediction results, and the final fused prediction result y fused ∈R N×d , d is the total number of motion control instructions, representing the crane motion control instruction prediction after combining local and long-term information decision.

[0172] Step 7, the motion control instruction of the operating end collected is taken as the label of the image, the loss function is optimized by minimizing the deviation between the predicted value and the true value, and the neural network model is trained to obtain the prediction model, specifically:

[0173] The depth image sequence, the RGB image sequence, and the motion control instruction of the operating end collected are input into the semantic RLSTM-Transformer crane end-to-end motion trajectory prediction model, the motion control instruction is taken as the label of the image, and the network calculates the predicted value according to the weight and bias; the predicted result obtained by forward propagation is compared with the actual label value, and the cross-entropy loss is calculated to measure the gap between the predicted value and the true value; the gradient calculated by back propagation is used to update the weight and bias of the network according to the gradient descent method, to reduce the value of the loss function, and train the semantic RLSTM-Transformer crane end-to-end motion trajectory prediction model.

[0174] Step 8, when executing online, the depth image and the RGB image collected in real time by the RGBD camera under the crane operation scene are obtained, the images are input into the trained semantic RLSTM-Transformer crane end-to-end motion trajectory prediction model, the crane motion control instruction is predicted, and the end-to-end intelligent motion control of the crane is realized.

Claims

1. A crane end-to-end motion trajectory prediction method based on RGBD images, characterized by: It includes two stages: training and prediction, and specifically includes the following steps: Step 1: Collect experimental data. Use an RGBD camera to obtain RGB images and depth images from the perspective of the crane cab during operation, and simultaneously obtain motion control commands from the operator. The motion control commands from the operator include: C1-ascend, C2-descend, C3-increase, C4-decrease, C5-rotate left, C6-rotate right, C7-forward, C8-backward, C9-grab, and C10-release. Step 2: Segment the RGB image based on the semantic segmentation model to obtain a pseudo-color image and a category index image. Overlay the pseudo-color image and the RGB image with a certain transparency to obtain a semantic RGB image. At the same time, fuse the category index image and the depth image. Fuse the single-channel category index image and the single-channel depth image, and set the third channel to blank to obtain a three-channel semantic depth image. Step 3: Establish a Multi-RLSTM local spatiotemporal feature extraction model by optimizing the model unit RLSTM; Step 4: Input the semantic depth image into the local spatiotemporal feature extraction sub-network. The local spatiotemporal feature extraction sub-network uses CNN, Multi-RLSTM and multi-head attention modules to extract the local temporal features of the input sub-image sequence and predict the output of each sub-sequence. Step 5: Input the semantic RGB image into the global spatiotemporal feature extraction sub-network. The global spatiotemporal feature extraction sub-network uses ResNet and Transformer to extract the deep spatial features of the image sequence and establish global dependencies, and predict the output of each sub-sequence. Step 6: Finally, the outputs of the two branches are fused through the weighted fusion network to obtain the final output of the model; X local ∈R N×d represents the crane motion state predicted by the local spatiotemporal prediction subnetwork, X global ∈R N×d Represents the crane motion state predicted by the global spatiotemporal prediction sub-network. The fusion of these two sub-networks adopts a weighted approach. fused =α·X local +β·X global , where α and β are weighted coefficients, indicating the different degrees of importance attached to local and long-term prediction results. The final fused prediction result y fused ∈R N×d , d is the total number of categories of crane motion control instructions, which represents the prediction of crane motion control instructions after combining local and long-term information decision; Step 7: The motion control instructions collected from the operator are used as labels for the images. By minimizing the deviation between the predicted value and the true value, the loss function is optimized and the above neural network model is trained to obtain a prediction model. Step 8: During the online execution phase, RGB images and depth images captured by the RGBD camera from the perspective of the crane cab in the crane operation scenario are obtained. The images are input into the trained semantic RLSTM-Transformer crane end-to-end motion trajectory prediction model. The required crane motion control instructions can be obtained through model prediction.

2. The method for predicting end-to-end crane motion trajectory based on RGBD images according to claim 1, characterized in that: In step 1, each set of data contains a complete crane loading process. Data is collected under different lighting conditions, by adjusting the relative position of the sand pile and the truck bed, simulating different wind levels, and changing the weight of the lifted load. The collected image resolution is 640*480, and it is later cropped to 160*210.

3. The crane end-to-end motion trajectory prediction method based on RGBD images according to claim 1 is characterized in that Step 2: The method for making a data set is: The grab, material, bucket, and background in the RGB image are segmented based on the semantic segmentation model. The model inference obtains a semantic segmentation pseudo-color image and a semantic segmentation category index image containing the grab, material, bucket, and background. The pseudo-color image containing the grab, material, bucket, and background is superimposed with the original image at a certain transparency to obtain an RGB image containing semantic information. The depth image and the semantic segmentation category index image are fused to obtain a depth image with semantic information. The fusion process can be expressed by the following formula. The output of the semantic segmentation model is expressed as: I pseudo ,I index =Segmentation(I RGB ) Among them, I RGB is the input RGB image, I pseudo is the pseudo-color image output by the semantic segmentation model, I index It is the single-channel category index map output by the semantic segmentation model; The fusion process of pseudo color image and RGB image is: I semantic_RGB =α·I pseudo +(1-α)·I RGB Among them, α is the transparency coefficient, I semantic_RGB It is the fused semantic RGB image.

4. The method for predicting end-to-end crane motion trajectory based on RGBD images according to claim 3, characterized in that: Three-channel semantic depth map I semantic_depth The way to generate is to use the category index map as the first channel, the depth map as the second channel, and the blank channel as the third channel, which is expressed as: I semantic_depth =Concatenate(I index ,I depth ,I empty ) I depth is the input single-channel depth map, I empty For blank channels, I empty Size and I depth and I index Similarly, Concatenate(·) means concatenating images in the channel dimension; Each semantic RGB image corresponds to a semantic depth image. The timestamp is used to align the image with the motion control instructions of the operator. The motion control instructions of the operator are used as the label of the image. The dataset is preprocessed to obtain the preprocessed dataset.

5. The method for predicting the end-to-end motion trajectory of a crane based on RGBD images according to claim 1, characterized in that: The forward propagation calculation formula of the optimized model unit RLSTM in step 3 is: f t =σ(W f ·[h t-1 ,x t ]+b f ) i t =σ(W i ·[h t-1 ,x t ]+b i ) the t =σ(W O [h t-1 ,x t ]+b O ) h′ t =o t *fishy(C) t ) h t =tanh(h′ t +W h ·h′ t-1 +b h ) σ is the Sigmoid activation function, W f , W i , W o are the weight parameters of the forget gate, input gate, and output gate respectively; b f , b i , b o are the bias parameters of the forget gate, input gate, and output gate, respectively, and f t ,i t , o t They are respectively the forget gate, input gate, and output gate, x t is the current sequence input, h t-1 is the state of the hidden layer at the previous moment, C t is the cell state under the current input sequence, is the cell state after the output of the forget gate and input gate is updated, tanh is the activation function, h′ t is the output of the hidden layer of the original LSTM at time t, h t is the state output of the hidden layer at time t after the change; The RLSTM model unit introduces a residual connection of the hidden state of the previous time step into the calculation of the hidden state of each time step in the LSTM, and introduces a learnable weight W h and bias b h To adjust the contribution of the residual connection.

6. The method for predicting end-to-end crane motion trajectory based on RGBD images according to claim 3, characterized in that: In step 3, the forward Multi-RLSTM is fed with data from time 1 to time t in chronological order. The local spatiotemporal feature extraction model of the Multi-RLSTM is: R(h i ,h j )=linear(concat(h i ,h j )) h1=tanh(LSTM1(x)) h2=tanh(LSTM2(h1)) h3=tanh(LSTM3(h2)) h4=tanh(LSTM4(R(h3,h1)))=tanh(LSTM4(linear(concat(h3,h1)))) h5=tanh(LSTM5(h4)) h6=tanh(LSTM6(R(h5,h2)))=tanh(LSTM6(linear(concat(h5,h2)))) where h i and h j are the outputs of the hidden states of the i-th layer and the j-th layer, respectively, and their shapes are (batch_size, seq_len, hidden_dim), R represents the dynamic residual connection function, concat(h i ,h j ) means concatenating h on the last dimension i and h j , linear is a linear transformation.

7. The method for predicting end-to-end motion trajectory of a crane based on RGBD images according to claim 1, characterized in that: The local spatiotemporal feature extraction subnetwork in step 4 is specifically: The local spatiotemporal feature extraction sub-network adopts a staged feature extraction strategy. First, the long image sequence X∈R N×C×T×H×W , where N is the batch size, C is the number of channels, T is the sequence length, H is the image height, and W is the image width; the input sequence is divided into image subsequences X with each sample length T (n) ∈R N×C×H×W , representing the image sequence of the nth sample; the image subsequence is input into the 3D convolution layer to extract spatiotemporal features and generate a temporal feature representation. After processing, the temporal data can still maintain its own temporal sequence; by optimizing the RLSTM model unit and establishing a Multi-RLSTM model, the temporal features of these subsequences are further extracted, gradually capturing higher-level temporal dependencies; A multi-head attention mechanism is introduced after the Multi-RLSTM layer to process the entire sequence and capture global dependencies. Specifically: F i =f(chunk i ) F concat =[F1;F2;…;F n ] chunk i represents the i-th subsequence, f(chunk i ) represents the process of feature extraction of the i-th subsequence through 3D convolution and Multi-RLSTM layer, F i Represents the feature representation of the i-th subsequence, with dimensions (N, T, D), F concat It is the feature representation after splicing in the time dimension, with dimensions (N,nT,D) Among them F concat is the representation of the input sequence, are the weight matrices of Q, K, and V of the i-th attention head, respectively. For each attention head, the similarity between Q and K is calculated by using the dot product operation and normalized by the Softmax function: MultiHead(Q,K,V)=Concat(head1,head2,head3)W O Concatenate the outputs of all heads and perform linear transformation to generate the output of the multi-head self-attention module with the dimension (N, nT, D); Finally, the tensor output by the multi-head attention module is subjected to a maximum pooling operation in the time dimension, and the extracted features are then processed through a fully connected layer and finally mapped to crane motion control instructions.

8. The method for predicting end-to-end motion trajectory of a crane based on RGBD images according to claim 1, characterized in that: The global spatiotemporal feature extraction subnetwork in step five combines the ResNet and Transformer architectures. The global spatiotemporal feature prediction subnetwork extracts features from the entire batch of images and uses position encoding and transformer modeling to understand the long-term motion of the crane and predict the crane's motion trend from a global perspective.

9. The method for predicting the end-to-end motion trajectory of a crane based on RGBD images according to claim 8 is characterized in that The specific steps of step five are: The global spatiotemporal feature extraction subnetwork predicts the input long time series image, and the input tensor is frames∈R N ×C×T×H×W , where N is the batch size, C is the number of channels, T is the number of time steps, H and W are the height and width of the image respectively, i.e. frames = {f n,c,t,h,w }; Reshape the input tensor and merge the N and T dimensions into N×T, i.e. frames′={f′ n′,c,h,w },n′=N×T,frames∈R NR×X×H×W ; First, the entire reshaped image sequence is input into the modified ResNet18 model for preliminary feature extraction; ResNet18 is used as the position encoder of the image. It extracts the spatial features of each image through convolution operations and converts them into fixed-size feature vectors. At the same time, a linear layer is used to map the extracted ResNet features to a low dimension as the input of the subsequent time series model. Then, the global spatiotemporal feature extraction subnetwork uses the position encoding module to introduce the position information of the sequence. By introducing position information into the feature sequence, the Transformer can perceive the sequential relationship in the frame sequence and help capture the global temporal dependency. The position encoding is generated by the predefined function PE(t), specifically: The global spatiotemporal feature extraction subnetwork inputs the position-encoded image sequence into the Transformer-based temporal encoder, specifically: It is the feature vector of the input image after Resnet18 extracts features and maps the features to low-dimensional features; Finally, the feature vector f={f t,n,k }∈R T×N×d , where T is the sequence length, N is the batch size, d is the feature dimension, and the maximum pooling operation is performed in the time dimension T, specifically: It is the feature after maximum pooling; The pooled features are subjected to dimensionality reduction processing, and then the reduced features are mapped to the final crane motion control instructions through a fully connected layer.

10. The method for predicting end-to-end motion trajectory of a crane based on RGBD images according to claim 1, characterized in that: The specific method of step seven is: input the depth image sequence, RGB image sequence, and the collected motion control instructions of the operating end into the semantic RLSTM-Transformer crane end-to-end motion trajectory prediction model, and use the motion control instructions as the image label. The network calculates the predicted value based on the weight and bias; compares the predicted result obtained by forward propagation with the actual label value, and calculates the cross-entropy loss to measure the gap between the predicted value and the true value; according to the gradient calculated by backpropagation, the weights and bias of the network are updated according to the gradient descent method to reduce the value of the loss function and train the semantic RLSTM-Transformer crane end-to-end motion trajectory prediction model.

Images