Method for predicting end-to-end movement track of crane based on RGBD image

Abstract

The invention discloses a crane end-to-end motion trail prediction method based on RGBD images, which belongs to the field of intelligent control, and comprises two stages of training and prediction: during training, inputting an RGB image sequence and a depth image sequence of a cab visual angle collected during crane operation and a motion control instruction of an operation end; segmenting the RGB image based on a semantic segmentation network to obtain a semantic segmentation map, and fusing the category index map and the depth map to obtain a semantic depth map; and the RLSTM-Transform double-branch network extracts the features of the semantic segmentation map and the semantic depth map and predicts a motion control command. Minimizing the deviation between the predicted value and the true value, and training the neural network model to obtain a prediction model; and during prediction, inputting the RGB image and the depth image into the trained semantic RLSTM-Transform crane end-to-end motion track prediction model, and predicting and outputting a motion control command of the crane by the model.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent control of cranes, and in particular relates to a method for predicting end-to-end motion trajectory of cranes based on RGBD images. Background Art

[0002] Against the backdrop of the rapid development of intelligent manufacturing and industrial automation, improving the autonomy level and overall performance of crane systems has become a key research direction.

[0003] The intelligent control of cranes has always faced many challenges, especially due to their under-actuated properties, the control system often needs to deal with redundant degrees of freedom and complex dynamic characteristics. Specifically, the crane's spreader and load will produce nonlinear swings, and external interference (such as wind, load instability, etc.) will also have a negative impact on control accuracy and operational safety. Therefore, existing crane control methods usually rely on local control, that is, segmented control of specific subsystems or operating processes. Although local control can improve operational performance to a certain extent, its limitations become more obvious when faced with more complex working conditions and multivariable control. This control method is difficult to achieve global optimization, resulting in low operational efficiency and the intelligence level of the system is still at a low stage.

[0004] In contrast, end-to-end control technology has made breakthrough progress in the field of autonomous vehicles in recent years. End-to-end control builds a unified model directly from sensor data (such as cameras, lidar, etc.) to control output by closely integrating perception, decision-making and control. This method no longer relies on the traditional hierarchical control structure, but uses artificial intelligence technologies such as deep learning to autonomously learn the optimal decision-making in complex environments from a large amount of data. This enables autonomous vehicles to perform global planning and control more accurately and flexibly when facing complex traffic environments, dynamic scenarios and diverse inputs, showing an excellent level of intelligence.

[0005] However, compared with the field of autonomous driving, the end-to-end control research of crane systems is still blank. Most of the current intelligent control of cranes relies on local algorithms and lacks a global and unified intelligent control framework, which shows certain deficiencies in dealing with complex working conditions, dynamic environments and diverse task requirements. However, the end-to-end control of vehicles is different from that of cranes. Cranes are typical under-actuated systems, especially because of the presence of ropes, which causes the load to swing during operation, increasing the uncertainty and complexity of the system; while autonomous vehicles are fully driven systems, and the control tasks are relatively straightforward. The operation scene of a crane is usually a small-scale unstructured scene with certain dynamics. External interference (such as wind, ground vibration) and rope swing make control more challenging; while the operation scene of an autonomous vehicle is a relatively structured road environment, and the control tasks focus more on global navigation and dynamic obstacle avoidance.

[0006] By introducing an end-to-end control method, the crane system can directly output control signals from sensor data, achieve global optimization and dynamic decision-making, and thus improve the automation level, operational efficiency and safety of the system. Especially in the face of complex environments or task requirements, end-to-end control is expected to overcome the bottleneck of traditional local control and provide a more intelligent and robust control solution.

[0007] Therefore, this paper proposes an end-to-end motion trajectory prediction method based on RGBD images for crane systems. This method collects RGB images, depth images and crane motion control instructions from the operator's cab in the crane operation scene, segments the RGB images based on the semantic segmentation model, obtains pseudo-color images and category index images, superimposes the pseudo-color images and RGB images with a certain degree of transparency, obtains semantic RGB images, and fuses the category index images and depth images at the same time. The single-channel category index images and single-channel depth images are fused, and the third channel is set to blank to obtain a three-channel semantic depth image. The features of the input image sequence are extracted through the RLSTM-Transformer model, the motion control instructions of the crane are predicted, and the end-to-end intelligent control of the crane is realized. Summary of the invention

[0008] The technical problem to be solved by the present invention is to propose an end-to-end motion trajectory prediction method based on RGBD images for crane systems, so as to solve the problems of low information transmission efficiency, difficulty in coping with various complex environments, and difficulty in achieving overall control in the control methods that separate perception and control in existing crane research. By introducing the end-to-end control method, the crane system can directly output control signals from sensor data, achieve global optimization and dynamic decision-making, and thus improve the automation level, operational efficiency, and safety of the system.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is:

[0010] An end-to-end crane motion trajectory prediction method based on RGBD images includes two stages: training and prediction. The steps of the training stage are steps 1 to 7, and the step of the prediction stage is step 8:

[0011] Step 1: Use the RGBD camera to obtain the RGB image and depth image from the cab's perspective when the crane is operating, and at the same time obtain the motion control instructions of the operator. The motion control instructions of the operator include: C1-rise, C2-fall, C3-increase, C4-decrease, C5-left rotation, C6-right rotation, C7-forward, C8-backward, C9-grasp, C10-release;

[0012] Step 2: Segment the RGB image based on the semantic segmentation model to obtain a pseudo-color image and a category index image. Overlay the pseudo-color image and the RGB image with a certain degree of transparency to obtain a semantic RGB image. At the same time, fuse the category index image and the depth image. Fuse the single-channel category index image and the single-channel depth image, and set the third channel to blank to obtain a three-channel semantic depth image. The output of the semantic segmentation model is expressed as:

[0013] I pseudo , I index =Segmentation(I RGB )

[0014] Among them, I RGB is the input RGB image, I pseudo is the pseudo-color image output by the semantic segmentation model, I index It is a single-channel category index map output by the semantic segmentation model.

[0015] The fusion process of pseudo color image and RGB image is:

[0016] I semantic_RGB =α·I pseudo +(1-α)·I RGB

[0017] Among them, α is the transparency coefficient, I semantic_RGB It is the fused semantic RGB image.

[0018] Three-channel semantic depth map I semantic_depth The way to generate is to use the category index map as the first channel, the depth map as the second channel, and the blank channel as the third channel, expressed as:

[0019] I semantic_depth =Concatenate(I index , I depth , Iempty )

[0020] I depth is the input single-channel depth map, I empty For blank channels, I empty Size and I depth and I index Concatenate(·) means concatenating images in the channel dimension.

[0021] Step 3: Establish a Multi-RLSTM local spatiotemporal feature extraction model by optimizing the model unit RLSTM;

[0022] Step 4: Input the semantic depth image into the local spatiotemporal feature extraction subnetwork. The local spatiotemporal feature extraction subnetwork uses CNN, Multi-RLSTM and multi-head attention modules to extract the local temporal features of the input sub-image sequence and predict the output of each sub-sequence. The local spatiotemporal feature extraction subnetwork adopts a staged feature extraction strategy. First, the long image sequence X∈R N×C×T×H×W , where N is the batch size, C is the number of channels, T is the sequence length, H is the image height, and W is the image width; the input sequence is divided into image subsequences X with each sample length T (n) ∈R N×C×H×w , representing the image sequence of the nth sample; the image subsequence is input into the 3D convolution layer to extract the spatiotemporal features and generate a temporal feature representation. The time series data after processing can still maintain its own temporal nature; by optimizing the RLSTM model unit, a Multi-RLSTM model is established to further extract the temporal features of these subsequences and gradually capture higher-level temporal dependencies; a multi-head attention mechanism is introduced after the Multi-RLSTM layer to process the entire sequence and capture global dependencies.

[0023] Step 5: Input the semantic RGB image into the global spatiotemporal feature extraction subnetwork, which uses ResNet and Transformer to extract the deep spatial features of the image sequence and establish global dependencies, and predict the output of each subsequence.

[0024] Step 6: Finally, the outputs of the two branches are fused through the weighted fusion network to obtain the final output of the model. local ∈R N×d represents the crane motion state predicted by the local spatiotemporal prediction subnetwork, X global ∈R N×d represents the crane motion state predicted by the global spatiotemporal prediction subnetwork. The fusion of these two subnetworks adopts a weighted approach. fused =α·X local +β·Xglobal , where α and β are weighted coefficients, indicating the different emphasis on local and long-term prediction results. The final fused prediction result y fused ∈R N×d , d is the total number of categories of crane motion control instructions, which represents the prediction of crane motion control instructions after combining local and long-term information decisions.

[0025] Step 7: The collected motion control instructions of the operator are used as the label of the image. By minimizing the deviation between the predicted value and the true value, the loss function is optimized, and the above neural network model is trained to obtain the prediction model.

[0026] Step 8: During online execution, obtain the RGB image and depth image captured by the RGBD camera from the perspective of the cab in the crane operation scenario, input the image into the trained semantic RLSTM-Transformer crane end-to-end motion trajectory prediction model, and obtain the required crane motion control instructions through model prediction.

[0027] A further improvement of the technical solution of the present invention is that the method for preparing the data set is:

[0028] The RGBD camera is used to obtain the RGB image and depth image from the perspective of the cab when the crane is operating, and at the same time obtain the motion control instructions of the operator;

[0029] Based on the semantic segmentation model, the grab, material, bucket, and background in the RGB image are segmented. The model inference obtains the semantic segmentation pseudo-color image and semantic segmentation category index image containing the grab, material, bucket, and background. The pseudo-color image containing the hook, material, bucket, and background output by the segmentation is superimposed with the original image at a certain transparency to obtain an RGB image containing semantic information; the depth image and the semantic segmentation category index image are fused to obtain a depth image with semantic information. The output of the semantic segmentation model is expressed as:

[0030] I pseudo , I index =Segmentation(I RGB )

[0031] Among them, I RGB is the input RGB image, I pseudo is the pseudo-color image output by the semantic segmentation model, I index It is a single-channel category index map output by the semantic segmentation model.

[0032] The fusion process of pseudo color image and RGB image is:

[0033] I semantic_RGB =α·I pseudo +(1-α)·IRGB

[0034] Among them, α is the transparency coefficient, I Semantic_RGB It is the fused semantic RGB image.

[0035] Three-channel semantic depth map I semantic_depth The way to generate is to use the category index map as the first channel, the depth map as the second channel, and the blank channel as the third channel, expressed as:

[0036] I semantic_depth =Concatenate(I index , I depth , I empty )

[0037] I depth is the input single-channel depth map, I empty is a blank channel, i empty Size and I depth and I index Concatenate(·) means concatenating images in the channel dimension.

[0038] Each semantic RGB image corresponds to a semantic depth image. The timestamp is used to align the image with the collected crane motion control instructions of the operator. The crane motion control instructions are used as the labels of the images, and the data set is preprocessed to obtain the preprocessed data set.

[0039] The further improvement of the technical solution of the present invention is that the specific calculation formula of the optimization model unit RLSTM is:

[0040] f t =σ(W f ·[h t-1 , x t ]+b f )

[0041] i t =σ(W i ·[h t-1 , x t ]+b i )

[0042]

[0043] o t =σ(W O [h t-1 , x t ]+b O )

[0044] h′ t =o t*tanh(C t )

[0045] h t =tanh(h′ t +W h ·h′ t-1 +b h )

[0046] σ is the Sigmoid activation function, W f , W i , W o are the weight parameters of the forget gate, input gate, and output gate respectively; b f , b i , b o are the bias parameters of the forget gate, input gate, and output gate, respectively, and f t ,i t , o t They are respectively the forget gate, input gate, and output gate, x t Input for the current sequence, h t-1 is the state of the hidden layer at the previous moment, C t is the cell state under the input sequence at the current moment, is the cell state after the output of the forget gate and input gate is updated, tanh is the activation function, h′ t To modify the output of the hidden layer of the previous LSTM at time t, h t is the output of the hidden layer of the modified RLSTM at time t. The RLSTM model unit introduces the residual connection of the hidden state of the previous time step to calculate the hidden state of each time step in the LSTM, and introduces the learnable weight W h and bias b h To adjust the contribution of the residual connection. The delayed residual connection can transfer the hidden state of the previous time step directly to the current time step more quickly, allowing the model to respond more quickly to rapid changes in the crane state.

[0047] The further improvement of the technical solution of the present invention is that the Multi-RLSTM local spatiotemporal feature extraction model is specifically:

[0048] R(h i ,h j )=linear(concat(h i ,h j ))

[0049] h1=tanh(LSTM1(x))

[0050] h2 = tanh(LSTM2(h1))

[0051] h3 = tanh(LSTM3(h2))

[0052] h4=tanh(LSTM4(R(h3, h1)))=tanh(LSTM4(linear(concat(h3, h1))))

[0053] h5=tanh(LSTM5(h4))

[0054] h6=tanh(LSTM6(R(h5, h2)))=tanh(LSTM6(linear(concat(h5, h2))))

[0055]

[0056] where h i and h j are the outputs of the hidden states of the i-th layer and the j-th layer, respectively, and their shapes are (batch_size, seq_len, hidden_dim), R represents the residual connection function, concat(h i ,h j ) means concatenating h in the last dimension i and h j , linear is a linear transformation. In the Multi-RLSTM local spatiotemporal feature extraction model, a stacked LSTM structure is used. Each layer of RLSTM can learn different levels of temporal features, so that the model can gradually extract rich high-dimensional features from shallow to deep layers. Dynamic residual connections are introduced between RLSTM layers to splice the output of the previous LSTM layer with the output of the subsequent LSTM layer in the feature dimension. This dynamic residual connection allows information to propagate quickly in the network, which helps to improve the model's response speed to changes in the joystick trajectory, especially when facing complex or rapid dynamic changes, it can capture the changing trend more quickly.

[0057] A further improvement of the technical solution of the present invention is that: the local spatiotemporal feature extraction subnetwork is specifically:

[0058] The local spatiotemporal feature extraction subnetwork adopts a staged feature extraction strategy. First, the long image sequence X∈R N×C×T×H×W , where N is the batch size, C is the number of channels, T is the sequence length, H is the image height, and W is the image width. Divide the input sequence into image subsequences X with each sample length T (n) ∈R N×C×H×w, represents the image sequence of the nth sample. The image subsequence is input into the 3D convolution layer. The convolution layer focuses on extracting the local spatiotemporal information of the input image sequence, reflecting the short-term changes of the system, and is suitable for processing short-term oscillations or emergencies of the crane system. The 3D convolution layer uses multiple convolution blocks (3×3 convolution + ELU activation function + batch normalization) to extract the features of the input image sequence. The residual convolution layer is introduced in the convolution layer to alleviate the gradient vanishing problem in the deep network, while improving the training efficiency and model performance.

[0059] By optimizing the RLSTM model unit and establishing a Multi-RLSTM model, we can further extract the temporal features of these subsequences and gradually capture higher-level temporal dependencies.

[0060] A multi-head attention mechanism is introduced after the Multi-RLSTM layer to process the entire sequence and capture global dependencies. Specifically:

[0061] F i =f(chunk i )

[0062] F co n cat =[F1; F2; ...; F n ]

[0063] chunk i represents the ith subsequence, f(chunk i ) represents the process of extracting features from the i-th subsequence through 3D convolution and Multi-RLSTM layers, F i represents the feature representation of the i-th subsequence, with dimensions (N, T, D), F concat It is the feature representation after splicing in the time dimension, with dimensions (N, nT, D)

[0064] Q i =F concat W i Q

[0065] K i =F concat W i K

[0066] V i =F concat W i V

[0067] Among them, F concat is the representation of the input sequence, W i Q , W iK , W i V are the weight matrices of Q, K, and V of the i-th attention head, respectively. For each attention head, the similarity between Q and K is calculated by using the dot product operation, and the similarity between Q and K is calculated by Sof t The max function performs normalization:

[0068]

[0069] MultiHead(Q,K,V)=Concat(head1,head2,head3)W O

[0070] The outputs of all heads are concatenated and linearly transformed to generate the output of the multi-head self-attention module with a dimension of (N, nT, D).

[0071] The features of each subsequence are aggregated using the maximum pooling method, and then a fully connected layer is introduced after the pooling layer to generate the final prediction result. Specifically:

[0072]

[0073] Among them, n represents the batch index, c represents the subsequence index, d represents the feature dimension index, and t represents the time step index. The above formula represents the average pooling of the time steps of each subsequence. The pooled input is linearly transformed to the fully connected layer to gradually reduce the number of features of the input tensor to the shape of the output tensor, and the crane motion control instructions are output.

[0074] A further improvement of the technical solution of the present invention is that: the global spatiotemporal feature extraction subnetwork is specifically:

[0075] The global spatiotemporal feature extraction subnetwork predicts the input long time series image, and the input tensor is frames∈R N×C×T×H×W , where N is the batch size, C is the number of channels, T is the number of time steps, H and W are the height and width of the image, that is, frames = {f n，c，t，h,w}. Reshape the input tensor and merge the N and T dimensions into N×T, that is, frames′={f′ n′，c，h，w}, n′=B×T, frames∈R NT×C×H×S. The global spatiotemporal feature extraction subnetwork combines the ResNet18 and Transformer architectures. First, the entire reshaped image sequence is input into the modified ResNet1 8 model for preliminary feature extraction. ResNet18, as the position encoder of the image, extracts the spatial features of each image through convolution operations and converts them into a fixed-size feature vector. At the same time, a linear layer is used to map the extracted ResNet features to a low dimension as the input of the subsequent time series model. In the network architecture, ResNet18 is selected as the encoder of the image sequence because ResNet18 has multiple layers of convolution and residual connections, which can extract complex spatial features in crane operation, such as the displacement of objects, the position change of the rocker, etc., thereby extracting rich spatial information at each time step to help the model better understand the operating status of the crane. Then the network uses the position encoding module to introduce the position information of the sequence. This step enables the Transformer to perceive the sequential relationship in the frame sequence by introducing the position information in the feature sequence, helping to capture the global temporal dependency. The position encoding is generated by the predefined function PE(t), specifically: Next, the network inputs the position-encoded image sequence into the Transformer-based temporal encoder, specifically: It is the feature vector of the input image after Resnet18 extracts features and maps the features to low-dimensional features. Transformer can consider all past states at the same time at each time step, and identify key time points and change trends through a multi-head attention mechanism. This enables the model to quickly capture short-term emergencies (such as sudden jitters) while keeping track of long-term trends, thereby improving the accuracy of predictions. The combination of ResNet18 and Transformer architectures means that the model is no longer limited to a single frame or a local time period, but can make decisions based on the comprehensive information of the entire operation sequence, and can better capture the changes in motion patterns brought about by complex dynamics.

[0076] Finally, the feature vector f = {f t，n，k}∈R T×N×d , where T is the sequence length, N is the batch size, and d is the feature dimension. The maximum pooling operation is performed in the time dimension T, specifically: It is the feature after maximum pooling. The pooled feature is processed by dimensionality reduction, and then the reduced feature is mapped to the final crane motion control command output through the fully connected layer. The global spatiotemporal feature prediction subnetwork extracts features from the entire batch of images and uses the temporal relationship of the position encoding and transformer modeling sequence to understand the long-term movement of the crane and predict the movement trend of the crane from a global perspective.

[0077] Compared with the prior art, the above technical solution adopted by the present invention has the following advantages:

[0078] This application fuses semantic information with RGB information, and also fuses semantic information with depth information. The semantic RGB image not only contains the original visual information, but also clearly distinguishes key objects such as grabs, materials, and truck buckets, allowing the model to better understand the objects and relationships in the scene. By fusing the depth image and the semantic category index image, the model can simultaneously obtain the geometric depth information and object category information of the scene, helping the model to more accurately determine the position and distance of the object, and improving the model's adaptability and robustness to complex scenes and prediction accuracy.

[0079] The present invention optimizes the model unit RLSTM and proposes a Multi-RLSTM local spatiotemporal feature extraction model. RLSTM retains the hidden state of the previous time step through residual connection, which can help cope with nonlinear changes in the system and improve the ability to track complex motion trajectories. RLSTM can retain and transmit important state information of the previous time step through residual connection, which helps the model better cope with local motion changes, making the model more sensitive to local feature changes and making precise adjustments.

[0080] In the Multi-RLSTM local spatiotemporal feature extraction model, a stacked LSTM structure is used, and each layer of RLSTM can learn different levels of temporal features, so that the model can gradually extract rich high-dimensional features from shallow to deep layers. Dynamic residual connections are introduced between RLSTM layers to splice the output of the previous LSTM layer with the output of the subsequent LSTM layer in the feature dimension. This dynamic residual connection allows information to propagate quickly in the network, which helps to improve the model's response speed to changes in the joystick trajectory, especially when facing complex or rapid dynamic changes, it can capture the change trend more quickly.

[0081] The RLSTM-Transformer crane system end-to-end motion trajectory prediction model proposed in the present invention is a unique dual-branch network structure. The local spatiotemporal feature extraction subnetwork uses CNN, Multi-RLSTM and multi-head attention modules to extract local temporal features of the input sub-image sequence; the global spatiotemporal feature extraction subnetwork uses ResNet and Transformer to extract deep spatial features of long image sequences and establish global dependencies; through the combination of local spatiotemporal feature extraction and global spatiotemporal feature extraction, the model can take into account both short-term action details and long-term trajectory continuity in crane motion prediction, and improve the understanding and modeling capabilities of complex spatiotemporal dynamics. It not only improves the prediction accuracy of the model, but also enhances its stability and robustness in actual operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 This is a schematic diagram of the RLSTM model unit structure;

[0083] Figure 2 This is a schematic diagram of the Multi-RLSTM model structure;

[0084] Figure 3 This is a schematic diagram of the structure of the semantic RLSTM-Transformer end-to-end crane motion trajectory prediction model;

[0085] Figure 4 It is a prediction flow chart. DETAILED DESCRIPTION

[0086] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or apparatus.

[0087] The present invention is further described in detail below with reference to the accompanying drawings and embodiments:

[0088] A crane end-to-end motion trajectory prediction method based on RGBD images includes two stages: training and prediction. The steps of the training stage are steps 1 to 7, and the step of the prediction stage is step 8:

[0089] Step 1: On the gantry crane real machine experimental platform built in the laboratory, the Azure Kinect camera is used to obtain the RGB image, depth image and motion control instructions of the operator from the perspective of the cab during crane operation, including: C1-rise, C2-fall, C3-increase, C4-decrease, C5-left rotation, C6-right rotation, C7-forward, C8-backward, C9-grasp, C10-release;

[0090] Specifically, the RGB image and depth image collected by the Azure Kinect sensor from the perspective of the portal crane cab and the crane motion control instructions from the operator are obtained at a collection frequency of 10FPS, and the timestamp is retained.

[0091] On the real crane experimental platform, Azure Kinect was installed on the boom of the crane to capture images of the grab bucket operation at 10FPS. The Azure Kinect sensor was used to collect data from the cab's perspective during crane operation through ROS and recorded as a rosbag file, and then the corresponding depth map and color map were extracted from the rosbag through a script. The acquisition includes 50,000 frames of 50 sets of RGB images, depth maps, and motion control instructions on the operator side. Each set of data contains a complete crane loading process. The image resolution is 640*480, which is later cropped to 160*210. Data is collected under different lighting conditions, the relative position of the sand pile and the bucket is adjusted, different levels of wind are simulated, and the mass of the lifted heavy objects is changed to improve data diversity.

[0092] The input port of the crane controller is connected to a switch, which is used to detect the operator's command. According to different input switch signals, the Siemens PLC program controls the relay connected to the output port to control the motor, thereby achieving precise control of different movements of the crane. This application captures and saves the state of the input switch at 10FPS and records the timestamp at the same time.

[0093] During acquisition, each RGB image corresponds to a depth image. The acquired image and the motion control instructions of the operator are aligned using the timestamp, and the motion control instructions of the operator are used as the label of the image.

[0094] Step 2: Manually annotate the grab, material, and bucket in the RGB image, train the PaddleSeg semantic segmentation model, input all RGB images into the trained semantic segmentation model, and use the model to infer the semantic segmentation pseudo-color image and semantic segmentation category index map containing the grab, material, bucket, and background. Superimpose the pseudo-color image output by the segmentation with the original image at a certain transparency to obtain an RGB image with fused semantic information. The superimposed image not only retains the rich color and texture information in the original image, but also highlights the target information that is focused on in the crane decision-making task. The single-channel depth map and the single-channel semantic segmentation category index map are fused, and the third channel is set to blank to form a three-channel semantic depth map. Each pixel value in the semantic segmentation category index map represents the category to which the pixel belongs, and the fusion obtains a depth map with semantic information.

[0095] The output of the semantic segmentation model is represented as:

[0096] I pseudo , I index =Segmentation(I RGB )

[0097] Among them, I RGB is the input RGB image, I pseudo is the pseudo-color image output by the semantic segmentation model, I index It is a single-channel category index map output by the semantic segmentation model.

[0098] The fusion process of pseudo color image and RGB image is:

[0099] I semantic_RGB =α·I pseudo +(1-α)·I RGB

[0100] Among them, α is the transparency coefficient, I semantic_RGB It is the fused semantic RGB image.

[0101] Three-channel semantic depth map I semantic_depth The way to generate is to use the category index map as the first channel, the depth map as the second channel, and the blank channel as the third channel, expressed as:

[0102] I semantic_depth =Concatenate(I index , I depth , I empty )

[0103] I depth is the input single-channel depth map, I empty For blank channels, I empty Size and I depth and Iindex Concatenate(·) means concatenating images in the channel dimension.

[0104] Step 3, by optimizing the model unit RLSTM, a Multi-RLSTM local spatiotemporal feature extraction model is established;

[0105] Specifically, the calculation of the hidden state of each time step introduces the residual connection of the hidden state of the previous time step, and introduces the learnable weight W h and bias b h To adjust the contribution of the residual connection, such a weighted residual connection can provide more flexibility, allowing the model to learn how to better combine information from the previous time step. The RLSTM forward propagation process is:

[0106] f t =σ(W f ·[h t-1 , x t ]+b f )

[0107] i t =σ(W i ·[h t-1 , x t ]+b i )

[0108]

[0109] o t =σ(W O [h t-1 , x t ]+b O )

[0110] h′ t =o t *tanh(C t )

[0111] h t =tanh(h′ t +W h ·h′ t-1 +b h )

[0112] σ is the Sigmoid activation function, S f , W i , W o are the weight parameters of the forget gate, input gate, and output gate respectively; b f , b i , b o are the bias parameters of the forget gate, input gate, and output gate, respectively, and f t,i t , o t They are respectively the forget gate, input gate, and output gate, x t Input for the current sequence, h t-1 is the state of the hidden layer at the previous moment, C t is the cell state under the input sequence at the current moment, is the cell state after the output of the forget gate and input gate is updated, tanh is the activation function, h′ t is the state of the hidden layer at time t before the model is improved. Its output value is changed by the formula to obtain h t .

[0113] In the forward Multi-RLSTM, the input is from time 1 to time t in chronological order. The mathematical expression of the Multi-RLSTM local spatiotemporal feature extraction model is as follows:

[0114] R(h i ,h j )=linear(concat(h i ,h j ))

[0115] h1=tanh(LSTM1(x))

[0116] h2 = tanh(LSTM2(h1))

[0117] h3 = tanh(LSTM3(h2))

[0118] h4=tanh(LSTM4(R(h3, h1)))=tanh(LSTM4(linear(concat(h3, h1))))

[0119] h5=tanh(LSTM5(h4))

[0120] h6=tanh(LSTM6(R(h5, h2)))=tanh(LSTM6(linear(concat(h5, h2))))

[0121]

[0122] where h i and h j are the outputs of the hidden states of the i-th layer and the j-th layer, respectively, and their shapes are (batch_size, seq_len, hidden_dim), R represents the residual connection function, concat(h i ,h j ) means concatenating h in the last dimension i and hj , linear is a linear transformation.

[0123] The Multi-RLSTM network layer adopts a stacked LSTM structure, stacks multiple LSTM layers, and introduces dynamic residual connections between LSTM layers, concatenates the output of the LSTM of the previous layer with the output tensor of the LSTM layer of the subsequent layer in the feature dimension, and the concatenated feature tensor is used as the input of the next layer of LSTM.

[0124] Step 4, using the local spatiotemporal feature extraction subnetwork to extract local spatiotemporal features of the semantic depth image sequence;

[0125] Specifically, the local spatiotemporal feature extraction subnetwork adopts a staged feature extraction strategy. First, the long image sequence X∈R N×C×T×H×S , where N is the batch size, C is the number of channels, T is the sequence length, H is the image height, and W is the image width; the input sequence is divided into image subsequences X with each sample length T (n) ∈R N×C×H×W , representing the image sequence of the nth sample; the image subsequence is input into the 3D convolution layer to extract the spatiotemporal features and generate a temporal feature representation. The time series data after processing can still maintain its own temporal nature; by optimizing the RLSTM model unit, a Multi-RLSTM model is established to further extract the temporal features of these subsequences and gradually capture higher-level temporal dependencies; a multi-head attention mechanism is introduced after the Multi-RLSTM layer to process the entire sequence and capture global dependencies.

[0126] First, a batch of long image sequences is input into a predefined five-dimensional tensor [N, T, C, H, W]. The batch size N is set to 8 and the image sequence length is set to 15. The long sequence is divided into subsequences with a sequence length of 15 each. The semantic depth image subsequence is input into the 3D convolutional network to extract the local spatiotemporal features of the image sequence; the input image sequence is processed by multiple 3D convolutional layers to extract spatiotemporal features and generate a temporal feature representation. The processed temporal data can still maintain its own temporal nature. The convolutional layer focuses on extracting the local spatiotemporal information of the input image sequence, reflecting the short-term changes of the system, and is suitable for processing short-term oscillations or emergencies of the crane system; the 3D convolutional layer uses multiple convolutional blocks (3×3 convolution + ELU activation function + batch normalization) to extract the features of the input image sequence, and introduces a residual convolutional layer in the convolutional layer to alleviate the gradient vanishing problem in the deep network, while improving training efficiency and model performance.

[0127] The 3D convolution kernel slides in the depth, height, and width directions of the input data, and calculates the weighted sum of the local area. The output Y(d, h, w) at each position (d, h, w) can be expressed as:

[0128]

[0129] Among them, Y(d, h, w) is the value of the output data at position (d, h, w); K(c, i, j, k) is the weight of the convolution kernel at channel c and position (i, j, k); X(c, d·S d +i,h·S h +j,w·S w +k) is the input data at channel c and position (d·S d +i,h·S h +j,w·S w +k); C is the number of channels of input data; K d , K h , K w They are the depth, height, and width of the convolution kernel respectively; S d , S h , S w are the strides in depth, height, and width, respectively.

[0130] After each convolution calculation, the size of the feature map obtained by the previous calculation will be reduced. The shape of the feature map output after applying the 3D convolution layer can be expressed as follows:

[0131] Output Depth:

[0132] Output height:

[0133] Output Width:

[0134] Among them, D in , H in , W in is the depth, width, and height of the input; K d , K h , K W is the depth, height and width of the convolution kernel; S d , S h , S W is the stride in depth, height and width; P is the padding size, which is 0.

[0135] The ELU activation function is introduced after each convolution operation to introduce nonlinear transformation to the network, so that the network can fit and represent complex nonlinear relationships and improve the performance of the model.

[0136] A 3D batch normalization layer is introduced after the ELU activation function to speed up the training process and reduce the sensitivity to initialization, which can be expressed as follows:

[0137]

[0138] Residual connections are introduced between different 3D convolutional layers to alleviate the gradient vanishing and gradient exploding problems and accelerate the training of the model. The residual can be expressed as follows:

[0139] Y=F(X,{S})+X

[0140] Where F represents the mapping function from input X to output Y, and {W} is the weight of the layer. Through back propagation, the gradient can be directly passed to the shallow layer:

[0141]

[0142] Secondly, by optimizing the model unit RLSTM, a Multi-RLSTM local spatiotemporal feature extraction model is established. The features extracted by 3D convolution on the subsequence are input into the Multi-RLSTM crane local spatiotemporal feature extraction model to further extract the high-dimensional temporal features of the subsequence, specifically:

[0143] R(h i ,h j )=linear(concat(h i ,h j ))

[0144] h1=tanh(LSTM1(x))

[0145] h2 = tanh(LSTM2(h1))

[0146] h3 = tanh(LSTM3(h2))

[0147] h4=tanh(LSTM4(R(h3, h1)))=tanh(LSTM4(linear(concat(h3, h1))))

[0148] h5=tanh(LSTM5(h4))

[0149] h6=tanh(LSTM6(R(h5, h2)))=tanh(LSTM6(linear(concat(h5, h2))))

[0150]

[0151] Finally, a multi-head attention mechanism is introduced after the Multi-RLSTM crane local spatiotemporal feature extraction model to combine the local features of the subsequences and input the entire sequence into the multi-head attention mechanism to capture the global dependencies of the entire sequence. Specifically:

[0152] Output features of the spliced ​​subsequences:

[0153] F i =f(chunk i )

[0154] F concat =[F1; F2; ...; F n ]

[0155] chunk i represents the ith subsequence, f(chunk i ) represents the process of extracting features from the i-th subsequence through 3D convolution and Multi-RLSTM layers, F i represents the feature representation of the i-th subsequence, with dimensions (N, T, D), F concat It is the feature representation after concatenation in the time dimension, and its dimension is (N, nT, D).

[0156] Input the concatenated features of the subsequences into the multi-head attention module:

[0157] Q i =F concat W i Q

[0158] K i =F concat W i K

[0159] V i =F concat W i V

[0160] where F concat is the representation of the input sequence, W i Q , W i K , W i V are the weight matrices of Q, K, and V of the i-th attention head, respectively. For each attention head, the similarity between Q and K is calculated by using the dot product operation and normalized by the Softmax function:

[0161]

[0162] MultiHead(Q,K,V)=Concat(head1,head2,head3)W O

[0163] The outputs of all heads are concatenated and linearly transformed to generate the output of the multi-head self-attention module, with dimensions (N, T, D). The features of the subsequences are aggregated and mapped to the final output using a fully connected layer. Specifically, the output of each subsequence contains not only the feature representation of the subsequence itself, but also the information of the sequence as a whole, which can generate a more comprehensive and detailed feature representation. The features of each subsequence are aggregated using the maximum pooling method, and then a fully connected layer is introduced after the pooling layer to generate the final prediction result. Specifically:

[0164]

[0165] Among them, n represents the batch index, c represents the subsequence index, d represents the feature dimension index, and t represents the time step index. The above formula represents the average pooling of the time steps of each subsequence.

[0166] The pooled input is linearly transformed to the fully connected layer to gradually reduce the feature number of the input tensor to the shape of the output tensor, and finally mapped to the motion control instructions of the crane.

[0167] The local spatiotemporal prediction subnetwork focuses on modeling the spatiotemporal relationship of the input sub-image sequence, understanding the movement trend of the crane from a local and short-term perspective, and predicting the movement trend of the crane.

[0168] Step 5, use the global spatiotemporal feature extraction subnetwork to extract features of the semantic RGB image sequence and predict motion control commands;

[0169] Specifically, the global spatiotemporal feature extraction subnetwork predicts the input long time series image, and the input tensor is frames∈R N×C×T×H×W , where N = 8 is the batch size, C = 3 is the number of channels, T = 15 is the number of time steps, H = 160 and W = 210 are the height and width of the image, that is, frames = {f n，c，t，h，w}. Reshape the input tensor and merge the N and T dimensions into N×T, that is, frames′={f′ n′，c，h，w}, n′=N×T, frames∈R 120×3×160×210. The global spatiotemporal feature extraction subnetwork combines the ResNetl8 and Transformer architectures. First, the entire reshaped image sequence is input into the modified ResNet18 model for preliminary feature extraction. ResNet18, as the position encoder of the image, extracts the spatial features of each image through convolution operations and converts them into a fixed-size feature vector. At the same time, a linear layer is used to map the extracted ResNet features to 512 dimensions as the input of the subsequent time series model. In the network architecture, ResNet18 is selected as the encoder of the image sequence because ResNetl 8 has multiple layers of convolution and residual connections, which can extract complex spatial features in crane operation, such as the displacement of objects, the position change of the rocker, etc., thereby extracting rich spatial information at each time step to help the model better understand the operating status of the crane. Then the network uses the position encoding module to introduce the position information of the sequence. This step enables the Transformer to perceive the sequential relationship in the frame sequence by introducing the position information in the feature sequence, helping to capture the global temporal dependency. The position encoding is generated by the predefined function PE(t), specifically: Next, the network inputs the position-encoded image sequence into the Transformer-based temporal encoder, specifically: It is the feature vector of the input image after Resnet18 extracts features and maps the features to 512-dimensional features. Transformer can consider all past states at the same time at each time step, and identify key time points and change trends through a multi-head attention mechanism. This enables the model to quickly capture short-term emergencies (such as sudden jitters) while keeping track of long-term trends, thereby improving the accuracy of predictions. The combination of ResNet18 and Transformer architectures means that the model is no longer limited to a single frame or a local time period, but can make decisions based on the comprehensive information of the entire operation sequence, and can better capture the changes in motion patterns brought about by complex dynamics.

[0170] Finally, the feature vector f = {f t，n，k}∈R T×N×d , where T = 15 is the sequence length, N = 8 is the batch size, and d is the feature dimension. The maximum pooling operation is performed in the time dimension T, specifically: It is the feature after maximum pooling. The pooled feature is reduced in dimension and then mapped to the final crane motion control instruction through the fully connected layer. The global spatiotemporal feature prediction subnetwork extracts features from the entire batch of images and uses the temporal relationship of the position encoding and transformer modeling sequence to understand the long-term motion of the crane and predict the motion trend of the crane from a global perspective.

[0171] Step 6: Finally, the outputs of the two branches are fused through a weighted fusion network to obtain the final output of the model. Specifically, X local ∈R N×d represents the crane motion state predicted by the local spatiotemporal prediction subnetwork, X global ∈R N×d represents the crane motion state predicted by the global spatiotemporal prediction subnetwork. The fusion of these two subnetworks adopts a weighted approach. fused =α·X local +β·X global , where α = 0.4, β = 0.6 are weighting coefficients, indicating the different emphasis on local and long-term prediction results. The final fused prediction result y fused ∈R N×d , d is the total number of categories of motion control instructions, which represents the prediction of crane motion control instructions after combining local and long-term information decisions.

[0172] Step 7: The collected motion control instructions of the operator are used as labels of the image. By minimizing the deviation between the predicted value and the true value, the loss function is optimized and the above neural network model is trained to obtain a prediction model. Specifically:

[0173] The depth image sequence, RGB image sequence, and the collected motion control instructions of the operator are input into the semantic RLSTM-Transformer crane end-to-end motion trajectory prediction model. The motion control instructions are used as the labels of the images, and the network calculates the predicted values ​​based on the weights and biases. The predicted results obtained by forward propagation are compared with the actual label values, and the cross entropy loss is calculated to measure the gap between the predicted value and the true value. The weights and biases of the network are updated according to the gradient descent method based on the gradient calculated by back propagation to reduce the value of the loss function and train the semantic RLSTM-Transformer crane end-to-end motion trajectory prediction model.

[0174] Step 8, when executed online, obtains the depth image and RGB image collected in real time by the RGBD camera in the crane operation scene, inputs the image into the trained semantic RLSTM-Transformer crane end-to-end motion trajectory prediction model, predicts the crane motion control instructions, and realizes end-to-end intelligent motion control of the crane.

Claims

1. A crane end-to-end motion trajectory prediction method based on RGBD images, characterized by: It includes two stages: training and prediction, which specifically include the following steps: Step 1: Collect experimental data. Use the RGBD camera to obtain the RGB image and depth image from the perspective of the cab when the crane is operating, and at the same time obtain the motion control instructions of the operator. The motion control instructions of the operator include: C1-rise, C2-fall, C3-increase, C4-decrease, C5-left rotation, C6-right rotation, C7-forward, C8-backward, C9-grasp, C10-release; Step 2: Segment the RGB image based on the semantic segmentation model to obtain a pseudo-color image and a category index image. Overlay the pseudo-color image and the RGB image with a certain degree of transparency to obtain a semantic RGB image. At the same time, fuse the category index image and the depth image. Fuse the single-channel category index image and the single-channel depth image, and set the third channel to blank to obtain a three-channel semantic depth image. Step 3: Establish a Multi-RLSTM local spatiotemporal feature extraction model by optimizing the model unit RLSTM; Step 4: Input the semantic depth image into the local spatiotemporal feature extraction subnetwork. The local spatiotemporal feature extraction subnetwork uses CNN, Multi-RLSTM and multi-head attention modules to extract the local temporal features of the input sub-image sequence and predict the output of each sub-sequence. Step 5: Input the semantic RGB image into the global spatiotemporal feature extraction subnetwork, which uses ResNet and Transformer to extract the deep spatial features of the image sequence and establish global dependencies, and predict the output of each subsequence; Step 6: Finally, the outputs of the two branches are fused through the weighted fusion network to obtain the final output of the model; X local ∈R N×d represents the crane motion state predicted by the local spatiotemporal prediction subnetwork, X global ∈R N×d represents the crane motion state predicted by the global spatiotemporal prediction subnetwork. The fusion of these two subnetworks adopts a weighted approach. fused =α·X local +β·X global , where α and β are weighted coefficients, indicating the different emphasis on local and long-term prediction results. The final fused prediction result y fused ∈R N×d , d is the total number of categories of crane motion control instructions, which represents the prediction of crane motion control instructions after combining local and long-term information decisions; Step 7: The collected motion control instructions of the operator are used as labels of the image, and the loss function is optimized by minimizing the deviation between the predicted value and the true value, and the above neural network model is trained to obtain a prediction model; Step 8: In the online execution phase, obtain the RGB image and depth image captured by the RGBD camera from the perspective of the cab in the crane operation scenario, input the image into the trained semantic RLSTM-Transformer crane end-to-end motion trajectory prediction model, and obtain the required crane motion control instructions through model prediction.

2. The method for predicting the end-to-end motion trajectory of a crane based on RGBD images according to claim 1 is characterized in that: Step 1: Each set of data contains a complete crane loading process. The data is collected under different lighting conditions, by adjusting the relative position of the sand pile and the truck bed, simulating different levels of wind, and changing the weight of the lifted object. The collected image resolution is 640*480, which is later cropped to 160*210 size.

3. The method for predicting the end-to-end motion trajectory of a crane based on RGBD images according to claim 1 is characterized in that Step 2: The method for making a data set is: Based on the semantic segmentation model, the grab, material, bucket and background in the RGB image are segmented. The model inference obtains the semantic segmentation pseudo-color map and semantic segmentation category index map containing the grab, material, bucket and background. The pseudo-color map containing the hook, material, bucket and background output by the segmentation is superimposed with the original image at a certain transparency to obtain an RGB image containing semantic information; the depth map and the semantic segmentation category index map are fused to obtain a depth image with semantic information; the fusion process can be expressed by the following formula, and the output of the semantic segmentation model is expressed as: I pseudo ,I index =Segmentation(I RGB ) Among them, I RGB is the input RGB image, I pseudo is the pseudo-color image output by the semantic segmentation model, I index It is a single-channel category index map output by the semantic segmentation model; The fusion process of pseudo color image and RGB image is: I semantic_RGB =α·I pseudo +(1-α)·I RGB Among them, α is the transparency coefficient, I semantic_RGB It is the fused semantic RGB image.

4. The method for predicting the end-to-end motion trajectory of a crane based on RGBD images according to claim 3 is characterized in that: Three-channel semantic depth map I semantic_depth The way to generate is to use the category index map as the first channel, the depth map as the second channel, and the blank channel as the third channel, expressed as: I semantic_depth =Concatenate(I index ,I depth ,I empty ) I depth is the input single-channel depth map, I empty For blank channels, I empty Size and I depth and I index Similarly, Concatenate(·) means concatenating images in the channel dimension; Each semantic RGB image corresponds to a semantic depth image. The image and the motion control instructions of the operator are aligned using timestamps. The motion control instructions of the operator are used as labels of the images. The data set is preprocessed to obtain the preprocessed data set.

5. The method for predicting the end-to-end motion trajectory of a crane based on RGBD images according to claim 1 is characterized in that: The forward propagation calculation formula of the optimized model unit RLSTM in step 3 is: f t =σ(W f ·[h t-1 ,x t ]+b f ) i t =σ(W i ·[h t-1 ,x t ]+b i ) the t =σ(W O [h t-1 ,x t ]+b O ) h' t =o t *tanh(C t ) h t =tanh(h′ t +W h ·h′ t-1 +b h ) σ is the Sigmoid activation function, W f , W i , W o are the weight parameters of the forget gate, input gate, and output gate respectively; b f , b i , b o are the bias parameters of the forget gate, input gate, and output gate, respectively, and f t ,i t , o t They are respectively the forget gate, input gate, and output gate, x t Input for the current sequence, h t-1 is the state of the hidden layer at the previous moment, C t is the cell state under the input sequence at the current moment, is the cell state after the output of the forget gate and input gate is updated, tanh is the activation function, h′ t is the output of the hidden layer of the original LSTM at time t, h t is the state output of the hidden layer at time t after the change; The RLSTM model unit introduces the residual connection of the hidden state of the previous time step into the calculation of the hidden state of each time step in the LSTM, and introduces the learnable weight W h and bias b h To adjust the contribution of the residual connection.

6. The method for predicting the end-to-end motion trajectory of a crane based on RGBD images according to claim 3 is characterized in that: In step 3, the input is made in the forward Multi-RLSTM in chronological order from time 1 to time t. The Multi-RLSTM local spatiotemporal feature extraction model is: R(h i ,h j )=linear(concat(h i ,h j )) h1=tanh(LSTM1(x)) h2 = tanh(LSTM2(h1)) h3 = tanh(LSTM3(h2)) h4=tanh(LSTM4(R(h3,h1)))=tanh(LSTM4(linear(concat(h3,h1)))) h5=tanh(LSTM5(h4)) h6=tanh(LSTM6(R(h5,h2)))=tanh(LSTM6(linear(concat(h5,h2)))) where h i and h j are the outputs of the hidden states of the i-th layer and the j-th layer, respectively, and their shapes are (batch_size, seq_len, hidden_dim), R represents the dynamic residual connection function, concat(h i ,h j ) means concatenating h in the last dimension i and h j , linear is a linear transformation.

7. The method for predicting the end-to-end motion trajectory of a crane based on RGBD images according to claim 1, characterized in that: The local spatiotemporal feature extraction subnetwork in step 4 is specifically: The local spatiotemporal feature extraction subnetwork adopts a staged feature extraction strategy. First, the long image sequence X∈R N×C×T×H×W , where N is the batch size, C is the number of channels, T is the sequence length, H is the image height, and W is the image width; the input sequence is divided into image subsequences X with each sample length T (n) ∈R N×C×H×W , represents the image sequence of the nth sample; the image subsequence is input into the 3D convolution layer to extract the spatiotemporal features and generate the time series feature representation. The time series data after processing can still maintain its own time series; by optimizing the RLSTM model unit and establishing the Multi-RLSTM model, the time series features of these subsequences are further extracted, and the higher-level time dependencies are gradually captured; A multi-head attention mechanism is introduced after the Multi-RLSTM layer to process the entire sequence and capture global dependencies. Specifically: F i =f(chunk i ) F concat =[F1;F2;…;F n ] chunk i represents the ith subsequence, f(chunk i ) represents the process of extracting features from the i-th subsequence through 3D convolution and Multi-RLSTM layers, F i represents the feature representation of the i-th subsequence, with dimensions (N, T, D), F concat It is the feature representation after splicing in the time dimension, with dimensions (N,nT,D) where F concat is the representation of the input sequence, are the weight matrices of Q, K, and V of the i-th attention head, respectively. For each attention head, the similarity between Q and K is calculated by using the dot product operation and normalized by the Softmax function: MultiHead(Q,K,V)=Concat(head1,head2,head3)W O Concatenate the outputs of all heads and linearly transform them to generate the output of the multi-head self-attention module with a dimension of (N,nT,D); Finally, the tensor output by the multi-head attention module is subjected to a maximum pooling operation in the time dimension, and then the extracted features are processed through a fully connected layer and finally mapped to the crane motion control instructions.

8. The method for predicting the end-to-end motion trajectory of a crane based on RGBD images according to claim 1 is characterized in that: The global spatiotemporal feature extraction subnetwork in step five combines the ResNet and Transformer architectures. The global spatiotemporal feature prediction subnetwork extracts features from the entire batch of images and uses position encoding and transformer modeling to build the temporal relationship of the sequence, thereby understanding the long-term motion of the crane and predicting the motion trend of the crane from a global perspective.

9. The method for predicting the end-to-end motion trajectory of a crane based on RGBD images according to claim 8 is characterized in that The specific steps of step five are: The global spatiotemporal feature extraction subnetwork predicts the input long time series image, and the input tensor is frames∈R N ×C×T×H×W , where N is the batch size, C is the number of channels, T is the number of time steps, H and W are the height and width of the image, that is, frames = {f n,c,t,h,w }; Reshape the input tensor and merge the N and T dimensions into N×T, that is, frames′={f′ n′,c,h,w },n′=N×T,frames∈R NR×X×H×W ; First, the entire reshaped image sequence is input into the modified ResNet18 model for preliminary feature extraction; ResNet18 is used as the position encoder of the image. It extracts the spatial features of each image through convolution operations and converts them into feature vectors of fixed size. At the same time, a linear layer is used to map the extracted ResNet features to a low dimension as the input of the subsequent time series model. Then, the global spatiotemporal feature extraction subnetwork uses the position encoding module to introduce the position information of the sequence. By introducing the position information in the feature sequence, the Transformer can perceive the sequential relationship in the frame sequence and help capture the global temporal dependency. The position encoding is generated by the predefined function PE(t), specifically: The global spatiotemporal feature extraction subnetwork inputs the position-encoded image sequence into the Transformer-based temporal encoder, specifically: It is the feature vector after the input image is extracted by Resnet18 and mapped to low-dimensional features; Finally, the feature vector f = {f t,n,k }∈R T×N×d , where T is the sequence length, N is the batch size, d is the feature dimension, and the maximum pooling operation is performed in the time dimension T, specifically: It is the feature after maximum pooling; The pooled features are subjected to dimensionality reduction processing, and then the reduced features are mapped to the final crane motion control instructions through a fully connected layer.

10. The method for predicting the end-to-end motion trajectory of a crane based on RGBD images according to claim 1, characterized in that: The specific method of step seven is: input the depth image sequence, RGB image sequence, and the collected motion control instructions of the operating end into the semantic RLSTM-Transformer crane end-to-end motion trajectory prediction model, use the motion control instructions as the image label, and the network calculates the prediction value based on the weight and bias; compare the prediction result obtained by forward propagation with the actual label value, and calculate the cross entropy loss to measure the gap between the predicted value and the true value; update the weight and bias of the network according to the gradient descent method based on the gradient calculated by back propagation, reduce the value of the loss function, and train the semantic RLSTM-Transformer crane end-to-end motion trajectory prediction model.

Images