Sensorless collision detection method for collaborative robot based on gat-lstm spatio-temporal feature fusion

Abstract

The application provides a kind of based on GAT-LSTM spatio-temporal feature fusion's collaborative robot sensorless collision detection method, and the technical field belongs to robot collision detection technical field.Its characterized in that the method models the current, position and speed time series data of robot joint motor as a graph structure, extracts the spatial topological features between joints through graph attention network (GAT), and learns the long-term time dependence of current signals through long short-term memory network (LSTM), realizing the deep fusion of spatio-temporal features.Based on the residual error between predicted current and actual current, combined with dynamic threshold strategy, collision is judged.The application deeply mines the robot body data, realizes the ability that whether the robot itself collides can be perceived without external sensors, greatly reduces the cost and system complexity of safety deployment, and brings a new technical way for the safe and reliable integration of collaborative robots into existing intelligent devices in more scenarios.

Description

Technical Field

[0001] This invention relates to the field of robot collision detection technology, specifically to a sensorless collision detection method for collaborative robots based on GAT-LSTM spatiotemporal feature fusion. Background Technology

[0002] As human-machine collaboration becomes increasingly prevalent in industries and medical fields, ensuring human safety in shared workspaces is paramount. The first step in safe collaboration is collision detection, which currently involves two main research areas: external sensing and body sensing.

[0003] Methods based on external sensors, such as vision, torque, and skin sensors, can directly sense external forces, but they generally suffer from drawbacks such as high manufacturing and maintenance costs, susceptibility to external environmental interference, and complex system structures with low integration, severely limiting the deployment scale and applicability of existing technologies. While combining body sensors with dynamic models to compare the difference between theoretically calculated and actually measured torques for collision detection, this method heavily relies on the accuracy of the dynamic model. When the model's uncertainty is high, it can easily lead to significant estimation errors, resulting in missed detections or false alarms.

[0004] In recent years, research has shifted towards data-driven solutions, with machine learning methods, such as deep neural networks, being widely adopted to reduce or even eliminate reliance on complex and difficult-to-obtain accurate prior models. While Long Short-Term Memory (LSTM) networks are strong at capturing temporal dependencies, they lack effective models for the complex spatial correlations arising from mechanical coupling between robot joints. Although graph neural network-based modeling methods are relatively simple and direct in representing structural relationships, their ability to extract and model long-range temporal features is weak, making it difficult to effectively capture deep correlations across time steps. This means existing methods cannot fully and effectively utilize the spatiotemporal characteristics of robot joint state data, thus failing to meet the combined requirements of high precision, robustness, and real-time performance for collision detection tasks, thereby limiting their practical application effectiveness. Summary of the Invention

[0005] To address the aforementioned problems, the present invention aims to propose a sensorless collision detection method for collaborative robots based on GAT-LSTM spatiotemporal feature fusion, which collaboratively extracts spatiotemporal features of joint signals. This model relies solely on body sensor data such as current, position, and velocity of joint motors to achieve high-precision collision detection. Specifically, it includes the following steps:

[0006] Step S10: Collect time-series data of each joint from the body sensors of the collaborative robot, including joint motor currents. Joint position and joint velocity ,in Indicates the sampling time;

[0007] Step S20: Based on the time series data, construct the spatiotemporal diagram structure of the robot joints. ;

[0008] Step S30: Input the graph structure data into the pre-trained GAT-LSTM hybrid prediction model. The predicted current values ​​of each joint will be output at the next moment. ;

[0009] Step S40: Calculate the predicted current value Compared with the actual current value residuals between ;

[0010] Step S50, based on the residual With preset dynamic threshold The comparison is used to determine whether a collision has occurred.

[0011] Furthermore, in step S20, the construction of the robot joint spatiotemporal graph structure... The process is as follows:

[0012] Step S201: Calculate the motor current of each joint at each moment. Joint position and joint velocity Each node serves as a separate node, forming a multi-level node set. ;

[0013] Step S202: Based on the robot joint connection sequence, construct a joint node sequence diagram. ;

[0014] Step S203: Construct a correlation graph based on the Pearson correlation coefficients between the sequences of each node. The formula for calculating the correlation coefficient is:

[0015]

[0016] in, Indicates the number of samples, in Time for a certain node In terms of its sample value Set a threshold ,when , If so, a connection is established between these two nodes, forming a directed edge;

[0017] Step S204: Obtain the joint node sequence diagram Correlation plot The union of the final graph structure ,in Let it be the set of edges.

[0018] Furthermore, the construction process of the GAT-LSTM hybrid prediction model described in step S30 is as follows:

[0019] Step S301: Establish the network structure of the prediction model. , For neural network information of the prediction model;

[0020] Step S302, construct the GAT neural network model, as follows:

[0021] Graph Structure A set of nodes with spatial characteristics in the data As input, where For the number of nodes, The number of features for each node, ;

[0022] At the nodes of the graph, a weight vector is defined. A self-attention mechanism implemented using a parameterized single-layer feedforward neural network Its function is to calculate and represent the characteristics of neighboring nodes based on the importance weight of the central node; this weight calculation process shares parameters among all edges:

[0023]

[0024] in, It is the first Each convolutional layer is used to achieve the effect from the original space. To the new higher dimension Feature dimension mapping, weight matrix Perform linear projection on the input features;

[0025] For each central node, the attention weights of itself and all its neighboring nodes are aggregated and then normalized using the softmax function to obtain an attention coefficient that represents the relative importance of each neighboring node.

[0026]

[0027] After obtaining the normalized attention coefficients, the transformed features of each neighboring node are processed by weighted summation and a nonlinear activation function to obtain the updated output feature vector of that node, which is calculated as follows:

[0028]

[0029] In the feature propagation process, a multi-head attention mechanism is used for calculation and aggregation to obtain variable-related features, and the specific calculation formula is as follows:

[0030] wherein, and are the features of the node and its adjacent nodes , is the normalized attention coefficient calculated by the first attention head, is a trainable weight matrix, is a nonlinear activation function, denotes vector splicing, and the total number of attention heads is

[0031] The output is a node embedding vector sequence enhanced by spatial features.

[0032] Step S303, establish an LSTM neural network model, and the steps are as follows:

[0033] First, after receiving the node embedding vector sequence of each node at the current time from the GAT layer, the input gate is sequentially transmitted, and the value is calculated by adding the hidden state of the previous time, the input at the current time, the first weight matrix and the first bias term. After transformation and mapping to the [0, 1] interval by the activation function, the degree of partial preservation of the input information at this time is obtained, and the calculation process is as follows:

[0034]

[0035] wherein, is a sigmoid activation function, is the hidden state of the previous time, is the input at the current time, , is the weight matrix of the input gate, is the corresponding bias term;

[0036] Then, according to the hidden state of the previous time, the input data at the current time, the second weight matrix and the second bias term are added to obtain an intermediate value, and then the control signal of the forgetting gate is obtained by compressing it between [0, 1] using the sigmoid function. Multiply the cell state at the previous time to obtain the cell state at the previous time that is preserved by the current time, and the calculation process is as follows:

[0037]

[0038] wherein, forget gate control signal, , forget gate weight matrix, forget gate bias term;

[0039] According to the hidden state of the previous moment and the input of the current moment, the candidate memory cell information of the current moment is generated through the third weight matrix and the third bias term via the activation function, and the calculation formula is as follows:

[0040]

[0041] wherein, candidate memory cell, , candidate memory cell weight matrix, candidate memory cell bias term;

[0042] Secondly, based on the forget gate control signal, the input gate control signal and the candidate memory cell, the unit state of the previous moment is updated to obtain the unit state of the current moment; the update formula is as follows:

[0043]

[0044] wherein, unit state of the current moment, unit state of the previous moment, symbol element-wise multiplication;

[0045] By fusing the hidden state of the previous moment and the input of the current moment, a gate signal with a value range of [0, 1] is generated through the sigmoid activation function after linear transformation by the fourth weight matrix and the fourth bias term, so as to adjust the information proportion of the current unit state transmitted to the hidden state, and the calculation formula is as follows:

[0046]

[0047] wherein, output gate control signal, , output gate weight matrix, output gate bias term;

[0048] Using the forget gate, input gate and output gate mechanism, each LSTM unit finally learns the long-term and short-term time dependence of the current evolution process of the corresponding joint, and finally generates a hidden state that integrates the spatial and temporal characteristics, and the calculation formula is as follows:

[0049]

[0050] Step S304, constructing a model output layer, mapping the hidden state of the LSTM layer to all joint current prediction values of the next time through a fully connected network fusing the spatio-temporal characteristics :

[0051]

[0052] wherein the weight matrix is a linear transformation to compress the high-dimensional hidden state to the target output dimension, while the bias term plays a role of non-linear compensation.

[0053] Further, in step S40, the dynamic threshold is set as follows:

[0054] Step S401, collecting historical collision-free data as a training set, training the model and recording its prediction residual, to form a sequence ;

[0055] Step S402, analyzing the statistical distribution characteristics of the residual sequence to determine the initial detection threshold ;

[0056] Step S403, according to the real-time motion state of the robot, adjusting the threshold adaptively using the speed change of the joint, and the adjustment formula is:

[0057]

[0058] wherein, is an adjustment coefficient.

[0059] Further, in step S50, the collision detection judgment step is as follows:

[0060] Step S501, at each joint, applying a collision decision function to decide the output signal :

[0061]

[0062]

[0063] Step S502: if the residual is greater than the preset limit value, it is preliminarily determined that a collision occurs; in order to improve reliability, a continuous triggering mode is adopted, i.e. if the residual of a joint continuously exceeds the threshold value for K sampling periods, it is considered that a collision occurs at the joint, and the collision time is saved.

[0064] Further, in step S30, the GAT-LSTM fusion prediction model is trained using the collision-free data set, using the mean square error as the loss function between the predicted current and the actual current, and using the Adam optimizer to update the model parameters.

[0065] Further, the method further comprises a collision detection performance evaluation step, using recall rate, accuracy rate and precision rate as the indicators of detection performance; using the receiver operating characteristic curve to optimize the threshold value; and evaluating the key performance indicators on a real robot platform, including detection delay and false alarm rate.

[0066] Further, the method is suitable for six degrees of freedom and above collaborative robots, and does not require additional external force / torque sensors. BRIEF DESCRIPTION OF DRAWINGS

[0067] Figure 1 The network architecture of the GAT-LSTM hybrid prediction model in the embodiments of the present application is shown in the figure;

[0068] Figure 2 The joint current prediction result example graph of the GAT-LSTM model in embodiment 1 of the present application under the collision-free working condition is shown in the figure;

[0069] Figure 3 The joint current prediction mean square error comparison graph of different models in embodiment 1 of the present application is shown in the figure;

[0070] Figure 4 The joint current prediction mean absolute percentage error comparison graph of different models in embodiment 1 of the present application is shown in the figure;

[0071] Figure 5 The collision detection result schematic diagram in embodiment 1 of the present application is shown in the figure;

[0072] Figure 6 The receiver operating characteristic curve of the collision detection method in embodiment 1 of the present application is shown in the figure;

[0073] Figure 7 The experimental scene graph of embodiment 2 of the present application on the AUBO-e5 six degrees of freedom collaborative robot physical platform is shown in the figure;

[0074] Figure 8 The collision detection result of embodiment 2 of the present application on the physical platform is shown in the figure. DETAILED DESCRIPTION

[0075] The present application will be further described in detail below with reference to the accompanying drawings. It can be understood that the specific implementation examples described herein are only used to illustrate the related application, and are not limited to the application.

[0076] Example 1: This example is used to illustrate the model training step of the method described in the present application and the all-around performance test on the standard public data set to establish the performance basis.

[0077] Step S10, based on the public collaborative robot time series data set, the collaborative robot used has 6 degrees of freedom, and various joint state data under different motion modes are recorded, and the data sampling rate is 24Hz; the entire data set includes 91 robot random trajectory motion collision-free segments, each segment collects about 10000 continuous data points, and the total effective data sample is about 910,000; among them, the sample selection ratio of model parameter implementation training learning is about 90% of the total data, and the sample selection ratio of about 10% of the total data amount is used as a verification set; in addition, there are 52 groups of collision motion segments with artificial intervention, each group has about 1000 sampling points, and a total of 208 explicit collision event data are included as a separate test set;

[0078] Step S20, based on the time series data, a robot joint space-time graph structure is constructed ;

[0079] Step S30, a GAT-LSTM hybrid prediction model is constructed, as shown in Figure 1 , including the following steps:

[0080] Step S301, the network structure of the prediction model is constructed , , the neural network information of the prediction model;

[0081] Step S302, a two-layer GAT network structure is established using GAT neural network, 9 independent attention heads are used, and LeakyReLU is selected as the activation function of the attention mechanism;

[0082] Step S303, an LSTM neural network model is established, the space-time feature sequence obtained by GAT is input into the LSTM network after reconstruction, and the dimension of each layer of the LSTM hidden state is set to 128. This module captures the long-range context information of each joint state evolution;

[0083] Step S304, the output layer of the model is constructed, the hidden state of the last time step of the LSTM is mapped into a 6-dimensional vector, which is the current joint current prediction value ;

[0084] As preferred, the loss function is selected as MSE, the optimizer is selected as Adam, the initial learning rate is 0.004, the batch size is set as 64; the trained model is iterated for 50 cycles on the training set, and the training is ended in advance after the last iteration (5 cycles) when the loss of the validation set no longer decreases, and the network parameters corresponding to this period are selected as the optimal values, Figure 2 The joint current prediction results of the GAT-LSTM model under the non-collision working condition.

[0085] Step S40, on the normal data segments of the test set, the prediction accuracy of the trained GAT-LSTM model for the joint current at the future time is evaluated. The predicted sequences of each joint output by the model are compared with the corresponding real value sequences point by point, and the mean square error reflecting the overall deviation and the mean absolute percentage error reflecting the relative error are calculated respectively. As a comparison, under the same data set and training conditions, the traditional LSTM model and the CNN-LSTM fusion model are trained synchronously as the baseline, and the comparison results are shown in Table 1. Figure 3 、 4 Table 1 shows the comparison results of the prediction errors of the three models on each joint.

[0086] Table 1 Comparison of prediction accuracy MSE and MAPE of different degrees of freedom joints

[0087]

[0088] From Table 1, it can be seen that the proposed GAT-LSTM model has a much lower MSE and MAPE for current prediction of all 6 joints than the two baseline models. On joint 6, the MSE is reduced by more than 98% compared with LSTM and CNN-LSTM, which shows that the spatial topological structure between joints is explicitly modeled, and LSTM is used to capture the time sequence dynamic changes between sequences, so that the proposed model can more accurately learn the operation rules of the robot, and thus provide a higher credibility prediction basis for subsequent collision detection of the robot system.

[0089] The trained GAT-LSTM model is used for forward propagation of the entire test set to obtain a continuous current prediction sequence; the prediction residual of the sequence is calculated, and the collision is determined according to the dynamic threshold strategy proposed in the application; Table 2 shows the various indicators of the system collision detection under different global threshold conditions, including recall rate, precision rate and accuracy;

[0090] Table 2 Comparison of recall rate, precision rate and accuracy of the collision detection algorithm under different threshold values

[0091]

[0092] The performance on the test set is the best in the whole process when the dynamic threshold is set to 1.5, the collision events as shown in Figure 5 can be detected, the recall rate of the method is as high as 99%, and 95% of the alarm points belong to real collisions; the total correct rate is 99%, as shown in Figure 6 , the ROC curve is drawn, and it can be seen that the method proposed in the application can well complete the collision recognition and has a very low false alarm probability.

[0093] Embodiment 2: This embodiment aims to verify whether the method of the application is feasible, real-time and robust on a real physical robot, and realizes the closed-loop verification from “laboratory data” to “industrial scene”.

[0094] An AUBO-e5 type six-degree-of-freedom collaborative robot is used as a verification platform, the repeatability accuracy is ±0.05mm, and high-precision encoders are installed at each joint part; the collision detection algorithm program of the application is written in the upper computer, and is provided to the Ethernet communication interface of the program through the robot controller, and all 6 joint motor raw current, position and speed data streams in the robot controller are subscribed and read in real time at a fixed frequency of 200Hz.

[0095] The GAT-LSTM model pre-trained on the public data set in embodiment 1 is loaded into the upper computer, the robot is controlled to move according to the preset complex random trajectory, and in the movement process of the robot, the experimenter applies three times of instantaneous external force collisions with different degrees to the robot in different positions of the robot (as shown in (b)-(d), close to the position of the end effector and the position of the elbow joint link). Figure 7 ).

[0096] When the robot is running, the whole experiment process is monitored and detected in real time by the online detection algorithm, and the predicted current of each joint, the dynamic threshold and the collision determination are output. The key results are shown in Figure 8 , three artificial collisions (occurring at about 0.260s, 0.485s and 0.740s) have been accurately detected by the system. At the same time, it can be observed that the current residual of joint 2 and joint 3 has obvious peak value after the collision occurs, and will soon exceed its own dynamic threshold.

[0097] From the collision occurrence ( ) to the delay time of the system to produce a collision alarm signal ( ) ​) are measured as 2.0 ms, 2.5 ms and 4.0 ms respectively. All the delays are much smaller than the sampling period of the system (5 ms), meeting the millisecond-level real-time requirement of safety response of the collaborative robot.

[0098] During the experiment test of about 10 seconds, the robot did not give false alarms in normal conditions (except for three intentional collisions) when the robot was in dynamic states such as high-speed operation, sharp stop and trajectory turning. Therefore, it is proved that the dynamic threshold judgment method adopted by the application can correctly judge whether the abnormal fluctuation of current during movement is caused by complex movement state or real collision impact, and has strong robustness.

Claims

1. A sensorless collision detection method for collaborative robots based on GAT-LSTM spatiotemporal feature fusion, characterized in that, The method comprises the following steps: Step S10, collect time series data of each joint of the body sensor of the collaborative robot, including joint motor current , joint position and joint speed , wherein represents the sampling time; Step S20, based on the time series data, construct a robot joint space-time graph structure ; Step S30, input the graph structure in the pre-trained GAT-LSTM hybrid prediction model At the next moment, output the predicted current value of each joint ; Step S40, calculating a predicted current value between the actual current value and the predicted current value ; Step S50, based on the residual error compared with a preset dynamic threshold to determine whether a collision occurs; Further, the robot joint space-time graph structure is constructed in step S20 The process is as follows: Step S201: Calculate the motor current of each joint at each moment. Joint position and joint velocity Each node serves as a separate node, forming a multi-level node set. ; Step S202, based on the joint connection sequence of the robot, construct a joint node sequence graph ; Step S203, a correlation graph is established according to Pearson correlation coefficients between the sequences of the nodes The formula for calculating the Pearson correlation coefficient is: ； wherein, denotes the Pearson correlation coefficient between nodes , , denotes the number of samples, and at time instant for a node , its sample value is at time instant for a node , is a temporary variable traversing the set of two target nodes; a threshold is set, , then a connection is established between the two nodes, forming a directed edge. Step S204, taking the joint node sequence graph and the relevance graph as the final graph structure where is the edge set; Further, the GAT-LSTM hybrid prediction model in step S30 is constructed in the following specific process: Step S301, establishing a network structure of a prediction model , is the neural network information of the prediction model; Step S302, constructing a GAT neural network model to a graph structure A set of nodes with spatial features in data As input, wherein The number of nodes is The number of features of each node is The output is a sequence of node embedding vectors enhanced by spatial features ; Step S303, constructing an LSTM neural network model to receive the node embedding vector sequence from the GAT layer Each LSTM unit learns the long and short term time dependence of the corresponding joint current evolution through its forget gate, input gate and output gate mechanism, and finally outputs a hidden state that integrates the spatial and temporal features ; Step S304, a model output layer is constructed to map the hidden state of the LSTM layer to the predicted joint current values at the next time instant through a fully connected network fusing spatio-temporal characteristics to map to all joint current predictions at the next time instant : ； wherein, represents the predicted output vector at time step , the weight matrix performs a linear transformation on the high-dimensional hidden state and compresses it to the target output dimension, while the bias term plays a role of non-linear compensation.

2. The sensorless collision detection method for collaborative robots based on GAT-LSTM spatiotemporal feature fusion according to claim 1, characterized in that, The GAT neural network model adopts the following structure: At the node of the graph structure, a weight vector A self-attention mechanism implemented by a parameterized single-layer feedforward neural network The function is to calculate and represent the importance weight of the center node by the neighborhood node features; the weight calculation process shares parameters between edges: ； wherein, represents a node between nodes , represents a feature vector of a node in the layer, represents a feature vector of a neighboring node of a node in the layer, ; to achieve the feature dimension mapping from the original space to the new high-dimensional space , the weight matrix linearly projects the input features; For each center node, the attention weights of all adjacent nodes are collected, and a Softmax function is input for normalization processing, so as to obtain an attention coefficient representing the relative importance of each adjacent node: ； wherein, denotes the normalized attention coefficient, is an exponential function, denotes the node and its any adjacent node between the original attention weight; denotes the sum of the correlation values of all adjacent nodes of the node ; the transformed features of each adjacent node are weighted and summed after obtaining the normalized attention coefficient, and a nonlinear activation function is processed, that is, the updated output feature vector of the node is obtained, which is calculated as: ； wherein, represents a weighted sum of features of all neighboring nodes of the node , represents the trainable weight matrix of the th layer; in the feature propagation process, a multi-head attention mechanism is used for calculation and aggregation to obtain variable-related features, and the specific calculation formula is as follows: ； wherein, denotes the node after multi-head attention aggregation of the current layer, the feature vector of the current layer; and the feature of the node and its adjacent nodes , the normalized attention coefficient computed by the th attention head, a trainable weight matrix, a nonlinear activation function, denotes vector concatenation, the total number of attention heads. 3.The sensorless collision detection method for collaborative robots based on GAT-LSTM spatio-temporal feature fusion according to claim 1, characterized in that, The dynamic threshold The setting method steps are as follows: Step S401, collect historical non-collision data as a training set, train the model and record its prediction residual, constitute a sequence ; Step S402, analyzing the statistical distribution characteristics of the sequence to determine the initial detection threshold ; Step S403, according to the real-time motion state of the robot, the speed of the joint is used to adjust the threshold value The adaptive adjustment threshold value is changed, and the adjustment formula is: ； wherein is a modulation coefficient.

4. The sensorless collision detection method for collaborative robots based on GAT-LSTM spatiotemporal feature fusion according to claim 1, characterized in that, The judgment step of the collision detection is specifically as follows: Step S501, at each joint, apply a collision decision function decision output signal : ； ； In step S502, if the absolute value of the residual error is greater than a set threshold, it is determined that a collision occurs; otherwise, it is determined that no collision accident occurs.

Images