Robot abnormal action detection method, device and system and storage medium

Abstract

The invention provides a robot abnormal action detection method, device and system and a storage medium, and belongs to the technical field of action detection.The method comprises the steps that original joint axis sequence data are imported, and a space-time diagram is constructed through the original joint axis sequence data; constructing a training model, and performing model analysis on the training model through the space-time diagram to obtain a detection model; and importing to-be-detected joint axis sequence data, and detecting the to-be-detected joint axis sequence data through the detection model to obtain a detection result of the abnormal action of the robot. According to the invention, the method achieves the quick positioning and detection of abnormal actions, achieves the higher response speed and detection precision, effectively reduces the damage risk, and can meet the requirements of the precision and time performance of an industrial scene.

Description

Technical Field

[0001] The present invention mainly relates to the technical field of motion detection, and in particular to a method, device, system and storage medium for detecting abnormal motion of a robot. Background Art

[0002] As key equipment in the field of intelligent manufacturing, industrial robots are a solid support for improving the level of automation and intelligence in industrial production. However, with the widespread application of industrial robots, robot safety accidents occur frequently. Due to the great power and high speed of robots, robot safety accidents can cause production interruptions and equipment damage, or even personal injury. Installing a visual system in industrial production scenarios to continuously monitor and analyze the movements of industrial robots can provide timely warnings of robot safety accidents.

[0003] Industrial robots involve complex and changeable motion patterns in the process of performing tasks, resulting in a variety of abnormal motions, which makes abnormal motion detection challenging. In order to meet this challenge, it is necessary to finely model the complex dynamic model of the industrial robot and accurately distinguish abnormal motions from normal motions. This modeling process not only considers time series data, but also involves a highly coupled relationship between time and space dimensions. This spatiotemporal coupling means that the changes in action frames are not limited to a single dimension of space or time, but requires the model to capture and understand the coupled changes in the temporal and spatial dimensions of industrial robot motions and the dynamic characteristics of the industrial robot structure. Existing studies have attempted to detect abnormal modes of industrial robots through different data-driven methods. Existing technologies distinguish abnormalities by reconstructing errors or use Gaussian process regression to improve Markov modeling to calculate and classify abnormal states of industrial robots. However, the above methods often show limitations when processing high-dimensional data and complex spatiotemporal relationships, and are prone to ignoring some important features or associations, and cannot accurately mine abnormal motion patterns hidden in normal motions, resulting in poor detection accuracy. Efficient understanding and grasp of the spatiotemporal characteristics and complex dynamic models of industrial robot movements usually requires deeper networks to capture the complex relationships of industrial robot movements, but this comes at the cost of a large amount of computational overhead. This high computing demand not only increases the consumption of hardware resources, but also significantly limits the response speed of abnormal motion detection. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a robot abnormal motion detection method, device, system and storage medium in view of the deficiencies in the prior art.

[0005] The technical solution of the present invention to solve the above technical problem is as follows: A method for detecting abnormal motion of a robot comprises the following steps:

[0006] Import multiple original joint axis sequence data at multiple moments, and construct a spatiotemporal graph through all the original joint axis sequence data;

[0007] Constructing a training model, and performing model analysis on the training model through the space-time graph to obtain a detection model;

[0008] The joint axis sequence data to be detected is imported, and the joint axis sequence data to be detected is detected by the detection model to obtain the detection result of the abnormal movement of the robot.

[0009] Another technical solution of the present invention to solve the above technical problem is as follows: A robot abnormal motion detection device, comprising:

[0010] Import module, used to import multiple original joint axis sequence data at multiple moments;

[0011] A building module for constructing a spatiotemporal graph from all raw joint axis sequence data;

[0012] A model analysis module is used to construct a training model, and perform model analysis on the training model through the space-time graph to obtain a detection model;

[0013] The import module is also used to import the joint axis sequence data to be detected;

[0014] The detection result acquisition module is used to detect the joint axis sequence data to be detected through the detection model to obtain the detection result of the abnormal movement of the robot.

[0015] Based on the above-mentioned robot abnormal motion detection method, the present invention also provides a robot abnormal motion detection system.

[0016] Another technical solution of the present invention to solve the above-mentioned technical problem is as follows: a robot abnormal motion detection system comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the robot abnormal motion detection method as described above is implemented.

[0017] Based on the above-mentioned method for detecting abnormal motion of a robot, the present invention also provides a computer-readable storage medium.

[0018] Another technical solution of the present invention to solve the above technical problem is as follows: a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the robot abnormal motion detection method as described above is implemented.

[0019] The beneficial effects of the present invention are as follows: a space-time graph is constructed through the original joint axis sequence data, a detection model is obtained through model analysis of the training model through the space-time graph, and the detection result of the robot's abnormal movement is obtained by detecting the joint axis sequence data to be detected by the detection model, thereby achieving rapid positioning and detection of abnormal movements, and also achieving higher response speed and detection accuracy, effectively reducing the risk of injury, and being able to meet the accuracy and time performance requirements of industrial scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A schematic diagram of a flow chart of a method for detecting abnormal movements of a robot provided by an embodiment of the present invention;

[0021] Figure 2 This is a module block diagram of a robot abnormal motion detection device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0023] Figure 1 A schematic flow chart of a method for detecting abnormal motion of a robot provided in an embodiment of the present invention.

[0024] like Figure 1 As shown, a method for detecting abnormal motion of a robot comprises the following steps:

[0025] Import multiple original joint axis sequence data at multiple moments, and construct a spatiotemporal graph through all the original joint axis sequence data;

[0026] Constructing a training model, and performing model analysis on the training model through the space-time graph to obtain a detection model;

[0027] The joint axis sequence data to be detected is imported, and the joint axis sequence data to be detected is detected by the detection model to obtain the detection result of the abnormal movement of the robot.

[0028] It should be understood that in order to obtain the joint axis sequence data (i.e., the original joint axis sequence data) of the six-axis industrial robot, the present invention uses existing work to detect and extract the rotation joint axis coordinate information of the six-axis industrial robot. Specifically, the present invention inputs the six-axis industrial robot action video sequence into the YOLOF industrial robot posture estimation detector, and the detector analyzes the action video sequence frame by frame to obtain the industrial robot joint axis category in each frame of the action sequence, and detects and outputs the normalized two-dimensional coordinates and confidence data of each joint axis.

[0029] Specifically, in the evaluation phase, the joint axis sequence data containing abnormal actions (i.e., the joint axis sequence data to be detected) is input into the variable spatiotemporal graph normalized flow model (i.e., the detection model). The abnormal action features will be distributed in the area with lower probability in the Gaussian distribution space, and the abnormal score obtained by the inverse transformation will be significantly lower than the abnormal score of the normal action. Then, the abnormal action of the industrial robot is judged based on the optimal threshold of the optimal area under the curve (AUC) obtained by training.

[0030] In the above embodiment, a space-time graph is constructed using the original joint axis sequence data, a detection model is obtained by analyzing the training model through the space-time graph, and the detection result of the abnormal movement of the robot is obtained by detecting the joint axis sequence data to be detected by the detection model, thereby achieving rapid positioning and detection of abnormal movements, and also achieving higher response speed and detection accuracy, effectively reducing the risk of injury, and being able to meet the accuracy and time performance requirements of industrial scenarios.

[0031] Optionally, as an embodiment of the present invention, the process of constructing a spatiotemporal graph using all original joint axis sequence data includes:

[0032] Respectively connecting each original joint axis sequence data at each moment and the original joint axis sequence data adjacent to each original joint axis sequence data at each moment to obtain spatial edges corresponding to each original joint axis sequence data at each moment, and collecting all the spatial edges to obtain a spatial edge set;

[0033] Respectively connect each original joint axis sequence data at each moment and each original joint axis sequence data at a time adjacent to each moment to obtain a time edge corresponding to each original joint axis sequence data at each moment, and collect all the time edges to obtain a time edge set, and collect all the original joint axis sequence data at all moments to obtain an original joint axis sequence data set;

[0034] The space edge set, the time edge set and the original joint axis sequence data set are constructed by the first formula to obtain a space-time graph, and the first formula is:

[0035] G=(V,E),

[0036] Among them, E∈(E Spatial ,E Temporal ),

[0037] Among them, G is the spatiotemporal graph, V is the original joint axis sequence data set, E is the target edge set, and E Spatial is the set of spatial edges, E Temporal is the set of time edges.

[0038] Specifically, the 2D coordinate representation of the joint axis sequence of a six-axis industrial robot efficiently and compactly describes the action type of the industrial robot, and the action data of a continuous joint frame sequence can intuitively represent the motion mode of the industrial robot. To this end, the present invention constructs a spatiotemporal graph structure of a six-axis industrial robot: the rotating joint axes of the industrial robot are used as graph nodes, and the nodes are naturally connected in sequence from the base to the end effector to form spatial edges, and the connections between the same nodes of consecutive frames form temporal edges. Specifically, the present invention constructs an undirected spatiotemporal graph G = (V, E) on an industrial robot joint axis sequence with N nodes and T frames, where V = {v it |t∈[1,τ],i∈[1,N]}, represents the set of all nodes E∈(E Spatial ,E Temporal ) represents the connection relationship between nodes, including spatial edges E Spatial and time edge E Temporal , E Spatial describes the natural connection of the joint axes at each frame, E Temporal It describes the connection of the same joint axis in consecutive frames. Therefore, the time-space diagram is constructed, from point to surface, from static to dynamic, to represent the motion trajectory of each joint axis node of the industrial robot in the time-space dimension.

[0039] In the above embodiment, a space-time graph is constructed through all the original joint axis sequence data, which breaks through the limitations of the predefined industrial robot graph structure modeling, improves the distinction of joint axis features, achieves a higher response speed and detection accuracy, effectively reduces the risk of injury, and can meet the accuracy and time performance requirements of industrial scenarios.

[0040] Optionally, as an embodiment of the present invention, the training model includes an activation normalization layer, a data permutation layer, and a stacked spatiotemporal coupling layer.

[0041] The process of constructing a training model and performing model analysis on the training model through the spatiotemporal graph to obtain a detection model includes:

[0042] Normalizing the spatiotemporal graph through the activation normalization layer to obtain a plurality of normalized joint axis data;

[0043] Performing replacement processing on each of the normalized joint axis data respectively through the data replacement layer to obtain replaced joint axis data corresponding to each of the normalized joint axis data;

[0044] The spatiotemporal graph and each of the replaced joint axis data are scored and analyzed respectively by the stacked spatiotemporal coupling layer to obtain a predicted action sequence score corresponding to each of the normalized joint axis data;

[0045] Importing the real action sequence scores corresponding to each of the normalized joint axis data, and performing loss function calculation on all the predicted action sequence scores and all the real action sequence scores using a negative log-likelihood minimization algorithm to obtain a loss function;

[0046] The parameters of the training model are updated according to the loss function to obtain a detection model.

[0047] It should be understood that the real action sequence score can be obtained through data analysis software, installing sensors on the user, etc.

[0048] It should be understood that the joint axis sequence data (i.e., the joint axis data to be trained) is first scaled, normalized, and inversely permuted and sorted through the activation normalization layer and the data permutation layer, and then the stacked spatiotemporal coupling layer (i.e., the stacked spatiotemporal coupling layer) is used to embed the complex motion data distribution of the industrial robot into the Gaussian distribution space, and bidirectional mapping is performed through affine coupling transformation to estimate the log-likelihood of each frame of action.

[0049] Specifically, the role of the activation normalization layer (ActNorm) is similar to the batch normalization layer. The present invention calculates the mean μ of the input data of each channel in the entire batch C and variance And the data of each channel is rescaled according to the mean and variance to achieve normalized data. The formula is expressed as:

[0050]

[0051] in, is the scaled and translated output, λ C With ε C are learnable parameters used to scale and translate the normalized data of each channel. J is a constant of 1e-6 to avoid division by zero. However, unlike batch normalization, the ActNorm layer dynamically initializes the mean deviation and scale scaling of the input data based on the data, making the data distribution at initialization close to the standard normal distribution, which helps to accelerate the convergence process of the model.

[0052] Specifically, the role of the data permutation layer is to perform a permutation operation on the channel dimension of the input tensor. Specifically, for an input tensor x with C channels, the permutation operation converts the channel order x = {x1, x2, …, x C} is transformed into x′={x ν(1) ,x ν(2) ,…,x ν(C)}, where ν is a permutation sequence of channels. The permutation layer can break the assumption of independence between channels, help the model learn the correlation between channels, and avoid the problem that some channels are never transformed in subsequent affine coupling operations. In addition, in order to restore the permuted data x′ to the original order x, the inverse permutation operation ν is used -1 This ensures the reversibility of the model.

[0053] It should be understood that the present invention uses log-likelihood to reflect the model to generate the real data distribution p X The log-likelihood can be expressed as:

[0054]

[0055] In the unsupervised training process, maximizing the log-likelihood is equivalent to maximizing the probability of generating data, that is, minimizing the negative log-likelihood (NLL). In this way, the present invention facilitates adjustment and better fits the data distribution. Therefore, the present invention trains the model by minimizing the loss function of the negative log-likelihood of the training sample, and the formula is as follows:

[0056]

[0057] The above formula describes the p generated by the evaluation model after generating sample x. X The negative log-likelihood between the probability density function of (x) and the corresponding probability of the actual data x.

[0058] In the above embodiment, the detection model is obtained by analyzing the training model through the space-time graph, which helps to accelerate the convergence process of the model, ensure the reversibility of the model, realize the rapid positioning and detection of abnormal actions, and also achieve a higher response speed and detection accuracy, effectively reducing the risk of injury, and can meet the accuracy and time performance requirements of industrial scenarios.

[0059] Optionally, as an embodiment of the present invention, the stacked spatiotemporal coupling layer includes a plurality of sequentially arranged variable spatiotemporal convolutional networks,

[0060] The process of performing score analysis on the spatiotemporal graph and each of the replaced joint axis data respectively through the stacked spatiotemporal coupling layer to obtain a predicted action sequence score corresponding to each of the normalized joint axis data includes:

[0061] Performing affine transformation analysis on the spatiotemporal graph and each of the displaced joint axis data respectively through the first variable spatiotemporal convolutional network, and inputting the analysis results into the next variable spatiotemporal convolutional network, until the last variable spatiotemporal convolutional network is passed to obtain affine transformed joint axis features corresponding to each of the normalized joint axis data;

[0062] Respectively splicing each of the first segmented joint axis features with the affine transformed joint axis features corresponding to each of the normalized joint axis data to obtain a spliced ​​joint axis feature corresponding to each of the normalized joint axis data;

[0063] Performing Gaussian spatial distribution mapping processing on each of the spliced ​​joint axis features respectively to obtain mapped joint axis features corresponding to each of the normalized joint axis data;

[0064] Respectively performing inverse mapping processing on each of the mapped joint axis features to obtain a predicted action sequence score corresponding to each of the normalized joint axis data;

[0065] The first variable spatiotemporal convolutional network is used to analyze the affine transformation of the spatiotemporal graph and each of the displaced joint axis data, specifically:

[0066] Segmenting each of the replaced joint axis data according to a preset channel dimension to obtain a first segmented joint axis feature corresponding to each of the normalized joint axis data and a second segmented joint axis feature corresponding to each of the normalized joint axis data;

[0067] Performing affine transformation processing on the spatiotemporal graph and each of the first segmented joint axis features respectively through a first variable spatiotemporal convolutional network to obtain the joint axis features to be segmented corresponding to each of the normalized joint axis data;

[0068] Segmenting each of the to-be-segmented joint axis features respectively to obtain a scale offset value corresponding to each of the normalized joint axis data and a scaling value corresponding to each of the normalized joint axis data;

[0069] The second formula is used to fuse each of the second segmented joint axis features, the scale offset value corresponding to each of the normalized joint axis data, and the scaling value corresponding to each of the normalized joint axis data, to obtain the affine transformed joint axis features corresponding to each of the normalized joint axis data. The second formula is:

[0070]

[0071] in, is the joint axis feature after affine transformation corresponding to the mth joint axis data to be trained, is the scaling value corresponding to the mth joint axis data to be trained, is the joint axis feature after the second segmentation corresponding to the mth joint axis data to be trained, is the scale offset value corresponding to the mth joint axis data to be trained.

[0072] It should be understood that the stacked spatiotemporal coupling layer is formed by stacking two layers of affine coupling layers (i.e., variable spatiotemporal convolutional networks). A single layer of affine coupling layer (i.e., variable spatiotemporal convolutional network) realizes reversible mapping of data through affine transformation, and stacked spatiotemporal coupling layers (i.e., variable spatiotemporal convolutional networks) realize more complex transformations to improve the expressive power of the model, so as to better approximate the real data space distribution. In order to capture the complex relationships and action patterns of industrial robots in time and space dimensions, the present invention uses a variable spatiotemporal graph convolutional network as a conditional network for affine transformation in the affine coupling layer, so as to obtain the spatiotemporal high-level semantics of the posture and action of the industrial robot.

[0073] Specifically, a single-layer affine coupling layer (i.e., variable spatiotemporal convolutional network) first transforms the input X∈R B×C×T×V (i.e., the joint axis data after permutation) is divided into two parts X1 (i.e., the joint axis feature after the first segmentation) and X2 (i.e., the joint axis feature after the second segmentation) according to the channel dimension, and then X2 (i.e., the joint axis feature after the second segmentation) is kept unchanged, and X1 (i.e., the joint axis feature after the first segmentation) is input into the affine coupling layer (i.e., the variable spatiotemporal convolutional network) for affine transformation. The result of X1 after affine transformation (i.e., the joint axis feature to be segmented) is segmented to obtain the scale offset value X 1_shift and the scaling value X 1_scale , and merge it with X2 (i.e., the joint axis feature after the second segmentation) to obtain Y2=X 1_scale (X2+X 1_shift ). Next, use Y2 (i.e., the joint axis feature after affine transformation) as the input of the next affine coupling layer to complete the second affine transformation. Subsequently, Y1 (i.e., the joint axis feature after the first segmentation) and Y2 (i.e., the joint axis feature after affine transformation) are concatenated to obtain the spatiotemporal high-level semantics Y (i.e., the joint axis feature after concatenation) of the industrial robot's posture action, where Y1 = X1. In the stage of implementing a reversible transformation to map the complex data of the skeleton sequence to a simple Gaussian distribution. The transformation function f of the reversible normalized flow NF :X→Z distributes the input data in the real data space X∈P X (x) is mapped to a Gaussian space distribution Z∈p Z (z), where Gaussian distribution Subsequently, the Gaussian distribution space is inversely mapped back to the real data space to perform action sequence scoring (i.e., predict action sequence scoring). According to the transformation rule of the probability density function, the probability density p of the original variable x distributed in the real data space isX (x) can be obtained by the probability density function p of the latent variable z distributed in the latent space Gaussian Z (z) and Jacobian determinant transformation, the formula is as follows:

[0074]

[0075] in, is the Jacobian matrix of z with respect to x.

[0076] In the above embodiment, the spatiotemporal graph and the displaced joint axis data are scored and analyzed by stacking the spatiotemporal coupling layers to obtain the predicted action sequence score, which can obtain the spatiotemporal high-level semantics of the industrial robot's posture movements, realize rapid positioning and detection of abnormal movements, and also achieve a higher response speed and detection accuracy, effectively reducing the risk of injury, and can meet the accuracy and time performance requirements of industrial scenarios.

[0077] Optionally, as an embodiment of the present invention, the variable spatiotemporal convolutional network includes a variable spatial graph convolution layer, a spatial channel attention layer, a first batch normalization layer, a first Relu activation function layer, a random dropout layer, a temporal graph convolution layer, a temporal attention layer, a second batch normalization layer and a second Relu activation function layer,

[0078] The process of performing affine transformation processing on the spatiotemporal graph and each of the first segmented joint axis features respectively through the first variable spatiotemporal convolutional network to obtain the joint axis features to be segmented corresponding to each of the normalized joint axis data includes:

[0079] Performing spatial convolution processing on the spatiotemporal graph and each of the first segmented joint axis features respectively through the variable spatial graph convolution layer to obtain spatially convolved joint axis features corresponding to each of the normalized joint axis data;

[0080] Performing spatial channel attention mechanism calculation on each of the spatial convolution joint axis features through the spatial channel attention layer to obtain original spatial channel joint axis features corresponding to each of the normalized joint axis data;

[0081] Normalizing each of the original spatial channel joint axis features respectively through the first batch normalization layer to obtain normalized spatial channel joint axis features corresponding to each of the normalized joint axis data;

[0082] Mapping each of the normalized spatial channel joint axis features through the first Relu activation function layer to obtain mapped spatial channel joint axis features corresponding to each of the normalized joint axis data;

[0083] Extracting features of the mapped spatial channel joint axis features respectively through the random inactivation layer to obtain extracted spatial channel joint axis features corresponding to the normalized joint axis data;

[0084] Performing temporal convolution processing on each of the extracted spatial channel joint axis features through the temporal graph convolution layer to obtain temporal convolutional spatial channel joint axis features corresponding to each of the normalized joint axis data;

[0085] The temporal attention layer performs temporal attention mechanism calculation on each of the temporal convolution spatial channel joint axis features to obtain original temporal joint axis features corresponding to each of the normalized joint axis data;

[0086] Normalizing each of the original temporal joint axis features respectively through the second batch normalization layer to obtain normalized temporal joint axis features corresponding to each of the normalized joint axis data;

[0087] Each of the normalized temporal joint axis features is mapped through the second Relu activation function layer to obtain the joint axis features to be segmented corresponding to each of the normalized joint axis data.

[0088] Specifically, the variable spatiotemporal graph convolution network of the present invention is mainly composed of a variable spatial graph convolution module (i.e., a variable spatial graph convolution layer), a temporal graph convolution module (i.e., a temporal graph convolution layer), a Dropout layer (i.e., a random inactivation layer), and a residual connection. Both the spatial graph convolution (i.e., a variable spatial graph convolution layer) and the temporal graph convolution (i.e., a temporal graph convolution layer) are processed by a batch normalization layer (i.e., a first batch normalization layer and a second batch normalization layer) and a Relu activation function layer (i.e., a first Relu activation function layer and a second Relu activation function layer). The batch normalization layer (i.e., the first batch normalization layer and the second batch normalization layer) is used to standardize the features of each batch so that they maintain a stable distribution. The Relu activation function (i.e., the first Relu activation function layer and the second Relu activation function layer) is used to introduce nonlinearity so as to improve the network's ability to learn complex data. In addition, when studying the actions of industrial robots, the present invention found that when a six-axis industrial robot performs different actions, the mapping of the motion mode of the joint axis sequence to the actual situation may appear one-to-many. This is because the number of joint axes of the six-axis industrial robot limits the obvious distinction between different actions. In order to alleviate this problem, the present invention adds a spatial channel attention mechanism (SC-Attn) (i.e., spatial channel attention layer) and a temporal attention mechanism (T-Attn) (i.e., temporal attention layer) to the variable spatial graph convolution module (i.e., variable spatial graph convolution layer) and the temporal graph convolution module (i.e., temporal graph convolution layer), respectively.

[0089] It should be understood that SC-Attn (i.e., spatial channel attention layer) enables the model to pay more attention to nodes with large motion amplitude in the graph structure in the spatial dimension. At the same time, T-Attn (i.e., temporal attention layer) enables the model to dynamically adjust the attention to key frames in the temporal dimension, so as to better capture important changes in the action sequence.

[0090] In the above embodiment, the first variable spatiotemporal convolutional network is used to perform affine transformation processing on the spatiotemporal graph and each first segmented joint axis feature to obtain the joint axis features to be segmented, thereby achieving rapid positioning and detection of abnormal movements, and also achieving a higher response speed and detection accuracy, effectively reducing the risk of injury, and being able to meet the accuracy and time performance requirements of industrial scenarios.

[0091] Optionally, as an embodiment of the present invention, the space-time graph further includes an original adjacency matrix and a plurality of graph nodes.

[0092] The process of performing spatial convolution processing on the spatiotemporal graph and each of the first segmented joint axis features respectively through the variable spatial graph convolution layer to obtain the spatially convolved joint axis features corresponding to each of the normalized joint axis data includes:

[0093] The original adjacency matrix, all graph nodes, and each of the first segmented joint axis features are spatially convolved by the third formula to obtain the spatially convolved joint axis features corresponding to each of the normalized joint axis data. The third formula is:

[0094]

[0095] in,

[0096] in, is the joint axis feature after spatial convolution corresponding to the mth joint axis data to be trained, K S is the convolution kernel size in the spatial domain, w S is the time domain weight function, is the joint axis feature after the first segmentation corresponding to the mth joint axis data to be trained, δ k For the k The weight function of the convolution kernel, Λ k For the k The preset normalized diagonal matrix of convolution kernels, G k For the k The original adjacency matrix of the convolution kernel, G′ k For the k The target adjacency matrix of the convolution kernel, Q k (v i ,v j) is the similarity between the i-th graph node and the j-th graph node, F is the residual connection function, α is a learnable parameter, φ k and All are k Gaussian function of the convolution kernel, N is the total number of graph nodes.

[0097] It should be understood that the spatiotemporal graph convolution uses a predefined graph structure for modeling. When considering the intersection with other nodes, it is limited to considering the distance from the target vertex to other nodes, which makes the connection between nodes that are not directly connected ignored. However, for the posture of the motion state of the industrial robot, the movement of the rotation axis node close to the base when the end effector performs different actions is also crucial. For example, when the industrial robot performs spot welding and arc welding operations, the angles between the end effector joint points and the neighborhood nodes of the graph structure are highly similar when performing welding operations, which makes it easy to confuse and misjudge the two tasks, thereby affecting feature extraction. The arc welding operation may extend along a complex path, resulting in the base rotation axis node on the graph changing at a greater angle than the spot welding operation. Therefore, if the nodes of the base rotation axis are considered in the graph structure, the feature discrimination can be improved.

[0098] Specifically, the present invention proposes a variable graph convolution layer, which realizes dynamic modeling of graph structure with different actions by introducing a learnable adjacency matrix and a graph node similarity matrix based on the original graph adjacency matrix. Specifically, the variable graph convolution layer of the present invention consists of an original adjacency matrix G and two supplementary matrices that can change with the graph structure: (1) The original adjacency matrix G is a graph node adjacency matrix of size N×N, which represents the spatial topological structure of the industrial robot spatiotemporal graph. (2) The learnable adjacency matrix G′ (i.e., the target adjacency matrix) is an N×N adjacency matrix that is continuously optimized according to the learnable parameter α in the reverse process, and its feature representation is generated based on the connection relationship and connection strength of the learning graph structure. Therefore, the learnable adjacency matrix (i.e., the target adjacency matrix) It will vary with the graph structure. (3) In order to determine whether there is a connection between two nodes and the strength of the connection, the invention uses a normalized Gaussian function φ k and The input features are embedded into the new feature space, and the two feature maps are reshaped into b×v×ct and b×ct ×v The matrix is ​​constructed and the dot product between two nodes is calculated to measure the similarity between the two representations, as follows:

[0099]

[0100] Then, the two matrices are multiplied by matrix multiplication and normalized by Tanh activation function to obtain a graph node similarity matrix of size N×N. Finally, the present invention integrates these three matrices by matrix addition to obtain a new N×N variable adjacency matrix. In order to stabilize the training, the present invention adds residual connections to each layer.

[0101] Therefore, the formula for variable spatial graph convolution can be expressed as:

[0102]

[0103] Among them, δ k is the weight function, F represents the residual connection, f in represents the input feature map, G k Represents the original adjacency matrix. GL k represents the learnable adjacency matrix, φ k and is the Gaussian function used to calculate the similarity.

[0104] In the above embodiment, spatial convolution processing is performed on the space-time graph and each joint axis feature after the first segmentation through a variable spatial graph convolution layer to obtain the joint axis features after spatial convolution, thereby realizing rapid positioning and detection of abnormal movements, and also realizing higher response speed and detection accuracy, effectively reducing the risk of injury, and being able to meet the accuracy and time performance requirements of industrial scenarios.

[0105] Optionally, as an embodiment of the present invention, the process of performing spatial channel attention mechanism calculation on each of the spatial convolution joint axis features through the spatial channel attention layer to obtain the original spatial channel joint axis features corresponding to each of the normalized joint axis data includes:

[0106] The fourth formula is used to perform spatial channel attention calculation on each of the joint axis features after spatial convolution, so as to obtain the original spatial channel joint axis features corresponding to each of the normalized joint axis data. The fourth formula is:

[0107]

[0108] in,

[0109] in, is the original space channel joint axis feature corresponding to the mth joint axis data to be trained, is the channel attention weight corresponding to the mth joint axis data to be trained, is the spatial attention weight corresponding to the mth joint axis data to be trained, ξ is the Sigmoid activation function, l 1×1is 1×1 convolution processing, AvgPool is average pooling processing, is the joint axis feature after spatial convolution corresponding to the mth joint axis data to be trained, fc1c is the fully connected layer function, and γ is the Relu activation function.

[0110] It should be understood that SC-Attn consists of spatial attention and channel attention. The spatial attention first focuses on the input feature vector f S ∈R B×C×V Perform average pooling, then perform a 1×1 convolution operation on the pooling result, and then use the Sigmoid activation function to obtain the spatial attention weight. The formula can be defined as:

[0111] A S =ξ(l 1×1 (AvgPool(f S ))),

[0112] in, represents the Sigmoid activation function. Channel attention first focuses on the input feature f C ∈R B×C Perform average pooling, then pass through the first fully connected layer and use the Relu activation function to get the intermediate feature se1, then pass through the second fully connected layer and use the Sigmoid activation function to get the attention weight se2. The formula can be defined as:

[0113] A C =ξ(fc2c(γ(fc1c(AvgPool(f C )))),

[0114] Among them, γ = max(0,x) represents the Relu activation function, and fc1c and fc2c are fully connected layers. The input of channel attention is the feature vector weighted by spatial attention, so the spatial channel attention (i.e., the original spatial channel joint axis feature) can be expressed as:

[0115] A SC =A C (A S (f)).

[0116] In the above embodiment, the spatial channel attention mechanism is used to calculate the original spatial channel joint axis features of the joint axis features after spatial convolution through the spatial channel attention layer, thereby realizing rapid positioning and detection of abnormal movements, and also realizing higher response speed and detection accuracy, effectively reducing the risk of injury, and being able to meet the accuracy and time performance requirements of industrial scenarios.

[0117] Optionally, as an embodiment of the present invention, the process of performing a temporal attention mechanism calculation on each of the spatial channel joint axis features after the temporal convolution through the temporal attention layer to obtain the original temporal joint axis features corresponding to each of the normalized joint axis data includes:

[0118] The fifth formula is used to calculate the temporal attention mechanism for each of the spatial channel joint axis features after the temporal convolution, so as to obtain the original temporal joint axis features corresponding to each of the normalized joint axis data. The fifth formula is:

[0119]

[0120] in, is the original time joint axis feature corresponding to the mth joint axis data to be trained, is the softmax activation function, l 1×1 is 1×1 convolution processing, AvgPool is average pooling processing, It is the spatial channel joint axis feature after temporal convolution corresponding to the mth joint axis data to be trained.

[0121] It should be understood that T-Attn is composed of a temporal attention mechanism, whose structure is similar to the spatial attention mechanism, except that the temporal attention is focused on the input feature f T ∈R B×C×T The time series features obtained by average pooling are then subjected to a 1×1 convolution operation, and finally a softmax activation function is used for nonlinear changes. The formula can be expressed as:

[0122]

[0123] In the above embodiment, the temporal attention mechanism is performed on the spatial channel joint axis features after temporal convolution through the temporal attention layer to obtain the original temporal joint axis features, thereby achieving rapid positioning and detection of abnormal movements, and also achieving higher response speed and detection accuracy, effectively reducing the risk of injury, and being able to meet the accuracy and time performance requirements of industrial scenarios.

[0124] Optionally, as another embodiment of the present invention, the rapid detection and response of abnormal actions of industrial robots can effectively reduce the risk of injury. Abnormal action detection requires modeling and detection of complex and changeable action patterns of industrial robots, which requires consideration of the spatiotemporal coupling and dynamic characteristics of the actions of industrial robots, and the efficient processing of spatiotemporal features and complex dynamic models requires a large amount of computing resources, which limits the accuracy and response speed of detection. To this end, the present invention proposes a variable spatiotemporal graph normalized flow (VSTG-NF) model for abnormal action detection of industrial robots. The model uses a variable spatiotemporal convolutional network (VSTGC) to capture and identify the complex action relationships and motion patterns of industrial robot actions in time and space dimensions. The frame-level action sequence is placed in a high-dimensional data space and a low-dimensional latent space through a normalized flow for bidirectional mapping, and the action is evaluated for anomaly scores based on the mapped data distribution to achieve the location and detection of anomalies. The model in this paper requires very little computational overhead (about 2.0K parameters) to achieve a high response speed and detection accuracy. Analysis experiments conducted on a self-made six-axis industrial robot action dataset (IRAD) show that the AUC score of frame-level detection is 92.18%, and the inference speed reaches 75FPS, which meets the accuracy and time performance requirements of industrial scenarios.

[0125] Optionally, as another embodiment of the present invention, the present invention proposes a variable spatiotemporal graph normalized flow (VSTG-NF) model for abnormal motion detection of a six-axis industrial robot. It uses highly semantic low-dimensional joint axis data obtained by an industrial robot posture estimation detector to construct an industrial robot spatiotemporal graph with highly coupled time and space relationships. In order to break through the limitations of predefined industrial robot graph structure modeling and improve the distinguishability of joint axis features, the present invention uses a variable spatiotemporal convolutional network (VSTGC) to realize the dynamic modeling of the industrial robot spatiotemporal graph structure with different action modes. At the same time, in order to alleviate the problem that the number of joint axes of the six-axis industrial robot limits the obvious distinction of different actions, the present invention also adds a spatial channel attention mechanism (SC-Attn) and a time attention mechanism (T-Attn) to the variable spatial graph convolution module and the temporal graph convolution module in the VSTGC. In this way, the complex relationship and motion pattern of the industrial robot action in the time and space dimensions are captured and distinguished, which enables the model of the present invention to more accurately distinguish abnormal behavior. In order to ensure low latency of detection, the present invention places the frame-level action sequence in the high-dimensional data space and the low-dimensional latent space for bidirectional mapping through the normalized flow, and evaluates the abnormal score of the behavior according to the mapped data distribution. This makes the model of the present invention not require a complex network structure and a large amount of computing resources. On the contrary, the model of the present invention is very lightweight (about 2.0K parameters), which ensures a higher detection speed of the model. It should be noted that the model parameter size of the posture estimation detector used for preprocessing of the present invention is 9.6M, and the reasoning speed based on this is 75FPS. However, the model of the present invention is not limited to a specific posture estimation detector, and the detection accuracy and reasoning speed will benefit from the improvement of the posture estimation detector.

[0126] Optionally, as another embodiment of the present invention, the method of the present invention is generally summarized as follows: given a six-axis industrial robot motion video sequence, the YOLOF industrial robot posture estimation detector is used to extract the industrial robot joint axis sequence data, and the joint axis sequence data is modeled as an industrial robot spatiotemporal graph. The present invention inputs the joint axis sequence data into the variable spatiotemporal graph normalization flow (VSTG-NF) model. In VSTG-NF, the joint axis sequence data is first scaled and normalized and inversely permuted and sorted by the activation normalization layer and the data permutation layer, and then the stacked spatiotemporal coupling layer is used to embed the complex motion data distribution of the industrial robot into the Gaussian distribution space, and bidirectional mapping is performed through the affine coupling transformation to estimate the logarithmic likelihood of each frame of action. The present invention uses a variable spatiotemporal convolutional network (VSTGC) as a conditional network (coupling layer) of the affine transformation. VSTGC takes the industrial robot spatiotemporal graph as input, and extracts the high-level semantic information of the motion mode of the industrial robot through variable space graph convolution and time graph convolution operations. Finally, the abnormal score of each frame of action is calculated by minimizing the logarithmic likelihood.

[0127] In the evaluation phase, the joint axis sequence data containing abnormal motion is input into the variable spatiotemporal graph normalized flow model. The abnormal motion features will be distributed in the area with low probability in the Gaussian distribution space, and the abnormal score obtained by the reverse transformation will be significantly lower than the abnormal score of the normal motion. Then, the abnormal motion of the industrial robot is judged according to the optimal threshold of the best area under the curve (AUC) obtained by training.

[0128] Optionally, as another embodiment of the present invention, in the spatial dimension, the present invention follows the partitioning strategy of STGCN to partition the vertices v of the spatiotemporal graph of the industrial robot i Divide the neighborhood into subsets v j , G represents the center of gravity of the single-frame industrial robot joint axis sequence, and the target node v i The neighbor subset v j1 and v j2 They represent centripetal nodes and centrifugal nodes respectively. The basis for distinguishing them is the distance from the center of gravity. That is, the nodes close to the center of gravity are the centripetal subset, and the nodes far from the center of gravity are the centrifugal subset. At the same time, through the mapping function m S (v j ,v i ) so that each neighbor subset v j With vertex v i All have a unique correspondence. Therefore, the spatial graph convolution can be expressed as:

[0129]

[0130] where f in is the input feature map, B i ∈{v j |d(v j ,v i )≤1} represents the vertex v i To the neighbor subset v j The set of subsets with the minimum distance for any path. Z ij =|{v g |m S (v g ,v i )=m S (v j ,v i )}| is the vertex v i The neighbor subset v j The cardinality of is used as a normalization factor to balance the impact of different nodes on the results during the spatial graph convolution process. S is the spatial domain weight function, through the spatial domain mapping function m S To determine a unique weight vector for each vertex.

[0131] Optionally, as another embodiment of the present invention, the unsupervised abnormal action detection model of the present invention will only learn the data distribution of normal action samples of industrial robots during training. Since the abnormal actions of industrial robots have different feature distributions from normal actions, when the model evaluates samples with abnormal actions, it will be assigned a lower probability density score, that is, the likelihood of the model calculating abnormal actions is low, indicating that the action may not belong to the normal action data distribution during training. Therefore, in the reasoning process, the present invention uses the model to calculate the probability density score of the distribution of each frame of action in the real data space, and minimizes the negative log-likelihood as the abnormal score score of each frame to distinguish normal actions from abnormal actions. It can be formally expressed as:

[0132]

[0133] Where η represents the threshold for distinguishing normal from abnormal actions, which is obtained by analyzing the data distribution of normal action samples during training.

[0134] Optionally, as another embodiment of the present invention, the present invention proposes a semi-supervised abnormal motion detection method for a six-axis industrial robot, which is based on the spatiotemporal graph representation of the joint axis sequence of the industrial robot. The present invention designs a new variable spatiotemporal graph normalization flow (VSTG-NF) architecture for processing industrial robot spatiotemporal graph data. And the proposed variable spatiotemporal convolutional network (VSTGC) is used to capture and distinguish the complex relationships and motion patterns of industrial robot movements in time and space dimensions, so as to improve the response speed and accuracy of abnormal motion detection. Experiments have shown that this method presents good abnormal motion detection capabilities, and the abnormal motion detection response speed and accuracy can reach the expected values.

[0135] Figure 2 A module block diagram of a robot abnormal motion detection device provided in an embodiment of the present invention.

[0136] Optionally, as another embodiment of the present invention, Figure 2 As shown, a robot abnormal motion detection device comprises:

[0137] Import module, used to import multiple original joint axis sequence data at multiple moments;

[0138] A building module for constructing a spatiotemporal graph from all raw joint axis sequence data;

[0139] A model analysis module is used to construct a training model, and perform model analysis on the training model through the space-time graph to obtain a detection model;

[0140] The import module is also used to import the joint axis sequence data to be detected;

[0141] The detection result acquisition module is used to detect the joint axis sequence data to be detected through the detection model to obtain the detection result of the abnormal movement of the robot.

[0142] Optionally, another embodiment of the present invention provides a robot abnormal motion detection system, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the robot abnormal motion detection method described above is implemented. The system may be a computer or other system.

[0143] Optionally, another embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the robot abnormal motion detection method as described above is implemented.

[0144] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0145] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0146] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0147] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the embodiments of the present invention.

[0148] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0149] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0150] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting abnormal motion of a robot, characterized in that: The steps include: Import multiple original joint axis sequence data at multiple moments, and construct a spatiotemporal graph through all the original joint axis sequence data; Constructing a training model, and performing model analysis on the training model through the space-time graph to obtain a detection model; The joint axis sequence data to be detected is imported, and the joint axis sequence data to be detected is detected by the detection model to obtain the detection result of the abnormal movement of the robot.

2. The method for detecting abnormal robot motion according to claim 1, characterized in that: The process of constructing a spatiotemporal graph using all original joint axis sequence data includes: Respectively connecting each original joint axis sequence data at each moment and the original joint axis sequence data adjacent to each original joint axis sequence data at each moment to obtain spatial edges corresponding to each original joint axis sequence data at each moment, and collecting all the spatial edges to obtain a spatial edge set; Respectively connect each original joint axis sequence data at each moment and each original joint axis sequence data at a time adjacent to each moment to obtain a time edge corresponding to each original joint axis sequence data at each moment, and collect all the time edges to obtain a time edge set, and collect all the original joint axis sequence data at all moments to obtain an original joint axis sequence data set; The space edge set, the time edge set and the original joint axis sequence data set are constructed by the first formula to obtain a space-time graph, and the first formula is: G=(V,E), Among them, E∈(E Spatial ,E Temporal ), Among them, G is the spatiotemporal graph, V is the original joint axis sequence data set, E is the target edge set, and E Spatial is the set of spatial edges, E Temporal is the set of time edges.

3. The method for detecting abnormal motion of a robot according to claim 1, characterized in that: The training model includes an activation normalization layer, a data permutation layer, and a stacked spatiotemporal coupling layer. The process of constructing a training model and performing model analysis on the training model through the spatiotemporal graph to obtain a detection model includes: Normalizing the spatiotemporal graph through the activation normalization layer to obtain a plurality of normalized joint axis data; Performing replacement processing on each of the normalized joint axis data respectively through the data replacement layer to obtain replaced joint axis data corresponding to each of the normalized joint axis data; The spatiotemporal graph and each of the replaced joint axis data are scored and analyzed respectively by the stacked spatiotemporal coupling layer to obtain a predicted action sequence score corresponding to each of the normalized joint axis data; Importing the real action sequence scores corresponding to each of the normalized joint axis data, and performing loss function calculation on all the predicted action sequence scores and all the real action sequence scores using a negative log-likelihood minimization algorithm to obtain a loss function; The parameters of the training model are updated according to the loss function to obtain a detection model.

4. The method for detecting abnormal robot motion according to claim 3, characterized in that: The stacked spatiotemporal coupling layer includes a plurality of sequentially arranged variable spatiotemporal convolutional networks, The process of performing score analysis on the spatiotemporal graph and each of the replaced joint axis data respectively through the stacked spatiotemporal coupling layer to obtain a predicted action sequence score corresponding to each of the normalized joint axis data includes: Performing affine transformation analysis on the spatiotemporal graph and each of the displaced joint axis data respectively through the first variable spatiotemporal convolutional network, and inputting the analysis results into the next variable spatiotemporal convolutional network, until the last variable spatiotemporal convolutional network is passed to obtain affine transformed joint axis features corresponding to each of the normalized joint axis data; Respectively splicing each of the first segmented joint axis features with the affine transformed joint axis features corresponding to each of the normalized joint axis data to obtain a spliced ​​joint axis feature corresponding to each of the normalized joint axis data; Performing Gaussian spatial distribution mapping processing on each of the spliced ​​joint axis features respectively to obtain mapped joint axis features corresponding to each of the normalized joint axis data; Respectively performing inverse mapping processing on each of the mapped joint axis features to obtain a predicted action sequence score corresponding to each of the normalized joint axis data; The first variable spatiotemporal convolutional network is used to analyze the affine transformation of the spatiotemporal graph and each of the displaced joint axis data, specifically: Segmenting each of the replaced joint axis data according to a preset channel dimension to obtain a first segmented joint axis feature corresponding to each of the normalized joint axis data and a second segmented joint axis feature corresponding to each of the normalized joint axis data; Performing affine transformation processing on the spatiotemporal graph and each of the first segmented joint axis features respectively through a first variable spatiotemporal convolutional network to obtain the joint axis features to be segmented corresponding to each of the normalized joint axis data; Segmenting each of the to-be-segmented joint axis features respectively to obtain a scale offset value corresponding to each of the normalized joint axis data and a scaling value corresponding to each of the normalized joint axis data; The second formula is used to fuse each of the second segmented joint axis features, the scale offset value corresponding to each of the normalized joint axis data, and the scaling value corresponding to each of the normalized joint axis data, to obtain the affine transformed joint axis features corresponding to each of the normalized joint axis data. The second formula is: in, is the joint axis feature after affine transformation corresponding to the mth joint axis data to be trained, is the scaling value corresponding to the mth joint axis data to be trained, is the joint axis feature after the second segmentation corresponding to the mth joint axis data to be trained, is the scale offset value corresponding to the mth joint axis data to be trained.

5. The method for detecting abnormal motion of a robot according to claim 4, characterized in that: The variable spatiotemporal convolutional network includes a variable spatial graph convolution layer, a spatial channel attention layer, a first batch normalization layer, a first Relu activation function layer, a random dropout layer, a temporal graph convolution layer, a temporal attention layer, a second batch normalization layer, and a second Relu activation function layer. The process of performing affine transformation processing on the spatiotemporal graph and each of the first segmented joint axis features respectively through the first variable spatiotemporal convolutional network to obtain the joint axis features to be segmented corresponding to each of the normalized joint axis data includes: Performing spatial convolution processing on the spatiotemporal graph and each of the first segmented joint axis features respectively through the variable spatial graph convolution layer to obtain spatially convolved joint axis features corresponding to each of the normalized joint axis data; Performing spatial channel attention mechanism calculation on each of the spatial convolution joint axis features through the spatial channel attention layer to obtain original spatial channel joint axis features corresponding to each of the normalized joint axis data; Normalizing each of the original spatial channel joint axis features respectively through the first batch normalization layer to obtain normalized spatial channel joint axis features corresponding to each of the normalized joint axis data; Mapping each of the normalized spatial channel joint axis features through the first Relu activation function layer to obtain mapped spatial channel joint axis features corresponding to each of the normalized joint axis data; Extracting features of the mapped spatial channel joint axis features respectively through the random inactivation layer to obtain extracted spatial channel joint axis features corresponding to the normalized joint axis data; Performing temporal convolution processing on each of the extracted spatial channel joint axis features through the temporal graph convolution layer to obtain temporal convolutional spatial channel joint axis features corresponding to each of the normalized joint axis data; The temporal attention layer performs temporal attention mechanism calculation on each of the temporal convolution spatial channel joint axis features to obtain original temporal joint axis features corresponding to each of the normalized joint axis data; Normalizing each of the original temporal joint axis features respectively through the second batch normalization layer to obtain normalized temporal joint axis features corresponding to each of the normalized joint axis data; Each of the normalized temporal joint axis features is mapped through the second Relu activation function layer to obtain the joint axis features to be segmented corresponding to each of the normalized joint axis data.

6. The method for detecting abnormal robot motion according to claim 5, characterized in that: The spatiotemporal graph also includes an original adjacency matrix and a plurality of graph nodes. The process of performing spatial convolution processing on the spatiotemporal graph and each of the first segmented joint axis features respectively through the variable spatial graph convolution layer to obtain the spatially convolved joint axis features corresponding to each of the normalized joint axis data includes: The original adjacency matrix, all graph nodes, and each of the first segmented joint axis features are spatially convolved by the third formula to obtain the spatially convolved joint axis features corresponding to each of the normalized joint axis data. The third formula is: in, is the joint axis feature after spatial convolution corresponding to the mth joint axis data to be trained, K S is the convolution kernel size in the spatial domain, w S is the time domain weight function, is the joint axis feature after the first segmentation corresponding to the mth joint axis data to be trained, δ k For the k The weight function of the convolution kernel, Λ k For the k The preset normalized diagonal matrix of convolution kernels, G k is the original adjacency matrix of the kth convolution kernel, G′ k is the target adjacency matrix of the kth convolution kernel, Q k (v i ,v j ) is the similarity between the i-th graph node and the j-th graph node, F is the residual connection function, α is a learnable parameter, φ k and are Gaussian functions of the kth convolution kernel, and N is the total number of graph nodes.

7. The method for detecting abnormal robot motion according to claim 5, characterized in that: The process of performing spatial channel attention mechanism calculation on each of the spatial convolution joint axis features through the spatial channel attention layer to obtain the original spatial channel joint axis features corresponding to each of the normalized joint axis data includes: The fourth formula is used to perform spatial channel attention calculation on each of the joint axis features after spatial convolution, so as to obtain the original spatial channel joint axis features corresponding to each of the normalized joint axis data. The fourth formula is: in, in, is the original space channel joint axis feature corresponding to the mth joint axis data to be trained, is the channel attention weight corresponding to the mth joint axis data to be trained, is the spatial attention weight corresponding to the mth joint axis data to be trained, ξ is the Sigmoid activation function, l 1×1 is 1×1 convolution processing, AvgPool is average pooling processing, is the joint axis feature after spatial convolution corresponding to the mth joint axis data to be trained, fc1c is the fully connected layer function, and γ is the Relu activation function.

8. The method for detecting abnormal motion of a robot according to claim 5, characterized in that: The process of performing temporal attention mechanism calculation on each of the spatial channel joint axis features after the temporal convolution through the temporal attention layer to obtain the original temporal joint axis features corresponding to each of the normalized joint axis data includes: The fifth formula is used to calculate the temporal attention mechanism for each of the spatial channel joint axis features after the temporal convolution, so as to obtain the original temporal joint axis features corresponding to each of the normalized joint axis data. The fifth formula is: in, is the original time joint axis feature corresponding to the mth joint axis data to be trained, is the softmax activation function, l 1×1 is 1×1 convolution processing, AvgPool is average pooling processing, It is the spatial channel joint axis feature after temporal convolution corresponding to the mth joint axis data to be trained.

9. A robot abnormal motion detection device, characterized in that: include: Import module, used to import multiple original joint axis sequence data at multiple moments; A building module for constructing a spatiotemporal graph from all raw joint axis sequence data; A model analysis module is used to construct a training model, and perform model analysis on the training model through the space-time graph to obtain a detection model; The import module is also used to import the joint axis sequence data to be detected; The detection result acquisition module is used to detect the joint axis sequence data to be detected through the detection model to obtain the detection result of the abnormal movement of the robot.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the robot abnormal motion detection method according to any one of claims 1 to 8 is implemented.

Images