Foot type robot fault detection method based on space-time diagram convolutional network

Abstract

The invention relates to a foot-type robot fault detection method based on a space-time diagram convolution network, and the method employs a deep learning model based on the space-time diagram convolution network, and the deep learning model comprises a time sequence gating convolution module and a space diagram convolution module. The method comprises the steps that a joint diagram is constructed according to the joint connection relation of the robot; performing long sequence feature extraction on the joint diagram by using a time sequence gating convolution module; the features obtained by the time sequence gating convolution module serve as input of a space graph convolution module, a space graph convolution layer conducts convolution operation on the node features through an adjacent matrix, and the space dependence relation between the nodes is captured; and feature extraction is carried out multiple times through a time sequence gating convolution gating module and a space graph convolution gating module, and a foot type robot fault detection result in the operation process is obtained. According to the invention, the accuracy of fault detection of the foot-type robot and the generalization performance of the algorithm can be improved.

Description

Technical Field

[0001] The present invention relates to a legged robot fault detection method based on a spatiotemporal graph convolutional network, and belongs to the field of legged robots. Background Art

[0002] As biomimetic robots, legged robots have broad application prospects in rescue, service, and reconnaissance due to their superior mobility and structural characteristics. However, when operating in complex environments, legged robots may encounter various faults, such as joint failure, sensor failure, motor failure, and battery failure. Fault detection technology is required to detect and diagnose these faults and take appropriate measures to repair or adjust them to ensure the robot's continued stable operation. This is particularly important for robots performing tasks in complex environments, such as post-disaster rescue and hazardous area detection. Predictive maintenance can be used to perform preventive maintenance before failures occur, further reducing maintenance costs. Furthermore, timely detection and resolution of faults during mission execution can improve mission success rates and ensure that robots complete their missions as planned. The research and application of fault detection technology can promote the development of robotics, enhance the intelligence of robots, and enable them to better adapt to complex and changing environments and complete a wider range of tasks.

[0003] Although legged robot fault detection plays a critical role in ensuring the continued stability of robotic operations, this field faces a number of challenges. First, the primary difficulty in legged robot fault detection lies in the high complexity and dynamic nature of the robot system. Because robots consist of multiple joints and actuators, these components interact in complex ways. During operation, a variety of fault types may occur, such as joint wear, sensor failure, and powertrain issues. However, due to the lack of effective multi-scale feature extraction techniques, these features are often overlooked by traditional diagnostic models, limiting their generalization capabilities. Second, the potential for multiple fault types and varying degrees of damage leads to differences in the distribution of fault features at different scales, and the dependencies of these distributions are difficult to capture using traditional networks. Furthermore, because faults are often progressive and multidimensional, potentially manifesting as subtle changes in motion behavior, traditional fault detection methods struggle to accurately capture and diagnose faults. Traditional methods are limited in extracting features at a single event scale, making it difficult to effectively extract deep features. Finally, the motion state of legged robots varies in different environments, making it difficult for models to achieve strong generalization capabilities. Deep learning, however, boasts powerful feature learning and generalization capabilities, can process large amounts of data, and can gradually learn complex fault patterns from low-level to high-level layers through a multi-layered architecture. In recent years, deep learning-based fault diagnosis methods have been proposed to overcome these difficulties. However, improving the accuracy of legged robot fault detection under complex conditions and the generalization of the model remains a significant research topic. Summary of the Invention

[0004] The present invention provides a legged robot fault detection method based on a spatiotemporal graph convolutional network, aiming to solve at least one of the technical problems existing in the prior art.

[0005] The technical solution of the present invention relates to a legged robot fault detection method based on a spatiotemporal graph convolutional network. The method according to the present invention comprises the following steps:

[0006] Among them, a deep learning model based on a spatiotemporal graph convolutional network is adopted, and the deep learning model includes a temporal gated convolution module and a spatial graph convolution module;

[0007] The method comprises the following steps:

[0008] S100, constructing a joint graph according to the joint connection relationship of the robot;

[0009] S200, using the temporal gated convolution module to perform long sequence feature extraction on the joint graph;

[0010] S300, using the features obtained by the sequential gated convolution module as the input of the spatial graph convolution module, and the spatial graph convolution layer performs a convolution operation on the node features through the adjacency matrix to capture the spatial dependency relationship between nodes;

[0011] S400 , performing feature extraction multiple times through the temporal gated convolution module and the spatial graph convolution module to obtain a fault detection result of the legged robot during operation.

[0012] Furthermore, in step S100:

[0013] The nodes of the joint graph represent the joints of the robot, and the edges of the joint graph represent the connection relationships between the joints. An adjacency matrix is ​​used to represent the graph structure, and the elements in the adjacency matrix represent the connection relationships between the nodes.

[0014] Furthermore, in step S200,

[0015] Each temporal convolution layer of the temporal gated convolution module includes two convolution modules, one of which is used to extract deep features and the other is a gated convolution layer for generating a gating signal; the gated convolution layer updates the features of the node by merging the features of the corresponding node on consecutive time slices.

[0016] Furthermore, in step S200,

[0017] Each of the gated convolutional layers includes a gated linear unit, which includes two one-dimensional convolutional layers; the gated convolutional layer is represented as follows:

[0018] y gated =(K1*x gated +b1)⊙σ(K2*x gated +b2)

[0019] Where K1 and K2 represent convolution kernels, b1 and b2 are biases, * represents convolution operation, ⊙ represents element-by-element multiplication operation, σ is the sigmoid function, and x gated and y gated are the input and output of the gated convolutional network, respectively.

[0020] Furthermore, in step S300,

[0021] In the graph convolutional neural network of the spatial graph convolution module, a graph is defined as a network consisting of nodes and edges, where nodes represent entities and edges represent relationships between nodes.

[0022] Furthermore, in step S300,

[0023] The graph convolutional neural network includes a multi-layer graph convolution layer with a hierarchical stacking structure, which is used to normalize the adjacency matrix and update the representation of the target node by weighting the features of the neighboring nodes, wherein the weighting coefficient is jointly determined by the adjacency matrix and the degree matrix.

[0024] Furthermore, the step S300 includes:

[0025] S310: For each node in the joint graph, the graph convolutional neural network aggregates the features of the node itself and the adjacent nodes to obtain a new node, which is represented as follows:

[0026] Z l + 1 =A·H (l) W (l)

[0027] Where Z l+1 It is a new node representation. The node set to be updated is represented by the adjacency matrix A and the node feature H (l) The product of W (l) is the weight matrix of the lth layer;

[0028] S320: Aggregation is achieved by weighted summing of features of neighboring nodes, where the weights are determined by the adjacency matrix and degree matrix of the graph, which are expressed as follows:

[0029]

[0030] Where, is the new node representation after normalization, and D is the degree matrix;

[0031] S330: Perform nonlinear changes on the updated node representation, which is expressed as follows:

[0032]

[0033] Where H (l+1) It is the node representation after the graph convolution operation, and ReLU(*) represents the activation function.

[0034] Furthermore, the deep learning model based on the spatiotemporal graph convolutional network includes a global average pooling layer, a fully connected layer and two STGCN modules; the output of one of the STGCN modules serves as the input of the other STGCN module;

[0035] Among them, each STGCN module first captures the spatiotemporal coupling characteristics of the joints through the temporal gated convolution module, then captures the spatial dependency between nodes through the spatial graph convolution module, and then extracts the temporal features again through another temporal gated convolution module, and finally obtains the output of the STGCN module through the batch normalization layer;

[0036] Among them, the obtained initial features pass through the two STGCN modules, then pass through the global average pooling layer, and finally pass through the fully connected layer to obtain the output of the deep learning model.

[0037] The technical solution of the present invention also relates to a computer-readable storage medium having program instructions stored thereon, and the above-mentioned method is implemented when the program instructions are executed by a processor.

[0038] The technical solution of the present invention also relates to a legged robot fault detection system based on a spatiotemporal graph convolutional network, wherein the system includes a computer device containing the above-mentioned computer-readable storage medium.

[0039] The beneficial effects of the present invention are as follows:

[0040] The present invention's legged robot fault detection method, based on a spatiotemporal graph convolutional network, can improve the accuracy of legged robot fault detection and the generalization performance of the algorithm. This method maps the joint features of the legged robot and uses a graph convolutional network for feature extraction and fault detection. A temporal gated convolution module learns the temporal characteristics of sensor signals to capture the spatiotemporal coupling characteristics of the joints. A spatial graph convolutional layer convolves node features using an adjacency matrix to capture the spatial dependencies between nodes, enabling the model to effectively detect faults during robot operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a basic flow chart of the legged robot fault detection method according to the present invention and a structural schematic diagram of the deep learning model.

[0042] Figure 2 Schematic diagram of the joints of the biped robot according to the present invention.

[0043] Figure 3 Schematic diagram of the joints of the quadruped robot according to the present invention.

[0044] Figure 4 and Figure 5 4 is a schematic diagram of a confusion matrix result for verifying the accuracy of fault diagnosis according to an embodiment of the present invention.

[0045] Figure 6a and Figure 6b as well as Figure 7a and Figure 7b2 is a comparison chart of detection results according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The following will provide a clear and complete description of the concept, specific structure and technical effects of the present invention in conjunction with the embodiments and drawings to fully understand the purpose, scheme and effects of the present invention.

[0047] It should be noted that, unless otherwise specified, when a feature is referred to as being "fixed" or "connected" to another feature, it may be directly fixed or connected to the other feature, or it may be indirectly fixed or connected to the other feature. The singular forms "a", "said" and "the" used herein are also intended to include the plural forms, unless the context clearly indicates otherwise. In addition, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art. The terms used in this specification are only for describing specific embodiments and are not intended to limit the invention. The term "and / or" used herein includes any combination of one or more related listed items.

[0048] Should be understood that, although the present disclosure may adopt the term first, second, third etc. to describe various elements, these elements should not be limited to these terms.These terms are only used to distinguish the elements of the same type from each other.For example, without departing from the scope of the present disclosure, the first element may also be referred to as the second element, and similarly, the second element may also be referred to as the first element.The use of any and all examples or exemplary language ("for example", "such as" etc.) provided herein is only intended to better illustrate embodiments of the present invention, and unless otherwise required, will not impose limitations on the scope of the present invention.

[0049] Reference Figures 1 to 3 In some embodiments, the legged robot fault detection method based on the spatiotemporal graph convolutional network of the present invention adopts a deep learning model based on the spatiotemporal graph convolutional network, wherein the deep learning model includes a temporal gated convolution module and a spatial graph convolution module; the method includes at least the following steps:

[0050] S100, constructing a joint graph according to the joint connection relationship of the robot;

[0051] S200, use the temporal gated convolution module to extract long sequence features from the joint graph;

[0052] S300: The features obtained by the temporal gated convolution module are used as the input of the spatial graph convolution module. The spatial graph convolution layer performs a convolution operation on the node features through the adjacency matrix to capture the spatial dependency between nodes.

[0053] S400, performing feature extraction multiple times through the temporal gated convolution module and the spatial graph convolution module to obtain the legged robot fault detection result during operation.

[0054] The present invention's fault detection method for legged robots, based on a spatiotemporal graph convolutional network, addresses various faults that legged robots may encounter when operating in complex environments, such as joint failures, sensor failures, motor failures, and battery failures. These failures not only affect the robot's normal operation but can also lead to mission failures and even safety issues. By employing an end-to-end spatiotemporal graph convolutional network and training the model on a large dataset, the present invention achieves accurate diagnosis of legged robot faults under complex conditions.

[0055] The present invention's legged robot fault detection method based on spatiotemporal graph convolutional network is used to improve the accuracy of legged robot fault detection, see Figure 1 , the method of the present invention is based on a legged robot fault detection system, and the system includes a robot joint graph construction module, a temporal gated convolution module (TGCM) and a spatial graph convolution module (SGCM). The present invention obtains the optimal model parameter value by training the model, and obtains a deep learning model based on a spatiotemporal graph convolution network that can achieve high accuracy in legged robot fault detection; and uses the optimal deep learning model obtained through training to conduct a legged robot fault detection experiment. The present invention adopts a legged robot fault detection algorithm based on a spatiotemporal graph convolution network. By mapping the joint features of the legged robot and using a graph convolution network for feature extraction and fault detection, its network model can effectively detect faults during the operation of the robot. The effectiveness of the proposed temporal gated convolution module and spatial graph convolution module has been verified through extensive experimental results and analysis, proving the advantages of the proposed method in fault diagnosis accuracy and adaptability.

[0056] In some embodiments, see Figure 1 The deep learning model based on the spatiotemporal graph convolutional network (STGCN) of an embodiment of the present invention is composed of a plurality of STGCN modules stacked together, each STGCN module includes a temporal gated convolution module (TGCM) and a spatial graph convolution module (SGCM), wherein feature extraction is achieved by alternating the combination of the temporal gated convolution module (TGCM) and the spatial graph convolution module (SGCM).

[0057] Specifically, the deep learning model includes two STGCN modules. Each STGCN module first captures the spatiotemporal coupling characteristics of the joints through TGCM, then captures the spatial dependencies between nodes through SGCM, and then extracts the temporal features through another TGCM again. Finally, the output of the STGCN module is obtained through a batch normalization layer (BN). The output of one STGCN module serves as the input of another STGCN module. Furthermore, the deep learning model of the present invention also includes a global average pooling layer (GAP) and a fully connected layer (FC). The initial features obtained are successively passed through the two STGCN modules, input into the GAP module, and finally output the model results through FC.

[0058] In some embodiments, the present invention first constructs a joint graph when constructing a deep learning model. Specifically, in order to capture the spatial dependencies between the joints of a legged robot, the present invention constructs a graph structure based on the joint connection relationships of the robot, wherein the nodes of the joint graph represent the joints of the robot, and the edges of the joint graph represent the connection relationships between the joints. The graph structure is represented by an adjacency matrix, and the elements in the adjacency matrix represent the connection relationships between the nodes. Figure 2 and Figure 3 In the simulation system of the specific embodiment of the present invention, the biped robot has a total of 33 joints and the quadruped robot has a total of 18 joints. The joint pairs with a value of 1 in the adjacency matrix indicate that the joints are connected to each other, and the adjacency matrix value corresponding to each joint itself is 1.

[0059] In some embodiments, after the joint graph is constructed, time series features are extracted from the constructed robot joint graph. Specifically, in order to extract sequence features of the robot sensor time series signals, the present invention uses a temporal gated convolution module (TGCM) to implement long sequence feature extraction.

[0060] Specifically, the present invention inputs robot joint graph data into a sequential gated convolutional module. Using convolution operations within the network, the flow of information is controlled by gating signals. This enables the sequential gated convolutional module to capture the spatiotemporal coupling of joint features, comprehensively understand joint state changes, and thus better identify complex fault modes. Each sequential convolutional layer contains two convolutional modules: one for extracting deep features and the other for generating gating signals. These are gated convolutional layers.

[0061] In one embodiment, the present invention employs stacked gated convolutional layers to update node features by merging the features of corresponding nodes over consecutive time slices. Each gated convolutional layer contains a gated linear unit, which is implemented by two conventional one-dimensional convolutional layers. Specifically, the mathematical model of the gated convolutional layer is expressed as follows:

[0062] ygated =(K1-x gated +b1)⊙σ(K2*x gated +b2)

[0063] Where K1 and K2 represent convolution kernels, b1 and b2 are biases, * represents convolution operation, ⊙ represents element-by-element multiplication operation, σ is the sigmoid function, and x gated and y gated are the input and output of the gated convolutional network, respectively.

[0064] In some embodiments, the features obtained by the temporal gated convolution module are used as the input of the spatial graph convolution module (SGCM), and the spatial graph convolution layer performs a convolution operation on the node features through the adjacency matrix to capture the spatial dependencies between the nodes. Specifically, the spatial graph convolution module of the present invention can learn the correlation between joint features, including the correlation of position, angle, speed, etc., through graph convolution. The spatial graph convolution module enables the robot fault detection task to adaptively learn the association between each joint and the fault state, thereby optimizing the accuracy of fault detection, and then extracting deeper features through multiple times of temporal gated convolution gating and spatial graph convolution modules.

[0065] In one application embodiment, the graph convolutional neural network (GCN) used in the present invention is used to process and analyze graph structured data. In the GCN of the present invention, the joint graph is defined as a network composed of nodes and edges, where nodes represent entities and edges represent the relationship between nodes. Unlike traditional fully connected neural networks, the GCN of the present invention makes full use of the structural information of the graph so that the representation of each node not only depends on its own characteristics, but is also closely related to the information of its neighboring nodes. This mechanism of the present invention enables the GCN to capture local and global patterns in the graph during training, thereby having stronger adaptability when processing complex system data.

[0066] Furthermore, the GCN of the present invention is composed of multiple graph convolutional layers (GCL), each of which has independent weight parameters. Through multi-layer combinations, the graph convolutional neural network can gradually learn more abstract and advanced feature representations. Among them, the hierarchical stacking structure enables GCN to perform deep learning on the graph structure, thereby capturing complex graph relationships. The core operation of the graph convolution layer is to normalize the adjacency matrix and update the representation of the target node by weighting the features of the neighboring nodes, where the weighting coefficient is jointly determined by the adjacency matrix and the degree matrix. This design of the present invention ensures that GCN can effectively propagate and learn the structural information in the graph while retaining the original features of the node.

[0067] Specifically, for each node in the joint graph, the graph convolutional neural network aggregates the features of its own node and adjacent nodes to obtain a new node representation:

[0068] Z l+1 =A·H (l) W (l)

[0069] Where Z l+1 It is a new node representation. The node set to be updated is represented by the adjacency matrix A and the node feature H (l) The product of W (l) is the weight matrix of layer l.

[0070] Furthermore, the aggregation method is implemented by weighted summing of the features of neighboring nodes, where the weights are determined by the adjacency matrix and degree matrix of the graph. This weighted strategy of the present invention can reflect the strength of the relationship between nodes and the topological structure of the graph.

[0071] Where, is the new node representation after normalization, and D is the degree matrix.

[0072] Furthermore, nonlinear changes are performed on the updated node representations, and nonlinearity is introduced using activation functions such as rectified linear units (ReLU).

[0073]

[0074] Where H (l+1) It is the node representation after the graph convolution operation, and ReLU(*) represents the activation function.

[0075] In some embodiments, preprocessed legged robot fault training data is used as model input. During model training, the temporal gated convolution module and the spatial graph convolution module are improved to obtain optimal model parameters, ultimately resulting in a deep learning model capable of accurately diagnosing robot faults. The Adam optimization algorithm is applied to this model training optimization task, replacing the traditional stochastic gradient descent method. Finally, the trained deep learning model is used to test the legged robot fault detection accuracy.

[0076] Specifically, when constructing a deep learning model for legged robot fault detection, the present invention first inputs the preprocessed robot fault training dataset into the model. During the model training phase, the temporal gated convolution module and the spatial graph convolution module are optimized to obtain the optimal model parameter configuration, thereby achieving accurate diagnosis of legged robot faults.

[0077] The legged robot fault detection algorithm based on the spatiotemporal graph convolutional network of the present invention maps the joint features of the legged robot and uses the graph convolutional network for feature extraction and fault detection. The temporal gated convolution module learns the temporal features of sensor signals, and the spatial graph convolution layer performs convolution operations on node features through the adjacency matrix to capture the spatial dependencies between nodes. The model can effectively detect faults during the operation of the robot.

[0078] In some embodiments, the training set preprocessing method of the present invention uses legged robot fault data generated through PyBullet simulation. Legged robot status types include normal state, joint failure, sensor failure, motor failure, and battery failure. Data preprocessing begins by reading data from the simulated robot fault dataset and dividing it into a training set, a validation set, and a test set in a 3:1:1 ratio. The data is then checked for missing values, outliers, and other issues. Necessary data cleaning and normalization are then performed to scale the data to a fixed range. Through these steps, the legged robot fault detection dataset is preprocessed into a data format suitable for deep learning model training.

[0079] In one application embodiment, the present invention adopts the Adam optimization algorithm in the optimization task of model training to replace the traditional stochastic gradient descent SGD method, accelerates the convergence of the model and improves the training efficiency, so that the model can be learned and optimized more effectively.

[0080] Specifically, the model training process includes:

[0081] A1: The robot joint graph data is input into the temporal gated convolution module. Convolution operations are used in the network to control the flow of information through gating signals, enabling the temporal gated convolution module to learn the long sequence features of the sensor timing signals.

[0082] A2: The extracted temporal features are fed into the spatial graph convolution module, which convolves the node features with the adjacency matrix to capture the spatial dependencies between nodes. This module adaptively learns the association between each joint and the fault state, optimizing fault detection accuracy. Feature extraction is then performed multiple times through the temporal gated convolution and spatial graph convolution modules.

[0083] A3: Globally average pool the output features generated in the above steps and input them into the fully connected layer. In the output layer, a softmax activation function is used to predict the conditional probability of each category. The cross-entropy function is used as the loss function to measure the difference between the model prediction and the actual label.

[0084] A4: Train the entire deep learning model by calculating the loss function and updating the network parameters until the loss function value stabilizes, thereby obtaining an optimized legged robot fault diagnosis model.

[0085] This paper tests a legged robot fault detection method based on a spatiotemporal graph convolutional network. Specifically, the present invention uses PyBullet simulation to generate a legged robot fault detection dataset. First, the PyBullet simulation environment is launched and the legged robot model is loaded. Gravity and the robot's initial position are set. The robot's joint indices are obtained and global variables are initialized to simulate different fault scenarios. A function is defined to simulate normal robot motion. By setting gait parameters, the target position of each joint is calculated and joint motion is controlled. A function is defined to simulate different types of faults, including joint faults, sensor faults, motor faults, and battery faults. The robot's joint angles, joint velocities, and force sensor data are obtained and concatenated into a vector. A loop is defined to generate time series data containing different fault types. Each fault sequence consists of 100 time steps, with the first 60 time steps representing normal walking and the last 40 time steps representing a faulty state. The dataset is divided into a training set containing 2400 samples, a validation set containing 800 samples, and a test set containing 800 samples. In this embodiment, the method of the present invention is compared with the existing method through experiments. The fault diagnosis accuracy results are shown in Tables 1 and 2.

[0086] Table 1: Fault diagnosis accuracy in bipedal robot fault dataset

[0087]

[0088]

[0089] Table 2: Fault diagnosis accuracy in the quadruped robot fault dataset

[0090]

[0091] From Table 1 and Table 2, it can be seen that the method of the present invention has a higher overall accuracy and better performance on the robot dataset than other methods mentioned above. Figure 4 and Figure 5 The confusion matrix results in show that the method of the present invention can achieve accurate fault diagnosis.

[0092] Furthermore, in order to intuitively demonstrate the effectiveness of the method proposed in this study, the t-SNE dimensionality reduction technology was used to visualize the classification results. Figure 6a and Figure 6b as well as Figure 7a and Figure 7bThe comparison results shown in the figure below show the original, unclassified signal on the left and the classification result after processing using this method on the right. It can be seen that the signal processed by this method visually demonstrates a more obvious classification effect. Therefore, this example verifies the effectiveness of this method and demonstrates its superior performance.

[0093] It should be appreciated that the method steps in the embodiments of the present invention can be implemented or executed by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer-readable memory. The method can use standard programming techniques. Each program can be implemented in a high-level procedural or object-oriented programming language to communicate with the computer system. However, if desired, the program can be implemented in assembly or machine language. In any case, the language can be a compiled or interpreted language. In addition, for this purpose, the program can be run on a programmed application-specific integrated circuit.

[0094] Furthermore, the operations of the processes described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by the context. The processes described herein (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors, by hardware, or a combination thereof. The computer program includes a plurality of instructions that can be executed by one or more processors.

[0095] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented as machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, an RSM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or portions thereof, can be transmitted over a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described herein includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention can also include the computer itself.

[0096] The computer program can be applied to input data to perform the functions described herein, thereby converting the input data to generate output data that is stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents a physical and tangible object, including a specific visual depiction of the physical and tangible object produced on the display.

[0097] The above description is merely a preferred embodiment of the present invention. The present invention is not limited to the aforementioned embodiments. As long as the technical effects of the present invention are achieved by the same means, any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention. Within the scope of protection of the present invention, various modifications and variations of the technical solutions and / or implementation methods are possible.

Claims

1. A legged robot fault detection method based on spatiotemporal graph convolutional network, characterized in that: A deep learning model based on a spatiotemporal graph convolutional network is used, which includes a temporal gated convolution module and a spatial graph convolution module; The method comprises the following steps: S100, constructing a joint graph according to the joint connection relationship of the robot; S200, using the temporal gated convolution module to perform long sequence feature extraction on the joint graph; S300, using the features obtained by the sequential gated convolution module as the input of the spatial graph convolution module, and the spatial graph convolution layer performs a convolution operation on the node features through the adjacency matrix to capture the spatial dependency relationship between nodes; S400 , performing feature extraction multiple times through the temporal gated convolution module and the spatial graph convolution module to obtain a fault detection result of the legged robot during operation.

2. The method according to claim 1, characterized in that In step S100: The nodes of the joint graph represent the joints of the robot, and the edges of the joint graph represent the connection relationships between the joints. An adjacency matrix is ​​used to represent the graph structure, and the elements in the adjacency matrix represent the connection relationships between the nodes.

3. The method according to claim 2, characterized in that In the step S200, Each temporal convolution layer of the temporal gated convolution module includes two convolution modules, one of which is used to extract deep features and the other is a gated convolution layer for generating a gating signal; the gated convolution layer updates the features of the node by merging the features of the corresponding node on consecutive time slices.

4. The method according to claim 3, characterized in that In the step S200, Each of the gated convolutional layers includes a gated linear unit, which includes two one-dimensional convolutional layers; the gated convolutional layer is represented as follows: y gated =(K1-x gated +b1)⊙σ(K2*x gated +b2) Where K1 and K2 represent convolution kernels, b1 and b2 are biases, * represents convolution operation, ⊙ represents element-by-element multiplication operation, σ is the sigmoid function, and x gated and y gated are the input and output of the gated convolutional network, respectively.

5. The method according to claim 2, characterized in that In the step S300, In the graph convolutional neural network of the spatial graph convolution module, a graph is defined as a network consisting of nodes and edges, where nodes represent entities and edges represent relationships between nodes.

6. The method according to claim 5, characterized in that In the step S300, The graph convolutional neural network includes a multi-layer graph convolution layer with a hierarchical stacking structure, which is used to normalize the adjacency matrix and update the representation of the target node by weighting the features of the neighboring nodes, wherein the weighting coefficient is jointly determined by the adjacency matrix and the degree matrix.

7. The method according to claim 6, characterized in that The step S300 includes: S310: For each node in the joint graph, the graph convolutional neural network aggregates the features of the node itself and the adjacent nodes to obtain a new node, which is represented as follows: Z l+1 =A·H (l) ·W (l) Where Z l+1 It is a new node representation. The node set to be updated is represented by the adjacency matrix A and the node feature H (l) The product of W (l) is the weight matrix of the lth layer; S320, achieve aggregation by weighted summing of neighbor node features, where the weights are determined by the adjacency matrix of the graph. Where, is the new node representation after normalization, and D is the degree matrix; S330: Perform nonlinear changes on the updated node representation, which is expressed as follows: Where H (l+1) It is the node representation after the graph convolution operation, and ReLU(*) represents the activation function.

8. The method according to claim 1, characterized in that The deep learning model based on the spatiotemporal graph convolutional network includes a global average pooling layer, a fully connected layer and two STGCN modules; the output of one of the STGCN modules serves as the input of the other STGCN module; Among them, each STGCN module first captures the spatiotemporal coupling characteristics of the joints through the temporal gated convolution module, then captures the spatial dependency between nodes through the spatial graph convolution module, and then extracts the temporal features again through another temporal gated convolution module, and finally obtains the output of the STGCN module through the batch normalization layer; Among them, the obtained initial features pass through the two STGCN modules, then pass through the global average pooling layer, and finally pass through the fully connected layer to obtain the output of the deep learning model.

9. A computer-readable storage medium, characterized in that Program instructions are stored thereon, and when the program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

10. A legged robot fault detection system based on spatiotemporal graph convolutional network, characterized in that: include: A computer device comprising the computer-readable storage medium according to claim 9.

Images