A robot anomaly prediction method and system based on multi-dimensional fusion and causal inference

Abstract

The application relates to the technical field of robot anomaly prediction, in particular to a robot anomaly prediction method and system based on multi-dimensional fusion and causal inference. The method comprises the following steps: performing multi-scale deep state representation based on acquired robot multi-joint sensing data; performing dynamic causal graph fusion based on the multi-scale deep state representation, including prior knowledge graph construction based on a kinematic chain and dynamic correlation attention mechanism construction based on data driving, state fusion guided by knowledge and attention, and global state vector generation; performing hierarchical spatiotemporal dependence prediction based on the fused features, including robot joint topology graph construction and spatial dependence dynamic modeling, long-range time evolution prediction, and future robot health state prediction; and the application exhibits excellent performance in multiple core dimensions such as prediction accuracy, early warning timeliness, diagnosis interpretability, and the like, and has extremely strong actual deployment value and engineering popularization potential.

Description

Technical Field

[0001] This invention relates to the field of robot anomaly prediction technology, and in particular to a robot anomaly prediction method and system based on multidimensional fusion and causal inference. Background Technology

[0002] In modern intelligent manufacturing systems, automated workstations centered around six-axis or multi-axis articulated industrial robots are the cornerstone for executing critical processes such as high-precision welding, assembly, painting, and material handling. These robots are designed to perform high-speed, highly repetitive precision tasks at fixed workstations day after day. To ensure their motion accuracy and stability, each joint is equipped with a high-precision servo motor, encoder, and reducer, and integrates various sensors such as torque and current sensors, continuously generating massive amounts of multimodal real-time status data during operation. The consistency and health status of the robot's working state directly determine the cycle time of the entire production line, the yield rate of products, and the safety of the production process.

[0003] However, industrial robots operate under high load and high dynamic conditions for extended periods, inevitably causing fatigue and wear to their core components due to cyclic stress. For example, wear on joint reducers, degradation of the electrical performance of servo motors, bearing lubrication failure, or minute drifts in control system parameters can all trigger potential anomalies. In their early stages, these anomalies may only manifest as micrometer-level trajectory deviations of the end effector or nanosecond-level response delays for specific actions, making them extremely difficult to detect by traditional quality inspection or equipment monitoring systems. But if left unchecked, these minute signs will gradually evolve into significant operational jitter, positioning inaccuracies ("lost steps"), and even ultimately lead to catastrophic shutdowns such as joint overload and reducer gear breakage, resulting in costly production interruptions and repair costs.

[0004] Current technologies for anomaly detection and predictive maintenance of robots face significant bottlenecks. On one hand, traditional methods often rely on monitoring single physical quantities, such as performing spectral analysis on the vibration signal of a joint motor or setting static torque thresholds. This approach misses the inherent integrity of the robot as a sophisticated electromechanical coupling system. In reality, robot motion is transmitted step-by-step from the base joints to the end effector via a precise kinematic and dynamic chain. Even a minor anomaly in any upstream joint will be propagated and amplified downstream through this chain. Analyzing a single joint or sensor in isolation cannot capture this cross-joint fault propagation pattern, easily leading to missed or false alarms.

[0005] On the other hand, most existing multimodal data fusion technologies employ simple feature vector concatenation or fixed weighted averaging strategies. This "static and blind" fusion approach cannot adapt to the dynamic changes in the importance of each joint's state under different motion postures and load conditions of the robot. For example, during high-speed, wide-range swinging, the torque information of the base joint is crucial; while during delicate operations at the end effector, the angular accuracy of the wrist joint becomes dominant. Static fusion will drown out truly critical abnormal signals under specific operating conditions. More importantly, most current deep learning-based detection models are like "black boxes." Even if they can predict anomalies, they cannot clearly answer a series of crucial questions for maintenance personnel, such as "Which joint is the root cause of the anomaly?", "Is it dominated by torque overload or encoder error?", and "What historical events are associated with this anomaly?" This lack of interpretability significantly reduces the value of early warnings in guiding accurate and efficient maintenance decisions.

[0006] Therefore, there is an urgent need to develop a new method for predicting abnormal working states of robots that can deeply integrate multi-joint and multi-dimensional state information of robots, accurately model their spatiotemporal evolution along the kinematic chain, and provide interpretable predictive diagnosis, so as to break through the bottleneck of existing technologies and improve the intelligent operation and maintenance level of high-end equipment. Summary of the Invention

[0007] To address the current challenges in robot anomaly detection and diagnosis, such as insufficient multi-dimensional state perception, lack of causal relationship modeling between joints, insufficient exploration of spatiotemporal evolution along the kinematic chain, and lagging anomaly prediction with uninterpretable diagnostic results, this invention provides a robot anomaly prediction method and system based on multi-dimensional fusion and causal inference. By constructing a novel technical framework integrating multi-scale state representation, dynamic causal fusion, hierarchical spatiotemporal prediction, and endogenous interpretability, it achieves high-precision, forward-looking, and traceable intelligent early warning and diagnosis of potential anomalies in robot systems.

[0008] In a first aspect, the present invention provides a robot anomaly prediction method based on multidimensional fusion and causal inference, which adopts the following technical solution:

[0009] A robot anomaly prediction method based on multidimensional fusion and causal inference includes:

[0010] Acquire multi-joint sensor data of the robot;

[0011] Multi-scale depth state representation is performed based on the acquired robot multi-joint sensing data, including time-frequency multi-scale decomposition and local fault mode coding.

[0012] Dynamic causal graph fusion based on multi-scale deep state representation includes the construction of prior knowledge graph based on kinematic chains and the construction of data-driven dynamic association attention mechanism, knowledge and attention-guided state fusion and global state vector generation.

[0013] Hierarchical spatiotemporal dependency prediction based on fused features includes robot joint topology graph construction and spatial dependency dynamic modeling, long-term temporal evolution prediction, and future robot health status prediction.

[0014] Predictive diagnosis and causal attribution are based on the prediction results.

[0015] Furthermore, the time-frequency multi-scale decomposition includes, in order to explicitly capture multi-scale information in the signal, firstly, using Discrete Wavelet Transform (DWT) to perform multi-level decomposition on each one-dimensional original sensor signal sequence, for a length of... signal sequence ,go through After layer wavelet decomposition, a set of high-frequency detail coefficient sequences is obtained. and low-frequency approximation coefficient sequence Together, they constitute the time spectrum of the signal at different time scales and frequencies, expressed as:

[0016] ,

[0017] Among them, the low-frequency approximation coefficient Representing the long-term trend of the signal, high-frequency detail coefficient sequence The instantaneous fluctuations of the signal at different frequencies are captured, and then the coefficient sequences representing dynamics at different scales are spliced ​​together along the channel dimension to form a multi-channel feature map. This serves as the input for the subsequent encoder.

[0018] Furthermore, the local fault mode encoding includes, after obtaining the multi-channel time-frequency feature map, performing end-to-end feature learning through a deep coding structure to automatically extract deep feature representations corresponding to various abnormal operating state modes. This involves first applying parallel multi-scale convolutional layers to the input... Convolutional kernels of different sizes capture local patterns at different time scales on the time-frequency plot, as shown below:

[0019] ,

[0020] in, Representing the A convolutional layer with different kernel sizes, It is the total number of convolutional kernels; then the feature maps output by all convolutional layers are... The data is concatenated along the channel dimension and then fused and dimensionality reduced using a 1x1 convolutional layer to obtain a compact feature sequence. , represented as:

[0021] ,

[0022] To capture the long-range dependency between any two time points in the feature sequence, Feed it into a standard Transformer encoder and add a learnable position code. To provide the model with temporal order information of the sequence, the Transformer encoder is composed of... The sequence consists of stacked identical layers, each containing a multi-head self-attention (MHSA) sublayer and a feedforward neural network sublayer. The MHSA dynamically calculates the dependency weights of each time point in the sequence on all other time points, thereby constructing a global context awareness, represented as:

[0023] ,

[0024] in, These are respectively composed of the input sequence The query, key, and value matrices obtained by linear transformation; This represents the dimension of the key vector for each attention head; The operation is used to normalize the similarity at different positions in a time series. Indicates the first The output of each attention head; Indicates the number of attention heads; This indicates that the outputs of all attention heads are concatenated; The output is a linear transformation matrix used to map the multi-head attention results back to the model dimension space.

[0025] Furthermore, the aforementioned construction of the prior knowledge graph based on kinematic chains and the construction of the data-driven dynamic association attention mechanism include those for robots. Each joint On Various sensor modes, at time Generate deep state representation All The feature vectors of each sensing mode are stacked along the mode dimension to form the current time step. Total characteristic matrix ,in This represents the total number of sensing modes, expressed as:

[0026] ,

[0027] Then, domain expert knowledge is introduced and encoded into a static kinematic prior adjacency matrix. This graph serves as a priori correlation strength map between different sensing modalities within the robot, where nodes represent various sensing modalities distributed across different joints, and the total number of nodes is [missing information]. Matrix elements The range of is [0, 1], representing the modality. and modality The correlation strength is determined by physical mechanisms; further, a data-driven dynamic correlation attention mechanism is introduced, utilizing the total feature matrix constructed above. A dynamic attention weight matrix is ​​calculated using a self-attention mechanism. , of which elements Represents the current moment In data scenarios, modality Modality should be considered The attention invested is represented as:

[0028] ,

[0029] in, These are the learnable query and key projection matrices, respectively. The dimensions of the query and key vectors.

[0030] Furthermore, the knowledge- and attention-guided state fusion and global state vector generation include combining static prior knowledge with dynamic data-driven attention, through learnable gating scalars. Dynamic trade-offs are made, resulting in the final fusion guidance matrix. Represented as:

[0031] ,

[0032] Among them, gated scalar Based on the current total feature matrix by the neural network Generation, after obtaining the fusion guiding matrix, of the original feature matrix Weighting is applied to enable information interaction and enhancement between modalities, represented as: In order to obtain a single, representative image of the robot at any given moment The final overall health status vector will be the enhanced feature matrix. To achieve integration and dimensionality reduction, the system first performs a residual connection with the original feature matrix to preserve the original information. Then, it uses a multilayer perceptron (MLP) network for nonlinear transformation and dimensionality compression to obtain the final global state vector. , represented as:

[0033] ,

[0034] The MLP network employs a bottleneck structure that includes activation functions to force the model to learn more informative and compact representations. The final output global state vector is highly condensed from all joints and all sensing modalities, guided by both knowledge and data, and mutually reinforcing collaborative information.

[0035] Furthermore, the robot joint topology graph construction and spatial dependency dynamic modeling include, after obtaining the fused state vector of each joint at a single time point, all By treating each joint as a whole, a spatiotemporal map is constructed, in which the robot's... Each joint is abstracted as a set of nodes in a graph. The connections between nodes are represented by an edge set. This constitutes a static space diagram characterizing the potential interactions between robot joints. At each time step ,all The state vectors of each joint constitute a graph signal, which represents the feature matrix. Each of its rows It represents the joint. Furthermore, the state vector after fusing internal sensor information is represented as:

[0036] ,

[0037] Then, a graph attention network (GAT) is used to aggregate spatial information. For any joint node in the graph, a single-layer feedforward network is used to calculate the relationship with neighboring joints. Attention coefficient between , It is a joint The set of first-order neighbors is represented as:

[0038] ,

[0039] Among them, the weight vector , It is a learnable linear transformation matrix; This represents the vector concatenation operation; LeakyReLU is the modified linear unit activation function with leakage. Represents a node At any moment The input feature vector; Indicates time Lower Neighbor Joint The input feature vector; This represents the learnable linear transformation matrix used in a single-layer GAT to map the original node features to the attention subspace, and finally, the softmax function is used to adjust the joints. The attention coefficients of all neighbors are normalized to obtain the final attention weight. :

[0040] ,

[0041] Normalized attention weights Indicates at time Neighbor joints Information on joints The importance of joints Feature vector updated after GAT layer That is, the weighted sum of the features of all its neighboring joints based on attention weights, expressed as:

[0042] ,

[0043] in, It is a non-linear activation function. After processing by the GAT layer, a state representation matrix incorporating dynamic spatial neighborhood information is obtained. .

[0044] Furthermore, the long-range temporal evolution prediction and future robot health state prediction include, after modeling the spatial dependence within each time slice, further capturing the evolutionary pattern of spatial state in the temporal dimension, employing a temporal Transformer encoder to model temporal evolution, and incorporating past... Spatial augmented state sequence at each time step As input to the temporal Transformer, the correlation between the states of any two historical moments is calculated through a multi-layer multi-head self-attention mechanism, thus learning... The continuous overload state at any moment is the cause Key precursors to deviations in time trajectory, the output of the Time Transformer Each joint is provided with a deep representation that incorporates the entire historical spatiotemporal context at each historical moment; then, a linear prediction head is used to decode the output of the temporal Transformer at the last time step to predict the future. The robot's state at a given time is represented as: ,in, This represents the feature matrix of the last time step of the Transformer output sequence, and the final output predicted state matrix. The model is based on the past. The spatiotemporal evolution pattern at a given moment.

[0045] Furthermore, the predictive diagnosis and causal attribution based on the prediction results include anomaly scoring based on prediction errors. First, the reconstruction error between the true state matrix and the predicted state matrix is ​​calculated, where the true state matrix... The row vector Represents joint The true state, and the predicted state matrix. The row vector The representative model predicts the joint state for each joint node. Its original outlier score Represented as:

[0046] ,

[0047] To enable the anomaly score to adapt to the historical fluctuation characteristics of each joint, a dynamic normalization method based on Z-score is used to calculate the final anomaly score, where is the score for each joint node. Maintenance length is A sliding time window is used to calculate the mean of the historical reconstruction error within that window. and standard deviation , represented as:

[0048] ,

[0049] in, It is a minimal stability constant added to prevent the denominator from being zero, ultimately resulting in anomaly fractions. This indicates the degree to which the prediction error of the current joint deviates from its historical normal level.

[0050] Furthermore, the predictive diagnosis and causal attribution based on the prediction results also include multi-dimensional attribution and interpretability diagnosis, and attribution analysis from three dimensions: sensing modality, space, and time. Modality attribution is based on a fusion guidance matrix. The row vectors represent the ability of each sensing mode to absorb information from other modes after fusion. The modal contribution score is defined and calculated using a matrix. The norm of the row sum is used to quantify the relative importance of each sensing modality at the moment of anomaly occurrence; spatial attribution includes, at the moment of anomaly occurrence, the relative importance of the joint nodes judged as anomalous. Check it from all neighbors Attention weights obtained at the location If a neighboring joint Corresponding weights If it is higher than other neighbors, then it is considered a joint. It causes joint problems The key spatial drivers of anomalies; temporal attribution includes when the model is predicting... When there is an anomaly in time, for Paying attention to the state of being at any given moment indicates that... Events occurring before a certain time step are important precursors to this anomaly; finally, when the abnormal score of any joint... Exceeding the threshold If so, a report will be generated automatically.

[0051] Secondly, a robot anomaly prediction system based on multidimensional fusion and causal inference includes:

[0052] The data acquisition module is configured to acquire sensor data from the robot's multiple joints.

[0053] The multi-scale characterization module is configured to perform multi-scale depth state characterization based on the acquired robot multi-joint sensing data, including time-frequency multi-scale decomposition and local fault mode coding.

[0054] The fusion module is configured to perform dynamic causal graph fusion based on multi-scale deep state representation, including the construction of prior knowledge graph based on kinematic chains and the construction of data-driven dynamic association attention mechanism, knowledge and attention-guided state fusion and global state vector generation.

[0055] The prediction module is configured to perform hierarchical spatiotemporal dependency prediction based on the fused features, including robot joint topology graph construction and spatial dependency dynamic modeling, long-term temporal evolution prediction and future robot health status prediction.

[0056] The attribution module is configured to perform predictive diagnosis and causal attribution based on the prediction results.

[0057] Thirdly, the present invention provides a computer-readable storage medium storing a plurality of instructions adapted for loading and execution by a processor of a terminal device of the robot anomaly prediction method based on multidimensional fusion and causal inference.

[0058] Fourthly, the present invention provides a terminal device, including a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, the instructions being adapted to be loaded and executed by the processor to provide a robot anomaly prediction method based on multidimensional fusion and causal inference.

[0059] In summary, the present invention has the following beneficial technical effects:

[0060] Compared to existing methods for predicting abnormal robot operating states, which suffer from limitations such as superficial multi-dimensional signal fusion, simplistic temporal dependency modeling, and the lag and "black box" nature of anomaly diagnosis, this invention addresses the core need for "accurate prediction and intelligent diagnosis of abnormal robot operating states" by constructing a novel end-to-end solution based on multi-dimensional state fusion and causal inference. The beneficial effects of this invention are mainly reflected in the following aspects:

[0061] First, this invention fundamentally solves the problem that traditional methods struggle to simultaneously capture transient impacts (such as collisions) and long-term trends (such as wear) in robot sensing signals through an innovative multi-scale depth state characterization module (MDSR). This greatly enhances the depth and breadth of extracting abnormally sensitive features from raw data, laying a solid data foundation for subsequent accurate predictions.

[0062] Secondly, the Dynamic Causal Graph Fusion Module (DCGF) designed in this invention overcomes the limitations of traditional methods that rely solely on "correlation" for information fusion. By combining data-driven attention with prior knowledge, it learns dynamic relationships that conform to the kinematics and dynamics logic of robots, and uses this to guide information fusion. This results in a fused state vector that is not only richer in information but also inherently possesses a high degree of interpretability.

[0063] Furthermore, the Hierarchical Spatiotemporal Dependency Prediction Module (HSTP) of this invention captures the dynamic spatial influence between joints through a graph attention network and combines it with a Transformer model to mine long-term temporal dependencies, thereby achieving accurate modeling of the evolution of the robot's future health state. This "prediction-verification" mode enables the system to provide early warnings before anomalies occur, gaining valuable response time for preventative maintenance.

[0064] Finally, this invention perfectly solves the "black box" problem of deep learning models through the Predictive Diagnosis and Causal Attribution (PCAD) module. The system can not only accurately identify anomalies, but also, relying on the model's inherent attention weights and causal relationships, clearly locate the root cause of the anomaly, trace its propagation path along the chain of joints, and automatically generate structured diagnostic reports. This inherent interpretability greatly enhances the credibility and decision support value of the technology in actual production and maintenance.

[0065] In summary, the test results of the method of this invention on a publicly available six-axis robot lifecycle dataset show that, in typical anomaly prediction tasks, the anomaly prediction accuracy (ACC) is improved to 96.2%, and the overall F1 score reaches 95.5%; the average lead time is 25.6 seconds; and the root cause joint localization accuracy (RCA-Acc) is as high as 94.1%. While ensuring high performance, the system's average inference time per sample is only 168 milliseconds. This invention demonstrates superior performance in multiple core dimensions such as prediction accuracy, early warning timeliness, and diagnostic interpretability, possessing strong practical deployment value and engineering promotion potential. Attached Figure Description

[0066] Figure 1 This is a schematic diagram of a robot anomaly prediction method based on multidimensional fusion and causal inference according to Embodiment 1 of the present invention;

[0067] Figure 2 This is a schematic diagram of the core performance indicators of Embodiment 1 of the present invention;

[0068] Figure 3 This is a schematic diagram illustrating the early warning and diagnostic capabilities of Embodiment 1 of the present invention;

[0069] Figure 4 This is a schematic diagram of inference efficiency in Embodiment 1 of the present invention;

[0070] Figure 5 This is a schematic diagram of the overall performance of Embodiment 1 of the present invention. Detailed Implementation

[0071] The present invention will be further described in detail below with reference to the accompanying drawings.

[0072] Example 1

[0073] Reference Figure 1 This embodiment proposes "a method for predicting abnormal working states of robots based on multi-dimensional state fusion and causal inference". Its overall technical framework includes four highly coupled core modules: (1) Multi-scale deep state representation module (MDSR); (2) Dynamic causal graph fusion module (DCGF); (3) Hierarchical spatiotemporal dependency prediction module (HSTP); (4) Predictive diagnosis and causal attribution module (PCAD). The specific scheme is as follows:

[0074] (1) Multi-scale Depth State Representation Module (MDSR)

[0075] This module forms the foundation for the entire system's perception and representation. Its core task is to transform raw data from multi-source, heterogeneous sensors at various robot joints (such as joint angles, angular velocities, torques, and motor currents) into a unified depth feature vector that comprehensively reflects the local operating state of the joints. Robot sensor signals exhibit typical non-stationary, multi-scale characteristics, including both slow trends indicative of long-term wear and degradation (such as increased reducer backlash and lubrication aging) and short-term, dramatic fluctuations caused by control commands or external collisions (such as emergency stops and abnormal workpiece pickup). Therefore, this module uses a combination of time-frequency decomposition and depth coding to decouple and extract multi-scale dynamic features related to abnormal operating states from the signals.

[0076] 1) Time-frequency multiscale decomposition

[0077] To explicitly capture multi-scale information in the signal, Discrete Wavelet Transform (DWT) is first used to perform multi-level decomposition on each one-dimensional original sensor signal sequence. DWT can decompose the signal into sub-bands of different frequencies while preserving its temporal locality. For a signal of length... signal sequence ,go through After layer wavelet decomposition, a set of high-frequency detail coefficient sequences can be obtained. and a low-frequency approximation coefficient sequence These coefficient sequences collectively constitute the "time spectrum" of the signal at different time scales and frequencies, which can be represented as:

[0078] ,

[0079] Among them, the low-frequency approximation coefficient This represents the long-term trend or baseline of the signal, which is crucial for identifying slowly evolving fault modes such as persistent wear of gears in a joint reducer or slow degradation of motor performance. High-frequency detail coefficient sequences... It accurately captures instantaneous fluctuations or high-frequency noise of signals at different frequencies, which plays an irreplaceable role in identifying short-term, sudden abnormal working states such as instantaneous collisions between the end effector and the environment, joint vibrations, and current spikes.

[0080] These coefficient sequences, which physically represent dynamics at different scales, are concatenated along the channel dimension to form a multi-channel feature map. This preprocessing method allows the model to directly extract features from the decomposed, multi-scale representation with clear physical meaning, without implicitly and in a black-box manner learning frequency information from the original signal. This greatly improves the efficiency and accuracy of subsequent fault mode coding.

[0081] 2) Local Fault Mode Coding

[0082] After obtaining the multi-channel time-frequency feature maps, this section aims to use a deep coding structure to perform end-to-end feature learning to automatically extract deep, abstract feature representations corresponding to various abnormal operating state patterns. This coding structure combines the powerful local feature extraction capabilities of one-dimensional convolutional neural networks (1D-CNN) with the excellent global dependency modeling capabilities of the Transformer.

[0083] First, a parallel set of multi-scale convolutional layers is applied to the input. This layer group contains multiple one-dimensional convolutional layers with different kernel sizes (e.g., 1x3, 1x5, 1x7). Different kernel sizes can capture local patterns at different time scales on the time-frequency map. For example, small kernels are good at capturing sharp torque impact signals, while large kernels can better cover a complete periodic motion pattern. This can be represented as:

[0084] ,

[0085] in, Representing the A convolutional layer with different kernel sizes, This represents the total number of convolutional kernels. Then, the feature maps output by all convolutional layers are... The data are concatenated along the channel dimension and then fused and dimensionality reduced using a 1x1 convolutional layer to obtain a compact feature sequence rich in local pattern information. , can be represented as:

[0086] ,

[0087] Next, in order to capture the long-range dependency between any two time points in the feature sequence, Feed a standard Transformer encoder. Before feeding it in, add a learnable position code to it. To provide the model with temporal sequence information, it can be represented as:

[0088] ,

[0089] Transformer encoder is composed of It consists of stacked identical layers, each containing a multi-head self-attention (MHSA) sublayer and a feedforward neural network sublayer. The core mechanism of MHSA lies in its ability to dynamically calculate the dependency weights of each time point in the sequence on all other time points, thereby constructing a global context-aware system, which can be represented as:

[0090] ,

[0091] in, These are respectively composed of the input sequence The query, key, and value matrices obtained by linear transformation;

[0092] This represents the dimension of the key vector for each attention head, used to scale the inner product to prevent excessively large values ​​from causing gradient instability; The operation is used to normalize the similarity of each position in the time series so that the weight sum is 1, thereby reflecting the importance of different time points to the current time point; Indicates the first The output of each attention head independently models feature correlations within a different subspace; Indicates the number of attention heads; This indicates that the outputs of all attention heads are concatenated; The output is a linear transformation matrix used to map the multi-head attention results back to the model dimension space.

[0093] Through this self-attention mechanism, the model can effectively capture long-distance causal relationships. For example, it can learn that a sudden change in end load a few seconds ago (reflected in the high-frequency detail coefficients of the torque signal) is a key precursor that causes the current joint trajectory deviation (reflected in the pattern change of the angle signal).

[0094] go through After the Transformer encoding, the vector of the output sequence at the last time step is taken as the final depth state representation of the sensor signal. For example, for the robot's... The torque sensor for each joint outputs a feature vector as follows: Similarly, other sensor signals such as the joint's angle and current are also independently encoded to obtain... And so on. These vectors together constitute the first... A multi-dimensional representation of the state of each joint at the current moment.

[0095] (2) Dynamic Cause-and-Effect Graph Fusion Module (DCGF)

[0096] After obtaining high-quality feature representations of each joint and sensing modality through the multi-scale deep state representation module, the core task of this module is to effectively connect these isolated information islands to form a unified, comprehensive, and profound understanding of the robot's overall health status. Simple feature stitching or averaging cannot capture the complex nonlinear dependencies and dynamic changes in importance between modalities. Therefore, this invention proposes a dynamic causal graph fusion mechanism, which innovatively combines domain expert knowledge with a data-driven attention mechanism to achieve dynamic, adaptive, and interpretable fusion of multimodal features.

[0097] This module first integrates the output of module (1). For the robot... Each joint On For various sensor modes (such as angle, torque, current, etc.), module (1) is used to measure the time at which the sensor is in motion. A deep state representation was generated. This module will include all The feature vectors of each sensing mode are stacked along the mode dimension to form the current time step. Total characteristic matrix ,in The total number of sensing modes can be expressed as:

[0098] ,

[0099] The overall characteristic matrix A complete description of the robot at any given moment. The status of all sensors is recorded and used as input for all subsequent calculations in this module.

[0100] 1) Construction of prior knowledge graph based on kinematic chains

[0101] To make the model fusion process more robust and consistent with the physical working mechanism of robots, this invention first introduces domain expert knowledge. This knowledge is encoded into a static "kinematic prior adjacency matrix". This is a graph representing the prior correlation strength between different sensor modes within a robot. The nodes in the graph represent various sensor modes distributed across different joints, and the total number of nodes is [number missing]. Matrix elements The range of is [0, 1], representing the modality. and modality The strength of the correlation at the physical mechanism level. For example, for a six-axis serial robot:

[0102] Strong correlation: Angle sensors and torque sensors at the same joint (such as the wrist joint J5) are highly correlated due to the direct correspondence between motion and force. High values ​​can be set, such as 0.9. Medium correlation: Two physically directly connected joints, such as the base joint (J1) and the waist joint (J2), have a direct kinematic transmission relationship in their motion states (such as angular velocity). The correlation between the corresponding sensors can be set to a medium value, such as 0.6. Weak correlation: Two physically far apart joints, such as the base joint (J1) and the end flange joint (J6), although there is indirect dynamic coupling, the direct correlation is weak in most operating conditions. The correlation between the corresponding sensors can be set to a low value, such as 0.1.

[0103] The construction of this knowledge matrix provides a robust foundation for the model, preventing it from learning spurious associations from the data that violate the principles of robot kinematics when the data is sparse or heavily disturbed.

[0104] 2) Data-driven dynamic associative attention

[0105] Static knowledge alone is insufficient to handle the complex and ever-changing working conditions of robots. For example, during high-speed, large-scale material handling, the torque changes of the base joints are dominant; while during precision assembly at the end effector, the minute angle changes of the wrist joints are even more critical. The model must be able to dynamically adjust its dependence on each sensing modality based on the real-time quality and content of the current data. To this end, this invention introduces a data-driven, dynamic associative attention mechanism.

[0106] Using the total feature matrix constructed above The model calculates a dynamic attention weight matrix through a self-attention mechanism. , of which elements Represents the current moment In data scenarios, modality Modality should be considered How much "attention" should be invested? The calculation process is as follows:

[0107] ,

[0108] in, These are the learnable query and key projection matrices, respectively. The dimensions are the query and key vectors; softmax is performed independently on each row of the matrix, ensuring that the sum of the attention weights of each modality to all other modalities is 1. This dynamic attention matrix gives the model great flexibility, enabling it to autonomously discover and strengthen cooperative signals between modalities, or suppress conflicting and noisy signals, based on the instantaneous characteristics of the input data.

[0109] 3) Integration of knowledge and attention-guided states

[0110] This step organically combines the static prior knowledge mentioned above with dynamic data-driven attention. This is achieved through a learnable gating scalar. The model can dynamically weigh whether to place more trust in expert knowledge. ), or does it rely more on relationships learned from the current data () The final fusion guidance matrix. The following formula is used to calculate:

[0111] ,

[0112] Among them, gated scalar A simple neural network based on the current total feature matrix Generate it, making it data-dependent:

[0113] ,

[0114] in, For matrix flattening operation, This is the Sigmoid activation function.

[0115] This design allows the model to rely more heavily on high-quality data and clear signals. However, when the data is noisy or the patterns are ambiguous, it is advisable to revert to robust expert knowledge. After obtaining the fusion guiding matrix, it is used to refine the original feature matrix. Weighting, to achieve information interaction and enhancement between modalities, can be represented as:

[0116] ,

[0117] 4) Global state vector generation

[0118] In order to obtain a single, representative image of the robot at any given moment The final overall health status vector requires the enhanced feature matrix. Integration and dimensionality reduction are then performed. First, a residual connection is made between the vector and the original feature matrix to preserve the original information. Then, a multilayer perceptron (MLP) network is used for nonlinear transformation and dimensionality compression to obtain the final global state vector. , can be represented as:

[0119] ,

[0120] This MLP network typically employs a bottleneck structure with activation functions (i.e., the hidden layer dimension is smaller than the input and output layers) to force the model to learn a more informative and compact representation. The final output global state vector $S_{global,t}$ highly condenses mutually reinforcing collaborative information from all joints and all sensing modalities, guided by both knowledge and data. It is no longer a simple accumulation of modal features, but a holistic and structural description of the robot's current health state, serving as input for the next module's temporal prediction.

[0121] (3) Hierarchical Spatiotemporal Dependency Prediction Module (HSTP)

[0122] This module is the core prediction engine of this invention, designed to model and predict the complex evolution patterns of the states of multiple robot joints in both time and space dimensions. After obtaining the fused state vector of each joint at a single time point, this module combines all... By treating each joint as a whole, constructing a spatiotemporal map, and learning its dynamic patterns, we can achieve accurate prediction of the robot's future health status. This is the key to achieving "early warning" rather than "alarm".

[0123] 1) Construction of robot joint topology diagram

[0124] Robot Each joint (e.g., J1 to J6) is abstracted as a set of nodes in a graph. The connections between nodes, i.e., the set of edges. The connection is constructed based on its inherent kinematic chain. Associativity can be defined based on several criteria: Physical connection: In the robot's kinematic model, connections are established between joints that are physically directly connected via links. Force / motion transmission path: According to the robot's dynamics model, directed connections are established along the main motion and torque transmission paths, for example, from the base joint (J1) to the end flange joint (J6). Historical data correlation: By analyzing a large amount of historical normal operation data, joint pairs with highly synchronized or causal motion states are identified and connections are established for them.

[0125] This creates a static space diagram that can characterize the potential interactions between robot joints. At each time step ,all The state vectors of each joint constitute a graphical signal. This graphical signal is represented as a feature matrix. Each of its rows It is the output of module (2) that represents the joint. The state vector after fusing information from all its internal sensors (i.e., in generating the global state vector) The previous joint-level feature aggregation results can be represented as:

[0126] ,

[0127] The matrix A complete description at time The health status of all joints of the robot is the starting point for this module to transmit spatial information.

[0128] 2) Spatial Dependency Dynamic Modeling

[0129] The transmission of abnormal states (such as vibration and torque overload) along the robot's joint chain is not homogeneous; it is affected by the robot's current posture and motion speed, causing the influence of different neighboring joints on the central joint to change dynamically. To capture this dynamic spatial dependency, this invention employs a Graph Attention Network (GAT) to aggregate spatial information. Unlike traditional Graph Convolutional Networks (GCNs) that use a fixed Laplacian matrix for neighborhood aggregation, GAT can dynamically learn an attention weight for each neighboring joint.

[0130] For any joint node in the diagram, its relationship with neighboring joints... ( It is a joint Attention coefficients between first-order neighbor sets Computed through a single-layer feedforward network, which consists of learnable weight vectors. Parameterization can be represented as:

[0131] ,

[0132] in, It is a learnable linear transformation matrix that acts on the features of each node, used to map the input features to a higher-level feature space; This represents the vector concatenation operation; LeakyReLU is the modified linear unit activation function with leakage. Represents a node At any moment The input feature vector; Indicates time Lower Neighbor Joint The input feature vector; This represents the learnable linear transformation matrix used in a single-layer GAT to map the original node features to the attention subspace.

[0133] To make the attention coefficients between different joints comparable, the softmax function is used to analyze the joints. The attention coefficients of all neighbors are normalized to obtain the final attention weight. :

[0134] ,

[0135] Normalized attention weights Indicates at time Neighbor joints Information on joints The importance of joints. Feature vector updated after GAT layer That is, the weighted sum of the features of all its neighboring joints based on attention weights, which can be expressed as:

[0136] ,

[0137] in, It is a non-linear activation function. To enhance the expressiveness and stability of the model, multiple independent attention heads can be used in parallel, and their outputs can be concatenated or averaged.

[0138] After processing by the GAT layer, a state representation matrix that incorporates dynamic spatial neighborhood information is obtained. .

[0139] 3) Long-range temporal evolution prediction

[0140] After modeling the spatial dependencies within each time slice, it is necessary to further capture the evolution of these spatial states over time. Robot performance degradation (such as reducer wear) is often the result of long-term accumulation, potentially taking tens of thousands of work cycles to manifest. Traditional recurrent neural networks (RNNs) are prone to gradient vanishing or exploding problems when processing such long sequences. Therefore, this invention employs a temporal Transformer encoder to model temporal evolution.

[0141] The past Spatial augmented state sequence at each time step This serves as input to the temporal Transformer. Similar to the encoder in the first module, this module employs a multi-layered multi-head self-attention mechanism, enabling the model to compute the correlation between the states of any two historical time points. For example, the model can learn... The continuous overload state at any moment is the cause A key precursor to deviations in the timeline, even when the two moments are far apart in time. Output of the Time Transformer. It provides a deep representation of each joint at each historical moment that incorporates the entire historical spatiotemporal context.

[0142] 4) Predicting the health status of future robots

[0143] To achieve "predictive" fault diagnosis, this invention utilizes a linear prediction head to decode the output of the temporal Transformer at the last time step in order to predict the future. The robot's state at a given time can be represented as:

[0144] ,

[0145] in, This represents the feature matrix of the last time step of the output sequence of the time-transformer.

[0146] The final output prediction state matrix The model is based on the past. The spatiotemporal evolution pattern at each moment provides a comprehensive prediction of the health status of the entire robot joint chain at the next moment.

[0147] (4) Predictive Diagnosis and Causal Attribution Module (PCAD)

[0148] This module serves as the final decision-making and interpretation output for the entire intelligent early warning and diagnostic system. Simply outputting a "normal" or "normal" label, or a raw prediction of the future state, is far from sufficient for robot maintenance engineers. The core task of this module is to compare the prediction results from the previous module with actual observation data. It must not only accurately determine the occurrence of abnormal operating states and locate their spatiotemporal position (i.e., which joint), but also deeply analyze the basis of the model's decision-making. In a "white-box" manner, it answers key questions such as "why is it abnormal?", "what sensor signals are driving it?", and "what is the root cause?", thereby generating a structured diagnostic report rich in information that decision-makers can directly use.

[0149] 1) Anomaly scoring based on prediction error

[0150] The core idea of ​​anomaly detection in this invention is that a well-trained spatiotemporal prediction model under normal working conditions should predict the robot's future state in a highly consistent manner with the actual observations. If there is a large deviation between the two, it is very likely that the robot's working state has deviated from the known normal mode, that is, an anomaly has occurred.

[0151] In obtaining the next moment The actual observation data is obtained, and the actual joint state matrix is ​​obtained through the processing of modules (1) and (2). Then, this module calculates its matrix with the matrix predicted by module (3). The reconstruction error between them. Wherein, the true state matrix... The first row vector Represents joints The true state, and the predicted state matrix. The row vector This represents the model's prediction of the joint state. For each joint node... Its original outlier score Defined as the square of the Euclidean distance (L2 norm) between these two vectors, it can be expressed as:

[0152] ,

[0153] However, the stability of different joints in different tasks varies, and their prediction error baselines under normal operating conditions also differ. For example, the normal error fluctuation of a base joint responsible for main weight-bearing and swinging may be higher than that of a wrist joint responsible only for fine-tuning posture. If a globally uniform fixed threshold is used, it is very easy to generate a large number of false alarms in the former and miss early weak abnormal signals in the latter.

[0154] To enable the anomaly score to adapt to the historical fluctuation characteristics of each joint, this invention employs a dynamic normalization method based on Z-score to calculate the final anomaly score. Specifically, the system will perform a dynamic normalization method for each joint node. Maintain a length of A sliding time window is used to calculate the mean of the historical reconstruction error within that window. and standard deviation , can be represented as:

[0155] ,

[0156] in, It is a minimal stability constant added to prevent the denominator from being zero.

[0157] The final abnormal score This visually represents the degree to which the prediction error of the current joint deviates from its historical normal level. When this score exceeds a preset, statistically significant threshold... (For example, 3 represents a deviation of 3 standard deviations from the normal mean), and the system can then determine the joint nodes. exist It can detect abnormal working conditions at any time, thus achieving adaptive and highly robust anomaly detection.

[0158] 2) Multidimensional attribution and interpretability diagnosis

[0159] After identifying an anomalies, answering the "why" question is key to enhancing the system's practical value. The framework design of this invention inherently preserves rich process information, making it possible to perform attribution analysis from three dimensions: sensing modality, space (joints), and time.

[0160] Modal attribution: What type of drastic change in observational data (angle, torque, or current) is driving the anomaly? The answer lies in the dynamic causal graph fusion module of module (2). Fusion guidance matrix The row vectors represent the ability of each sensing mode to absorb information from other modes after fusion. A modal contribution score can be defined and calculated using a matrix. The norm of the row sum is used to quantify the relative importance of each sensing modality at the moment the anomaly occurs. The higher the contribution score of a modality, the greater the influence of its features on the final fused representation in the current context, and the more likely it is to be the main driving force of the anomaly.

[0161] Spatial attribution (joint attribution): Is an abnormality in a joint caused by its own internal factors, or is it strongly influenced by other joints (such as the transmission of abnormal vibrations)? The answer can be found in the graph attention network (GAT) of module (3). At the moment the abnormality occurs, for the joint node that is judged as abnormal... You can check it from all neighbors. Attention weights obtained at the location If a neighboring joint Corresponding weights If it is significantly higher than that of its neighbors, there is ample reason to believe that the joint... It causes joint problems The critical spatial driving force of anomalies. By tracing these highly attention-weighted connections, the propagation path of anomalous states along the robot's joint chain can be mapped.

[0162] Temporal attribution: Is the current anomaly a sudden event, or has it gradually evolved from key events at certain points in the past? The answer lies hidden in the temporal Transformer encoder of module (3). By analyzing its internal self-attention map, especially the attention weights connecting the final prediction output with each historical time step, the key historical moments that contribute the most to the prediction of the current anomaly can be identified. For example, if the model predicts... When there is an anomaly in time, for The fact that the state of mind at any given moment received extremely high attention indicates that... An event that occurred a time step earlier (such as a strong external collision) is an important precursor to this anomaly.

[0163] 3) Generation of structured early warning reports

[0164] The final output of this module is not an isolated outlier score or attribution value, but a structured diagnostic report that integrates all the above information and is human-readable and understandable. When the outlier score of any joint... Exceeding the threshold The system will automatically generate a report. The report content includes:

[0165] Basic information: warning time, robot ID, abnormal joint location (e.g., joint J3).

[0166] Anomaly Level: Classified according to the anomaly score (e.g.: The result is described as "mildly abnormal". (Highly abnormal).

[0167] Dominant mode: List the sensing modes that contribute the most and the changes in their key indicators (e.g., "Torque mode is dominant, with the joint J3 torque index overshooting by 30% within 6 milliseconds").

[0168] Spatial source: Indicate the nearest joint that is most affected and its condition (e.g., "mainly affected by abnormal vibration transmission from joint J2").

[0169] Key precursors: Identify the most impactful historical events (e.g., "Highly correlated with the end effector overload event 2 seconds ago").

[0170] By generating such a comprehensive, in-depth, and interpretable diagnostic report, this invention transforms complex model outputs into intuitive and clearly guiding decision-making basis, thereby playing a significant role in practical applications such as robot fault prevention, rapid troubleshooting, and intelligent operation and maintenance.

[0171] 3. Beneficial effects

[0172] Compared to existing methods for predicting abnormal robot operating states, which suffer from limitations such as superficial multi-dimensional signal fusion, simplistic temporal dependency modeling, and the lag and "black box" nature of anomaly diagnosis, this invention addresses the core need for "accurate prediction and intelligent diagnosis of abnormal robot operating states" by constructing a novel end-to-end solution based on multi-dimensional state fusion and causal inference. The beneficial effects of this invention are mainly reflected in the following aspects:

[0173] First, this invention fundamentally solves the problem that traditional methods struggle to simultaneously capture transient impacts (such as collisions) and long-term trends (such as wear) in robot sensing signals through an innovative multi-scale depth state characterization module (MDSR). This greatly enhances the depth and breadth of extracting abnormally sensitive features from raw data, laying a solid data foundation for subsequent accurate predictions.

[0174] Secondly, the Dynamic Causal Graph Fusion Module (DCGF) designed in this invention overcomes the limitations of traditional methods that rely solely on "correlation" for information fusion. By combining data-driven attention with prior knowledge, it learns dynamic relationships that conform to the kinematics and dynamics logic of robots, and uses this to guide information fusion. This results in a fused state vector that is not only richer in information but also inherently possesses a high degree of interpretability.

[0175] Furthermore, the Hierarchical Spatiotemporal Dependency Prediction Module (HSTP) of this invention captures the dynamic spatial influence between joints through a graph attention network and combines it with a Transformer model to mine long-term temporal dependencies, thereby achieving accurate modeling of the evolution of the robot's future health state. This "prediction-verification" mode enables the system to provide early warnings before anomalies occur, gaining valuable response time for preventative maintenance.

[0176] Finally, this invention perfectly solves the "black box" problem of deep learning models through the Predictive Diagnosis and Causal Attribution (PCAD) module. The system can not only accurately identify anomalies, but also, relying on the model's inherent attention weights and causal relationships, clearly locate the root cause of the anomaly, trace its propagation path along the chain of joints, and automatically generate structured diagnostic reports. This inherent interpretability greatly enhances the credibility and decision support value of the technology in actual production and maintenance.

[0177] In summary, the test results of the method of this invention on a publicly available six-axis robot lifecycle dataset show that, in typical anomaly prediction tasks, the anomaly prediction accuracy (ACC) is improved to 96.2%, and the overall F1 score reaches 95.5%; the average lead time is 25.6 seconds; and the root cause joint localization accuracy (RCA-Acc) is as high as 94.1%. While ensuring high performance, the system's average inference time per sample is only 168 milliseconds. This invention demonstrates superior performance in multiple core dimensions such as prediction accuracy, early warning timeliness, and diagnostic interpretability, possessing strong practical deployment value and engineering promotion potential.

[0178] Experimental verification

[0179] To systematically verify the performance advantages of the method of this invention in predicting and diagnosing abnormal working states of robots, we used a publicly available simulation dataset for a six-axis collaborative robot (similar to UR5). This dataset records the complete state information of the robot during the execution of repetitive "pick-and-place" task cycles until a simulated failure occurs. The data is collected by multi-channel sensors and mainly includes: ① command / actual position, velocity, and torque of the six joints; ② motor current; ③ Cartesian space coordinates of the end effector. The dataset contains 1000 labeled task cycles, which we divided into a training set (700 cycles) and a test set (300 cycles) to ensure coverage of four typical robot health stages: "healthy working state," "early performance degradation," "mid-term anomaly," and "imminent failure."

[0180] To comprehensively evaluate the performance of the method of this invention, the following four mainstream comparative methods, representative of both academia and industry, are set up: LSTM: A classic recurrent neural network method, which flattens all sensor channels and inputs them into an LSTM network for prediction, representing a pure temporal analysis benchmark that ignores structural information. Concat-MLP: A basic multimodal fusion method, which simply concatenates the feature vectors of all sensors at the same time point and inputs them into a multilayer perceptron (MLP) for classification, representing a shallow static fusion approach. Transformer: It uses a standard Transformer encoder to directly process the concatenated multi-channel temporal data, representing an advanced pure temporal analysis model. GCN-LSTM: An advanced spatiotemporal graph network method, which constructs the robot joint chain as a static graph, uses a graph convolutional network (GCN) to handle spatial relationships, and uses LSTM to handle temporal evolution, making it a strong competitor in the current field of robot anomaly detection.

[0181] All methods were evaluated under the same training and test set conditions. Evaluation metrics included: accuracy (ACC, %), F1 score (F1-Score, %), lead time (Lead Time, s), root cause joint localization accuracy (RCA-Acc, %), and inference time (Time, ms).

[0182] Table 1. Comparison of data from different methods under five major indicators.

[0183] Method Name ACC (%) F1-Score (%) Lead Time (s) RCA-Acc (%) Time (ms) LSTM 82.1 79.5 N / A N / A 115 Concat-MLP 80.6 77.3 N / A 48.2 85 Transformer 88.5 86.2 5.1 N / A 150 GCN-LSTM 91.3 90.5 8.5 75.6 195 Method of the present invention 96.2 95.5 25.6 94.1 168

[0184] The experimental results are shown in Table 1 and Figure 2 , Figure 3 , Figure 4 , Figure 5As shown in the diagram, the five indicators—accuracy, F1 score, early warning lead time, root cause joint location accuracy, and inference time—have different dimensions and evaluation directions, making them unsuitable for direct display on a single radar chart. To achieve a fair and intuitive comparison of overall performance, this invention first standardizes the indicators. This invention retains the positive indicators (higher values ​​are better, i.e., accuracy, F1 score, early warning lead time, and root cause joint location accuracy). For the negative indicators (lower values ​​are better, i.e., inference time), this invention first normalizes all indicators to their maximum and minimum values, then applies a reverse processing of "1 - normalized value," thus converting all indicators into a uniform "higher values ​​are better" format, clearly displaying performance differences on the same chart. To prevent the contours of poorly performing models from "collapsed" on the radar chart due to zero values, this invention also introduces an offset mechanism to ensure clear comparability of the contours of all methods.

[0185] Core performance indicators ( Figure 2 In the field of physics, traditional single-modal methods like LSTM suffer from performance limitations due to missing information. Concat-MLP, employing blind static splicing, fails to effectively utilize multi-joint information, resulting in even worse performance. Transformer models, with their powerful long-range dependency capture capabilities, outperform the former two, but their accuracy still has significant room for improvement due to the lack of modeling of physical relationships between joints. GCN-LSTM significantly improves performance by introducing a fixed kinematic chain graph structure to aggregate joint information. The method of this invention, however, uses the MDSR module for deeper feature extraction and the DCGF module for dynamic causal fusion, achieving the most accurate identification of abnormal operating states and thus achieving the best performance on both ACC and F1-Score.

[0186] In terms of early warning and diagnostic capabilities ( Figure 3 On the other hand, LSTM, Concat-MLP, and Transformer models either fail to predict or lack interpretability mechanisms, resulting in lead time and RCA-Acc scores of N / A or extremely low. GCN-LSTM possesses some predictive ability, but its grasp of long-term performance degradation trends is inferior to the hierarchical prediction module of this invention, leading to a shorter warning lead time. Its fixed graph structure also limits its dynamic attribution capabilities. This invention's method, with its "predictive" framework and unique causal attribution mechanism, not only achieves an average warning time of up to 25.6 seconds but also achieves an accuracy of 94.1% in locating the root cause of anomalies, demonstrating significant practical value.

[0187] In reasoning efficiency ( Figure 4In terms of performance, the method of this invention (168ms) is faster than GCN-LSTM (195ms). This is because the core computing units (Transformer, GAT) of this invention have high parallel computing capabilities, while the LSTM units in GCN-LSTM have serial computing dependencies, resulting in lower efficiency when processing long sequences. This proves that this invention achieves better performance while also maintaining high computational efficiency, meeting the needs of near real-time applications in industrial scenarios.

[0188] In terms of overall performance ( Figure 5 The radar chart visually illustrates the overall performance of all methods. The area enclosed by the outline of the method of this invention is significantly larger than that of all the comparative methods, and it has no obvious weaknesses in any dimension, indicating that it achieves the best overall performance in the five core dimensions. In particular, methods such as LSTM and Transformer show a significant "collapse" in their performance outlines due to the lack of early warning and diagnostic capabilities. While GCN-LSTM performs more evenly, its outline is completely enveloped by the method of this invention, further highlighting the comprehensive superiority of the method of this invention in achieving multiple objectives such as high accuracy, high timeliness, and high interpretability.

[0189] A computer-readable storage medium storing a plurality of instructions adapted for loading and execution by a processor of a terminal device of the robot anomaly prediction method based on multidimensional fusion and causal inference.

[0190] A terminal device includes a processor and a computer-readable storage medium, the processor being configured to implement various instructions; the computer-readable storage medium being configured to store multiple instructions adapted for loading and execution by the processor of the aforementioned robot anomaly prediction method based on multidimensional fusion and causal inference.

[0191] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A robot anomaly prediction method based on multi-dimensional fusion and causal inference, characterized in that, The method comprises the following steps: obtaining multi-joint sensing data of a robot; performing multi-scale deep state representation based on the obtained multi-joint sensing data of the robot, including time-frequency multi-scale decomposition and local fault mode coding; performing dynamic causal graph fusion based on the multi-scale deep state representation, including prior knowledge graph construction based on kinematic chains and data-driven dynamic correlation attention mechanism construction, knowledge and attention guided state fusion, and global state vector generation; performing hierarchical spatio-temporal dependence prediction based on the fused features, including robot joint topology graph construction and spatial dependence dynamic modeling, long-range time evolution prediction, and future robot health state prediction; performing predictive diagnosis and causal attribution based on the prediction result. 2.The robot anomaly prediction method based on multi-dimensional fusion and causal inference according to claim 1, characterized in that, The time-frequency multi-scale decomposition, including explicitly capturing multi-scale information in the signal, first uses a discrete wavelet transform (DWT) to perform multi-layer decomposition on each one-dimensional original sensing signal sequence. For a signal sequence of length , after layer wavelet decomposition, a high-frequency detail coefficient sequence set and a low-frequency approximation coefficient sequence are obtained, which together constitute the time-frequency spectrum of the signal at different time scales and frequencies, denoted as:​ ， where the low-frequency approximation coefficients represent the long-term trend of the signal, and the high-frequency detail coefficients sequence captures the instantaneous fluctuations of the signal at different frequencies, and the coefficient sequences representing different scales of dynamics are spliced in the channel dimension to form a multi-channel feature map as the input of the subsequent encoder. 3.The robot anomaly prediction method based on multi-dimensional fusion and causal inference of claim 2, wherein, The local fault mode coding comprises: after obtaining a multi-channel time-frequency feature map, performing end-to-end feature learning through a deep coding structure to automatically extract deep feature representations corresponding to various abnormal working state modes, wherein a parallel multi-scale convolutional layer group is first applied to the input Convolution kernels of different sizes capture local patterns of different time scales on the time-frequency map, and are represented as: ， wherein, represents the first convolutional layer with different kernel sizes, is the total number of convolution kernels; then the feature maps output by all convolutional layers are spliced in the channel dimension and information fusion and dimension reduction are performed through a 1x1 convolutional layer to obtain a compact feature sequence , denoted as: ， To capture the long-range dependencies between any two time points in the sequence of features, we feed them into a standard Transformer encoder augmented with a learnable position encoding to provide the model with the temporal order information of the sequence, which is stacked by identical layers, each of which contains a multi-head self-attention (MHSA) sub-layer and a feed-forward neural network sub-layer. The MHSA builds a global context awareness by dynamically computing the dependency weights of each time point in the sequence to all other time points, which is represented as: ​ ， wherein, are query, key and value matrices resulting from a linear transformation of the input sequence denotes the dimension of the key vectors for each attention head; denotes the dimension of the key vectors for each attention head; is an operation for normalizing the similarity of positions in the time series, denotes the output of the th attention head; denotes the number of attention heads; denotes concatenating the outputs of all attention heads; is an output linear transformation matrix for mapping the multi-head attention result back to the model dimension space.

4. The method of claim 3, wherein, The prior knowledge graph construction based on kinematic chain and the dynamic correlation attention mechanism construction based on data driving include a sensor modality on each joint of the robot generating a deep state representation at a time point stacking feature vectors of all sensor modalities in the modal dimension to form a total feature matrix at the current time point wherein is a total number of sensor modalities, and is represented as:​​​ ， Then the domain expert knowledge is introduced to encode a static kinematic prior adjacency matrix , which is a graph representing the prior correlation strength between different sensor modalities distributed on the joints of the robot, where the number of nodes is , and the value range of the matrix element is [0, 1], representing the correlation strength between modality and modality in the physical mechanism; The data-driven dynamic correlation attention mechanism is introduced again, and the total feature matrix constructed above is used The dynamic attention weight matrix is calculated by the self-attention mechanism , wherein the element represents the attention that the mode should pay to the mode under the data scene at the current time , and is expressed as: ， wherein, are learnable Query and Key projection matrices, respectively; is the dimension of the Query and Key vectors.

5. The method of claim 4, wherein, The knowledge and attention-guided state fusion and global state vector generation includes combining static prior knowledge with dynamic data-driven attention, through a learnable gating scalar dynamically trade off, the final fusion guidance matrix is represented as: ， wherein the gating scalar is generated by the neural network based on the current total feature matrix is obtained, the original feature matrix is weighted to realize the interaction and enhancement of information between modalities, represented as: To obtain a single final overall health state vector representing the robot at time , the enhanced feature matrix is integrated and reduced in dimension. First, a residual connection is performed with the original feature matrix to retain the original information, and then a multi-layer perception (MLP) network is used for nonlinear transformation and dimension compression to obtain the final global state vector , represented as: ， The MLP network adopts a bottleneck structure containing an activation function to force the model to learn a more informative compact representation. The final output global state vector is highly concentrated from all joints, all sensing modalities, and knowledge and data double-guided, mutually enhanced collaborative information.

6. The method of claim 5, wherein the method further comprises: The robot joint topology graph construction and spatial dependency dynamic modeling include, after obtaining the fused state vector of each joint at a single time point, all By treating each joint as a whole, a spatiotemporal map is constructed, in which the robot's... Each joint is abstracted as a set of nodes in a graph. The connections between nodes are represented by an edge set. This constitutes a static space diagram characterizing the potential interactions between robot joints. At each time step ,all The state vectors of each joint constitute a graph signal, which represents the feature matrix. Each of its rows It represents the joint. Furthermore, the state vector after fusing internal sensor information is represented as: ， Then the graph attention network GAT is used to aggregate the spatial information. For any joint node in the graph, the attention coefficient between the joint node and its neighbor joints is calculated through a single-layer feedforward network , is the first-order neighbor set of the joint , which is represented as: ， Among them, the weight vector , It is a learnable linear transformation matrix; This represents the vector concatenation operation; LeakyReLU is the modified linear unit activation function with leakage. Represents a node At any moment The input feature vector; Indicates time Lower Neighbor Joint The input feature vector; This represents the learnable linear transformation matrix used in a single-layer GAT to map the original node features to the attention subspace, and finally, the softmax function is used to adjust the joints. The attention coefficients of all neighbors are normalized to obtain the final attention weight. : ， Normalized attention weights Indicates at time Neighbor joints Information on joints The importance of joints Feature vector updated after GAT layer That is, the weighted sum of the features of all its neighboring joints based on attention weights, expressed as: ， wherein, is a nonlinear activation function, and after processing by the GAT layer, a state representation matrix fusing dynamic spatial neighborhood information is obtained .

7. The method of claim 6, wherein the method further comprises: The long-range temporal evolution prediction and future robot health state prediction include, after modeling the spatial dependence within each time slice, further capturing the evolutionary pattern of spatial state in the temporal dimension, using a temporal Transformer encoder to model temporal evolution, and incorporating past... Spatial augmented state sequence at each time step As input to the temporal Transformer, the correlation between the states of any two historical moments is calculated through a multi-layer multi-head self-attention mechanism, thus learning... The continuous overload state at any moment is the cause Key precursors to deviations in time trajectory, the output of the Time Transformer Each joint is provided with a deep representation that incorporates the entire historical spatiotemporal context at each historical moment; then, a linear prediction head is used to decode the output of the temporal Transformer at the last time step to predict the future. The robot's state at a given time is represented as: ， wherein, represents the feature matrix of the last time step of the output sequence of the temporal Transformer, the final output prediction state matrix is the spatio-temporal evolution rule based on the past time instants.

8. The method of claim 7, wherein the method further comprises: The predictive diagnosis and causal attribution based on the prediction result includes performing an anomaly score based on a prediction error, and first calculating a reconstruction error of a real state matrix and a predicted state matrix, wherein the real state matrix The first row vector of the real state matrix represents the real state of the joint, and the first row vector of the predicted state matrix represents the prediction of the joint state by the model, and for each joint node , the original anomaly score is represented as: ， To make the abnormal score adaptive to the historical fluctuation characteristics of each joint itself, a dynamic normalization method based on Z-score is adopted to calculate the final abnormal score, wherein, for each joint node a sliding time window with a length of is maintained to calculate the mean value and the standard deviation of the historical reconstruction error in the window, which is represented as: ， wherein, is a very small stability constant added to prevent the denominator from being zero, the resulting abnormal fraction represents the degree to which the prediction error of the current joint deviates from its historical normal level.

9. The method of claim 8, wherein, The predictive diagnosis and causal attribution based on the prediction results also include multi-dimensional attribution and interpretable diagnosis, and attribution analysis from three dimensions: sensing modality, space, and time. Modality attribution is based on a fusion guidance matrix. The row vectors represent the ability of each sensing mode to absorb information from other modes after fusion. The modal contribution score is defined and calculated using a matrix. The norm of the row sum is used to quantify the relative importance of each sensing modality at the moment of anomaly occurrence; spatial attribution includes, at the moment of anomaly occurrence, the relative importance of the joint nodes judged as anomalous. Check it from all neighbors Attention weights obtained at the location If a neighboring joint Corresponding weights If it is higher than other neighbors, then it is considered a joint. It causes joint problems The key spatial drivers of anomalies; temporal attribution includes when the model is predicting... When there is an anomaly in time, for Paying attention to the state of being at any given moment indicates that... Events occurring before a certain time step are important precursors to this anomaly; finally, when the abnormal score of any joint... Exceeding the threshold If so, a report will be generated automatically.

10. A robot anomaly prediction system based on multi-dimensional fusion and causal inference, characterized in that, The method comprises the following steps: a data acquisition module configured to obtain multi-joint sensing data of a robot; a multi-scale representation module configured to perform multi-scale deep state representation based on the obtained multi-joint sensing data of the robot, including time-frequency multi-scale decomposition and local fault mode coding; a fusion module configured to perform dynamic causal graph fusion based on the multi-scale deep state representation, including prior knowledge graph construction based on kinematic chains and data-driven dynamic correlation attention mechanism construction, knowledge and attention guided state fusion, and global state vector generation; a prediction module configured to perform hierarchical spatio-temporal dependence prediction based on the fused features, including robot joint topology graph construction and spatial dependence dynamic modeling, long-range time evolution prediction, and future robot health state prediction; an attribution module configured to perform predictive diagnosis and causal attribution based on the prediction result.

Images