A dynamic perception method for hand contact state of force-tactile spatiotemporal fusion

Abstract

This invention discloses a dynamic perception method for hand contact state through force-touch spatiotemporal fusion. This invention relates to the field of robot perception and control technology, including synchronous acquisition of multimodal sensor data, construction of a tactile spatiotemporal graph network, force-touch spatiotemporal fusion, and dynamic contact state recognition. The specific steps of the dynamic perception method are as follows: Step 1, synchronously acquire multi-channel pressure data from the tactile sensor array and global force / torque data from the force sensor. The advantages of this invention are: the graph topology construction method does not rely on a regular rectangular sensor array; as long as the spatial coordinates of each tactile node are provided, an adapted weighted graph model can be automatically generated. It can be easily transferred to dexterous hands or end effectors with different degrees of freedom and different sensor layouts, exhibiting good versatility. It employs sparse adjacency matrix storage and normalized graph convolution operations, combined with global pooling dimensionality reduction, to reduce single-frame processing latency and meet the real-time control requirements of robots.

Description

Technical Field

[0001] This invention relates to the field of robot perception and control technology, specifically to a method for dynamic perception of hand contact state through force-touch spatiotemporal fusion. Background Technology

[0002] Currently, with the widespread application of robotic dexterous hands and end effectors in fields such as medical rehabilitation, intelligent manufacturing, and home services, the perception of the contact state between the hand and the object has become a key technology for achieving precise grasping and safe operation. Existing commercial dexterous hands generally integrate tactile sensor arrays and wrist six-dimensional force / torque sensors. The applicant discovered through a search that a Chinese patent, "A Tactile Fingertip with Fingerprint Microstructure and a Method for Perceiving Tactile Information," with publication number "CN 119115994 B," mainly employs a hybrid structure of a flexible skin layer, a tactile sensing layer, and a rigid phalanx. Four sets of sensitive elements are embedded within the flexible skin, each set containing two sensitive elements that perceive dynamic and static signals, located at different skin layer depths. The fingertip surface is designed with raised fingerprint microstructures to further enhance tactile perception. This tactile fingertip can identify surface attributes based on the contact area and perceive gripping and contact states using the fingertip. However, this method lacks the ability to identify the contacting object. Therefore, we propose a method for dynamically perceiving hand contact states through a force-tactile spatiotemporal fusion approach. Summary of the Invention

[0003] The purpose of this invention is to provide a method for dynamically sensing the state of hand contact through a spatiotemporal fusion of force and touch.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a method for dynamic perception of hand contact state through force-touch spatiotemporal fusion, comprising synchronous acquisition of multimodal sensor data, construction of a tactile spatiotemporal graph network, force-touch spatiotemporal fusion, and dynamic contact state recognition. The specific steps of the dynamic perception method are as follows: Step 1: Synchronously acquire multi-channel pressure data from the tactile sensor array and global force / torque data from the force sensor; Step 2: Based on the spatial coordinates of each node of the tactile sensor, construct a weighted graph model, where nodes represent tactile units and edge weights are determined by spatial distance. Step 3: Encode the force data into global force field features and inject them as bias terms into all nodes in each layer of graph convolution. At the same time, calculate the dynamic gating factor for each node to achieve layer-by-layer spatiotemporal fusion of force and touch. Step 4: Input the fused spatiotemporal features into the temporal attention classifier and output the hand contact dynamic state.

[0005] As a further aspect of the present invention: in step two, the adjacency matrix of the weighted graph model adopts the Gaussian radial basis function as follows: ; in For distance scale parameters, For tactile nodes Three-dimensional spatial coordinate vectors on the surface of a dexterous hand. For tactile nodes The three-dimensional spatial coordinate vector, The Euclidean distance between two tactile nodes controls how quickly spatial affinity decreases with distance. For the natural constant An exponential function with base 1 treats each node in the tactile sensor array as a vertex of a graph. Its initial characteristic is the pressure value at the current moment. and three-dimensional spatial coordinates .

[0006] As a further aspect of the present invention: in step three, the dynamic gating factor is: ; in, For at any time tactile nodes The force-touch dynamic gating factor, with a value range of (0,1), It is an sigmoid function that maps any real number to the interval (0,1). For learnable weight row vectors, For a moment The force / torque vector measured by the force sensor. For a moment tactile nodes The measured pressure value, This is the bias term for the gating factor.

[0007] As a further aspect of the present invention: in step three, the formula for graph convolution update is: ; in, For the first After layer graph convolution, nodes eigenvectors, For the first After layer graph convolution, nodes eigenvectors, For activation function, For the first The learnable weight matrix of the layer is used to transform the aggregated neighborhood features. For nodes The set of neighboring nodes, For elements of an adjacency matrix with self-loops, For degrees with self-loop, As the symmetry normalization factor, For the first Layer force injection matrix, For a moment The global force-encoding feature vector, For the first The layer's gated transformation matrix is ​​used to map the original features of the nodes to the same dimension as the output.

[0008] The first step: Collect "neighbor pressure information" from adjacent tactile nodes to achieve spatial propagation; The second item: Global force perception features are used as a bias term and forcibly injected into each node (this is the core innovation, which is equivalent to constantly reminding each tactile point of the current overall force). The third item: The dynamic gating factor adjusts the influence of the original features of this node to prevent local noise from misleading.

[0009] As a further aspect of the present invention: In step one, a high-density flexible tactile sensor array is deployed on the surface of the dexterous hand and the end effector, while a six-dimensional force / torque sensor is installed at the wrist and the base. A synchronous trigger signal is generated using the same hardware clock source, so that the tactile array and the force sensor collect data simultaneously at the same sampling frequency. After collection, the tactile signal is subjected to temperature compensation, baseline drift elimination, and single-point median filtering, while the force signal is subjected to low-pass filtering and self-weight bias subtraction. Finally, the timestamp, tactile pressure matrix, and six-dimensional force vector are packaged into a multimodal data frame and sent to a circular buffer for subsequent real-time processing.

[0010] As a further aspect of the present invention: In step two, each effective sensing unit in the tactile sensor array is defined as a node in the graph, and the precise spatial coordinates (x, y, z) of each node on the hand surface are obtained according to the CAD model of the dexterous hand. For each pair of nodes, the spatial Euclidean distance is calculated, and the Gaussian radial basis function is used to map the distance into edge weights. The closer the distance, the closer the weight is to 1. When the distance is greater than a threshold, the weight approaches 0. All non-zero weights are stored to form a sparse adjacency matrix, and a self-loop is added to each node. Then, the degree of each node is calculated, and each edge is symmetrically normalized to prevent the numerical instability of the deep network. For rigid hand structures, the graph model can remain statically unchanged. For soft hands, the coordinates and weights are recalculated every few frames to achieve adaptive topology.

[0011] As a further aspect of the present invention: In step three, the original six-dimensional force / torque vector is first input into a small multilayer perceptron, which outputs a global force field feature vector with the same dimension as the graph convolutional hidden layer. Simultaneously, for each tactile node, its current pressure value is concatenated with the global force vector, and then processed through a single fully connected layer. The function calculates a dynamic gating factor between 0 and 1. .

[0012] As a further aspect of the present invention: In step three, a multi-layer graph convolutional network is constructed, and in each layer, the following steps are performed sequentially: the features of each node and its neighbors in the previous layer are weighted and summed according to the normalized edge weights; the global force field feature vector is linearly transformed and broadcast to the aggregated result of all nodes; and the features of the node itself in the previous layer are linearly transformed and multiplied by a gating factor. Similarly, the features are added to the aggregation result, and finally a bias is added and the output features of this layer are obtained through the ReLU activation function. Global pooling is performed on all node features of the last layer to obtain a one-dimensional global vector representing the current state of the whole. The global vectors of consecutive T frames are stacked along the time dimension to form a spatiotemporal feature sequence.

[0013] As a further aspect of the present invention: in step four, the spatiotemporal feature sequence obtained in the previous step is input into the temporal attention module to obtain the enhanced fusion global vector, and then sent to a classification head composed of 2-3 fully connected layers to output the original score for the preset state category.

[0014] Compared with the prior art, the beneficial effects of the present invention by adopting the above technical solution are as follows: 1. This invention uses a graph topology construction method that does not rely on a regular rectangular sensor array. It can automatically generate an adapted weighted graph model as long as the spatial coordinates of each tactile node are provided. It can be easily transferred to dexterous hands or end effectors with different degrees of freedom and different sensor layouts. It has good versatility. It uses sparse adjacency matrix storage and normalized graph convolution operation, combined with global pooling dimensionality reduction, to reduce single-frame processing latency and meet the real-time control requirements of robots. 2. This invention can intuitively reflect the degree of dependence of each tactile node on force information at the current moment through dynamic gating factors, and can generate an attention heatmap accordingly, clearly showing the dominant modal contribution of the contact area, providing a transparent basis for decision backtracking and system debugging in safety-critical tasks; 3. Through the layer-by-layer spatiotemporal fusion of force and touch, the system can capture the synergistic precursors between local pressure gradient changes and global force perception decline. In the standard slip detection experiment, it improves the early warning time of the later fusion method, reserves sufficient response window for upper-level control, and the information of adjacent healthy nodes can still play a compensatory role through graph convolution. At the same time, force perception is injected into each layer as a global bias, which effectively suppresses the misleading effect of local tactile anomalies on the overall judgment. Attached Figure Description

[0015] Figure 1 This is a flowchart of the hand contact state dynamic sensing method in an embodiment of the present invention. Detailed Implementation

[0016] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0017] Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0018] Please see the appendix Figure 1 This invention discloses a method for dynamically sensing hand contact states through force-touch spatiotemporal fusion, comprising synchronous acquisition of multimodal sensor data, construction of a tactile spatiotemporal graph network, force-touch spatiotemporal fusion, and dynamic contact state recognition. The specific steps of the dynamic sensing method are as follows: Step 1: Synchronously acquire multi-channel pressure data from the tactile sensor array and global force / torque data from the force sensor; Step 2: Based on the spatial coordinates of each node of the tactile sensor, construct a weighted graph model, where nodes represent tactile units and edge weights are determined by spatial distance. Step 3: Encode the force data into global force field features and inject them as bias terms into all nodes in each layer of graph convolution. At the same time, calculate the dynamic gating factor for each node to achieve layer-by-layer spatiotemporal fusion of force and touch. Step 4: Input the fused spatiotemporal features into the temporal attention classifier and output the hand contact dynamic state.

[0019] In one embodiment of the present invention: In step two, the adjacency matrix of the weighted graph model adopts the Gaussian radial basis function as follows: ; in For distance scale parameters, For tactile nodes Three-dimensional spatial coordinate vectors on the surface of a dexterous hand. For tactile nodes The three-dimensional spatial coordinate vector, The Euclidean distance between two tactile nodes controls how quickly spatial affinity decreases with distance. For the natural constant An exponential function with base 1 treats each node in the tactile sensor array as a vertex of a graph. Its initial characteristic is the pressure value at the current moment. and three-dimensional spatial coordinates .

[0020] In one embodiment of the present invention: in step three, the dynamic gating factor is: ; in, For at any time tactile nodes The force-touch dynamic gating factor, with a value range of (0,1), It is an sigmoid function that maps any real number to the interval (0,1). For learnable weight row vectors, For a moment The force / torque vector measured by the force sensor. For a moment tactile nodes The measured pressure value, This is the bias term for the gating factor.

[0021] In one embodiment of the present invention: in step three, the formula for graph convolution update is: ; in, For the first After layer graph convolution, nodes eigenvectors, For the first After layer graph convolution, nodes eigenvectors, For activation function, For the first The learnable weight matrix of the layer is used to transform the aggregated neighborhood features. For nodes The set of neighboring nodes, For elements of an adjacency matrix with self-loops, For degrees with self-loop, As the symmetry normalization factor, For the first Layer force injection matrix, For a moment The global force-encoding feature vector, For the first The layer's gated transformation matrix is ​​used to map the original features of the nodes to the same dimension as the output.

[0022] The first step: Collect "neighbor pressure information" from adjacent tactile nodes to achieve spatial propagation; The second item: Global force perception features are used as a bias term and forcibly injected into each node (this is the core innovation, which is equivalent to constantly reminding each tactile point of the current overall force). The third item: The dynamic gating factor adjusts the influence of the original features of this node to prevent local noise from misleading.

[0023] Example: Contact state sensing system for three-finger dexterity hand: Hardware configuration: 1. Dexterous Hand: A three-finger underactuated dexterous hand, with tactile sensor arrays covering the fingertip and fingertip area of ​​each finger. The array size of a single finger is 16×16, and there are a total of 768 effective tactile nodes across the three fingers, with a spatial resolution of approximately 0.5mm. Tactile sensor: It adopts a flexible piezoresistive thin film sensor with a sampling rate of 2000Hz and outputs a 12-bit AD value, which is calibrated and converted into a pressure value of 0kPa-200kPa; Force sensor: A six-dimensional force / torque sensor (ATIMini45) is installed on the dexterous wrist, with a measurement range of ±145N / ±6N·m and a sampling rate of 2000Hz synchronized with tactile feedback; Computing platform: Edge computing module, used to run graph neural networks in real time; Synchronization method: The FPGA generates a 2000Hz square wave synchronous trigger signal, which simultaneously triggers the AD acquisition of the tactile array and the data output of the force sensor, with the timestamp deviation of each channel being less than 0.1ms; II. Implementation Steps: Step 1: Synchronous acquisition of multimodal sensor data: After the system is powered on, the FPGA control unit outputs a synchronization pulse at a frequency of 2000Hz. When each pulse arrives, the tactile acquisition circuit scans all 768 sensing nodes in sequence, converting the analog voltage into a digital pressure value through a multiplexer and a 16-bit ADC. At the same time, the force sensor outputs six-dimensional force / torque data through the SPI interface. The data preprocessing module performs temperature compensation and baseline drift elimination on the tactile pressure value, performs Butterworth low-pass filtering with a cutoff frequency of 200Hz on the six-dimensional force signal, and deducts the sensor's own weight. Then, the timestamp, the 768-dimensional tactile pressure vector, and the 6-dimensional force / torque vector are encapsulated into a data frame and stored in a circular buffer with a length of 200 frames. Step 2: Construct a weighted graph model based on spatial coordinates: Read the CAD model file of the dexterous hand, extract the three-dimensional coordinates of each tactile node on the hand surface, and calculate the Euclidean distance for each pair of nodes, using all 768 nodes as vertices. Set scale parameters mm (approximately 1.5 times the average distance between adjacent nodes), according to the Gaussian radial basis function Calculate edge weights and set weight thresholds. =0.05, edges with values ​​less than 0.05 are discarded, leaving only about 32,000 edges (the original complete graph had about 290,000 edges), forming a sparse adjacency matrix. A self-loop is then added to each node (let = 0.05). =1); Step 3: Layer-by-layer spatiotemporal fusion of force and touch: First, we build a force encoder: an MLP containing two fully connected layers, with an input dimension of 6, a hidden layer dimension of 64 (ReLU activation), and an output dimension of 64 (global force field feature vector). During training, the MLP is jointly optimized with the graph convolutional network; For each frame of data, calculate the dynamic gating factor for each tactile node: [This involves] the six-dimensional force vector of the current frame... (6-dimensional) and the pressure value of the node (1-dimensional) concatenation into a 7-dimensional vector, fed into a fully connected layer (input 7, output 1) with sigmoid activation, yields The value range is (0,1); Construct a 3-layer graph convolutional network (with hidden layer dimensions of 64, 128, and 64 respectively), and the computation process of each layer (taking the l-th layer as an example): Input: Feature matrix of all nodes in the previous layer (The first layer input is the original pressure value, which is expanded to 64 dimensions, where the 0th dimension is pressure and the other 63 dimensions are 0). Neighborhood aggregation: using a pre-calculated normalized adjacency matrix ,calculate The aggregated neighborhood feature matrix is ​​obtained. Force bias injection: injecting global force field feature vectors Learnable matrix After a 64×64 dimension transformation, the broadcast is added to the aggregated features of each node; Gated residuals: The input features of a node are... Learnable matrix After transformation (64×64 dimensions), and with the gating factor vector (768-dimensional) Element-by-element multiplication (broadcast); Adding the three together, plus the bias Activated by ReLU ; After the three convolutional layers are completed, the node features (768×64) of the last layer are subjected to global average pooling along the node dimension to obtain a 64-dimensional global vector. The global vectors of T=16 frames (corresponding to an 8ms time window) are stacked continuously to form a 16×64 spatiotemporal feature sequence. Step 4: Dynamic Contact Status Recognition The spatiotemporal feature sequence is input into the temporal attention module: first, a learnable positional encoding (16×64) is added, then it is passed through a multi-head self-attention layer (4 heads, 64 output dimensions), followed by a gated recurrent unit (GRU, 64 hidden layers), finally resulting in a 16×64 augmented sequence. The output of the last frame is taken as the fusion global vector (64 dimensions). The classification head consists of two fully connected layers (64→32→6), outputting the raw scores of 6 categories. Temperature scaling is used (T=1.5 is determined on the validation set). After dividing the score by 1.5, Softmax is performed to obtain the calibration probability. Post-processing uses a sliding window voting with a length of 5 frames to output the majority state and its average confidence. III. Training and Testing: Data collection: Using the three-finger dexterity hand, grasp 10 different objects (smooth metal ball, rough wooden block, flexible sponge, glass, clay, etc.). Perform 500 grasp-lift-slide-release actions for each object. The total data volume is about 500,000 frames. Six states are labeled: stable contact (0), slight slippage trend (1), significant slippage (2), about to be released (3), collision impact (4), flexible settlement (5). The training / validation / test sets are divided into 8:1:1. Train for 50 epochs on Jetson XavierNX using the Adam optimizer with an initial learning rate of 0.001 and a batch size of 32. IV. Implementation Results: Recognition accuracy: The overall accuracy of the test set was 97.3%, with an average F1-score of 0.96 for the "significant slip" and "about to slip" states. Warning lead time: For the final grasping process that results in slippage, this method can detect the "significant slippage" state on average 158ms before the contact force begins to decrease, which is a significant improvement compared to pure haptic graph networks that do not integrate force perception (warning lead time of only 42ms) or simple late-stage fusion methods (warning lead time of 65ms). Fault tolerance test: Randomly setting 30% of the haptic node outputs to a fixed zero value (simulating damage), the overall system accuracy still reached 92.1%, while the accuracy of the pure haptic graph network dropped to 68.4%. Real-time performance: The average processing time per frame is 4.2ms (including data reading, graph convolution, and temporal attention), which is much higher than the sampling period of 0.5ms, meeting the requirements of real-time closed-loop control; V. Comparative Experiment: On the same three-finger dexterous hand platform, the performance of this invention was evaluated against the following three comparative methods: Method A: Use only the maximum and mean values ​​of tactile pressure in a single frame as features + SVM classifier; Method B: Uses a haptic graph network but without force injection, only graph convolution in step two, without force bias and gating in step three; Method C: Post-fusion, extracting tactile and force features separately, and finally splicing them into MLP classification; The above embodiments demonstrate that the present invention significantly improves slip warning time, noise robustness, and interpretability.

[0024] While the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Any variations and modifications can be made by those skilled in the art without departing from the spirit and scope of the invention. Therefore, any modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention, without departing from the scope of the invention, fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for dynamic perception of hand contact state through force-touch spatiotemporal fusion, comprising synchronous acquisition of multimodal sensor data, construction of a tactile spatiotemporal graph network, force-touch spatiotemporal fusion, and dynamic contact state recognition, characterized in that: The specific steps of the dynamic sensing method are as follows: Step 1: Synchronously acquire multi-channel pressure data from the tactile sensor array and global force / torque data from the force sensor; Step 2: Based on the spatial coordinates of each node of the tactile sensor, construct a weighted graph model, where nodes represent tactile units and edge weights are determined by spatial distance. Step 3: Encode the force data into global force field features and inject them as bias terms into all nodes in each layer of graph convolution. At the same time, calculate the dynamic gating factor for each node to achieve layer-by-layer spatiotemporal fusion of force and touch. Step 4: Input the fused spatiotemporal features into the temporal attention classifier and output the hand contact dynamic state.

2. The method for dynamically sensing hand contact state through force-touch spatiotemporal fusion according to claim 1, characterized in that: In step two, the adjacency matrix of the weighted graph model uses the Gaussian radial basis function as follows: ； in For distance scale parameters, For tactile nodes Three-dimensional spatial coordinate vectors on the surface of a dexterous hand. For tactile nodes The three-dimensional spatial coordinate vector, The Euclidean distance between two tactile nodes controls how quickly spatial affinity decreases with distance. For the natural constant An exponential function with base 1 treats each node in the tactile sensor array as a vertex of a graph. Its initial characteristic is the pressure value at the current moment. and three-dimensional spatial coordinates .

3. The method for dynamic perception of hand contact state through force-touch spatiotemporal fusion according to claim 2, characterized in that: In step three, the dynamic gating factor is: ； in, For at any time tactile nodes The force-touch dynamic gating factor, with a value range of (0,1), It is an sigmoid function that maps any real number to the interval (0,1). For learnable weight row vectors, For a moment The force / torque vector measured by the force sensor. For a moment tactile nodes The measured pressure value, This is the bias term for the gating factor.

4. The method for dynamic perception of hand contact state through force-touch spatiotemporal fusion according to claim 3, characterized in that: In step three, the formula for graph convolution update is: ； in, For the first After layer graph convolution, nodes eigenvectors, For the first After layer graph convolution, nodes eigenvectors, For activation function, For the first The learnable weight matrix of the layer is used to transform the aggregated neighborhood features. For nodes The set of neighboring nodes, For elements of an adjacency matrix with self-loops, For degrees with self-loop, The symmetry normalization factor is For the first Layer force injection matrix, For a moment The global force-encoding feature vector, For the first The layer's gated transformation matrix is ​​used to map the original features of the nodes to the same dimension as the output.

5. The method for dynamic perception of hand contact state through force-touch spatiotemporal fusion according to claim 4, characterized in that: In step one, a high-density flexible tactile sensor array is deployed on the surface of the dexterous hand and the end effector. At the same time, a six-dimensional force / torque sensor is installed at the wrist and the base. A synchronous trigger signal is generated using the same hardware clock source, so that the tactile array and the force sensor collect data simultaneously at the same sampling frequency. After collection, the tactile signal is subjected to temperature compensation, baseline drift elimination, and single-point median filtering. The force signal is subjected to low-pass filtering and self-weight bias subtraction. Finally, the timestamp, tactile pressure matrix, and six-dimensional force vector are packaged into a multimodal data frame and sent to a circular buffer for subsequent real-time processing.

6. The method for dynamically sensing hand contact state through force-touch spatiotemporal fusion according to claim 5, characterized in that: In step two, each effective sensing unit in the tactile sensor array is defined as a node in the graph, and the precise spatial coordinates (x, y, z) of each node on the hand surface are obtained according to the CAD model of the dexterous hand. For each pair of nodes, the spatial Euclidean distance is calculated, and the Gaussian radial basis function is used to map the distance into edge weights.

7. The method for dynamic perception of hand contact state through force-touch spatiotemporal fusion according to claim 6, characterized in that: In step three, the original six-dimensional force / torque vector is first input into a small multilayer perceptron, which outputs a global force field feature vector with the same dimension as the graph convolutional hidden layer. Simultaneously, for each tactile node, its current pressure value is concatenated with the global force vector, and then processed through a single fully connected layer. The function calculates a dynamic gating factor between 0 and 1. .

8. The method for dynamically sensing hand contact state through force-touch spatiotemporal fusion according to claim 7, characterized in that: In step three, a multi-layer graph convolutional network is constructed. In each layer, the following steps are performed sequentially: the features of each node and its neighbors in the previous layer are weighted and summed according to the normalized edge weights; the global force field feature vector is linearly transformed and broadcast to the aggregated result of all nodes; and the features of the node itself in the previous layer are linearly transformed and multiplied by a gating factor. Similarly, the features are added to the aggregation result, and finally a bias is added and the output features of this layer are obtained through the ReLU activation function. Global pooling is performed on all node features of the last layer to obtain a one-dimensional global vector representing the current state of the whole. The global vectors of consecutive T frames are stacked along the time dimension to form a spatiotemporal feature sequence.

9. A method for dynamically sensing hand contact state through force-touch spatiotemporal fusion according to claim 8, characterized in that: In step four, the spatiotemporal feature sequence obtained in the previous step is input into the temporal attention module to obtain the enhanced fusion global vector, and then sent to a classification head composed of 2-3 fully connected layers to output the original score for the preset state category.

Images