Event-driven tactile perception method and system based on pulse graph neural network

Abstract

The invention relates to an event-driven tactile sensing method and system based on a pulse graph neural network, and belongs to the technical field of tactile sensing and robot control. The method comprises the following steps: S1, extracting main features from data collected by an event-driven touch sensor; s2, encoding the main features extracted from the tactile sensor data; s3, using a spiking neural network to further learn feature embedding; s4, the encoded features are processed through a graph neural network; and S5, identifying the category or state of the target object by adopting a voting mechanism. According to the scheme, through the graph neural network, the graph based on the space structure of the tactile sensing unit can be automatically constructed, and local and global features are efficiently extracted; the pulse neural network is used for processing event-driven data, so that the energy consumption is remarkably reduced and the calculation efficiency is improved; the method has strong generalization ability, can adapt to different tactile sensor configurations and application scenes, and has excellent performance when processing real-time tactile perception tasks.

Description

Technical Field

[0001] The present invention belongs to the technical field of tactile perception and robot control, in particular to the technical field of event-driven tactile perception methods based on impulse graph neural networks, and relates to an event-driven tactile perception method and system based on impulse graph neural networks. Background Art

[0002] Tactile sensing technology is one of the key means for robots to interact with the external environment. Traditional tactile sensors capture information such as pressure, temperature, and vibration through synchronous output, helping robots complete tasks such as object recognition, grasping, and manipulation. However, the frame data output mode of these sensors easily leads to high energy consumption and data processing delays, which makes it difficult to meet the needs of real-time applications. In order to solve this problem, event-driven tactile sensors have emerged in recent years. They can output data asynchronously based on events, greatly reducing latency and improving energy efficiency. This type of sensor has been applied to tasks such as object recognition and grasping stability detection. Related studies have shown that event-driven tactile sensors perform well in dynamic environments.

[0003] Currently, the main methods for processing tactile data are based on convolutional neural networks (CNNs). These methods require converting tactile data into a regular grid form before feature extraction. However, the distribution of sensory units (called taxels) in tactile sensors is usually irregular, and forcibly mapping them into a grid will introduce redundant calculations and information loss. In addition, event-driven tactile data is event-based discrete data, which is incompatible with traditional frame data processing methods. Therefore, existing CNN-based solutions not only consume large computing resources, but also find it difficult to fully utilize the advantages of event-driven sensors.

[0004] To this end, researchers began to apply Graph Neural Networks (GNN) to the field of tactile perception to better represent and process irregularly distributed taxel data. TactileGCN is an early graph convolutional network model that extracts features from tactile data by constructing the taxel of a tactile sensor as a graph node and generating graph edges based on their spatial proximity. However, TactileGCN has problems such as poor generalization ability, high energy consumption, and excessive demand for large-scale data, and it is difficult to adapt to changes in new objects or scenes.

[0005] Another type of neural network, the Spiking Neural Network (SNN), processes spatiotemporal data by simulating the working mode of biological neurons. This type of network has significant energy efficiency advantages when processing event-driven data and can achieve low-latency data processing. Models such as SLAYER have achieved some success in using SNN to process event-driven visual data. However, SNN still faces many challenges in processing graph-structured data, such as how to integrate spiking dynamics with graph convolution operations, and how to maintain training stability in deep networks.

[0006] In general, the existing technologies have the following main problems when processing event-driven tactile data: 1. Traditional CNN methods are difficult to effectively process irregular taxel structures, 2. GNN-based methods have high energy consumption and poor generalization ability, and 3. Although SNN has advantages in processing spatiotemporal data, it has not yet been well combined with graph structure data. Therefore, it is necessary to propose a new method that can combine the advantages of pulse neural networks and graph neural networks, which can not only process irregular tactile data structures, but also efficiently process event-driven data. Summary of the invention

[0007] In view of this, the purpose of the present invention is to provide an event-driven tactile perception method and system based on a pulse graph neural network, which can effectively process irregular tactile sensor unit data and improve the processing efficiency of event-driven data. The method and system organize the tactile data into a graph structure and utilize the ability of the pulse neural network to efficiently process event-driven data, thereby solving the computational bottleneck problem of the prior art in processing irregular tactile data structures and event-driven data.

[0008] In order to achieve the above object, the present invention provides the following technical solutions:

[0009] An event-driven tactile perception method based on a pulse graph neural network comprises the following steps:

[0010] S1. Collect data through an event-driven tactile sensor and extract main features from the data collected by the event-driven tactile sensor;

[0011] S2, encoding the main features extracted from the tactile sensor data;

[0012] S3, using Spiking Neural Network (SNN) to further learn feature embedding;

[0013] S4. Process the encoded features through Graph Neural Networks (GNN);

[0014] S5. Use a voting mechanism to identify the category or state of the target object.

[0015] Furthermore, in step S1, main features are extracted when acquiring tactile sensor data, and relevant features are extracted from 39 tactile element taxels in total, including pressure, temperature, and vibration data of each taxel; by using a graph structure, these data are converted into a graph that can describe the spatial relationship between tactile elements, thereby retaining the spatial locality and globality of the data.

[0016] Further, in step S1, a graph G=(V, E) is used to represent the tactile data, where V is a set of nodes of the graph, representing each Taxel, and E represents a set of edges between nodes; the process of constructing the graph includes but is not limited to being implemented by the following two methods:

[0017] Manual method: Manually determine the edge by the physical distance between Taxels: set a threshold d th , when the physical distance between Taxeli and Taxel j satisfies d(u i ,u j ) <d th , establish an edge between nodes Taxel i and Taxel j, which can be expressed as:

[0018]

[0019] Where d(u i ,u j ) is calculated using the Euclidean distance:

[0020]

[0021] Here, x i ,y j Represents the two-dimensional coordinates of Taxel;

[0022] k-nearest neighbor method: In order to improve the efficiency of automatic graph construction, the k-nearest neighbor algorithm is used to automatically construct edges; in this method, each Taxel is connected to its k nearest neighbors in space, that is:

[0023]

[0024] Here kNN(u i ) means Tax u i The k nearest neighbors of ;

[0025] Through these construction methods, the generated maps can directly reflect the spatial distribution of tactile sensors and the relationship between individual tactile elements.

[0026] Furthermore, in step S2, pulse encoding is performed after acquiring the tactile data. In order to improve computing efficiency, the LIF (leaky integration and firing) model is used for encoding. Each tactile data is time-processed through this model, and a pulse is triggered when the membrane potential reaches the set threshold. Compared with the traditional convolutional neural network method, the LIF model can effectively process event-driven asynchronous data and significantly improve the reasoning speed.

[0027] Furthermore, in step S2, a LIF neuron model is used. The LIF model accumulates input pulses and triggers a pulse signal when the accumulated potential exceeds a certain threshold. The dynamic equation of the LIF neuron is:

[0028]

[0029] Where V(t) represents the membrane potential of the neuron, I(t) is the input current, and τ is the time constant of the membrane potential;

[0030] When the membrane potential V(t) exceeds the set firing threshold V th When the neuron fires a pulse, it resets the membrane potential to V reset ; expressed as:

[0031]

[0032] Through this mechanism, LIF neurons can effectively process time series information and generate sparse spike outputs.

[0033] Furthermore, in steps S3 and S4, feature embedding learning is performed by combining SNN and GNN. The dynamic characteristics of pulse events are captured in the time dimension by SNN, and the graph structure data is processed by GNN, so that the system can extract complex spatiotemporal relationships in the tactile map.

[0034] Further, in step S5, a voting mechanism is used to determine the final output of each category. The voting mechanism is based on the pulse emission of LIF neurons, accumulates the number of pulses of each neuron in the time window, and selects the category with the largest number of pulses as the final classification result; the classification result is expressed as:

[0035]

[0036] Among them, k (t) represents the number of output pulses of category k at time step t, and T is the length of the time window.

[0037] The present invention also provides an event-driven tactile perception system based on a pulse graph neural network.

[0038] The beneficial effects of the present invention are:

[0039] 1. Efficient processing of irregular tactile sensor unit layouts: Different from traditional CNNs, this invention can effectively deal with the complex topological structure of tactile sensors by modeling the tactile sensor unit as a graph structure and combining the advantages of graph neural networks in processing irregular layout data. It has good adaptability when facing different sensor configurations.

[0040] 2. Low energy consumption and real-time response: Due to the use of spiking neural networks, the system has significant energy consumption advantages when processing event-driven data and can achieve low-latency real-time response. Compared with traditional frame-based neural network methods, spiking neural networks only emit pulses when tactile events occur, avoiding redundant calculations caused by frame data, thereby significantly reducing system energy consumption.

[0041] 3. Dense connection improves information fluidity and generalization ability: This invention introduces a dense connection mechanism to reuse information between layers of the network, ensuring stronger gradient fluidity and information transmission efficiency. Compared with existing cascade networks, dense connections reduce information loss and enhance the generalization ability of the model, especially when processing complex multi-dimensional tactile data.

[0042] 4. High classification accuracy: The present invention adopts a classification method based on a voting mechanism, and performs classification judgment by counting the pulse emission frequency of neurons in the output layer, which significantly improves the accuracy of classification. In a variety of event-driven tactile perception tasks, the present invention is significantly better than the existing technology in classification accuracy. For example, in some tactile perception scenarios, the classification accuracy rate is improved by about 10%-15%.

[0043] 5. Strong scalability and adaptability to different application scenarios: Since the present invention does not rely on fixed sensor configurations or specific environmental structures, the system can be well generalized to different tactile perception applications. The irregular graph structure processed by its graph neural network can flexibly adapt to various new sensor layouts, and the pulse neural network ensures the application effect in different real-time tactile scenarios.

[0044] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0046] Figure 1 A schematic diagram showing the comparison of recognition accuracy between the present invention and the benchmark method;

[0047] Figure 2 Use T-SNE visualization for the ST-MINIST dataset;

[0048] Figure 3 It is the confusion matrix of different models under the ST-MINIST dataset;

[0049] Figure 4 A comparison diagram of the time delay between the present invention and the baseline method;

[0050] Figure 5 A comparison chart of the present invention and the baseline method under the K nearest neighbor mapping method;

[0051] Figure 6 Comparison diagram of different models learning feature density on the ST-MNIST dataset;

[0052] Figure 7 It is an overall summary diagram of the method of the present invention;

[0053] Figure 8 It is the overall framework diagram of the present invention. DETAILED DESCRIPTION

[0054] The technical solution of the present invention is described in detail below with reference to the accompanying drawings.

[0055] The present invention solves the computational bottleneck problem of the prior art in processing irregular tactile data structures and event-driven data by organizing tactile data into a graph structure and utilizing the ability of pulse neural networks to efficiently process event-driven data. Figure 7 This is an overall summary of the method of the present invention. Figure 8 It is the overall framework diagram of the present invention.

[0056] like Figure 7 As shown, it takes event-driven tactile sensing data as input and organizes it into a tactile graph. The present invention uses the LIF model, which consists of an adaptive graph convolution (TAGConv) layer and a fully connected (FC) layer. Figure 8 The overall structure of the present invention is shown, which consists of a TAGConv layer, two Deep blocks, a transition layer (also a TAGConv layer), two FC layers and a voting layer. Each Deep block consists of three DeepLayer layers, which are connected by dense connections. Please note that each TAGConv or FC layer is followed by LIF activation and event-based batch normalization layers, which are not shown in the figure for clarity. The following will describe the various steps of the method in detail, including the derivation of the formula and its physical meaning.

[0057] First, tactile data is collected through the NeuTouch sensor. The sensor consists of multiple tactile elements (Taxels) distributed on an irregular grid. Each Taxel generates events at irregular time intervals. These events represent the force, pressure or vibration applied to the tactile element by the environment. In order to effectively process these irregularly structured tactile data, they must be constructed as a graph structure.

[0058] In this embodiment, a graph G = (V, E) is used to represent tactile data, where V is a set of nodes of the graph, representing each Taxel, and E represents a set of edges between nodes. The process of constructing the graph can be implemented by the following two methods:

[0059] Manual method: Manually determine the edge by the physical distance between Taxels. Set a threshold d th , when the physical distance between Taxeli and Taxel j satisfies d(u i ,u j ) <d th , establish an edge between nodes Taxel i and Taxel j. This can be expressed as:

[0060]

[0061] Where d(u i ,u j ) is usually calculated using the Euclidean distance:

[0062]

[0063] Here, x i ,y j Represents the two-dimensional coordinates of Taxel.

[0064] K-nearest neighbor method: In order to improve the efficiency of automatic graph construction, the K-nearest neighbor algorithm is also used to automatically construct edges. In this method, each Taxel is connected to its k nearest neighbors in space. That is:

[0065]

[0066] Here kNN(u i ) means Tax u i The k nearest neighbors of .

[0067] Through these construction methods, the generated maps can directly reflect the spatial distribution of tactile sensors and the relationship between individual tactile elements.

[0068] On the constructed tactile graph, a topology adaptive graph convolution layer (TAGConv) is used to extract the features of the graph. Different from the traditional graph convolution, TAGConv captures the multi-hop information between nodes by introducing multi-order adjacency matrix powers. This operation can be expressed by the following formula:

[0069]

[0070] Among them, H (l) is the feature matrix of the lth layer, A k is the normalized adjacency matrix of the graph, α k is the learned coefficient, and K is the maximum power of the adjacency matrix. This formula captures the local and global information of the graph by propagating node information to nodes within K hops.

[0071] The features extracted by the TAGConv layer are input into the deep module (DeepBlock) for further processing. Each DeepBlock consists of multiple deep layers (DeepLayer), each of which contains two TAGConv layers and two event-based batch normalization (EBN) layers. EBN is introduced to prevent the gradient from disappearing or exploding and to adapt to event-driven data in spiking neural networks.

[0072] The SNN in the present invention uses the LIF neuron model. The LIF model accumulates the input pulses and triggers a pulse signal when the accumulated potential exceeds a certain threshold. The dynamic equation of the LIF neuron is:

[0073]

[0074] Among them, V(t) represents the membrane potential of the neuron, I(t) is the input current, and τ is the time constant of the membrane potential.

[0075] When the membrane potential V(t) exceeds the set firing threshold V th When the neuron fires a pulse, it resets the membrane potential to V reset . This can be expressed as:

[0076]

[0077] Through this mechanism, LIF neurons can effectively process time series information and generate sparse spike outputs.

[0078] In order to better capture features, the model introduces dense connections between deep layers in each DeepBlock. This connection method allows the output of each layer to be directly passed to all subsequent layers, thereby achieving feature reuse and better flow of information. The mathematical expression of this operation is:

[0079]

[0080] in, Represents a splicing operation.

[0081] After passing through multiple TAGConv layers and deep modules, the extracted features are input into the fully connected layer (FC) for dimensionality reduction and classification. The operation of the fully connected layer can be described by the following formula:

[0082] h=Wx+b#(8)

[0083] Among them, W is the weight matrix, x is the input feature, and b is the bias term.

[0084] The final classification layer uses a voting mechanism to determine the final output of each category. The voting mechanism is based on the pulse emission of LIF neurons, accumulating the number of pulses of each neuron in the time window, and selecting the category with the most pulses as the final classification result. The classification result can be expressed as:

[0085]

[0086] Among them, k (t) represents the number of output pulses of category k at time step t, and T is the length of the time window.

[0087] Since the pulse emission function of the spiking neural network is discontinuous, the traditional gradient descent method cannot be directly applied to the spiking neural network. To solve this problem, the surrogate gradient method is used to approximate the derivative of the pulse function. Specifically, a rectangular function is used to approximate the pulse derivative:

[0088]

[0089] Here, a is a parameter that controls the width of the rectangle. This approximation allows for efficient error backpropagation during training.

[0090] The loss function of the entire model consists of two parts: supervision loss and graph structure regularization loss. Supervision loss uses mean squared error (MSE) to measure the gap between the predicted output and the actual label:

[0091]

[0092] Among them, y i is the true label of the i-th sample, is the predicted value of the model.

[0093] At the same time, in order to ensure the structural characteristics of the graph, the graph structure regularization loss is introduced:

[0094]

[0095] Among them, A ij is the adjacency matrix, h i ,h j is the feature representation of nodes i and j.

[0096] The total loss function is:

[0097]

[0098] Among them, λ is the regularization coefficient.

[0099] In this embodiment, PyTorch was used for the experiment, and the Adam optimizer was used to train the model. The initial learning rate was set to 0.001 and the number of training iterations was 100. In order to ensure the reliability of the experimental results, this embodiment repeated the training and testing process 5 times with different random seeds, and reported the average test accuracy of each experiment. The experimental evaluation of this embodiment was carried out on three public event-driven datasets, namely EvTouch-Objects, EvTouch-Containers, and ST-MNIST. Next, the experimental results on these datasets and their comparison with the benchmark model will be analyzed in detail.

[0100] In this embodiment, the recognition accuracy of the method proposed by the present invention and the benchmark method on the tactile recognition dataset are compared. The experimental results on the EvTouch-Objects and EvTouch-Containers datasets show that the method proposed by the present invention is significantly better than multiple benchmark models in classification accuracy. Figure 1The classification accuracy of different models on three data sets is shown. It can be seen that the present invention has achieved a classification accuracy of about 84.3% on the EvTouch-Objects data set, which is significantly higher than benchmark models such as TactileSGNet, TactileGCN, SLAYER and ResNet-18. In comparison, the classification accuracy of TactileSGNet on this data set is 67.0%, TactileGCN is 51.3%, and SLAYER and ResNet-18 perform at 65.0% and 62.7%, respectively. In particular, on the EvTouch-Containers data set, the performance improvement of the present invention is more significant, with a classification accuracy of 84.3%, which is much higher than the performance of the benchmark model. This result shows that the present invention has a stronger feature extraction capability when processing event-driven tactile data, especially when the data set is small, it can still show excellent recognition performance.

[0101] Figure 3 is the confusion matrix of different models under the ST-MNIST dataset. Through t-SNE visualization analysis, experiments on the ST-MNIST dataset further verify the effectiveness of the proposed method. Figure 2 As shown, it can be intuitively seen that the features learned by the present invention have clearer category boundaries in two-dimensional space, which shows that the model is able to learn more discriminative feature embeddings. In particular, compared with the traditional convolutional neural network (ResNet-18), the present invention performs particularly well in capturing the temporal information of event-driven data.

[0102] In order to further analyze the feature expression ability of the model, this embodiment also compares the compactness of the features through the PCA dimensionality reduction method. Figure 6 The feature compactness comparison of different models on the ST-MNIST dataset is shown in the figure. The experimental results show that the present invention performs well in feature dimensionality compression. Even in the extreme case where the feature dimension is reduced to only two dimensions, the present invention can still maintain a classification accuracy of 52%, which is much higher than the performance of other models. This result shows that the proposed model can extract more compact and discriminative features while maintaining high classification accuracy, which is of great significance for processing large-scale data and resource-constrained application scenarios.

[0103] Next, the influence of graph connectivity is analyzed, and the influence of different graph structures on classification performance is explored by adjusting the number of edges from each node to its neighbors. Figure 5As shown in the figure, on the EvTouch-Containers, EvTouch-Objects and ST-MNIST datasets, the performance stability of the present invention is better than other benchmark models, especially under different graph connectivity settings (k changes from 1 to 9), the classification performance of the model fluctuates less. In contrast, TactileGCN is very sensitive to the connectivity of the graph, and its classification performance gradually decreases as the number of edges increases. This shows that the proposed model of the present invention has strong robustness and can maintain a relatively stable classification performance in complex graph structures.

[0104] Finally, the impact of time delay on model performance is analyzed, which is a critical factor in event-driven data processing. Figure 4 As shown in the figure, on the EvTouch-Containers dataset, as the time delay increases, the classification accuracy of the present invention gradually improves. When the time delay reaches 70% of the sample collection time, the accuracy of the present invention reaches about 74%, which is better than other models. On the EvTouch-Objects and ST-MNIST datasets, the present invention can achieve a higher classification accuracy in a shorter time window, which shows that its advantages in real-time application scenarios are particularly significant.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. An event-driven tactile perception method based on impulse graph neural network, characterized in that: The method comprises the following steps: S1. Collect data through an event-driven tactile sensor and extract main features from the data collected by the event-driven tactile sensor; S2, encoding the main features extracted from the tactile sensor data; S3, using Spiking Neural Network (SNN) to further learn feature embedding; S4. Process the encoded features through Graph Neural Networks (GNN); S5. Use a voting mechanism to identify the category or state of the target object.

2. The event-driven tactile perception method based on impulse graph neural network according to claim 1, characterized in that: In step S1, the main features are extracted when acquiring the tactile sensor data, and relevant features are extracted from a total of 39 tactile elements taxel, and these features include pressure, temperature and vibration data of each taxel; By using a graph structure, this data is transformed into a graph that can describe the spatial relationships between tactile elements, thereby preserving the spatial locality and globality of the data.

3. The event-driven tactile perception method based on impulse graph neural network according to claim 2, characterized in that: In step S1, a graph G = (V, E) is used to represent the tactile data, where V is a set of nodes of the graph, representing each Taxel, and E represents a set of edges between nodes; the process of constructing the graph includes but is not limited to the following two methods: Manual method: Manually determine the edge by the physical distance between Taxels: set a threshold d th , when the physical distance between Taxeli and Taxel j satisfies d(u i ,u j ) <d th , establish an edge between nodes Taxel i and Taxel j, which can be expressed as: Where d(u i ,u j ) is calculated using the Euclidean distance: Here, x i ,y j Represents the two-dimensional coordinates of Taxel; k-nearest neighbor method: In order to improve the efficiency of automatic graph construction, the k-nearest neighbor algorithm is used to automatically construct edges; in this method, each Taxel is connected to its k nearest neighbors in space, that is: Here kNN(u i ) means Tax u i The k nearest neighbors of ; Through these construction methods, the generated maps can directly reflect the spatial distribution of tactile sensors and the relationship between individual tactile elements.

4. The event-driven tactile perception method based on impulse graph neural network according to claim 3, characterized in that: In step S2, pulse encoding is performed after acquiring the tactile data, and the LIF model is used for encoding. Each tactile data is time-series processed through this model, and a pulse is triggered when the membrane potential reaches the set threshold.

5. The event-driven tactile perception method based on impulse graph neural network according to claim 4, characterized in that: In step S2, the LIF neuron model is used. The LIF model accumulates the input pulses and triggers a pulse signal when the accumulated potential exceeds a certain threshold. The dynamic equation of the LIF neuron is: Where V(t) represents the membrane potential of the neuron, I(t) is the input current, and τ is the time constant of the membrane potential; When the membrane potential V(t) exceeds the set firing threshold V th When the neuron fires a pulse, it resets the membrane potential to V reset ; expressed as: Through this mechanism, LIF neurons can effectively process time series information and generate sparse spike outputs.

6. The event-driven tactile perception method based on impulse graph neural network according to claim 5, characterized in that: In steps S3 and S4, feature embedding learning is performed by combining SNN and GNN. SNN is used to capture the dynamic characteristics of pulse events in the time dimension, and GNN is used to process graph structure data, so that the system can extract complex spatiotemporal relationships in the tactile map.

7. The event-driven tactile perception method based on impulse graph neural network according to claim 6, characterized in that: In step S5, the final output of each category is determined by a voting mechanism, which is based on the pulse emission of LIF neurons, accumulates the number of pulses of each neuron in the time window, and selects the category with the largest number of pulses as the final classification result; the classification result is expressed as: Among them, k (t) represents the number of output pulses of category k at time step t, and T is the length of the time window.

8. An event-driven tactile perception system based on impulse graph neural network, characterized in that: The system adopts the method as claimed in any one of claims 1 to 7.

Images