An event-driven tactile perception method and system based on a pulse graph neural network

Abstract

The present application relates to a kind of event-driven haptic perception method and system based on pulse graph neural network, belong to haptic perception and robot control technical field.The method includes the following steps: S1, extracting main features from the data collected from event-driven haptic sensor;S2, the main features extracted in haptic sensor data are encoded;S3, further learn feature embedding using pulse neural network;S4, the features after coding are processed by graph neural network;S5, the class or state of target object is identified using voting mechanism.The scheme can automatically construct the atlas based on the spatial structure of haptic sensing unit through graph neural network, and efficiently extract local and global features;Event-driven data is processed using pulse neural network, which significantly reduces energy consumption and improves computing efficiency;It has strong generalization ability and can adapt to different haptic sensor configurations and application scenarios, and performs excellently when processing real-time haptic perception tasks.

Description

Technical Field

[0001] This invention belongs to the field of tactile perception and robot control technology, and in particular to the field of event-driven tactile perception method based on pulse graph neural network. It relates to an event-driven tactile perception method and system based on pulse graph neural network. Background Technology

[0002] Tactile sensing technology is one of the key means for robots to interact with their external environment. Traditional tactile sensors capture information such as pressure, temperature, and vibration through synchronous output, helping robots complete tasks such as object recognition, grasping, and manipulation. However, the frame-based data output mode of these sensors tends to lead to high energy consumption and data processing latency, making it difficult to meet the needs of real-time applications. To solve this problem, event-driven tactile sensors have emerged in recent years. These sensors can output data asynchronously based on events, significantly reducing latency and improving energy efficiency. These sensors have been applied to tasks such as object recognition and grasping stability detection. Related research shows that event-driven tactile sensors perform excellently in dynamic environments.

[0003] Currently, the main methods for processing tactile data are based on Convolutional Neural Networks (CNNs). These methods require converting tactile data into a regular grid format for feature extraction. However, the distribution of sensory units (called taxels) in tactile sensors is usually irregular, and forcibly mapping them into a grid introduces redundant computation and information loss. Furthermore, event-driven tactile data is discrete data based on events, which is incompatible with traditional frame data processing methods. Therefore, existing CNN-based solutions not only consume large amounts of computational resources but also struggle to fully utilize the advantages of event-driven sensors.

[0004] To address this, researchers have begun applying Graph Neural Networks (GNNs) to the field of tactile perception to better represent and process irregularly distributed taxel data. TactileGCN, an early graph convolutional network model, extracts features from tactile data by constructing graph nodes from the taxels of tactile sensors and generating graph edges based on their spatial proximity. However, TactileGCN suffers from poor generalization ability, high energy consumption, excessive requirements for large-scale data, and difficulty adapting to changes in new objects or scenes.

[0005] Another type of neural network, the Spiking Neural Network (SNN), processes spatiotemporal data by mimicking the workings of biological neurons. These networks offer significant energy efficiency advantages when processing event-driven data and can achieve low-latency data processing. Models such as SLAYER have achieved some success using SNNs to process event-driven visual data. However, SNNs still face many challenges when processing graph-structured data, such as how to integrate spiking dynamics with graph convolution operations, and how to maintain training stability in deep networks.

[0006] In summary, existing technologies face the following main challenges when processing event-driven haptic data: 1. Traditional CNN methods struggle to effectively handle irregular taxel structures; 2. GNN-based methods suffer from high energy consumption and poor generalization ability; 3. While SNNs excel in processing spatiotemporal data, they haven't been well integrated with graph-structured data. Therefore, a novel method is needed that combines the advantages of spiking neural networks and graph neural networks, capable of handling both irregular haptic data structures and efficiently processing event-driven data. Summary of the Invention

[0007] In view of this, the purpose of this invention is to provide an event-driven tactile perception method and system based on a spiking graph neural network, which can effectively process irregular tactile sensing unit data and improve the processing efficiency of event-driven data. This method and system solve the computational bottleneck problem of existing technologies when processing irregular tactile data structures and event-driven data by organizing tactile data into a graph structure and utilizing the efficient processing capabilities of spiking neural networks.

[0008] To achieve the above objectives, the present invention provides the following technical solution:

[0009] An event-driven tactile perception method based on a pulse graph neural network, the method comprising the following steps:

[0010] S1. Collect data through event-driven haptic sensors and extract key features from the data collected by event-driven haptic sensors;

[0011] S2. Encode the main features extracted from the tactile sensor data;

[0012] S3. Use a spiking neural network (SNN) to further learn feature embeddings;

[0013] S4. Process the encoded features using Graph Neural Networks (GNNs);

[0014] S5. Use a voting mechanism to identify the category or state of the target object.

[0015] Furthermore, in step S1, key features are extracted when acquiring tactile sensor data. A total of 39 tactile elements (taxels) are extracted, including pressure, temperature, and vibration data of each taxel. By using a graph structure, these data are transformed into a map that can describe the spatial relationships between tactile elements, thereby preserving the spatial locality and globality of the data.

[0016] Furthermore, in step S1, a graph G = (V, E) is used to represent the tactile data, where V is the set of nodes in the graph, V represents each Taxel, and E represents the set of edges between nodes; the process of constructing the graph includes, but is not limited to, the following two methods:

[0017] Manual method: Determine edges manually by physical distance between Taxels: Set a threshold d th When the physical distance between Taxeli and Taxelj satisfies d(u i ,u j ) <d th To establish an edge between nodes Taxel i and Taxel j, this can be represented as:

[0018]

[0019] Where d(u i ,u j ) Use Euclidean distance to calculate:

[0020]

[0021] Here, x i ,y j Represents the two-dimensional coordinates of a Taxel;

[0022] k-Nearest Neighbors Method: To improve the efficiency of automated graph construction, the k-nearest neighbor algorithm is used to automatically construct edges; in this method, each Taxel is connected to its k nearest neighbors in space, i.e.:

[0023]

[0024] Here kNN(u) i ) indicates Taxel u i k nearest neighbors;

[0025] These construction methods generate graphs that can directly reflect the spatial distribution of tactile sensors and the relationships between various tactile elements.

[0026] Furthermore, in step S2, pulse encoding is performed after acquiring tactile data. To improve computational efficiency, the LIF (Leak Integration and Release) model is used for encoding. Each tactile data is processed in sequence through this model. When the membrane potential reaches a set threshold, a pulse is triggered. Compared with traditional convolutional neural network methods, the LIF model can effectively handle event-driven asynchronous data and significantly improve inference speed.

[0027] Furthermore, in step S2, a LIF neuron model is employed. The LIF model accumulates the input pulses and triggers a pulse signal when the accumulated potential exceeds a certain threshold. The dynamic equation of the LIF neuron is:

[0028]

[0029] Where V(t) represents the membrane potential of the neuron, I(t) is the input current, and τ is the time constant of the membrane potential;

[0030] When the membrane potential V(t) exceeds the set discharge threshold V th At this time, the neuron will emit a pulse and reset the membrane potential to V. reset ; indicates as:

[0031]

[0032] Through this mechanism, LIF neurons are able to effectively process time-series information and generate sparse pulse output.

[0033] Furthermore, in steps S3 and S4, feature embedding learning is performed by combining SNN and GNN. SNN captures the dynamic features of impulse events in the time dimension, while GNN processes graph structure data, enabling the system to extract complex spatiotemporal relationships in the tactile map.

[0034] Furthermore, in step S5, a voting mechanism is used to determine the final output for each category. This voting mechanism is based on the spiking of LIF neurons, accumulating the number of spiking events for each neuron within a time window, and selecting the category with the most spiking events as the final classification result. The classification result is expressed as follows:

[0035]

[0036] Among them, o k (t) represents the number of output pulses of category k at time step t, where T is the length of the time window.

[0037] The present invention also provides an event-driven tactile perception system based on a pulse graph neural network.

[0038] The beneficial effects of this invention are as follows:

[0039] 1. Efficiently handles irregular tactile sensor unit layouts: Unlike traditional CNNs, this invention models tactile sensor units as graph structures, combining the advantages of graph neural networks in processing irregular layout data, thus effectively addressing the complex topologies of tactile sensors. It exhibits excellent adaptability to different sensor configurations.

[0040] 2. Low energy consumption and real-time response: Due to the use of spiking neural networks, the system has significant energy consumption advantages when processing event-driven data and can achieve low-latency real-time response. Compared with traditional frame-based neural network methods, spiking neural networks only emit pulses when tactile events occur, avoiding redundant calculations caused by frame data, thereby greatly reducing system energy consumption.

[0041] 3. Dense Connections Enhance Information Flow and Generalization Ability: This invention introduces a dense connection mechanism to reuse information between different layers of the network, ensuring stronger gradient flow and information transfer efficiency. Compared with existing layered networks, dense connections reduce information loss and enhance the model's generalization ability, especially when processing complex multidimensional tactile data.

[0042] 4. High Classification Accuracy: This invention employs a voting-based classification method, which uses the pulse firing frequency of output layer neurons to make classification judgments, significantly improving classification accuracy. In various event-driven tactile perception tasks, this invention significantly outperforms existing technologies in classification accuracy. For example, in some tactile perception scenarios, the classification accuracy is improved by approximately 10%-15%.

[0043] 5. High scalability and adaptability to different application scenarios: Because this invention does not rely on fixed sensor configurations or specific environmental structures, the system can be well generalized to different tactile sensing applications. Its graph neural network processing of irregular graph structures can flexibly adapt to various novel sensor layouts, while the spiking neural network ensures effective application in different real-time tactile scenarios.

[0044] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0045] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0046] Figure 1 This is a schematic diagram comparing the recognition accuracy of the present invention with that of the benchmark method;

[0047] Figure 2 Visualizing the ST-MINIST dataset using T-SNE;

[0048] Figure 3 This is the confusion matrix for different models on the ST-MINIST dataset;

[0049] Figure 4 This is a comparison graph showing the time delay between the present invention and the baseline method;

[0050] Figure 5 This is a comparison diagram of the present invention and the baseline method under the K-nearest neighbor mapping method;

[0051] Figure 6 A comparison plot showing the feature tightness of different models on the ST-MNIST dataset;

[0052] Figure 7 This is an overall overview diagram of the method of the present invention;

[0053] Figure 8 This is an overall framework diagram of the present invention. Detailed Implementation

[0054] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0055] This invention addresses the computational bottleneck in existing technologies when processing irregular tactile data structures and event-driven data by organizing tactile data into a graph structure and leveraging the efficient processing capabilities of spiking neural networks for event-driven data. Figure 7 This is an overall overview diagram of the method of the present invention. Figure 8 This is an overall framework diagram of the present invention.

[0056] like Figure 7 As shown, it takes event-driven haptic sensing data as input and organizes it into a haptic graph. This invention uses a LIF model and consists of adaptive graph convolutional (TAGConv) layers and fully connected (FC) layers. Figure 8 The overall structure of the invention is illustrated, consisting of a TAGConv layer, two Deep blocks, a transition layer (also a TAGConv layer), two FC layers, and a voting layer. Each Deep block comprises three DeepLayer layers connected by dense connections. Note that each TAGConv or FC layer is followed by a LIF activation and event-based batch normalization layer; these layers are not shown in the figure for clarity. The following describes the various steps of the method in detail, including formula derivation and their physical significance.

[0057] First, tactile data is acquired using the NeuTouch sensor. This sensor consists of multiple tactile elements (Taxels) distributed across an irregular grid. Each Taxel generates events at irregular time intervals, representing forces, pressures, or vibrations exerted on the tactile element by the environment. To effectively process tactile data from this irregular structure, it must be constructed as a graph structure.

[0058] In this embodiment, a graph G = (V, E) is used to represent tactile data, where V is the set of nodes in the graph, representing each Taxel, and E represents the set of edges between nodes. The graph construction process can be implemented using the following two methods:

[0059] Manual method: Determine edges manually by physical distance between Taxels. Set a threshold d. th When the physical distance between Taxeli and Taxelj satisfies d(u i ,u j ) <d th Establish an edge between nodes Taxel i and Taxel j. This can be represented as:

[0060]

[0061] Where d(u i ,u j Euclidean distance is usually used for calculation:

[0062]

[0063] Here, x i ,y j Represents the two-dimensional coordinates of the Taxel.

[0064] k-Nearest Neighbors Method: To improve the efficiency of automated graph construction, the k-nearest neighbor algorithm is used to automatically construct edges. In this method, each Taxel is connected to its k nearest neighbors in space. That is:

[0065]

[0066] Here kNN(u) i ) indicates Taxel u i The k nearest neighbors.

[0067] These construction methods generate graphs that can directly reflect the spatial distribution of tactile sensors and the relationships between various tactile elements.

[0068] On the constructed haptic map, a Topological Adaptive Graph Convolutional Layer (TAGConv) is used to extract graph features. Unlike traditional graph convolution, TAGConv captures multi-hop information between nodes by introducing a multi-order power of the adjacency matrix. This operation can be expressed by the following formula:

[0069]

[0070] Among them, H (l) A is the feature matrix of the l-th layer. k Let α be the normalized adjacency matrix of the graph. k These are the learned coefficients, and K is the maximum power of the adjacency matrix. This formula captures both local and global information about the graph by propagating node information to nodes within a K-hop range.

[0071] The features extracted by the TAGConv layer are fed into the Deep Block for further processing. Each Deep Block consists of multiple DeepLayers, each containing two TAGConv layers and two Event-Based Batch Normalization (EBN) layers. EBN is introduced to prevent vanishing or exploding gradients and to accommodate event-driven data in spiking neural networks.

[0072] The SNN in this invention uses the LIF neuron model. The LIF model accumulates input pulses and triggers a pulse signal when the accumulated potential exceeds a certain threshold. The dynamic equation of the LIF neuron is:

[0073]

[0074] Where V(t) represents the membrane potential of the neuron, I(t) is the input current, and τ is the time constant of the membrane potential.

[0075] When the membrane potential V(t) exceeds the set discharge threshold V th At this time, the neuron will emit a pulse and reset the membrane potential to V. reset This can be expressed as:

[0076]

[0077] Through this mechanism, LIF neurons are able to effectively process time-series information and generate sparse pulse output.

[0078] To better capture features, the model introduces dense connections between the deeper layers within each DeepBlock. This connection method allows the output of each layer to be directly passed to all subsequent layers, thereby enabling feature reuse and better information flow. The mathematical expression for this operation is:

[0079]

[0080] in, This indicates a splicing operation.

[0081] After passing through multiple TAGConv layers and deep modules, the extracted features are input into a fully connected (FC) layer for dimensionality reduction and classification. The operation of the fully connected layer can be described by the following formula:

[0082] h=Wx+b#(8)

[0083] Where W is the weight matrix, x is the input feature, and b is the bias term.

[0084] The final classification layer uses a voting mechanism to determine the final output for each category. This voting mechanism is based on the spiking of LIF neurons, accumulating the number of spiking events for each neuron within a time window, and selecting the category with the most spiking events as the final classification result. The classification result can be represented as:

[0085]

[0086] Among them, o k (t) represents the number of output pulses of category k at time step t, where T is the length of the time window.

[0087] Because the pulse firing function of a spiking neural network is discontinuous, traditional gradient descent methods cannot be directly applied to spiking neural networks. To address this issue, a surrogate gradient method is used to approximate the derivative of the pulse function. Specifically, the pulse derivative is approximated using a rectangular function:

[0088]

[0089] Here, 'a' is a parameter that controls the width of the rectangle. This approximation allows for effective backpropagation of errors during training.

[0090] The loss function of the entire model consists of two parts: supervision loss and graph structure regularization loss. The supervision loss uses mean squared error (MSE) to measure the difference between the predicted output and the actual label.

[0091]

[0092] Among them, y i It is the true label of the i-th sample. These are the model's predicted values.

[0093] Meanwhile, to ensure the structural properties of the graph, a graph structure regularization loss is introduced:

[0094]

[0095] Among them, A ij It is an adjacency matrix, h i ,h j It is the feature representation of nodes i and j.

[0096] The overall loss function is:

[0097]

[0098] Where λ is the regularization coefficient.

[0099] In this embodiment, PyTorch was used for experiments, and the Adam optimizer was used for model training. The initial learning rate was set to 0.001, and the number of training iterations was 100. To ensure the reliability of the experimental results, this embodiment repeated the training and testing process 5 times using different random seeds, and the average test accuracy for each experiment is reported. The experimental evaluation of this embodiment was conducted on three publicly available event-driven datasets: EvTouch-Objects, EvTouch-Containers, and ST-MNIST. The experimental results on these datasets and their comparison with benchmark models will be analyzed in detail below.

[0100] This embodiment compares the recognition accuracy of the proposed method with benchmark methods on tactile recognition datasets. Experimental results on the EvTouch-Objects and EvTouch-Containers datasets show that the proposed method significantly outperforms several benchmark models in classification accuracy. Figure 1The classification accuracy of different models on three datasets is shown. It can be seen that the present invention achieves a classification accuracy of approximately 84.3% on the EvTouch-Objects dataset, significantly higher than benchmark models such as TactileSGNet, TactileGCN, SLAYER, and ResNet-18. In comparison, TactileSGNet achieves a classification accuracy of 67.0% on this dataset, TactileGCN 51.3%, while SLAYER and ResNet-18 achieve 65.0% and 62.7%, respectively. Particularly on the EvTouch-Containers dataset, the performance improvement of the present invention is even more significant, with a classification accuracy of 84.3%, far exceeding the performance of benchmark models. This result demonstrates that the present invention possesses stronger feature extraction capabilities when processing event-driven haptic data, exhibiting excellent recognition performance even with smaller datasets.

[0101] Figure 3 This represents the confusion matrix for different models on the ST-MNIST dataset. t-SNE visualization analysis and experiments on the ST-MNIST dataset further validate the effectiveness of the proposed method. Figure 2 As shown, it can be intuitively seen that the features learned by this invention have clearer category boundaries in two-dimensional space, indicating that the model can learn more discriminative feature embeddings. In particular, compared with traditional convolutional neural networks (ResNet-18), this invention performs exceptionally well in capturing the temporal information of event-driven data.

[0102] To further analyze the model's feature representation capability, this embodiment also compares the compactness of features using the PCA dimensionality reduction method. Figure 6 The paper presents a comparison of feature compactness among different models on the ST-MNIST dataset. Experimental results show that the proposed invention excels in feature dimensionality compression. Even in the extreme case where the feature dimension is reduced to only two dimensions, the proposed invention still maintains a classification accuracy of 52%, far exceeding the performance of other models. This result demonstrates that the proposed model can extract more compact and discriminative features while maintaining high classification accuracy, which is of great significance for handling large-scale data and resource-constrained applications.

[0103] Next, the impact of graph connectivity was analyzed. By adjusting the number of edges from each node to its neighbors, the influence of different graph structures on classification performance was explored. For example... Figure 5As shown, on the EvTouch-Containers, EvTouch-Objects, and ST-MNIST datasets, the proposed model demonstrates superior stability compared to other benchmark models, particularly exhibiting smaller fluctuations in classification performance under different graph connectivity settings (k varying from 1 to 9). In contrast, TactileGCN is highly sensitive to graph connectivity, with its classification performance gradually decreasing as the number of edges increases. This indicates that the proposed model possesses strong robustness and maintains relatively stable classification performance in complex graph structures.

[0104] Finally, the impact of time latency on model performance was analyzed, which is a key factor in event-driven data processing. Figure 4 As shown, on the EvTouch-Containers dataset, the classification accuracy of this invention gradually improves with increasing time delay. When the time delay reaches 70% of the sample collection time, the accuracy of this invention reaches approximately 74%, which is superior to other models. Furthermore, on the EvTouch-Objects and ST-MNIST datasets, this invention achieves high classification accuracy within a shorter time window, demonstrating its significant advantages in real-time application scenarios.

[0105] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications should be covered within the scope of the claims of the present invention.

Claims

1. An event-driven tactile perception method based on a pulse graph neural network, characterized in that: The method includes the following steps: S1. Collect data through event-driven haptic sensors and extract key features from the data collected by event-driven haptic sensors; S2. Encode the main features extracted from the tactile sensor data. The main features include pressure, temperature, and vibration data of each taxel. S3. Use a spiking neural network (SNN) to further learn feature embeddings; S4. Process the encoded features using a graph neural network (GNN); S5. Use a voting mechanism to identify the category or state of the target object; In step S1, key features are extracted when acquiring tactile sensor data. A total of 39 tactile elements (taxels) are extracted, including pressure, temperature, and vibration data of each taxel. By using a graph structure, these data are transformed into a graph that can describe the spatial relationships between tactile elements, thereby preserving the spatial locality and globality of the data. In step S1, using the diagram To represent tactile data, Here, is the set of nodes in the graph, representing each Taxel. The graph represents the set of edges between nodes; the process of constructing the graph is achieved in one of the following two ways: Manual method: Manually determine edges by physical distance between Taxels: Set a threshold. When Taxel and Taxel When the physical distance between them meets the requirements At node Taxel and Taxel Establish an edge between them, represented as: in Calculated using Euclidean distance: here, , Represents the two-dimensional coordinates of a Taxel; k-Nearest Neighbors Method: To improve the efficiency of automated graph construction, the k-nearest neighbor algorithm is used to automatically construct edges; in this method, each Taxel is connected to its k nearest neighbors in space, i.e.: here Taxel k nearest neighbors; The generated graphs, constructed using these methods, can directly reflect the spatial distribution of tactile sensors and the relationships between various tactile elements. In step S2, pulse coding is performed after acquiring tactile data. The LIF model is used for coding. Each tactile data is processed in time through this model. When the membrane potential reaches the set threshold, a pulse is triggered. In step S2, a LIF neuron model is used. The LIF model accumulates the input pulses and triggers a pulse signal when the accumulated potential exceeds a certain threshold. The dynamic equation of the LIF neuron is: in, Represents the membrane potential of a neuron. It is the input current. It is the time constant of the membrane potential; When membrane potential Exceeding the set distribution threshold At this time, the neuron will emit a pulse and reset the membrane potential to ; indicates as: Through this mechanism, LIF neurons can effectively process time-series information and generate sparse pulse outputs; the model introduces dense connections between the deep layers in each DeepBlock.

2. The event-driven tactile perception method based on a pulse graph neural network according to claim 1, characterized in that: In steps S3 and S4, feature embedding learning is performed by combining SNN and GNN. SNN captures the dynamic features of impulse events in the time dimension, while GNN processes graph structure data, enabling the system to extract complex spatiotemporal relationships in the tactile map.

3. The event-driven tactile perception method based on a pulse graph neural network according to claim 2, characterized in that: In step S5, a voting mechanism is used to determine the final output of each category. The voting mechanism is based on the pulse firing of LIF neurons, accumulating the number of pulses of each neuron within the time window, and selecting the category with the most pulses as the final classification result. The classification results are represented as follows: in, Indicates time step Time Category The number of output pulses, It is the length of the time window.

4. An event-driven tactile perception system based on a pulse graph neural network, characterized in that: The system employs the method described in any one of claims 1 to 3.

Images