Robot perception data word segmentation representation and learning method based on graph neural network

Abstract

The application relates to a kind of robot perception data word segmentation representation and learning method based on graph neural network, comprising the following steps: obtaining multiple kinds of perception data of robot;According to the class of perception data, word segmentation representation is carried out;Word segmentation representation learning: based on the perception data after word segmentation representation, initial feature map is constructed;Based on autoencoder, compact representation of feature map is learned, graph structure is reconstructed, and the relationship between the edges of different nodes in graph structure is represented;After the learning of graph structure is completed by autoencoder, the graph structure is fixed;Perception data is converted into node feature vector, and based on the learned graph structure, feature map is constructed, each node feature vector is numerically encoded using graph neural network, and high-dimensional feature vector representation of perception data is obtained.Compared with the prior art, the application has the advantages of better expressing the complexity and multidimensional information of perception data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot perception data processing, and in particular to a robot perception data word segmentation representation and learning method based on a graph neural network. BACKGROUND

[0002] Currently, in the field of natural language processing, word segmentation, as an important step in text preprocessing, has a crucial impact on subsequent tasks such as sentiment analysis, machine translation, and question-answering systems. Traditional word segmentation methods mainly rely on dictionary matching and statistical models, which often perform poorly when faced with new words, polysemous words, and long and difficult sentences. With the development of deep learning, methods based on neural networks have gradually become mainstream, especially recurrent neural networks (RNN) and long short-term memory networks (LSTM) have achieved good results in word segmentation tasks. Subsequently, embedding technology converts these discrete words into vector representations in high-dimensional space, not only enabling computers to process text information, but more importantly, these vectors can mathematically capture the semantic and grammatical relationships between words, enabling machines to understand the content of the text to some extent. As a result, whether for text classification, sentiment analysis, or machine translation, word segmentation and embedding provide the system with the ability to understand and process natural language, and they together form the cornerstone of the modern natural language processing technology stack, greatly improving the accuracy and efficiency of various language processing tasks.

[0003] Modern robot technology has been widely applied in industrial manufacturing, service industries, and other fields. In order to enable robots to better understand and execute tasks, it is often necessary to convert perception data (such as degrees of freedom, end pose, visual perception, and tactile perception) into a form that can be processed by a computer. However, existing perception data processing methods mostly rely on fixed data structures and simple feature extraction methods, which limit the understanding ability of robots in complex environments and the flexibility of task execution.

[0004] With the development of artificial intelligence technology, especially the application of large pre-trained models, robot systems have made significant progress in decision control. These large models can handle complex tasks and learn patterns from large amounts of data, enabling them to make more intelligent decisions. However, in order for these models to function in actual robot applications, they need to be closely integrated with the embodied perception inputs of the robot.

[0005] Robot perception input includes but is not limited to data in multiple dimensions such as degrees of freedom, end pose, visual perception, and tactile perception. In order for large models to effectively utilize this information, these perception data must be converted into a form that the model can understand. Ideally, this data should be converted into a high-dimensional vector, and in this conversion process, the mutual relationship between different dimensional perception data and the information they carry respectively needs to be preserved.

[0006] Currently, although the field of natural language processing has developed relatively mature segmentation and embedding methods, these methods can well convert text into semantic-rich vector representations, but similar methods for robot perception data are relatively lacking. Existing technologies usually use relatively simple methods when processing robot perception data, such as directly using raw numerical values or simple feature engineering, which cannot fully express the complexity and multi-dimensional information of perception data.

[0007] In addition, due to the lack of effective segmentation and embedding methods, existing robot systems have difficulty fully utilizing the capabilities of large pre-trained models to process perception input. This has limited the decision-making and control performance of robots in dynamic and complex environments, and they cannot respond to various situations as flexibly as humans.

[0008] Therefore, there is an urgent need to develop a method that can effectively convert robot perception data into high-dimensional vectors, and this method should be able to preserve the mutual relationship between perception data in order to better serve the decision-making and control system of the robot. Currently, research in this field is still in its infancy, and there is no mature technical solution to solve this problem. SUMMARY

[0009] The purpose of the present application is to provide a robot perception data segmentation and representation method based on graph neural networks (GNN), which aims to convert the multi-dimensional perception input of the robot into a high-dimensional vector and preserve the relationship information between different dimensional perception data.

[0010] The purpose of the present application can be achieved by the following technical solutions:

[0011] A robot perception data segmentation and representation method based on graph neural networks, comprising the following steps:

[0012] S1, data acquisition: acquiring multiple perception data of the robot;

[0013] S2, segmentation and representation according to the category of perception data;

[0014] S3, segmentation and representation learning:

[0015] S31, constructing an initial feature map based on the segmented perception data;

[0016] S32, learning a compact representation of the feature map based on a self-encoder, and reconstructing a graph structure representing the relationship between edges of different nodes, wherein the self-encoder comprises an encoder and a decoder, wherein the encoder adopts a graph attention mechanism network to map node features to a latent space, and the decoder reconstructs the original graph structure from the latent space;

[0017] S33, fixing the graph structure after the self-encoder completes learning of the graph structure;

[0018] S34, converting the perception data into node feature vectors, and constructing a feature map based on the learned graph structure, and using a graph neural network to numerically encode each node feature vector to obtain a high-dimensional feature vector representation of the perception data.

[0019] The perception data includes degree of freedom state data, end pose data, visual perception data, tactile perception data, and pressure sensor data.

[0020] In S2, for discrete data types, different categories of perception data are considered as different segmented tokens for segmented representation, and different nodes are correspondingly represented in the graph neural network.

[0021] In S2, for continuous numerical input, a graph attention mechanism network is directly used as an embedding network to learn a high-dimensional representation of the numerical data and capture the relationship and structure between different numbers.

[0022] In S2, for time series data, it is divided into multiple time periods according to a preset time length, and feature extraction or encoding processing is performed on each time period to obtain segmented representation; or the time series data is transformed to obtain features in the time domain or frequency domain, and the features are used as segmented representation.

[0023] The method for learning a compact representation of the feature map based on the self-encoder and reconstructing the graph structure comprises the following steps:

[0024] Random masking: selecting part of the nodes for masking, wherein the masking method uses a random selection strategy, and for the masked nodes, the feature vectors are replaced with zero vectors or a special masking marker;

[0025] Encoding: using a graph attention mechanism network as an encoder to encode the nodes that are not masked;

[0026] Decoding: the decoder receives the node representation generated by the encoder and the masking information to predict the features of the masked nodes;

[0027] Loss function definition: define the mean square error loss function to measure the difference between the decoder output and the actual mask node feature, and introduce the actual physical constraint in the loss function;

[0028] Backpropagation and iterative training: use the loss function to perform gradient descent on the autoencoder model parameters to minimize the loss until the model converges or reaches a predetermined number of training rounds;

[0029] Use the trained autoencoder to learn the node mapping of different types of perception data on the graph neural network, establish the connection between nodes according to the internal relationship between perception data, and form an undirected graph.

[0030] The graph attention mechanism network aggregates the information of neighbors by calculating the attention weight between nodes, represented as:

[0031]

[0032] where, is the representation of node i at the l-th layer, σ is the activation function, w (l) is the weight matrix, α ij is the attention coefficient between nodes i and j, N(i) is the set of neighbors of node i.

[0033] The decoder uses a multi-layer perceptron, specifically, a 3-layer fully connected layer to decode the encoded information.

[0034] The actual physical constraints include:

[0035] Dynamic constraints: use the dynamics model of the robot to define the dynamics constraints at each time step;

[0036] Geometric constraints: use the kinematics model of the robot to define geometric constraints, including constant link length and joint angle range constraints;

[0037] Contact constraints: define torque and force constraints at contact points, including the constraint that friction cannot exceed the maximum static friction;

[0038] Energy conservation constraints: ensure the total energy conservation of the system, i.e. the conversion between potential energy, kinetic energy and work done complies with the law of conservation of energy.

[0039] The specific steps of using the graph neural network to numerically encode each node feature vector to obtain a high-dimensional feature vector representation of the perception data are as follows:

[0040] Initialize each perception data node, and use a multi-layer perceptron to convert the original numerical value to an initial node feature vector;

[0041] The message passing mechanism of the graph neural network is used to make each node exchange information with its adjacent nodes, and in each layer, each node updates its own representation to contain the information of the neighbor nodes, so as to capture the mutual relationship between the perception data.

[0042] Multiple iteration optimization: after multiple rounds of iteration of the graph neural network, each node obtains a new embedding vector containing the features of the node itself and the fused information of the nodes around it, and the embedding vector is used as a learned high-dimensional feature vector representation.

[0043] Compared with the prior art, the present application has the following beneficial effects:

[0044] (1) The present application introduces the word segmentation and embedding technology and ideas in the field of large models into the processing of robot perception data, filling the technical gap in this field.

[0045] (2) The present application uses graph neural network (GNN) as the core technical means for processing robot perception data, which can effectively model the relationship between different perception data and convert it into a high-dimensional vector representation, thereby preserving the mutual relationship between the perception data. Compared with the traditional method of using only raw numerical values or simple feature engineering, it can better express the complexity and multi-dimensional information of the perception data.

[0046] (3) The present application proposes a learning method based on physical systems to autonomously learn the word segmentation representation of multi-dimensional perception of robots while preserving the relationship between different dimensional perception data, which is beneficial to large models in processing robot input information and better understanding the decision control of robots in complex environments. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 The flowchart of the method of the present application is shown in the figure;

[0048] Figure 2 The graph attention mechanism network model architecture of the present application is shown in the figure;

[0049] Figure 3 The perception data graph structure construction flowchart of the present application is shown in the figure;

[0050] Figure 4 The graph node feature vector numerical encoding flowchart of the present application is shown in the figure. DETAILED DESCRIPTION

[0051] The present application will be described in detail below in conjunction with the drawings and specific embodiments. The present embodiment is implemented on the basis of the technical solution of the present application, and detailed implementation and specific operation processes are given, but the protection scope of the present application is not limited to the following examples.

[0052] This embodiment provides a robot perception data segmentation representation and learning method based on graph neural network, such as Figure 1 As shown, the following steps are included:

[0053] S1, data acquisition: Acquire various perception data of the robot.

[0054] In this embodiment, a variety of perception data from the robot are collected, including but not limited to the states of each degree of freedom, end position, visual perception, touch, pressure sensors, etc., to construct a robot multimodal perception dataset.

[0055] S2, word segmentation representation according to the category of perception data.

[0056] For various input perception data, it is necessary to perform word segmentation representation and convert it into a representation that the model can process. Unlike tokenization in natural language processing, the input robot perception data is a discrete data type of different types. Each data type is a continuous value. Therefore, a word segmentation representation paradigm needs to be established separately. The details are as follows:

[0057] S21, for discrete data types, different categories of perception data are regarded as different segmentations for word segmentation representation, corresponding to different nodes in the graph neural network.

[0058] S22, for continuous numerical input, directly use the graph attention mechanism network as the embedding network to learn the high-dimensional representation of numerical data and capture the relationship and structure between different numerical values.

[0059] S23: For time series data (such as acceleration, temperature, distance and other sensor data), divide it into multiple time periods according to a preset time length, perform feature extraction or encoding processing on each time period to obtain a word segmentation representation; or, transform the time series data to obtain time domain or frequency domain features, and use the features as word segmentation representation.

[0060] S3, word segmentation representation learning.

[0061] The purpose of constructing a perceptual data graph is to encode the robot's various perceptual data while preserving both its own information and the relationships between these different perceptual data. The collected perceptual data all originates from the robot's physical system and, therefore, conforms to the underlying laws of the physical world. An autoencoder framework is used to learn a compact representation of the graph, where the encoder maps node features to a latent space, and the decoder reconstructs the original graph structure from the latent space.

[0062] Specifically, such as Figure 3 As shown, S3 includes the following steps:

[0063] S31, constructing an initial feature graph G=(V, E) based on the segmented perception data, where V is a node set and E is an edge set, and each node has a feature vector.

[0064] S32, learning a compact representation of the feature graph based on an autoencoder and reconstructing the graph structure, where the graph structure represents the relationship between edges of different nodes.

[0065] S321, random masking: selecting part of the nodes for masking, where the masking method uses a random selection strategy, and for the masked nodes, their feature vectors are replaced with zero vectors or a special masking marker.

[0066] S322, encoding: using a graph attention mechanism network (GAT) as an encoder to encode the nodes that are not masked. As shown in Figure 2 In the GAT network, each node not only pays attention to its own features, but also pays attention to the features of other nodes connected to it. Specifically, each node in the graph has an attention coefficient that determines the importance of its relationship with other nodes. First, the attention coefficient is calculated, which is usually a scalar value representing the degree of attention of node i to node j. This means that attention can be allocated to all relevant nodes, not just a single node. Then multiply the attention coefficient by the corresponding node feature to get a weighted average value that reflects the weighted summary of node i's neighbor node features. Finally, combine this weighted summary with the features of node i itself to generate a new node representation.

[0067] That is, the graph attention mechanism network aggregates the information of neighbors by calculating the attention weight between nodes, represented as:

[0068]

[0069] where, is the representation of node i at the l-th layer, σ is the activation function, w (l) is the weight matrix, α ij is the attention coefficient between nodes i and j, and N(i) is the set of node i's neighborhood.

[0070] S323, decoding: the decoder receives the node representation generated by the encoder and the masking information to predict the features of the masked nodes.

[0071] This embodiment uses a multi-layer perceptron as a decoder, specifically, a 3-layer fully connected layer is used to decode the encoded information.

[0072] S324, loss function definition: define the mean square error loss function to measure the difference between the decoder output and the actual mask node features, in addition to this, considering the actual physical system, introduce the actual physical constraint into the loss function.

[0073] In this embodiment, the actual physical constraint includes:

[0074] 1) Dynamic constraint: use the dynamics model of the robot to define the dynamics constraint at each time step, such as Newton's second law;

[0075] 2) Geometric constraint: use the kinematics model of the robot to define geometric constraints, such as constant link length and joint angle range constraints, etc;

[0076] 3) Contact constraint: define the torque and force constraints at the contact point, such as the friction force cannot exceed the maximum static friction constraint;

[0077] 4) Energy conservation constraint: ensure the total energy conservation of the system, that is, the conversion between potential energy, kinetic energy and work done complies with the law of conservation of energy.

[0078] Then, the original loss function is combined with one or more additional loss terms, which are used to punish predictions that do not comply with physical laws. The final loss function can be represented as:

[0079] L = L base + λL phys

[0080] Where λ is a hyperparameter used to adjust the importance of physical constraints.

[0081] The specific form of the physical constraint can be exemplified as

[0082] For a general second-order system, the loss function under dynamic constraint can be represented as

[0083]

[0084] Here m is the mass, c, k is the physical coefficient, F(t) is the external force. x and are the predicted position and velocity, respectively.

[0085] For a robot arm, if the length of the link is known to be fixed, a constraint term can be added to punish those predicted joint angle combinations, such as

[0086] L phys = ||l1 cos(θ1)+l2 cos(θ1+θ2)||-d 12 | 2

[0087] wherein, l i represents the length of the ith link, θ i is the angle of joint i, d 12 is the theoretical fixed distance between two ends.

[0088] The establishment of other loss functions can refer to the above method, and the present embodiment will not be described here.

[0089] S325, back propagation and iterative training: using the loss function to perform gradient descent on the autoencoder model parameters to minimize the loss. The Adam optimization algorithm can be used, and the above steps are repeated until the model converges or a predetermined number of training iterations is reached.

[0090] S326, using the trained autoencoder to learn the node mapping of different types of perception data on the graph neural network, establishing the connection between the nodes according to the internal relationship between the perception data (such as causal relationship, spatial relationship, etc.), and forming an undirected graph.

[0091] S33, after the autoencoder completes the learning of the graph structure, the graph structure is fixed.

[0092] S34, convert the perception data into node feature vectors, and construct a feature graph based on the learned graph structure, use the graph neural network to numerically encode each node feature vector to obtain a high-dimensional feature vector representation of the perception data.

[0093] As Figure 4 shown, the specific steps are as follows:

[0094] S341, initialize each perception data node, in the present embodiment, a multilayer perceptron is used to convert the original numerical value into an initial node feature vector.

[0095] S342, use the message passing mechanism of the graph neural network (GNN) to let each node exchange information with its adjacent nodes, in each layer, each node updates its own representation to include the information of the neighbor nodes, so as to capture the mutual relationship between the perception data;

[0096] S343, multiple iterations of optimization: after multiple iterations of the graph neural network, each node obtains a new embedding vector, which contains the features of the node itself and the information of the surrounding nodes, and the embedding vector is used as the learned high-dimensional feature vector representation.

[0097] The preferred embodiments of the present application have been described above in detail. It should be understood that modifications and variations to the preferred embodiments could be made by those skilled in the art in light of the teachings above without departing from the spirit of the present application. It is therefore to be understood that what is desired to be protected by letters patent is defined by the scope of the claims below and that on the basis of the teachings of the present application, obvious modifications and equivalents can be adopted by those skilled in the art in their possession of the teachings of the present application without departing from the spirit and scope of the application.

Claims

1. A robot perception data segmentation representation and learning method based on graph neural network, characterized by: The following steps are involved: S1, data acquisition: obtaining various sensory data of the robot; S2, word segmentation representation according to the category of perception data; S3, word segmentation representation learning: S31, constructing the initial feature map based on the perception data after word segmentation representation; S32, reconstructing a graph structure based on a compact representation of a feature graph learned by an autoencoder, wherein the graph structure represents edge relationships between different nodes, the autoencoder comprising an encoder and a decoder, wherein the encoder employs a graph attention mechanism network to map node features to a latent space, and the decoder reconstructs the original graph structure from the latent space; The method of learning a compact representation of the feature graph based on the autoencoder and reconstructing the graph structure includes the following steps: Random masking: select some nodes for masking. The masking method uses a random selection strategy. For the masked nodes, their feature vectors are replaced with zero vectors or a special mask mark. Encoding: Use the graph attention mechanism network as an encoder to encode the unmasked nodes; Decoding: The decoder receives the node representation and mask information generated by the encoder to predict the features of the masked nodes; Loss function definition: Define the mean square error loss function to measure the difference between the decoder output and the actual mask node features, and introduce actual physical constraints into the loss function; Backpropagation and iterative training: Gradient descent of the autoencoder model parameters using the loss function to minimize the loss until the model converges or reaches a predetermined number of training rounds; The trained autoencoder is used to learn the node mapping of different types of sensory data on the graph neural network, and connections between nodes are established based on the intrinsic connections between the sensory data to form an undirected graph. S33, after the autoencoder completes the learning of the graph structure, the graph structure is fixed; S34, converting the perception data into node feature vectors, and constructing a feature graph based on the learned graph structure, using a graph neural network to numerically encode each node feature vector to obtain a high-dimensional feature vector representation of the perception data; The specific steps of using the graph neural network to numerically encode the feature vectors of each node to obtain a high-dimensional feature vector representation of the perception data are as follows: Initialize each perception data node and use a multi-layer perceptron to convert the original value into the initial node feature vector; Utilizing the message passing mechanism of graph neural networks, each node exchanges information with its neighboring nodes. In each layer, each node updates its own representation to include information about neighboring nodes, thereby capturing the mutual relationships between sensory data. Multiple iterative optimization: After multiple rounds of iterations of the graph neural network, each node obtains a new embedding vector, which contains the characteristics of the node itself and the information of its surrounding nodes. The embedding vector is represented as the learned high-dimensional feature vector.

2. A robot perception data word segmentation representation and learning method based on graph neural network according to claim 1, characterized in that: The perception data includes state data of each degree of freedom, end position data, visual perception data, tactile perception data and pressure sensor data.

3. The method for word segmentation representation and learning of robot perception data based on graph neural network according to claim 1 is characterized in that: In S2, for discrete data types, different categories of perception data are regarded as different word segmentations for word segmentation representation, corresponding to different nodes in the graph neural network.

4. The method for word segmentation representation and learning of robot perception data based on graph neural network according to claim 1 is characterized in that: In S2, for continuous numerical input, the graph attention mechanism network is directly used as the embedding network to learn the high-dimensional representation of numerical data and capture the relationship and structure between different numerical values.

5. The method for word segmentation representation and learning of robot perception data based on graph neural network according to claim 1 is characterized in that: In S2, for the time series data, it is divided into multiple time periods according to a preset time length, and feature extraction or encoding processing is performed on each time period to obtain a word segmentation representation; or, the time series data is transformed to obtain time domain or frequency domain features, and the features are used as word segmentation representations.

6. The method for word segmentation representation and learning of robot perception data based on graph neural network according to claim 1, characterized in that: The graph attention mechanism network weights the information of the neighbors by calculating the attention weights between nodes, which can be expressed as: in, is a node i In the l The layer representation, is the activation function, is the weight matrix, is a node i and j The attention coefficient between Is a node i The neighborhood set of .

7. The method for word segmentation representation and learning of robot perception data based on graph neural network according to claim 1, characterized in that: The decoder uses a multi-layer perceptron, specifically, three fully connected layers to decode the encoded information.

8. The method for word segmentation representation and learning of robot perception data based on graph neural network according to claim 1 is characterized in that: The actual physical constraints include: Dynamic constraints: Use the robot's dynamic model to define the dynamic constraints at each time step; Geometric constraints: Use the robot's kinematic model to define geometric constraints, including link length invariance and joint angle range constraints; Contact constraints: define the torque and force constraints on the contact points, including the constraint that the friction force cannot exceed the maximum static friction force; Energy conservation constraint: ensures that the total energy of the system is conserved, that is, the conversion between potential energy, kinetic energy and work done obeys the law of conservation of energy.

Images