Robot perception data word segmentation characterization and learning method based on graph neural network

Abstract

The invention relates to a robot sensing data word segmentation characterization and learning method based on a graph neural network. The method comprises the following steps: acquiring various sensing data of a robot; performing word segmentation representation according to the category of the perception data; word segmentation representation learning: constructing an initial feature map based on the perception data after word segmentation representation; a compact representation of the feature graph is learned based on an auto-encoder, a graph structure is reconstructed, and the graph structure represents the relationship of edges between different nodes; after the self-encoder completes the learning of the graph structure, the graph structure is fixed; and converting the perception data into node feature vectors, constructing a feature graph based on the learned graph structure, and performing numerical coding on each node feature vector by using a graph neural network to obtain high-dimensional feature vector representation of the perception data. Compared with the prior art, the method has the advantages that the complexity and multi-dimensional information of perception data can be better expressed, and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of robot perception data processing, and in particular to a robot perception data word segmentation representation and learning method based on graph neural network. Background Art

[0002] At present, in the field of natural language processing, word segmentation, as an important step in text preprocessing, has a crucial impact on subsequent tasks such as sentiment analysis, machine translation, and question-answering systems. Traditional word segmentation methods mainly rely on dictionary matching and statistical models, which often perform poorly when faced with new words, polysemous words, and long and difficult sentences. With the development of deep learning, neural network-based methods have gradually become mainstream, especially recurrent neural networks (RNNs) and long short-term memory networks (LSTMs), which have achieved good results in word segmentation tasks. Next, embedding technology converts these discrete words into vector representations in high-dimensional space, which not only enables computers to process text information, but more importantly, these vectors can mathematically capture the semantic and grammatical relationships between words, allowing machines to understand the content of the text to some extent. Therefore, whether for applications such as text classification, sentiment analysis, or machine translation, word segmentation and embedding provide the system with the ability to understand and process natural language. Together, they constitute the cornerstone of the modern natural language processing technology stack and greatly improve the accuracy and efficiency of various language processing tasks.

[0003] Modern robotics technology has been widely used in many fields such as industrial manufacturing and service industries. In order for robots to better understand and perform tasks, it is usually necessary to convert sensory data (such as degrees of freedom, end position, visual perception, touch, etc.) into a form that can be processed by computers. However, most existing sensory data processing methods rely on fixed data structures and simple feature extraction methods, which limits the robot's ability to understand complex environments and the flexibility of task execution.

[0004] With the development of artificial intelligence technology, especially the application of large pre-trained models, robotic systems have made significant progress in decision-making and control. These large models are able to handle complex tasks and learn patterns from large amounts of data to make more intelligent decisions. However, in order for these models to work in practical robotic applications, they need to be closely integrated with the robot's embodied sensory input.

[0005] Robot perception input includes but is not limited to data in multiple dimensions such as degrees of freedom, end-point posture, visual perception, and touch. In order for large models to effectively use this information, these perception data must be converted into a form that the model can understand. Ideally, these data should be converted into high-dimensional vectors, and in this conversion process, the relationship between perception data of different dimensions and the information they carry should be preserved.

[0006] At present, although relatively mature word segmentation and embedding methods have been developed in the field of natural language processing, which can well transform text into semantically rich vector representations, similar methods for robot perception data are relatively lacking. Existing technologies usually use relatively simple methods to process robot perception data, such as directly using raw values ​​or simple feature engineering. Such processing methods cannot fully express the complexity and multi-dimensional information of perception data.

[0007] In addition, due to the lack of effective word segmentation and embedding methods, existing robotic systems have difficulty fully utilizing the capabilities of large pre-trained models to process sensory inputs. This results in the robot's decision-making and control performance being limited when facing dynamic and complex environments, and it cannot respond to various situations as flexibly as humans.

[0008] Therefore, it is urgent to develop a method that can effectively transform robot perception data into high-dimensional vectors, and this method should be able to preserve the relationship between perception data in order to better serve the robot's decision-making control system. At present, research in this field is still in its infancy, and there is no mature technical solution to solve this problem. Summary of the invention

[0009] The purpose of the present invention is to provide a robot perception data segmentation representation and learning method based on Graph Neural Networks (GNN), which aims to convert the robot's multi-dimensional perception input into a high-dimensional vector and retain the relationship information between perception data of different dimensions.

[0010] The purpose of the present invention can be achieved by the following technical solutions:

[0011] A robot perception data word segmentation representation and learning method based on graph neural network includes the following steps:

[0012] S1, data acquisition: obtaining various sensory data of the robot;

[0013] S2, word segmentation representation according to the category of perception data;

[0014] S3, word segmentation representation learning:

[0015] S31, constructing an initial feature map based on the perception data after word segmentation representation;

[0016] S32, reconstructing a graph structure based on a compact representation of a feature graph learned by an autoencoder, wherein the graph structure represents the relationship between edges of different nodes, wherein the autoencoder includes an encoder and a decoder, wherein the encoder uses a graph attention mechanism network to map node features to a latent space, and the decoder reconstructs the original graph structure from the latent space;

[0017] S33, after the autoencoder completes the learning of the graph structure, the graph structure is fixed;

[0018] S34, converting the perception data into node feature vectors, and constructing a feature graph based on the learned graph structure, using a graph neural network to numerically encode the feature vectors of each node to obtain a high-dimensional feature vector representation of the perception data.

[0019] The perception data includes state data of each degree of freedom, end position data, visual perception data, tactile perception data and pressure sensor data.

[0020] In S2, for discrete data types, different categories of perception data are regarded as different word segmentations for word segmentation representation, corresponding to different nodes in the graph neural network.

[0021] In S2, for continuous numerical input, the graph attention mechanism network is directly used as the embedding network to learn the high-dimensional representation of numerical data and capture the relationship and structure between different numerical values.

[0022] In S2, for the time series data, it is divided into multiple time periods according to a preset time length, and feature extraction or encoding processing is performed on each time period to obtain a word segmentation representation; or, the time series data is transformed to obtain time domain or frequency domain features, and the features are used as word segmentation representations.

[0023] The method of learning a compact representation of a feature graph based on an autoencoder and reconstructing a graph structure comprises the following steps:

[0024] Random masking: select some nodes for masking. The masking method uses a random selection strategy. For the masked nodes, their feature vectors are replaced with zero vectors or a special mask mark.

[0025] Encoding: Use the graph attention network as an encoder to encode the unmasked nodes;

[0026] Decoding: The decoder receives the node representation and mask information generated by the encoder to predict the features of the masked nodes;

[0027] Loss function definition: Define the mean square error loss function to measure the difference between the decoder output and the actual mask node features, and introduce actual physical constraints into the loss function;

[0028] Back propagation and iterative training: Use the loss function to perform gradient descent on the autoencoder model parameters to minimize the loss until the model converges or reaches a predetermined number of training rounds;

[0029] The trained autoencoder is used to learn the node mapping of different types of perceptual data on the graph neural network, and connections between nodes are established based on the intrinsic connections between the perceptual data to form an undirected graph.

[0030] The graph attention mechanism network weights the information of aggregated neighbors by calculating the attention weights between nodes, expressed as:

[0031]

[0032] in, is the representation of node i in layer l, σ is the activation function, and w (l) is the weight matrix, α ij is the attention coefficient between nodes i and j, and N(i) is the neighborhood set representing node i.

[0033] The decoder uses a multi-layer perceptron, specifically, uses 3 fully connected layers to decode the encoded information.

[0034] The actual physical constraints include:

[0035] Dynamic constraints: Use the robot's dynamic model to define the dynamic constraints at each time step;

[0036] Geometric constraints: Use the robot's kinematic model to define geometric constraints, including link length invariance and joint angle range constraints;

[0037] Contact constraints: define the torque and force constraints on the contact points, including the constraint that the friction force cannot exceed the maximum static friction force;

[0038] Energy conservation constraint: Ensures that the total energy of the system is conserved, that is, the conversion between potential energy, kinetic energy, and work done obeys the law of conservation of energy.

[0039] The specific steps of using the graph neural network to numerically encode the feature vector of each node to obtain a high-dimensional feature vector representation of the perception data are as follows:

[0040] Initialize each perception data node and use a multi-layer perceptron to convert the original value into an initial node feature vector;

[0041] Using the message passing mechanism of graph neural networks, each node exchanges information with its neighboring nodes. In each layer, each node updates its own representation to include the information of neighboring nodes, thereby capturing the mutual relationship between perception data.

[0042] Multiple iterative optimization: After multiple rounds of iterations of the graph neural network, each node obtains a new embedding vector, which contains the characteristics of the node itself and the information of its surrounding nodes. The embedding vector is represented as a learned high-dimensional feature vector.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] (1) The present invention introduces the natural language word segmentation and embedding technology and ideas in the large model field into robot perception data processing, filling the technical gap in this field.

[0045] (2) The present invention adopts graph neural network (GNN) as the core technical means to process robot perception data. GNN can effectively model the relationship between different perception data and convert it into a high-dimensional vector representation, thereby retaining the relationship between the perception data. Compared with the traditional method of using only raw numerical values ​​or simple feature engineering, it can better express the complexity and multi-dimensional information of perception data.

[0046] (3) The present invention proposes a learning method that autonomously learns the word segmentation representation of the robot's multi-dimensional perception based on the physical system, while retaining the relationship between perception data of different dimensions, which is conducive to the large model to better understand the robot's decision-making and control in a complex environment when processing the robot's input information. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flow chart of the method of the present invention;

[0048] Figure 2 This is a diagram of the graph attention mechanism network model architecture of the present invention;

[0049] Figure 3 Constructing a flow chart for the perception data graph structure of the present invention;

[0050] Figure 4 This is a flowchart of numerical encoding of graph node feature vectors of the present invention. DETAILED DESCRIPTION

[0051] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0052] This embodiment provides a robot perception data segmentation representation and learning method based on graph neural network. Figure 1 As shown, the following steps are included:

[0053] S1, data acquisition: obtain various perception data of the robot.

[0054] In this embodiment, various perception data from the robot are collected, including but not limited to the state of each degree of freedom, end position, visual perception, touch, pressure sensor, etc., to construct a robot multimodal perception data set.

[0055] S2, word segmentation representation is performed according to the category of the perception data.

[0056] For various input perception data, it is necessary to perform word segmentation representation and convert it into a representation that the model can process. Different from tokenization in natural language processing, the input robot perception data is a discrete data type of different types. Each data is a continuous value. Therefore, it is necessary to establish a word segmentation representation paradigm. The details are as follows:

[0057] S21, for discrete data types, different categories of perception data are regarded as different word segmentations for word segmentation representation, corresponding to different nodes in the graph neural network.

[0058] S22, for continuous numerical input, the graph attention mechanism network is directly used as the embedding network to learn the high-dimensional representation of numerical data and capture the relationship and structure between different numerical values.

[0059] S23: For time series data (such as acceleration, temperature, distance and other sensor data), divide it into multiple time periods according to a preset time length, perform feature extraction or encoding processing on each time period to obtain a segmentation representation; or, transform the time series data to obtain time domain or frequency domain features, and use the features as segmentation representation.

[0060] S3, word segmentation representation learning.

[0061] The purpose of constructing the perception data graph is to encode the robot's various perception data and retain its own information and the relationship information between different perception data. The collected perception data all come from the robot's physical system, so they all conform to the laws of the physical world at the bottom. The autoencoder framework is used to learn a compact representation of the graph, where the encoder maps node features to the latent space, and the decoder reconstructs the original graph structure from the latent space.

[0062] Specifically, Figure 3 As shown, S3 includes the following steps:

[0063] S31, construct an initial feature graph G = (V, E) based on the perception data after word segmentation representation, where V is a node set, E is an edge set, and each node has a feature vector.

[0064] S32, based on the compact representation of the feature graph learned by the autoencoder, reconstructs the graph structure, where the graph structure represents the relationship between the edges of different nodes.

[0065] S321, random masking: select some nodes for masking, wherein the masking method uses a random selection strategy, and for the masked nodes, their feature vectors are replaced with a zero vector or a special mask mark.

[0066] S322, encoding: Use the Graph Attention Networks (GAT) as an encoder to encode the unmasked nodes. Figure 2 As shown in the figure, in the GAT network, each node will pay attention not only to its own features, but also to the features of other nodes to which it is connected. Specifically, each node in the graph will have an attention coefficient that determines the importance of its relationship with other nodes. First, the attention coefficient is calculated, which is usually a scalar value indicating the degree of attention of node i to node j. This means that attention can be allocated to all relevant nodes, not just a single node. The attention coefficient is then multiplied by the corresponding node features to obtain a weighted average that reflects the weighted summary of node i's features of its neighboring nodes. Finally, this weighted summary is combined with node i's own features to generate a new node representation.

[0067] That is, the graph attention mechanism network weights the information of aggregated neighbors by calculating the attention weights between nodes, expressed as:

[0068]

[0069] in, is the representation of node i in layer l, σ is the activation function, and w (l) is the weight matrix, α ij is the attention coefficient between nodes i and j, and N(i) is the neighborhood set representing node i.

[0070] S323, decoding: The decoder receives the node representation and mask information generated by the encoder to predict the features of the masked nodes.

[0071] This embodiment uses a multi-layer perceptron as a decoder, specifically, uses three fully connected layers to decode the encoded information.

[0072] S324, loss function definition: define a mean square error loss function to measure the difference between the decoder output and the actual mask node features. In addition, consider the actual physical system and introduce actual physical constraints into the loss function.

[0073] In this embodiment, the actual physical constraints include:

[0074] 1) Dynamic constraints: Use the robot's dynamic model to define the dynamic constraints at each time step, such as Newton's second law;

[0075] 2) Geometric constraints: Use the robot's kinematic model to define geometric constraints, such as constant link length and joint angle range constraints;

[0076] 3) Contact constraints: define the torque and force constraints on the contact point, such as the friction force cannot exceed the maximum static friction force constraint;

[0077] 4) Energy conservation constraint: Ensure that the total energy of the system is conserved, that is, the conversion between potential energy, kinetic energy and work done obeys the law of conservation of energy.

[0078] The original loss function is then combined with one or more additional loss terms that penalize predictions that do not conform to the laws of physics. The final loss function can be expressed as:

[0079] L=L base +λL phys

[0080] where λ is a hyperparameter used to tune the importance of the physical constraint.

[0081] The specific forms of physical constraints in dynamic constraints and geometric constraints can be exemplified as

[0082] For a general second-order system, the loss function under dynamic constraints can be expressed as

[0083]

[0084] Here m is mass, c and k are physical coefficients, and F(t) is external force. are the predicted position and velocity respectively.

[0085] For a robotic arm, if the length of the links is known to be fixed, a constraint can be added to penalize those predicted joint angle combinations, such as

[0086] L phys =||l1 cos(θ1)+l2 cos(θ1+θ2)||-d 12 | 2

[0087] Among them, l i represents the length of the i-th connecting rod, θ i is the angle of joint i, d 12 is the theoretical fixed distance between the two ends.

[0088] The establishment of other loss functions can refer to the above method, which will not be described in detail in this embodiment.

[0089] S325, back propagation and iterative training: Use the loss function to perform gradient descent on the autoencoder model parameters to minimize the loss. The Adam optimization algorithm can be used to repeat the above steps until the model converges or reaches the predetermined training rounds.

[0090] S326, using the trained autoencoder to learn the node mapping of different types of perceptual data on the graph neural network, establish connections between nodes based on the intrinsic connections between the perceptual data (such as causal relationships, spatial relationships, etc.), and form an undirected graph.

[0091] S33, after the autoencoder completes the learning of the graph structure, the graph structure is fixed.

[0092] S34, converting the perception data into node feature vectors, and constructing a feature graph based on the learned graph structure, using a graph neural network to numerically encode the feature vectors of each node to obtain a high-dimensional feature vector representation of the perception data.

[0093] like Figure 4 As shown, the specific steps are as follows:

[0094] S341, initializing each perception data node. In this embodiment, a multi-layer perceptron is used to convert the original value into an initial node feature vector.

[0095] S342, using the message passing mechanism of the graph neural network (GNN), allows each node to exchange information with its neighboring nodes. In each layer, each node updates its own representation to include the information of neighboring nodes, thereby capturing the mutual relationship between the perception data;

[0096] S343, multiple iterative optimization: After multiple rounds of iterations of the graph neural network, each node obtains a new embedding vector, which contains the characteristics of the node itself and the information of its surrounding nodes. The embedding vector is represented as a high-dimensional feature vector obtained by learning.

[0097] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning, or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A robot perception data word segmentation representation and learning method based on graph neural network, characterized in that: The following steps are involved: S1, data acquisition: obtaining various sensory data of the robot; S2, word segmentation representation according to the category of perception data; S3, word segmentation representation learning: S31, constructing an initial feature map based on the perception data after word segmentation representation; S32, reconstructing a graph structure based on a compact representation of a feature graph learned by an autoencoder, wherein the graph structure represents the relationship between edges of different nodes, wherein the autoencoder includes an encoder and a decoder, wherein the encoder uses a graph attention mechanism network to map node features to a latent space, and the decoder reconstructs the original graph structure from the latent space; S33, after the autoencoder completes the learning of the graph structure, the graph structure is fixed; S34, converting the perception data into node feature vectors, and constructing a feature graph based on the learned graph structure, using a graph neural network to numerically encode the feature vectors of each node to obtain a high-dimensional feature vector representation of the perception data.

2. According to claim 1, a robot perception data segmentation representation and learning method based on graph neural network is characterized in that: The perception data includes state data of each degree of freedom, end position data, visual perception data, tactile perception data and pressure sensor data.

3. According to claim 1, a robot perception data word segmentation representation and learning method based on graph neural network is characterized in that: In S2, for discrete data types, different categories of perception data are regarded as different word segmentations for word segmentation representation, corresponding to different nodes in the graph neural network.

4. According to claim 1, a robot perception data word segmentation representation and learning method based on graph neural network is characterized in that: In S2, for continuous numerical input, the graph attention mechanism network is directly used as the embedding network to learn the high-dimensional representation of numerical data and capture the relationship and structure between different numerical values.

5. According to claim 1, a robot perception data segmentation representation and learning method based on graph neural network is characterized in that: In S2, for the time series data, it is divided into multiple time periods according to a preset time length, and feature extraction or encoding processing is performed on each time period to obtain a word segmentation representation; or, the time series data is transformed to obtain time domain or frequency domain features, and the features are used as word segmentation representations.

6. The method for representing and learning robot perception data word segmentation based on graph neural network according to claim 1, characterized in that: The method of learning a compact representation of a feature graph based on an autoencoder and reconstructing a graph structure comprises the following steps: Random masking: select some nodes for masking. The masking method uses a random selection strategy. For the masked nodes, their feature vectors are replaced with zero vectors or a special mask mark. Encoding: Use the graph attention network as an encoder to encode the unmasked nodes; Decoding: The decoder receives the node representation and mask information generated by the encoder to predict the features of the masked nodes; Loss function definition: Define the mean square error loss function to measure the difference between the decoder output and the actual mask node features, and introduce actual physical constraints into the loss function; Back propagation and iterative training: Use the loss function to perform gradient descent on the autoencoder model parameters to minimize the loss until the model converges or reaches a predetermined number of training rounds; The trained autoencoder is used to learn the node mapping of different types of perceptual data on the graph neural network, and connections between nodes are established based on the intrinsic connections between the perceptual data to form an undirected graph.

7. The method for representing and learning robot perception data word segmentation based on graph neural network according to claim 1 is characterized in that: The graph attention mechanism network weights the information of aggregated neighbors by calculating the attention weights between nodes, expressed as: in, is the representation of node i in layer l, σ is the activation function, and w (l) is the weight matrix, α ij is the attention coefficient between nodes i and j, and N(i) is the neighborhood set representing node i.

8. The method for representing and learning robot perception data word segmentation based on graph neural network according to claim 1, characterized in that: The decoder uses a multi-layer perceptron, specifically, uses 3 fully connected layers to decode the encoded information.

9. The method for representing and learning robot perception data word segmentation based on graph neural network according to claim 6 is characterized in that: The actual physical constraints include: Dynamic constraints: Use the robot's dynamic model to define the dynamic constraints at each time step; Geometric constraints: Use the robot's kinematic model to define geometric constraints, including link length invariance and joint angle range constraints; Contact constraints: define the torque and force constraints on the contact points, including the constraint that the friction force cannot exceed the maximum static friction force; Energy conservation constraint: Ensures that the total energy of the system is conserved, that is, the conversion between potential energy, kinetic energy, and work done obeys the law of conservation of energy.

10. The method for representing and learning robot perception data word segmentation based on graph neural network according to claim 1, characterized in that: The specific steps of using the graph neural network to numerically encode the feature vector of each node to obtain a high-dimensional feature vector representation of the perception data are as follows: Initialize each perception data node and use a multi-layer perceptron to convert the original value into an initial node feature vector; Using the message passing mechanism of graph neural networks, each node exchanges information with its neighboring nodes. In each layer, each node updates its own representation to include the information of neighboring nodes, thereby capturing the mutual relationship between perception data. Multiple iterative optimization: After multiple rounds of iterations of the graph neural network, each node obtains a new embedding vector, which contains the characteristics of the node itself and the information of its surrounding nodes. The embedding vector is represented as a learned high-dimensional feature vector.

Images