Knowledge-enhanced robot long-sequence operation decision-making method

Abstract

The knowledge-enhanced robot long-sequence operation decision-making method comprises the following steps: step A, collecting description statements and action sequences of various tasks, and constructing a plurality of task description-action sequence pairs as training samples; step B, constructing a professional domain knowledge graph, and for each entity in the professional domain knowledge graph, generating a brief description of basic information of the entity through a manual or dialogue large model; c, inputting each entity and the corresponding description thereof into the general field entity embedding model to obtain general field vector representation; step D, training a knowledge enhancement sequence reasoning model based on the graph convolutional neural network and the Seq2Seq model; e, inputting task description and a professional domain knowledge graph, and generating an action sequence; f, the robot sequentially selects and executes corresponding packaging action functions from the action primitive library according to the generated action sequence; according to the method, the understanding ability and the long sequence operation reasoning ability of the robot are effectively improved, and high-level decision and bottom-level execution are effectively decoupled.

Description

Technical Field

[0001] The present invention relates to the technical field of robot operation decision-making, and in particular to a knowledge-enhanced robot long-sequence operation decision-making method. Background Art

[0002] With the rapid development of artificial intelligence (AI) technology and the growing demand for intelligent lifestyles and a more responsive workforce, the integration of robots into daily life and across all industries is inevitable. These robots will need to possess a high degree of adaptability and autonomous decision-making capabilities to cope with diverse and complex environments and tasks. However, existing robots are primarily designed for simple tasks and fixed scenarios. They lack the decision-making capabilities for complex, long-term operations and struggle to adapt to dynamic, multi-factor, and unstructured environments. Therefore, improving robots' ability to perceive and understand their environment and tasks, as well as their decision-making and execution accuracy in long-term operations, is crucial for their widespread application in complex social environments.

[0003] Among the existing robot operation decision-making technology solutions, most are based on methods such as reinforcement learning, imitation learning, and pre-trained large models. Examples of these methods are as follows:

[0004] 1. Patent CN116402164A discloses a robot task generation method, dedicated device and medium based on a pre-trained language model. Through prompt engineering, the pre-trained model outputs skill descriptions and matches them with the skill list to obtain the optimal skills.

[0005] 2. Patent CN118906066A discloses a robot decision-making method based on link prediction relevance, which models the motion trajectory of each time step into a professional domain knowledge graph, calculates the relevance of trajectories at different time steps through a link prediction method, and then optimizes the complete operation trajectory.

[0006] 3. Patent CN 111618862A discloses a robot operation skill learning system and method guided by prior knowledge. It is based on the reinforcement learning method and improves the model learning efficiency by pre-establishing an expert knowledge base that maps the operation contact state to the robot arm movement.

[0007] However, the operation decision-making method based on reinforcement learning is suitable for basic action learning, but it is difficult to learn in long sequence operations due to the huge action space; the decision-making method based on imitation learning can refine the underlying action learning, but lacks understanding and adaptation to complex environments; the method based on pre-training models is often used for high-level operation decision-making, but due to the lack of domain-specific data and high computational overhead, it is not conducive to efficient and accurate domain operation decision-making. Summary of the Invention

[0008] The present invention aims to overcome the aforementioned shortcomings of the prior art. Specifically, it addresses the difficulties robots face in learning long-sequence operations and their poor adaptability to complex environments. By proposing a knowledge-enhanced decision-making method for long-sequence operations, the method effectively improves robots' understanding and reasoning capabilities for long-sequence operations, effectively decouples high-level decision-making from low-level execution, and offers strong scalability.

[0009] To achieve the above objectives, the present invention provides a knowledge-enhanced robot long-sequence operation decision-making method, which includes the following steps:

[0010] Step A: Collect descriptions and action sequences of various tasks and construct multiple task description-action sequence pairs as training samples;

[0011] Step B: Build a professional domain knowledge graph and generate a brief description of the basic information for each entity in the professional domain knowledge graph through manual or dialogue model;

[0012] Step C: Input each entity and its corresponding description in the professional domain knowledge graph into the pre-trained general domain entity embedding model to obtain the general domain vector representation of each entity;

[0013] Step D: Use the domain knowledge graph and task description-action sequence pairs as training data, and use the general domain vector obtained in step C above as the initialization vector to train a knowledge-enhanced sequential reasoning model based on a graph convolutional neural network and a Seq2Seq model;

[0014] Step E: In actual application, the task description and professional domain knowledge graph are input. First, the pre-trained general domain entity embedding model is used to obtain the general domain vector representation of the relevant entities. Then, the pre-trained knowledge-enhanced sequential reasoning model is input to generate the action sequence.

[0015] Step F: The robot selects and executes the corresponding encapsulated action functions from the action primitive library according to the generated action sequence;

[0016] The action sequence is composed of multiple action descriptions arranged in execution order, each action description includes an action primitive and several action parameters, and each action description is distinguished by a segmentation word interval;

[0017] The action primitive library is a variety of pre-prepared encapsulated action functions, each of which has several parameters.

[0018] Preferably, in step B, the professional domain knowledge graph includes five types of nodes: task, action, object, place and attribute. The attribute type nodes point to the task, action, object and place type nodes through directed edges to represent their attributes.

[0019] Preferably, the action type nodes in the professional domain knowledge graph include the following action primitives:

[0020] (1) Movement: translation, steering and positioning;

[0021] (2) Operations: grabbing, releasing, pushing, pulling, inserting, pressing, end-end translation, end-end steering, and end-end positioning;

[0022] (3) Perception: target recognition, sound recognition, target positioning, and sensor reading;

[0023] (4). Other categories: path planning.

[0024] Preferably, in step C, the pre-trained general domain entity embedding model adopts a BERT model, and the model is pre-trained based on a dual-tower architecture designed for a text-entity similarity task;

[0025] Among them, the left tower is a text encoder, which encodes the input entity description text into a feature vector;

[0026] The right tower acts as an entity encoder, encoding entities into feature vectors based on their attributes.

[0027] Preferably, in step D, the Seq2Seq model is implemented based on a gated recurrent unit (GRU) and a Bahdanau attention mechanism.

[0028] Preferably, in step D, the graph convolutional neural network is based on a task-aware graph attention mechanism. Specifically, under the task task, the attention calculation process is:

[0029]

[0030] in, It is the node e in the professional field knowledge graph i The general domain vector representation, W e is the learnable node mapping matrix, h i for The hidden vector after mapping is represented, N(i) represents the node e i The set of adjacent nodes, the ELU activation layer is used to introduce nonlinear relationships, h i Hidden vector representation after message aggregation. α ij Represents the adjacent node e j For e i The attention weight, α ij It is calculated by additive attention and introduces a learnable vector a to capture more complex feature interactions. The calculation process is expressed as:

[0031]

[0032] in, It is the relationship r in the professional field knowledge graph k The general domain vector representation, W r is the learnable relation mapping matrix, z k for The hidden vector representation after mapping, h task Represents the hidden vector representation of the task entity, R(i,j,task) represents the node e under the task task i and adjacent node e j The set of all relations s ij Represents node e j For e i The attention score is adjusted by the task task and can represent the different influences between nodes in different tasks. ij Score s ij The result after Softmax normalization.

[0033] Preferably, in step D, the graph convolutional neural network and the Seq2Seq model are combined in such a way that the initial vector of the input entity of the Seq2Seq model is the professional domain vector representation of the corresponding entity output by the graph convolutional neural network.

[0034] Preferably, in step E, the loss function ζ1 of the knowledge-enhanced sequential reasoning model is a negative likelihood logarithmic loss, and the calculation process is:

[0035]

[0036] Where Y is the set of all words that appear in the sequence, L is the length of the nth output sentence, and y ij is the binary representation of the jth word appearing in the i-th position of the output sentence, 1 represents y ij It is exactly the true label, 0 means it is not the true label, p ij y represents the model judgment ij The probability size, N represents the total number of sentences in the training set.

[0037] Compared with the prior art, the present invention has the following beneficial effects:

[0038] 1. This invention decouples high-level decision-making from low-level execution, generates high-level operational decisions through a knowledge-enhanced sequential reasoning model, and then selects action functions from a library of action primitives for execution, effectively overcoming the learning difficulties of traditional reinforcement learning or imitation learning methods for long sequences of operations;

[0039] This invention is based on professional domain knowledge graphs and small-scale general domain pre-training models to effectively enhance the model's ability to understand entities;

[0040] The knowledge-enhanced sequential reasoning model proposed in this invention is smaller in scale and more interpretable than the pre-training model and reinforcement learning model.

[0041] 2. In summary, the present invention provides a knowledge-enhanced robot decision-making method for long-sequence operations. This method addresses the difficulties robots face in learning long-sequence operations and their poor adaptability to complex environments. This method leverages domain-specific knowledge graphs and small-scale pre-trained models to improve the robot's understanding of different entities. It further utilizes a knowledge-enhanced sequential reasoning model to reason about action sequences for different tasks, and finally selects corresponding action functions from a library of action primitives for execution. This method effectively improves the robot's understanding and reasoning capabilities for long-sequence operations, effectively decouples high-level decision-making from low-level execution, and offers strong scalability. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1 A flowchart of a knowledge-enhanced robot long-sequence operation decision-making method provided by an embodiment of the present invention;

[0044] Figure 2 A schematic diagram of task description-action sequence pairs in a robot 3C assembly embodiment provided by an embodiment of the present invention;

[0045] Figure 3 A schematic diagram of a professional domain knowledge graph triple in a robot 3C assembly embodiment provided by an embodiment of the present invention;

[0046] Figure 4 A schematic diagram of ChatGPT generating entity description is provided for an embodiment of the present invention;

[0047] Figure 5 A model framework diagram provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The following will clearly and completely describe the technical solution in this embodiment of the present invention in conjunction with the drawings in this embodiment of the present invention. Obviously, the embodiment described is only one embodiment of the present invention, not all embodiments of the present invention. Based on this embodiment of the present invention, all other embodiments of the present invention obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0049] See also Figure 1 The embodiment of the present invention provides a knowledge-enhanced robot long sequence operation decision-making method. In this embodiment, the method is applied to the field of robot 3C assembly. In other embodiments, the method can be applied to other fields of robots. The method includes the following steps:

[0050] Step A: Collect descriptions and action sequences of various tasks and construct multiple task description-action sequence pairs as training samples;

[0051] Here are some examples of how to collect description statements and action sequences for various tasks:

[0052] like Figure 2 As shown, in this embodiment, the task description is a structured statement "task name location object target object rule"; the action sequence is a structured statement "action 1 parameter 1 parameter 2..., action 2 parameter 1, action 3 parameter 1 parameter 2..., action 4...", which is composed of multiple action descriptions arranged in order of execution, each action description includes an action primitive and several action parameters, and each action description is distinguished by a separator ",".

[0053] "Get the male connector of the soft flat cable, material area, male connector of the soft flat cable, none" means that the current task is "Get the male connector of the soft flat cable", the target location is the "material area", the object to be obtained is the "male connector of the soft flat cable", and the target object and rule do not exist.

[0054] "Move the male connector of the soft flat cable to the transfer area and fit it to the secondary positioning table" means that the current task is to "move the male connector of the soft flat cable", the target position is the "transfer area", the object is the "male connector of the soft flat cable", and it needs to be placed on the target object "secondary positioning table", and the placement rule is "fitting to the positioning table".

[0055] For example, in the "Get Flex Cable Male Connector" task, the action parameters for the "Detect" action are "Material Area," "Flex Cable," "Flex Cable Position," and "Flex Cable Shape." This means the target needs to identify the material area and the flex cable male connector, and then detect the position and shape of the flex cable male connector. The detection results are stored in the database as key-value pairs (variable name - variable value).

[0056] The action parameter of the "approach" action is "soft cable position value", which means that the function needs to query the corresponding variable value from the database as a parameter based on the variable name.

[0057] Step B: Build a professional domain knowledge graph and generate a brief description of the basic information for each entity in the professional domain knowledge graph through manual or dialogue model;

[0058] Among them, in step B, the professional domain knowledge graph in the 3C assembly field contains at least five types of nodes: tasks, actions, objects, locations, and attributes. The attribute type nodes point to the task, action, object, and location type nodes through directed edges to represent their attributes. Traditional knowledge graphs organize knowledge in the form of triples (head entity, relationship, tail entity), ignoring the dynamic nature of the relationship between entities, that is, different weights under different conditions. Therefore, knowledge is organized into a four-tuple form (task, head entity, relationship, tail entity) that includes task conditions, thereby effectively representing knowledge under different task conditions.

[0059] Since part attributes have a greater impact on operational decisions in 3C assembly scenarios, the professional domain knowledge graph mainly includes parts and their attributes. Figure 3 As shown in the figure, part attributes include functions, assembly rules, precision, type, material and other attributes.

[0060] like Figure 4 As shown, for all entities that appear (such as the parts mentioned above), ChatGPT is used to batch generate a popular and brief description of the entity and store it in text.

[0061] More specifically, in step B, the action type nodes in the professional domain knowledge graph include the following action primitives:

[0062] (1) Movement: translation, steering and positioning;

[0063] (2) Operations: grabbing, releasing, pushing, pulling, inserting, pressing, end-end translation, end-end steering, and end-end positioning;

[0064] (3) Perception: target recognition, sound recognition, target positioning, and sensor reading;

[0065] (4). Other categories: path planning.

[0066] At this point, three types of data are obtained from steps A and B: professional domain knowledge graph triples, task description-action sequence pairs, and entity description sentences.

[0067] Step C: Input each entity and its corresponding description in the professional domain knowledge graph into the pre-trained general domain entity embedding model to obtain the general domain vector representation of each entity;

[0068] In step C, the pre-trained general domain entity embedding model adopts the BERT model, and the model is pre-trained based on a dual-tower architecture designed for text-entity similarity tasks;

[0069] Among them, the left tower is a text encoder, which encodes the input entity description text into a feature vector;

[0070] The right tower acts as an entity encoder, encoding entities into feature vectors based on their attributes.

[0071] Through training, the encoded feature vectors of the left and right towers become more similar, thereby strengthening the association between text and entities. More specifically, through training, the left tower can effectively encode text containing entity descriptions into the entity's feature representation, while the right tower directly encodes the entity itself, aligning the two in the feature space. Ultimately, the feature vectors of the left and right towers gradually converge, strengthening the semantic connection between text and entities. This enables the model to capture entity features through easy-to-understand descriptions, avoiding the impact of overly technical and complex entity names that can affect representation.

[0072] The BERT model is the trained Zuota BERT model.

[0073] The specific input method of step C is as follows: Figure 5 As shown, each entity is represented by <e>and <s>Surround to tell the BERT model the entity that needs to be encoded, followed by a description sentence, and the concatenated sentence is input into the BERT model. The first vector output by the BERT model represents the general domain vector representation of the entity.

[0074] Step D: Use the domain knowledge graph and task description-action sequence pairs as training data, and use the general domain vector obtained in step C above as the initialization vector to train a knowledge-enhanced sequential reasoning model based on a graph convolutional neural network and a Seq2Seq model;

[0075] like Figure 5 As shown in Figure 1, the knowledge-enhanced sequential reasoning model consists of a graph convolutional neural network and a Seq2Seq model. Specifically, the input method is to feed the domain-specific knowledge graph represented by the general domain vector into the graph convolutional neural network to obtain the domain-specific vector representation of the entity. The task description represented by the domain-specific vector is then fed into the Seq2Seq model to generate the action sequence corresponding to the task. This action sequence is then compared with the labeled action sequence to calculate the loss for model training.

[0076] The Seq2Seq model is implemented based on the gated recurrent unit GRU and the Bahdanau attention mechanism.

[0077] The graph convolutional neural network is based on a task-aware graph attention mechanism. Specifically, under the task task, the attention calculation process is:

[0078]

[0079] in, It is the node e in the professional field knowledge graph i The general domain vector representation, W e is the learnable node mapping matrix, h i for The hidden vector after mapping is represented, N(i) represents the node e i The set of adjacent nodes, the ELU activation layer is used to introduce nonlinear relationships, h i Hidden vector representation after message aggregation. α ij Represents the adjacent node e j For e i The attention weight, α ij It is calculated by additive attention and introduces a learnable vector a to capture more complex feature interactions. The calculation process is expressed as:

[0080]

[0081] in, It is the relationship r in the professional field knowledge graph k The general domain vector representation, W r is the learnable relation mapping matrix, z k for The hidden vector representation after mapping, h task Represents the hidden vector representation of the task entity, R(i,j,task) represents the node e under the task tansk i and adjacent node e j The set of all relations s ij Represents node e j For e i The attention score is adjusted by the task task and can represent the different influences between nodes in different tasks. ij Score s ij The result after Softmax normalization.

[0082] It can be seen that with this design, the attention between nodes is not only affected by the general domain vector representation of the nodes, but also adjusted by the task, making the model more flexible.

[0083] The graph convolutional neural network and the Seq2Seq model are combined in such a way that the initial vector of the input entity of the Seq2Seq model is the professional field vector representation of the corresponding entity output by the graph convolutional neural network.

[0084] When using the domain knowledge graph and task description-action sequence pairs as training data, the task description-action sequence pairs are divided into training set, validation set, and test set in a ratio of 3:1:1. Evaluation indicators include:

[0085] (1) Whole sentence accuracy, that is, the entire action sequence generated by the model is completely consistent with the label action sequence, which can be formulated as:

[0086]

[0087] (2) Perplexity. The lower the perplexity, the higher the confidence of each word generated by the model. It can be formulated as:

[0088]

[0089] Where M is the total number of output sentences in the dataset, L is the length of each output sentence, and N is the total number of output words in the dataset. ij P(w ij ) represents w ij The predicted probability of .

[0090] Step E: In actual application, the task description and professional domain knowledge graph are input. First, the pre-trained general domain entity embedding model is used to obtain the general domain vector representation of the relevant entities. Then, the pre-trained knowledge-enhanced sequential reasoning model is input to generate the action sequence.

[0091] In step E, the loss function ζ1 of the knowledge-enhanced sequential reasoning model is the negative likelihood logarithmic loss, and the calculation process is:

[0092]

[0093] Where Y is the set of all words that appear in the sequence, L is the length of the nth output sentence, and y ij is the binary representation of the jth word appearing in the i-th position of the output sentence, 1 represents y ij It is exactly the true label, 0 means it is not the true label, p ij y represents the model judgment ij The probability size, N represents the total number of sentences in the training set.

[0094] Step F: The robot selects and executes the corresponding encapsulated action functions from the action primitive library according to the generated action sequence;

[0095] The action sequence is composed of multiple action descriptions arranged in execution order, each action description includes an action primitive and several action parameters, and each action description is distinguished by a segmentation word interval;

[0096] The action primitive is a predefined minimum indivisible action;

[0097] The action primitive library is a variety of pre-prepared encapsulated action functions, each of which has several parameters.

[0098] Among them, the action primitives in this embodiment include "detection", "approach", "suction", "lifting", "posture adjustment", "releasing", "alignment", "pressing", "pulling", "pushing", "lifting", "inserting", "grasping", "clamping", "rotation" and "twisting". These action primitives are implemented through reinforcement learning, hard programming or algorithm models. For example, detection includes general target detection, visual positioning and other algorithm models, while complex actions such as grasping and lifting are implemented through reinforcement learning.

[0099] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.< / s> < / e>

Claims

1. A knowledge-enhanced robot long-sequence operation decision-making method, characterized by: It includes the following steps: Step A: Collect description sentences and action sequences of various tasks; Step B: Build a professional domain knowledge graph, and generate a brief description of the basic information for each entity in the knowledge graph; Step C: Input the pre-trained general domain entity embedding model to obtain the general domain vector representation of each entity; Step D: Use the domain knowledge graph and task description-action sequence pairs as training data, and use the general domain vector obtained in step C above as the initialization vector to train a knowledge-enhanced sequential reasoning model based on a graph convolutional neural network and a Seq2Seq model; Step E: In actual application, the task description and professional domain knowledge graph are input. First, the pre-trained general domain entity embedding model is used to obtain the general domain vector representation of the relevant entities. Then, the pre-trained knowledge-enhanced sequential reasoning model is input to generate the action sequence. Step F: The robot selects and executes the corresponding encapsulated action functions from the action primitive library according to the generated action sequence; The action sequence is composed of multiple action descriptions arranged in execution order, each action description includes an action primitive and several action parameters, and each action description is distinguished by a segmentation word interval; The action primitive library is a variety of pre-prepared encapsulated action functions, each of which has several parameters.

2. A knowledge-enhanced robot long sequence operation decision-making method according to claim 1, characterized in that: In step B, the professional domain knowledge graph includes five types of nodes: task, action, object, location and attribute. The attribute type nodes point to the task, action, object and location type nodes through directed edges to represent their attributes.

3. A knowledge-enhanced robot long sequence operation decision-making method according to claim 2, characterized in that: The action type nodes in the professional domain knowledge graph include the following action primitives: (1) Movement: translation, steering and positioning; (2) Operations: grabbing, releasing, pushing, pulling, inserting, pressing, end-end translation, end-end steering, and end-end positioning; (3) Perception: target recognition, sound recognition, target positioning, and sensor reading; (4). Other categories: path planning.

4. The knowledge-enhanced robot long sequence operation decision-making method according to claim 1 is characterized in that: In step C, the pre-trained general domain entity embedding model adopts the BERT model, and the model is pre-trained based on a dual-tower architecture designed for text-entity similarity tasks; Among them, the left tower is a text encoder, which encodes the input entity description text into a feature vector; The right tower acts as an entity encoder, encoding entities into feature vectors based on their attributes.

5. The knowledge-enhanced robot long-sequence operation decision-making method according to claim 1 is characterized in that: In step D, the Seq2Seq model is implemented based on the gated recurrent unit GRU and the Bahdanau attention mechanism.

6. The knowledge-enhanced robot long sequence operation decision-making method according to claim 1, characterized in that: In step D, the graph convolutional neural network is based on a task-aware graph attention mechanism. Specifically, under the task task, the attention calculation process is: in, It is the node e in the professional field knowledge graph i The general domain vector representation, W e is the learnable node mapping matrix, h i for The hidden vector after mapping is represented, N(i) represents the node e i The set of adjacent nodes, the ELU activation layer is used to introduce nonlinear relationships, h i Hidden vector representation after message aggregation. α ij Represents the adjacent node e j For e i The attention weight, α ij It is calculated by additive attention and introduces a learnable vector a to capture more complex feature interactions. The calculation process is expressed as: s ij =∑ k∈R(i,j,task) LeakyReLU(a(h i ||h j ||z k ||h task )) in, It is the relationship r in the professional field knowledge graph k The general domain vector representation, W r is the learnable relation mapping matrix, z k for The hidden vector representation after mapping, h task Represents the hidden vector representation of the task entity, R(i,j,task) represents the node e under the task task i and adjacent node e j The set of all relations s ij Represents node e j For e i The attention score is adjusted by the task task and can represent the different influences between nodes in different tasks. ij Score s ij The result after Softmax normalization.

7. A knowledge-enhanced robot long sequence operation decision-making method according to claim 1 or 6, characterized in that: In step D, the graph convolutional neural network and the Seq2Seq model are combined in such a way that the initial vector of the input entity of the Seq2Seq model is the professional domain vector representation of the corresponding entity output by the graph convolutional neural network.

8. The knowledge-enhanced robot long sequence operation decision-making method according to claim 1 is characterized in that: In step E, the loss function ζ1 of the knowledge-enhanced sequential reasoning model is the negative likelihood logarithmic loss, and the calculation process is: Where Y is the set of all words that appear in the sequence, L is the length of the nth output sentence, and y ij is the binary representation of the jth word appearing in the i-th position of the output sentence, 1 represents y ij It is exactly the true label, 0 means it is not the true label, p ij y represents the model judgment ij The probability size, N represents the total number of sentences in the training set.

Images