Robot operation task planning method and system based on knowledge graph and large model

Abstract

The application discloses a robot operation task planning method and system based on a knowledge graph and a large model. The robot operation task planning method based on the knowledge graph and the large model comprises the following steps: constructing a knowledge graph, obtaining a large model, and adjusting the large model to obtain a target large model; obtaining a target operation task of a robot, obtaining a task subgraph according to the target operation task based on the knowledge graph; and obtaining an operation plan of the robot according to the task subgraph based on the target large model. The application can realize accurate reasoning and planning of complex and fine operation tasks of the robot, and improve the execution efficiency of the robot in completing the task target.

Description

Technical Field

[0001] This application relates to the field of robot operation technology, and in particular to a robot operation task planning method and system based on knowledge graphs and large models. Background Technology

[0002] Existing robot operation reasoning technologies are mainly divided into reasoning based on distributed representation learning, reasoning based on neural networks, and operation reasoning based on reinforcement learning.

[0003] The central idea of ​​representation-based reasoning methods is to find a mapping function that maps symbolic representations to vector spaces for numerical representation, thereby reducing the curse of dimensionality and capturing implicit associations between entities and relationships. Importantly, these methods can be directly computed and are fast. However, this approach struggles to handle complex physical interactions, lacks fine-grained manipulation capabilities, and is difficult to maintain consistency and coherence.

[0004] Neural network-based reasoning methods train large amounts of data using deep learning models (such as convolutional neural networks, recurrent neural networks, or transformers) to automatically extract features and perform reasoning. The advantage of this method is that it can handle complex nonlinear relationships and has strong generalization ability. However, this method has disadvantages such as difficulty in handling high-dimensional continuous action spaces, insufficient modeling of the physical world, and poor generalization ability.

[0005] Operational reasoning methods based on reinforcement learning learn how to maximize a certain reward signal through a series of actions by interacting with the environment. They are suitable for solving sequential decision-making problems, and have strong adaptability and flexibility, especially in robot operation tasks. However, this method has the disadvantages of low sample efficiency and difficulty in handling long-term dependent tasks. In particular, reinforcement learning methods have poor robustness in fine operation tasks.

[0006] Therefore, although reasoning methods based on distributed representation learning, neural networks, and reinforcement learning have made some progress in robot operation reasoning, they still have significant limitations when dealing with fine manipulation tasks.

[0007] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0008] The main purpose of this application is to provide a robot operation task planning method and system based on knowledge graphs and large models, aiming to solve the problem of low efficiency of existing robot operation reasoning technology when handling fine operation tasks.

[0009] The first aspect of this application provides a robot operation task planning method based on knowledge graphs and large models. The robot operation task planning method based on knowledge graphs and large models includes: constructing a knowledge graph and obtaining a large model, and adjusting the large model to obtain a target large model; obtaining the target operation task of the robot, and obtaining a task subgraph based on the knowledge graph and the target operation task; and obtaining the robot operation plan based on the target large model and the task subgraph.

[0010] Optionally, in one embodiment of this application, the construction of the knowledge graph specifically includes: obtaining the robot's operation task; decomposing the operation task into multiple sub-tasks, wherein the multiple sub-tasks are associated with corresponding skills, actions, intelligent agents, and scenarios; associating the decomposed multiple sub-tasks with the domain knowledge corresponding to the robot to obtain the knowledge graph required for task execution.

[0011] Optionally, in one embodiment of this application, the construction of the knowledge graph further includes: obtaining feedback information from the robot during task execution; updating the knowledge graph based on the feedback information; and using the updated knowledge graph as the knowledge graph.

[0012] Optionally, in one embodiment of this application, adjusting the large model to obtain the target large model specifically includes: decomposing the weight matrix of the large model into two low-rank matrices; obtaining the operation dataset corresponding to the robot; and training the two low-rank matrices based on the operation dataset to obtain the target large model.

[0013] Optionally, in one embodiment of this application, obtaining a task subgraph based on the knowledge graph and the target operation task specifically includes: parsing the target operation task to obtain entities corresponding to the target operation task, wherein the entities include any one of objects, operation actions, scene environments, task objectives, and skill types; retrieving knowledge nodes and edge relationships corresponding to the target operation task from the knowledge graph based on the entities; and organizing the knowledge nodes and edge relationships to obtain a task subgraph oriented towards the target operation task.

[0014] Optionally, in one embodiment of this application, the step of parsing the target operation task to obtain the entity corresponding to the target operation task specifically includes: performing language processing on the target operation task using a bidirectional long short-term memory network to obtain a language information vector; processing the language information vector using a graph attention network to obtain feature information corresponding to the target operation task; and predicting the feature information using a conditional random field to obtain the entity corresponding to the target operation task.

[0015] Optionally, in one embodiment of this application, obtaining the robot's operation plan based on the target large model and the task subgraph specifically includes:

[0016] The target operation task is decomposed into sub-problems to obtain the corresponding prompt words for each sub-problem; based on the task sub-graph, the prompt words are input into the target large model to obtain the robot's operation plan.

[0017] A second aspect of this application also provides a robot operation task planning system based on knowledge graphs and large models, wherein the robot operation task planning system based on knowledge graphs and large models includes:

[0018] The knowledge graph construction and model adjustment module is used to construct a knowledge graph and obtain a large model, and adjust the large model to obtain the target large model;

[0019] The subgraph generation module is used to obtain the robot's target operation task and, based on the knowledge graph, generate a task subgraph according to the target operation task.

[0020] The planning and derivation module is used to obtain the robot's operation plan based on the target large model and the task subgraph.

[0021] A third aspect of this application also provides a terminal, wherein the terminal includes: a memory, a processor, and a robot operation task planning program based on knowledge graphs and large models stored in the memory and executable on the processor. When the robot operation task planning program based on knowledge graphs and large models is executed by the processor, it implements the steps of the robot operation task planning method based on knowledge graphs and large models as described above.

[0022] A fourth aspect of this application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a robot operation task planning program based on knowledge graphs and large models, and when the robot operation task planning program based on knowledge graphs and large models is executed by a processor, it implements the steps of the robot operation task planning method based on knowledge graphs and large models as described above.

[0023] Beneficial Effects: This application provides a robot operation task planning method and system based on knowledge graphs and large models. In this method, a knowledge graph containing complete domain knowledge is constructed through task decomposition and knowledge association. Through low-rank matrix decomposition, specialized knowledge of the robot operation domain is injected into the large model. Thus, given a robot operation task, relevant knowledge is retrieved through the knowledge graph to generate a subgraph. The finely tuned large model is then used for reasoning to output the operation plan. This enables the robot to accurately reason and plan for complex and precise operation tasks, improving the robot's execution efficiency in completing task objectives. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of a preferred embodiment of the robot operation task planning method based on knowledge graphs and large models in this application;

[0026] Figure 2 This is a knowledge graph hierarchy diagram of the robot operation task planning method based on knowledge graphs and large models proposed in this application;

[0027] Figure 3 This is a knowledge graph example diagram of the robot operation task planning method based on knowledge graphs and large models proposed in this application;

[0028] Figure 4 This is an entity extraction structure diagram in a preferred embodiment of the robot operation task planning method based on knowledge graphs and large models in this application;

[0029] Figure 5 This is a flowchart of robot operation reasoning in a preferred embodiment of the robot operation task planning method based on knowledge graphs and large models in this application;

[0030] Figure 6 This is a structural diagram of a preferred embodiment of the robot operation task planning system based on knowledge graphs and large models in this application;

[0031] Figure 7 This is a structural diagram of a preferred embodiment of the terminal of this application.

[0032] Explanation of reference numerals in the attached figures:

[0033] 100. Graph construction and model adjustment module; 200. Subgraph generation module; 300. Planning derivation module. Detailed Implementation

[0034] To make the objectives, technical solutions, and effects of this application clearer and more explicit, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. The described embodiments are only possible technical implementations of this application and not all possible implementations. Based on the embodiments in this application, those skilled in the art can obtain other embodiments without creative effort, and these embodiments are also within the protection scope of this application.

[0035] First, let's introduce the terms used in the embodiments of this application:

[0036] The full name of a large model is Large Language Model (LLM), which can also be simply called a large language model. In the fields of machine learning and artificial intelligence, a large model is a model with a large number of parameters that can capture and learn more complex data patterns, thus performing well on various tasks.

[0037] BiLSTM stands for Bidirectional Long Short-Term Memory Network. It's a special type of recurrent neural network (RNN) with wide applications in natural language processing (NLP) and time series analysis. Compared to unidirectional LSTMs, BiLSTM runs two independent LSTMs at each time step: one from the beginning to the end of the sequence (forward LSTM) and the other from the end to the beginning of the sequence (backward LSTM). The outputs of these two LSTMs are combined, allowing for simultaneous consideration of contextual information.

[0038] The specific meaning of an entity refers to an object or concept that is closely related to the robot's operation tasks and can be clearly identified and associated in the knowledge graph. These entities may represent concrete objects in the physical world (such as robots, tools, target objects, etc.), operation actions (such as grasping, moving, placing, etc.), scene environments (such as factory workshops, warehouse shelves, etc.), or abstract concepts (such as task objectives, skill types, etc.). Entities typically exist as nodes in the knowledge graph, connected to other nodes through relationships to form a rich semantic network.

[0039] The following description, with reference to the accompanying drawings, describes a robot operation task planning method and system based on knowledge graphs and large models, according to embodiments of this application. Addressing the issue of low efficiency in robot operation reasoning techniques for handling fine-grained operation tasks mentioned above, this application provides a robot operation task planning method based on knowledge graphs and large models. In this method, a knowledge graph containing complete domain knowledge is constructed through task decomposition and knowledge association. Furthermore, specialized knowledge of the robot operation domain is injected into the large model through low-rank matrix decomposition. Thus, given a robot operation task, relevant knowledge is retrieved from the knowledge graph to generate a subgraph, and the finely tuned large model is used for reasoning to output the operation plan. This enables the robot to accurately reason and plan for complex and fine-grained operation tasks, improving the robot's execution efficiency in completing task objectives. Therefore, this solves the technical problem of low efficiency in robot operation reasoning techniques for handling fine-grained operation tasks in related technologies.

[0040] This application addresses the problems of insufficient reasoning accuracy, low task execution efficiency, and poor adaptability to complex scenarios in existing technologies when facing complex and delicate operational tasks. This application can simultaneously utilize the rich knowledge of knowledge graphs and the powerful reasoning capabilities of large-scale models to achieve precise operational reasoning for robots. First, the large-scale model is fine-tuned based on robot operational knowledge. Specifically, the parameters of the large-scale model are frozen and decomposed into two low-rank matrices. By training these two low-rank matrices, the large-scale model acquires knowledge related to robot operations. Further details can be found in [link to further details]. Figure 1 This paper decomposes robot operation skills into five layers from top to bottom: the first layer is the task layer, containing the robot operation task representation; the second layer is the skill layer, containing the skills available for robot operation; the third layer is the action layer, containing the actions included in the robot skills; the fourth layer is the agent layer, containing the robot and end effector used to perform the task; and the fifth layer is the scene layer, containing the scene information in which the robot performs the task. By decomposing robot operation tasks from the first layer to the fourth layer, complex and detailed robot operation tasks can be broken down, thereby constructing a knowledge graph containing complete knowledge. Given a robot operation task, relevant knowledge is retrieved through the task description, thus forming a subgraph oriented towards the given task. The task description is decomposed into several sub-problems to form several prompt words. These prompt words are input into a large model, which combines the generated subgraphs to perform reasoning, thereby outputting the corresponding robot operation plan. This application can be applied to complex robot task application scenarios, using knowledge graphs to enhance the reasoning accuracy of the large model and improve the execution efficiency of the robot in completing task objectives.

[0041] The technical solutions of this application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0042] The robot operation task planning method based on knowledge graphs and large models described in the preferred embodiments of this application, such as... Figure 1 As shown, the robot operation task planning method based on knowledge graphs and large models includes the following steps:

[0043] In step S101, a knowledge graph is constructed and a large model is obtained, and the large model is adjusted to obtain the target large model.

[0044] In one possible implementation, during the construction of the knowledge graph, the robot's operational task is obtained, and the operational task is decomposed into multiple sub-tasks, wherein each sub-task is associated with a corresponding skill, action, agent, and scene; the decomposed sub-tasks are associated with the domain knowledge corresponding to the robot to obtain the knowledge graph required for task execution. Feedback information from the robot during task execution is obtained; the knowledge graph is updated based on the feedback information, and the updated knowledge graph is used as the final knowledge graph.

[0045] like Figure 2 As shown, to better understand and reason about complex robot operation tasks, this application decomposes robot operation tasks into five layers from top to bottom. The first layer is the task layer, which represents the overall task that the robot needs to complete, such as "assembling a device" or "removing an item from a shelf." The representation of the task layer can be a natural language description or a structured task definition. The second layer is the skill layer, which contains various skills required by the robot to perform the task. Each skill represents the ability required by the robot during task execution, such as "grasping an object," "moving an object," and "rotating an object." The representation of the skill layer can be a predefined skill library or a skill representation acquired through learning. The third layer is the action layer, which contains the specific operational actions corresponding to the skills. Each skill can be further decomposed into a series of actions. For example, the skill "grasping an object" can be decomposed into specific actions such as "approaching the object," "adjusting the grasping posture," and "closing the gripper." The representation of the action layer is usually the low-level action instructions that the robot controller can execute. The fourth layer is the agent layer, which contains the robots and their end effectors used to perform the tasks. Different robots and end effectors have different capabilities and limitations, so the appropriate robot and end effector need to be selected based on the specific requirements of the task at the agent layer. The fifth layer is the scene layer, which contains environmental information about the robot's environment when performing the task. Scene information includes the physical environment in which the robot is located, the attributes of objects in the task (such as the shape, material, and position of objects), and dynamic changes in the environment. The scene layer can be represented through sensor data, environmental modeling, and other methods.

[0046] This application, through this five-layer decomposition, can progressively refine complex robot operation tasks, forming a knowledge graph containing complete domain knowledge.

[0047] Specifically, the process begins with task decomposition. Complex robot operation tasks are broken down into five layers from top to bottom: task layer, skill layer, action layer, agent layer, and scenario layer. Each layer further refines and specifies the task at the previous layer, forming a hierarchical task decomposition system. Next, knowledge association is performed. The decomposed tasks, skills, actions, agents, and scenarios are linked to knowledge in the robot operation domain, constructing a knowledge graph required for task execution. This ensures that nodes in the knowledge graph (such as tasks, skills, and actions) are interconnected through edges (such as relationships), forming a complete knowledge network (structured knowledge representation). Then, knowledge updates are implemented. Feedback from the robot's execution in actual tasks is used to dynamically update the information in the knowledge graph, ensuring that it reflects the latest task execution status and environmental changes, improving the accuracy and adaptability of the knowledge graph. Task decomposition and knowledge association are the foundation for building the knowledge graph; their logical relationship is one of progressive refinement, ensuring the completeness and accuracy of the knowledge graph.

[0048] Furthermore, such as Figure 3 As shown, the nodes in a knowledge graph represent tasks, skills, actions, agents, and scene information, and the edges between nodes represent the relationships between these elements. For example, the edge between a task node and a skill node indicates which skills are needed to complete the task; the edge between a skill node and an action node indicates which specific actions a skill can be decomposed into; the edge between an agent node and an action node indicates how a specific robot or end effector performs these actions; and the edge between a scene node and task, skill, and action nodes indicates the environmental factors that need to be considered during task execution. The construction process of a knowledge graph includes task decomposition, knowledge association, and knowledge updating. Task decomposition refers to breaking down a complex task into multiple sub-tasks and associating each sub-task with corresponding skills, actions, agents, and scene information. Knowledge association refers to associating the decomposed tasks, skills, actions, agents, and scenes with knowledge in the robot's operational domain to construct the knowledge graph required for task execution. Knowledge updating refers to dynamically updating the information in the knowledge graph based on the robot's execution feedback in actual tasks, ensuring that the knowledge graph reflects the latest task execution status and environmental changes.

[0049] In one possible implementation, during the adjustment of the large model, the weight matrix of the large model is decomposed into two low-rank matrices; the operation dataset corresponding to the robot is obtained, and the two low-rank matrices are trained based on the operation dataset to obtain the target large model.

[0050] Specifically, during large-scale model tuning, the first step is low-rank matrix decomposition. The weight matrix of the large model is decomposed into two low-rank matrices, and other parameters of the large model are frozen. This is to preserve the original general knowledge and reasoning ability of the large model while injecting robot-specific knowledge. Next, the low-rank matrices are trained. Without changing the core structure of the large model, these two low-rank matrices are trained to inject robot-specific knowledge into the large model. Training is performed using robot-specific datasets (such as data from tasks like grasping, assembly, and handling). The training process ensures that the large model can learn key skills and knowledge related to robot operation, improving its reasoning accuracy in robot operation tasks. Fine-tuning of the large model aims to enhance its reasoning ability in robot operation tasks, and the reasoning process depends on the fine-tuned model parameters and the generated task subgraph.

[0051] Furthermore, by fine-tuning the large model, it is endowed with knowledge relevant to the robot manipulation domain. To retain the original general knowledge and reasoning ability of the large model, this application employs a low-rank matrix factorization method, decomposing the weight matrix of the large model into two low-rank matrices and freezing other parameters of the large model. By training these two low-rank matrices, robot manipulation domain-specific knowledge can be injected into the large model without changing its core structure, enabling the large model to perform more accurate reasoning in robot manipulation tasks. During training, robot manipulation domain-specific datasets are used, such as data from robot grasping, assembly, and handling tasks, to ensure that the large model learns key skills and knowledge related to robot manipulation.

[0052] In this application, a multi-level knowledge graph structure is used to divide the knowledge graph into five levels: task, skill, action, agent, and scenario, which facilitates knowledge graph retrieval and makes knowledge representation clearer.

[0053] In step S102, the target operation task of the robot is obtained, and a task subgraph is obtained based on the knowledge graph and the target operation task.

[0054] In one possible implementation, during the generation of the task subgraph, the target operation task is parsed to obtain the entity corresponding to the target operation task, wherein the entity includes any one of object, operation action, scene environment, task target and skill type; based on the entity, the knowledge graph is retrieved to obtain the knowledge node and edge relationship corresponding to the target operation task; the knowledge node and the edge relationship are organized to obtain the task subgraph oriented towards the target operation task.

[0055] Specifically, task description parsing and entity extraction involve parsing the task description to extract task-related entities. Entity extraction is the foundation for subsequent steps, ensuring that key information is accurately extracted from the task description. Generating a task-oriented subgraph involves retrieving task-related knowledge nodes and relationships from the knowledge graph based on keywords and structured information in the task description, and generating a task-oriented subgraph. This depends on the construction and updating of the knowledge graph, as well as the parsing and entity extraction of the task description.

[0056] In one possible implementation, during entity extraction, a bidirectional long short-term memory network is used to perform language processing on the target operation task to obtain a language information vector; a graph attention network is used to process the language information vector to obtain feature information corresponding to the target operation task; and a conditional random field is used to predict the feature information to obtain the entity corresponding to the target operation task.

[0057] Specifically, in the process of determining the task subgraph based on the knowledge graph, task description parsing and entity extraction involve performing natural language processing on the task description, using a BiLSTM network structure to capture information from the input text and words; then, a graph attention network is used to capture feature information between words, within text and between words, and between words and text; finally, a conditional random field is used to predict entities in the text, which will serve as keywords for subsequent knowledge graph retrieval. Knowledge graph retrieval, based on the entity extraction results, retrieves knowledge nodes and relationships related to the task description from the knowledge graph. This may involve traversing or querying the knowledge graph to find all nodes related to the task and the edges connecting them. Constructing the task subgraph involves organizing the retrieved knowledge nodes and relationships according to their logical relationships and dependencies in task execution, forming a task-oriented subgraph. This subgraph contains all the relevant knowledge required for task execution, such as skill selection, action execution order, agent configuration, and environmental information. Subgraph validation and optimization involves validating the constructed task subgraph to ensure its completeness and accuracy, and optimizing it as needed to eliminate redundant information or add necessary details.

[0058] Furthermore, after constructing the knowledge graph, given a robot operation task (target operation task), relevant knowledge is retrieved from the knowledge graph based on the task description, generating a subgraph oriented towards that task. For example... Figure 4 and Figure 5Specifically, the task description is first parsed to extract task-related entities. Entity extraction utilizes a BiLSTM network (Bidirectional Long Short-Term Memory) structure to capture information from the input text and words. Three different graph attention networks are then used to capture feature information between words, within and between words in the text, and between words and text, respectively. Finally, a Conditional Random Field (CRF) is used to predict entities in the text. Next, based on keywords and structured information in the task description, task-related knowledge nodes and relationships are retrieved from the knowledge graph to generate a task-specific subgraph. This subgraph contains all the relevant knowledge required for task execution, including skill selection, action execution order, agent configuration, and environmental information. Figure 4 MGAT extracts features from dictionary words, RGAT extracts features within text and between words, and LGAT extracts features between text and words. MGAT is mentioned as a technique or algorithm for extracting features from dictionary words, which can be understood as a technique or algorithm related to word feature extraction. RGAT is mentioned as an advanced graph neural network library, specifically used for researching and applying graph neural networks. LGAT is a technique or algorithm related to feature extraction between text and words.

[0059] It's important to note that entity extraction extracts entities from a given text and then uses these entities to retrieve other related entities and relationships from the knowledge graph, forming a subgraph. A bidirectional long short-term memory network is used for language processing of the task, resulting in vectors containing linguistic information. These vectors are then further processed by a graph attention network, and the final output is the probability that the input text contains entities; those with higher probabilities are considered entities. The input text can be understood as the input corresponding to the target task, and the vocabulary is a dictionary introduced for the task, which helps the network better identify words in the input text.

[0060] This application constructs three novel graph attention structures through entity extraction to analyze task instructions, enabling more accurate extraction of entities related to the knowledge graph from the instructions.

[0061] In step S103, the robot's operation plan is obtained based on the target large model and the task subgraph.

[0062] In one possible implementation, during the process of generating the operation plan, the target operation task is decomposed into sub-problems to obtain the prompt words corresponding to the sub-problems; based on the task sub-graph, the prompt words are input into the target large model to obtain the robot's operation plan.

[0063] Specifically, generating prompts involves breaking down the task description into several sub-problems and generating corresponding prompts for each sub-problem. The purpose of prompt generation is to guide the large model in reasoning, which depends on the parsing of the task description and the decomposition of sub-problems. Large model reasoning involves inputting the generated prompts into the fine-tuned large model, which then performs reasoning based on the generated task subgraph. The large model's reasoning depends on the fine-tuned model parameters, the generated task subgraph, and the prompts. Based on the reasoning results of the large model, the corresponding robot operation plan is output. The output of the operation plan is the final goal, which depends on the reasoning results of the large model, as well as the task description and scene information.

[0064] Further, see Figure 4 The task description corresponding to the target operation task is decomposed into several sub-problems, forming several prompt words. The prompt word generation process includes sub-problem decomposition and prompt word generation. Sub-problem decomposition refers to further breaking down each sub-task in the task description into specific operational problems. For example, the task of "grabbing an object" can be decomposed into sub-problems such as "how to approach the object" and "how to adjust the grasping posture." Prompt word generation refers to generating corresponding prompt words for each sub-problem. The prompt words can be natural language questions or structured instructions, prompting the large model to perform reasoning. The generated prompt words are input into the fine-tuned large model, which combines the generated task subgraph for reasoning. The large model utilizes its built-in reasoning capabilities and structured knowledge in the knowledge graph to deduce the specific steps and operational plan for task execution.

[0065] Based on the inference results of the large model, the corresponding robot operation plan is output. The operation plan includes task execution steps, skill invocation, motion command generation, and adjustments to the plan based on scene information. Task execution steps refer to generating specific execution steps and operation sequences for each subtask. Skill invocation refers to selecting appropriate skills for each step and assigning corresponding parameters to the skills, such as grasping force and movement speed. Motion commands refer to generating low-level motion commands that the robot controller can execute, ensuring the robot can complete the task according to the plan. Scene adaptability refers to adjusting the parameters and execution sequence in the operation plan based on scene-level information, ensuring the robot can adapt to changes in the current environment.

[0066] The robot action reasoning proposed in this application combines knowledge graphs, entity extraction, and fine-tuning of a large language model for task reasoning, thereby improving the accuracy of action reasoning.

[0067] Next, referring to the accompanying drawings, a robot operation task planning system based on knowledge graphs and large models, according to an embodiment of this application, is described.

[0068] Figure 6 This is a structural diagram of a robot operation task planning system based on knowledge graphs and large models, according to an embodiment of this application.

[0069] like Figure 6 As shown, the robot operation task planning system based on knowledge graphs and large models includes: a graph construction and model adjustment module 100, a subgraph generation module 200, and a planning derivation module 300.

[0070] Specifically, the knowledge graph construction and model adjustment module 100 is used to construct a knowledge graph and obtain a large model, and adjust the large model to obtain a target large model;

[0071] The subgraph generation module 200 is used to obtain the robot's target operation task and, based on the knowledge graph, obtain a task subgraph according to the target operation task.

[0072] The planning and derivation module 300 is used to obtain the robot's operation plan based on the target large model and the task subgraph.

[0073] Figure 7 A structural diagram of a terminal provided in an embodiment of this application. The terminal may include:

[0074] The memory 501, the processor 502, and the computer program stored on the memory 501 and capable of running on the processor 502.

[0075] When the processor 502 executes the program, it implements the robot operation task planning method based on knowledge graphs and large models provided in the above embodiments.

[0076] Furthermore, the terminal also includes:

[0077] Communication interface 503 is used for communication between memory 501 and processor 502.

[0078] The memory 501 is used to store computer programs that can run on the processor 502.

[0079] The memory 501 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0080] If the memory 501, processor 502, and communication interface 503 are implemented independently, they can be interconnected via a bus to communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EIS) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus.

[0081] Optionally, in a specific implementation, if the memory 501, processor 502, and communication interface 503 are integrated on a single chip, then the memory 501, processor 502, and communication interface 503 can communicate with each other through an internal interface.

[0082] Processor 502 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.

[0083] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the robot operation task planning method based on knowledge graphs and large models as described above.

[0084] One embodiment of this application provides a computer program product, including a computer program that, when executed by a processor, implements the features described in this application. Figure 1 The corresponding embodiments provide robot operation task planning methods based on knowledge graphs and large models.

[0085] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0086] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0087] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.

[0088] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable storage medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable storage medium could be paper or other suitable media on which the program can be printed, since the program can be obtained electronically by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0089] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0090] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0091] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0092] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.

[0093] It should be understood that the application of this application is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A robot operation task planning method based on knowledge graphs and large models, characterized in that, The robot operation task planning method based on knowledge graphs and large models includes: Construct a knowledge graph and obtain a large model, and adjust the large model to obtain the target large model; Obtain the robot's target operation task, and based on the knowledge graph, obtain a task subgraph according to the target operation task; Based on the target model, the robot's operation plan is obtained according to the task subgraph; The construction of the knowledge graph specifically includes: The robot's operation task is obtained, and the operation task is decomposed into multiple sub-tasks, wherein the multiple sub-tasks are associated with corresponding skills, actions, intelligent agents and scenes; The decomposed subtasks are associated with the domain knowledge corresponding to the robot to obtain the knowledge graph required for task execution; The step of obtaining a task subgraph based on the knowledge graph and the target operation task specifically includes: The target operation task is parsed to obtain the entity corresponding to the target operation task, wherein the entity includes any one of the following: object, operation action, scene environment, task target, and skill type; Based on the entity, the knowledge nodes and edge relationships corresponding to the target operation task are retrieved from the knowledge graph. The knowledge nodes and edge relationships are organized to obtain a task subgraph oriented towards the target operation task; The step of obtaining the robot's operation plan based on the target large model and the task subgraph specifically includes: The target operation task is decomposed into sub-problems to obtain the prompt words corresponding to the sub-problems; Based on the task subgraph, the prompt words are input into the target large model to obtain the robot's operation plan; The operation plan includes task execution steps, skill invocation, action instruction generation, and adjustments to the plan based on scene information. The task execution steps refer to generating specific execution steps and operation sequences for each sub-task. Skill invocation refers to selecting appropriate skills for each step and assigning corresponding parameters to the skills. Action instructions refer to generating low-level action instructions that the robot controller can execute to ensure that the robot can complete the task according to the plan. Adjusting the plan based on scene information refers to adjusting the parameters and execution sequence in the operation plan based on scene-level information to ensure that the robot can adapt to current environmental changes.

2. The robot operation task planning method based on knowledge graphs and large models according to claim 1, characterized in that, The construction of the knowledge graph then includes: Obtain feedback information from the robot during task execution; The knowledge graph is updated based on the feedback information, and the updated knowledge graph is used as the knowledge graph.

3. The robot operation task planning method based on knowledge graphs and large models according to claim 1, characterized in that, The adjustment of the large model to obtain the target large model specifically includes: The weight matrix of the large model is decomposed into two low-rank matrices; Obtain the operation dataset corresponding to the robot, and train the two low-rank matrices based on the operation dataset to obtain the target large model.

4. The robot operation task planning method based on knowledge graphs and large models according to claim 1, characterized in that, The step of parsing the target operation task to obtain the entity corresponding to the target operation task specifically includes: A bidirectional long short-term memory network is used to perform language processing on the target operation task to obtain a language information vector. The language information vector is processed using a graph attention network to obtain the feature information corresponding to the target operation task; The feature information is predicted using a conditional random field to obtain the entity corresponding to the target operation task.

5. A robot operation task planning system based on knowledge graphs and large models, characterized in that, The robot operation task planning system based on knowledge graphs and large models is applied to the robot operation task planning method based on knowledge graphs and large models as described in any one of claims 1-4; the robot operation task planning system based on knowledge graphs and large models includes: The knowledge graph construction and model adjustment module is used to construct a knowledge graph and obtain a large model, and adjust the large model to obtain the target large model; The subgraph generation module is used to obtain the robot's target operation task and, based on the knowledge graph, generate a task subgraph according to the target operation task. The planning and derivation module is used to obtain the robot's operation plan based on the target large model and the task subgraph.

6. A terminal, characterized in that, The terminal includes: a memory, a processor, and a robot operation task planning program based on knowledge graphs and large models stored in the memory and executable on the processor. When the robot operation task planning program based on knowledge graphs and large models is executed by the processor, it implements the steps of the robot operation task planning method based on knowledge graphs and large models as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a robot operation task planning program based on knowledge graphs and large models. When the robot operation task planning program based on knowledge graphs and large models is executed by a processor, it implements the steps of the robot operation task planning method based on knowledge graphs and large models as described in any one of claims 1-4.

Images