Robot operation task planning method and system based on knowledge graph and large model

Abstract

The invention discloses a robot operation task planning method and system based on a knowledge graph and a large model, and the method comprises the steps: constructing the knowledge graph, obtaining the large model, and adjusting the large model to obtain a target large model; obtaining a target operation task of the robot, and obtaining a task sub-graph according to the target operation task based on the knowledge graph; and based on the target large model, obtaining an operation plan of the robot according to the task sub-graph. Accurate reasoning and planning of the robot on complex and fine operation tasks can be achieved, and the execution efficiency of the robot for completing task targets is improved.

Description

Technical Field

[0001] The present application relates to the field of robot operation technology, and in particular to a robot operation task planning method and system based on knowledge graph and large model. Background Art

[0002] Existing robot operation reasoning technologies are mainly divided into reasoning based on distributed representation learning, reasoning based on neural networks, and operation reasoning based on reinforcement learning.

[0003] The central idea of ​​the reasoning method based on representation learning is to find a mapping function to map symbolic representation to vector space for numerical representation, thereby reducing the curse of dimensionality while capturing the implicit associations between entities and relations. The key point is that it can be calculated directly and quickly; however, this method has difficulty in handling complex physical interactions, lacks fine operation capabilities, and has difficulty maintaining consistency and coherence.

[0004] The neural network-based reasoning method trains a large amount of data through a deep learning model (such as convolutional neural network, recurrent neural network or transformer, etc.), automatically extracts features and performs reasoning. The advantage of this method is that it can handle complex nonlinear relationships and has strong generalization ability; however, this method has the disadvantages of difficulty in handling high-dimensional continuous action space, insufficient modeling of the physical world, and poor generalization ability.

[0005] The operational reasoning method based on reinforcement learning learns how to maximize a certain reward signal through a series of actions through the interaction between the intelligent agent and the environment. It is suitable for solving sequential decision-making problems, especially in robot operation tasks, and has strong adaptability and flexibility. However, this method has the disadvantages of low sample efficiency and difficulty in handling long-term dependent tasks. Especially in fine operation tasks, the robustness of reinforcement learning methods is poor.

[0006] Therefore, although reasoning methods based on distributed representation learning, neural networks, and reinforcement learning have made some progress in robotic manipulation reasoning, they still have significant limitations when dealing with fine manipulation tasks.

[0007] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0008] The main purpose of this application is to provide a robot operation task planning method and system based on knowledge graphs and large models, aiming to solve the problem of low efficiency of robot operation reasoning technology in the prior art when processing fine operation tasks.

[0009] A first aspect of an embodiment of the present application provides a robot operation task planning method based on a knowledge graph and a large model, the robot operation task planning method based on a knowledge graph and a large model comprising: constructing a knowledge graph and obtaining a large model, and adjusting the large model to obtain a target large model; obtaining the target operation task of the robot, and based on the knowledge graph, obtaining a task subgraph according to the target operation task; based on the target large model, obtaining the operation plan of the robot according to the task subgraph.

[0010] Optionally, in one embodiment of the present application, the constructing of the knowledge graph specifically includes: obtaining the operation task of the robot, decomposing the operation task into multiple subtasks, wherein the multiple subtasks are associated with corresponding skills, actions, agents and scenarios; associating the decomposed multiple subtasks with the domain knowledge corresponding to the robot to obtain the knowledge graph required for task execution.

[0011] Optionally, in one embodiment of the present application, the constructing of the knowledge graph further includes: obtaining feedback information of the robot in performing a task; updating the knowledge graph according to the feedback information, and using the updated knowledge graph as the knowledge graph.

[0012] Optionally, in one embodiment of the present application, the large model is adjusted to obtain a target large model, specifically comprising: decomposing the weight matrix of the large model into two low-rank matrices; obtaining an operation data set corresponding to the robot, and training the two low-rank matrices according to the operation data set to obtain the target large model.

[0013] Optionally, in one embodiment of the present application, based on the knowledge graph, obtaining a task subgraph according to the target operation task specifically includes: parsing the target operation task to obtain an entity corresponding to the target operation task, wherein the entity includes any one of an object, an operation action, a scene environment, a task goal and a skill type; according to the entity, searching from the knowledge graph to obtain knowledge nodes and edge relationships corresponding to the target operation task; organizing the knowledge nodes and the edge relationships to obtain a task subgraph for the target operation task.

[0014] Optionally, in one embodiment of the present application, parsing the target operation task to obtain an entity corresponding to the target operation task specifically includes: using a bidirectional long short-term memory network to perform language processing on the target operation task to obtain a language information vector; using a graph attention network to process the language information vector to obtain feature information corresponding to the target operation task; and using a conditional random field to predict the feature information to obtain the entity corresponding to the target operation task.

[0015] Optionally, in one embodiment of the present application, obtaining the operation plan of the robot based on the target macro model and according to the task subgraph specifically includes:

[0016] The target operation task is decomposed into sub-problems to obtain prompt words corresponding to the sub-problems; according to the task subgraph, the prompt words are input into the target large model to obtain the operation plan of the robot.

[0017] The second aspect of the embodiment of the present application further provides a robot operation task planning system based on a knowledge graph and a large model, wherein the robot operation task planning system based on a knowledge graph and a large model includes:

[0018] A graph construction and model adjustment module is used to construct a knowledge graph and obtain a large model, and adjust the large model to obtain a target large model;

[0019] A subgraph generation module, used to obtain a target operation task of the robot, and based on the knowledge graph, obtain a task subgraph according to the target operation task;

[0020] A planning derivation module is used to obtain the operation plan of the robot based on the target large model and the task subgraph.

[0021] The third aspect of an embodiment of the present application also provides a terminal, wherein the terminal includes: a memory, a processor, and a robot operation task planning program based on a knowledge graph and a large model stored in the memory and executable on the processor, wherein the robot operation task planning program based on a knowledge graph and a large model implements the steps of the robot operation task planning method based on a knowledge graph and a large model as described above when executed by the processor.

[0022] The fourth aspect of an embodiment of the present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a robot operation task planning program based on a knowledge graph and a large model, and when the robot operation task planning program based on the knowledge graph and the large model is executed by a processor, the steps of the robot operation task planning method based on the knowledge graph and the large model as described above are implemented.

[0023] Beneficial effects: The present application provides a robot operation task planning method and system based on knowledge graph and large model. In this method, a knowledge graph containing complete domain knowledge is constructed through task decomposition and knowledge association, and the specialized knowledge in the robot operation field is injected into the large model through low-rank matrix decomposition. Therefore, when a robot operation task is given, relevant knowledge is retrieved through the knowledge graph, a subgraph is generated, and the fine-tuned large model is used for reasoning to output the operation plan, thereby realizing the robot's precise reasoning and planning of complex and precise operation tasks, and improving the robot's execution efficiency in completing task objectives. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0025] Figure 1 It is a flowchart of a preferred embodiment of the robot operation task planning method based on knowledge graph and large model of the present application;

[0026] Figure 2 This is a knowledge graph hierarchical structure diagram of the robot operation task planning method based on the knowledge graph and the large model of this application;

[0027] Figure 3 It is a knowledge graph example diagram of the robot operation task planning method based on the knowledge graph and the large model of the present application;

[0028] Figure 4 It is an entity extraction structure diagram in a preferred embodiment of the robot operation task planning method based on knowledge graph and large model of the present application;

[0029] Figure 5 It is a robot operation reasoning flow chart in a preferred embodiment of the robot operation task planning method based on knowledge graph and large model of the present application;

[0030] Figure 6 It is a structural diagram of a preferred embodiment of the robot operation task planning system based on knowledge graph and large model of the present application;

[0031] Figure 7 This is a structural diagram of a preferred embodiment of the terminal of this application.

[0032] Description of reference numerals:

[0033] 100. Graph construction and model adjustment module; 200. Subgraph generation module; 300. Planning derivation module. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical scheme and effect of the present application clearer and more specific, the technical scheme in the embodiments of the present application will be described clearly and completely in combination with the drawings in the embodiments of the present application. The described embodiments are only possible technical implementations of the present application, not all possible implementations. Based on the embodiments in the present application, those skilled in the art can completely combine the embodiments of the present application to obtain other embodiments without creative work, and these embodiments are also within the scope of protection of the present application.

[0035] First, the nouns involved in the embodiments of the present application are introduced:

[0036] The full name of the big model is Large Language Model (LLM), which can also be referred to as a large language model. In the field of machine learning and artificial intelligence, a large model refers to a model with a large number of parameters that can capture and learn more complex data patterns, thereby performing well in various tasks.

[0037] The full name of the BiLSTM network structure is Bidirectional Long Short-Term Memory Network. BiLSTM is a special recurrent neural network (RNN) that is widely used in fields such as natural language processing (NLP) and time series analysis. Compared with the unidirectional LSTM, BiLSTM runs two independent LSTMs at each time step, one from the beginning to the end of the sequence (forward LSTM) and the other from the end to the beginning of the sequence (backward LSTM). The outputs of these two LSTMs are combined to consider the context information at the same time.

[0038] The specific meaning of entity refers to objects or concepts that are closely related to robot operation tasks and can be clearly identified and associated in the knowledge graph. These entities may represent specific objects in the physical world (such as robots, tools, target objects, etc.), operation actions (such as grasping, moving, placing, etc.), scene environments (such as factory workshops, warehouse shelves, etc.), or abstract concepts (such as task goals, skill types, etc.). Entities usually exist as nodes in the knowledge graph, connected to other nodes through relationships, forming a rich semantic network.

[0039] The following describes the robot operation task planning method and system based on knowledge graph and large model of the embodiment of the present application with reference to the accompanying drawings. In view of the problem that the robot operation reasoning technology in the above-mentioned related technology is inefficient in processing fine operation tasks, the present application provides a robot operation task planning method based on knowledge graph and large model, in which a knowledge graph containing complete domain knowledge is constructed through task decomposition and knowledge association, and the special knowledge of the robot operation field is injected into the large model through low-rank matrix decomposition, so that when a robot operation task is given, the relevant knowledge is retrieved through the knowledge graph, a subgraph is generated, and the fine-tuned large model is used for reasoning, and the operation plan is output, thereby realizing the robot's accurate reasoning and planning of complex and fine operation tasks, and improving the execution efficiency of the robot in completing the task objectives. As a result, the technical problem of the low efficiency of the robot operation reasoning technology in the related technology when processing fine operation tasks is solved.

[0040] This application aims to solve the problems of insufficient reasoning accuracy, low task execution efficiency and poor adaptability to complex scenarios in the existing technology when facing complex and delicate operation tasks. This application can simultaneously utilize the rich knowledge of the knowledge graph and the powerful reasoning ability of the large model to achieve accurate operation reasoning of the robot. First, fine-tune the large model according to the robot operation knowledge. Specifically, freeze the parameters of the large model and decompose them into two low-rank matrices. By training these two low-rank matrices, the large model can obtain knowledge related to robot operation. For further information, see Figure 1 , the robot operation skills are decomposed into five levels from top to bottom. The first level is the task layer, which contains the robot operation task representation, the second level is the skill layer, which contains the skills available for robot operation, the third level is the action layer, which contains the actions contained in the robot skills, the fourth level is the agent layer, which contains the robot and end effector used to perform the task, and the fifth level is the scene layer, which contains the scene information in which the robot performs the task. By decomposing the robot operation task from the first level to the fourth level, it is possible to decompose the complex and sophisticated robot operation tasks, thereby constructing a knowledge graph containing complete knowledge. Given a robot operation task, relevant knowledge is retrieved through the task description to form a subgraph for a given task, the task description is decomposed into several sub-problems to form several prompt words, the prompt words are input into the large model, and the large model is combined with the generated subgraph for reasoning, thereby outputting the corresponding robot operation plan. The present application can be applied to the application scenario of complex robot tasks, using the knowledge graph to enhance the reasoning accuracy of the large model and improve the execution efficiency of the robot to complete the task objectives.

[0041] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0042] The robot operation task planning method based on knowledge graph and large model described in the preferred embodiment of the present application is as follows: Figure 1 As shown, the robot operation task planning method based on knowledge graph and large model includes the following steps:

[0043] In step S101, a knowledge graph is constructed and a large model is obtained, and the large model is adjusted to obtain a target large model.

[0044] In a possible implementation, in the process of constructing the knowledge graph, the operation task of the robot is obtained, and the operation task is decomposed into multiple subtasks, wherein the multiple subtasks are associated with corresponding skills, actions, agents, and scenarios; the decomposed multiple subtasks are associated with the domain knowledge corresponding to the robot to obtain the knowledge graph required for task execution. Feedback information of the robot in executing the task is obtained; the knowledge graph is updated according to the feedback information, and the updated knowledge graph is used as the knowledge graph.

[0045] like Figure 2 As shown, in order to better understand and reason about complex robot operation tasks, this application decomposes robot operation tasks from top to bottom into five levels. The first level is the task layer, which is used to represent the overall task that the robot needs to complete, such as "assembling a device" or "taking an item from a shelf". The representation of the task layer can be a natural language description or a structured task definition. The second level is the skill layer, which contains various skills required by the robot when performing tasks. Each skill represents the ability required by the robot during the task execution process, such as "grasping objects", "moving objects", "rotating objects", etc. The representation of the skill layer can be a predefined skill library or a skill representation acquired through learning. The third level is the action layer, which contains the specific operation actions corresponding to the skills. Each skill can be further decomposed into a series of actions. For example, the skill of "grasping objects" can be decomposed into specific actions such as "approaching objects", "adjusting grasping posture", and "closing grippers". The representation of the action layer is usually a low-level action instruction that can be executed by the robot controller. The fourth layer is the agent layer, which includes the robot and its end effector used to perform the task. Different robots and end effectors have different capabilities and limitations, so the agent layer needs to select the appropriate robot and end effector according to the specific requirements of the task. The fifth layer is the scene layer, which contains the environmental information of the robot when performing the task. The scene information includes the physical environment where the robot is located, the object attributes in the task (such as the shape, material, position, etc. of the object), and the dynamic changes in the environment. The representation of the scene layer can be obtained through sensor data, environmental modeling, etc.

[0046] Through the decomposition of this five-layer structure, the present application can gradually refine the complex robot operation tasks and form a knowledge graph that contains complete domain knowledge.

[0047] Specifically, first of all, the task is decomposed, and the complex robot operation task is decomposed from top to bottom into five levels (task level, skill level, action level, intelligent agent level, and scene level). Each level is a further refinement and concretization of the previous level task, forming a hierarchical task decomposition system. Then, the recognition association is performed, and the decomposed tasks, skills, actions, intelligent agents, and scenes are associated with the knowledge of the robot operation field to build the knowledge graph required for task execution, ensuring that the nodes in the knowledge graph (such as tasks, skills, actions, etc.) are interconnected through edges (such as relationships) to form a complete knowledge network (structured knowledge representation). Then, the knowledge is updated. Through the execution feedback of the robot in the actual task, the information in the knowledge graph is dynamically updated to ensure that the knowledge graph can reflect the latest task execution and environmental changes, and improve the accuracy and adaptability of the knowledge graph. Task decomposition and knowledge association are the basis for building the knowledge graph. The logical relationship between them is to be refined layer by layer to ensure the integrity and accuracy of the knowledge graph.

[0048] Further, if Figure 3 As shown in the figure, the nodes of the knowledge graph represent tasks, skills, actions, agents and scene information, and the edges between the nodes represent the relationship between these elements. For example, the edge between the task node and the skill node represents which skills are needed to complete the task, the edge between the skill node and the action node represents which specific actions the skill can be decomposed into, the edge between the agent node and the action node represents how a specific robot or end effector performs these actions, and the edge between the scene node and the task, skill and action nodes represents the environmental factors that need to be considered during the task execution. The construction process of the knowledge graph includes task decomposition, knowledge association and knowledge update. Task decomposition refers to decomposing a complex task into multiple subtasks and associating each subtask with the corresponding skills, actions, agents and scene information. Knowledge association refers to associating the decomposed tasks, skills, actions, agents and scenes with the knowledge of the robot operation field to build the knowledge graph required for task execution. Knowledge update refers to dynamically updating the information in the knowledge graph through the execution feedback of the robot in the actual task, ensuring that the knowledge graph can reflect the latest task execution status and environmental changes.

[0049] In one possible implementation, during the process of adjusting the large model, the weight matrix of the large model is decomposed into two low-rank matrices; an operation data set corresponding to the robot is obtained, and the two low-rank matrices are trained according to the operation data set to obtain the target large model.

[0050] Specifically, when adjusting the large model, first perform low-rank matrix decomposition to decompose the weight matrix of the large model into two low-rank matrices, and freeze the other parameters of the large model. This is to retain the original general knowledge and reasoning ability of the large model, while injecting specialized knowledge in the field of robot operation into it; train the low-rank matrix, without changing the core structure of the large model, and inject specialized knowledge in the field of robot operation into the large model by training these two low-rank matrices. Use specialized data sets in the field of robot operation (such as data for tasks such as grasping, assembly, and handling) for training. The training process ensures that the large model can learn key skills and knowledge related to robot operation and improve its reasoning accuracy in robot operation tasks. Fine-tuning the large model is to enhance its reasoning ability in robot operation tasks, and the reasoning process depends on the fine-tuned model parameters and the generated task subgraphs.

[0051] Furthermore, by fine-tuning the large model, it is given relevant knowledge in the field of robot operation. In order to retain the original general knowledge and reasoning ability of the large model, this application adopts a low-rank matrix decomposition method to decompose the weight matrix of the large model into two low-rank matrices and freeze the other parameters of the large model. By training these two low-rank matrices, the specialized knowledge in the field of robot operation can be injected into the large model without changing the core structure of the large model, so that the large model can perform more accurate reasoning in robot operation tasks. During the training process, special data sets in the field of robot operation are used, such as data on robot grasping, assembly, and handling tasks, to ensure that the large model can learn key skills and knowledge related to robot operation.

[0052] In this application, through a multi-level knowledge graph structure, the knowledge graph is divided into five levels: tasks, skills, actions, agents, and scenarios, which facilitates the retrieval of the knowledge graph and more clear knowledge expression.

[0053] In step S102, the target operation task of the robot is obtained, and based on the knowledge graph, a task subgraph is obtained according to the target operation task.

[0054] In one possible implementation, in the process of generating a task subgraph, the target operation task is parsed to obtain an entity corresponding to the target operation task, wherein the entity includes any one of an object, an operation action, a scene environment, a task goal, and a skill type; based on the entity, a search is performed from the knowledge graph to obtain knowledge nodes and edge relationships corresponding to the target operation task; the knowledge nodes and the edge relationships are organized to obtain a task subgraph for the target operation task.

[0055] Specifically, task description parsing and entity extraction parse the task description and extract the entities related to the task. Entity extraction is the basis for subsequent steps, which ensures that key information is accurately extracted from the task description. Task-oriented subgraphs are generated by retrieving task-related knowledge nodes and relationships from the knowledge graph based on the keywords and structured information in the task description, and generating a subgraph for the task. This depends on the construction and updating of the knowledge graph, as well as the parsing and entity extraction of the task description.

[0056] In a possible implementation, during the entity extraction process, a bidirectional long short-term memory network is used to perform language processing on the target operation task to obtain a language information vector; a graph attention network is used to process the language information vector to obtain feature information corresponding to the target operation task; and a conditional random field is used to predict the feature information to obtain an entity corresponding to the target operation task.

[0057] Specifically, in the process of determining the task subgraph based on the knowledge graph, task description parsing and entity extraction are natural language processing of the task description, using the BiLSTM network structure to capture the information of the input text and vocabulary; then, the graph attention network is used to capture the feature information between words, within the text and within the vocabulary, and between vocabulary and text; finally, the conditional random field is used to predict the entities in the text, which will be used as keywords for subsequent retrieval of the knowledge graph. Knowledge graph retrieval, based on the results of entity extraction, retrieves knowledge nodes and relationships related to the task description from the knowledge graph, which may involve traversal or query operations on the knowledge graph to find all nodes related to the task and the edges connecting them. Constructing the task subgraph, organizing the retrieved knowledge nodes and relationships according to their logical relationships and dependencies in task execution, forming a subgraph for the task, which contains all relevant knowledge required for task execution, such as skill selection, action execution order, agent configuration, and scene environment information. Subgraph verification and optimization, verifying the constructed task subgraph to ensure its completeness and accuracy, and optimizing the subgraph as needed to eliminate redundant information or add necessary details.

[0058] Furthermore, after constructing the knowledge graph, given a robot operation task (target operation task), relevant knowledge is retrieved from the knowledge graph through the task description to generate a subgraph for the task. Figure 4 and Figure 5As shown in the figure, specifically, the task description is first parsed to extract entities related to the task. Entity extraction uses the BiLSTM network structure (bidirectional long short-term memory network) to capture the information of the input text and vocabulary, and three different graph attention networks are used to capture the feature information between words, within the text and within the vocabulary, and between vocabulary and text. Finally, the entities in the text are predicted through the conditional random field. Then, based on the keywords and structured information in the task description, the knowledge nodes and relationships related to the task are retrieved from the knowledge graph to generate a subgraph for the task. This subgraph contains all the relevant knowledge required for task execution, including the selection of skills, the execution order of actions, the configuration of the agent, and the environmental information of the scene. Figure 4 MGAT extracts features of dictionary vocabulary, RGAT extracts features between text and vocabulary, and LGAT extracts features between text and vocabulary. Among them, MGAT is mentioned as a technology used to extract features of dictionary vocabulary, which can be understood as a technology or algorithm related to vocabulary feature extraction, RGAT is mentioned as an advanced graph neural network library specifically used for the research and application of graph neural networks, and LGAT is a technology or algorithm related to feature extraction between text and vocabulary.

[0059] It should be noted that the role of entity extraction is to extract entities from a given paragraph, and then use these entities to retrieve other entities and relationships related to these entities from the knowledge graph to form a subgraph. The bidirectional long short-term memory network is used to process the language of the operation task, and the vector containing language information is obtained. These vectors are then further processed by the graph attention network, and the final output is the probability of the input language containing the entity. The higher the probability, the entity is considered. The input text can be understood as the input corresponding to the target operation task, and the vocabulary is a dictionary introduced corresponding to the operation task. This dictionary can help the network better recognize the vocabulary in the input text.

[0060] This application constructs three novel graph attention structures through entity extraction to realize the analysis of task instructions, which can more accurately extract entities related to the knowledge graph in the instructions.

[0061] In step S103, based on the target macro model, the operation plan of the robot is obtained according to the task subgraph.

[0062] In a possible implementation, in the process of generating an operation plan, the target operation task is decomposed into sub-problems to obtain prompt words corresponding to the sub-problems; according to the task subgraph, the prompt words are input into the target large model to obtain the operation plan of the robot.

[0063] Specifically, generating prompt words is to decompose the task description into several sub-problems and generate corresponding prompt words for each sub-problem. The generation of prompt words is to guide the large model to perform reasoning, which depends on the parsing of the task description and the decomposition of the sub-problems; large model reasoning is to input the generated prompt words into the fine-tuned large model, and the large model performs reasoning in combination with the generated task subgraph. The reasoning of the large model depends on the fine-tuned model parameters, the generated task subgraph and the prompt words; based on the reasoning results of the large model, the corresponding robot operation plan is output. The output of the operation plan is the ultimate goal, which depends on the reasoning results of the large model as well as the task description and scene information.

[0064] For further information, see Figure 4 , decompose the task description corresponding to the target operation task into several sub-problems to form several prompt words. The prompt word generation process includes sub-problem decomposition and prompt word generation. Sub-problem decomposition refers to further decomposing each sub-task in the task description into specific operation problems. For example, the "grasp the object" task can be decomposed into sub-problems such as "how to approach the object" and "how to adjust the grasping posture". Prompt word generation refers to generating corresponding prompt words for each sub-problem. Prompt words can be natural language questions or structured instructions to prompt the large model to reason. The generated prompt words are input into the fine-tuned large model, and the large model performs reasoning in combination with the generated task subgraph. The large model uses its built-in reasoning ability and structured knowledge in the knowledge graph to deduce the specific steps and operation plans for task execution.

[0065] Based on the reasoning results of the large model, the corresponding robot operation plan is output. The operation plan includes the steps of task execution, the calling of skills, the generation of action instructions, and the adjustment of the plan according to the scene information. The task execution steps refer to the generation of specific execution steps and operation sequences for each subtask. Skill calling refers to the selection of appropriate skills for each step and the assignment of corresponding parameters to the skills, such as the strength of grasping, the speed of movement, etc. Action instructions refer to the generation of low-level action instructions that can be executed by the robot controller to ensure that the robot can complete the task according to the plan. Scene adaptability refers to adjusting the parameters and execution sequence in the operation plan according to the information of the scene layer to ensure that the robot can adapt to the current environmental changes.

[0066] The robot action reasoning of this application combines knowledge graphs, entity extraction, and fine-tuning of large language models to perform task reasoning, thereby improving the accuracy of action reasoning.

[0067] Secondly, the robot operation task planning system based on knowledge graph and large model proposed in accordance with the embodiment of the present application is described with reference to the accompanying drawings.

[0068] Figure 6 It is a structural diagram of a robot operation task planning system based on a knowledge graph and a large model in an embodiment of the present application.

[0069] like Figure 6 As shown, the robot operation task planning system based on knowledge graph and large model includes: a graph construction and model adjustment module 100, a sub-graph generation module 200 and a planning derivation module 300.

[0070] Specifically, the graph construction and model adjustment module 100 is used to construct a knowledge graph and obtain a large model, and adjust the large model to obtain a target large model;

[0071] A subgraph generation module 200 is used to obtain a target operation task of the robot and obtain a task subgraph according to the target operation task based on the knowledge graph;

[0072] The planning derivation module 300 is used to obtain the operation plan of the robot based on the target large model and the task subgraph.

[0073] Figure 7 This is a structural diagram of a terminal provided in an embodiment of the present application. The terminal may include:

[0074] A memory 501 , a processor 502 , and a computer program stored in the memory 501 and executable on the processor 502 .

[0075] When the processor 502 executes the program, the robot operation task planning method based on the knowledge graph and the large model provided in the above embodiment is implemented.

[0076] Furthermore, the terminal further includes:

[0077] The communication interface 503 is used for communication between the memory 501 and the processor 502 .

[0078] The memory 501 is used to store computer programs that can be executed on the processor 502 .

[0079] The memory 501 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0080] If the memory 501, the processor 502 and the communication interface 503 are implemented independently, the communication interface 503, the memory 501 and the processor 502 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EIS) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0081] Optionally, in a specific implementation, if the memory 501, the processor 502 and the communication interface 503 are integrated on a chip, the memory 501, the processor 502 and the communication interface 503 can communicate with each other through an internal interface.

[0082] The processor 502 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0083] This embodiment also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned robot operation task planning method based on knowledge graph and large model.

[0084] One embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the present application Figure 1 A robot operation task planning method based on knowledge graph and large model is provided in any embodiment of the corresponding embodiment.

[0085] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0086] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0087] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present application belong.

[0088] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable storage medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, a "computer-readable storage medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable storage media include the following: an electrical connection with one or N wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable storage medium may even be paper or other suitable medium on which the program is printed, since the program can be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways as necessary, and then storing it in a computer memory.

[0089] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiment, the N steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0090] A person skilled in the art may understand that all or part of the steps in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one or a combination of the steps of the method embodiment.

[0091] In addition, each functional unit in each embodiment of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0092] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present application. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present application.

[0093] It should be understood that the application of the present application is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to this application.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A robot operation task planning method based on knowledge graph and large model, characterized in that: The robot operation task planning method based on knowledge graph and large model includes: Constructing a knowledge graph and obtaining a large model, and adjusting the large model to obtain a target large model; Obtain a target operation task of the robot, and based on the knowledge graph, obtain a task subgraph according to the target operation task; Based on the target macro model, the operation plan of the robot is obtained according to the task subgraph.

2. The robot operation task planning method based on knowledge graph and large model according to claim 1 is characterized in that: The construction of the knowledge graph specifically includes: Acquire an operation task of the robot, and decompose the operation task into a plurality of subtasks, wherein the plurality of subtasks are associated with corresponding skills, actions, agents, and scenarios; The decomposed multiple subtasks are associated with the domain knowledge corresponding to the robot to obtain the knowledge graph required for task execution.

3. The robot operation task planning method based on knowledge graph and large model according to claim 2 is characterized in that: The construction of the knowledge graph further includes: Obtaining feedback information of the robot during task execution; The knowledge graph is updated according to the feedback information, and the updated knowledge graph is used as the knowledge graph.

4. The robot operation task planning method based on knowledge graph and large model according to claim 1 is characterized in that: The adjusting the large model to obtain the target large model specifically includes: Decomposing the weight matrix of the large model into two low-rank matrices; An operation data set corresponding to the robot is obtained, and the two low-rank matrices are trained according to the operation data set to obtain a target large model.

5. The robot operation task planning method based on knowledge graph and large model according to claim 1 is characterized in that: The step of obtaining a task subgraph based on the knowledge graph and according to the target operation task specifically includes: Parsing the target operation task to obtain an entity corresponding to the target operation task, wherein the entity includes any one of an object, an operation action, a scene environment, a task goal, and a skill type; According to the entity, searching from the knowledge graph to obtain knowledge nodes and edge relationships corresponding to the target operation task; The knowledge nodes and the edge relationships are organized to obtain a task subgraph oriented to the target operation task.

6. The robot operation task planning method based on knowledge graph and large model according to claim 5 is characterized in that: The parsing of the target operation task to obtain an entity corresponding to the target operation task specifically includes: Using a bidirectional long short-term memory network to perform language processing on the target operation task to obtain a language information vector; Using a graph attention network to process the language information vector to obtain feature information corresponding to the target operation task; The feature information is predicted using a conditional random field to obtain an entity corresponding to the target operation task.

7. The robot operation task planning method based on knowledge graph and large model according to claim 1 is characterized in that: The step of obtaining the operation plan of the robot based on the target macro model and according to the task subgraph specifically includes: Decomposing the target operation task into sub-problems to obtain prompt words corresponding to the sub-problems; According to the task subgraph, the prompt word is input into the target large model to obtain the operation plan of the robot.

8. A robot operation task planning system based on knowledge graph and large model, characterized in that: The robot operation task planning system based on knowledge graph and large model includes: A graph construction and model adjustment module is used to construct a knowledge graph and obtain a large model, and adjust the large model to obtain a target large model; A subgraph generation module, used to obtain a target operation task of the robot, and based on the knowledge graph, obtain a task subgraph according to the target operation task; A planning derivation module is used to obtain the operation plan of the robot based on the target large model and the task subgraph.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a robot operation task planning program based on a knowledge graph and a large model stored in the memory and executable on the processor. When the robot operation task planning program based on the knowledge graph and the large model is executed by the processor, the steps of the robot operation task planning method based on the knowledge graph and the large model are implemented as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a robot operation task planning program based on a knowledge graph and a large model. When the robot operation task planning program based on a knowledge graph and a large model is executed by a processor, the steps of the robot operation task planning method based on a knowledge graph and a large model as described in any one of claims 1 to 7 are implemented.

Images