A method and apparatus for autonomous decision-making and execution of robot tasks

Abstract

This invention discloses a method for autonomous decision-making and execution of robot tasks, comprising the following steps: classifying and constructing robot-related knowledge, including static knowledge and dynamic knowledge; expressing different categories of knowledge in different forms based on static and dynamic knowledge to construct a multi-level multimodal knowledge graph; storing the multimodal knowledge graph using a differentiated knowledge storage method to make different categories of knowledge interconnected; and enabling the robot to autonomously make decisions and execute tasks based on the interconnected multimodal knowledge graph. This method of classifying and storing various types of knowledge facilitates autonomous decision-making by the robot based on these types of knowledge.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of knowledge base and robotics, and relates to a method and apparatus for autonomous decision-making and execution of robot tasks. Background Technology

[0002] Knowledge graphs are a cutting-edge technology in computer science, with market applications primarily focused on financial fraud prevention, question answering, and search. Their application in robotics is limited, although knowledge-based task planning and execution are currently at the forefront of this field. Using knowledge graphs to store basic everyday experiences and common sense, record historical task execution sequences, and atomic action knowledge, can simulate the human brain's memory of common sense, actions, and task execution. Simultaneously, knowledge reasoning techniques can be used to infer and understand unknown and unfamiliar environments from existing knowledge, enabling autonomous learning and task planning. This represents the future direction of robot intelligence.

[0003] With the development of intelligent technology, robots are playing a valuable role in an increasing number of scenarios, such as industrial, service, and home sectors. Most existing robot task execution methods involve a fixed task execution sequence or are controlled and guided by humans. For example, patent document CN214560938A discloses a robot with a behavioral decision control system, and patent document CN110497404A discloses a robot bionic intelligent decision-making system. These robot task execution methods rely heavily on knowledge application and lack adaptability and flexibility. For instance, in a tour guide scenario, task execution is simply a sequential arrangement, lacking flexibility in the content and scope of questions and answers. Summary of the Invention

[0004] To address the aforementioned technical problems in the existing technology, this invention proposes a method and apparatus for autonomous decision-making and execution of robot tasks. By classifying and storing various types of knowledge, it facilitates autonomous decision-making by the robot based on these types of knowledge.

[0005] To achieve the above-mentioned objectives, an embodiment provides a method for autonomous decision-making and execution of robot tasks, comprising the following steps:

[0006] Robot-related knowledge is categorized and constructed, including static knowledge and dynamic knowledge;

[0007] Based on static and dynamic knowledge, different categories of knowledge are expressed in different forms to construct a multi-level, multimodal knowledge graph;

[0008] Multimodal knowledge graphs are stored using a differentiated knowledge storage approach, enabling interrelation between different categories of knowledge.

[0009] Robots can make autonomous decisions and execute tasks based on interconnected multimodal knowledge graphs.

[0010] Preferably, the static knowledge describes the knowledge required for the robot's work, and the content of the static knowledge does not change over time except for knowledge update operations, and exists in a static form;

[0011] The static knowledge includes object knowledge, algorithm knowledge, static scene knowledge, hardware knowledge, and action knowledge;

[0012] The knowledge about objects includes common knowledge about objects, physical phenomena of objects, and rules for using objects;

[0013] The algorithm knowledge is used for sharing among robots to achieve hardware adaptation. The algorithm knowledge includes algorithm functions, algorithms, and algorithm files.

[0014] The static scene knowledge describes static spatial information, which includes rooms, fixed objects, and object positions.

[0015] The hardware knowledge enables action executability and constraint action behavior reasoning, including the robot body and components, and multi-source sensors;

[0016] The action knowledge describes action information, which includes action function and action attribute information.

[0017] Preferably, the dynamic knowledge introduces the concepts of time and operation, describing the process of state changes of the environment, objects, and robot during task execution, and is dynamically changing;

[0018] The dynamic knowledge includes dynamic environment knowledge, task knowledge, and interaction knowledge;

[0019] The dynamic environment knowledge describes the real-time perception information that changes dynamically over time in the scene. The real-time perception information includes information on changes in the position of objects in the scene and information on changes in the state of the robot.

[0020] The task knowledge description includes the task name, initial state, task description information, and subtasks with the task ending state. It also describes the execution process of the subtasks, which is a continuously changing process. The corresponding execution process information includes the start time, the actions and objects involved, the algorithms involved, the state change process of the objects, and the end time.

[0021] The interactive knowledge recorder logs the robot's operations during subtask execution, forming contextual memory for action learning, experience-assisted operation, task learning, and data mining. The logs include information on trajectory, actions, changes in object state, and images.

[0022] Preferably, based on static knowledge, different categories of knowledge are expressed in different forms to construct a multi-level, multimodal knowledge graph, including:

[0023] Based on object knowledge, algorithm knowledge, hardware knowledge, and action knowledge, construct object multimodal knowledge graphs, algorithm multimodal knowledge graphs, hardware multimodal knowledge graphs, and action multimodal knowledge graphs;

[0024] The object multimodal knowledge graph is used to describe object knowledge. It uses common sense about objects and physical phenomena of objects as entity nodes. Common sense about objects includes object structure, object function, and object attributes. The relationships between entity nodes include category relationship, position relationship, and object usage rules. Entity nodes in the object multimodal knowledge graph are represented by text symbols. The content of entity nodes is associated with text symbols through mapping relationships. This content is presented in a multimodal manner.

[0025] The algorithm multimodal knowledge graph is used to describe algorithm knowledge. The algorithm function, the algorithm that performs the algorithm function, the input data and the result data of the algorithm are the entity nodes. The relationships between entity nodes include functional classification relationships, input and output relationships and association relationships with other categories of knowledge. The entity nodes in the algorithm multimodal knowledge graph are represented by text symbols. The text symbols are associated with multimodal algorithm files, specific input data and output data through mapping relationships.

[0026] The hardware multimodal knowledge graph is used to describe hardware knowledge. The robot body and components, multi-source sensors and hardware functions are used as entity nodes. The relationships between entity nodes include hardware classification relationships, ownership relationships and association relationships with other categories of knowledge. Entity nodes in the hardware multimodal knowledge graph are represented by text symbols, and the text symbols are associated with multimodal data collected through hardware functions through mapping relationships.

[0027] The action multimodal knowledge graph is used to describe action knowledge. It uses meta-actions, action functions, action attribute information, and action operation objects as entity nodes. The relationships between entity nodes include action classification relationships, ownership relationships, and association relationships with other categories of knowledge. Entity nodes in the action multimodal knowledge graph are represented by text symbols, and the text symbols are associated with specific action attribute information of multimodality through mapping relationships.

[0028] Preferably, the step of expressing different categories of knowledge in different forms based on static and dynamic knowledge to construct a multi-level, multimodal knowledge graph includes:

[0029] A scene multimodal knowledge graph is constructed based on static and dynamic scene knowledge. The scene multimodal knowledge graph is used to describe static and dynamic scene knowledge. Rooms, fixed objects, object positions, and robot states are used as entity nodes. The relationships between entity nodes include classification relationships, belonging relationships, and association relationships with other categories of knowledge. Entity nodes in the scene multimodal knowledge graph are represented by text symbols. The text symbols are associated with multimodal object position change information and robot state information through mapping relationships.

[0030] Preferably, based on dynamic knowledge, different categories of knowledge are expressed in different forms to construct a multi-layered, multimodal knowledge graph, including:

[0031] A task multimodal knowledge graph is constructed based on task knowledge. The task multimodal knowledge graph is used to describe subtasks and their execution processes. Subtasks are used as entity nodes, and the relationships between entity nodes include the turning relationships between subtasks. Turning probabilities and turning times are configured for the turning relationships. The execution process of a subtask is deconstructed into a subgraph. In the subgraph, the start time, participating actions and objects, participating algorithms, object state change processes, and end time are used as entity nodes. The relationships in the subgraph are association relationships.

[0032] Preferably, the mapping relationship includes path mapping, and the multimodal representation includes point cloud, image, video, audio, estimation, and model file formats.

[0033] Preferably, the method of storing the multimodal knowledge graph using differentiated knowledge storage, enabling interrelation between different categories of knowledge, includes:

[0034] The robot ontology rule description is stored separately, the entities and relationships in the multimodal knowledge graph are stored as text, the multimodal data and information mapped to entity nodes are stored separately, and the mapping relationship between multimodal data and information and entity nodes is constructed. Multiple multimodal knowledge graphs are associated by the association relationship between entity nodes.

[0035] Preferably, the robot achieves autonomous decision-making and execution of tasks based on an associated multimodal knowledge graph, including:

[0036] The robot starts with a task, confirms and executes the decisions of each step by matching them in the associated multimodal knowledge graph, and completes the autonomous decision-making and execution of the task.

[0037] To achieve the above-mentioned objectives, the embodiments also provide a device for autonomous decision-making and execution of robot tasks, including a knowledge construction module, a knowledge graph construction module, a storage and association module, and a decision execution module;

[0038] The knowledge construction module is used to classify and construct robot-related knowledge, including static knowledge and dynamic knowledge;

[0039] The knowledge graph construction module is used to express different categories of knowledge in different forms based on static and dynamic knowledge, and to construct a multi-level, multimodal knowledge graph.

[0040] The storage association module is used to store multimodal knowledge graphs using a differentiated knowledge storage method, so that different categories of knowledge are interconnected.

[0041] The decision execution module is used by the robot to achieve autonomous decision-making and execution of tasks based on the associated multimodal knowledge graph.

[0042] Compared with existing technologies, the beneficial effects of this invention include at least the following: Firstly, it provides a more comprehensive classification of knowledge, dividing it into two main categories: dynamic and static. Static knowledge describes long-term, unchanging factual knowledge, while dynamic knowledge introduces the concept of time, describing the state changes during robot operation. The specific knowledge categories include the seven types of knowledge required for robot operation. Secondly, robot scene information has multimodal characteristics. To better express this multimodal information, a multimodal knowledge representation and storage method is adopted. Multimodal information includes images, model files, trajectories, text symbols, speech features, and point cloud data. Different storage strategies are designed for these different modalities of knowledge, resulting in clearer storage, more convenient matching and retrieval, and improved versatility of autonomous decision-making and execution. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram of a method for autonomous decision-making and execution of robot tasks according to the present invention;

[0045] Figure 2 This is a schematic diagram of an object multimodal knowledge graph according to an embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram of the multimodal knowledge graph algorithm according to an embodiment of the present invention;

[0047] Figure 4 This is a schematic diagram of a hardware multimodal knowledge graph according to an embodiment of the present invention;

[0048] Figure 5This is a schematic diagram of an action multimodal knowledge graph according to an embodiment of the present invention;

[0049] Figure 6 This is a schematic diagram of a scene-based multimodal knowledge graph according to an embodiment of the present invention;

[0050] Figure 7 This is a schematic diagram of a task-based multimodal knowledge graph according to an embodiment of the present invention;

[0051] Figure 8 This is a schematic diagram of the knowledge base storage framework according to an embodiment of the present invention;

[0052] Figure 9 This is a schematic diagram of the structure of the robot task autonomous decision-making and execution device provided in the embodiment. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the scope of protection of this invention.

[0054] Before using knowledge graphs to assist robot decision-making, technologies such as knowledge representation, knowledge extraction, and knowledge fusion are needed to construct the knowledge graph. This invention describes the knowledge representation module, which defines knowledge categories, relationships between knowledge, and attributes, and uses a hierarchical knowledge storage method to express knowledge in different modalities. This invention uses ontology engineering technology to summarize information such as objects, spaces, tasks, and actions involved in home scenarios and guided tour scenarios, designs a knowledge ontology, and establishes constraints between different knowledge, laying the foundation for subsequent knowledge extraction and application.

[0055] This embodiment provides a method for autonomous decision-making and execution of robot tasks. First, knowledge is categorized and constructed; second, different categories of knowledge are expressed in different forms; and finally, different knowledge base storage methods are designed for the different expression forms of different categories of knowledge, thereby assisting the robot in achieving autonomous task decision-making. Figure 1 As shown, the specific steps include:

[0056] S110 categorizes and constructs robot-related knowledge, including static knowledge and dynamic knowledge.

[0057] In this embodiment, in order to enable autonomous decision-making for robot tasks in unstructured scenarios, knowledge is divided into static knowledge and dynamic knowledge.

[0058] In this embodiment, static knowledge describes the knowledge required for the robot's work. The content of static knowledge does not change over time except for knowledge update operations and exists in a static form. Static knowledge includes object knowledge, algorithm knowledge, static scene knowledge, hardware knowledge, and action knowledge.

[0059] Knowledge of objects includes general knowledge about objects, physical phenomena related to objects, and rules for using objects. General knowledge includes an object's structure, function, and properties. For example, describing the structure of a cup includes its characteristics, categories, and functions; a cup is made of glass. Physical phenomena refer to the physical phenomena that an object can produce. For example, a lamp emits light, and emitting light is a physical phenomenon. Rules for using an object refer to how it is used to achieve a certain function. For example, a self-heating cup's rule of use is to activate its self-heating function by clicking a switch.

[0060] Algorithmic knowledge is used for sharing among robots to achieve hardware adaptation. Algorithmic knowledge includes algorithm functions, algorithms, and algorithm files. Algorithm functions refer to the capabilities of an algorithm, such as visual perception, semantic segmentation, association, and invocation. An algorithm is the algorithm that performs these functions, including interface functions, deep learning models, and traditional algorithms. Algorithm files are configuration files for algorithms. For interface functions and traditional algorithms, the configuration file contains function parameters, while for deep learning models, the configuration file contains the model structure and model parameters.

[0061] Static scene knowledge describes static spatial information, which includes rooms, fixed objects, and object positions. This static spatial information exists statically in the knowledge graph. Rooms include bedrooms, kitchens, exhibition halls, etc. Fixed objects refer to objects fixed to the scene, such as cabinets, kitchenware, household appliances, exhibits, etc. Object positions refer to the location of objects, which can be on the floor, walls, or other objects in the room.

[0062] Hardware knowledge enables the execution of actions and the reasoning of constrained action behaviors, including the robot body and its components, and multi-source sensors. Components refer to parts belonging to the robot, such as the arm and wheels, while multi-source sensors are used to acquire sensory data, including image, sound, and video acquisition, and distance measurement.

[0063] Motion knowledge describes motion information, which includes motion function and motion attribute information. Motion function includes moving, turning, speaking, etc., while motion attribute information refers to attribute information related to the motion, including movement distance, movement direction, etc.

[0064] In this embodiment, dynamic knowledge introduces the concepts of time and operation to describe the dynamic changes in the state of the environment, objects, and robot during task execution. Dynamic knowledge includes dynamic environment knowledge, task knowledge, and interaction knowledge.

[0065] Dynamic environment knowledge describes real-time perceived information that changes dynamically over time in a scene. This real-time perceived information includes information on changes in the position of objects in the scene and changes in the robot's state.

[0066] The task knowledge description includes the task name, task description information, and subtasks of the task's target state. It also describes the execution process of the subtasks, which is a continuously changing process. The corresponding execution process information includes the start time, the actions and objects involved, the algorithms involved, the state changes of the objects, and the end time. For example, the execution process information for the subtask of opening a door includes: door (closed) - recognition and opening action -> door (open).

[0067] Interactive knowledge records the robot's operation logs during subtask execution, forming contextual memories for action learning, experience-assisted operation, task learning, and data mining. The logs include information such as trajectory, actions, object state changes, and images.

[0068] S120, based on static and dynamic knowledge, expresses different categories of knowledge in different forms to construct a multi-level, multimodal knowledge graph.

[0069] In this embodiment, based on object knowledge, algorithm knowledge, hardware knowledge, action knowledge, static scene knowledge, dynamic scene knowledge, and task knowledge, multimodal knowledge graphs of objects, algorithms, hardware, actions, scenes, and tasks are constructed respectively. Among them, objects, algorithms, hardware, actions, static scenes, dynamic scenes, and tasks are multi-layered to form a multi-layered multimodal knowledge graph.

[0070] For object knowledge, a multimodal knowledge graph is constructed to describe object knowledge. Object common sense and physical phenomena are used as entity nodes. Object common sense includes object structure, object function, and object attributes. Relationships between entity nodes include category affiliation, positional relationship, and usage rules. Entity nodes in the multimodal knowledge graph are represented by text symbols, and the content of entity nodes is associated with these text symbols through mapping relationships. This content is presented in a multimodal manner. The mapping relationship is a path relationship, mapping the actual content corresponding to the entity node through paths. This actual content can be point cloud data or pixel semantic interpretations of images, etc.

[0071] Figure 2 An example of an object multimodal knowledge graph is given, in Figure 2 In this context, objects, containers, cups, mugs, handles, desks, etc., belong to the object structure; containing solutions belongs to the object function; ceramic as a material belongs to the object attribute; subclasses belong to the relationships described in the category, such as ownership, position, and combination, all of which belong to positional relationships; and prohibited operations and operation methods belong to the object's usage rules.

[0072] For algorithms, a multimodal knowledge graph is constructed to describe algorithmic knowledge. Entity nodes are defined as algorithm functions, the algorithms that perform those functions, the input data, and the output data. Relationships between entity nodes include functional classification relationships, input and output relationships, and associations with other categories of knowledge. These associations include connections to hardware knowledge, action knowledge, and task knowledge, such as the invocation relationship with action knowledge. Entity nodes in the multimodal knowledge graph are represented by text symbols, which are mapped to multimodal algorithm files, specific input data, and output data through mapping relationships. These mapping relationships are path mappings, meaning that specific algorithm files are associated with algorithms through path mappings.

[0073] Figure 3 An example of an algorithmic multimodal knowledge graph is given. Figure 3 In this context, algorithms, visual perception, and semantic segmentation belong to the algorithm function, Segnet belongs to the algorithm, image 1 is the input data, image 2 is the output data, and subclasses and functions belong to the functional classification relationship.

[0074] For hardware knowledge, a constructed hardware multimodal knowledge graph is used to describe hardware knowledge. The robot body and its components, multi-source sensors, and hardware functions are represented as entity nodes. Relationships between entity nodes include hardware classification relationships, ownership relationships, and associations with other categories of knowledge. Here, associations refer to the relationships between hardware knowledge and algorithmic knowledge, task knowledge, environmental knowledge, etc., such as the invocation relationship between hardware knowledge and algorithmic knowledge. Entity nodes in the hardware multimodal knowledge graph are represented by text symbols, which are mapped to multimodal data collected through hardware functions. This mapping is a path mapping, and the multimodal data includes point cloud data, image data, video data, and audio data.

[0075] Figure 4 An exemplary hardware multimodal knowledge graph is provided, in Figure 4 In this context, perception and movement are hardware functions, and subclasses belong to hardware classification relationships. Skills and ownership belong to ownership relationships. For example, a robot (robot1) has movement capabilities, and an RGBD camera has perception capabilities.

[0076] For action knowledge, a multimodal action knowledge graph is constructed to describe action knowledge. Entity nodes are defined as meta-actions, action functions, action attribute information, and action operation objects. Relationships between entity nodes include action classification relationships, ownership relationships, and association relationships with other categories of knowledge. Here, association relationships refer to relationships related to task knowledge, object knowledge, hardware knowledge, etc. Entity nodes in the multimodal action knowledge graph are represented by text symbols, and these text symbols are associated with specific action attribute information of the multimodality through mapping relationships. These mapping relationships are path relationships, and the specific action attribute information of the multimodality is stored in the form of trajectories, point clouds, images, audio, and video.

[0077] Figure 5 An example is provided of an action multimodal knowledge graph, in Figure 5 In this context, movement is both an action function and a meta-action; coordinates, angle, and speed are action attribute information; subclasses belong to action classification relationships; and input and output belong to ownership relationships, etc.

[0078] A multimodal scene knowledge graph is constructed to describe both dynamic and static scene knowledge. Rooms, fixed objects, object positions, and robot states are used as entity nodes. Relationships between entity nodes include classification relationships, membership relationships, and associations with other categories of knowledge. Here, associations include relationships related to hardware knowledge, action knowledge, etc. Entity nodes in the multimodal scene knowledge graph are represented by text symbols, and these text symbols are linked to multimodal object position change information and robot state information through mapping relationships. Path mapping is used for these mapping relationships, and the multimodal object position change information and robot state information are presented in formats such as images, point clouds, videos, and audio.

[0079] Figure 6 An exemplary multimodal knowledge graph for a scenario is provided, in Figure 6 In this context, subclasses belong to a classification relationship, and possess a belonging relationship, etc.

[0080] For task knowledge, a multimodal knowledge graph is constructed to describe subtasks and their execution processes. Subtasks are used as entity nodes, and the relationships between entity nodes include the turning relationships between subtasks. Turning probabilities and turning times are configured for the turning relationships. The execution process of a subtask is deconstructed into a subgraph. In the subgraph, the start time, participating actions and objects, participating algorithms, object state change processes, and end time are used as entity nodes, and the relationships in the subgraph are association relationships.

[0081] Figure 7 An example of a task-based multimodal knowledge graph is given. Figure 7In this system, finding books, finding bookshelves, and opening bookshelves are all sub-tasks. Each sub-task has an initial state and an ending state. For example, when finding a book, the initial state is empty and the ending state is a book. A sub-graph is also constructed for the execution process of the sub-task of finding books (i.e., the book-finding process), which includes robots involved in finding books, movement actions, and perception data.

[0082] In practical applications, the task multimodal knowledge graph will be supplemented. For a task, the task state changes and operations are compared with those of the existing task multimodal knowledge graph. If any difference exists, it will be stored in the task multimodal knowledge graph.

[0083] S130 uses a differentiated knowledge storage method to store multimodal knowledge graphs, enabling interrelation between different categories of knowledge.

[0084] In this embodiment, when storing the multimodal knowledge graph using a differentiated knowledge storage approach, the robot ontology rule description is stored separately, including using Jena software to store the robot ontology. Entities and relationships in the multimodal knowledge graph are stored as text, including using Neo4j software to store entities and relationships in text form, to improve retrieval efficiency. Considering the real-time and multimodal nature of the knowledge graph in the robot and scene, the multimodal data and information mapped to entity nodes are stored separately, including: using MinIO to store static data in modes such as images, videos, and point clouds, and using Redis to store dynamic data in the scene, such as... Figure 8 As shown.

[0085] The embodiment also constructs a mapping relationship between multimodal data and information and entity nodes, and associates multiple multimodal knowledge graphs by means of the association relationship between entity nodes.

[0086] In this embodiment, since the knowledge storage methods for different levels and modalities are different, it is still necessary to use modal knowledge alignment technology based on distributed representation learning to effectively connect multi-level knowledge, so as to form an associated multimodal knowledge graph, thereby assisting the robot in making autonomous decisions for tasks.

[0087] S140, the robot achieves autonomous decision-making and execution of tasks based on an associated multimodal knowledge graph.

[0088] In this embodiment, when the robot autonomously decides and executes tasks based on the associated multimodal knowledge graph, it starts with the task and confirms and executes each decision by matching within the associated multimodal knowledge graph, thus completing the autonomous decision-making and execution of the task. For example, in a home scenario, the robot receives the task "Get me a magazine." First, other modules (intent understanding) understand the semantics of the text and clarify the task objective (such as a magazine name). Then, it searches the task knowledge to find the task node "Get an object," which includes sub-task nodes such as "Find an object," "Get an object," and "Transport an object." Taking the "Find an object" sub-task as an example, completing this sub-task requires different algorithms and hardware, such as object recognition algorithms, navigation algorithms, positioning algorithms, cameras, and motion chassis. During the execution of this sub-task, information such as objects and state changes in the scene is recorded simultaneously, generating an interaction log for subsequent autonomous learning.

[0089] Based on the same inventive concept, the embodiment also provides a device 900 for autonomous decision-making and execution of robot tasks, such as... Figure 9 As shown, it includes a knowledge construction module 910, a knowledge graph construction module 920, a storage and association module 930, and a decision execution module 940;

[0090] The knowledge construction module 910 is used to classify and construct robot-related knowledge, including static knowledge and dynamic knowledge; the knowledge graph construction module 920 is used to express different categories of knowledge in different forms based on static and dynamic knowledge, and construct a multi-level multimodal knowledge graph; the storage and association module 930 is used to store the multimodal knowledge graph in a differentiated way, so that different categories of knowledge are related to each other; and the decision execution module 940 is used by the robot to realize autonomous decision-making and execution of tasks based on the associated multimodal knowledge graph.

[0091] The specific embodiments described above illustrate the technical solution and beneficial effects of the present invention in detail. It should be understood that the above description is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for autonomous decision-making and execution of robot tasks, characterized in that, Includes the following steps: Robot-related knowledge is categorized and constructed, including static knowledge and dynamic knowledge. Static knowledge describes the knowledge required for robot operation. The content of static knowledge does not change over time except for knowledge update operations and exists in a static form. Static knowledge includes object knowledge, algorithm knowledge, static scene knowledge, hardware knowledge, and action knowledge. Dynamic knowledge introduces the concepts of time and operation to describe the process of state changes of the environment, objects, and robots during task execution. It is dynamic and includes dynamic environment knowledge, task knowledge, and interaction knowledge. Based on static knowledge, different categories of knowledge are expressed in different forms to construct a multi-layered, multimodal knowledge graph, including: Based on object knowledge, algorithm knowledge, hardware knowledge, and action knowledge, construct object multimodal knowledge graphs, algorithm multimodal knowledge graphs, hardware multimodal knowledge graphs, and action multimodal knowledge graphs; The object multimodal knowledge graph is used to describe object knowledge. It uses common sense about objects and physical phenomena of objects as entity nodes. Common sense about objects includes object structure, object function, and object attributes. The relationships between entity nodes include category relationship, position relationship, and object usage rules. Entity nodes in the object multimodal knowledge graph are represented by text symbols. The content of entity nodes is associated with text symbols through mapping relationships. This content is presented in a multimodal manner. The algorithm multimodal knowledge graph is used to describe algorithm knowledge. The algorithm function, the algorithm that performs the algorithm function, the input data and the result data of the algorithm are the entity nodes. The relationships between entity nodes include functional classification relationships, input and output relationships and association relationships with other categories of knowledge. The entity nodes in the algorithm multimodal knowledge graph are represented by text symbols. The text symbols are associated with multimodal algorithm files, specific input data and output data through mapping relationships. The hardware multimodal knowledge graph is used to describe hardware knowledge. The robot body and components, multi-source sensors and hardware functions are used as entity nodes. The relationships between entity nodes include hardware classification relationships, ownership relationships and association relationships with other categories of knowledge. Entity nodes in the hardware multimodal knowledge graph are represented by text symbols, and the text symbols are associated with multimodal data collected through hardware functions through mapping relationships. The action multimodal knowledge graph is used to describe action knowledge. Meta-actions, action functions, action attribute information, and action operation objects are used as entity nodes. The relationships between entity nodes include action classification relationships, ownership relationships, and association relationships with other categories of knowledge. Entity nodes in the action multimodal knowledge graph are represented by text symbols, and the text symbols are associated with specific action attribute information of multimodality through mapping relationships. Based on static and dynamic knowledge, different categories of knowledge are expressed in different forms to construct a multi-layered, multimodal knowledge graph, including: A scene multimodal knowledge graph is constructed based on static and dynamic scene knowledge. The scene multimodal knowledge graph is used to describe static and dynamic scene knowledge. Rooms, fixed objects, object positions, and robot states are used as entity nodes. The relationships between entity nodes include classification relationships, belonging relationships, and association relationships with other categories of knowledge. Entity nodes in the scene multimodal knowledge graph are represented by text symbols. The text symbols are associated with multimodal object position change information and robot state information through mapping relationships. Multimodal knowledge graphs are stored using a differentiated knowledge storage approach, enabling interrelation between different categories of knowledge. Robots can make autonomous decisions and execute tasks based on interconnected multimodal knowledge graphs.

2. The method for autonomous decision-making and execution of robot tasks according to claim 1, characterized in that, The knowledge about objects includes common knowledge about objects, physical phenomena of objects, and rules for using objects; The algorithm knowledge is used for sharing among robots to achieve hardware adaptation. The algorithm knowledge includes algorithm functions, algorithms, and algorithm files. The static scene knowledge describes static spatial information, which includes rooms, fixed objects, and object positions. The hardware knowledge enables action executability and constraint action behavior reasoning, including the robot body and components, and multi-source sensors; The action knowledge describes action information, which includes action function and action attribute information.

3. The method for autonomous decision-making and execution of robot tasks according to claim 1, characterized in that, The dynamic environment knowledge describes the real-time perception information that changes dynamically over time in the scene. The real-time perception information includes information on changes in the position of objects in the scene and information on changes in the robot's state. The task knowledge description includes the task name, initial state, task description information, and subtasks with the task ending state. It also describes the execution process of the subtasks, which is a continuously changing process. The corresponding execution process information includes the start time, the actions and objects involved, the algorithms involved, the state change process of the objects, and the end time. The interactive knowledge recorder logs the robot's operations during subtask execution, forming contextual memory for action learning, experience-assisted operation, task learning, and data mining. The logs include information on trajectory, actions, changes in object state, and images.

4. The method for autonomous decision-making and execution of robot tasks according to claim 3, characterized in that, Based on dynamic knowledge, different categories of knowledge are expressed in different forms to construct a multi-layered, multimodal knowledge graph, including: A task multimodal knowledge graph is constructed based on task knowledge. The task multimodal knowledge graph is used to describe subtasks and their execution processes. Subtasks are used as entity nodes, and the relationships between entity nodes include the turning relationships between subtasks. Turning probabilities and turning times are configured for the turning relationships. The execution process of a subtask is deconstructed into a subgraph. In the subgraph, the start time, participating actions and objects, participating algorithms, object state change processes, and end time are used as entity nodes. The relationships in the subgraph are association relationships.

5. The method for autonomous decision-making and execution of robot tasks according to claim 1, characterized in that, The mapping relationship includes path mapping, and the multimodal forms include point cloud form, image form, video form, audio form, estimation form, and model file.

6. The method for autonomous decision-making and execution of robot tasks according to claim 1, wherein the method of storing multimodal knowledge graphs using differentiated knowledge storage to enable interrelation between different categories of knowledge includes: The robot ontology rule description is stored separately, the entities and relationships in the multimodal knowledge graph are stored as text, the multimodal data and information mapped to entity nodes are stored separately, and the mapping relationship between multimodal data and information and entity nodes is constructed. Multiple multimodal knowledge graphs are associated by the association relationship between entity nodes.

7. The method for autonomous decision-making and execution of a robot task according to claim 1, wherein the robot achieves autonomous decision-making and execution of the task based on an associated multimodal knowledge graph, comprising: The robot starts with a task, confirms and executes the decisions of each step by matching them in the associated multimodal knowledge graph, and completes the autonomous decision-making and execution of the task.

8. A device for autonomous decision-making and execution of robot tasks, characterized in that, It includes a knowledge construction module, a knowledge graph construction module, a storage and association module, and a decision execution module; The knowledge construction module is used to classify and construct robot-related knowledge, including static knowledge and dynamic knowledge. Static knowledge describes the knowledge required for robot operation. The content of static knowledge does not change with time except for knowledge update operations and exists in a static form. Static knowledge includes object knowledge, algorithm knowledge, static scene knowledge, hardware knowledge, and action knowledge. Dynamic knowledge introduces the concepts of time and operation to describe the process of state changes of the environment, objects, and robots during task execution. It is dynamic and includes dynamic environment knowledge, task knowledge, and interaction knowledge. The knowledge graph construction module is used to express different categories of knowledge in different forms based on static knowledge, and construct a multi-level, multimodal knowledge graph, including: Based on object knowledge, algorithm knowledge, hardware knowledge, and action knowledge, construct object multimodal knowledge graphs, algorithm multimodal knowledge graphs, hardware multimodal knowledge graphs, and action multimodal knowledge graphs; The object multimodal knowledge graph is used to describe object knowledge. It uses common sense about objects and physical phenomena of objects as entity nodes. Common sense about objects includes object structure, object function, and object attributes. The relationships between entity nodes include category relationship, position relationship, and object usage rules. Entity nodes in the object multimodal knowledge graph are represented by text symbols. The content of entity nodes is associated with text symbols through mapping relationships. This content is presented in a multimodal manner. The algorithm multimodal knowledge graph is used to describe algorithm knowledge. The algorithm function, the algorithm that performs the algorithm function, the input data and the result data of the algorithm are the entity nodes. The relationships between entity nodes include functional classification relationships, input and output relationships and association relationships with other categories of knowledge. The entity nodes in the algorithm multimodal knowledge graph are represented by text symbols. The text symbols are associated with multimodal algorithm files, specific input data and output data through mapping relationships. The hardware multimodal knowledge graph is used to describe hardware knowledge. The robot body and components, multi-source sensors and hardware functions are used as entity nodes. The relationships between entity nodes include hardware classification relationships, ownership relationships and association relationships with other categories of knowledge. Entity nodes in the hardware multimodal knowledge graph are represented by text symbols, and the text symbols are associated with multimodal data collected through hardware functions through mapping relationships. The action multimodal knowledge graph is used to describe action knowledge. Meta-actions, action functions, action attribute information, and action operation objects are used as entity nodes. The relationships between entity nodes include action classification relationships, ownership relationships, and association relationships with other categories of knowledge. Entity nodes in the action multimodal knowledge graph are represented by text symbols, and the text symbols are associated with specific action attribute information of multimodality through mapping relationships. It is also used to express different categories of knowledge in different forms based on static and dynamic knowledge, constructing multi-level, multimodal knowledge graphs, including: A scene multimodal knowledge graph is constructed based on static and dynamic scene knowledge. The scene multimodal knowledge graph is used to describe static and dynamic scene knowledge. Rooms, fixed objects, object positions, and robot states are used as entity nodes. The relationships between entity nodes include classification relationships, belonging relationships, and association relationships with other categories of knowledge. Entity nodes in the scene multimodal knowledge graph are represented by text symbols. The text symbols are associated with multimodal object position change information and robot state information through mapping relationships. The storage association module is used to store multimodal knowledge graphs using a differentiated knowledge storage method, so that different categories of knowledge are interconnected. The decision execution module is used by the robot to achieve autonomous decision-making and execution of tasks based on the associated multimodal knowledge graph.

Images