Task planning method, device and equipment for robot and medium

Abstract

The invention relates to the technical field of robots, and discloses a task planning method and device for a robot, equipment and a medium, and the method comprises the steps: building a scene relation graph model containing environment space topology and object attributes; inputting the received natural language instruction into the first language model, and outputting a structured task target; on the basis of the structured task target and the real-time environment perception data, auxiliary information is generated through a second language model, and the scene relation graph model is updated; inputting the updated scene relation graph model into a reinforcement learning model, and outputting an action decision sequence; and the target robot is controlled to execute the action decision sequence, and reinforcement learning strategies of the scene relation graph model and the reinforcement learning model are adjusted according to the environment feedback information. According to the invention, efficient, flexible and accurate task planning of the robot in a complex dynamic scene can be realized, and the efficiency and adaptability of task planning of the robot are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, in particular to a task planning method, device and equipment for robots and a medium. BACKGROUND

[0002] With the rapid development of intelligent robot technology, robots are increasingly widely used in complex dynamic scenarios such as industrial logistics, home service, disaster rescue, and the like, and autonomous task planning capability is crucial for the effective work of robots in these scenarios.

[0003] Traditional task planning methods mainly rely on pre-defined rule systems or classical planning algorithms, but the traditional methods have the following problems when facing open dynamic environments: on the one hand, it is difficult to model dynamic environments, and it is difficult to accurately and timely capture dynamic changes in environmental information, such as the appearance of moving obstacles, changes in task targets, and the like. On the other hand, the generalization ability of traditional methods is insufficient, and when facing scenarios or tasks that are not pre-set, they often cannot make reasonable planning. In addition, the search space explosion problem also seriously restricts the application of traditional task planning methods in large-scale complex scenarios, resulting in low planning efficiency and difficulty in meeting actual needs. It can be seen that the existing task planning technology has the defects of difficulty in environment modeling and insufficient generalization ability when facing open dynamic environments.

[0004] The foregoing narrative is to provide general background information and does not necessarily constitute the prior art. SUMMARY

[0005] The embodiments of the present application provide a task planning method, device, equipment and medium for robots, which can realize efficient, flexible and accurate task planning of robots in complex dynamic scenarios, improve the efficiency and adaptability of robot task planning, and solve the problems of difficulty in environment modeling and insufficient generalization ability of existing task planning technology when facing open dynamic environments.

[0006] In a first aspect, the embodiments of the present application provide a task planning method for robots, comprising:

[0007] establishing a scene relation graph model containing environmental space topology and object attributes;

[0008] inputting the received natural language instruction into a first language model to output a structured task target;

[0009] generating auxiliary information based on the structured task target and real-time environmental perception data through a second language model, and updating the scene relation graph model;

[0010] inputting the updated scene relation graph model into a reinforcement learning model to output a sequence of action decisions;

[0011] The target robot is controlled to perform the action decision sequence, and the scene graph model and the reinforcement learning strategy of the reinforcement learning model are adjusted according to environment feedback information.

[0012] Optionally, in some embodiments of the present application, the scene graph model containing environment space topology and object attributes is established, including:

[0013] Node features in the current environment are extracted, including node types, physical attributes, and inter-node topology connection relationships;

[0014] The node features are converted into vectorized space representations through embedding coding technology to generate mathematical representations of the scene graph model;

[0015] Based on real-time state data of the target robot and the mathematical representations of the scene graph model, a dynamically expandable scene graph model data structure is constructed.

[0016] Optionally, in some embodiments of the present application, the received natural language instruction is input into the first language model to output a structured task target, including:

[0017] The task subject, spatial relationship predicate, and constraint condition in the natural language instruction are parsed through the first language model to obtain parsed elements;

[0018] The parsed elements are converted into a target state logical expression in planning domain definition language;

[0019] The structured task target is output based on the target state logical expression, including a target object identifier, a target container identifier, and a reachability constraint set.

[0020] Optionally, in some embodiments of the present application, the scene graph model is updated based on the structured task target and real-time environment perception data through the second language model, including:

[0021] Unexplored region nodes are marked in the scene graph model;

[0022] The relevance score of each unexplored region to the structured task target is calculated through the second language model combined with a common sense knowledge base;

[0023] A region exploration priority sequence is generated in descending order of relevance score as input basis for path planning.

[0024] Optionally, in some embodiments of the present application, the scene graph model is updated based on the structured task target and real-time environment perception data through the second language model, further including:

[0025] receiving a semantic scene description text of an external input;

[0026] identifying, by a second language model, an environmental state change event in the semantic scene description text, the environmental state change event including an added obstacle, a device outage, or a path closure;

[0027] mapping the environmental state change event to an attribute update operation of a corresponding node in the scene relation graph model.

[0028] Optionally, in some embodiments of the present application, the inputting the updated scene relation graph model into a reinforcement learning model and outputting a sequence of action decisions comprises:

[0029] converting the updated scene relation graph model into a state vector required by the reinforcement learning model;

[0030] calculating a value function of each action by a deep Q network, and outputting an action decision with the highest value;

[0031] adopting a post-experience replay mechanism to reconstruct a state that fails to achieve a target into an alternative target and recalculate a reward value, and generating a final sequence of action decisions after optimizing the sequence of decision.

[0032] Optionally, in some embodiments of the present application, the controlling a target robot to execute the sequence of action decisions and adjusting a reinforcement learning strategy of the scene relation graph model and the reinforcement learning model according to environmental feedback information comprises:

[0033] triggering a new area discovery event when the sequence of action decisions is executed, extracting spatial topology features and object attributes of the new area, and expanding a node set of the scene relation graph model;

[0034] storing state transition data fed back by the environment into an experience replay pool when a task is not completed, and driving a policy network parameter update of the reinforcement learning model.

[0035] In a second aspect, embodiments of the present application provide a task planning device for a robot, comprising:

[0036] a model establishing module configured to establish a scene relation graph model containing environmental spatial topology and object attributes;

[0037] a first processing module configured to input a received natural language instruction into a first language model and output a structured task target;

[0038] a second processing module configured to generate auxiliary information based on the structured task target and real-time environmental perception data by a second language model, and update the scene relation graph model;

[0039] a third processing module, configured to input the updated scene graph model into a reinforcement learning model, and output an action decision sequence;

[0040] a control adjustment module, configured to control the target robot to execute the action decision sequence, and adjust the scene graph model and a reinforcement learning strategy of the reinforcement learning model according to environment feedback information.

[0041] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the task planning method for a robot according to the first aspect when executing the computer program.

[0042] In a fourth aspect, an embodiment of the present application provides a storage medium storing a computer program capable of being loaded and executed by a processor to implement the task planning method for a robot according to the first aspect.

[0043] The present application provides a task planning method, device, equipment and medium for a robot. First, a scene graph model containing environmental space topology and object attributes is established. Then, a received natural language instruction is input into a first language model to output a structured task target. Next, auxiliary information is generated through a second language model based on the structured task target and real-time environmental perception data, and the scene graph model is updated. The updated scene graph model is input into a reinforcement learning model to output an action decision sequence. Finally, a target robot is controlled to execute the action decision sequence, and the scene graph model and the reinforcement learning strategy of the reinforcement learning model are adjusted according to environment feedback information. In the task planning scheme for a robot provided in the embodiments of the present application, a structured task target is generated by analyzing a natural language instruction through a large language model to provide clear task guidance for the robot, improving the flexibility and accuracy of task planning. Through the reinforcement learning model, the robot can make real-time decisions in a dynamic environment, adapt to environmental changes, and avoid the shortcomings of traditional methods in dynamic response and long-term planning. The second language model is used to generate auxiliary information and update the scene graph model, which can provide more rich environmental information for the reinforcement learning model, enhancing the accuracy and adaptability of decision-making. The robot executes the action decision sequence and adjusts the model according to the environment feedback, realizing continuous optimization and learning of the robot in actual operation, and further improving the efficiency and success rate of task planning. It can be seen that the present application can realize efficient, flexible and accurate task planning of the robot in a complex dynamic scene, improve the efficiency and adaptability of the robot task planning, and solve the problems of difficulty in environment modeling and insufficient generalization ability of existing task planning technology in the face of open dynamic environment. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the description of the embodiments will be briefly introduced. Obviously, the drawings in the following description only represent some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0045] Figure 1 is an application environment diagram of a task planning method for a robot provided by an embodiment of the present application;

[0046] Figure 2 is a flowchart of a task planning method for a robot provided by an embodiment of the present application;

[0047] Figure 3 is a flowchart of a specific embodiment of a task planning method for a robot provided by an embodiment of the present application;

[0048] Figure 4 is a schematic diagram combining a large language model and a reinforcement learning model provided by an embodiment of the present application;

[0049] Figure 5 is an example diagram of unknown area exploration suggestions provided by an embodiment of the present application;

[0050] Figure 6 is a flowchart of a dynamic scene task planning algorithm of a large model assisted reinforcement learning model provided by an embodiment of the present application;

[0051] Figure 7 is a structural diagram of a task planning device for a robot provided by an embodiment of the present application;

[0052] Figure 8 is a structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0053] The exemplary embodiments will be described in detail herein with reference to the drawings. Unless otherwise specified, the same numbers in different drawings indicate the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all the embodiments consistent with the present application. Rather, they are merely examples of systems and methods consistent with some aspects of the present application, as detailed in the appended claims, or as otherwise described in this application.

[0054] It should be noted that, in this document, the terms "comprise", "contain" or any other variant thereof are intended to encompass non-exclusive inclusion, such that processes, methods, articles, or devices that "comprise", "contain" or "comprise" a list of elements do not only include those elements, but also include other elements that are not expressly listed, or further include elements inherent in such processes, methods, articles, or devices. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of additional identical elements in the process, method, article, or device that includes the element, in addition, components, features, elements with the same name in different embodiments of the present application can have the same meaning or different meanings, and the specific meaning thereof should be determined in conjunction with the explanation thereof in the specific embodiment or further in conjunction with the context in the specific embodiment.

[0055] It should be understood that the specific embodiments described herein are merely intended to explain the present application and are not intended to limit the present application.

[0056] In the following description, the suffixes used for elements such as "module", "part", or "unit" are used only to facilitate explanation of the present application, and have no specific meaning by themselves. Therefore, "module", "part", or "unit" can be used interchangeably.

[0057] With the rapid development of intelligent robot technology, autonomous task planning for complex dynamic scenarios has become an important research direction in the field of robots. In application scenarios such as industrial logistics, home service, disaster rescue, etc., robots not only need to process dynamic environmental information (such as moving obstacles, variable task targets, etc.), but also need to have the ability to plan long-term task sequences. Traditional task planning methods are mainly based on pre-defined rule systems or classical planning algorithms, and face serious challenges such as difficulty in environment modeling, insufficient generalization ability, and explosion of search space in open dynamic environments.

[0058] In order to solve the above technical problems and overcome the defects of the prior art, the embodiments of the present application provide a task planning method, device, equipment and medium for a robot, which can realize efficient, flexible and accurate task planning of the robot in a complex dynamic scenario, and improve the efficiency and adaptability of the robot task planning.

[0059] Figure 1 An application environment diagram for the task planning method for a robot in an embodiment. Referring to Figure 1The task planning method for robots is applied to a task planning system for robots. The task planning system for robots includes a terminal 110 and a server 120. The terminal 110 and the server 120 are connected through a network. The terminal 110 can be a desktop terminal or a mobile terminal. The mobile terminal can be at least one of a mobile phone, a tablet computer, a notebook computer, and the like. The server 120 can be implemented by an independent server or a server cluster composed of multiple servers. The server 120 can be used to establish a scene graph model containing environmental space topology and object attributes; input a received natural language instruction into a first language model to output a structured task target; generate auxiliary information through a second language model based on the structured task target and real-time environmental perception data, and update the scene graph model; input the updated scene graph model into a reinforcement learning model to output a sequence of action decisions; control a target robot to execute the sequence of action decisions, and adjust a reinforcement learning strategy of the scene graph model and the reinforcement learning model according to environmental feedback information.

[0060] Please refer to Figure 2 , Figure 2 is a flowchart of a task planning method for robots provided by an embodiment of the present application. The embodiment mainly takes the task planning method for robots as an example applied to a computer device. The task planning method for robots provided by an embodiment of the present application can specifically include the following steps:

[0061] S1. Establish a scene graph model containing environmental space topology and object attributes;

[0062] Specifically, for step S1, the scene graph model is a basic module of robot task planning. Specifically, this step first needs to extract key information in the current environment, including node features (such as the types and attributes of objects such as rooms, articles, and containers) and the spatial topology connection relationships between them. Through embedding coding technology, these node features are converted into vectorized spatial representation to generate a mathematical representation of the scene graph model. Based on the real-time state data of the target robot and the mathematical representation of the scene graph model, a dynamically expandable scene graph model data structure is constructed, which can be updated in real time as the environment changes.

[0063] S2. Input a received natural language instruction into a first language model to output a structured task target;

[0064] Specifically, for step S2, the user's natural language instruction is input into a trained first language model. The model converts the natural language instruction into a structured task goal by parsing the task subject, spatial relation predicates, and constraints in the instruction. Specifically, the first language model extracts key information from the instruction and converts it into a goal state logical expression in Planning Domain Definition Language (PDDL), thereby generating a structured task goal containing target object identifiers, target container identifiers, and a set of reachability constraints.

[0065] S3. Based on the structured task goal and real-time environment perception data, generate auxiliary information through a second language model, and update the scene graph model;

[0066] Specifically, for step S3, the second language model generates auxiliary information based on the structured task goal and real-time environment perception data to update the scene graph model. Specifically, the second language model uses its understanding of world background knowledge to generate task-related auxiliary information, such as exploration priority sequences for unexplored areas or semantic supplementary instructions for node attributes in the scene graph. By integrating these auxiliary information into the scene graph model, the model can more accurately reflect the current environment state, thereby providing more comprehensive information support for subsequent decision-making.

[0067] S4. Input the updated scene graph model into the reinforcement learning model, and output the action decision sequence;

[0068] Specifically, for step S4, the updated scene graph model is converted into a state vector required by the reinforcement learning model. The reinforcement learning model (such as Deep Q Network, DQN) outputs the action decision sequence with the highest value by calculating the value function of each action. To improve the efficiency and adaptability of the reinforcement learning model, the Hindsight Experience Replay (HER) mechanism is adopted. HER increases the number of effective feedback by reconstructing the state that did not achieve the goal as an alternative goal and recalculating the reward value, thereby optimizing the decision sequence.

[0069] S5. Control the target robot to execute the action decision sequence, and adjust the scene graph model and the reinforcement learning strategy of the reinforcement learning model according to the environmental feedback information;

[0070] Specifically, for step S5, the robot executes the generated action decision sequence and adjusts the scene graph model and the reinforcement learning model according to the environmental feedback information. During execution, if a new region discovery event is triggered, the robot extracts the spatial topology features and object attributes of the new region, and expands the node set of the scene graph model. At the same time, when the task is not completed, the robot stores the state transition data of the environmental feedback into the experience replay pool, and drives the policy network parameter update of the reinforcement learning model to optimize its decision strategy.

[0071] The embodiment provides accurate environmental representation for the robot by establishing a detailed scene graph model. Then, the first language model is used to convert natural language instructions into structured task objectives, enhancing the robot's understanding of user intent. Next, the second language model generates auxiliary information and updates the scene graph model, improving the robot's semantic understanding and dynamic adaptation ability to the environment. The reinforcement learning model generates the optimal action decision sequence based on the updated scene graph, and continuously optimizes the task planning strategy through real-time feedback and model adjustment. Thus, the robot's task planning efficiency and success rate in complex dynamic environments are improved, solving the deficiencies of traditional methods in environmental modeling, generalization ability, and real-time decision-making, and providing strong technical support for the robot's application in industrial logistics, home service, disaster rescue, and other fields.

[0072] Optionally, in some embodiments, step S1 "establishes a scene graph model containing environmental spatial topology and object attributes" can specifically include:

[0073] S11. Extract node features in the current environment, including node type, physical attributes, and topological connection relationships between nodes;

[0074] Specifically, first, the current environment needs to be perceived and data collected to identify the node features of various objects in the environment. These node features specifically include node type (such as room, object, container, etc.), physical attributes of nodes (such as size, weight, color, material, etc.), and topological connection relationships between nodes (such as connectivity relationships between rooms, relationships between objects and their locations, etc.). Through the extraction of this information, a preliminary framework of the scene graph can be constructed, providing basic data for subsequent processing and analysis.

[0075] S12. Convert node features into vectorized spatial representation through embedding coding technology to generate mathematical representation of the scene graph model;

[0076] Specifically, the extracted node features are converted into vectorized spatial representations, for example, using embedding encoding techniques. Different features of each node (type, physical properties, topological connection relationships, etc.) are mapped into a high-dimensional vector space, so that these features can be effectively processed and analyzed mathematically. The mathematical representation of the generated scene graph model can describe the characteristics and relationships of the entire environment in a structured manner, facilitating subsequent algorithm processing and utilization.

[0077] S13. Based on the real-time state data of the target robot and the mathematical representation of the scene graph model, a dynamically expandable scene graph model data structure is constructed;

[0078] Specifically, when constructing the dynamically expandable scene graph model data structure, the real-time state data of the target robot (such as position, attitude, load, etc.) and the mathematical representation of the scene graph model need to be considered comprehensively. By integrating the state information of the robot into the scene graph model, the model can more accurately reflect the actual situation of the current task execution. At the same time, a dynamically expandable data structure is designed, so that when the environment changes or new information is perceived, the scene graph model can be conveniently and quickly updated and expanded, such as adding new nodes or modifying the connection relationships between nodes.

[0079] This embodiment extracts node features in the current environment and converts them into vectorized spatial representations to construct a dynamically expandable scene graph model data structure, which not only accurately captures various objects and their relationships in the environment, but also reflects the state changes of the target robot in real time.

[0080] Optionally, in some embodiments, step S2 "inputting the received natural language instruction into the first language model and outputting a structured task target" can specifically include:

[0081] S21. Analyzing the task subject, spatial relationship predicate, and constraint condition in the natural language instruction through the first language model to obtain parsed elements;

[0082] Specifically, the core of this step is to use the first language model to deeply analyze the user's natural language instruction and extract the key elements of the task, including the task subject (such as "vase"), the spatial relationship predicate (such as "put into"), and the constraint condition (such as "the sink in the kitchen"). Through natural language processing technology, the model can understand the semantic information in the instruction and convert it into a structured data form

[0083] S22. Converting the parsed elements into a target state logical expression of the planning domain definition language;

[0084] Specifically, after extracting the task elements, this step converts these elements into a goal state logical expression in Planning Domain Definition Language (PDDL). PDDL is a formal language used to describe the initial state and goal state of a task, which is the foundation of robot task planning. By mapping the parsed elements to predicates and logical expressions in PDDL, the model can generate a structured task goal description.

[0085] S23. Output a structured task goal based on the goal state logical expression, the structured task goal including a target object identifier, a target container identifier, and a set of reachability constraints;

[0086] Specifically, this step generates a structured task goal based on the PDDL goal state logical expression. The structured task goal explicitly specifies key information of the task, including the target object (e.g., "vase"), the target container (e.g., "sink"), and the reachability constraints (e.g., "the path from the living room to the kitchen must be clear"). These information provide clear task guidance for the reinforcement learning model.

[0087] This embodiment parses natural language instructions through a first language model, extracts task subjects, spatial relationship predicates, and constraint conditions as parsed elements, and converts them into PDDL goal state logical expressions, finally generates a structured task goal, significantly improves the understanding ability of robots for natural language instructions, and realizes the conversion from fuzzy natural language to precise task planning.

[0088] Optionally, in some embodiments, the auxiliary information includes at least one of the following:

[0089] A priority sequence of unexplored areas based on task relevance ranking;

[0090] Semantic supplement instructions for node attributes in the scene graph model.

[0091] Specifically, in the process of robot task planning, one of the auxiliary information is a priority sequence of unexplored areas based on task relevance ranking. This sequence is generated by a second language model combined with a common sense knowledge base, and is used to guide the priority order of the robot when exploring unknown areas. For example, if the task is to find "banana", according to the common sense knowledge base, the relevance score of the kitchen will be higher than that of the bedroom, so the kitchen will be explored first.

[0092] Another important component of auxiliary information is semantic supplement instructions for node attributes in the scene graph model. These instructions are generated by a second language model to update and enrich the attribute information of nodes in the scene graph model. For example, if the sensor detects that a new obstacle appears in a certain area, the second language model can generate semantic supplement instructions to add the type, location and impact range of the obstacle to the scene graph model.

[0093] Optionally, in some embodiments, the step S3 "generating auxiliary information through the second language model based on the structured task goal and the real-time environment perception data, and updating the scene graph model", specifically can include:

[0094] S31. Marking unexplored area nodes in the scene graph model;

[0095] Specifically, in the scene graph model, the robot needs to identify and mark unexplored area nodes. These unexplored area nodes represent the parts of the environment that the robot has not yet obtained information. Marking these nodes provides a basis for subsequent exploration strategies, helping the robot to determine the areas that need to be explored first.

[0096] S32. Calculating the relevance score of each unexplored area to the structured task goal through the second language model combined with the common sense knowledge base;

[0097] Specifically, this step uses the second language model and the common sense knowledge base to calculate the relevance score of the unexplored area to the task goal. Specifically, the second language model combines common sense knowledge (for example, "bananas are usually placed in the kitchen") to evaluate the potential contribution of unexplored areas to the completion of the task. Areas with high relevance scores will be explored first.

[0098] S33. Generating a sequence of area exploration priorities in descending order of relevance scores as input for path planning;

[0099] Specifically, according to the calculated relevance scores, the robot generates a sequence of area exploration priorities in descending order. This sequence determines which areas the robot should explore first, thereby optimizing the exploration path. The priority sequence will be used as input for the path planning algorithm to guide the robot to efficiently complete the task.

[0100] This embodiment optimizes the robot's exploration strategy by marking unexplored area nodes, calculating relevance scores, and generating exploration priority sequences. By combining the common sense knowledge base and the second language model, the robot can intelligently evaluate the relevance of unexplored areas to the task goal, prioritize high-relevance areas, significantly improve the robot's exploration efficiency and task planning success rate in complex dynamic environments, reduce the time and resource consumption of task execution, and enhance the autonomy and intelligence of the robot.

[0101] Optionally, in some embodiments, the step S3 "generating auxiliary information through the second language model based on the structured task goal and the real-time environment perception data, and updating the scene graph model", specifically can also include:

[0102] S34. Receiving externally input semantic scene description text;

[0103] Specifically, in the robot task planning process, the externally input semantic scene description text provides important information about the changes in the environment state. These texts can come from user input, sensor data, or notifications from other systems. For example, a user might input "the left corridor is congested" or a sensor detects "elevator 1 is out of service today." These semantic scene description texts provide additional environmental information for the robot, helping it to better understand and adapt to the current environment.

[0104] S35. Identify the environmental state change events in the semantic scene description text through the second language model, including the addition of obstacles, equipment downtime, or path closure;

[0105] Specifically, this step uses the second language model to analyze the semantic scene description text and identify the environmental state change events therein. These events include the addition of obstacles, equipment downtime, or path closure, etc. The second language model extracts key information from the text through natural language processing techniques and converts it into a structured event description. For example, "the left corridor is congested" is identified as an "addition of obstacles" event, and its location and impact range are recorded.

[0106] S36. Map the environmental state change events to attribute update operations of the corresponding nodes in the scene graph model;

[0107] Specifically, after identifying the environmental state change events, this step maps these events to attribute update operations of the corresponding nodes in the scene graph model. For example, the "addition of obstacles" event is mapped to adding a new node in the scene graph and updating its location and attributes; the "equipment downtime" event is mapped to modifying the state attribute of the corresponding node. Through these update operations, the scene graph model can reflect the latest state of the environment in real time.

[0108] This embodiment significantly enhances the robot's perception and adaptation to dynamic environments by receiving externally input semantic scene description texts, identifying environmental state change events, and mapping these events to attribute update operations of the scene graph model. This enables the robot to quickly respond to environmental changes, update the scene graph model in real time, thereby improving the accuracy and success rate of task planning, and enhancing the robot's autonomy and intelligence in complex dynamic environments.

[0109] Optionally, in some embodiments, the step S4 "input the updated scene graph model into the reinforcement learning model to output the action decision sequence" can specifically include:

[0110] S41. Convert the updated scene graph model into a state vector required by the reinforcement learning model;

[0111] Specifically, before inputting the scene graph model into the reinforcement learning model, it needs to be converted into a state vector. The state vector is the input form that the reinforcement learning model can process, which contains the current state information of the environment. Among them, the state vector is composed of multiple features, including the position of the robot, the position of the target object, the correlation score of the unexplored area, and the attribute information of the nodes in the scene graph. These information is converted into a fixed length vector through feature extraction and dimension reduction technology.

[0112] S42. Calculate the value function of each action through the deep Q network, and output the action decision with the highest value;

[0113] Specifically, the deep Q network (DQN) is an algorithm that combines deep learning and reinforcement learning, which is used to calculate the value function of each action. DQN approximates the Q value function through a neural network, with the input being the current state vector and the output being the Q value of each possible action. The action with the highest Q value is selected as the current optimal action decision. DQN uses the experience replay mechanism and target network to stabilize the training process, ensuring that the model can effectively learn the optimal policy.

[0114] S43. Use the post-experience replay mechanism to reconstruct the state that has not achieved the target as an alternative target and recalculate the reward value, optimize the decision sequence to generate the final action decision sequence;

[0115] Specifically, the post-experience replay mechanism (HER) increases the number of effective feedback by reconstructing the state that has not achieved the target as an alternative target and recalculating the reward value. After each round ends, HER takes the state actually achieved by the agent as a new target and recalculates the corresponding reward value. In this way, even in a sparse reward environment, the agent can obtain more learning signals, thereby optimizing the decision sequence.

[0116] This embodiment significantly improves the decision-making ability and task planning efficiency of the reinforcement learning model in complex dynamic environments by converting the scene graph model into a state vector, using the deep Q network to calculate the action value function, and using the post-experience replay mechanism to optimize the decision sequence. It effectively solves the learning efficiency problem of reinforcement learning in a sparse reward environment and improves the accuracy and success rate of robot task planning.

[0117] Optionally, in some embodiments, step S5 "control the target robot to execute the action decision sequence, and adjust the scene graph model and the reinforcement learning strategy of the reinforcement learning model according to the environmental feedback information", specifically can include:

[0118] S51. Trigger a new area discovery event when executing the action decision sequence, extract the spatial topology features and object attributes of the new area, and expand the node set of the scene graph model;

[0119] Specifically, during the process of the robot executing the sequence of action decisions, a new region discovery event can be triggered. When the robot enters an unexplored region, the system needs to extract the spatial topology features of the new region (such as the shape, size, and connection relationship with other regions of the room) and object attributes (such as the type, location, and state of the items). These new information will be integrated into the scene graph model, and the node set of the graph model will be extended by adding new nodes and updating the connection relationship between the nodes.

[0120] S52. When the task is not completed, the state transition data fed back by the environment is stored in the experience replay pool to drive the policy network parameter update of the reinforcement learning model;

[0121] Specifically, if the robot discovers during execution that the task has not been completed, the state transition data fed back by the environment (such as the current state, the action taken, the reward obtained, and the next state) will be stored in the experience replay pool. These data are used to train the reinforcement learning model, and by continuously updating the parameters of the policy network, the decision-making ability of the model is optimized.

[0122] The embodiment significantly improves the adaptability and learning efficiency of the robot in a complex dynamic environment by dynamically expanding the scene graph model during the process of the robot executing the sequence of action decisions and updating the reinforcement learning model using the experience replay pool, so that the robot can continuously learn and adapt to the new environment during the execution of the task, and optimize the task planning strategy.

[0123] Optionally, in some embodiments, the deep Q network works in the following way:

[0124] A separate main network and target network are constructed, and the parameters of the target network are periodically synchronized with the parameters of the main network;

[0125] The Q value is iteratively updated based on the Bellman optimality equation.

[0126] Specifically, the deep Q network (DQN) adopts a dual-network structure of a main network and a target network. The main network is responsible for calculating the Q value of each action according to the current state, while the target network is used to generate the target Q value. The parameters of the target network are periodically synchronized from the main network to maintain stability. This separate structure helps to reduce instability during training and improve the convergence speed of the model.

[0127] The core of DQN is to update the Q value based on the Bellman optimality equation:

[0128]

[0129] where r is the reward in the current state, g is the discount factor, s' is the next state, and a' is the possible action in the next state. The main network updates the network parameters by minimizing the mean squared error between the predicted Q value and the target Q value.

[0130] The embodiment significantly improves the training efficiency and decision quality of the deep Q network by constructing a separate main network and target network structure and an iterative update mechanism based on the Bellman optimal equation.

[0131] Optionally, in some embodiments, the dynamic updating of the scene relationship graph model is implemented through the following architecture:

[0132] Language model interaction interface: receives auxiliary information and converts it into graph update instructions;

[0133] Specifically, the language model interaction interface is an important module in the robot task planning system, responsible for receiving auxiliary information from the second language model and converting it into specific graph update instructions. Auxiliary information includes priority sequences of unexplored areas, environmental state change events, etc., which need to be converted into instructions that the scene relationship graph model can understand and process. For example, converting the "new obstacle" event into an instruction to add a new node and update its attributes.

[0134] Real-time perception parser: maps sensor data to node attribute changes;

[0135] Specifically, the real-time perception parser is responsible for processing raw data from various sensors and mapping it to changes in node attributes in the scene relationship graph model. For example, if a sensor detects that an obstacle in a certain area has moved, the real-time perception parser will extract this information and convert it into an update to the corresponding node's location attribute. This process requires efficient data processing and real-time performance to ensure the timeliness of the scene relationship graph model.

[0136] Graph structure decision engine: performs node addition, deletion, and attribute modification operations and verifies topological consistency.

[0137] Specifically, the graph structure decision engine is the core operation module of the scene relationship graph model, responsible for performing node addition, deletion, and attribute modification operations. When performing these operations, the graph structure decision engine also needs to verify the topological consistency of the graph to ensure that the structure of the graph is still valid after updating. For example, when deleting a node, it needs to check and update the connection relationships of other nodes connected to the node to avoid isolated nodes or invalid connections.

[0138] The embodiment realizes efficient dynamic updating of the scene relationship graph model by constructing a language model interaction interface, a real-time perception parser and a graph structure decision engine, ensures that the robot can update the understanding of the environment in real time and accurately, provides solid data support for task planning and decision making, and improves the adaptability and task execution success rate of the robot in a complex dynamic environment.

[0139] In specific embodiments, in order to obtain feedback on the planning of the scene graph, it is necessary to convert the scene graph into an environment that can be used by the agent to take actions, obtain observation space, action space, and reward feedback. Since the embodiment focuses on reasoning under the scene graph, the requirement for visualization is not high, so Gymnasium (the latest version of gym, hereinafter referred to as gym) is selected as the training environment of the reinforcement learning model.

[0140] Gym is a Python library developed by OpenAI for reinforcement learning research, which provides a rich simulation environment to help developers and researchers more easily conduct reinforcement learning experiments and algorithm development. Its core goal is to provide a standardized interface so that users can test and compare reinforcement learning algorithms in different environments. Gym provides a unified interface, such as creating an environment through gym.make(), and interacting with the environment through env.reset(), env.step(), etc. It also has good definitions of observation space (Observation Space) and action space (Action Space). The observation space describes the current state of the environment, while the action space defines the actions that the agent can take. In addition, Gym supports environment encapsulation and customization, and users can also create custom environments to meet specific research needs.

[0141] In the open source reinforcement learning scene, there is no gym environment that meets the long-term dynamic task planning of the scene relationship graph-based robot discussed in this patent, so a custom gym environment is needed. The specific operation is to inherit the gym.Env class, initialize the environment using init(), define the action space (actionSpace) and observation space (observationSpace), then write the reset() function to reset the environment to the initial state and the step(action) function to execute actions. In addition, in order to represent the scene graph information, it is also necessary to define the room class and the item class, and then use data structures such as dictionaries and lists to represent the scene graph.

[0142] The reinforcement learning model can only parse vector type data when interacting with the gym environment, so embedding coding of the observation space is also needed. The specific implementation is to first define node features, that is, define feature vectors for each node (room, area, location, item, container), which can include node types (e.g., room, item, container, etc.), node attributes (e.g., whether the container needs to be opened, whether the item is in stock), and relationships between nodes (e.g., connection relationships between rooms), etc., then use a graph neural network (GNN) to generate node embeddings, and finally combine the node embeddings with other state information (e.g., agent location, inventory status, etc.) to construct the observation space for reinforcement learning.

[0143] As shown in Figure 3 , the embodiment provides the overall flow of the robot task planning method combining large language models and reinforcement learning, mainly including the following parts:

[0144] Training environment part:

[0145] Dynamic task planning custom gym environment: the system first extracts scene relationship graph information to build a dynamic task planning custom gym environment. In this environment, the robot can perform actions and obtain corresponding state, reward and goal information. These information will be stored in the experience storage to provide data support for subsequent reinforcement learning training.

[0146] Performing actions: according to the action decision sequence output by the DQN network, the robot performs corresponding actions in the environment and returns the execution results to the environment.

[0147] Perception and planning part

[0148] Perception module: in real scene tasks, the perception module is responsible for collecting environmental information and converting it into a local scene relationship graph. This module is equivalent to the "eyes" and "ears" of the robot, helping the robot perceive the surrounding world.

[0149] Local scene relationship graph: integrate the perceived information into a local scene relationship graph to represent the robot's partial understanding of the current environment. The relationship graph will be represented in vector form and input into the planning large model.

[0150] Text representation and scene graph supplement: convert the local scene relationship graph into a text representation and input it into the planning large model. The planning large model supplements and perfects the local scene relationship graph according to the text representation and additional scene information, providing the robot with more comprehensive environmental information.

[0151] DQN network part

[0152] Online Network and Target Network: The DQN network consists of an online network and a target network. The online network calculates Q-values based on the current state, while the target network is used to generate target Q-values. Through a parameter synchronization mechanism, the parameters of the target network are periodically synchronized from the online network to maintain stability.

[0153] Q-Value Update: Based on the Bellman Optimality Equation, the system calculates the TD target and updates the Q-values of the online network. This process is iterated continuously, allowing the network to gradually learn the optimal action decision policy.

[0154] Trained DQN Model: After extensive training, the DQN model can output the next action, guiding the robot to perform tasks in real-world scenarios.

[0155] HER Module Part

[0156] Experience Replay Pool: The HER module retrieves original experiences from the experience storage and performs target relabeling to generate new experiences. These new experiences are stored in the experience replay pool for subsequent training.

[0157] Random Sampling: Randomly sample batches of experiences from the experience replay pool for training the DQN network. This random sampling method can break the correlation between experiences, improving the stability and efficiency of training.

[0158] Real-World Scenario Task Part

[0159] Real-World Scenario Task: In actual applications, the robot executes tasks in real-world scenarios based on the action decision sequence output by the trained DQN model.

[0160] Scenario Graph Update: During task execution, the robot continuously collects new environmental information and updates the local scenario graph. These updated information will be fed back to the entire system for further optimization of task planning strategies.

[0161] where DQN (Deep Q-Network) is an algorithm that combines deep learning and reinforcement learning. Its core idea is to approximate the Q-value function through a deep neural network, thereby solving the limitations of traditional Q-Learning in high-dimensional state spaces. The goal of DQN is to learn a Q-value function Q(s, a) that represents the expected value of future cumulative rewards when taking action a in state s. The Q-value function is updated through the Bellman equation:

[0162]

[0163] Unlike traditional Q-Learning, DQN uses a deep neural network to approximate the Q-value function, avoiding the need to store and update a large Q table in traditional Q-Learning. The input of the neural network is the state s, and the output is the Q value corresponding to each action a. In addition, the experience replay mechanism is used to reduce the correlation between samples, and a target network with the same structure as the main network is constructed to keep its parameters unchanged for a period of time.

[0164] In the scene relationship graph of a large scene, the agent often encounters the problem of sparse rewards because it cannot complete the task all the time. Therefore, the method of the embodiment also adds the HER method on the basis of DQN. HER (Hindsight Experience Replay) is a technique used to solve the problem of sparse rewards in reinforcement learning, and is particularly suitable for goal-oriented tasks such as robot grasping and path planning. HER redefines the goal and calculates the reward by temporarily modifying the task goal and using the experience of the agent that did not achieve the goal during exploration, thereby increasing the number of effective feedback. Specifically, HER takes the state actually reached by the agent as the new goal after each round ends, recalculates the reward, and generates new experience data.

[0165] Large language models (LLMs) have shown great potential in developing general task planning agents. However, in complex multi-floor, multi-room dynamic environments, how to effectively implement these planning schemes remains a major challenge in the field of robotics. Although significant progress has been made in LLM-based planning frameworks, inherent limitations such as high response latency and error accumulation in long sequence tasks still need to be addressed. To this end, the embodiment proposes an innovative architecture (as shown in Figure 4 ) that combines large language models with reinforcement learning models: a lightweight reinforcement learning model is used as the core planning module, and is only called when the large language model provides semantic reasoning support. This framework not only avoids the system latency caused by frequent calls to large models, but also effectively suppresses the error accumulation effect caused by large model hallucinations in long-range tasks through the planning mechanism dominated by small models.

[0166] The large language model plays an important role in three aspects: parsing natural language instructions provided by the user, supplementing additional knowledge of the local scene relationship graph, and providing exploration suggestions for unknown areas.

[0167] 1) Parsing natural language instructions provided by the user:

[0168] The large language model can be used to parse natural language instructions provided by the user. After training through prompt engineering and supervision engineering, the large model can convert natural language instructions such as "put the vase in the sink in the kitchen" into:

[0169] (:goal (and (in receptacle item61_vase_medium item receptacle52_sink))

[0170] Through this standard pddl language, the specific goal of the task is understood by the planning module centered on the small model, making the whole agent framework more flexible in receiving tasks.

[0171] 2) Supplement additional local scene relationship graph knowledge:

[0172] There is a custom data structure in the observation space of the small model module centered on DQN-HER reinforcement learning, called the local scene relationship graph. Since the scene of the experiment is defined as dynamic, the model does not have a god's perspective of the global scene, but rather gradually enriches the local scene relationship graph through step-by-step exploration to gradually understand the overall environment and make better task planning decisions. However, in some cases, some human hints and observations from the robot's external sensors will provide additional information that will speed up the agent's movement towards the correct destination. The small model is difficult to utilize this information due to its lack of understanding of general world knowledge and human natural language. The agent framework of this patent uses the understanding of general world knowledge and human natural language of the large model to analyze additional scene information such as "the left hallway is congested" and "the No. 1 elevator is out of service" into changes in the local scene relationship graph, thereby assisting the small model in better planning.

[0173] 3) Unknown area exploration suggestions:

[0174] When the agent explores unknown areas, the reinforcement learning model often refers to historical experience to use the most efficient exploration method, with the goal of exploring all areas as quickly as possible. In other words, in the concept of the reinforcement learning model, the value of all rooms is the same, and the only factor that affects the exploration order is the topology between rooms and the distance between them. However, in real-world robot tasks, different rooms have different relevance to the task, and the priority of exploration should also change due to this relevance. The large language model, relying on its understanding of world background knowledge, can provide suggestions when choosing the next exploration area (as shown in Figure 5 ), such as the large model can understand general prior knowledge such as "bananas are more likely to be in the kitchen than in the bedroom", thereby optimizing the exploration path of the reinforcement learning algorithm and ultimately reducing the average planning steps of the agent.

[0175] The specific algorithm flow is as follows: Figure 6As shown, Figure 6 An algorithm flowchart in pseudocode is provided, and the specific process is as follows:

[0176] (1) Initialization phase

[0177] Input and output definition: At the beginning of the algorithm, it is clear that the input is the environmental parameters (including room information, item information, etc.), and the output is the behavior sequence of the agent.

[0178] Environment initialization: Initialize the environment env, create a basic framework for task planning.

[0179] Memory initialization: Initialize the internal memory of the agent, i.e. the local scene relationship graph memory is empty, and prepare for information storage in subsequent task planning.

[0180] Task mode judgment: Determine whether to enable natural language task mode. If yes, enter the natural language task processing flow; otherwise, use the default task target information.

[0181] (2) Natural language task processing

[0182] Natural language input: User inputs a natural language representation of a task.

[0183] Task generation: Call the "task generation" large language model to convert abstract natural language into specific task target information, including target item target_item_id, target room target_room_id, etc.

[0184] Load task information: Call env.reset(), load task target information, and initialize task-related item and room information.

[0185] (3) Main loop

[0186] Loop start: Enter the main loop, continuously execute the task planning steps until the task is completed or the maximum step limit is reached.

[0187] (4) Scene description processing

[0188] Get scene description: Get the scene description scene_description from the user or sensor.

[0189] Determine the type of scene description: If it is "find", call the "unknown area rearrangement" large language model to rearrange the local scene relationship graph memory according to the task relevance. If it is not empty and not "find", call the "local scene graph supplement" large language model to update the local scene relationship graph memory.

[0190] (5) Reinforcement learning and action execution

[0191] Reinforcement learning model decision: The reinforcement learning model reads the memory, target_item_id, target_room_id, and current_room_id information, and gives the recommended action.

[0192] Perform action: The agent performs the action and interacts with the environment by calling env.step(action).

[0193] Task completion judgment: Check if the environment judges that the task is completed. If so, record and output the task completion information and exit the loop.

[0194] (6) New area exploration and memory update

[0195] New area exploration judgment: If the agent explores a new area, update the area information to the local scene relationship graph memory.

[0196] (7) Behavior recording and step number updating

[0197] Behavior recording: Record the behavior of the agent.

[0198] Step number updating: Update the total number of steps total_steps.

[0199] (8) Loop end

[0200] Loop end condition judgment: Check if the exit condition is met, if so, end the loop and return the behavior sequence of the agent.

[0201] This embodiment uses three different large language models to optimize the planning algorithm in three places: task generation, unknown area rearrangement, and local scene graph supplement. The "task generation" large language model is used to convert abstract natural language into specific task target information for the reinforcement learning model, improving the flexibility of task representation. The "unknown area rearrangement" large language model uses its general knowledge to sort unknown areas according to task relevance from large to small, prioritizing areas that are more likely to complete the task to reduce the average number of steps in task planning. The "local scene graph supplement" large language model is used to read scene descriptions to update the local scene relationship graph, which also reduces the average number of steps in task planning.

[0202] In a specific embodiment, this embodiment is divided into two parts, including the construction of a custom gym simulation environment, combined with a robot task planning algorithm based on large language models and reinforcement learning models.

[0203] Simulation environment setup: Use the API interface provided by openai gym to build a simulation environment for robot dynamic long-term task planning scene. In order to focus on the planning module based on scene graph, the experiment focuses on the relationship graph in the 3DSceneGraph-medium and 3DSceneGraph-tiny datasets of iGibson 2.0 open source, uses the task sampler to extract the planning task represented by PDDL (Planning Domain Definition Language) in it, and converts the PDDL file into a data form available to the custom Gym environment through regular expressions. The custom environment is implemented based on the Gymnasium interface, and by rewriting the observation space, action space, state transition method and reward function, the physical engine module unrelated to planning is stripped, greatly improving the training speed. An important step in data preprocessing is to extract object and predicate information in PDDL planning language, for example, using the regular expression pattern below:

[0204] \(\s*inreceptacle\s+([^\s]+)\s+([^\s)]+)

[0205] to extract the parameter information of the inreceptacle predicate, then through program processing, save the result in the python dictionary data structure, and finally save it as a.pkl file for easy loading at any time.

[0206] Robot task planning algorithm combining large language model and reinforcement learning model: The DQN-HER reinforcement learning model is trained in the test environment of the custom gym environment, and the large language model is used to enhance the understanding of the robot task planning problem through prompt engineering and mechanism engineering. In the test environment, the natural language understanding ability and general knowledge understanding ability of the large language model are used to modify the local scene graph, and then the decision of the reinforcement learning model is optimized.

[0207] Compared with the prior art, the task planning method for robots provided by the embodiment can generate structured task targets by analyzing natural language instructions through a large language model, provide clear task guidance for robots, improve the flexibility and accuracy of task planning, make robots able to make real-time decisions in dynamic environments through a reinforcement learning model, adapt to environmental changes, and avoid the shortcomings of traditional methods in dynamic response and long-term planning. The second language model is used to generate auxiliary information and update the scene relationship graph model, which can provide more rich environmental information for the reinforcement learning model, enhance the accuracy and adaptability of decision-making, and through the robot executing the action decision sequence and adjusting the model according to the environmental feedback, realize the continuous optimization and learning of the robot in actual operation, and further improve the efficiency and success rate of task planning.

[0208] It should be understood that, although Figure 2 The steps in the flowchart of FIG. 1 are not necessarily performed in the order indicated by the arrows. Unless specifically stated, the order of performance of the steps is not strictly limited to that shown and can be changed. Also, Figure 2 At least some of the steps in the flowchart of FIG. 1 can include multiple sub-steps or multiple stages, which are not necessarily performed at the same time, but can be performed at different times, and the order of the sub-steps or stages is not necessarily sequential, but can be round-robin or alternating with at least some of the other steps or sub-steps or stages of other steps.

[0209] To better implement the task planning method for robots of the embodiments of the present application, the embodiments of the present application further provide a task planning device for robots based on the above task planning method for robots. The meanings of the terms are the same as in the above task planning method for robots, and the specific implementation details can be referred to the description in the method embodiment.

[0210] Please refer to Figure 7 , Figure 7 The structure diagram of the task planning device for robots provided by the embodiments of the present application is specifically as follows, which can specifically include a model establishing module 201, a first processing module 202, a second processing module 203, a third processing module 204, and a control adjustment module 205, and can be specifically as follows:

[0211] The model establishing module 201 is configured to establish a scene relationship graph model containing environment space topology and object attributes.

[0212] The first processing module 202 is configured to input the received natural language instruction into a first language model, and output a structured task target.

[0213] The second processing module 203 is configured to generate auxiliary information through a second language model based on the structured task target and real-time environment perception data, and update the scene relationship graph model.

[0214] The third processing module 204 is configured to input the updated scene relationship graph model into a reinforcement learning model, and output a sequence of action decisions.

[0215] The control adjustment module 205 is configured to control the target robot to execute the sequence of action decisions, and adjust the reinforcement learning strategy of the scene relationship graph model and the reinforcement learning model according to the environment feedback information.

[0216] Optionally, in some embodiments, the model establishing module 201 can specifically include:

[0217] a node extraction unit configured to extract node features in a current environment, the node features including node types, physical attributes, and inter-node topological connection relationships;

[0218] a vectorization unit configured to convert the node features into vectorized space representations through embedding coding techniques, to generate a mathematical representation of the scenario graph model;

[0219] a construction unit configured to construct a dynamically expandable scenario graph model data structure based on real-time state data of a target robot and the mathematical representation of the scenario graph model.

[0220] Optionally, in some embodiments, the first processing module 202 can specifically include:

[0221] a parsing unit configured to parse a task subject, a spatial relationship predicate, and a constraint condition in the natural language instruction through a first language model, to obtain parsed elements;

[0222] a conversion unit configured to convert the parsed elements into a target state logical expression in a planning domain definition language;

[0223] a target unit configured to output a structured task target based on the target state logical expression, the structured task target including a target object identifier, a target container identifier, and a reachability constraint set.

[0224] Optionally, in some embodiments, the second processing module 203 can specifically include:

[0225] a marking unit configured to mark unexplored area nodes in the scenario graph model;

[0226] a calculation unit configured to calculate a relevance score of each unexplored area with respect to the structured task target through a second language model in combination with a common sense knowledge base;

[0227] a sorting unit configured to generate a sequence of area exploration priorities in descending order of the relevance scores, as an input basis for path planning.

[0228] Optionally, in some embodiments, the third processing module 204 can specifically include:

[0229] a receiving unit configured to receive an externally input semantic scene description text;

[0230] an identification unit configured to identify an environmental state change event in the semantic scene description text through a second language model, the environmental state change event including an added obstacle, a device outage, or a path closure;

[0231] a mapping unit configured to map the environmental state change event into an attribute update operation for a corresponding node in the scenario graph model.

[0232] Optionally, in some embodiments, the third processing module 204 can specifically include:

[0233] a state unit configured to convert the updated scene graph model into a state vector required by the reinforcement learning model;

[0234] a value unit configured to calculate a value function of each action by the deep Q network and output an action decision with the highest value;

[0235] an optimization unit configured to use a post-experience replay mechanism to reconstruct a state that fails to achieve a target into an alternative target and recalculate a reward value, and generate a final action decision sequence after optimizing a decision sequence.

[0236] Optionally, in some embodiments, the control adjustment module 205 can specifically include:

[0237] an expansion unit configured to trigger a new region discovery event when the action decision sequence is executed, extract spatial topology features and object attributes of the new region, and expand a node set of the scene graph model;

[0238] an update unit configured to store state transition data fed back by the environment into an experience replay pool when a task is not completed, and drive policy network parameter update of the reinforcement learning model.

[0239] Specific limitations on the task planning device for the robot can be referred to the limitations on the task planning method for the robot in the foregoing, which will not be repeated here. Each module in the task planning device for the robot described above can be realized by software, hardware, and a combination thereof, in whole or in part. Each module described above can be embedded in or independent of a processor in a computer device in hardware form, or can be stored in a memory in the computer device in software form, so as to be called and executed by the processor to perform operations corresponding to each module.

[0240] The task planning device for the robot provided in the embodiment can generate a structured task target by analyzing a natural language instruction through a large language model, provide clear task guidance for the robot, and improve flexibility and accuracy of task planning. The robot can make real-time decisions in a dynamic environment through a reinforcement learning model, adapt to environmental changes, and avoid shortcomings of traditional methods in dynamic response and long-term planning. The second language model is used to generate auxiliary information and update a scene graph model, which can provide more rich environmental information for the reinforcement learning model, and enhance accuracy and adaptability of decisions. The robot executes an action decision sequence and adjusts the model according to environmental feedback, realizes continuous optimization and learning of the robot in actual operation, and further improves efficiency and success rate of task planning.

[0241] In addition, the embodiment of the application further provides an electronic device, such as Figure 8As shown in FIG. 1, which shows a structural schematic diagram of an electronic device related to embodiments of the present application, in particular:

[0242] The electronic device can include a processor 301 having one or more processing cores, a memory 302 having one or more computer-readable storage media, a power supply 303, and an input unit 304, etc. Those skilled in the art can understand that the electronic device structure shown in FIG. 1 does not constitute a limitation on the electronic device, and can include more or fewer components than shown, or combine certain components, or different component arrangements. Among them: Figure 8

[0243] The processor 301 is the control center of the electronic device, which connects various parts of the entire electronic device through various interfaces and lines, and performs various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 302, and calling data stored in the memory 302, thereby overall monitoring the electronic device. Optionally, the processor 301 can include one or more processing cores; preferably, the processor 301 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application program, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 301.

[0244] The memory 302 can be used to store software programs and modules, and the processor 301 executes various functions and task planning methods for robots by running the software programs and modules stored in the memory 302. The memory 302 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), etc.; the data storage area can store data created according to the use of the electronic device, etc. In addition, the memory 302 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 302 can also include a memory controller to provide access for the processor 301 to the memory 302.

[0245] The electronic device also includes a power supply 303 for powering various components, and preferably the power supply 303 can be logically connected to the processor 301 through a power management system, so as to realize functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 303 can also include one or more direct current or alternating current power supplies, a recharging system, a power supply failure detection circuit, a power supply converter or inverter, a power supply state indicator, etc. any component.

[0246] ​The electronic device can further include an input unit 304, which can be used to receive inputted digital or character information, and to generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0247] Although not shown, the electronic device can further include a display unit or the like, which will not be described here. In particular in the present embodiment, the processor 301 in the electronic device will load the executable file corresponding to the process of one or more application programs into the memory 302 according to the following instructions, and run the application program stored in the memory 302 by the processor 301, thereby realizing various functions, as follows:

[0248] A scene graph model containing environmental space topology and object attributes is established; a received natural language instruction is input into a first language model to output a structured task target; auxiliary information is generated through a second language model based on the structured task target and real-time environmental perception data, and the scene graph model is updated; an updated scene graph model is input into a reinforcement learning model to output a sequence of action decisions; the target robot is controlled to execute the sequence of action decisions, and the reinforcement learning strategy of the scene graph model and the reinforcement learning model is adjusted according to environmental feedback information.

[0249] The specific implementation of each of the above operations can be referred to the previous embodiments, which will not be described here.

[0250] The embodiments of the present application parse natural language instructions through a large language model to generate a structured task target, providing clear task guidance for the robot and improving the flexibility and accuracy of task planning; through a reinforcement learning model, the robot can make real-time decisions in a dynamic environment, adapting to environmental changes and avoiding the shortcomings of traditional methods in dynamic response and long-term planning; auxiliary information is generated through a second language model and the scene graph model is updated, which can provide more rich environmental information for the reinforcement learning model, enhancing the accuracy and adaptability of decision-making; the robot executes the sequence of action decisions and adjusts the model according to environmental feedback, realizing continuous optimization and learning of the robot in actual operation, and further improving the efficiency and success rate of task planning.

[0251] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by instructions controlling related hardware, which can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0252] To this end, the embodiments of the present application provide a storage medium, which stores a plurality of instructions, which can be loaded by a processor to execute the steps in any of the task planning methods for robots provided by the embodiments of the present application. For example, the instructions can execute the following steps:

[0253] establishing a scene graph model containing environmental space topology and object attributes; inputting the received natural language instruction into a first language model to output a structured task target; generating auxiliary information through a second language model based on the structured task target and real-time environmental perception data, and updating the scene graph model; inputting the updated scene graph model into a reinforcement learning model to output a sequence of action decisions; controlling a target robot to execute the sequence of action decisions, and adjusting the reinforcement learning strategy of the scene graph model and the reinforcement learning model according to environmental feedback information.

[0254] The specific implementation of each operation can refer to the foregoing embodiments, which will not be described here again.

[0255] The storage medium can include a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, and the like.

[0256] Due to the instructions stored in the storage medium, the steps of any of the task planning methods for robots provided in the embodiments of the present application can be executed, and thus the beneficial effects of any of the task planning methods for robots provided in the embodiments of the present application can be achieved. Details can be found in the foregoing embodiments, which will not be described here again.

[0257] The task planning method for robots, the device, the equipment and the medium provided in the embodiments of the present application are described in detail above, and the principle and implementation manner of the present application are described by applying specific examples in this paper. The above embodiment description is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range will be changed, and the above description should not be understood as the limitation of the present application.

Claims

1. A task planning method for a robot, characterized by, The method comprises the following steps: establishing a scene graph model containing environmental space topology and object attributes; inputting the received natural language instruction into a first language model to output a structured task target; generating auxiliary information through a second language model based on the structured task target and real-time environmental perception data, and updating the scene graph model; inputting the updated scene graph model into a reinforcement learning model to output a sequence of action decisions; controlling a target robot to execute the sequence of action decisions, and adjusting the scene graph model and the reinforcement learning strategy of the reinforcement learning model according to environmental feedback information.

2. The task planning method for a robot according to claim 1, characterized in that, The scene graph model containing environmental space topology and object attributes is established by: extracting node features in the current environment, the node features including node types, physical attributes and topological connection relationships between nodes; converting the node features into vectorized space representations through embedding coding technology to generate mathematical representations of the scene graph model; constructing a dynamically expandable scene graph model data structure based on real-time state data of the target robot and the mathematical representations of the scene graph model.

3. The task planning method for a robot according to claim 1, wherein, The natural language instruction is inputted into the first language model to output a structured task target by: analyzing a task subject, a spatial relationship predicate and a constraint condition in the natural language instruction through the first language model to obtain analysis elements; converting the analysis elements into a target state logical expression in a planning domain definition language; outputting a structured task target based on the target state logical expression, the structured task target including a target object identifier, a target container identifier and a reachability constraint set.

4. The task planning method for a robot according to claim 1, wherein The auxiliary information is generated through the second language model based on the structured task target and real-time environmental perception data, and the scene graph model is updated by: labeling unexplored area nodes in the scene graph model; calculating relevance scores of each unexplored area and the structured task target through the second language model combined with a common sense knowledge base; generating a sequence of area exploration priorities in descending order of relevance scores as input basis for path planning.

5. The task planning method for a robot according to claim 1, wherein, The auxiliary information is generated through the second language model based on the structured task target and real-time environmental perception data, and the scene graph model is updated by: receiving externally input semantic scene description texts; identifying environmental state change events in the semantic scene description texts through the second language model, the environmental state change events including newly added obstacles, device shutdowns or path closures; mapping the environmental state change events into attribute update operations of corresponding nodes in the scene graph model. 6.The task planning method for a robot according to claim 1, wherein, The updated scene graph model is inputted into the reinforcement learning model to output a sequence of action decisions by: converting the updated scene graph model into a state vector required by the reinforcement learning model; calculating a value function of each action through a deep Q network to output an action decision with the highest value; adopting a post-experience replay mechanism to reconstruct a state that fails to achieve a target into an alternative target and recalculate a reward value, thereby optimizing the sequence of decisions to generate a final sequence of action decisions. 7.The task planning method for a robot according to claim 1, wherein The control target robot executes the action decision sequence, and adjusts the scene relation graph model and the reinforcement learning strategy of the reinforcement learning model according to environment feedback information, comprising: Trigger a new region discovery event when executing the action decision sequence, extract the spatial topology features and object attributes of the new region, and expand the node set of the scene relation graph model; When the task is not completed, store the state transition data of the environment feedback into an experience replay pool, and drive the policy network parameter update of the reinforcement learning model.

8. A task planning device for a robot, characterized by comprising: Comprise: A model establishing module is configured to establish a scene relation graph model containing environment spatial topology and object attributes; A first processing module is configured to input the received natural language instruction into a first language model, and output a structured task target; A second processing module is configured to generate auxiliary information through a second language model based on the structured task target and real-time environment perception data, and update the scene relation graph model; A third processing module is configured to input the updated scene relation graph model into a reinforcement learning model, and output an action decision sequence; A control adjustment module is configured to control a target robot to execute the action decision sequence, and adjust the scene relation graph model and the reinforcement learning strategy of the reinforcement learning model according to environment feedback information.

9. An electronic device, comprising: Comprise: A memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the steps of the task planning method for the robot according to any one of claims 1-7 when executing the computer program.

10. A storage medium, characterized by A computer program capable of being loaded and executed by the processor to implement the task planning method for the robot according to any one of claims 1-7 is stored.

Images