Task planning method and system for reinforcement learning driven robot with body

Abstract

The invention, which relates to the technical field of path optimization, discloses a task planning method and system for a reinforced learning-driven robot with a body, comprising the following steps: acquiring semantic features of a robot task instruction and real-time environment state data and performing joint coding; a semantic constraint vector is obtained, a candidate action sequence is screened from a preset action primitive library, an action causal association graph is obtained, and a first action execution path is generated; constructing a probability time sequence field model, and predicting the state of a dynamic object around the robot to obtain probability prediction distribution; acquiring a robot action space occupation area, performing state conflict evaluation on the robot action space occupation area based on probability prediction distribution to obtain a state conflict set, dynamically adjusting the first action execution path and performing reinforcement learning to obtain a second action execution path, the problem that deviation occurs when the robot executes tasks due to dynamic changes of obstacles and objects in the environment and the motion sequence of the obstacles and the objects is solved.

Description

Technical Field

[0001] This invention relates to the field of path optimization technology, and more specifically, to a reinforcement learning-driven method and system for planning embodied robot tasks. Background Technology

[0002] With the continuous development of animate robot technology, especially in fields such as intelligent manufacturing, logistics and distribution, and service robots, animate robots not only need to handle routine physical operations when performing tasks, but also need to cope with complex and dynamically changing environments. Therefore, animate robots need to possess the ability to perceive and understand environmental changes in real time and react promptly to ensure that tasks can be completed efficiently and safely. This ability to handle complex and dynamic environments is key to realizing the autonomy and intelligence of animate robots and their widespread application in real-world scenarios.

[0003] However, the dynamic changes of obstacles and objects in the environment, and their movement sequence, can cause deviations in the embodied robot's task execution. Specifically, in real-world environments, the position and shape of obstacles are often dynamically changing, potentially shifting or altering due to human activity, the movement of other robots, or objects. These changes are often unpredictable, especially in complex work scenarios such as warehousing, production lines, or indoor service environments. For example, in warehousing and logistics, stacked goods may be moved by other robots or human operators, altering the robot's planned path or task execution sequence; in smart manufacturing, temporary equipment shutdowns or adjustments during production can also cause the robot's operation timing to deviate from expectations. Furthermore, the movement sequence of objects also impacts task execution. In environments with multiple objects or human interactions, the movement of objects is not merely random; there may be specific sequences or dependencies between them—for example, moving one object before another, otherwise, conflicts or path blockages may occur. Because these object movements are continuous and mutually influential, robots often cannot anticipate future changes, leading to deviations between path planning and actual execution. These deviations may cause the robot to fail to execute tasks along the predetermined path, thereby affecting the efficiency and accuracy of the task. To address these problems, this invention proposes a solution. Summary of the Invention

[0004] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a reinforcement learning-driven embodied robot task planning method and system. By jointly encoding the semantic features of task instructions and real-time environmental data, a probabilistic temporal field model is constructed to predict the state of dynamic objects. Combined with state conflict evaluation, the robot's action path is dynamically adjusted to solve the problem of deviations in the embodied robot's task execution caused by the dynamic changes of obstacles and objects in the environment and their movement sequence.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A reinforcement learning-driven embodied robot task planning method includes the following steps: acquiring semantic features of robot task instructions and real-time environmental state data and jointly encoding them to obtain a semantic constraint vector; selecting candidate action sequences from a pre-defined action primitive library based on the semantic constraint vector to obtain an action causal relationship graph and generating a first action execution path based on the action causal relationship graph; constructing a probabilistic temporal field model and predicting the state of dynamic objects around the robot based on the probabilistic temporal field model to obtain a probability prediction distribution; acquiring the robot's action space occupancy area and evaluating state conflicts in the robot's action space occupancy area based on the probability prediction distribution to obtain a state conflict set; dynamically adjusting the first action execution path based on the state conflict set and performing reinforcement learning to obtain a second action execution path.

[0006] In a preferred embodiment, the step of obtaining the semantic features of the robot task instructions and real-time environmental state data and performing joint encoding to obtain a semantic constraint vector specifically involves: parsing the natural language description of the robot task instructions, extracting verb-object structures and modifiers, and generating a semantic dependency tree. The task target entity and action constraints are extracted from the semantic dependency tree to form a structured semantic graph; real-time environmental state data is acquired, and the motion trajectory of dynamic objects is extracted based on the real-time environmental state data to obtain an environmental state map; the structured semantic graph and the environmental state map are spatiotemporally aligned, and the corresponding instances of semantic entities in the environment and their spatial relationships are identified to obtain a coupled graph; graph embedding encoding is performed on the coupled graph to extract node features and edge relationship features to obtain semantic constraint vectors.

[0007] In a preferred embodiment, the step of selecting candidate action sequences from a preset action primitive library based on semantic constraint vectors to obtain an action causal relationship graph specifically involves: extracting the task objective and spatial boundary from the semantic constraint vectors, and constructing a first constraint condition based on the task objective and spatial boundary; retrieving candidate action sequences that meet the first constraint condition from the preset action primitive library; performing causal reasoning analysis on the candidate action sequences, and identifying the preconditions and post-conditions between the candidate action sequences based on the analysis results to obtain an action causal directed graph.

[0008] In a preferred embodiment, the step of performing causal reasoning analysis on candidate action sequences and identifying the preconditions and post-action states between candidate action sequences based on the analysis results to obtain a causal directed graph of actions specifically involves: defining the environmental state conditions required for the execution of each candidate action sequence as precondition nodes; analyzing the changes in the environmental state after the execution of each candidate action sequence and constructing post-action state nodes based on the analysis results; establishing directed edges from precondition nodes to preset action nodes and from preset action nodes to post-action state nodes to obtain causal edges; constructing a state transition function and performing state rolling updates on the causal edges based on the state transition function to obtain a state transition sequence; and merging and connecting the precondition nodes, preset action nodes, post-action state nodes, and corresponding causal edges in the state transition sequence to obtain a causal directed graph of actions.

[0009] In a preferred embodiment, generating the first action execution path based on the action causal relationship graph specifically involves: performing a path search on the action causal relationship graph to filter out actionable paths from the initial state to the preset task target state; extracting the action node set of the actionable paths and constructing a first action set based on semantic constraint vectors; performing a set operation on the action node set and the first action set to obtain the Jaccard similarity of each actionable path; and filtering the actionable paths based on the Jaccard similarity to obtain the first action execution path.

[0010] In a preferred embodiment, the step of constructing a probabilistic temporal field model and predicting the state of dynamic objects around the robot based on the probabilistic temporal field model to obtain a probabilistic prediction distribution specifically involves: acquiring the motion trajectory of the dynamic object and extracting the motion pattern features of the dynamic object based on the motion trajectory; constructing a probabilistic temporal field model with spatiotemporal positions as nodes based on the motion pattern features of the dynamic object; performing multi-step temporal inference based on the probabilistic temporal field model and predicting the position probability distribution of the dynamic object in future time periods to obtain a probabilistic prediction distribution.

[0011] In a preferred embodiment, the step of obtaining the robot's motion space occupancy area and evaluating the state conflict of the robot's motion space occupancy area based on the probability prediction distribution to obtain a state conflict set specifically involves: obtaining the robot's physical model and kinematic constraints, and identifying the motion space occupancy area of ​​the robot when executing the first motion execution path; discretizing the motion space occupancy area into several grid occupancy units, and mapping the probability prediction distribution to the grid occupancy units; calculating the probability density of dynamic objects appearing in each grid occupancy unit, and comparing the probability density with a preset probability threshold; identifying conflicts in the grid occupancy units based on the comparison results, obtaining conflicting grid units, and obtaining the pose mapping relationship of the conflicting grid units in the robot's first motion execution path; identifying the motion state nodes corresponding to the conflicting grid units based on the pose mapping relationship, and summarizing all motion state nodes to form a state conflict set.

[0012] In a preferred embodiment, the step of dynamically adjusting the first action execution path based on the state conflict set and performing reinforcement learning to obtain the second action execution path specifically involves: extracting the conflict time window and spatial constraints corresponding to the first action execution path based on the state conflict set; locating the conflict segment in the first action execution path based on the conflict time window and spatial constraints, and calculating the adjustable time margin and adjustable spatial offset range of the conflict segment; inputting the adjustable time margin and adjustable spatial offset range into a preset reinforcement learning network and outputting the conflict resolution action; and performing time shifting or spatial rerouting on the conflict segment based on the conflict resolution action to generate the second action execution path.

[0013] The technical effects and advantages of the reinforcement learning-driven embodied robot task planning method and system of the present invention are as follows: This invention, by acquiring the semantic features of task instructions and real-time environmental state data and performing joint encoding, tightly integrates the task objective with the current environmental state, forming a precise semantic constraint vector. This step ensures that the robot can fully understand the task instructions and perceive and adapt to the dynamic changes in the surrounding environment in real time, providing a solid foundation for subsequent task execution. Next, based on the semantic constraint vector, candidate action sequences are selected from a pre-set action primitive library, and an action causal relationship graph is generated. Through causal reasoning analysis of the sequential relationships between candidate action sequences, it is ensured that each candidate action meets the requirements of the current task and can avoid path conflicts or execution deviations caused by environmental changes during actual execution. Furthermore, the system constructs a probabilistic temporal field model and performs multi-step temporal reasoning based on this model to predict the future state and trajectory of dynamic objects. This step effectively solves the path conflict problem caused by the movement of dynamic objects in the environment. Through accurate prediction of the behavior of dynamic objects, the robot can proactively adjust its execution path, avoiding task deviations caused by obstacles or changes in the movement sequence of objects. Subsequently, by acquiring the robot's motion space occupancy area and performing state conflict assessment, the motion patterns of objects are matched with the robot's motion paths to identify potential conflict areas. The paths are then dynamically adjusted using a set of state conflicts. Finally, based on a reinforcement learning mechanism, the first motion path is optimized according to the conflict information. Reinforcement learning is used to adjust the conflict path and generate a second motion execution path, ensuring that the task can be executed smoothly in dynamically changing environments. This effectively addresses task execution deviations caused by environmental obstacles and changes in the order of object motion, greatly improving the adaptability and execution efficiency of the embodied robot in complex environments and ensuring that tasks are completed efficiently and accurately. Attached Figure Description

[0014] Figure 1 This is a flowchart illustrating the reinforcement learning-driven embodied robot task planning method of the present invention.

[0015] Figure 2 This is a schematic diagram of the reinforcement learning-driven embodied robot task planning system of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0017] Example 1, Figure 1The present invention provides a reinforcement learning-driven task planning method for embodied robots, comprising the following steps: S1, acquire the semantic features of the robot task instructions and the real-time environmental state data and perform joint encoding to obtain the semantic constraint vector; In this example, the semantic features of the robot's task instructions and real-time environmental state data are obtained and jointly encoded to obtain a semantic constraint vector, specifically: Parse the natural language description of robot task instructions, extract verb-object structures and modifiers, and generate a semantic dependency tree; Extract the task target entity and action constraints from the semantic dependency tree to form a structured semantic graph; Acquire real-time environmental state data, and extract the motion trajectory of dynamic objects based on the real-time environmental state data to obtain an environmental state map; The structured semantic graph is spatiotemporally aligned with the environment state graph, and the corresponding instances of semantic entities in the environment and their spatial relationships are identified to obtain a coupled graph. Graph embedding encoding is performed on the coupled graph to extract node features and edge relationship features, resulting in a semantic constraint vector.

[0018] It's important to note that, firstly, the natural language description of the robot's task instructions is broken down into lexical units and tagged with parts of speech. For example, the sentence "put the box on the table" is tagged as "put (verb), box (noun), put (verb), to (preposition), table (noun), on (locative word)". Next, the relationship between the verb and the object is extracted, and modifiers (such as locative words, time words, etc.) are identified. This information is used to form a semantic dependency tree through dependency parsing, where the connections between each lexical node represent their grammatical or semantic relationships. For example, "put" as a verb points to "box" as the object, and "to the table" as a modifier points to the verb "put", indicating the target location.

[0019] Secondly, the task target entity and action constraints can be extracted from the semantic dependency tree. The task target entity is usually a noun or noun phrase, such as "box" being the target, while the action constraints are modifiers or phrases related to the action, such as "place on the table" being the target location for the action. By traversing the dependency tree, verbs and the entities they modify are identified, and the core task target and constraints are extracted. This information is ultimately organized into a structured semantic graph, where nodes represent target entities and constraints, and edges represent the relationships between them. For example, the graph might have a "box" node, connected to a "place" action node, and then connected to a target location "table" node.

[0020] Furthermore, real-time environmental state data is acquired through sensors (such as cameras and LiDAR) to capture information about surrounding objects, including static objects (such as walls and tables) and dynamic objects (such as people and moving objects). Based on this data, the motion trajectories of dynamic objects are detected, and their spatial positions and temporal changes are recorded. For example, if a person is detected moving from left to right, their displacement path is recorded. By analyzing the motion trajectories, the motion pattern features of objects are extracted to form an environmental state map, where each node represents an object and its position, and the edges between objects represent their spatial relationships and motion trajectories.

[0021] Furthermore, the spatiotemporal alignment of the structured semantic graph with the environment state graph involves mapping target entities in the semantic graph to actual objects in the environment. For example, "box" is a target entity in the semantic graph, while an object that is a box may have already been detected in the environment state graph. During alignment, it is also necessary to identify the spatial relationships between entities; for example, in "put the box on the table," the "table" is the location where the target is placed. Through alignment, a coupling graph is established, matching entities in the semantic graph with instances in the environment graph and representing the spatial relationships between them in the graph. For example, nodes in the coupling graph include both "box" as a target entity and physical "box" instances detected in the environment.

[0022] Finally, graph embedding encoding is performed on the coupled graph, converting the nodes and edges into high-dimensional vector representations for subsequent processing. Node features can include information such as entity type, location, and state, while edge features describe the spatial relationships between nodes (such as "near" or "above"). By integrating the information of each node and edge into a vector representation through graph embedding methods, not only is the semantic information in the task preserved, but also the state features of the environment are incorporated.

[0023] S2, based on semantic constraint vectors, candidate action sequences are selected from the preset action primitive library to obtain an action causal relationship graph and the first action execution path is generated based on the action causal relationship graph; In this example, candidate action sequences are selected from a pre-defined action primitive library based on semantic constraint vectors to obtain an action causal relationship graph, specifically as follows: Extract the task objective and spatial boundary from the semantic constraint vector, and construct the first constraint condition based on the task objective and spatial boundary; Retrieve candidate action sequences that meet the first constraint from the preset action primitive library; Causal reasoning analysis is performed on candidate action sequences, and the antecedent conditions and consequent states between candidate action sequences are identified based on the analysis results, resulting in a causal directed graph of actions.

[0024] It's important to note that the process of extracting the task objective and spatial boundary from the semantic constraint vector first focuses on the task objective (e.g., "box" in "put the box on the table") and the spatial boundary (e.g., "table" and its spatial extent). The task objective is typically the object of the action, such as the target object or task state, while the spatial boundary defines the physical limitations or range of the action's execution. By analyzing the semantic graph, the task objective (e.g., "box") and constraints (e.g., "position on the table") are extracted. The spatial boundary is typically a specific region, such as a specific spatial area or the surface of an object, restricting the object's reachable positions. Based on the task objective and spatial boundary, the first constraint is constructed; this is a crucial step in modeling the constraints during task execution. The first constraint includes requirements for the target object (e.g., "box" must be placed on the "table") and specifies that the object should be located within a specific spatial range. For example, if the task objective is "put the box on the table," then the spatial boundary is the surface of the table, which constitutes the spatial constraint for the action's execution.

[0025] Secondly, when retrieving candidate action sequences that meet the first constraint from the pre-defined action primitive library, it's essential to understand the concept of the "action primitive library." The action primitive library contains a series of predefined action units, each describing the basic behaviors and actions the robot can perform. These behaviors are typically physically executable and have certain spatial requirements. For example, the action primitive library might include basic actions such as "grasping," "placing," and "rotating," which can be combined into more complex tasks. The retrieval process filters candidate action sequences that meet the first constraint (e.g., "put the box on the table"). For instance, suppose the action primitive library contains actions such as "grasping the box," "moving the box to a designated location," and "placing the box." By matching these actions with the spatial boundary (tabletop) in the first constraint, action sequences that meet the constraint can be selected, such as "grasping the box" -> "moving the box" -> "placing the box on the table." These action sequences meet the target task and satisfy the spatial requirements, thus becoming candidate action sequences.

[0026] Furthermore, when performing causal reasoning analysis on candidate action sequences, it is first necessary to understand the preconditions and post-action states of each action. Preconditions refer to the conditions that must be met before an action is executed, while post-action states refer to the effects or states produced after the action is executed. For example, in an action sequence, the precondition for "grabbing the box" is that the box must be in a grabbable state, while "placing the box" has a post-action state, namely, the box is successfully placed on the table.

[0027] Finally, during the causal reasoning analysis, the dependencies between actions in each candidate action sequence are analyzed. For example, "grabbing the box" is a prerequisite action before "placing the box on the table"; the box can only be placed after it has been successfully grabbed. Based on this analysis, the dependencies between actions are organized into a causal directed graph. In this directed graph, each node represents an action, and directed edges represent causal relationships between actions, such as the causal relationship between "grabbing the box" and "placing the box." This causal directed graph not only shows the order of actions but also clarifies the conditions (preconditions) that must be met to execute each action and the resulting state (aftereffect state).

[0028] In this example, causal reasoning analysis is performed on candidate action sequences, and the antecedent conditions and consequential states between candidate action sequences are identified based on the analysis results, resulting in a directed causal graph of actions, specifically: Define the environmental state conditions required for the execution of each candidate action sequence and use them as prerequisite condition nodes; Analyze the changes in the environmental state after the execution of each candidate action sequence, and construct the subsequent state node based on the analysis results; Construct directed edges from the precondition node to the preset action node and from the preset action node to the subsequent state node to obtain causal edges; Construct a state transition function and perform rolling state updates on causal edges based on the state transition function to obtain a state transition sequence; By merging and connecting the precondition nodes, preset action nodes, subsequent state nodes, and corresponding causal edges in the state transition sequence, we obtain an action-causal directed graph.

[0029] It's important to note that when defining the environmental conditions required for each candidate action sequence, the specific environmental conditions that must be met before the action can be executed must first be identified. For example, if a candidate action is "grab a box," then its prerequisites would be either "the box must be within the robot's reach" or "the box must not be obstructed." These environmental conditions are typically related to the object's position, state (e.g., whether it can be grasped), or the dynamics of the surrounding environment (e.g., the presence of obstacles). Each candidate action sequence's prerequisite node contains these conditions and indicates that the action can only proceed if these conditions are met.

[0030] Taking the action of "grabbing a box" as an example, the prerequisite nodes may include the following: the box must be located within the robot's grasping workspace, the box must not be obstructed by other objects, and the box must be stationary. These environmental conditions ensure that the "grabbing a box" action can be safely performed under specific environmental conditions.

[0031] Secondly, when analyzing the environmental state changes after each candidate action sequence is executed, the key is to assess the impact of the action on the environment. For example, after performing the "grab a box" action, the environmental state changes—the box's state changes from "stationary" to "grabbed," and its spatial position also changes (e.g., it moves to the robot's arm). This state change can be recorded as a follow-up state node. A follow-up state node describes the new environmental or object state after the action is executed, reflecting the direct impact of the action on the environment. Taking the "grab a box" action as an example, the follow-up state nodes might include: the box being grasped, a change in the box's posture (e.g., being lifted from the table), and a change in the box's position (moving to the robot arm's predetermined position). These follow-up states provide state change information for subsequent action sequences and also provide a basis for constructing a causal relationship graph.

[0032] Furthermore, causal edges are crucial in constructing a causal directed graph. First, directed edges must be established from precondition nodes to pre-action nodes. This indicates that the execution of an action is conditional, and the precondition node describes the environmental state that must be satisfied before the action can be executed. For example, the precondition for the action "grab the box" is that the box is in an accessible position, so there is a directed edge from the precondition node (box is grabbable) to the "grab the box" action node. Then, directed edges also need to be established from the pre-action node to the subsequent state node, representing the environmental change caused by the action. For example, after the "grab the box" action is executed, the box changes from a "static" state to a "grabbed" state, and these two states are connected by a directed edge. Each directed edge represents a clear causal relationship: the action can be executed after the precondition is satisfied, and the environmental state changes after the action is executed.

[0033] Furthermore, the state transition function is a function that describes how the environment state changes over time; it defines how the environment state changes after a specific action is performed. Based on the preceding analysis, the state transition function updates the environment state according to the execution of actions. For example, after performing the action "grab the box," the state transition function updates "box is stationary" to "box has been grabbed" and records this change. By performing state rolling updates on the causal edges of the action execution, the state transition function can reflect these changes.

[0034] State transition functions simulate the impact of actions on the environment and update the state of each action based on these impacts. For example, the action of "placing the box on the table" updates the state from "the box was picked up" to "the box is placed on the table." The state transition sequence is a chain that records all these environmental changes, providing dynamic environmental feedback for subsequent task planning.

[0035] Finally, all the precondition nodes, preset action nodes, post-action state nodes, and causal edges are merged and connected to obtain a complete action-causal directed graph. This graph represents each step in the task execution process and their interrelationships. Each node represents an environmental state, action, or condition, and each directed edge represents the causal relationship between the precondition, action, and post-action state. For example, suppose the task is "grab a box and put it on the table." The construction process of the action-causal directed graph might be as follows: starting from the initial state "the box is on the table," proceeding through the action "grab the box" (precondition: the box is grabbable, post-action state: the box is grabbed), then the action "move the box," and finally the action "place the box on the table" (precondition: the box is grabbed, post-action state: the box is placed on the table). All these nodes and edges are connected by causal relationships to form a complete task execution graph, representing how the robot reaches the final goal step by step from the initial state.

[0036] In this example, the execution path of the first action is generated based on the action causal relationship graph, specifically as follows: Perform path search on the action causal relationship graph to filter out the action paths from the initial state to the preset task target state; Extract the set of action nodes for possible action paths, and construct the first action set based on semantic constraint vectors; Perform a set operation between the action node set and the first action set to obtain the Jaccard similarity of each possible action path; Based on Jaccard similarity, the possible action paths are filtered to obtain the first action execution path.

[0037] In this example, the formula for calculating the Jaccard similarity of action paths is as follows: in, For Jaccard similarity, This is the set of action nodes for the first possible action path. This is the set of action nodes for the second possible action path. For intersection operators, This is the union operator.

[0038] It's important to understand that path search involves finding a feasible path from an initial state to a pre-defined task target state within an action causal relationship graph. In this process, the initial state represents the robot's current environmental state, such as "the box is on the table." The pre-defined task target state is the desired state upon task completion, such as "the box is successfully placed in the designated location." Path search analyzes the action causal relationship graph to find a sequence of actions from the initial state to the target state, ensuring that the preconditions for each action are met and that executing these actions ultimately leads to the target state. For example, suppose the initial state is "the box is on the table," and the target state is "the box is placed in the designated location." Path search might first check if the action "grab the box" can be performed, then the action "move the box to the designated location," and finally the action "place the box." This smooth transition from the initial state to the target state satisfies the task objective. The pre-defined task target state is the final desired state after task completion, typically determined during the task planning phase by analyzing the task's natural language description or other instruction information. For instance, the task objective might be "place the box in a certain location on the table," and the target state is the box precisely in the designated location on the table.

[0039] Secondly, during the path search process, we extract all action nodes from a possible action path, forming an action node set. The action node set contains every basic action performed on the path, such as "grab the box", "move the box", and "place the box".

[0040] Based on the semantic constraint vector, we can filter actions that conform to the task semantics from the action node set to construct the first action set. The semantic constraint vector contains information such as the task objective and environmental constraints, which helps us select the most suitable actions from the action library. For example, assuming the task is to place a box on a table, the semantic constraint vector might include "box" and "table" as entities, limiting the actions to those related to these objects, thus constructing an action set that meets the semantic requirements. For example, the action set might only include the actions of "grabbing the box," "moving the box to the table," and "placing the box," excluding other irrelevant actions.

[0041] Furthermore, Jaccard similarity is a metric for measuring the similarity between two sets. Calculating Jaccard similarity involves comparing the overlap between the action node set and the first action set. Specifically, the action node set in each possible action path is first subjected to a set operation with the first action set selected based on semantic constraint vectors, and the ratio of their intersection to their union is calculated. For example, suppose the action node set of a possible action path contains "grab a box," "move a box," and "place a box," while the first action set only contains "grab a box" and "place a box." Then their intersection is "grab a box" and "place a box," and their union is "grab a box," "move a box," and "place a box." Jaccard similarity is the ratio of the intersection to the union, i.e., 2 / 3, representing the degree to which this path conforms to the semantic constraints.

[0042] Finally, based on the calculated Jaccard similarity, all feasible action paths can be filtered. Paths with higher Jaccard similarity indicate a higher degree of conformity with the task objective and semantic constraints, and therefore these paths will be prioritized. Paths with lower similarity may not meet the task requirements or have significant deviations, and therefore will be excluded. For example, if there are multiple feasible action paths, one path has a Jaccard similarity of 0.9, indicating a high degree of match with the semantic constraints; while another path has a similarity of only 0.5, indicating a lower degree of match with the constraints. After filtering, the path with higher similarity will be selected as the first action execution path, meaning the robot will execute the task under this path. This ensures that the executed path not only conforms to the task objective but is also the most efficient under the semantic constraints.

[0043] S3. Construct a probabilistic temporal field model and predict the state of dynamic objects around the robot based on the probabilistic temporal field model to obtain the probability prediction distribution. In this example, a probabilistic temporal field model is constructed, and the state of dynamic objects around the robot is predicted based on the probabilistic temporal field model to obtain the probability prediction distribution, specifically: Acquire the motion trajectory of a dynamic object and extract the motion pattern features of the dynamic object based on the motion trajectory; Construct a probabilistic temporal field model with spatiotemporal location as the node based on the motion pattern characteristics of dynamic objects; Multi-step temporal reasoning is performed based on a probabilistic temporal field model, and the probability distribution of the position of dynamic objects in future time periods is predicted to obtain the probability prediction distribution.

[0044] It's important to note that acquiring the trajectory of a dynamic object first requires real-time acquisition of its position data in the environment using sensors (such as cameras, LiDAR, or other environmental monitoring devices). This position data is typically stored in time-series format, describing the object's spatial position changes at different points in time. For example, images captured by a camera or point cloud data captured by LiDAR can be used to track the object's trajectory over specific time intervals. The trajectory of a dynamic object is a collection of this position data, representing the object's path in three-dimensional space as it changes over time. Extracting the motion pattern features of a dynamic object involves analyzing its motion patterns from its trajectory. Motion pattern features describe the object's typical motion behavior, such as speed, acceleration, and changes in direction. These features can be extracted by analyzing the trajectory data. For example, an object might move at a specific speed (such as constant speed) along a straight line or curve, or exhibit periodic motion patterns (such as reciprocating motion). Motion pattern features may include the object's direction of movement, the pattern of speed change, and the pattern of acceleration change; these features can help predict the object's future motion behavior.

[0045] Furthermore, when constructing a probabilistic temporal field model, the motion trajectory of a dynamic object first needs to be discretized in a spatiotemporal coordinate system. Spatiotemporal positions serve as nodes in the model, encompassing not only the object's spatial coordinates (e.g., x, y, z coordinates) but also the timestamp of the object's current time. Each node represents the object's spatial position at a given moment, and multiple nodes together form the object's complete motion trajectory over a period of time. To construct the probabilistic temporal field model, the relationships between these spatiotemporal position nodes need to be modeled. Specifically, the model analyzes the relationships between adjacent nodes in the object's historical motion trajectory (i.e., the object's relative position and motion behavior between a given moment and the next), defining a transition probability for each pair of adjacent nodes. This represents the probability distribution of the object moving from one position to the next. The probabilistic temporal field model captures the temporal and spatial dependencies of object motion through the transition probabilities between spatiotemporal position nodes. During the construction process, the connection methods and transition probabilities of spatiotemporal position nodes are influenced by the object's motion pattern characteristics. If the object typically moves along a straight line, the model considers the direction and speed of each movement and assigns corresponding probability weights to these movements. If the motion of an object follows a pattern, a probabilistic time series field model can effectively reflect these patterns and capture the trends and patterns of motion.

[0046] Finally, multi-step temporal inference is based on the known historical trajectory of a dynamic object, using a probabilistic temporal field model to predict its future state. First, using the model's initial state (i.e., the current spatiotemporal position of the object) as a starting point, the possible position of the object at the next moment is inferred based on the information of the current spatiotemporal position node. This is based on the transition probabilities of the probabilistic temporal field model, where each step of the inference depends on the object's position at the previous moment and its motion pattern characteristics. Predicting the probability distribution of the dynamic object's position in future time periods means that, given the object's current state and motion pattern, the probabilistic temporal field model infers the possible positions the object may appear in the future. For example, assuming the object is currently located at coordinates (10, 20) and moving forward at a high speed, the model will predict the object's possible positions in the next few time steps based on historical data. These predicted positions are not a single, definite location, but a probability distribution, meaning the object may appear in multiple locations with different probabilities. For example, an object might have a 40% probability of appearing at (12, 20), a 30% probability at (13, 20), a 20% probability at (14, 20), and a remaining 10% probability at other locations. Through multi-step temporal reasoning, the model can predict not only the object's single location but also its spatial distribution over multiple future moments, thus obtaining a probability distribution of its positions. This probability prediction provides strong support for robot path planning, obstacle avoidance, and decision-making because it helps the robot predict the possible behaviors of surrounding dynamic objects and adjust its behavioral strategies based on these predictions.

[0047] S4. Obtain the robot's motion space occupied area and evaluate the state conflict of the robot's motion space occupied area based on the probability prediction distribution to obtain the state conflict set. In this example, the robot's motion space occupies the area, and a state conflict assessment is performed on the robot's motion space occupies the area based on a probability prediction distribution, resulting in a state conflict set, specifically: Obtain the robot's physical model and kinematic constraints, and identify the motion space occupied by the robot when executing the first action path; The action space is discretized into several grid cells, and the probability prediction distribution is mapped to the grid cells. Calculate the probability density of dynamic objects appearing in each grid cell and compare the probability density with a preset probability threshold; Based on the comparison results, conflict identification is performed on the grid occupancy cells to obtain conflicting grid cells and the pose mapping relationship of the conflicting grid cells in the robot's first action execution path is obtained. Based on the pose mapping relationship, the action state nodes corresponding to the conflict mesh cells are identified, and all action state nodes are summarized to form a state conflict set.

[0048] It's important to note that a robot's physical model and kinematic constraints refer to a series of physical attributes describing its form and motion behavior, such as structure, size, joint degrees of freedom, and motion limitations. The physical model includes information such as the robot's geometry, mass distribution, and the maximum range of motion of its joints. For example, a robot may have multiple joints, and the range of motion of each joint is constrained within a certain angular range. Kinematic constraints, on the other hand, include the robot's motion capabilities, particularly constraints related to its joint movements, such as speed and acceleration limits, or limitations imposed by its mechanical structure that prevent certain actions from being performed.

[0049] When executing the first action path, the robot completes the task step by step according to a certain sequence of actions. At this time, it is necessary to identify the motion space occupancy area, that is, the area occupied by each part of the robot (such as the robotic arm, gripper, moving platform, etc.) in space when performing each action. Identifying these areas is crucial because they are directly related to the space occupancy of other objects in the environment and can affect the robot's actions. For example, when the robot performs a "grasping" action, its robotic arm may require a certain amount of space to complete the grasping action; this space is the "motion space occupancy area." If the robot performs an action within this area, it may collide with other objects; therefore, accurate calculation and identification of the motion space occupancy area are essential.

[0050] Secondly, discretization transforms a continuous spatial region into several small discrete units, typically represented by a grid. In this process, the spatial region that the robot might occupy during task execution needs to be divided into many small grid units based on the robot's physical model and kinematic constraints. Each grid unit represents a small spatial region, and these grids cover the robot's entire action space.

[0051] Once these grid cells are determined, the next step is to map the probability prediction distribution onto them. This probability prediction distribution originates from the motion pattern prediction of dynamic objects and is calculated using a probabilistic temporal field model. The future positions of these dynamic objects represent a probability distribution; therefore, this probability distribution can be mapped onto the grid cells of the robot's motion space based on the object's possible future positions. For example, if the predicted trajectory of a dynamic object at a certain moment indicates a high probability of it appearing at a certain location in the future, then the probability density of the grid cell containing that location will be higher.

[0052] Furthermore, the probability density of dynamic objects appearing in each grid cell is calculated based on the probability distribution of the dynamic object's position in different grid cells. Each grid cell has a probability density value, which represents the likelihood of an object appearing at that location. If the motion pattern prediction of a dynamic object shows a 70% probability of it appearing at a certain location, then the probability density of that grid cell is 0.7. Next, these probability densities need to be compared with a preset probability threshold. The probability threshold is usually a value set based on the task's safety requirements and physical constraints. For example, if the probability density of an object in a certain grid cell exceeds the threshold, it indicates that there may be a risk of collision in that grid cell, meaning that the probability of a dynamic object existing in that area is high, which may cause the robot to collide while performing the task.

[0053] Furthermore, during conflict identification, we compare the probability density of each grid cell with a preset threshold. If the probability density of a grid cell exceeds the threshold, that grid cell is considered to have a potential conflict. Conflicting grid cells represent grid cells where objects may appear during robot actions, potentially leading to a collision between the robot and the object.

[0054] Pose mapping refers to the relationship between conflicting mesh cells and the robot's position and orientation. Pose includes the robot's position (usually coordinates) and orientation (direction). In path planning, pose mapping indicates that when the robot performs a certain action, a mesh cell may conflict with the robot's current action state (such as position and orientation). For example, if the robot needs to enter a mesh cell when performing a "grasp" action, and that cell is currently occupied by a dynamic object, then that cell becomes a conflicting cell, and this conflict is related to the robot's current position and orientation.

[0055] Finally, identifying the action state nodes corresponding to conflicting mesh cells means finding the action state nodes associated with each conflicting mesh cell based on the pose mapping relationship. An action state node represents the robot's action or state at a specific moment. For example, if the robot's action space overlaps with a mesh cell while performing an action, then that action state will be associated with the conflicting mesh cell.

[0056] By aggregating all the action state nodes corresponding to conflicting mesh cells, a state conflict set can be formed. This set includes all action state nodes that may lead to conflict; these nodes represent which actions or states in the robot's execution path will conflict with dynamic objects in the environment. By identifying these conflicts, the robot can adjust its path to avoid collisions. For example, suppose a robot is performing a path planning task and needs to traverse a mesh region at a certain point in the path. Within this region, probability prediction shows a 50% chance of a dynamic object being present, and this region happens to be the robot's action space. If this probability density exceeds a preset threshold, the robot will identify the potential conflict in this region and find the action state nodes associated with that region based on the pose mapping relationship.

[0057] S5. Based on the state conflict set, the execution path of the first action is dynamically adjusted and reinforcement learning is performed to obtain the execution path of the second action.

[0058] In this example, the execution path of the first action is dynamically adjusted and reinforcement learning is performed based on the set of state conflicts to obtain the execution path of the second action, specifically: Based on the state conflict set, extract the conflict time window and spatial constraints corresponding to the first action execution path; Based on the conflict time window and spatial constraints, the conflict segment is located in the first action execution path, and the adjustable time margin and adjustable spatial offset range of the conflict segment are calculated. Input the adjustable time margin and adjustable spatial offset range into the preset reinforcement learning network, and output conflict resolution actions; Based on the conflict resolution action, the conflict segment is shifted in time or rerouted in space to generate a second action execution path.

[0059] It's important to note that conflict time windows and spatial constraints are crucial concepts in robot task planning. Based on the state conflict set, the specific conflict time periods and spatial constraints can be extracted by analyzing the relationship between each conflict state node and the robot's motion path. A conflict time window refers to the time period within which the robot is likely to collide with dynamic objects in the environment during its execution path. In other words, the conflict time window represents the period during which an object might appear on the robot's path at a certain moment, leading to a collision. For example, if an object has a high probability of colliding with the robot between the 3rd and 5th seconds of the robot's execution path, then this period is the conflict time window. Spatial constraints refer to specific areas or spaces that the robot needs to avoid at a certain moment or during a certain action segment to prevent collisions with dynamic objects. For example, if a region's grid cells are occupied by a dynamic object, the robot needs to avoid this region. In this case, the spatial constraints describe the spatial locations the robot needs to avoid.

[0060] Secondly, once the conflict time window and spatial constraints are identified, the next step is to determine which segments in the first action execution path will be affected by the conflict and analyze the adjustable range of these conflict segments. This is done by comparing the conflict time window with the robot's path to find the path portions that intersect with the conflicting object. These path portions are the conflict segments. For example, if the robot's path segment from 3 seconds to 5 seconds overlaps with the conflict time window of an object, then this path segment is a conflict segment.

[0061] Furthermore, adjustable time margin refers to the amount of time a robot can delay or advance within a conflict segment to avoid conflict. For example, if the conflict time window is 3 to 5 seconds, and the robot can perform its next action 5 seconds later, then the adjustable time margin is the time difference between the two. By adjusting the time, collisions can be avoided. Adjustable spatial offset range refers to the range of space a robot can move within to avoid collisions with objects. If a dynamic object in a certain area affects a grid cell in the robot's path, the robot can deviate in certain directions to bypass the obstacle. Therefore, the adjustable spatial offset range is the degree of freedom a robot has in space to adjust its path.

[0062] Furthermore, reinforcement learning is a highly effective technique in path planning and decision-making, especially suitable for tasks requiring dynamic adaptation to changing environments. Here, a reinforcement learning network is used to output a conflict resolution action based on the conflict time window and spatial constraints. In reinforcement learning, the network learns how to take appropriate actions (such as time adjustment or spatial rerouting) based on the current state (in this scenario, the conflict time window and spatial constraints). The network optimizes its decision-making strategy through a continuous trial-and-error process, based on reward and penalty signals. In this case, the network's input is the conflict time window, spatial constraints, and the state of the current path; the output is the specific conflict resolution action. For example, the network might decide to delay a certain action by 2 seconds or slightly shift the path to avoid collisions. Specifically, the reinforcement learning model learns which path adjustments are most effective through extensive historical data and simulations. When encountering a new conflict scenario, the network calculates the most suitable solution based on the learned strategies. In this process, reinforcement learning not only relies on current environmental information but also considers the success rates of historical paths and actions, continuously optimizing path adjustment decisions through feedback.

[0063] Once the reinforcement learning network outputs conflict resolution actions, the robot adjusts its path accordingly based on these suggestions. These adjustments are typically achieved through temporal translation or spatial rerouting. If the conflict arises from a dynamic object in the path, the robot might avoid collisions by delaying certain actions (i.e., temporal translation). For example, if an action requires overlapping with an object and causing a conflict, the robot might wait for the object to move out of the area before performing the action, or avoid the conflict by changing the timing of the actions. If the conflict occurs in a specific spatial region, the robot can adjust its path to avoid the conflict area through spatial rerouting. For example, the robot can bypass dynamic objects by choosing a different path or shifting its position spatially.

[0064] Finally, the second action execution path is a new execution path after conflict resolution. The main difference between the second and first action execution paths is that the second path avoids the conflict area or time period. The advantage is that the second path can effectively avoid collisions with dynamic objects, improving the safety and efficiency of task execution while maintaining the achievement of the task objective. For example, the first path might need to pass through an area containing a dynamic object, but the second path bypasses that area through time translation or spatial rerouting, thus avoiding collisions and increasing the task success rate.

[0065] Example 2, Figure 2 The present invention provides a reinforcement learning-driven embodied robot task planning system, comprising a semantic encoding module, an action planning module, a probability prediction module, a conflict evaluation module, and an execution adjustment module. The semantic encoding module is used to acquire the semantic features of robot task instructions and real-time environmental state data and perform joint encoding to obtain a semantic constraint vector; The action planning module is used to filter candidate action sequences from a preset action primitive library based on semantic constraint vectors, obtain an action causal relationship graph, and generate the first action execution path based on the action causal relationship graph; The probability prediction module is used to construct a probabilistic temporal field model and predict the state of dynamic objects around the robot based on the probabilistic temporal field model to obtain the probability prediction distribution. The conflict assessment module is used to obtain the robot's motion space occupied area and to perform state conflict assessment on the robot's motion space occupied area based on probability prediction distribution, so as to obtain a state conflict set. The execution adjustment module is used to dynamically adjust the execution path of the first action based on the state conflict set and perform reinforcement learning to obtain the execution path of the second action.

[0066] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0067] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.

[0068] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0069] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0070] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0071] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A reinforcement learning driven embodied robot task planning method, characterized in that, The method comprises the following steps: obtaining semantic features of robot task instructions and real-time environment state data and jointly encoding to obtain a semantic constraint vector; screening a candidate action sequence from a preset action primitive library based on the semantic constraint vector, obtaining an action causal correlation graph and generating a first action execution path based on the action causal correlation graph; constructing a probabilistic time series field model and predicting dynamic object states around the robot based on the probabilistic time series field model to obtain a probability prediction distribution; obtaining a robot action space occupation area and performing state conflict evaluation on the robot action space occupation area based on the probability prediction distribution to obtain a state conflict set; dynamically adjusting the first action execution path according to the state conflict set and performing reinforcement learning to obtain a second action execution path.

2. The reinforcement learning driven embodied robotic task planning method of claim 1, wherein, The obtaining of the semantic features of the robot task instructions and the real-time environment state data and the joint encoding to obtain the semantic constraint vector specifically comprises: parsing natural language descriptions of the robot task instructions and extracting verb-object structures and modifiers to generate a semantic dependency tree; extracting task target entities and action constraint conditions from the semantic dependency tree to form a structured semantic graph; obtaining real-time environment state data and extracting dynamic object motion trajectories based on the real-time environment state data to obtain an environment state atlas; spatiotemporally aligning the structured semantic graph and the environment state atlas and identifying corresponding instances of semantic entities in the environment and their spatial relationships to obtain a coupled graph; performing graph embedding coding on the coupled graph to extract node features and edge relationship features to obtain the semantic constraint vector.

3. The reinforcement learning driven embodied robotic task planning method of claim 2, wherein, The screening of the candidate action sequence from the preset action primitive library based on the semantic constraint vector to obtain the action causal correlation graph specifically comprises: extracting task targets and spatial boundaries in the semantic constraint vector and constructing a first constraint condition based on the task targets and the spatial boundaries; retrieving candidate action sequences that meet the first constraint condition from the preset action primitive library; performing causal reasoning analysis on the candidate action sequences and identifying preconditions and post-effect states between the candidate action sequences according to the analysis results to obtain an action causal directed graph.

4. The reinforcement learning driven embodied robotic task planning method of claim 3, wherein, The performing of the causal reasoning analysis on the candidate action sequences and the identifying of the preconditions and the post-effect states between the candidate action sequences according to the analysis results to obtain the action causal directed graph specifically comprises: defining environment state conditions required for execution of each candidate action sequence as precondition nodes; analyzing environment state changes after execution of each candidate action sequence and constructing post-effect state nodes according to the analysis results; establishing directed edges from the precondition nodes to preset action nodes and from the preset action nodes to the post-effect state nodes to obtain causal edges; constructing a state transition function and performing state rolling update on the causal edges based on the state transition function to obtain a state transition sequence; merging and connecting the precondition nodes, the preset action nodes, the post-effect state nodes and the corresponding causal edges in the state transition sequence to obtain the action causal directed graph.

5. The reinforcement learning driven embodied robotic task planning method of claim 4, wherein, The generating of the first action execution path based on the action causal correlation graph specifically comprises: performing path search on the action causal correlation graph to screen out actionable paths from an initial state to a preset task target state; extract an action node set of the actionable path, and construct a first action set based on the semantic constraint vector; perform set operation on the action node set and the first action set to obtain Jaccard similarity of each actionable path; perform path screening on the actionable paths based on the Jaccard similarity to obtain a first action execution path.

6. The reinforcement learning driven embodied robotic task planning method of claim 5, wherein, The probability temporal field model is constructed, and the state of the dynamic object around the robot is predicted based on the probability temporal field model to obtain a probability prediction distribution, specifically as follows: acquire the motion trajectory of the dynamic object and extract the motion pattern feature of the dynamic object based on the motion trajectory; construct a probability temporal field model with a space-time position as a node based on the motion pattern feature of the dynamic object; perform multi-step temporal reasoning based on the probability temporal field model, and predict the position probability distribution of the dynamic object in the future time period to obtain the probability prediction distribution.

7. The reinforcement learning driven embodied robotic task planning method of claim 6, wherein, The robot action space occupation area is acquired, and state conflict set is obtained by state conflict evaluation on the robot action space occupation area based on the probability prediction distribution, specifically as follows: acquire the physical model and kinematic constraint of the robot, and identify the action space occupation area of the robot when the first action execution path is executed; discretize the action space occupation area into a plurality of grid occupation units, and map the probability prediction distribution into the grid occupation units; calculate the probability density of the dynamic object appearing in each grid occupation unit, and compare the probability density with a preset probability threshold; perform conflict identification on the grid occupation units according to the comparison result to obtain the conflict grid unit and acquire the pose mapping relationship of the conflict grid unit in the first action execution path of the robot; identify the action state node corresponding to the conflict grid unit based on the pose mapping relationship, and aggregate all the action state nodes to form the state conflict set.

8. The reinforcement learning driven embodied robotic task planning method of claim 7, wherein, The first action execution path is dynamically adjusted and reinforcement learning is performed according to the state conflict set to obtain a second action execution path, specifically as follows: extract the conflict time window and the spatial constraint condition corresponding to the first action execution path based on the state conflict set; locate the conflict section in the first action execution path based on the conflict time window and the spatial constraint condition, and calculate the adjustable time allowance and the adjustable spatial offset range of the conflict section; input the adjustable time allowance and the adjustable spatial offset range into a preset reinforcement learning network to output a conflict resolution action; perform time translation or spatial rerouting on the conflict section based on the conflict resolution action to generate the second action execution path.

9. A reinforcement learning driven embodied robot task planning system applying the reinforcement learning driven embodied robot task planning method of any one of claims 1-8, characterized in that, The semantic encoding module, the action planning module, the probability prediction module, the conflict evaluation module, and the execution adjustment module are included: The semantic encoding module is configured to acquire semantic features of a robot task instruction and real-time environment state data and perform joint encoding to obtain a semantic constraint vector; The action planning module is configured to select a candidate action sequence from a preset action primitive library based on the semantic constraint vector, obtain an action causal correlation graph, and generate a first action execution path based on the action causal correlation graph; The probability prediction module is configured to construct a probability temporal field model, and predict the state of a dynamic object around the robot based on the probability temporal field model to obtain a probability prediction distribution; A conflict evaluation module is configured to obtain a robot action space occupation region, and perform state conflict evaluation on the robot action space occupation region based on a probability prediction distribution to obtain a state conflict set. An execution adjustment module is configured to perform dynamic adjustment on a first action execution path according to the state conflict set and perform reinforcement learning to obtain a second action execution path.

Images