Task planning capability evaluation method and device, electronic equipment and program product

Abstract

The invention relates to the technical field of intelligent technology, and provides a task planning capability evaluation method and device, electronic equipment and a program product. The method comprises the steps of obtaining a first node list based on a sample video; obtaining a second node list according to the to-be-evaluated task planning sequence; inputting the input data into a calling interface of the first multi-modal large language model, and at least obtaining node accuracy; wherein the input data comprises a first node list, a second node list and a first prompt word; the node correct rate represents the matching degree between a plurality of nodes in the second node list and a plurality of nodes in the first node list; the first prompt word comprises first prompt information, and the first prompt information is used for prompting the first multi-mode large language model to recognize whether nodes in the first node list are matched with nodes in the second node list or not; and obtaining an evaluation result of the to-be-evaluated task planning sequence at least according to the node accuracy. The method can improve the evaluation credibility.

Description

Technical Field

[0001] This invention relates to the field of embodied intelligence technology, and in particular to a method and apparatus for evaluating task planning capabilities, electronic devices, and program products. Background Technology

[0002] Embodied intelligence emphasizes that intelligent agents achieve autonomous learning and evolution through dynamic interactions between their bodies and the environment, deeply integrating perception, action, and cognition. Intelligent agents can perceive the world through their eyes (cameras), ears (microphones), hands (robotic arms), and body (mobile chassis), and influence the world through their actions, learning and understanding in the process. Intelligent agents not only need low-level precise control capabilities but also high-level cognitive abilities, such as understanding human instructions, perceiving complex scenes, formulating long-term task plans, reasoning about availability, and reflecting on and correcting task plans after failures. Task planning ability is one of the core capabilities required of intelligent agents. For example, large models such as Multimodal Large Language Models (MLLMs), acting as the core brain of intelligent agents, require long-term task planning. Accurately evaluating the task planning capabilities of such large models is crucial for improving the task planning ability of intelligent agents. Some unreasonable evaluation methods can lead to inaccurate and unreliable evaluation results. Summary of the Invention

[0003] This application provides a method, apparatus, electronic device, and program product for evaluating task planning capabilities, which can accurately evaluate the task planning capabilities of large models such as MLLM and solve the technical problem of low reliability of evaluation results.

[0004] In a first aspect, embodiments of this application provide a method for evaluating task planning capabilities. The method includes: obtaining a first node list based on sample videos; obtaining a second node list based on a task planning sequence to be evaluated; the task planning sequence is output by the target model to be evaluated and includes multiple actions; the first node list and the second node list adopt a unified standardized data format and each includes multiple nodes, each node including an action function name and the name of the operated object; inputting input data to the calling interface of a first multimodal large language model to obtain at least a node accuracy rate; wherein the input data includes the first node list, the second node list, and a first prompt word; the node accuracy rate represents the matching degree between multiple nodes in the second node list and multiple nodes in the first node list; the first prompt word includes first prompt information, which is used to prompt the first multimodal large language model to identify whether nodes in the first node list match nodes in the second node list; and obtaining the evaluation result of the task planning sequence to be evaluated based at least on the node accuracy rate.

[0005] In some embodiments, the input data further includes a directed acyclic graph and an initial scene image; before inputting the input data to the calling interface of the first multimodal large language model, the method further includes: obtaining a directed acyclic graph based on a first node list; the directed acyclic graph includes vertices and edges; vertices represent actions, edges are used to connect different vertices, and an edge indicates that there is a dependency relationship between the two actions represented by the two connected vertices; extracting the first frame image or the image of a preset frame from the sample video as the initial scene image; inputting the input data to the calling interface of the first multimodal large language model to obtain at least the node accuracy, including: inputting the input data to the calling interface of the multimodal large language model to obtain the node accuracy and key state completion rate; wherein, the key state completion rate represents the completion rate of the key state obtained based on the task planning sequence to be evaluated relative to the key states in the key state set; the key state represents the state after the spatial position and / or structure of an object changes; the key state set is obtained based on the directed acyclic graph and includes multiple key states that need to be achieved; At least based on the node accuracy, obtain the evaluation results of the task planning sequence to be evaluated, including: obtaining the evaluation results of the task planning sequence to be evaluated based on the node accuracy and the completion rate of key states.

[0006] In some embodiments, each of the plurality of nodes further includes action parameters; a first prompt message is used to prompt the first multimodal large language model to identify whether the nodes in the first node list match the nodes in the second node list according to the following rules: if the nth node in the first node list and the mth node in the second node list respectively include the same action function name, the semantics corresponding to the name of the operated object are of the same category or the same, and the semantics of the action parameters are of the same category or the same, then the nth node and the mth node are determined to match; n = m or n ≠ m.

[0007] In some embodiments, the first prompt word further includes a second prompt message; the second prompt message is used to prompt the first multimodal large language model to perform the following tasks: based on the input directed acyclic graph, predict the states to be reached by performing multiple actions in the directed acyclic graph, and obtain a key state set including multiple key states; based on the input initial scene image and the directed acyclic graph, gradually simulate each of the multiple actions included in the task planning sequence to be evaluated, and obtain the updated state; determine whether the updated state hits the key state in the first key state set, and output the key state completion degree.

[0008] In some embodiments, the second prompt information is specifically used to prompt the first multimodal large language model to perform the following tasks: under the current environmental state, based on the directed acyclic graph, determine whether the preconditions of each of the multiple actions included in the task planning sequence to be evaluated have been met; the preconditions include the preconditions; the preconditions represent the actions that the current action depends on; if the preconditions are met, simulate the execution of the current action, deduce the result of the execution of the current action, and update the environmental state.

[0009] In some embodiments, obtaining a first node list based on a sample video includes: inputting the sample video into a multimodal large-scale model, obtaining a segmented description of the sample video through the multimodal large-scale model; determining a predefined normalized list; the predefined normalized list is used to indicate the data format of each node in the list; obtaining a corrected segmented description; calling a second multimodal large-scale language model, inputting the corrected segmented description, a second prompt word, and the predefined normalized list into the second multimodal large-scale language model, obtaining a first node list with the same data format as the predefined normalized list; the second prompt word is used to prompt the second multimodal large-scale language model to output a node list with the same data format as the predefined normalized list.

[0010] In some embodiments, obtaining a second node list based on the task planning sequence to be evaluated includes: calling a second multimodal large language model, inputting the task planning sequence to be evaluated, the second prompt word, and a predefined standardized list into the second multimodal large language model, and obtaining a second node list with the same data format as the predefined standardized list.

[0011] Secondly, this application also provides a task planning capability evaluation device, comprising: a first standardization module for obtaining a first node list based on sample videos; a second standardization module for obtaining a second node list based on a task planning sequence to be evaluated; the task planning sequence is the output of the target model to be evaluated and includes multiple actions; the first node list and the second node list each include multiple nodes, each of the multiple nodes including an action function name and the name of the operated object; an index acquisition module for inputting input data to the calling interface of a first multimodal large language model to obtain at least a node accuracy rate; wherein, the input data includes a first node list, a second node list, and a first prompt word; the node accuracy rate represents the matching degree between multiple nodes in the second node list and multiple nodes in the first node list; the first prompt word includes a first prompt message, which is used to prompt the first multimodal large language model to identify whether the nodes in the first node list match the nodes in the second node list; and an evaluation result output module for obtaining the evaluation result of the task planning sequence to be evaluated based at least on the node accuracy rate.

[0012] Thirdly, embodiments of this application provide an electronic device, including a processor and a memory storing a computer program, wherein the processor executes the program to implement the task planning capability evaluation method of the first aspect.

[0013] Fourthly, embodiments of this application provide a non-transitory computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor, implements the task planning capability evaluation method described in the first aspect.

[0014] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the task planning capability evaluation method described in the first aspect.

[0015] The task planning capability evaluation method, apparatus, electronic device, and program product provided in this application obtain a first node list based on sample videos as a reference standard list, and obtain a second node list based on the task planning sequence to be evaluated. The first and second node lists use standardized data formats. A multimodal large language model is used to predict the node accuracy of the second node list relative to the first node list. At least the evaluation result of the task planning sequence to be evaluated is obtained based on this node accuracy metric. Obtaining the first node list from the sample videos and the second node list from the task planning sequence to be evaluated involves converting natural language output into a standardized function structure (nodes include action function names). The model can more accurately identify the node matching degree between the two lists based on the structured node list, avoiding misjudgments caused by differences in natural language expression and the commutativity of action sequences. Therefore, compared to traditional natural language-based evaluation methods, it more accurately reflects the model's task planning capability. Furthermore, in some embodiments, while standardizing the natural language output into a function structure, a directed acyclic graph is also constructed. Combined with the initial scene image and simulation inference mechanism, this not only avoids misjudgments caused by differences in language expression and commutativity, but also verifies the feasibility of actions under physical constraints through simulation, thus more comprehensively and realistically reflecting the model's planning capabilities. In addition, during the evaluation process, key states that directly affect the achievement of the task objective are used as scoring items, highlighting the contribution of key actions. This clearly presents the differences in planning capabilities among different models, improving the evaluation's discriminatory power. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the system architecture of the task planning capability evaluation method provided in the embodiments of this application; Figure 2 This is a flowchart illustrating the task planning capability evaluation method provided in the embodiments of this application; Figure 3 This is a schematic diagram illustrating the acquisition of the first node list in some embodiments of the task planning capability evaluation method provided in this application; Figure 4 This is a flowchart illustrating the process of obtaining the first node list in some embodiments of the task planning capability evaluation method provided in this application. Figure 5This is a schematic diagram illustrating the acquisition of a second node list in some embodiments of the task planning capability evaluation method provided in this application. Figure 6 These are schematic flowcharts of some other embodiments of the task planning capability evaluation method provided in this application. Figure 7 This is a schematic diagram of the overall system processing flow in a specific embodiment of the task planning capability evaluation method provided in this application. Figure 8 This is a schematic diagram of the software module structure of the task planning capability evaluation device provided in the embodiments of this application; Figure 9 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0019] Embodied intelligence is a form of intelligence that combines cognitive computing and physical interaction, aiming to enable intelligent agents to perceive, understand, and interact efficiently with their environment in the real world. In this paradigm, intelligent agents not only need low-level precise control capabilities but also high-level cognitive abilities, such as understanding human instructions, perceiving complex scenarios, developing long-term task plans, reasoning about availability, and reflecting on and correcting failures. The combined manifestation of these capabilities determines whether an embodied intelligent agent can operate stably in dynamic, unstructured environments.

[0020] In recent years, Multimodal Large Language Models (MLLMs) have demonstrated enormous potential in the field of embodied intelligence. Leveraging their advantages in instruction understanding, common-sense reasoning, and general planning, MLLMs have gradually become the core cognitive module of embodied intelligence systems. Some solutions employ a dual-system architecture, using the MLLM as a high-level "brain" for reasoning and decision-making, combined with a low-level controller to implement specific operations. In VLA (Vision-Language-Action) models, the MLLM is often used as a core backbone for fine-tuning; in multi-agent frameworks, the MLLM acts as a high-level planner to guide the executors in completing tasks. Regardless of the specific form, task planning and task decomposition capabilities remain crucial prerequisites for ensuring the versatility and robustness of the agent. Task planning capability requires the model to reasonably decompose the overall goal into a series of executable sub-tasks based on complex natural language instructions and environmental states, and to complete them in the correct order.

[0021] However, some evaluation schemes for task planning capabilities in related technologies have the following shortcomings: For example, text similarity-based evaluation methods have biases: they use natural language generation metrics such as Bilingual Evaluation Understudy (BLEU) to evaluate planning results. However, these methods only measure the similarity between natural language expressions, making it difficult to reflect the structural rationality and physical feasibility of task planning. This can lead to logically sound but differently worded solutions being misjudged, resulting in a lack of credibility in the results.

[0022] On the other hand, multiple-choice evaluation schemes are disconnected from real-world tasks: some methods transform planning evaluation into multiple-choice questions, requiring the model to select an answer from a pre-set sequence of candidate actions. However, in real-world environments, task execution paths are often not unique, and there are no manually provided candidate options. This approach fails to reflect the model's true reasoning and planning capabilities in open environments, and there is a significant gap between this approach and the actual needs of embodied intelligence.

[0023] Furthermore, some scoring methods lack objectivity: another approach attempts to use LLM to score the model planning results. While semantic rationality can be considered to some extent, this method has the following problems: the evaluation results rely on the subjective judgment of LLM, lacking stability and consistency; when the plan and the reference answer are not exactly the same in expression but are logically reasonable, LLM often fails to accurately identify them, leading to reasonable solutions being incorrectly judged as incorrect.

[0024] In summary, the relevant technologies generally suffer from insufficient reliability, lack of objectivity, and significant discrepancies with real-world execution environments in assessing task planning capabilities. These shortcomings make it difficult to comprehensively and accurately reflect the planning and task decomposition capabilities of multimodal large language models in complex embodied tasks. These deficiencies have become key bottlenecks restricting the further development of embodied intelligence.

[0025] Therefore, there is an urgent need to propose a new evaluation mechanism that can measure the task planning capability of MLLM in a more objective, systematic and operable way under real physical and visual constraints, so as to more accurately evaluate its effectiveness.

[0026] In view of this, embodiments of this application propose a task planning capability evaluation method, apparatus, electronic device, and program product. The planning results (the task planning sequence to be evaluated) are standardized to obtain a node list with a standardized data structure. By comparing the node matching degree between the node lists, task structure alignment is ensured, effectively reducing misjudgments caused by differences in expression and the commutability of planning order. Furthermore, a directed acyclic graph is constructed simultaneously, and the execution of actions and the achievement of key states are gradually deduced from the initial scene image using a visual language model. This ensures task structure alignment while accurately measuring the physical feasibility of different planning paths by whether key states are achieved. This method not only effectively reduces misjudgments caused by differences in expression and the commutability of planning order but also highlights the evaluation value of key states. It effectively solves the problems of lack of objectivity and authenticity in long-term task planning evaluation, and the difficulty in balancing structural rationality and physical executability, significantly improving the robustness and authenticity of the evaluation.

[0027] like Figure 1 As shown, the task planning capability evaluation method provided in this application embodiment can be based on Figure 1 The system architecture shown is executed.

[0028] For example, a first node list with a standardized structure is extracted from the sample video and used as a real reference list. The task planning sequence output by the MLLM to be evaluated is standardized to obtain a second node list with the same data structure as the first node list. Furthermore, in some embodiments, a corresponding DAG is generated based on the first node list. The DAG, along with the initial scene image extracted from the sample video (usually the first frame), the first node list, the second node list, and the corresponding prompt (first prompt word), is input into another MLLM model (a model used to evaluate the MLLM to be evaluated, hereinafter referred to as the evaluation model). Through the design of the prompt word, the evaluation model is controlled to perform the tasks of predicting node accuracy and key state completion based on the above input. On the one hand, based on the matching degree of nodes between the standardized first node list and the second node list, the coverage of nodes in the second node list with nodes in the first node list can be identified, thereby outputting the node accuracy. The higher the node accuracy, the higher the evaluation score. On the other hand, optionally, based on the initial scene image and the information provided by the DAG, the various actions in the task planning sequence are simulated in the real physical environment to perform action deduction and state verification, verify the physical feasibility of the actions in the task planning sequence, and determine whether the key states in the key state set have been reached based on the state obtained by deducing the task planning sequence. The higher the completion rate of the key states, the higher the evaluation score.

[0029] It should be noted that both the large model to be evaluated and the evaluation model can be multimodal large language models, capable of simultaneously processing multimodal information such as vision and language. The MLLM to be evaluated is used for task planning and reasoning in embodied intelligence, while the evaluation model is used to evaluate the MLLM to be evaluated. The two models are different models. To avoid confusion, the evaluation model is defined as the first multimodal large language model, or simply the first MLLM below.

[0030] To help understand the methods proposed in the embodiments of this application, the technical terms that may be involved are explained: A Directed Acyclic Graph (DAG) is a structured representation of task decomposition, consisting of vertices and edges. Vertices represent actions, and edges connect different vertices. An edge indicates a dependency relationship between the two actions represented by the two vertices it connects. A DAG can graphically describe the order of actions, parallel relationships between actions, serial relationships, and other relationships.

[0031] Critical State (Milestone): An object-state change that needs to be achieved during task execution, such as "the drawer is open" or "the object has been placed on the table".

[0032] World-simulator Rollout: This uses a multimodal large language model as a simulator, such as a visual language model. Combining initial scene images and a Directed Acyclic Graph (DAG), it progressively deduces the feasibility of actions and the resulting state, determining whether critical states have been achieved. The multimodal large language model used as the simulator here can be the same model as the first multimodal large language model (MLLM) mentioned above, or it can be a different model. Using the same model reduces the cost and resource overhead of deploying different models.

[0033] Figure 2 This is one of the flowcharts illustrating the task planning capability evaluation method provided in this application. (Refer to...) Figure 2 This application provides a method for evaluating task planning capabilities, which may include: Step 101: Obtain the first node list based on the sample video.

[0034] A multimodal large model can be used to segment and describe the sample video, followed by correction. Correction can be achieved through model correction or manual annotation, resulting in high-quality standard reference data. Specifically, combined with... Figure 3 and Figure 4 As shown, the first node list can be extracted from the sample video in the following way: Step 401: Input the sample video into the multimodal large model, and obtain the segmented description of the sample video through the multimodal large model.

[0035] The multimodal large model can be a multimodal large model that supports video modal input, such as the large model of the Gemini series or the large model of the GPT-4o series.

[0036] Step 402: Determine the predefined standardization list.

[0037] A predefined standardized list is used to indicate the data format of each node in the list. For example, the data format of a node includes the action function name, the name of the object being operated on, and may also include action parameters.

[0038] Step 403: Obtain the corrected segment description.

[0039] The video segment descriptions obtained in step 401 are corrected, which can be done manually by correcting and annotating them. In other embodiments, the correction can also be done by an artificial intelligence model with correction function.

[0040] Step 404: Call the second multimodal large language model, input the corrected segmented description, the second prompt word, and the predefined normalized list into the second multimodal large language model, and obtain the first node list with the same data format as the predefined normalized list.

[0041] The second prompt word is used to indicate a list of nodes with the same data format as the predefined normalized list output by the second multimodal large language model.

[0042] The second multimodal large language model (second MLLM) and the first multimodal large language model (first MLLM) can be the same model or different models.

[0043] Step 102: Obtain the second node list based on the task planning sequence to be evaluated.

[0044] The task planning sequence is the output of the target model to be evaluated. The task planning sequence can be understood as an action sequence, including descriptions of multiple actions in natural language.

[0045] The first and second node lists use the same standardized data format, each including multiple nodes. For example, the standardized data format could be that each of the multiple nodes includes the action function name and the name of the object being operated on.

[0046] For example, combining Figure 5 As shown, the second node list is obtained in the following way: The second multimodal large language model (second MLLM) is invoked. The task planning sequence to be evaluated, the second prompt word, and the predefined normalized list are input into the second multimodal large language model to obtain a second node list with the same data format as the predefined normalized list.

[0047] Step 103: Input the input data into the calling interface of the first multimodal large language model to obtain at least the node accuracy.

[0048] Node accuracy refers to the degree of matching between multiple nodes in the second node list and multiple nodes in the first node list. For example, if the first node list contains 5 nodes and the second node list contains 4 nodes, and after the first MLLM identification, 3 of the 4 nodes match 3 of the 5 nodes in the first node list, then the node accuracy can be 3 / 5 = 0.6 or other values ​​obtained by linearly mapping 3 / 5, such as dividing 3 / 5 by 10, multiplying by 10, or multiplying by 100.

[0049] Combination Figure 1 As shown, the input data includes a first node list, a second node list, and a first prompt word. The first prompt word includes first prompt information, which is used to prompt the first multimodal large language model to identify whether the nodes in the first node list match the nodes in the second node list.

[0050] Step 104: Obtain the evaluation results of the task planning sequence to be evaluated, based at least on the node accuracy rate.

[0051] In some embodiments, node accuracy can be used as the evaluation metric alone. The resulting evaluation score could be 3 / 5 = 0.6, or other scores obtained by linearly mapping 0.6, such as 6 or 60. In other embodiments, the critical state completion rate is also used for evaluation; that is, the evaluation result is obtained based on both node accuracy and critical state completion rate, for example, by weighted summation of the two metrics.

[0052] For example, such as Figure 6 As shown, in some other embodiments, the method proposed in this application may include the following process: Step 601: Obtain the first node list based on the sample video.

[0053] Step 602: Obtain the second node list based on the task planning sequence to be evaluated.

[0054] Step 603: Obtain the directed acyclic graph based on the first node list.

[0055] A directed acyclic graph consists of vertices and edges; vertices represent actions, and edges are used to connect different vertices. An edge indicates that there is a dependency between two actions represented by two connected vertices.

[0056] Step 604: Extract the first frame image or a preset frame image from the sample video as the initial scene image.

[0057] For example, the image of the preset frame can be any one of the first 5 frames.

[0058] Step 605: Input the input data into the multimodal large language model's API to obtain node accuracy and key state completion rate.

[0059] In this embodiment, the input data also includes a directed acyclic graph and an initial scene image.

[0060] Here, the critical state completion rate represents the degree of completion of the critical states obtained based on the task planning sequence to be evaluated relative to the critical states in the critical state set.

[0061] A critical state refers to the state of an object after a change in its spatial position and / or structure. For example, objects that may appear in a video can be divided into four categories: rigid bodies, articulated objects, flexible objects, and liquids. The state of these four categories of objects after a change in their spatial position and / or structure is considered a critical state. For example, if the strawberry's position changes from the table to the plate, then "the strawberry is in the plate" is considered a critical state, while the state of the strawberry during its movement (before it reaches the plate) is not a critical state.

[0062] Rigid bodies are objects whose structures do not deform during movement. Hinged objects include doors, microwave ovens, and other objects that move along an axis. Flexible objects include clothing, whose state can change, such as being folded into several folds or not folded at all. Changes in the state of a liquid can include whether water or other liquids have been added, changes in water level, and so on.

[0063] The set of critical states is derived from a directed acyclic graph and includes multiple critical states that need to be achieved. In other words, the critical states that need to be achieved are obtained from sample videos as a reference standard.

[0064] Step 606: Based on the node accuracy and critical state completion rate, obtain the evaluation results of the task planning sequence to be evaluated.

[0065] Figure 6The illustrated embodiment standardizes the natural language output into a function structure while constructing a directed acyclic graph. Combined with the initial scene image and simulation inference mechanism, this not only avoids misjudgments caused by differences in language expression and commutativity, but also verifies the feasibility of actions under physical constraints through simulation, thus more comprehensively and realistically reflecting the model's planning capabilities. During the evaluation process, key states that directly affect the achievement of the task objective are used as scoring items, highlighting the contribution of key actions and clearly demonstrating the differences in planning capabilities among different models.

[0066] Below, we will provide some specific examples based on real-world application scenarios.

[0067] refer to Figure 7 As shown, in one specific embodiment, the initial scene image is as follows: a toy kitchen stove with plastic vegetables, a green plate, and a small pot on the stove. It should be noted that... Figure 7 In the first node list, there are several nodes as follows: Move to carrot: move_to(none, carrot); where none indicates that the object being operated on is empty; the same applies to none in other actions, so it will not be described again. Pick up a carrot: pick_up(carrot); Place carrots onto a plate: place(carrot, plate); Move to strawberry: move_to(none, strawberry); Pick up a strawberry: pick_up(strawberry); Place strawberries on a plate: place(strawberry, plate).

[0068] The resulting DAG consists of two parallel branches: move_to(none, carrot) → pick up(carrot) → place(carrot, plate); move_to(none, strawberry) → pick up(strawberry) → place(strawberry, plate).

[0069] The second node list obtained from the task planning sequence to be evaluated includes the following nodes: (1) move_to(none, strawberry), (2) ick_up(strawberry), (3) place(strawberry, plate), (4) place(carrot, plate).

[0070] Evaluation model (i.e.) Figure 7 The evaluation shown is visually constrained. Figure 7 The process involves multiple steps, including Visual Constraints, Critical Object States, State Dependency Construction, Rollout Simulation, and Scores Calculation. It identifies two critical states that need to be achieved: "carrot in plate" and "strawberry in plate." Based on the Directed Acyclic Graph (DAG), these two critical states are identified as parallel and independent. Executing action (1) results in no state change, as does executing action (2). Executing action (3) achieves the critical state "strawberry in plate," but executing action (4) does not achieve this critical state. Thus, the task planning sequence is simulated and verified, identifying one critical state and two required critical states, resulting in a critical state completion rate of 1 / 2 × 10 = 5. The second node list has 4 nodes, and the first node list has 6 nodes. Four nodes in the second node list match four nodes in the first node list, resulting in a node accuracy rate of 4 / 6 × 10 = 6.

[0071] in, Figure 7 In this context, "DAG is parallel" indicates that the DAG graph has parallel relationships; "are independent" means that there are no dependencies between states; "no state change" means that there is no state change; and "precondition not met" means that the critical state has not been reached. "Achieved states" indicates achieved states, and "Matched nodes" indicates the number of matched nodes. "Num of nodes" indicates the total number of nodes in the first node list.

[0072] Specifically, in this embodiment, a long-term mission planning and evaluation method based on world simulation is proposed, comprising the following five stages: Phase 1: Standardization of task planning sequence.

[0073] First, obtain the first node list as the reference standard. The node list consists of multiple nodes, each using a fixed and standardized data structure.

[0074] Specifically, for example, the sample video is segmented and described by the Gemini model, and then manually corrected and annotated to form a high-quality standard reference.

[0075] Building upon this, a large language model (such as a second MLLM) is invoked to standardize the output. Using prompts [Prompt - predefined action list] and [Prompt - action extraction and standardization], the natural language expressions in the corrected video segment descriptions are transformed into a unified functional action sequence, such as: In the video segment description, "pick up a tomato" is converted to "pick_up(tomato)"; where pick_up is the action function name and tomato is the name of the object being operated on.

[0076] Similarly, "place on the table" is converted to place(tomato, table), where place is the action function name, tomato is the name of the object being operated on, and table is the action parameter, indicating that the destination of the place action is table.

[0077] Similarly, the task planning sequence output by the MLLM to be evaluated is standardized to obtain the second node list.

[0078] The output of this stage is a standardized list of first nodes L1 and second nodes L2, which removes language style and format differences, making it easier for subsequent automatic evaluation.

[0079]

Prompt - Predefined Action List

Prompt - Action Extraction and Standardization

Prompt - Predefined Action List

Prompt - Predefined Action List

Prompt - Action Extraction and Standardization

[0080] For example, a specific example of

Prompt - predefined action list

[0081] instruction: 1. Extract key information: Identify the task target from the prompt field and assign it to the "task_summary" field.

[0082] Extract action functions from the previous_step and response fields to construct the necessary sequence of steps in "plan_step".

[0083] 2. Strict action function format: Use only the predefined action functions listed below. Do not modify function names or introduce new functions.

[0084] Ensure that all extracted function names exactly match the provided function names.

[0085] The parameters (object, target_object, carry_object, direction) should be generalized based on the input information, but at the same time faithful to the task.

[0086] 3. Maintain execution order: The "plan_step" list should strictly follow the order in which the robot should execute the steps.

[0087] 4. Determine the action format based on the input: Single-arm task: If the action in the response field does not contain the left: or right: prefix, the action is extracted in a single-arm format (e.g., move_to(object, target)).

[0088] Two-armed task: If the action in the response field contains the prefix left: or right:, the action is extracted in a two-armed format (e.g., left:move_to(object, target), right:no_ops).

[0089] Automatic detection: Analyze the input to determine whether it is a single-arm or dual-arm task, and format the output accordingly.

[0090] 5. Assumptions are strictly prohibited: Extract only actions that are explicitly present in the input.

[0091] Do not add missing steps based on assumptions.

[0092] Do not reinterpret or correct potentially incorrect actions or parameters—keep the input as is.

[0093] 6. Reasoning Explanation: Provide a "reason" field explaining how "task_summary" and "plan_step" are derived, including how you determine the format (single-arm vs. double-arm).

[0094] Predefined action functions """ + PREDEFINED_ACTIONS + """ Output format (JSON) Your response must be strictly formatted as a single-line JSON object: For single-arm tasks: json {{ "task_summary": "<task objective>", "plan_step": ["<action function_1>", "<action function_2>", ...], "reason": "<your reasoning>" }} For bi-arm tasks: json {{ "task_summary": "<task objective>", "plan_step": [ "<Action Function_1>", left:<action function_2>, right:<action function_3>", ... ], "reason": "<your reasoning>" }} Example Example 1: Single-arm task enter: json {{ "prompt": "With the goal of <placing a roll of toilet paper on the stand>, and having completed some steps, what is the next step?" "previous_step": "1-move_to(toilet_paper)", "response": "From the image sequence, it appears a robotic system is placing a toilet paper roll onto its holder. Here's what needs to be done next: \\n\\n1. Align the toilet paper roll with the holder's lever. \\n2. Insert the lever into the toilet paper roll. \\n3. Attach the lever and toilet paper roll to the holder mechanism. \\n4. Setup complete." }} Expected output: json {{ "task_summary": "<Place a roll of toilet paper on the stand>", "plan_step": ["move_to(toilet_paper)", "place(toilet_paper,holder)"], "reason": "Based on the absence of a left / right prefix in the response format, it was detected as a single-arm task. The first step, 'move_to(toilet_paper)', comes from the 'previous_step' field, and the second step, 'place(toilet_paper,holder)', is derived by summarizing the remaining steps in the 'prompt' and 'response' fields." }} Example 2: Two-arm task enter: json {{ "prompt": "With the goal of cooking shrimp in a pan and serving them in a bowl, and having completed some steps, what is the next step?" "previous_step": "1-observation(shrimp)", "response": "To plan the remaining steps to achieve the goal of 'cooking shrimp in a pan and serving it in a bowl,' we will organize the steps required by the robot...\\n\\n1-left:move_to(none, shrimp), right:no_ops\\n2-left:pick_up(shrimp), right:no_ops\\n3-left:no_ops, right:move_to(none, pan)\\n4-left:no_ops, right:turn_on(stove)\\n5-left:no_ops, right:pour(oil, pan)\\n6-left:no_ops, right:move_to(shrimp, pan)\\n7-left:no_ops,right:scoop(shrimp, pan)\\n8-left:no_ops, right:move_to(pan, bowl)\\n9-left:no_ops, right:pour(shrimp, bowl)" }} Expected output: json {{ "task_summary": "<Cook shrimp in a pot and serve in a bowl>", "plan_step": [ "observation(shrimp)", "left:move_to(none, shrimp), right:no_ops", "left:pick_up(shrimp), right:no_ops", "left:no_ops, right:move_to(none, pan)", "left:no_ops, right:turn_on(stove)", "left:no_ops, right:pour(oil, pan)", "left:no_ops, right:move_to(shrimp, pan)", "left:no_ops, right:scoop(shrimp, pan)", "left:no_ops, right:move_to(pan, bowl)", "left:no_ops, right:pour(shrimp, bowl)" ], "reason": "Detected as a two-armed task based on the left / right prefix in the response format. The 'task_summary' field is extracted from the prompt. 'plan_step' follows the order in 'previous_step' and 'response'. 'observation' steps are included as independent actions, and all actions use the 'left:' and 'right:' format. No missing steps are assumed or inferred." }} The data I provided is as follows: {data} Please output your results as required.

[0095] Final Directive • Automatic task type detection: Analyze the input to determine whether it is a single-arm or dual-arm task and format it accordingly.

[0096] • Do not modify the function name.

[0097] • Do not add missing steps in addition to the information entered.

[0098] • Ensure that the correct function is selected based on the context.

[0099] • Ensure the "plan_step" field is strictly sorted.

[0100] • For two-arm tasks: Unless it's an explicit global action, all actions should be formatted using `left:` and `right:`. If one arm doesn't perform any action in a step, use `no_ops`.

[0101] • Independent movement and observation actions in two-arm tasks must not have the prefix left: or right:.

[0102] • Outputs a single-line JSON object without any additional content.

[0103] • Do not reinterpret or correct potentially incorrect actions or parameters—keep the input as is. For example, a specific example of

Prompt - Action Extraction and Standardization

[0104] instruction: 1. Extract key information: Identify the task target from the prompt field and assign it to the "task_summary" field.

[0105] Extract action functions from the previous_step and response fields to construct the necessary sequence of steps in "plan_step".

[0106] 2. Strict action function format: Use only the predefined action functions listed below. Do not modify function names or introduce new functions.

[0107] Ensure that all extracted function names exactly match the provided function names.

[0108] The parameters (object, target_object, carry_object, direction) should be generalized based on the input information, but at the same time faithful to the task.

[0109] 3. Maintain execution order: The "plan_step" list should strictly follow the order in which the robot should execute the steps.

[0110] 4. Determine the action format based on the input: Single-arm task: If the action in the response field does not contain the left: or right: prefix, the action is extracted in a single-arm format (e.g., move_to(object, target)).

[0111] Two-armed task: If the action in the response field contains the prefix left: or right:, the action is extracted in a two-armed format (e.g., left:move_to(object, target), right:no_ops).

[0112] Automatic detection: Analyze the input to determine whether it is a single-arm or dual-arm task, and format the output accordingly.

[0113] 5. Assumptions are strictly prohibited: Extract only actions that are explicitly present in the input.

[0114] Do not add missing steps based on assumptions.

[0115] Do not reinterpret or correct potentially incorrect actions or parameters—keep the input as is.

[0116] 6. Reasoning Explanation: Provide a "reason" field explaining how "task_summary" and "plan_step" are derived, including how you determine the format (single-arm vs. double-arm).

[0117] Predefined action functions """ + PREDEFINED_ACTIONS + """ Output format (JSON) Your response must be strictly formatted as a single-line JSON object: For single-arm tasks: json {{ "task_summary": "<task objective>", "plan_step": ["<action function_1>", "<action function_2>", ...], "reason": "<your reasoning>" }} For bi-arm tasks: json {{ "task_summary": "<task objective>", "plan_step": [ "<Action Function_1>", left:<action function_2>, right:<action function_3>", ... ], "reason": "<your reasoning>" }} Example Example 1: Single-arm task enter: json {{ "prompt": "With the goal of <placing a roll of toilet paper on the stand>, and having completed some steps, what is the next step?" "previous_step": "1-move_to(toilet_paper)", "response": "From the image sequence, it appears a robotic system is placing a toilet paper roll onto its holder. Here's what needs to be done next: \\n\\n1. Align the toilet paper roll with the holder's lever. \\n2. Insert the lever into the toilet paper roll. \\n3. Attach the lever and toilet paper roll to the holder mechanism. \\n4. Setup complete." }} Expected output: json {{ "task_summary": "<Place a roll of toilet paper on the stand>", "plan_step": ["move_to(toilet_paper)", "place(toilet_paper,holder)"], "reason": "Based on the absence of a left / right prefix in the response format, it was detected as a single-arm task. The first step, 'move_to(toilet_paper)', comes from the 'previous_step' field, and the second step, 'place(toilet_paper,holder)', is derived by summarizing the remaining steps in the 'prompt' and 'response' fields." }} Example 2: Two-arm task enter: json {{ "prompt": "With the goal of cooking shrimp in a pan and serving them in a bowl, and having completed some steps, what is the next step?" "previous_step": "1-observation(shrimp)", "response": "To plan the remaining steps to achieve the goal of 'cooking shrimp in a pan and serving it in a bowl,' we will organize the steps required by the robot...\\n\\n1-left:move_to(none, shrimp), right:no_ops\\n2-left:pick_up(shrimp), right:no_ops\\n3-left:no_ops, right:move_to(none, pan)\\n4-left:no_ops, right:turn_on(stove)\\n5-left:no_ops, right:pour(oil, pan)\\n6-left:no_ops, right:move_to(shrimp, pan)\\n7-left:no_ops,right:scoop(shrimp, pan)\\n8-left:no_ops, right:move_to(pan, bowl)\\n9-left:no_ops, right:pour(shrimp, bowl)" }} Expected output: json {{ "task_summary": "<Cook shrimp in a pot and serve in a bowl>", "plan_step": [ "observation(shrimp)", "left:move_to(none, shrimp), right:no_ops", "left:pick_up(shrimp), right:no_ops", "left:no_ops, right:move_to(none, pan)", "left:no_ops, right:turn_on(stove)", "left:no_ops, right:pour(oil, pan)", "left:no_ops, right:move_to(shrimp, pan)", "left:no_ops, right:scoop(shrimp, pan)", "left:no_ops, right:move_to(pan, bowl)", "left:no_ops, right:pour(shrimp, bowl)" ], "reason": "Detected as a two-armed task based on the left / right prefix in the response format. The 'task_summary' field is extracted from the prompt. 'plan_step' follows the order in 'previous_step' and 'response'. 'observation' steps are included as independent actions, and all actions use the 'left:' and 'right:' format. No missing steps are assumed or inferred." }} The data I provided is as follows: {data} Please output your results as required.

[0118] Final Directive • Automatic task type detection: Analyze the input to determine whether it is a single-arm or dual-arm task and format it accordingly.

[0119] • Do not modify the function name.

[0120] • Do not add missing steps in addition to the information entered.

[0121] • Ensure that the correct function is selected based on the context.

[0122] • Ensure the "plan_step" field is strictly sorted.

[0123] • For two-arm tasks: Unless it's an explicit global action, all actions should be formatted using `left:` and `right:`. If one arm doesn't perform any action in a step, use `no_ops`.

[0124] • Independent movement and observation actions in two-arm tasks must not have the prefix left: or right:.

[0125] • Outputs a single-line JSON object without any additional content.

[0126] • Do not reinterpret or correct potentially incorrect actions or parameters—keep the input as is. Phase 2: Construction of Directed Acyclic Graph (DAG).

[0127] Based on the first node list L1 obtained in Phase 1, the large language model is invoked, for example, the third MLLM. The third MLLM can be the same model as the first MLLM or a different model. Using the prompt word "Prompt-DAG construction", the sequential and dependency relationships between actions are analyzed, and a reference standard directed acyclic graph G is constructed. gt .

[0128] Directed acyclic graph G gt In the diagram, nodes represent action functions; edges represent the dependencies between actions. For example, open(drawer) → pick_up(towel), the towel is in the drawer, and the action pick_up(towel) (picking up the towel) can only be executed after the action open(drawer) (opening the drawer) is completed.

[0129] A directed acyclic graph (DAG) fully records the decomposition structure of the task logic.

[0130] Step 3: Key State Extraction and Alignment Based on the first and second node lists, the large language model is invoked, for example, the fourth MLLM. The fourth MLLM can be the same model as the first MLLM or a different model. Using the prompt word "Prompt-DAG construction," the model is derived from the reference task graph. The system identifies actions and determines the key target states that the task should achieve (such as a drawer being opened or a towel being placed on a table), thereby generating a set of key states K.

[0131] For example, if the reference plan includes the actions open(drawer) and place(towel, table), then the critical states drawer=open and towel=on(table) will be stored in the critical state set K.

[0132] The final set of critical states K is the critical state set used as a reference standard and is used as the core criterion for task completion. In subsequent evaluations, it is used to determine whether the task planning sequence output by the first MLLM has achieved the goal.

[0133] For example, a specific example of the prompt word "Prompt-DAG construction" is as follows: "Hint words - Directed Acyclic Graph Construction:" Task Description: You will receive a structured robot task description, including: Task Summary: Overall Objective Step Sequence of the Robot Task: Describe in sequence the list of standard action functions to be performed by the robot task. Your task is to construct a directed acyclic graph that accurately represents the dependencies between these steps, while ensuring that each step is included and that there are no cycles.

[0134] Operating Instructions Extract and construct the directed acyclic graph: The action function in each step sequence should be used as the key of the directed acyclic graph as is. If a certain action function appears multiple times, add a numeric suffix according to the execution order. Example: If "move to (sweater)" appears twice, then it is marked as "move to (sweater)" in the directed acyclic graph. <1> "and" moves to (the sweater). <2> " Each key must include a "follow-up steps" field, which lists its direct follow-up steps.

[0135] To ensure the validity of a directed acyclic graph (DAG): all actions in the sequence of steps must be included as keys in the DAG. Circularity should not be formed (i.e., a node should not reference itself or any of its ancestors through "subsequent steps"). The first occurrence of an action should not depend on its subsequent occurrence. Dependency handling: Identify direct dependencies between steps based on execution order. Ensure that each step lists only the direct follow-up steps in the "Follow-up Steps" section. If a step has no direct follow-up step, "follow-up steps" should be an empty array. Numbering of repetitive actions: If a function appears multiple times, add a suffix according to its execution order. Numbering consistency should be maintained in the "Subsequent Steps" references. Output format: The output must be a JSON object with the following structure: json { "Task Summary": "<Task Objective>", "Directed Acyclic Graph": { "<Step 1>": { "Subsequent Steps": ["<Step 2>", "<Step 3>"]}, "<Step 2>": { "Subsequent Steps": ["<Step 4>"]}, "<Step 4>": { "Subsequent Steps": []} } } Ensure that subsequent steps follow the execution order given in the step sequence. Predefined action functions [Insert predefined action function here] Example enter json { Task Summary: "<I can't see the entire surface of the table. Can you fix this?>" "Step sequence": [ "Move to (sweater)", "Pick up (the sweater)", "Move to (sweater)", "Place (the sweater)" ] } Expected output json { Task Summary: "<I can't see the entire surface of the table. Can you fix this?>" "Directed Acyclic Graph": { Move to (sweater) <1> ": { "Next steps": ["Place (the sweater)"] }, "Pick up (the sweater)": { "Next steps": ["Move to (sweater)" <2> "] }, Move to (sweater) <2> ": { "Next steps": ["Place (the sweater)"] }, "Place (sweater)": { "Next steps": [] } } } The data I provided is as follows: json [Insert the provided data here] Please output the results as required.

[0136] Final Guide Strictly maintain the order of the step sequence to ensure that all step sequence elements are contained within the directed acyclic graph (DAG). The DAG should not contain cycles. If actions are repeated, assign an identifier number that matches the execution order. Phase Four: World Simulation and State Verification.

[0137] In this embodiment, a Visual-Language Model (VLM) can be selected as the evaluation model, and the input includes: an initial scene image. Reference Directed Acyclic Graph (DAG) The set of critical states, K.

[0138] Using the prompts "[Prompt - Key State Extraction]" and "[Prompt - Action Deduction and State Verification]", the action sequence predicted by the model is simulated step by step, and state verification is performed. The specific steps include the following: Step 701: Prerequisite check.

[0139] Judge the action a t Preconditions (pre) a t Is it in the current state? S { t-1} The following conditions must be met. Furthermore, based on the dependency analysis of the DAG, if the upstream state is not met, subsequent actions that depend on that state will be blocked.

[0140] It's important to note that the relationships between different actions in a Directed Acyclic Graph (DAG) include sequential dependencies and parallel relationships. A sequential dependency means that the execution of a next action depends on the execution of a previous action. For example, the action "put the apple on the plate" depends on the action "pick up the apple." These two actions have a sequential dependency. A parallel relationship means that achieving a critical state requires two or more actions to be performed in parallel. For example, the critical state "both the apple and the banana are on the plate" requires two parallel actions: putting the apple on the plate and putting the banana on the plate.

[0141] The evaluation model infers the state reached after each action based on the prompts. For example, when the action "open the drawer" is performed, the drawer's state changes from closed to open. The state is updated after each simulated action. Based on the serial or parallel dependencies between actions represented by the points in the input Directed Acyclic Graph (DAG), the dependencies between states can be derived. When simulating the execution of each action step by step, it is necessary to determine whether the preconditions for the current action have been met. Preconditions can be understood as the states that need to be achieved to execute the current action. For example, if a towel is in a drawer, the action of picking up the towel can only be performed if the drawer is already open. Therefore, when simulating the action of picking up the towel, it is necessary to first determine whether the state of "drawer is open" has been achieved. The state of "drawer is open" is achieved after performing the action "open the drawer." The determination of preconditions can be achieved by first inferring the action. a t Corresponding state S t Then, based on the dependencies between states, find the state. S t If the dependent state has been achieved, then the precondition is considered satisfied. The precondition check can also be combined with the action. a tThe judgment is aided by whether the previous action has been executed, with the main criterion being whether the state has been achieved.

[0142] Step 702: Execution effect prediction.

[0143] If the conditions are met, then the deduced action result is post( a t ), and update the environment status. S t .

[0144] Step 703, Critical State Detection.

[0145] After the state is updated, check whether the critical state in K (such as drawer=open) is triggered.

[0146] This process allows us to obtain the execution trajectory of the action sequence and the achievement of key states, thereby verifying the feasibility of model planning under physical and logical constraints.

[0147] For example, a specific example of

Prompt - Key State Extraction

[0148] You will receive the following input information: • The first image of the scene, containing initial conditions and visual constraints.

[0149] • Real-world motion list: Generated by Gemini through segmented descriptions of static videos and manually optimized.

[0150] • Realistic Directed Acyclic Graph: Manually annotated from the video, including the order and parallel relationships between subtasks.

[0151] The content you need to extract is: Construct a set of K critical states, which are the key object state milestones that must be achieved to complete the task. Focus on the state changes of the objects themselves, rather than the movements of the robotic arm.

[0152] These rules should be followed when identifying critical states: • Only calculate the changes in the attributes of the manipulated object, including posture, support relationship, containment relationship, orientation angle, open state, active state, assembly integrity, etc.

[0153] • Purely robotic actions are not included, such as moving to, approaching, aligning, pre-grabbing, grasping, holding, releasing, retracting, viewing, scanning, planning, and thinking.

[0154] • Derive the state from actual actions: Actions such as opening, closing, pulling, and pushing drawer doors will change the corresponding drawer door status to "open" or "closed".

[0155] Place an object into the container area corresponding to the object being in the container or on the surface.

[0156] Turning the knob to a specific angle changes the faucet's angle state, such as turning it on or off.

[0157] Picking up an object is only considered a change of state when the support relationship changes from the surface to the gripper.

[0158] • Use a true directed acyclic graph to assign dependencies, distinguishing between sequential and parallel relationships. Parallel branches are independent of each other, while sequential branches must adhere to prerequisites.

[0159] • Use a standardized format for describing statuses, such as drawer = open, towel = on the table, apple = in the bowl, faucet = on.

[0160] The output should be in JSON format and include the following: A list of critical L2 states, each containing a state number, state description, dependent preceding state numbers, whether the execution mode is sequential or parallel, and corresponding evidence of actual actions. Finally, a brief description of the directed acyclic graph (DAG) is required, illustrating the parallel and sequential relationship chains derived from the graph. For example, a specific example of

Prompt - Action Deduction and State Verification

[0161] The input you will receive: The first image of the scene (containing initial conditions and constraints); Real Directed Acyclic Graph (Text + Optional Image): Manually labeled, encoding sequential and parallel relationships. K-Critical State Set: The set of critical states generated in the "Critical State Extraction" step (including ID, state predicate, dependency, and threshold). List of planning actions for the model to be evaluated: Planning actions that need to be evaluated (generated based on the same scene image and question); Predefined action sets: A standardized catalog of skills you must use for standardized processing; Standardization process: Standardize skills: Use lowercase letters; remove spaces, underscores, and hyphens (e.g., turn_on / turn-on / turn on → turnon). Do not infer new skills.

[0162] Parameter normalization and flexible equivalence (standard mode only): minor deviations are allowed if the function of the parameter remains unchanged in the scene; strict matching is required in CSS mode.

[0163] Object matching: If objects have the same function in the scene, use flexible equivalence, allowing aliases or part-whole relationships (e.g., opening drawer_handle can be considered as a drawer). Provide an object matching table.

[0164] Evaluation dimensions (0-10 points for each item, rounded down to the lower integer).

[0165] Node correctness: Treat each (skill, object, parameter) as a node.

[0166] Perform a one-to-one optimal match between model nodes and real nodes.

[0167] Calculate the number of Y nodes matched among X real nodes, with flexible equivalence allowed only in standard mode.

[0168] Score = floor((Y / X)10); also calculate (Y10) / / X and ensure that the two are equal.

[0169] Task completion rate (based on simulation): Using the first image, a real directed acyclic graph, and a set of K key states, a rigorous sequential deduction of the model planning is performed.

[0170] Enforce prerequisites and physical realism; apply the "skill → effect" rule.

[0171] Dependency enforcement: If a prerequisite critical state in K is not achieved, then any subsequent state that depends on that state cannot be achieved later (based on the processing order and parallel relationships of the directed acyclic graph).

[0172] Track the state changes after each step; only the states actually achieved during the simulation are included in Y.

[0173] Score = floor((Y / X)10); also calculate (Y10) / / X and ensure that the two are equal.

[0174] Inference Engine: A. Object State Machine: Tracks the attitude / support / contact, orientation / angle, open state, containment / attachment, active state, and integrity / assembly state of each object.

[0175] B. Skill → Effect Mapping Matrix.

[0176] C. Visual and physical constraints: accessibility / occlusion restrictions; orientation correctness; obstacles can interrupt movement; application of gravity and stability rules.

[0177] D. Dependency enforcement in directed acyclic graphs: sequential dependencies must be satisfied in sequence; parallel branches are independent of each other.

[0178] E. Deduction Log (Required): Record the original and normalized actions, the pass / fail of the preconditions and the reasons, the state changes (before → after), the relevant numerical opening ratio / angle, and the newly satisfied L2 state for each step.

[0179] Output format (JSON only; includes detailed reasoning process) json { "Node correctness": { "Inference Process": "True Nodes: [...]. Model Nodes: [...]. Object Matching Table: [...]. One-to-One Node Matching: [...]. X, Y, Z = Y / X. Lower Score Limit (Z*10) = W; (Y*10) / / X = W_Substitution; Final Score = W.", "Result": <0-10 integer> }, "Task Completion Rate": { "Reasoning Process": "Visual Constraints: [...]. L2 States (including dependencies and thresholds): [...]. Deduction Trajectory Table (prerequisites / effects / state changes at each step): [...]. Reachable States: [...]. X, Y, Z = Y / X. Lower Score Limit (Z*10) = W; (Y*10) / / X = W_substitution; Final Score = W.", "Result": <0-10 integer> }, "Comprehensive Assessment": "Overall assessment + actionable improvement recommendations." } " It should be noted that the above prompt examples are merely examples. The above prompts can be split and combined to obtain other specific implementations of prompt words. Any other prompt with equivalent semantics or function that is generated by appropriately splitting or combining, changing the order of description, or changing the description method based on the above examples should fall within the scope of protection of this application.

[0180] Phase 5: Output of indicators and evaluation results.

[0181] Based on the results of Phase 3 and Phase 4, the following two types of indicators were calculated: Node_Correctness: Measures whether the actions output by the model cover the key nodes of a standard DAG. Where (X) represents the number of reference nodes and (Y) represents the number of matches.

[0182] Critical_Object_Completion: Measures the achievement of critical states during the simulation process. Where (X) represents the total number of critical states and (Y) represents the number of states achieved.

[0183] The final overall score is: Overall = 0.5 × Node_Correctness + 0.5 × Critical_Object_Completion This scoring system examines not only the rationality of the planning structure, but also assesses its feasibility and the degree of goal achievement in a real-world environment.

[0184] The evaluation method proposed in this application encompasses the standardization of task output, the structured representation of reference actions, the extraction and alignment of the key state set K, model prediction verification based on world simulation, and the final index scoring, forming a complete evaluation closed loop. All processes involving the conversion from natural language to functional representation, action dependency identification, and state deduction are implemented using a multimodal large language model. In practical applications, it can be further refined and expanded to meet different task requirements.

[0185] Specifically, the task planning capability evaluation method proposed in this application has the following technical advantages: The evaluation results are more accurate and objective: By standardizing natural language output into a function structure and constructing a directed acyclic graph, combined with initial scene images and world simulation inference mechanisms, this method can not only avoid misjudgments caused by differences in language expression and commutativity of order, but also verify whether actions are truly executable under physical constraints, thus more comprehensively and realistically reflecting the planning capabilities of the model.

[0186] The model has high discrimination ability: During the scoring process, only key actions and key states that directly affect the achievement of task objectives are considered as the main scoring items, while connecting actions such as "movement" and "grabbing" are given lower weights. This design highlights the contribution of key steps, clearly presents the differences in planning capabilities among different models, and improves the differentiation of the evaluation.

[0187] High compatibility and scalability: The method proposed in this application utilizes a large language model as a standardization and simulation tool, which can adapt to various output formats of embodied intelligence models, including natural language, code snippets, and functional action sequences, all of which can be processed uniformly. Furthermore, evaluation can be completed based on any initial scene image and reference action list, demonstrating strong scene transferability and scalability.

[0188] Highly automated, suitable for large-scale evaluation: From action extraction and structured representation to world simulation and index calculation, the entire process can be automated, significantly reducing manual intervention and annotation costs. This characteristic makes this method particularly suitable for large-scale model comparison experiments and benchmark construction.

[0189] It should be noted that, based on the solutions proposed in the embodiments of this application, more embodiments can be obtained by combining the following methods: Alternative model: In the process of motion standardization and world simulation, in addition to general large language models, special instruction understanding models, graph neural networks or rule engines can also be used to achieve more efficient structure extraction and planning parsing to meet the needs of edge device deployment or data privacy protection.

[0190] Task graph structure expansion and cross-scene migration: Current representations based on directed acyclic graphs primarily cover sequence and partial dependencies. These can be further extended to support task state machines that support conditional branching, concurrent execution, and looping, thereby evaluating complex scenarios such as multi-agent collaboration and anomaly recovery. Furthermore, by constructing cross-task libraries and transfer learning mechanisms, existing task structures can be migrated to new domains, achieving universal evaluation across various scenarios.

[0191] Interpretable enhancement: During the world simulation process, a visual explanation mechanism for action execution can be added to record the precondition verification, state update and failure reasons for each step, and generate a diagnostic report with causal links to help researchers quickly locate model weaknesses and guide improvements.

[0192] Lightweight and efficient: In large-scale evaluations or computationally constrained environments, lightweight or distilled visual language models can be used as world simulators to reduce computational overhead. Simultaneously, directed acyclic graph analysis and key state annotation can be pre-computed for benchmark tasks, allowing direct access to the reference library during the evaluation phase, reducing redundant computations and significantly improving evaluation efficiency.

[0193] Online assessment: Based on the method proposed in the embodiments of this application, the offline mode can be extended to an online mode, which can perform real-time inference and scoring during the execution of the intelligent agent, providing the system with instant feedback and dynamic monitoring capabilities, thereby supporting continuous evaluation and adaptive optimization in actual deployment.

[0194] In summary, this solution possesses excellent model substitutability, task structure scalability, result interpretability, potential for lightweight and high efficiency, and online dynamic evaluation capabilities, providing ample room for future application in more complex, larger-scale, and more realistic embodied intelligence scenarios.

[0195] The task planning capability evaluation device provided in the embodiments of this application is described below. The task planning capability evaluation device described below and the task planning capability evaluation method described above can be referred to in correspondence.

[0196] This application provides a task planning capability evaluation device. The corresponding product of this system can be an electronic device used to implement the corresponding function, such as a server or server cluster. Figure 8 As shown, the device 80 may include: The first standardization module 801 is used to obtain the first node list based on the sample video.

[0197] The second standardization module 802 is used to obtain a second node list based on the task planning sequence to be evaluated; the task planning sequence is the output of the target model to be evaluated and includes multiple actions; the first node list and the second node list each include multiple nodes, and each of the multiple nodes includes an action function name and the name of the object being operated on.

[0198] The indicator acquisition module 803 is used to input input data into the calling interface of the first multimodal large language model to obtain at least the node accuracy rate. The input data includes a first node list, a second node list, and a first prompt word. The node accuracy rate represents the matching degree between multiple nodes in the second node list and multiple nodes in the first node list. The first prompt word includes a first prompt message, which is used to prompt the first multimodal large language model to identify whether the nodes in the first node list match the nodes in the second node list.

[0199] The evaluation result output module 804 is used to obtain the evaluation results of the task planning sequence to be evaluated, based at least on the node accuracy.

[0200] Figure 9 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communication interface 920, and the memory 930 communicate with each other via the communication bus 940. The processor 910 can call a computer program in the memory 930 to execute the steps of a task planning capability assessment method, such as including: Based on sample videos, a first node list is obtained; based on the task planning sequence to be evaluated, a second node list is obtained; the task planning sequence is the output of the target model to be evaluated, including multiple actions; the first and second node lists adopt a unified standardized data format, each including multiple nodes, and each node includes an action function name and the name of the operated object; the input data is input to the calling interface of the first multimodal large language model to obtain at least the node accuracy; wherein, the input data includes the first node list, the second node list, and the first prompt word; the node accuracy represents the matching degree between multiple nodes in the second node list and multiple nodes in the first node list; the first prompt word includes the first prompt information, which is used to prompt the first multimodal large language model to identify whether the nodes in the first node list match the nodes in the second node list; based at least on the node accuracy, the evaluation result of the task planning sequence to be evaluated is obtained.

[0201] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0202] On the other hand, embodiments of this application also provide a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the steps of the task planning capability evaluation method provided in the above embodiments, such as including: Based on sample videos, a first node list is obtained; based on the task planning sequence to be evaluated, a second node list is obtained; the task planning sequence is the output of the target model to be evaluated, including multiple actions; the first and second node lists adopt a unified standardized data format, each including multiple nodes, and each node includes an action function name and the name of the operated object; the input data is input to the calling interface of the first multimodal large language model to obtain at least the node accuracy; wherein, the input data includes the first node list, the second node list, and the first prompt word; the node accuracy represents the matching degree between multiple nodes in the second node list and multiple nodes in the first node list; the first prompt word includes the first prompt information, which is used to prompt the first multimodal large language model to identify whether the nodes in the first node list match the nodes in the second node list; based at least on the node accuracy, the evaluation result of the task planning sequence to be evaluated is obtained.

[0203] On the other hand, embodiments of this application also provide a processor-readable storage medium storing a computer program for causing a processor to perform the steps of the methods provided in the above embodiments, such as including: Based on sample videos, a first node list is obtained; based on the task planning sequence to be evaluated, a second node list is obtained; the task planning sequence is the output of the target model to be evaluated, including multiple actions; the first and second node lists adopt a unified standardized data format, each including multiple nodes, and each node includes an action function name and the name of the operated object; the input data is input to the calling interface of the first multimodal large language model to obtain at least the node accuracy; wherein, the input data includes the first node list, the second node list, and the first prompt word; the node accuracy represents the matching degree between multiple nodes in the second node list and multiple nodes in the first node list; the first prompt word includes the first prompt information, which is used to prompt the first multimodal large language model to identify whether the nodes in the first node list match the nodes in the second node list; based at least on the node accuracy, the evaluation result of the task planning sequence to be evaluated is obtained.

[0204] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0205] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0206] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0207] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for evaluating task planning ability, characterized in that, The method includes: Based on the sample video, obtain the first node list; Based on the task planning sequence to be evaluated, a second node list is obtained; the task planning sequence is the output of the target model to be evaluated, including multiple actions; the first node list and the second node list adopt a unified standardized data format, each including multiple nodes, and each of the multiple nodes includes an action function name and the name of the object being operated on; Input data is fed into the API of a first multimodal large language model to obtain at least the node accuracy rate; wherein, the input data includes a first node list, a second node list, and a first prompt word; the node accuracy rate represents the matching degree between multiple nodes in the second node list and multiple nodes in the first node list; the first prompt word includes first prompt information, which is used to prompt the first multimodal large language model to identify whether the nodes in the first node list match the nodes in the second node list; Based at least on the accuracy of the nodes, the evaluation result of the task planning sequence to be evaluated is obtained.

2. The method according to claim 1, characterized in that, The input data also includes a directed acyclic graph and an initial scene image; Before inputting the input data into the API call interface of the first multimodal large language model, the method further includes: Based on the first node list, a directed acyclic graph is obtained; the directed acyclic graph includes vertices and edges; the vertices represent actions, the edges are used to connect different vertices, and the edges indicate that there is a dependency relationship between the two actions represented by the two connected vertices; Extract the first frame or a preset frame from the sample video as the initial scene image; Input data is fed into the API of the first multimodal large language model to obtain at least the node accuracy, including: Input data is fed into the API of the multimodal large language model to obtain node accuracy and key state completion rate; Wherein, the critical state completion degree represents the completion degree of the critical state obtained based on the task planning sequence to be evaluated relative to the critical states in the critical state set; the critical state represents the state of an object after its spatial position and / or structure has changed; the critical state set is obtained based on the directed acyclic graph and includes multiple critical states that need to be achieved. Based at least on the node accuracy, the evaluation result of the task planning sequence to be evaluated is obtained, including: The evaluation results of the task planning sequence to be evaluated are obtained based on the node accuracy and critical state completion rate.

3. The method according to claim 1, characterized in that, Each of the plurality of nodes also includes action parameters; The first prompt message is used to prompt the first multimodal large language model to identify whether the nodes in the first node list match the nodes in the second node list according to the following rules: If the nth node in the first node list and the mth node in the second node list are found to have the same action function name, the semantics corresponding to the name of the operated object are of the same category or the same, and the semantics of the action parameters are of the same category or the same, then the nth node and the mth node are determined to be a match; n = m or n ≠ m.

4. The method according to claim 2 or 3, characterized in that, The first prompt word also includes a second prompt message; The second prompt message is used to prompt the first multimodal large language model to perform the following task: Based on the input directed acyclic graph, predict the states to be reached by performing multiple actions in the directed acyclic graph, and obtain a set of key states including multiple key states. Based on the input initial scene image and the directed acyclic graph, each action in the multiple actions included in the task planning sequence to be evaluated is simulated step by step to obtain the updated state; Determine whether the updated state matches a key state in the first key state set, and output the key state completion rate.

5. The method according to claim 4, characterized in that, The second prompt message is specifically used to prompt the first multimodal large language model to perform the following task: In the current environment, based on the directed acyclic graph, it is determined whether the preconditions of each action in the multiple actions included in the task planning sequence to be evaluated have been met; the preconditions include pre-actions; the pre-actions represent the actions that the current action depends on. Once the preconditions are met, the current action is simulated, the result of the current action is deduced, and the environmental state is updated.

6. The method according to any one of claims 1-5, characterized in that, Based on the sample video, the first node list is obtained, including: The sample video is input into a multimodal large model, and the segmented description of the sample video is obtained through the multimodal large model; Determine a predefined normalization list; the predefined normalization list is used to indicate the data format of each node in the list; Obtain the corrected segment description; The second multimodal large language model is invoked, and the corrected segmented description, the second prompt word, and the predefined normalized list are input into the second multimodal large language model to obtain a first node list with the same data format as the predefined normalized list; the second prompt word is used to prompt the second multimodal large language model to output a node list with the same data format as the predefined normalized list.

7. The method according to claim 6, characterized in that, Based on the task planning sequence to be evaluated, obtain the second node list, including: The second multimodal large language model is invoked, and the task planning sequence to be evaluated, the second prompt word, and the predefined standardized list are input into the second multimodal large language model to obtain a second node list with the same data format as the predefined standardized list.

8. A task planning capability evaluation device, characterized in that, The device includes: The first standardization module is used to obtain the first node list based on the sample video; The second standardization module is used to obtain a second node list based on the task planning sequence to be evaluated; the task planning sequence is the output of the target model to be evaluated and includes multiple actions; the first node list and the second node list each include multiple nodes, and each of the multiple nodes includes an action function name and the name of the object being operated on; The metric acquisition module is used to input input data into the calling interface of the first multimodal large language model to obtain at least the node accuracy rate; wherein, the input data includes the first node list, the second node list, and the first prompt word; the node accuracy rate represents the matching degree between multiple nodes in the second node list and multiple nodes in the first node list; the first prompt word includes the first prompt information, which is used to prompt the first multimodal large language model to identify whether the nodes in the first node list match the nodes in the second node list; The evaluation result output module is used to obtain the evaluation result of the task planning sequence to be evaluated, based at least on the node accuracy.

9. An electronic device comprising a processor and a memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the task planning capability evaluation method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the task planning capability evaluation method according to any one of claims 1 to 7.

Images