Long-time-sequence task planning method and system based on adaptive symbol world construction

Abstract

The invention discloses a long-time-sequence task planning method and system based on adaptive symbol world construction. The method comprises the following steps: based on a pre-trained visual language model VLM, extracting an object, a state, an attribute and a relationship from an environment image, and constructing and dynamically maintaining a symbol world; generating a PDDL problem file in combination with a natural language task and a symbol world, and outputting a PDDL action sequence by the VLM under the constraint of a domain file; if the action sequence does not pass the verification, updating the sequence and the symbol world according to plan-level and environment-level feedback, repeating verification until the action sequence and the symbol world pass the verification or reach the maximum round, executing the action after the action sequence passes the verification, and recording actual feedback; and finally, abstracting symbolic execution feedback and actual feedback into structured symbolic memory, and merging the structured symbolic memory into a system prompt template to enhance the VLM reasoning capability and realize interpretable cross-task knowledge generalization.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of artificial intelligence and intelligent planning control, and particularly relates to a long-time sequence task planning method and system based on an adaptive symbolic world construction. BACKGROUND

[0002] With the rapid development of embodied intelligence and large-scale vision-language models (VLMs), task planning based on perception-language-action is gradually applied to virtual interaction, home service robots, industrial inspection, and digital twins. In order to enable agents to perform multi-step tasks according to instructions, the industry usually adopts the route of symbolic abstraction + planner: continuous visual observations and text goals are converted into verifiable and inferable symbolic representations, and then a planner generates an action sequence and organizes execution.

[0003] In the existing process, the environment and the task are generally described in the planning domain description language: the domain file gives the action mode, the precondition, and the effect, and the problem file gives the object set, the initial state, and the target condition. By writing the objects, attributes, and spatial relationships obtained through perception into the problem file, the system can call a classic planner to generate an action sequence that meets the precondition and causal chain.

[0004] However, for long-time sequence, strong constraint, and partially observable tasks, the above process exposes common deficiencies. First, the symbolic representation of the environment is mostly generated at the beginning of the task, and it is difficult to incorporate new observations or interaction facts in a timely manner, resulting in incomplete or biased symbolic worlds for a long time. Second, the feedback generated by verification or execution is mostly used for local adjustment of the action sequence itself, and less for updating and correcting the structured signals of the environment symbolic model, so that structural problems caused by perception noise or information loss occur repeatedly in different tasks. Third, classic planners usually require a relatively complete and consistent state description, and it is difficult to generate executable solutions when information is missing; while high-level plans that only rely on VLMs are prone to produce steps that are inconsistent with the current environment when symbolic support is insufficient, and may trigger a chain failure once in the execution phase, with high trial and error costs.

[0005] The reasons for the above problems are that the real scene has dynamicity and local visibility, the mapping of the perceived symbols has noise and uncertainty, and the existing process is static and single-round modeling mode, lacking a closed-loop mechanism that can convert feedback into structured signals before execution, and use them for action planning and environment symbol model collaborative update, and lacking the ability to deposit stable rules into structured memory that can be reused across tasks, and use them in subsequent modeling and planning stages. This brings problems such as insufficient plan executability, repeated errors, and weak cross-task generalization, which cannot meet the requirements of long-time complex tasks for robustness and efficiency. How to solve the defects in the prior art is a technical problem that needs to be solved urgently. SUMMARY

[0006] The purpose of the present application is to solve the following problems existing in the prior art: in a dynamic and partially observable environment, the existing VLM+symbol planning process lacks a closed-loop capability of consistent verification and collaborative update of action planning and environment symbol model before execution, and lacks a mechanism of depositing stable rules into structured memory that can be injected and reused to serve subsequent modeling and planning, resulting in incomplete symbol description for a long time, insufficient plan executability and robustness, repeated errors, and high trial and error cost; a scheme is needed that can convert verification / interaction feedback into structured signals, and realize synchronous update of plan and environment model in pure symbol space, while forming a symbolized memory that can be transferred across tasks, to significantly improve the reliability and generalization ability of long-time tasks. A long-time task planning method and system based on adaptive symbol world construction are provided.

[0007] In order to achieve the above application purpose, the present application specifically adopts the following technical solutions:

[0008] In the first aspect, the present application provides a long-time task planning method based on adaptive symbol world construction, which includes the following steps:

[0009] S1, extracting the object, object state, object attribute and relationship between objects contained in the environment image by a pre-trained visual language model, for constructing and incrementally maintaining the symbol world;

[0010] S2, obtaining a domain file in a planning domain definition language, using a visual language model to generate a problem file in a planning domain definition language based on a natural language task description and the symbol world of S1, and then generating an action sequence in a planning domain definition language style by the visual language model under the constraints of the domain file and the problem file;

[0011] S3, calling a symbolic execution simulator to verify the action sequence in the symbolic world: when the verification fails, outputting plan-level feedback and environment-level feedback, updating the action sequence and the symbolic world according to the plan-level feedback and the environment-level feedback, calling the symbolic execution simulator to verify the updated action sequence in the updated symbolic world again until the verification passes or a preset maximum verification round is reached; if the verification passes, executing the action sequence in the environment where the environment image is located and recording actual execution feedback;

[0012] S4, taking the plan-level feedback and the environment-level feedback generated by the symbolic execution simulator as structured feedback, abstracting the structured feedback and the actual execution feedback into structured symbolic memory, and adding the structured symbolic memory into a system prompt template to enhance the reasoning ability of a visual language model and realize cross-task and interpretable knowledge generalization.

[0013] On the basis of the above scheme, each step can be implemented in the following preferred specific manner.

[0014] As a preferred embodiment of the first aspect, in step S1, the construction and incremental maintenance method of the symbolic world is as follows:

[0015] S11, using a pre-trained visual language model to recognize the obtained environment image, extracting the object, object state, object attribute and relationship between objects contained in the environment image, generating corresponding object name, state predicate, attribute predicate and relationship predicate, and forming a symbolic world for structured description of the environment state in the environment image;

[0016] S12, the visual language model makes a decision based on the current symbolic world and the natural language task description, and selects a planning task or continues to explore the environment: if the planning task is selected, step S2 is entered; if the exploration of the environment is continued, a new environment image is obtained and detected by the visual language model, when a new object is generated or the state of the object changes or there is a conflict between predicates, if a new object is generated, the visual language model generates the corresponding object name, and extracts the state predicate, attribute predicate and relationship predicate corresponding to the new object as new content to supplement to the symbolic world, if the state of the object changes or there is a conflict between predicates, the visual language model locates the associated object name, generates the modified state predicate, attribute predicate and relationship predicate to update the symbolic world.

[0017] As a preferred embodiment of the first aspect, in step S2, the generation method of the action sequence is as follows:

[0018] S21, based on the S1 constructed symbolic world and the natural language task description, the visual language model generates a problem file in the planning domain definition language, which includes three parts of an object set, an initial state predicate set and a target condition predicate set;

[0019] S22, based on the problem file and the domain file, the visual language model generates an action sequence in the planning domain definition language style using a few-shot prompt.

[0020] As a preferred embodiment of the first aspect, in step S21, the object set consists of the object names recognized in the symbolic world, the initial state predicate set consists of the state predicates, attribute predicates and relationship predicates in the symbolic world, and the target condition predicate set consists of the state predicates and relationship predicates.

[0021] As a preferred embodiment of the first aspect, in step S3, the symbolic execution simulator verifies the action sequence and outputs the plan-level feedback and the environment-level feedback in the following manner: using a verifier of the planning domain definition language to perform step-by-step simulation execution on the symbolic world, if the simulation execution fails, recording meta-information consisting of four parts and converting it into the plan-level feedback and the environment-level feedback; wherein the meta-information consists of the step position of the failed action, the name of the failed action, the object related to the failed action, and the precondition that causes the action to fail; the plan-level feedback is a structured annotation of the problems existing in the action sequence itself, including the following types: invalid action, parameter mismatch, unsatisfied precondition, and causal chain break; the environment-level feedback is a structured annotation of the incompleteness or errors in the modeling of the symbolic world, including the following types: undefined object and object relationship error.

[0022] As a preferred embodiment of the first aspect, in step S4, the specific process of forming the structured symbolic memory is as follows: extracting the execution failure reasons from the plan-level feedback and the actual execution feedback to form structured action semantic memory triples, extracting the environmental knowledge defects from the environment-level feedback and the actual execution feedback to form structured environment perception memory triples, thereby forming the structured symbolic memory from the two kinds of memory triples; wherein the semantic memory triples consist of the failed action name, the predicate set of the preconditions that need to be satisfied for the successful execution of the action, and the expected effect predicate set that the action should produce; the environment perception memory triples consist of the missing state predicate set, the attribute predicate error set and the relationship predicate error set.

[0023] Secondly, the present application provides a long-time sequence task planning system based on an adaptive symbolic world, which comprises:

[0024] A symbolic world construction module is configured to extract, by a pre-trained visual language model, an item object, an item object state, an item object attribute, and a relationship between item objects contained in an environment image, and to construct and incrementally maintain a symbolic world;

[0025] An action sequence generation module is configured to obtain a domain file in a planning domain definition language, generate, by a visual language model, a problem file in the planning domain definition language based on a natural language task description and the symbolic world, and generate an action sequence in the planning domain definition language under the constraints of the domain file and the problem file.

[0026] A feedback generation module is configured to verify the action sequence by calling a symbolic execution simulator in the symbolic world, output a plan-level feedback and an environment-level feedback when the verification fails, update the action sequence and the symbolic world according to the plan-level feedback and the environment-level feedback, verify the updated action sequence again in the updated symbolic world by calling the symbolic execution simulator, until the verification passes or a preset maximum verification round is reached, and execute the action sequence in an environment where the environment image is located and record an actual execution feedback when the verification passes.

[0027] A symbolic memory generation module is configured to abstract the plan-level feedback and the environment-level feedback generated by the symbolic execution simulator as structured feedback, abstract the structured feedback and the actual execution feedback as structured symbolic memory, and add the structured symbolic memory into a system prompt template to enhance the reasoning ability of the visual language model and realize cross-task and interpretable knowledge generalization.

[0028] In a third aspect, the present application provides a computer program product, which comprises computer programs / instructions that, when executed by a processor, can implement the long-time sequence task planning method based on adaptive symbolic world construction according to any one of the above first aspect.

[0029] In a fourth aspect, the present application provides a computer readable storage medium, which stores computer programs, and when the computer programs are executed by a processor, the long-time sequence task planning method based on adaptive symbolic world construction according to any one of the above first aspect is implemented.

[0030] In a fifth aspect, the present application provides a computer electronic device, which comprises a memory and a processor.

[0031] The memory is configured to store computer programs.

[0032] The processor is configured to implement the long-time sequence task planning method based on adaptive symbolic world construction according to any one of the above first aspect when the computer programs are executed.

[0033] Compared with the prior art, the present application has the following beneficial effects:

[0034] 1. Symbolic environment modeling is more complete: By constructing and verifying feedback correction mechanisms in parallel, the present invention can significantly improve the completeness of the symbolic world.

[0035] 2. The plan is more executable and robust: The present invention uses a symbolic execution simulator to verify and correct the action sequence step by step, avoiding execution failures caused by missing preconditions or incorrect causal relationships.

[0036] 3. Self-adaptive memory improves cross-task generalization ability: Through the symbolic long-term memory mechanism, the present invention can retain verified action conditions and environmental laws, and reuse them in subsequent tasks to avoid repeated mistakes.

[0037] 4. Strong anti-environmental uncertainty ability: The present invention supports dynamic correction and supplementary reasoning in partially observable environments, and can still generate feasible plans under conditions of missing observations or perception noise.

[0038] 5. Closed-loop adaptive iteration: Through the closed-loop mechanism of "planning-verification-correction-memory", the present invention can continuously optimize itself during execution, showing continuous learning and long-term evolution ability, which is not possessed by traditional static symbolic modeling and single-step correction methods. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 The step flowchart of the method of the present invention;

[0040] Figure 2 The overall framework schematic diagram of the method of the present invention;

[0041] Figure 3 The system block diagram of the present invention. DETAILED DESCRIPTION

[0042] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention will be described in detail below. In the following description, a large number of specific details are set forth in order to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the spirit of the present invention, so the present invention is not limited by the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflict.

[0043] In the description of the present invention, it should be understood that the terms "first", "second" are only used for distinguishing description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features.

[0044] Before introducing the technical solutions of the present application, first introduce the related background knowledge involved in the present application.

[0045] In the field of embodied intelligence, long-horizon task planning refers to the process of generating a series of action plans for an agent that satisfy stage goals and temporal dependencies in an environment with complex task structure and long time span. Such tasks often involve the sequential arrangement of multiple sub-goals, the satisfaction of conditional constraints, the coordination of stage switching, and the ability to perceive and adapt to external environmental changes during execution. Compared to immediate reaction tasks, long-horizon tasks place more emphasis on the logical continuity between tasks, the feasibility of execution, and the long-term management and regulation of process states.

[0046] Long-horizon task planning has the following notable features: first, there are a large number of task steps, and there are significant dependencies between sub-tasks; second, the task execution period is long, covering multiple physical or cognitive stages; third, the environment state may change continuously during execution, and the agent needs to have dynamic perception and real-time adjustment capabilities; fourth, the stability and fault tolerance requirements are high during task execution, and the task strategy needs to be adjusted in time in the event of local failure to ensure the realization of the overall goal.

[0047] In the current task planning system based on large-scale visual language models, there are the following outstanding challenges in the face of the above long-horizon scenarios: 1. Insufficient reasoning depth and logical consistency. Although language models can generate task steps based on semantic relationships, they often exhibit instability when dealing with conditional dependencies between tasks, stage transitions, or cyclic structures. For example, when a sub-task fails due to unmet prerequisites, the model lacks effective backward reasoning capabilities to adjust the plan structure, making it difficult to reorder or insert additional steps to achieve task recovery. 2. Plan and execution layer decoupling is serious. The task plans generated by existing methods often do not consider the feasibility and cost evaluation of underlying actions. Task steps may appear reasonable at the language level, but when mapped to specific actions such as movement and grasping, they often result in unexecutable or inefficient action sequences due to the lack of symbolic support or execution constraint checking. 3. Lack of robustness and explainability. In the face of new environments or handling abnormal situations, existing models lack the ability to constrain the real state, often producing planning outputs that are inconsistent with the environment state, or even generating steps that lack basis. At the same time, when a task fails or an exception occurs, the system often has difficulty providing explicit failure reasons or subsequent correction suggestions, limiting the model's debuggability and safety.

[0048] Compared with the prior art, the application has the following advantages in long-time task planning: the application realizes the robust planning of a visual language model for a long-time task in a complex dynamic environment by constructing a structured symbolic world, introducing a verifiable execution mechanism at a symbolic level, and continuously accumulating structured symbolic memories. The main advantages are as follows:

[0049] In terms of task planning continuity and executability, the application converts environment images into an operable symbolic world, and automatically generates a problem description acceptable to a traditional planner based on a natural language task target, so that the action sequence output by the visual language model has clear semantic constraints and causal logic, and the problem of skip, redundancy or illusion action in task planning is solved.

[0050] In terms of the efficiency and explainability of plan verification, the application introduces a symbolic execution simulator to replace real environment execution, and combines structured plan-level and environment-level feedback to accurately mark the type, location and reason of action failure. This mechanism not only significantly improves the verification efficiency and quickly identifies plan defects without real physical execution, but also enhances the explainability of task failure, providing clear basis for subsequent planning adjustment.

[0051] In terms of self-enhancement and task generalization of the model itself, the application abstracts the structured feedback of the plan level and the environment level of the symbolic execution simulator and the actual execution feedback of the action into structured symbolic memories, which are used to guide the visual language model to more accurately construct the symbolic world in subsequent tasks, and to guide the generation of action sequences with strong logic, thereby forming a long-time planning method with memory, feedback, self-correction and generalization capability, that is, a long-time task planning method based on adaptive symbolic world construction.

[0052] As shown in Figure 1 , in a preferred implementation of the application, the long-time task planning method based on adaptive symbolic world construction includes the following S1-S4 steps. The specific implementation process will be described below.

[0053] S1, extract the object, object state, object attribute and relationship between objects contained in the environment image by the pre-trained visual language model, for constructing and incrementally maintaining the symbolic world.

[0054] It should be noted that in the symbolic world construction stage of the application S1, the visual language model used for object recognition and relationship extraction is not limited to a specific model, and any multi-modal pre-training model can be used, such as GPT-4o, Qwen2.5-VL, PaLM-E, etc., or its task fine-tuned variant, so it is not limited in the application.

[0055] It should be noted that in S1 of the present application, the environment image is an image actually collected in a simulation environment or in a real environment. In the subsequent process, the agent can continuously construct and update this image through perception, and perform complex task decomposition, logical reasoning and long-term planning based on it.

[0056] It should be noted that in S1 of the present application, the construction and incremental maintenance method of the symbolic world is as follows:

[0057] S11, the pre-trained visual language model is used to recognize the acquired environment image, and the object, object state, object attribute and relationship between objects contained in the environment image are extracted, and the corresponding object name, state predicate, attribute predicate and relationship predicate are generated, and a symbolic world is formed for structured description of the environment state in the environment image.

[0058] S12, the visual language model makes a decision based on the current symbolic world and the natural language task description, and selects a planning task or continues to explore the environment: if the planning task is selected, step S2 is entered; if the exploration of the environment is continued, a new environment image is acquired and detected by the visual language model, when a new object is detected, the corresponding object name is generated by the visual language model, and the state predicate, attribute predicate and relationship predicate corresponding to the new object are extracted as new content to supplement the symbolic world, if there is a change in the state of the object or there is a conflict between the predicates, the visual language model is used to locate the associated object name, and the modified state predicate, attribute predicate and relationship predicate are generated to update the symbolic world.

[0059] In this embodiment, as shown in Figure 2 , first, symbol extraction is performed: an environment image is collected, the object set and its state, attribute and relationship are recognized from the image by using the above-mentioned visual language model, and a predicate set describing the above-mentioned environment is established, including state predicate, attribute predicate and relationship predicate, and all extracted predicates are recorded in the symbolic world The symbol world is used for subsequent long-time sequence task planning and verification. Secondly, an incremental strategy is designed. The visual language model makes a decision based on the current symbolic world and the natural language task description. If the planning task is selected, step S2 is entered. If more environment is explored, a new environment image is obtained, and the visual language model detects whether there is a new object, a state change or a conflict predicate in the image. If the state of the existing object is found to have changed, or there is a semantic conflict with the existing predicate, the visual language model generates a set of predicates that need to be modified, and locates the object name associated with the set of predicates, to replace the original corresponding set of predicates, thereby realizing incremental modification and consistency maintenance of the symbolic world, maintaining the consistency of the symbolic world and the actual environment, and avoiding reasoning errors or constraint violations in the symbolic world.

[0060] S2, a domain file in a planning domain definition language (PDDL) is obtained, and a visual language model is used to generate an action sequence in a planning domain definition language style based on a natural language task description and the symbol world of S1 under the constraints of the domain file and the problem file.

[0061] It should be noted that in the present application, the domain file (Domain File) and the problem file (Problem File) in the planning domain definition language (PDDL) are in a closely coupled and complementary relationship, and they together describe a complete planning problem. The domain file extension is usually.dl or.pddl, which defines general actions (Actions), predicates (Predicates), and types (Types). The problem file extension is usually.p or.pddl, which defines specific objects (Objects), initial states (Initial State), and goals (Goal).

[0062] In the above step S2, the action sequence generation method is as follows:

[0063] S21, based on the symbol world constructed in S1 and the natural language task description, a visual language model generates a problem file in a planning domain definition language, which includes three parts: an object set, an initial state predicate set and a target condition predicate set. The object set is composed of object names recognized in the symbol world, the initial state predicate set is composed of state predicates, attribute predicates and relationship predicates in the symbol world, and the target condition predicate set is composed of state predicates and relationship predicates.

[0064] It should be noted that the above problem file includes three core components: an item object set, an initial state predicate set, and a target condition predicate set. The item object set and the initial state predicate set are directly obtained from the corresponding parts of the symbolic world constructed in step S1; and the target condition predicate set is intelligently parsed by the visual language model according to the current task semantics to ensure the correctness and feasibility of the task target. Specifically, the item object set is composed of the names of the identified item objects in the symbolic world, which is used to define all entity objects in the current environment that can be referenced by the method of the present application; the initial state predicate set is composed of state predicates, attribute predicates, and relationship predicates in the symbolic world, which is used to describe the environment state before task execution; and the target condition predicate set is composed of state predicates and relationship predicates, which is generated by the visual language model from the symbolic world in combination with the natural language task description, and is used to represent the target conditions that should be met when the task is completed.

[0065] S22, based on the problem file and the domain file, using a few-shot prompt to guide the visual language model to generate a planning domain definition language style action sequence for subsequent symbolic execution verification and correction; wherein, is the first action of the action sequence.

[0066] In addition, it should be noted that in the action sequence generation phase of S2, the method of generating the action sequence is not limited to the visual language model based on few-shot prompt, but can also be replaced by a reinforcement learning driven action generator, a program synthesizer, or a retrieval-based planner based on a large-scale symbolic database.

[0067] S3, calling a symbolic execution simulator in the symbolic world to verify the action sequence: when the verification fails, outputting plan-level feedback and environment-level feedback, updating the action sequence and the symbolic world according to the plan-level feedback and the environment-level feedback, calling the symbolic execution simulator in the updated symbolic world to verify the updated action sequence again until the verification passes or the maximum number of verification rounds is reached; if the verification passes, executing the action sequence in the environment where the environment image is located and recording the actual execution feedback.

[0068] It should be noted that in step S3 of the present application, the symbolic execution simulator verifies the action sequence and outputs the plan level feedback and the environment level feedback as follows: using a planner domain definition language verifier to simulate the execution of the action sequence step by step in the symbolic world, if the simulation execution fails, record the meta-information composed of four parts and convert it into the plan level feedback and the environment level feedback; wherein, the meta-information is composed of the step position where the failed action is located, the name of the failed action, the object related to the failed action, and the precondition that causes the action to fail (i.e. the predicate constraint that the action does not satisfy in the current environment); the plan level feedback is a structured annotation of the problem existing in the action sequence itself, including the following types: invalid action, parameter mismatch, unsatisfied precondition, and causal chain break; the environment level feedback is a structured annotation of the incomplete or incorrect modeling of the symbolic world, including the following types: undefined object and relationship error between objects.

[0069] It should be noted that in the present embodiment, the above-mentioned plan level feedback will be precisely positioned to the action at the corresponding step position and its related predicate, guiding the visual language model to make targeted modifications to the action sequence; the above-mentioned environment level feedback is used to clearly point out the environmental modeling defects that cause the verification failure, guiding the visual language model to make incremental modifications and updates to the symbolic world, thereby improving the feasibility of subsequent planning and the actual execution success rate. Specifically, the plan level feedback refers to a structured annotation of the problem existing in the action sequence itself, which includes the following types: 1) invalid action, which means that the action is undefined or not applicable in the symbolic world; 2) parameter mismatch, which means that the action parameters do not match the object type or quantity; 3) unsatisfied precondition, which means that the object state or attribute predicate that needs to be satisfied before the action is executed does not hold in the symbolic world; 4) causal chain break, which means that the result of a certain action in the action sequence does not satisfy the precondition of the subsequent action. The environment level feedback refers to a structured annotation of the incomplete or incorrect modeling of the symbolic world, which includes the following types: 1) undefined object, which means that the object involved in the action is not defined in the symbolic world; 2) relationship error between objects, such as incorrect relationship predicates between objects.

[0070] It should be noted that in the S3 symbolic execution simulator, the planner domain definition language verifier can be replaced by ASP (Answer Set Programming) or other solvers, and the structured verification can also be realized through a self-defined symbolic logic reasoning engine.

[0071] It should be noted that in step S3 of the present application, the visual language model generates a new action sequence as the updated action sequence and a new symbolic world as the updated symbolic world based on the action sequence, the symbolic world, the plan level feedback, and the environment level feedback, i.e. the modified action plan and the environment representation:

[0072]

[0073] After generating the updated action sequence and the updated symbolic world, the symbolic execution simulator is invoked in the updated symbolic world to verify the updated action sequence, and when the updated action sequence fails to pass the verification again, new planning-level feedback and new environment-level feedback are generated, and the action sequence and the symbolic world are updated again according to the new planning-level feedback and the new environment-level feedback, and the symbolic execution simulator is invoked in the second updated symbolic world to continue verifying the second updated action sequence, and the iteration is repeated until a preset termination condition is met. In this embodiment, the iteration is terminated when any of the following conditions is met: (1) the action sequence is executable throughout the verification in the symbolic world and reaches the task goal; (2) a preset maximum verification round is reached.

[0074] S4, the planning-level feedback and the environment-level feedback generated by the symbolic execution simulator are taken as structured feedback, and the structured feedback and the actual execution feedback are abstracted into structured symbolic memories and added to the system prompt template to enhance the reasoning ability of the visual language model and realize cross-task and interpretable knowledge generalization.

[0075] It should be noted that the specific process of forming the structured symbolic memories is as follows:

[0076] S41, the execution failure reason is extracted from the planning-level feedback and the actual execution feedback to form a structured action semantic memory triple, which consists of a failed action name, a predicate set of preconditions required for successful execution of the action, and an expected effect predicate set generated by the action.

[0077] S42, the environment knowledge defects are extracted from the environment-level feedback and the actual execution feedback to form a structured environment perception memory triple, which consists of a missing state predicate set, an attribute predicate error set, and a relationship predicate error set.

[0078] S43, the structured symbolic memories are composed of the semantic memory triple and the environment perception memory triple.

[0079] In this embodiment, for the action type that repeatedly fails to pass the verification during the task execution, the key failure reason is extracted from the planning-level feedback and the actual execution feedback to form a structured action semantic memory triple , wherein represents the failed action name; represents the predicate set of preconditions required for successful execution of the action, that is, the state predicate, the attribute predicate, or the relationship predicate that should be satisfied before the action is successfully executed; a set of predicate of expected effect of the action, used to describe the changes of state predicate and relation predicate that the environment should satisfy after the action is executed, such as an object being grasped, a container being opened, etc., reflecting the influence of the action on the state of the items and the relations of the items in the symbolic world. For task failure caused by incomplete modeling of the symbolic world, structured environment knowledge defects are extracted from the structured environment perception memory triplets from the environment-level feedback and the actual execution feedback wherein, a set of missing state predicates, such as the state of a switch of an object not being labeled, a holding state, etc. a set of attribute predicate errors, such as an object that cannot be grasped being incorrectly labeled as being graspable; a set of relation predicate errors, such as missing or incorrect labeling of spatial position relations.

[0080] The above action semantic memory triplets can be embedded into system prompt templates, used to guide the visual language model to avoid generating actions that violate the semantics of the task or violate physical logic in new tasks, thereby improving the feasibility and robustness of planning. Similarly, the above environment perception memory triplets can also be embedded into system prompt templates, improving the modeling quality and planning stability of subsequent tasks. Thus, the two types of structured triplets will be injected into the inference process of the visual language model in subsequent tasks in the form of templated prompts, improving the modeling quality of the visual language model in low-observation, perception deviation, etc. and the planning reliability of the visual language model in long-time tasks.

[0081] It should be noted that, for step S3, in a single task execution process, the visual language model generates an action sequence based on the currently constructed symbolic world and the natural language task description. The action sequence is verified in the symbolic world through a symbolic execution simulator. If the verification fails, plan-level feedback and environment-level feedback are generated to accurately locate the reasons, positions, and constraint condition missing situations of the failed action. The structured feedback information will directly act on the visual language model, used to guide it to adjust the action sequence or update the symbolic world incrementally, and repeat the verification until the action sequence is executable. The above operation constitutes a closed feedback loop of “planning-verification-correction”, which directly guides the visual language model to correct the action sequence or update the symbolic world, significantly improving the adaptability and verification efficiency of the model to complex tasks. Without actual physical execution, the model can complete efficient iterative optimization of task planning, improve the instant adaptability and robustness of the system to complex task structures.

[0082] In addition, for step S4, the application further introduces a symbolic memory abstraction mechanism during the multiple task planning and execution processes, extracts, induces and modularly represents the structural failure information repeatedly appearing in the tasks. Specifically, it includes action semantic abstraction and environment perception abstraction. The symbolic memory will be injected into the subsequent task reasoning process of the visual language model in the form of modular prompts, helping it to more accurately construct a symbolic world and generate action sequences with physical logic and semantic consistency, thereby achieving the improvement of modeling robustness and the generalization of planning ability in low observation and uncertainty scenarios.

[0083] In order to better show the specific implementation and technical effects of the application, the long-time sequence task planning method based on adaptive symbolic world construction shown in steps S1-S4 in the above preferred implementation mode will be applied to a specific example.

[0084] Embodiment

[0085] The specific implementation process of the long-time sequence task planning method based on adaptive symbolic world construction adopted in this embodiment is as described above, and will not be repeated here.

[0086] This embodiment verifies the effect in the VirtualHome simulation platform. The platform simulates the daily environment of a family, supports visual observation, language instruction input and multi-object interaction, and is suitable for testing the planning and modeling capabilities of embodied agents.

[0087] Three kinds of environment configurations are selected in the experiment, including different house types (marked as env1, env2 and env3 respectively) in terms of space structure, lighting conditions and material, in order to increase the complexity of the scene and the diversity of the perception interference. The task sources include the official task templates of VirtualHome, covering five categories of daily tasks such as personal hygiene, food preparation, housekeeping, entertainment and leisure, and a total of 62 task instances are sampled as evaluation benchmarks.

[0088] In the construction phase, the application constructs structured memory through interactive exploration, symbolic verification and execution feedback; in the test phase, the memory content is fixed and no longer updated, in order to verify the knowledge transfer and generalization ability.

[0089] In order to objectively evaluate the system performance, the initial starting point, viewing angle and other conditions are randomly set in each group of experiments, and the average value is reported. All methods are run under the unified action domain file and initial observation conditions to ensure fairness of comparison.

[0090] The evaluation indexes include: (1) ClassicalSR: classical planner support rate, reflecting the support degree of the symbolic world to the planning; (2) SymRecall: task-related predicate recall rate, measuring the integrity of symbolic modeling; (3) SymHalluc: symbolic hallucination rate, measuring the proportion of irrelevant or false predicates; (4) GCR: goal condition recall rate, measuring the satisfaction degree of the final goal; (5) Exec: executable action proportion, measuring the executability of the plan in the environment; (6) R-Count: total modification times of the plan and the environment, measuring the convergence efficiency.

[0091] The comparison methods include: (1) Classical Planner: directly calling the classical planner to solve in the symbolic world; (2) OnlyPDDL: constructing the symbolic world and generating the plan once, without incremental correction; (3) ForPlan: only adjusting the action sequence according to the symbolic verification feedback, without updating the environment symbols; (4) PASS: the method of the application, updating the plan and the symbolic world in the symbolic world, and generalizing the structured memory.

[0092] As shown in Table 1, in the symbolic modeling and plan executability evaluation, the symbolic world generated by the method of the application is more complete and has stronger planning support, and the executability of the plan is also stronger. At the same time, the method of the application shows lower modification frequency in R-Count, indicating that the system can converge to a feasible plan more quickly in the symbolic space.

[0093] Table 1. Symbolic modeling and execution indexes in different environments

[0094]

[0095] Further, Table 2 shows the robustness results in the partially observable environment. Compared with other methods, the method of the application can still maintain a high plan success rate and goal achievement rate in the case of incomplete observation of the environment, verifying the supporting role of the structured memory on the stability of cross-task modeling and planning.

[0096] Table 2. Planning robustness under partially observable conditions

[0097]

[0098] In addition, it should be noted that the long-term task planning method based on adaptive symbolic world construction in the above embodiment can be essentially executed by a computer program or a module. Therefore, based on the same inventive concept, another preferred embodiment of the application also provides a long-term task planning system based on adaptive symbolic world construction corresponding to the long-term task planning method based on adaptive symbolic world construction provided by the above embodiment, as shown in Figure 3 which includes:

[0099] a symbolic world construction module configured to extract, by a pre-trained visual language model, item objects, item object states, item object attributes, and relationships between the item objects contained in the environment image, and to construct and incrementally maintain a symbolic world;

[0100] an action sequence generation module configured to obtain a domain file in a planning domain definition language, generate, by a visual language model, a problem file in the planning domain definition language based on a natural language task description and the symbolic world, and generate, by the visual language model, an action sequence in the planning domain definition language under the constraints of the domain file and the problem file;

[0101] a feedback generation module configured to verify the action sequence by calling a symbolic execution simulator in the symbolic world, output planning-level feedback and environment-level feedback when the verification fails, update the action sequence and the symbolic world according to the planning-level feedback and the environment-level feedback, verify the updated action sequence again in the updated symbolic world by calling the symbolic execution simulator, until the verification passes or a preset maximum verification round is reached, and execute the action sequence in an environment in which the environment image is located and record actual execution feedback if the verification passes;

[0102] a symbolic memory generation module configured to abstract the planning-level feedback and the environment-level feedback generated by the symbolic execution simulator as structured feedback, abstract the structured feedback and the actual execution feedback as structured symbolic memory, and add the structured symbolic memory into a system prompt template to enhance the reasoning ability of the visual language model and realize cross-task and interpretable knowledge generalization.

[0103] It can be understood that the long sequential task planning method based on adaptive symbolic world construction described in S1-S4 can be realized by a computer program. Therefore, based on the same inventive concept, another preferred embodiment of the present application also provides a computer program product corresponding to the long sequential task planning method based on adaptive symbolic world construction provided by the above-mentioned embodiment, which includes computer programs / instructions that can be executed by a processor to realize the long sequential task planning method based on adaptive symbolic world construction as described in the above-mentioned embodiment.

[0104] Similarly, based on the same inventive concept, another preferred embodiment of the present application also provides a computer electronic device corresponding to the long sequential task planning method based on adaptive symbolic world construction provided by the above-mentioned embodiment, which includes a memory and a processor;

[0105] the memory is configured to store a computer program;

[0106] The processor is configured to implement the long-time task planning method based on the adaptive symbol world construction when the computer program is executed.

[0107] In addition, the logic instructions in the memory described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application.

[0108] Therefore, based on the same inventive concept, another preferred embodiment of the present application also provides a computer readable storage medium corresponding to the long-time task planning method based on the adaptive symbol world construction provided by the above-mentioned embodiments. The storage medium stores a computer program, and when the computer program is executed by a processor, the long-time task planning method based on the adaptive symbol world construction in the above-mentioned embodiments can be implemented.

[0109] Specifically, in the computer readable storage medium of the above three embodiments, the computer program stored therein can be executed by a processor to perform the steps S1-S3.

[0110] It can be understood that the storage medium can include a random access memory (RAM) and can also include a non-volatile memory (NVM), such as at least one disk memory. Meanwhile, the storage medium can also be a U disk, a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0111] It can be understood that the processor described above can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0112] In addition, it should be noted that, for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the foregoing method embodiments, and will not be described here. In the embodiments provided in the present application, the division of steps or modules in the system and method described is only a logical functional division, and there can be another division manner in actual implementation, for example, multiple modules or steps can be combined or integrated together, or a module or step can be split.

[0113] The above-described embodiments are only a preferred scheme of the present application, and are not intended to limit the present application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present application. Therefore, any technical scheme obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present application.

Claims

1. A long-term task planning method based on adaptive symbolic world construction, characterized in that, Includes the following steps: S1. The pre-trained visual language model extracts the objects, states, attributes, and relationships between objects contained in the environment image to construct and incrementally maintain the symbolic world. S2. Obtain the domain file in the planning domain definition language. Using the visual language model based on the natural language task description and the symbolic world of S1, generate the problem file in the planning domain definition language. Under the constraints of the domain file and the problem file, the visual language model generates an action sequence in the style of the planning domain definition language. S3. Verify the action sequence by calling the symbolic execution simulator in the symbolic world: When the verification fails, output plan-level feedback and environment-level feedback. Update the action sequence and symbolic world based on the plan-level feedback and environment-level feedback. Then, call the symbolic execution simulator again in the updated symbolic world to verify the updated action sequence until the verification passes or the preset maximum number of verification rounds is reached. If the verification passes, execute the action sequence in the environment where the environment image is located and record the actual execution feedback. S4. The plan-level feedback and environment-level feedback generated by the symbolic execution simulator are used as structured feedback. The structured feedback and actual execution feedback are abstracted into structured symbolic memory and added to the system prompt template to enhance the reasoning ability of the visual language model and achieve cross-task, interpretable knowledge generalization.

2. The long-term task planning method based on adaptive symbolic world construction as described in claim 1, characterized in that, In step S1, the method for constructing and incrementally maintaining the symbolic world is as follows: S11. Using a pre-trained visual language model, the acquired environmental image is identified, and the object, state, attribute, and relationship between the objects are extracted. Then, the corresponding object name, state predicate, attribute predicate, and relation predicate are generated to form a symbolic world for the structured description of the environmental state in the environmental image. S12. The visual language model makes a decision based on the current symbolic world and natural language task description, choosing to plan the task or continue exploring the environment: If the planning task is selected, proceed to step S2; if the environment is continued to be explored, a new environmental image is acquired and detected by the visual language model. When a new item object is detected, or there is a change in the state of an item object, or there is a conflict between predicates, if a new item object is detected, the visual language model generates the corresponding item object name and extracts the state predicate, attribute predicate, and relation predicate corresponding to the new item object, adding them to the symbolic world as new content. If there is a change in the state of an item object or a conflict between predicates, the visual language model locates the associated item object name and generates modified state predicates, attribute predicates, and relation predicates to update the symbolic world.

3. The long-term task planning method based on adaptive symbolic world construction as described in claim 2, characterized in that, In step S2, the action sequence is generated as follows: S21. Based on S1, the symbolic world and natural language task description are constructed, and the visual language model generates a problem file in the planning domain definition language. The problem file contains three parts: a set of item objects, a set of initial state predicates, and a set of target condition predicates. S22. Based on the problem file and domain file, a few-shot prompt guides the visual language model to generate action sequences in the planning domain-defined language style.

4. The long-term task planning method based on adaptive symbolic world construction as described in claim 3, characterized in that, In step S21, the set of item objects consists of the names of item objects identified in the symbolic world, the set of initial state predicates consists of state predicates, attribute predicates and relation predicates in the symbolic world, and the set of target condition predicates consists of state predicates and relation predicates.

5. The long-term task planning method based on adaptive symbolic world construction as described in claim 3, characterized in that, In step S3, the symbolic execution simulator verifies the action sequence and outputs plan-level and environment-level feedback as follows: The action sequence is simulated step-by-step in the symbolic world using a verifier based on the planning domain definition language. If the simulation fails, meta-information consisting of four parts is recorded and transformed into plan-level and environment-level feedback. The meta-information comprises the step location of the failed action, the name of the failed action, the item objects related to the failed action, and the preconditions that caused the action failure. Plan-level feedback is a structured annotation indicating problems with the action sequence itself, including the following types: invalid action, parameter mismatch, unmet preconditions, and broken causal chains. Environment-level feedback is a structured annotation indicating incomplete or incorrect modeling of the symbolic world, including the following types: undefined item objects and incorrect relationships between item objects.

6. The long-term task planning method based on adaptive symbolic world construction as described in claim 5, characterized in that, In step S4, the specific process of forming structured symbolic memory is as follows: extracting the reasons for execution failure from planning-level feedback and actual execution feedback to form structured action semantic memory triples; extracting environmental knowledge deficiencies from environmental-level feedback and actual execution feedback to form structured environmental perception memory triples; thus, structured symbolic memory is composed of two types of memory triples. Among them, the semantic memory triples consist of the name of the failed action, the set of predicates for the preconditions that the action needs to be met for successful execution, and the set of predicates for the expected effect that the action should produce; the environmental perception memory triples consist of the set of missing state predicates, the set of incorrect attribute predicates, and the set of incorrect relational predicates.

7. A long-term task planning system based on an adaptive symbolic world, characterized in that, include: The symbolic world construction module is used to extract the objects, states, attributes, and relationships between objects contained in the environment image from the pre-trained visual language model, and is used to construct and incrementally maintain the symbolic world. The action sequence generation module is used to obtain the domain file in the planning domain definition language, and then use the visual language model to generate the problem file in the planning domain definition language based on the natural language task description and symbolic world. Under the constraints of the domain file and the problem file, the visual language model generates action sequences in the style of the planning domain definition language. The feedback generation module is used to call the symbolic execution simulator in the symbolic world to verify the action sequence. When the verification fails, it outputs plan-level feedback and environment-level feedback. After updating the action sequence and the symbolic world based on the plan-level feedback and environment-level feedback, it calls the symbolic execution simulator again in the updated symbolic world to verify the updated action sequence until the verification passes or the preset maximum number of verification rounds is reached. If the verification passes, the action sequence is executed in the environment where the environment image is located and the actual execution feedback is recorded. The symbolic memory generation module is used to take the plan-level feedback and environment-level feedback generated by the symbolic execution simulator as structured feedback. After abstracting the structured feedback and actual execution feedback into structured symbolic memory, it is added to the system prompt template to enhance the reasoning ability of the visual language model and achieve cross-task, interpretable knowledge generalization.

8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it can implement the long-term task planning method based on adaptive symbolic world construction as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the long-term task planning method based on adaptive symbolic world construction as described in any one of claims 1 to 6.

10. A computer electronic device, characterized in that, Including memory and processor; The memory is used to store computer programs; The processor is configured to implement, when executing the computer program, the long-term task planning method based on adaptive symbolic world construction as described in any one of claims 1 to 6.

Images