Long-range planning method and system of open body environment fusion model and symbol solver

Abstract

The invention provides a long-range planning method and system for an open body environment fusion model and a symbol solver. The long-range planning system comprises a sensing module, a parallel planning module and a supervision fine tuning module, the sensing module receives multi-modal input information of various sensors, generates a problem description file and transmits the problem description file to the parallel planning module; the parallel planning module generates a planning scheme according to the problem description file and the field description file, and verifies and outputs an optimal planning scheme; the supervision fine tuning module collects a planning scheme and a problem description file, and constructs a supervision fine tuning data set iteration parallel planning module. According to the method, the success rate of generating the long-range planning scheme in the open body environment is high, the real-time performance is good, the robustness of the generated plan is good, the number of execution steps is small, and the problems of inherent planning ability limitation of the vision-language model, environment part observability and intelligent body long-range planning in the open body environment are solved.

Description

Technical Field

[0001] This invention belongs to the field of computer application technology, specifically, it relates to a long-range planning method and system for an open embodied environment fusion model and symbolic solver, and in particular, a long-range planning method for an open embodied environment fusion visual language model, large language model and symbolic solver. Background Technology

[0002] The visual-language model (VLM), by introducing a pre-trained visual encoder and a visual-language modality fusion module, can perform visual question-answering reasoning by combining visual and text input. In open embodied scenarios, the introduction of the visual-language model enables agents to reason and plan based on natural language instructions and visual observations of the environment, thereby improving the agent's dynamic perception and reasoning ability of the environment.

[0003] Symbolic solver-based task and motion planning methods can solve complex long-range reasoning problems in embodied scenarios. Planners (such as Fast Downward) utilize PDDL domain files and problem files as formal inputs, enabling them to output sound and complete planning sequences. However, symbolic solver-based planning methods face the following challenges in open embodied scenarios: 1) they lack the ability to handle open-world semantics; 2) they rely on detailed and accurate environment models; and 3) the computational scale increases dramatically with problem complexity (number of objects, length of planning schemes, etc.).

[0004] For long-range planning problems of intelligent agents in open embodied scenarios, fusing a visual-language model with a PDDL-style planning solver can significantly enhance the model's performance in planning and reasoning. Multiple studies have transformed visual and textual inputs into formalized planning problem statements (e.g., PDDL form) using methods such as LatPlan (utilizing variational or state-action autoencoders), LLL+P, ViLaIn, and DKPROMPT. These methods can generate long-range optimal plans in most complex robotic tasks, achieving an average success rate of 66.5% in open embodied scenarios.

[0005] Hybrid neural-symbolic architectures offer an effective and reliable solution for enhancing the long-range planning capabilities of agents in open embodied scenarios. They can compensate for the shortcomings of visual-language models in long-range planning feasibility and address the issues of symbolic solvers lacking the ability to handle open-world semantics and relying on precise environment models. For example, integrated solutions like LLM-DP demonstrate a 96% task completion rate in embodied planning. Frameworks such as ThinkProgram reCtify and PEIL enhance the spatial and causal reasoning capabilities of visual-language models by alternately executing inference and actions. Compositional Foundation Models further separate language, visual, and action models to achieve hierarchical planning and incorporate visual cues for manipulation and grasping tasks. However, existing model-symbolic solver combinations are insufficient to fully address the long-range planning problem in open embodied scenarios.

[0006] 1. Computational efficiency and real-time issues: The complexity of problems increases dramatically in open embodied scenarios, the computational scale of symbolic programming solvers expands rapidly, and the execution speed of the sequential solution that simply connects the visual-language model and the symbolic programming solver is slow and cannot meet the real-time processing requirements in embodied scenarios.

[0007] 2. Robustness issues in long-range planning generation: The phased architecture system that sequentially connects the visual-language model and the symbolic planning solver has poor robustness. Failure in either the model phase or the symbolic planning phase will directly lead to planning generation failure, and there is no cross-phase restart mechanism.

[0008] 3. The inherent planning capability limitation of the visual-language model: Under the sequential architecture, the long-range planning capability of the visual-language model as a large model planner directly limits the success rate of long-range planning in the hybrid neural-symbolic system.

[0009] For example, the patent document "Training Method and System for Embodied Intelligent Task Planner Based on Multimodal Large Model" (CN118036750A) discloses a training method for embodied intelligent task planner based on multimodal large model. It enhances the dataset format, behavior tree structure and feasibility verification through image-text-robot motion trajectory, and achieves uniformity of task planning granularity and environmental assessment. It also optimizes the planning process by combining a breadth-first search algorithm. However, the tree structure has poor robustness and does not have a cross-stage restart mechanism.

[0010] The patent document "Visual Language Navigation Method and Device Based on Inference Link Autonomous Evolution Strategy" (CN117290479A) discloses a method that uses an inference link autonomous evolution strategy. It deploys a large language model on a cloud server and combines the functional description information of multiple program modules deployed in an intelligent robot to determine the module scheduling sequence, thereby achieving a more accurate visual language navigation task. However, due to the need to go to the cloud, the real-time performance is poor.

[0011] Therefore, there is still a need for a long-range planning method that can generate high success rates and good real-time performance in open embodied environments, and can solve the inherent planning capability limitations of visual-language models and the problem of environmental partial observability. Summary of the Invention

[0012] To address the shortcomings of existing technologies, the purpose of this invention is to provide a long-range planning method and system that integrates open embodied environment fusion models and symbolic solvers.

[0013] A long-range programming method based on an open embodied environment fusion model and symbolic solver provided by the present invention includes:

[0014] Step S1: Use a perception module to dynamically perceive the environment and integrate multimodal inputs, and output a planning problem description file;

[0015] Step S2: Construct a parallel planning module to generate planning schemes based on the planning problem description file, verify and output the best planning scheme;

[0016] Step S3: The supervised fine-tuning module collects planning schemes and problem description files, and constructs a supervised fine-tuning dataset iterative parallel planning module.

[0017] Preferably, step S1 includes:

[0018] Step S1.1: Use multiple sensors to acquire multimodal input information, acquire environmental visual input information through a camera, perform open vocabulary target detection and semantic segmentation, perform semantic annotation and image enhancement on the image, acquire semantic input information through an audio device, convert it into text information, and write or review it into a domain description file;

[0019] Step S1.2: Align all input information through a vision-language model, combine the domain description file to understand open-world semantics and solve the observability problem of the scene, and output a planning problem description file in PDDL form;

[0020] Step S1.3: Use planning verification software to check whether the planning problem description file conforms to the PDDL syntax specification.

[0021] If the error does not conform to the syntax rules, the historical error information is recorded, and the problem description file is generated again based on the error feedback. If the error conforms to the syntax rules, the verified planning problem description file is output to the parallel planning module.

[0022] Preferably, in step S2, a large language model is used as a fast planning module to generate a planning scheme based on the planning problem description file, and a planning scheme is proposed in the planning-feedback-re-prompt generation loop;

[0023] The symbolic solver is used as a slow planning module to search for planning solutions in the given search space.

[0024] The fast and slow planning central scheduler signals control the fast and slow planning modules. If the slow planning module finds a planning solution earlier, the planning solution generated by the slow planning module is adopted as the best planning solution. If the fast planning module outputs a planning solution earlier, the planning solution proposed by the fast planning module is adopted as the best planning solution.

[0025] In step S3, the planning schemes generated by the slow planning module, the planning problem description file, and the error feedback generated when the fast planning module generates the planning schemes are collected, and a supervised fine-tuning dataset is constructed by combining them to iteratively optimize the fast planning module.

[0026] The slow planning module is executed alternately offline to generate, collect, and supervise fine-tuning of planning schemes.

[0027] Preferably, step S2 includes:

[0028] The fast planning step uses a large language model as a fast planning module, takes a domain description file and a planning problem description file as input, and outputs a planning scheme in PDDL form.

[0029] The verification module checks the legality of the planning problem description file and the feasibility of the planning scheme, and constructs a planning-error feedback-planning regeneration loop;

[0030] The slow planning step involves using a traditional planning solver as a slow planning module, searching for and outputting the optimal planning solution based on the domain description file and the planning problem description file.

[0031] The scheduling process involves using a fast and slow planning central scheduler, coordinating the outputs of the fast and slow planning modules, and selecting the optimal planning scheme that meets real-time requirements.

[0032] The domain description file is provided by a large model and then reviewed by human experts, or it is written directly by human experts.

[0033] Preferably, the fast planning step includes:

[0034] Step S2.1.1: The fast planning module outputs a planning scheme based on the planning problem description file and domain description file output by the perception module;

[0035] Step S2.1.2: For the generated planning scheme, use planning verification software as a verification module to check whether the planning scheme conforms to the PDDL syntax specification;

[0036] If the plan conforms to the syntax, the verified plan is mapped to an executable robot primitive, and a fast planning success signal is output to the fast and slow planning central scheduler. If the plan does not conform to the syntax, historical error information is recorded, and a new plan is generated and combined with error feedback.

[0037] Preferably, the slow planning step includes:

[0038] Step S2.2.1: Use the traditional planning solver as a slow planning module to search for the optimal planning scheme;

[0039] Step S2.2.2: Control the slow planning module through the fast and slow planning central scheduler signal. If the fast planning success signal is received from the fast and slow planning central scheduler, the scheduling step is executed. If the slow planning module does not receive the fast planning success signal from the fast and slow planning central scheduler, it sends the slow planning success signal to the fast and slow planning central scheduler and maps the generated best planning scheme to the executable robot primitive.

[0040] Preferably, in the scheduling step, the fast and slow planning central scheduler signals to control the fast planning module to determine whether the number of generation times is less than the set maximum number of planning generation times. If so, the fast planning step is executed; if not, a fast planning failure signal is sent to the fast and slow planning central scheduler, the current planning problem description file and error feedback information are recorded, and the slow planning module is notified to collect planning schemes.

[0041] The fast and slow planning central scheduler judges the received signals. If it receives a fast planning success signal, the planning scheme of the fast planning module is verified and the fast planning success signal is forwarded to the slow planning module.

[0042] If a slow planning success signal is received, the slow planning success signal is forwarded to the fast planning module to stop the generation process, and the current planning problem description file and planning scheme are recorded. The best planning scheme is saved to the supervision and fine-tuning module.

[0043] A long-range planning system for an open embodied environment fusion model and symbolic solver provided by the present invention includes: a perception module, a parallel planning module, and a supervised fine-tuning module;

[0044] The perception module receives multimodal input information from various sensors, generates a problem description file, and transmits it to the parallel planning module.

[0045] The parallel planning module generates planning schemes based on the problem description file and the domain description file, verifies and outputs the best planning scheme;

[0046] The supervised fine-tuning module collects planning schemes and problem description files, and constructs a supervised fine-tuning dataset for iterative parallel planning.

[0047] Preferably, the perception module adopts a vision-language model, and the supervised fine-tuning module collects planning schemes in an offline manner.

[0048] The parallel planning module uses a large language model as the fast planning module and a symbolic solver as the slow planning module. The fast and slow planning central scheduler works together to generate the optimal plan. The fast and slow planning central scheduler is started first, and then the fast planning module and the slow planning module are started synchronously.

[0049] Preferably, the perception module uses multiple sensors to acquire multimodal input information, acquires environmental visual input information through a camera, performs open-vocabulary target detection and semantic segmentation, performs semantic annotation and image enhancement on images, acquires semantic input information through an audio device, converts it into text information, and writes or reviews it into a domain description file.

[0050] By aligning all input information through a visual-language model, combining domain description files to understand open-world semantics and solve the scene partial observability problem, the output planning problem description file in PDDL form is generated.

[0051] Use planning verification software to check whether the planning problem description document conforms to the PDDL syntax specification:

[0052] If the error does not conform to the syntax rules, the historical error information is recorded, and the problem description file is generated again based on the error feedback. If the error conforms to the syntax rules, the verified planning problem description file is output to the parallel planning module.

[0053] The parallel planning module uses a large language model as a fast planning module to generate planning schemes based on the planning problem description file and proposes planning schemes in the planning-feedback-re-prompt generation loop.

[0054] The symbolic solver is used as a slow planning module to search for planning solutions in the given search space.

[0055] The fast and slow planning central scheduler signals control the fast and slow planning modules. If the slow planning module finds a planning solution earlier, the planning solution generated by the slow planning module is adopted as the best planning solution. If the fast planning module outputs a planning solution earlier, the planning solution proposed by the fast planning module is adopted as the best planning solution.

[0056] The supervised fine-tuning module collects the planning schemes generated by the slow planning module, the planning problem description file, and the error feedback generated when the fast planning module generates the planning schemes. It combines these to construct a supervised fine-tuning dataset and performs supervised iteration on the fast planning module.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] 1. The perception and planning separation architecture of the present invention utilizes a visual-language model to participate only in the perception stage, separated from the subsequent planning module, which solves the inherent planning capability limitations of the visual-language model and the observability problem of the environment.

[0059] 2. The supervised fine-tuning data collection method of the present invention collects high-quality planning data verified by traditional planners offline in the planning generation pipeline for supervised fine-tuning of VLM / LLM, thus solving the problem of the inherent planning capability limitation of large models.

[0060] 3. The fast and slow parallel planning module of the present invention uses a large language model as a fast planner and a symbolic solver as a slow planner, and the fast and slow parallel planning results in a high success rate and good real-time performance in generating long-range planning schemes in an open embodied environment. In addition, the generated plan has good robustness and fewer execution steps, thus solving the long-range planning problem of intelligent agents in an open embodied environment. Attached Figure Description

[0061] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0062] Figure 1 A schematic diagram of the long-range planning method for the open embodied environment fusion model and symbolic solver provided by the present invention;

[0063] Figure 2 This is a flowchart illustrating the visual-language model as a perception module in an embodiment of the present invention.

[0064] Figure 3 This is a flowchart illustrating the large language model as a fast planning module in an embodiment of the present invention.

[0065] Figure 4 This is a flowchart illustrating the traditional planning solver as a slow planning module in an embodiment of the present invention.

[0066] Figure 5 This is a flowchart illustrating the offline collection of planning data to supervise and fine-tune the model in an embodiment of the present invention. Detailed Implementation

[0067] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0068] This invention provides a long-range planning method based on an open embodied environment fusion model and a symbolic solver, to address, or at least partially address, the technical problems of lack of long-range planning capability, low computational efficiency, and poor real-time performance in open embodied scenarios. A system architecture of perception and parallel planning is proposed, utilizing the hierarchical differences between visual-language models, large language models, and traditional symbolic planning solvers in terms of inference speed, computational resource consumption, and inference and perception capabilities to construct a hierarchical planning system, specifically including:

[0069] Step S1: Utilize a visual-language model to dynamically perceive the environment, integrate multimodal inputs, and output a problem description.

[0070] Specifically, a vision-language model is used as the perception module: leveraging the multimodal alignment, visual, and spatial reasoning capabilities of the Visual Model (VLM) to understand open-world semantics, address some observability issues in the scene, dynamically perceive the open embodied environment, provide an environment model for the subsequent planner, and integrate multimodal information to obtain a problem description in PDDL form. This includes:

[0071] Step S1.1: Acquire multimodal input information using sensors; acquire environmental visual input through an RGB-D camera; in the visual image processing part: use Grounding-SAM for open-vocabulary target detection and semantic segmentation; use DINOv2 for semantic annotation and image enhancement; acquire semantic input of human commands through an audio device and convert it into text information.

[0072] Step S1.2: Align multimodal inputs such as images and text through a visual-language model, combine with domain description files, utilize its spatial reasoning capabilities to understand open-world semantics, solve the observable problems of the scene, and output a planning problem description in PDDL form.

[0073] The domain description file needs to be provided in advance by a large model and then reviewed by human experts, or it can be written directly by human experts.

[0074] Step S1.3, as follows Figure 2 As shown, a reasoning-error feedback-re-generation loop is constructed: For the planning problem description file output by the visual-language model, the planning verification software VAL is used to check whether the description file conforms to the PDDL syntax specification; if the visual-language model output does not conform to the syntax specification, historical error information is recorded, and the visual-language model is re-prompted to generate the problem description file based on the error feedback; if it conforms to the syntax specification, the verified planning problem description file (PD) is output to the parallel planning module.

[0075] Step S2: Construct a fast and slow parallel planning module, using a large language model as the fast planner and a symbolic solver as the slow planner to collaboratively generate the optimal plan.

[0076] Specifically, a fast and slow parallel planning module is constructed by using a large language model and a symbolic solver as parallel planning modules: the large language model planner acts as the fast planner, directly generating planning schemes based on the planning problem description, and proposing reasonable plans in the planning-feedback-re-prompt generation loop. At the same time, the symbolic solver acts as the slow planner, searching for the best plan in the given search space. If it finds the best plan earlier than the LLM planner (i.e., the fast planner of the large language model), the agent accepts the plan generated by the symbolic solver, and the fast planning module and the slow planning module start synchronously.

[0077] The optimal plan is the fastest solution that can complete the task.

[0078] Fast and slow parallel collaborative planning, which balances the real-time nature and feasibility of planning, specifically includes:

[0079] Step S2.1: The large language model, as the fast planning module, takes the domain description file and the planning problem description file generated by the perception module as input, and outputs a planning scheme in PDDL form.

[0080] The traditional planning verification software VAL is used as a planning feasibility verifier, i.e., the verification module, to check the legality of the PDDL problem description generated by VLM and the feasibility of the plan generated by LLM, and to build a planning-error feedback-plan regeneration loop.

[0081] By combining existing traditional planning validators with an error feedback-re-reminder mechanism, the legality and feasibility of the problem description are ensured.

[0082] In more preferred embodiments, step S2.1 includes:

[0083] Step S2.1.1: Large Language Model (LLM) as Fast Planning Module: Based on the planning problem description file and domain description file output by the perception module as input, the Large Language Model directly outputs the planning scheme.

[0084] Step S2.1.2: Constructing an Error Feedback-Re-prompt Generation Loop: For the generated planning scheme, use the planning verification software VAL to check whether the planning scheme conforms to the PDDL syntax specification; if the planning scheme does not conform to the syntax specification, record the historical error information, and combine the error feedback with the large language model to generate a new planning scheme; if it conforms to the syntax specification, map the verified PDDL planning scheme to executable robot primitives, such as... Figure 3 As shown.

[0085] Step S2.2: The traditional planning solver, as a slow planning module, searches for the optimal planning scheme, i.e. the best planning scheme, based on the domain description file and the problem description file generated by the vision-language model.

[0086] Specifically, in the fast-slow parallel planning architecture, a traditional planning solver is used as the slow planning module. The traditional planner searches for the optimal planning scheme based on the given domain description and the planning problem description. The system, in conjunction with the outputs of the fast and slow planning modules, selects the optimal planning scheme that meets real-time requirements. This includes:

[0087] Step S2.2.1: Traditional programming solver as slow planning module: As input, the traditional programming solver is used to search for the optimal planning scheme.

[0088] Step S2.2.2, as follows Figure 4 As shown, the execution logic of the slow planning module is controlled by the signal from the central scheduler for fast and slow planning:

[0089] If a "fast planning successful" signal is received from the fast and slow planning central scheduler, the slow planning module will terminate prematurely.

[0090] If the symbolic programming solver finds the optimal planning solution before the fast module, it sends a "slow planning successful" signal to the fast and slow programming central scheduler, thus terminating the fast programming module's generation process.

[0091] Step S2.3: Construct a fast and slow parallel collaborative planning module, which coordinates the fast and slow modules through a fast and slow planning central scheduler; the large language model provides the real-time performance of the fast planning module, while the traditional planning solver provides the accuracy and reliability of the slow planning module.

[0092] Specifically, such as Figure 3 As shown, the fast and slow planning central scheduler starts first, and controls the execution logic of the fast planning module through the number of planning generation times and the signals from the fast and slow planning central scheduler:

[0093] Set a maximum number of planning attempts. When the number of planning failures exceeds the threshold, send a "Fast Planning Failure" signal to the fast and slow planning central scheduler, record the current planning problem description file and error feedback information, and notify the slow planning module to collect planning solutions for subsequent monitoring and fine-tuning.

[0094] If the fast and slow planning central scheduler receives a "fast planning successful" signal, the planning scheme is verified and the slow planning module ends early; if the fast and slow planning central scheduler receives a "slow planning successful" signal, it forwards the signal and stops the fast planning module generation process.

[0095] If the slow module eventually finds the optimal planning scheme, it saves the planning scheme and related information of the fast module for subsequent supervised fine-tuning of the large language model. At the same time, it maps the generated PDDL planning scheme to executable robot primitives.

[0096] Step S3: Collect high-quality planning data verified by traditional planners offline for iterative supervised fine-tuning of large models (VLM / LLM) to improve the reasoning and inherent long-range planning capabilities of large models in open embodied scenarios.

[0097] by Figure 5 For example, offline collection of well-defined planning data validated by traditional planners is used as a Supervised Fine-Tuning (SFT) dataset to iteratively improve the reasoning and planning capabilities of visual-language models and large language models. Specifically, this includes:

[0098] Step S3.1: Collect good planning data verified by traditional solvers offline as a supervised fine-tuning dataset.

[0099] Specifically, high-quality planning data is collected offline: high-quality planning schemes generated by the traditional solver of the slow planning module are collected, while error feedback of the planning generated by the corresponding fast planning module large language model is saved, and the supervised fine-tuning dataset is constructed by combining them for supervised fine-tuning of the large language model.

[0100] Step S3.2: Iteratively and alternately execute the offline planning data generation and collection stage and the large model supervised fine-tuning stage to continuously improve the inherent reasoning and planning capabilities of the large model.

[0101] This invention provides a long-range programming system that integrates an open embodied environment fusion model and a symbolic solver. Figure 1 For example, it adopts a hybrid neural-symbolic architecture, which integrates a visual-language model, a large language model, and a symbolic programming solver module. It uses a perception and parallel programming system architecture to form a hierarchical reasoning system.

[0102] The perception module receives multimodal input information from various sensors, generates a problem description file, and transmits it to the parallel planning module.

[0103] The parallel planning module generates planning schemes based on the problem description file and the domain description file, verifies and outputs the best planning scheme;

[0104] The supervised fine-tuning module collects planning schemes and problem description files, and constructs a supervised fine-tuning dataset for iterative parallel planning.

[0105] Specifically, the perception module adopts a vision-language model, and the supervised fine-tuning module collects planning schemes offline.

[0106] The parallel planning module uses a large language model as the fast planning module and a symbolic solver as the slow planning module. The fast and slow planning central scheduler works together to generate the optimal plan. The fast and slow planning central scheduler is started first, and then the fast planning module and the slow planning module are started synchronously.

[0107] In more preferred embodiments, the perception module uses multiple sensors to acquire multimodal input information, acquires environmental visual input information through a camera, performs open-vocabulary target detection and semantic segmentation, performs semantic annotation and image enhancement on images, acquires semantic input information through an audio device, converts it into text information, and writes or reviews it into a domain description file.

[0108] By aligning all input information through a visual-language model, combining domain description files to understand open-world semantics and solve the scene partial observability problem, the output planning problem description file in PDDL form is generated.

[0109] Use planning verification software to check whether the planning problem description document conforms to the PDDL syntax specification:

[0110] If the error does not conform to the syntax rules, the historical error information is recorded, and the problem description file is generated again based on the error feedback. If the error conforms to the syntax rules, the verified planning problem description file is output to the parallel planning module.

[0111] The parallel planning module uses a large language model as a fast planning module to generate planning schemes based on the planning problem description file and proposes planning schemes in the planning-feedback-re-prompt generation loop.

[0112] The symbolic solver is used as a slow planning module to search for planning solutions in the given search space.

[0113] The fast and slow planning central scheduler signals control the fast and slow planning modules. If the slow planning module finds a planning solution earlier, the planning solution generated by the slow planning module is adopted as the best planning solution. If the fast planning module outputs a planning solution earlier, the planning solution proposed by the fast planning module is adopted as the best planning solution.

[0114] The supervised fine-tuning module collects the planning schemes generated by the slow planning module, the planning problem description file, and the error feedback generated when the fast planning module generates the planning schemes. It combines these to construct a supervised fine-tuning dataset and performs supervised iteration on the fast planning module.

[0115] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A long-range programming method that integrates an open embodied environment model and a symbolic solver, characterized in that, include: Step S1: Use a perception module to dynamically perceive the environment and integrate multimodal inputs, and output a planning problem description file; Step S2: Construct a parallel planning module to generate planning schemes based on the planning problem description file, verify and output the best planning scheme; Step S3: The supervised fine-tuning module collects planning schemes and problem description files, and constructs a supervised fine-tuning dataset iterative parallel planning module.

2. The long-range programming method of the open embodied environment fusion model and symbolic solver according to claim 1, characterized in that, Step S1 includes: Step S1.1: Use multiple sensors to acquire multimodal input information, acquire environmental visual input information through a camera, perform open vocabulary target detection and semantic segmentation, perform semantic annotation and image enhancement on the image, acquire semantic input information through an audio device, convert it into text information, and write or review it into a domain description file; Step S1.2: Align all input information through a vision-language model, combine the domain description file to understand open-world semantics and solve the observability problem of the scene, and output a planning problem description file in PDDL form; Step S1.3: Use planning verification software to check whether the planning problem description file conforms to the PDDL syntax specification. If the error does not conform to the syntax rules, the historical error information is recorded, and the problem description file is generated again based on the error feedback. If the error conforms to the syntax rules, the verified planning problem description file is output to the parallel planning module.

3. The long-range programming method based on the open embodied environment fusion model and symbolic solver according to claim 1, characterized in that, In step S2, a large language model is used as a fast planning module to generate a planning scheme based on the planning problem description file, and a planning scheme is proposed in the planning-feedback-re-prompt generation loop. The symbolic solver is used as a slow planning module to search for planning solutions in the given search space. The fast and slow planning central scheduler signals control the fast and slow planning modules. If the slow planning module finds a planning solution earlier, the planning solution generated by the slow planning module is adopted as the best planning solution. If the fast planning module outputs a planning solution earlier, the planning solution proposed by the fast planning module is adopted as the best planning solution. In step S3, the planning schemes generated by the slow planning module, the planning problem description file, and the error feedback generated when the fast planning module generates the planning schemes are collected, and a supervised fine-tuning dataset is constructed by combining them to iteratively optimize the fast planning module. The slow planning module is executed alternately offline to generate, collect, and supervise fine-tuning of planning schemes.

4. The long-range programming method of the open embodied environment fusion model and symbolic solver according to claim 1, characterized in that, Step S2 includes: The fast planning step uses a large language model as a fast planning module, takes a domain description file and a planning problem description file as input, and outputs a planning scheme in PDDL form. The verification module checks the legality of the planning problem description file and the feasibility of the planning scheme, and constructs a planning-error feedback-planning regeneration loop; The slow planning step involves using a traditional planning solver as a slow planning module, searching for and outputting the optimal planning solution based on the domain description file and the planning problem description file. The scheduling process involves using a fast and slow planning central scheduler, coordinating the outputs of the fast and slow planning modules, and selecting the optimal planning scheme that meets real-time requirements. The domain description file is provided by a large model and then reviewed by human experts, or it is written directly by human experts.

5. The long-range programming method for the open embodied environment fusion model and symbolic solver according to claim 4, characterized in that, The fast planning steps include: Step S2.1.1: The fast planning module outputs a planning scheme based on the planning problem description file and domain description file output by the perception module; Step S2.1.2: For the generated planning scheme, use planning verification software as a verification module to check whether the planning scheme conforms to the PDDL syntax specification; If the plan conforms to the syntax, the verified plan is mapped to an executable robot primitive, and a fast planning success signal is output to the fast and slow planning central scheduler. If the plan does not conform to the syntax, historical error information is recorded, and a new plan is generated and combined with error feedback.

6. The long-range programming method for the open embodied environment fusion model and symbolic solver according to claim 4, characterized in that, The slow planning steps include: Step S2.2.1: Use the traditional planning solver as a slow planning module to search for the optimal planning scheme; Step S2.2.2: Control the slow planning module through the fast and slow planning central scheduler signal. If the fast planning success signal is received from the fast and slow planning central scheduler, the scheduling step is executed. If the slow planning module does not receive the fast planning success signal from the fast and slow planning central scheduler, it sends the slow planning success signal to the fast and slow planning central scheduler and maps the generated best planning scheme to the executable robot primitive.

7. The long-range programming method for the open embodied environment fusion model and symbolic solver according to claim 4, characterized in that, In the scheduling step, the fast and slow planning central scheduler signals to control the fast planning module to determine whether the number of generation times is less than the set maximum number of planning generation times. If so, the fast planning step is executed; otherwise, a fast planning failure signal is sent to the fast and slow planning central scheduler, the current planning problem description file and error feedback information are recorded, and the slow planning module is notified to collect planning solutions. The fast and slow planning central scheduler judges the received signals. If it receives a fast planning success signal, the planning scheme of the fast planning module is verified and the fast planning success signal is forwarded to the slow planning module. If a slow planning success signal is received, the slow planning success signal is forwarded to the fast planning module to stop the generation process, and the current planning problem description file and planning scheme are recorded. The best planning scheme is saved to the supervision and fine-tuning module.

8. A long-range programming system that integrates an open embodied environment model and a symbolic solver, characterized in that, include: The module consists of a perception module, a parallel planning module, and a supervised fine-tuning module. The perception module receives multimodal input information from various sensors, generates a problem description file, and transmits it to the parallel planning module. The parallel planning module generates planning schemes based on the problem description file and the domain description file, verifies and outputs the best planning scheme; The supervised fine-tuning module collects planning schemes and problem description files, and constructs a supervised fine-tuning dataset for iterative parallel planning.

9. The long-range planning system of the open embodied environment fusion model and symbolic solver according to claim 8, characterized in that, The perception module uses a visual-language model, and the supervised fine-tuning module collects planning schemes offline. The parallel planning module uses a large language model as the fast planning module and a symbolic solver as the slow planning module. The fast and slow planning central scheduler works together to generate the optimal plan. The fast and slow planning central scheduler is started first, and then the fast planning module and the slow planning module are started synchronously.

10. The long-range planning system of the open embodied environment fusion model and symbolic solver according to claim 8, characterized in that, The perception module uses multiple sensors to acquire multimodal input information, acquires environmental visual input information through a camera, performs open-vocabulary target detection and semantic segmentation, performs semantic annotation and image enhancement on images, acquires semantic input information through audio devices, converts it into text information, and writes or reviews it into a domain description file. By aligning all input information through a visual-language model, combining domain description files to understand open-world semantics and solve the scene partial observability problem, the output planning problem description file in PDDL form is generated. Use planning verification software to check whether the planning problem description document conforms to the PDDL syntax specification: If it does not conform to the syntax rules, record the historical error information and generate the problem description file again based on the error feedback. If it conforms to the syntax rules, output the verified planning problem description file to the parallel planning module. The parallel planning module uses a large language model as a fast planning module to generate planning schemes based on the planning problem description file and proposes planning schemes in the planning-feedback-re-prompt generation loop. The symbolic solver is used as a slow planning module to search for planning solutions in the given search space. The fast and slow planning central scheduler signals control the fast and slow planning modules. If the slow planning module finds a planning solution earlier, the planning solution generated by the slow planning module is adopted as the best planning solution. If the fast planning module outputs a planning solution earlier, the planning solution proposed by the fast planning module is adopted as the best planning solution. The supervised fine-tuning module collects the planning schemes generated by the slow planning module, the planning problem description file, and the error feedback generated when the fast planning module generates the planning schemes. It combines these to construct a supervised fine-tuning dataset and performs supervised iteration on the fast planning module.

Images