Optimal global planning method and system for long-range task

Abstract

The invention discloses an optimal global planning method and system for a long-range task, and relates to the technical field of artificial intelligence, and the method comprises the steps: generating a plurality of initial global plans for the long-range task; determining the initial global plan with the task completion rate reaching the standard as a high-quality global plan; performing optimization training on the initial planner model based on the high-quality global plan to obtain an intermediate planner model; generating a target global plan based on the intermediate planner model, and giving a corresponding reward score to the intermediate planner model of which the actual task completion rate reaches the standard; performing iterative training optimization on the intermediate planner model by taking the reward score as a signal to obtain a final global planner; and generating a plurality of to-be-planned global plans for the to-be-planned long-range tasks, inputting the to-be-planned global plans into the final global planner to output the optimal global plan applied to the long-range tasks, thereby facilitating solving the problem that the optimal global planning of the long-range tasks cannot be realized in the prior art.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an optimal global planning method and system for long-term tasks. Background Technology

[0002] In the field of artificial intelligence, agents based on large language models (LLM) are increasingly used in the processing of "long-term tasks" with multi-step and long-term planning. However, due to the limitations of the model's autoregressive characteristics, agents generally lack global planning capabilities, often exhibiting "brainless trial and error," and are prone to planning illusions, generating unreasonable or erroneous action sequences, which seriously affect the quality of task completion.

[0003] To address this issue, existing technologies have developed two core approaches: one is implicit planning, which directly trains the agent through supervised fine-tuning (SFT) or reinforcement learning (RL), enabling it to implicitly learn planning capabilities from expert trajectories or environmental interactions. However, this approach focuses on local planning and has limited effectiveness for long-term tasks. Furthermore, SFT relies on a large amount of expert-labeled data, and RL suffers from reward sparsity and latency issues, resulting in low training efficiency. The other approach is explicit planning, which provides explicit guidance by training an independent global planner or building an external knowledge base. However, this approach requires manually writing and modifying plan data and building a knowledge base, which is costly and difficult to quickly transfer to new tasks and environments.

[0004] In summary, existing technologies have not yet developed an automated planner training scheme that combines effective global planning capabilities, no human intervention, high training efficiency, and flexible adaptation, making it difficult to meet the practical application needs of intelligent agents for long-term tasks.

[0005] Therefore, there is an urgent need for an optimal global planning method for long-term tasks. Summary of the Invention

[0006] In view of this, the present invention proposes an optimal global planning method and system for long-term tasks, which can realize the optimal global planning method for long-term tasks.

[0007] To achieve the above objectives, the present invention provides the following technical solution: An optimal global planning method for long-term tasks includes: Generate several initial global plans for long-term tasks, including multi-step robotic operations and intelligent office processes; The initial global plan that achieves the target task completion rate is identified as a high-quality global plan; Based on the high-quality global plan, the initial planner model is optimized and trained to obtain the intermediate planner model; Based on the intermediate planner model, a target global plan is generated, and the intermediate planner model that achieves the target completion rate in the actual task is given a corresponding reward score. The intermediate planner model is iteratively trained and optimized using the reward score as a signal to obtain the final global planner; Several global plans are generated for the long-term task to be planned. The global plans to be planned are input into the final global planner to output the optimal global plan applied to the long-term task.

[0008] Based on the above technical solution, the present invention can be further improved as follows: Optionally, the generation of several initial global plans for the long-term task includes: The long-term task is input into a pre-trained large language model, which then generates several initial global plans based on the historical successful execution trajectories corresponding to the task instructions.

[0009] Optionally, before the step of determining the initial global plan that achieves the task completion rate target as a high-quality global plan, the following steps are included: Introduce at least two homogeneous executor agents. The two homogeneous executor agents have the same model architecture and pre-training background, but different task completion capabilities.

[0010] Optionally, determining the initial global plan that achieves the task completion rate target as a high-quality global plan includes: The long-term task is executed by the same executor agent based on each initial global plan, and the initial global plan that achieves the task completion rate is determined as a high-quality global plan.

[0011] Optionally, determining the initial global plan that achieves the task completion rate target as a high-quality global plan includes: Determine whether the initial global plan can simultaneously improve the task completion rate of both novice and expert agents. If so, the initial global plan is determined to be a high-quality global plan.

[0012] Optionally, the reward score given to the intermediate planner model that achieves the target actual task completion rate includes: Obtain the actual task completion rate of each of the aforementioned homogeneous executor agents when using the target global plan; Obtain the baseline task completion rate for each of the aforementioned homogeneous executor agents when the target global plan is not used; If the actual task completion rate is greater than the baseline task completion rate, the intermediate planner model is given a corresponding reward score.

[0013] Optionally, the optimal global planning method for the long-range task further includes: The intermediate planner model is iteratively trained and optimized using a reinforcement learning strategy optimization algorithm based on the reward score.

[0014] An optimal global programming system for long-duration tasks includes: The plan generation module is used to generate several initial global plans for long-term tasks, including multi-step robot operations and intelligent office processes. The filtering module is used to identify initial global plans that meet the task completion rate target as high-quality global plans. The intermediate model training module is used to optimize and train the initial planner model based on the high-quality global plan to obtain the intermediate planner model. The reward module is used to generate a target global plan based on the intermediate planner model and give corresponding reward scores to the intermediate planner model that achieves the target completion rate in the actual task. The final model optimization module is used to iteratively train and optimize the intermediate planner model using the reward score as a signal to obtain the final global planner. The optimal task acquisition module is used to generate several global plans to be planned for a long-term task to be planned, and input the global plans to be planned into the final global planner to output the optimal global plan applied to the long-term task.

[0015] An electronic device includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the method described herein.

[0016] A non-transitory computer-readable storage medium having a computer program stored thereon, the computer program implementing the steps of the method when executed by a processor.

[0017] The present invention has the following advantages: The optimal global planning method for long-term tasks in this invention provides a reliable foundation for training the initial planner model by selecting high-quality plans based on task completion rates. Combined with reward-driven iterative optimization, it significantly improves the global planning capability of the intermediate planner model to obtain the final global planner. This avoids the defects of data dependence or reward sparsity in traditional methods and shortens the model convergence cycle. Attached Figure Description

[0018] For illustrative and not limiting purposes, the present invention will now be described in conjunction with embodiments and accompanying drawings, wherein: Figure 1 This is a flowchart illustrating the optimal global planning method for long-range tasks in an embodiment of the present invention. Figure 2 This is a schematic diagram of the main components of the optimal global planning system for long-range tasks in an embodiment of the present invention; Figure 3This is a schematic diagram of the physical structure of the electronic device provided by the present invention. Detailed Implementation

[0019] To enable those skilled in the art to better understand the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0020] It should be noted that the terms "first," "second," etc., in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be used interchangeably where appropriate for the embodiments of the invention described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0021] It should be noted that, where there is no conflict, the embodiments and features of the present invention can be combined with each other. The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0022] Figure 1 This is a flowchart illustrating the optimal global planning method for long-range tasks in an embodiment of the present invention, as shown below. Figure 1 As shown, the optimal global planning method for long-term tasks provided in this embodiment of the invention includes the following steps S101 to S106.

[0023] S101 generates several initial global plans for long-term tasks.

[0024] The long-term tasks include multi-step robotic operations and intelligent office processes; For long-term tasks (such as intelligent office process automation, multi-step robot operations, etc.), 10-20 initial global plans are automatically generated by using a pre-trained large language model (such as DeepSeek-V3.1-Think), combined with the task's instruction description (such as "complete file organization - data statistics - report generation - email sending") and historical successful execution trajectories.

[0025] S102, the initial global plan that meets the task completion rate target is identified as a high-quality global plan; Introduce at least two homogeneous executor agents (such as novice and expert agents fine-tuned based on the same base model, with identical model architecture and pre-training background, but with gradient differences in task completion capabilities), and use each initial global plan as a guiding signal to drive the two agents to execute the long-term task respectively.

[0026] Calculate the task completion rate corresponding to each initial global plan, set a preset threshold (such as 80%), and determine the initial global plans with a completion rate not lower than the threshold as high-quality global plans, and remove low-quality plans that do not meet the standard.

[0027] S103, the initial planner model is optimized and trained based on a high-quality global plan to obtain an intermediate planner model.

[0028] A basic language model (such as lightweight LLM) that has not undergone special training for long-term task planning is selected as the initial planner model, and the high-quality global plan obtained in step S101 is used as the training data.

[0029] The initial planner model is optimized and trained using supervised fine-tuning (SFT) to adjust the model parameters to learn the planning logic and step arrangement of high-quality plans. After training, an intermediate planner model with basic global planning capabilities is output.

[0030] S104 generates a target global plan based on the intermediate planner model, and gives corresponding reward scores to the intermediate planner model that achieves the target completion rate in the actual task.

[0031] Input new long-term task instructions into the intermediate planner model, driving the model to generate a target global plan for that task.

[0032] The same executor agent from step S101 is still used to execute tasks according to the target global plan and the actual task completion rate is recorded; at the same time, the baseline completion rate when the agent does not use the target global plan is obtained.

[0033] Set reward rules: If the actual task completion rate is higher than the baseline completion rate (or higher than the preset target threshold), the intermediate planner model will be given a reward score of 1 point; if it is not met, the reward score will be 0 points.

[0034] S105 uses the reward score as a signal to iteratively train and optimize the intermediate planner model to obtain the final global planner.

[0035] A reinforcement learning optimization algorithm (such as the GRPO algorithm) is used to iteratively train and optimize the intermediate planner model, with the reward score generated in step S103 as the core reward signal.

[0036] During the iteration process, the model continuously learns the characteristics of "high-reward plans," adjusts its planning strategy, and gradually improves the rationality and effectiveness of the global plan until the model converges and outputs the final global planner. This planner can directly provide planning guidance for various long-term task agents without additional adaptation or adjustment.

[0037] S106 generates several global plans to be planned for the long-term task to be planned, and inputs the global plans to be planned into the final global planner to output the optimal global plan applied to the long-term task.

[0038] The planner trained using this invention provides high-quality global guidance for various executor agents (including open-source and closed-source models), enabling them to achieve state-of-the-art performance on multiple long-term task benchmarks. The training process of this invention is fully automated, requiring no manual intervention, and reduces training costs by 8 times compared to traditional RL training methods. Under the guidance of the planner, agents can complete tasks with fewer interaction steps, improving task execution efficiency and speed. The entire training process, from data synthesis and selection to model optimization, requires no manual annotation or modification, significantly reducing application costs and making it easy to extend and migrate. The trained planner is a standalone, plug-and-play module that can be flexibly used with any new executor agent without retraining the executor itself, demonstrating strong versatility.

[0039] This invention proposes for the first time a global planner training framework combining cold-start SFT and RL optimization. This framework is fully automated, addressing the core pain points of existing technologies in terms of efficiency and reliance on manual intervention. It also innovatively proposes a data filtering method that eliminates the need for manual annotation. By evaluating the universal gain of a plan across multiple homogeneous agents with different ability levels, it automatically and efficiently identifies and retains high-quality training data. Furthermore, it designs a novel, unbiased reinforcement learning reward signal. By measuring the real ability gain a plan brings to multiple homogeneous agents and combining it with execution efficiency, it provides more accurate and effective guidance for planner optimization, thereby training a planner with stronger generalization capabilities.

[0040] Figure 2 This is a schematic diagram illustrating the main components of the optimal global planning system for long-range tasks in an embodiment of the present invention. For example... Figure 2 As shown, the optimal global planning system 1 for long-term tasks provided in this embodiment of the invention includes a plan generation module 10, a screening module 20, an intermediate model training module 30, a reward module 40, a final model optimization module 50, and an optimal task acquisition module 60.

[0041] The plan generation module 10 is used to generate several initial global plans for long-term tasks, including multi-step robot operations and intelligent office processes. The filtering module 20 is used to identify initial global plans that meet the task completion rate target as high-quality global plans; The intermediate model training module 30 is used to optimize and train the initial planner model based on the high-quality global plan to obtain the intermediate planner model. Reward module 40 is used to generate a target global plan based on the intermediate planner model and give corresponding reward scores to the intermediate planner model that achieves the target completion rate of the actual task. The final model optimization module 50 is used to iteratively train and optimize the intermediate planner model using the reward score as a signal to obtain the final global planner. The optimal task acquisition module 60 is used to generate several global plans to be planned for the long-term task to be planned, and input the global plans to be planned into the final global planner to output the optimal global plan applied to the long-term task.

[0042] Figure 3 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as... Figure 3 As shown, the electronic device 70 includes: a processor 701, a memory 702, and a bus 703; The processor 701 and the memory 702 communicate with each other via the bus 703. The processor 701 is used to call program instructions in the memory 702 to execute the methods provided in the above-described method embodiments, and to execute the methods provided in the embodiments of the present invention.

[0043] This embodiment provides a non-transitory computer-readable storage medium that stores computer instructions, which cause a computer to execute the method provided in this embodiment of the invention.

[0044] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various storage media capable of storing program code, such as ROM, RAM, magnetic disk, or optical disk.

[0045] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for optimal global planning of long-range missions, characterized in that, The method comprises the following steps: Generate a plurality of initial global plans for long-term tasks, which include robot multi-step jobs and intelligent office processes; Determine the initial global plan with a task completion rate meeting a standard as a high-quality global plan; Optimize and train an initial planner model based on the high-quality global plan to obtain an intermediate planner model; Generate a target global plan based on the intermediate planner model and give a reward score corresponding to the intermediate planner model with a task completion rate meeting a standard; Iteratively train and optimize the intermediate planner model with the reward score as a signal to obtain a final global planner; Generate a plurality of to-be-planned global plans for a to-be-planned long-term task, input the to-be-planned global plans into the final global planner, and output an optimal global plan applied to the long-term task.

2. The method of optimal global planning for long-range missions according to claim 1, wherein, The method for generating a plurality of initial global plans for a long-term task comprises the following steps: Input the long-term task into a pre-trained large language model, so that the pre-trained large language model generates a plurality of initial global plans based on a historical successful execution trajectory corresponding to the task instruction.

3. The method of claim 1, wherein, Before the step of determining the initial global plan with a task completion rate meeting a standard as a high-quality global plan, the method comprises the following steps: Introduce at least two homologous executor agents, and the two homologous executor agents have the same model architecture and pre-training background but different task completion capabilities.

4. The method of claim 3, wherein, The method for determining the initial global plan with a task completion rate meeting a standard as a high-quality global plan comprises the following steps: Use the homologous executor agents to execute the long-term task based on each initial global plan, and determine the initial global plan with a task completion rate meeting a standard as a high-quality global plan.

5. The method of claim 3, wherein, The method for determining the initial global plan with a task completion rate meeting a standard as a high-quality global plan comprises the following steps: Determine whether the initial global plan can simultaneously improve the task completion rates of novice agents and expert agents, and if so, determine the initial global plan as a high-quality global plan.

6. The method of optimal global planning for long-range missions according to claim 3, wherein, The method for giving a reward score corresponding to the intermediate planner model with a task completion rate meeting a standard comprises the following steps: Obtain an actual task completion rate of each homologous executor agent when the target global plan is used; Obtain a benchmark task completion rate of each homologous executor agent when the target global plan is not used; If the actual task completion rate is greater than the benchmark task completion rate, give a reward score corresponding to the intermediate planner model.

7. The method of claim 1, wherein, The optimal global planning method for a long-term task further comprises the following steps: Iteratively train and optimize the intermediate planner model based on the reward score using a reinforcement learning strategy optimization algorithm.

8. A system for optimal global planning of long-range missions, characterized by, The method comprises the following steps: A plan generation module is configured to generate a plurality of initial global plans for long-term tasks, which include robot multi-step jobs and intelligent office processes; A screening module is configured to determine the initial global plan with a task completion rate meeting a standard as a high-quality global plan; An intermediate model training module is configured to optimize and train an initial planner model based on the high-quality global plan to obtain an intermediate planner model; A reward module is configured to generate a target global plan based on the intermediate planner model and give a reward score corresponding to the intermediate planner model with a task completion rate meeting a standard. An end model optimization module is configured to iteratively train and optimize the intermediate planner model based on the reward score to obtain an end global planner. An optimal task acquisition module is configured to generate a plurality of global plans to be planned for a long-range task to be planned, and input the global plans to be planned into the end global planner to output an optimal global plan applied to the long-range task.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor implements the steps of the method of any one of claims 1-7 when executing the computer program.

10. A non-transitory computer readable medium having stored thereon a computer program, characterized in that, The computer program, when executed by the processor, implements the steps of the method of any one of claims 1-7.

Images